How we build an autonomous SRE Agent for Kubernetes Deployments

I'm a Deployed Engineer at LangChain, and a good part of my job lives on top of Kubernetes. I maintain our internal self-hosted cluster (also where new self-hosted features land first, so our team can test them as they arrive), and I help customers stand up and upgrade their own self-hosted environments. Doing this requires a mental model of a live cluster deep enough that when ours goes down I can read the architecture, find the fault, and fix it. Doing that continuously, on top of a full-time job, is exhausting, and it isn't unique to me. Anyone who works on infrastructure knows these pains.

So we built an autonomous SRE Agent to reduce time to triage and time to remediation. The goal was to improve infrastructure reliability and reduce cognitive load on the team. We wanted it to triage Kubernetes health, propose fixes, and pull in a human only when a cluster or infrastructure changes need to happen. This post covers why, how we built it, why LangSmith makes it trustworthy, and the benefits we've seen.

Kubernetes emits a firehose of signals (pod phases, restart counts, HPA (Horizontal Pod Autoscaling) states, node conditions, warning events, deployment readiness, across dozens of namespaces) and almost no synthesis. On-call engineers use these signals to answer three questions: Is anything broken now (a crash loop, an OOM kill, zero ready endpoints); Is anything about to break (an HPA pinned at max, a single-replica service, :latest image tags); What do we do about it? Answering these questions well takes judgment, so it falls to infrastructure engineers or subject matter experts, but 90% of this work is mechanical triage that mostly comes back clean. This is the toil that burns people out and trains them to inadvertently skim past important alerts.

Proactive monitoring. A scheduler checks health every N minutes without waking the full agent. It collects raw cluster state through the Kubernetes Python client (zero LLM tokens), then makes one Claude Haiku call with forced tool-use to produce a structured health report that lands in Slack, sorted by severity.

On-demand investigation. When an issue needs diagnosis, the orchestrator fans out to specialized subagents in parallel: pod-inspector, scaling-analyzer, performance-analyzer, log-analyzer, security-auditor, reliability-auditor, and more. Each reads the cluster independently before it synthesizes one prioritized report.

The agent can read the entire cluster but change nothing on its own. Every write (scaling a deployment, restarting a rollout, patching an HPA) lives inside a single change-executor subagent, and each write tool is gated by a human-in-the-loop (HITL) interrupt. The agent proposes a remediation, a person approves, rejects, or edits, right from a Slack message. Read is autonomous and writing is always gated through HITL. It’s enforced structurally and mirrored by in-cluster RBAC (cluster-wide read, tightly scoped write).

Each choice below was a fork where the obvious path and the right path diverged.

The through-line in our architecture is to keep things as simple and cost-effective as possible. We spend tokens, model power, and increase network surface only where it improves agent outcomes for the infrastructure we’re managing. That's what makes the agent cheap enough to run every few minutes and safe enough to point at production.

Every decision in Part 2 runs as a LangSmith trace: the scheduled Haiku check, each subagent investigation, every read and proposed write. The Deep Agents and LangGraph pieces trace automatically; the scheduler's direct Anthropic calls are wrapped with @traceable.

Those labeled runs form the backbone of our evaluations. A misclassified pod or a missed OOM gets promoted into a LangSmith dataset with the correct answer attached, and every prompt or model tweak then runs against it with LLM-as-judge and code-based evaluators, so a change that regresses shows up as a red number and doesn't merge. The single-replica false positive above becomes a permanent test case for our eval suite.

The improvement loop above still relies on a human noticing undesirable behavior, finding the trace, and promoting it to a dataset. It’s time intensive for whoever maintains the agent. LangSmith Engine：https://docs.langchain.com/langsmith/engine-overview automates this manual work. It's like a proactive agent engineer that watches our tracing project in three stages.

Engine has grouped our traces into open issues we had not filed ourselves. One example is that the scheduled health check was collecting no utilization data at all. The collector queried nodes, pods, warning events, HPAs, and deployments, and the analysis step was a single forced-tool call with no tools available, so the model couldn’t fetch what the collector skipped. Every hourly report raised a capacity question it structurally could not answer and handed it back to us as a recommended action like "check pod CPU/memory metrics". The kubectl_top_pods and kubectl_top_nodes capability was already in the repo for the interactive agent and had never been wired into the scheduled path. Engine proposed wiring pod and node metrics into the collector, reusing the existing unit parsing rather than inventing new logic, and scoped the change to the collector so the analysis prompt and the zero-token property stay untouched. It arrived as a pull request that was reviewed, added as an example to our dataset, tested against our dataset to prevent regressions and prove the fix worked, and then merged.

We will continue to use SRE Agent internally and have begun rolling this out to some current LangSmith customers. We’re actively working to develop it and expand its capabilities for Kubernetes and other parts of the stack. Some of the next improvements will be making the state durable for HITL, and making the monitoring loop stateful so it has memory of the recent incidents reported. It is open source and available here. https://github.com/langchain-samples/sre-agent：https://github.com/langchain-samples/sre-agent . Feel free to try it out, contribute, or provide feedback.

LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.