我们如何为 Kubernetes 部署构建一个自主 SRE 代理
我是 LangChain 的一名部署工程师，我工作的很大一部分都依赖于 Kubernetes。我维护我们内部的自建集群（也是新自建功能首先落地的地方，这样我们的团队可以在功能上线时进行测试），并帮助客户启动和升级他们自己的自建环境。做这些工作需要对活跃集群有足够深入的心理模型，以至于当我们的集群宕机时，我可以阅读架构，找出故障并修复它。在全职工作的基础上持续做这种事情是非常耗费精力的，而且这并不只是我一个人的经历。任何从事基础设施工作的人都知道这些痛苦。
所以我们构建了一个自主 SRE 代理，以减少排查时间和修复时间。目标是提高基础设施的可靠性并减轻团队的认知负担。我们希望它能够排查 Kubernetes 的健康状况、提出修复方案，并且仅在集群或基础设施需要更改时才引入人工干预。这篇文章涵盖了原因、我们如何构建它、为什么 LangSmith 让它值得信赖，以及我们看到的好处。
Kubernetes 发出大量信号（Pod 阶段、重启计数、HPA（水平 Pod 自动扩缩）状态、节点状态、警告事件、部署就绪状态，覆盖数十个命名空间），几乎没有任何综合信息。值班工程师使用这些信号来回答三个问题：现在是否有故障（崩溃循环、OOM 杀死、零就绪端点）；是否有即将发生的故障（HPA 锁定在最大值、单副本服务、:latest 镜像标签）；我们应该如何处理？有效回答这些问题需要判断能力，因此责任通常落在基础设施工程师或主题专家上，但90%的工作是机械性的排查，大部分情况下结果都是良好的。这种重复繁琐的工作容易让人精疲力竭，也会训练他们无意中忽略重要警报。
主动监控。调度器每N分钟检查一次健康状况，而无需唤醒完整代理。它通过 Kubernetes Python 客户端收集原始集群状态（不消耗任何 LLM 令牌），然后通过强制使用工具的一次 Claude Haiku 调用生成结构化健康报告，并按严重性排序发送到 Slack。
按需调查。当需要诊断问题时，协调器会并行扩展到专门的子代理：pod-inspector、scaling-analyzer、performance-analyzer、log-analyzer、security-auditor、reliability-auditor 等。每个子代理在综合一个优先报告之前会独立读取集群信息。
代理可以读取整个集群，但自身不会更改任何内容。每一次写操作（扩展部署、重启滚动更新、修补 HPA）都存在于单独的 change-executor 子代理中，并且每个写工具都由人工干预（HITL）控制。代理提出修复方案，由人工直接在 Slack 消息中批准、拒绝或编辑。读取是自主的，而写入始终通过 HITL 控制执行。这在结构上得到强制执行，并由集群内的 RBAC 反映（集群范围内读取，写入权限严格限定）。
下面的每一个选择都是一个分岔点，在明显路径与正确路径之间分开。
我们架构中的主线是尽量保持事情简单且高效。我们只在能改善我们所管理基础设施的代理效果的地方消耗代币、模型算力并增加网络表面。这使得代理足够便宜，可以每隔几分钟运行一次，并足够安全，可指向生产环境。
第 2 部分的每一个决策都以 LangSmith 跟踪运行：计划的 Haiku 检查、每个子代理调查、每一次读取和提出的写入。Deep Agents 和 LangGraph 部分自动跟踪；调度器的直接 Anthropic 调用使用 @traceable 包装。
这些标记的运行构成了我们评估的骨架。被错误分类的 pod 或错过的 OOM 会被提升为 LangSmith 数据集并附上正确答案，然后每条提示或模型调整都会在其上运行，使用 LLM 作为评判者和基于代码的评估器，这样回退的改动会显示为红色数字且不会合并。以上单副本的误报会成为我们评估套件的永久测试用例。
上面的改进循环仍然依赖于人类发现不良行为、找到追踪记录并将其提升为数据集。这对维护代理的人来说是耗时的工作。LangSmith 引擎：https://docs.langchain.com/langsmith/engine-overview 可以自动完成这项手动工作。它就像一个主动的代理工程师，在三个阶段监视我们的追踪项目。
引擎已经将我们的追踪记录分组为我们自己尚未提交的未解决问题。一个例子是，计划的健康检查根本没有收集任何利用率数据。收集器查询了节点、Pods、警告事件、HPAs 和部署，而分析步骤只是一次单一的强制工具调用，而没有可用工具，因此模型无法获取收集器跳过的内容。每小时报告都会提出一个结构上无法回答的容量问题，并以“检查 Pod 的 CPU/内存指标”之类的建议操作交回给我们。kubectl_top_pods 和 kubectl_top_nodes 功能已经在交互式代理的代码库中，但从未接入计划路径。引擎提出将 Pod 和节点指标接入收集器，重用现有的单位解析，而不是发明新的逻辑，并将变更范围限定在收集器中，以便分析提示和零令牌属性保持不变。它以拉取请求的形式到来，经过审核，添加为我们数据集的示例，针对我们的数据集进行测试以防止回归并验证修复有效，然后合并。
我们将继续在内部使用 SRE Agent，并已开始向一些现有的 LangSmith 客户推广。我们正在积极开发它，并扩展它在 Kubernetes 和栈的其他部分的能力。下一步的一些改进将是使状态对于 HITL 耐久化，并使监控循环具有状态性，以便它记忆最近报告的事件。它是开源的，可在此获取：https://github.com/langchain-samples/sre-agent：https://github.com/langchain-samples/sre-agent。欢迎尝试、贡献或提供反馈。
LangSmith，我们的代理工程平台，帮助开发者调试每一个代理决策、评估变更，并一键部署。
How we build an autonomous SRE Agent for Kubernetes Deployments
I'm a Deployed Engineer at LangChain, and a good part of my job lives on top of Kubernetes. I maintain our internal self-hosted cluster (also where new self-hosted features land first, so our team can test them as they arrive), and I help customers stand up and upgrade their own self-hosted environments. Doing this requires a mental model of a live cluster deep enough that when ours goes down I can read the architecture, find the fault, and fix it. Doing that continuously, on top of a full-time job, is exhausting, and it isn't unique to me. Anyone who works on infrastructure knows these pains.
So we built an autonomous SRE Agent to reduce time to triage and time to remediation. The goal was to improve infrastructure reliability and reduce cognitive load on the team. We wanted it to triage Kubernetes health, propose fixes, and pull in a human only when a cluster or infrastructure changes need to happen. This post covers why, how we built it, why LangSmith makes it trustworthy, and the benefits we've seen.
Kubernetes emits a firehose of signals (pod phases, restart counts, HPA (Horizontal Pod Autoscaling) states, node conditions, warning events, deployment readiness, across dozens of namespaces) and almost no synthesis. On-call engineers use these signals to answer three questions: Is anything broken now (a crash loop, an OOM kill, zero ready endpoints); Is anything about to break (an HPA pinned at max, a single-replica service, :latest image tags); What do we do about it? Answering these questions well takes judgment, so it falls to infrastructure engineers or subject matter experts, but 90% of this work is mechanical triage that mostly comes back clean. This is the toil that burns people out and trains them to inadvertently skim past important alerts.
Proactive monitoring. A scheduler checks health every N minutes without waking the full agent. It collects raw cluster state through the Kubernetes Python client (zero LLM tokens), then makes one Claude Haiku call with forced tool-use to produce a structured health report that lands in Slack, sorted by severity.
On-demand investigation. When an issue needs diagnosis, the orchestrator fans out to specialized subagents in parallel: pod-inspector, scaling-analyzer, performance-analyzer, log-analyzer, security-auditor, reliability-auditor, and more. Each reads the cluster independently before it synthesizes one prioritized report.
The agent can read the entire cluster but change nothing on its own. Every write (scaling a deployment, restarting a rollout, patching an HPA) lives inside a single change-executor subagent, and each write tool is gated by a human-in-the-loop (HITL) interrupt. The agent proposes a remediation, a person approves, rejects, or edits, right from a Slack message. Read is autonomous and writing is always gated through HITL. It’s enforced structurally and mirrored by in-cluster RBAC (cluster-wide read, tightly scoped write).
Each choice below was a fork where the obvious path and the right path diverged.
The through-line in our architecture is to keep things as simple and cost-effective as possible. We spend tokens, model power, and increase network surface only where it improves agent outcomes for the infrastructure we’re managing. That's what makes the agent cheap enough to run every few minutes and safe enough to point at production.
Every decision in Part 2 runs as a LangSmith trace: the scheduled Haiku check, each subagent investigation, every read and proposed write. The Deep Agents and LangGraph pieces trace automatically; the scheduler's direct Anthropic calls are wrapped with @traceable.
Those labeled runs form the backbone of our evaluations. A misclassified pod or a missed OOM gets promoted into a LangSmith dataset with the correct answer attached, and every prompt or model tweak then runs against it with LLM-as-judge and code-based evaluators, so a change that regresses shows up as a red number and doesn't merge. The single-replica false positive above becomes a permanent test case for our eval suite.
The improvement loop above still relies on a human noticing undesirable behavior, finding the trace, and promoting it to a dataset. It’s time intensive for whoever maintains the agent. LangSmith Engine：https://docs.langchain.com/langsmith/engine-overview automates this manual work. It's like a proactive agent engineer that watches our tracing project in three stages.
Engine has grouped our traces into open issues we had not filed ourselves. One example is that the scheduled health check was collecting no utilization data at all. The collector queried nodes, pods, warning events, HPAs, and deployments, and the analysis step was a single forced-tool call with no tools available, so the model couldn’t fetch what the collector skipped. Every hourly report raised a capacity question it structurally could not answer and handed it back to us as a recommended action like "check pod CPU/memory metrics". The kubectl_top_pods and kubectl_top_nodes capability was already in the repo for the interactive agent and had never been wired into the scheduled path. Engine proposed wiring pod and node metrics into the collector, reusing the existing unit parsing rather than inventing new logic, and scoped the change to the collector so the analysis prompt and the zero-token property stay untouched. It arrived as a pull request that was reviewed, added as an example to our dataset, tested against our dataset to prevent regressions and prove the fix worked, and then merged.
We will continue to use SRE Agent internally and have begun rolling this out to some current LangSmith customers. We’re actively working to develop it and expand its capabilities for Kubernetes and other parts of the stack. Some of the next improvements will be making the state durable for HITL, and making the monitoring loop stateful so it has memory of the recent incidents reported. It is open source and available here. https://github.com/langchain-samples/sre-agent：https://github.com/langchain-samples/sre-agent . Feel free to try it out, contribute, or provide feedback.
LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.
扫码直达文章详情