代理设计很难，很大程度上是因为代理的评估也很难。当我们开发 Deep Agents——我们的开源、模型无关的代理框架——时，我们不断面临许多决策：如何提示、包含哪些工具、包含哪些中间件等。在迭代过程中，我们希望拥有一套稳健的评估集合来衡量我们的决策。我们最近重新设计了评估框架，在这篇博客中，我们将分享我们如何看待 Deep Agents 的评估。
我们最近的评估工作集中在确定一组端到端评估。之前，我们有较小的“单元”式测试。我们仍然保留这些测试，但随着代理任务运行时间越来越长，我们转向了更多的端到端评估。
为此，我们使用了 Harbor：https://www.langchain.com/blog/unified-stack-for-evaluating-agents 作为评估运行器。Harbor 是一个流行的开源框架，用于运行代理评估，以支持 Terminal Bench（一个领先的编码基准）而闻名。
使用 Harbor 时，你需要提供三个内容：
每个数据集（https://www.harborframework.com/docs/core-concepts#dataset）都有任务（https://www.harborframework.com/docs/core-concepts#task），其组成部分包括：
与简单的 LLM 评估相比，有两个主要区别：
目前我们运行三个基准测试，每个测试涵盖不同类型的代理工作。Deep Agents 是一个通用框架，因此我们需要在多个不同领域中基准其能力。
这只是一个开始——我们会随着时间增长这些任务集合。
我们使用这些基准测试来自信地进行迭代。当我们做出决策时，基准测试的变化让我们有信心朝着正确的方向前进。
一个具体的例子是我们如何使用它为 Deep Agents 0.7 发布做准备。作为该版本的一部分，我们希望精简框架，并移除曾经必要但现在不再需要的提示。这对使用 Deep Agents 的人来说是有好处的：每次运行的 token 数更少，模型可以将更多注意力集中在他们编写的指令上。
我们正在考虑的两个具体变化：移除待办事项中间件，以及显著精简系统提示。我们正在使用这个基准来帮助决定这些包含项对于代理工具是否仍然必要。
LangSmith，我们的代理工程平台，可以帮助开发者调试每个代理决策、评估变更，并一键部署。
Agent design is hard, in no small part because evaluation of agents is hard. As we develop Deep Agents - our open source, model agnostic agent harness - we are constantly faced with many decisions: how to prompt, which tools to include, which middleware to include, etc. As we iterate, we want to have a robust evaluation set to benchmark our decisions against. We recently revamped our evaluation framework, and in this blog we will share how we think about evaluating Deep Agents.
Our recent evaluation work has centered around identifying a set of end-to-end evals. Previously, we had smaller, “unit”-style tests. We still have those, but as we’ve seen agent tasks get longer and longer running, we’ve moved to more end-to-end evals.
In order to accomplish this, we used Harbor：https://www.langchain.com/blog/unified-stack-for-evaluating-agents as an eval runner. Harbor is a popular open source framework for running agent evals, most well known for powering Terminal Bench (a leading coding benchmark).
To use Harbor, you provide three things:
Each dataset：https://www.harborframework.com/docs/core-concepts#dataset has tasks：https://www.harborframework.com/docs/core-concepts#task, which consist of:
Compared to simpler LLM evaluation, there are two main differences:
Today we run three benchmarks, each covering a distinct kind of agent work. Deep Agents is a general purpose harness, so we need to benchmark its capability across multiple different domains.
This is just the start - we will grow these sets of tasks over time.
We use these benchmarks to iterate with confidence. When we are making decisions, benchmarking changes allows to have confidence we are moving in the right direction.
A concrete example of this is how we used it to prepare for a 0.7 release of Deep Agents. As part of this release, we are looking to slim down the harness and remove prompting that may have once been necessary but no longer is. This pays off for someone running Deep Agents: fewer tokens per run and more of the model's attention on the instructions they wrote.
Two concrete changes we are considering: removing the todo-list middleware, and significantly slimming down the system prompt. We are using this benchmark to help decide whether those inclusions are still necessary for the agent harness or not.
LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.
扫码直达文章详情