Agent design is hard, in no small part because evaluation of agents is hard. As we develop Deep Agents - our open source, model agnostic agent harness - we are constantly faced with many decisions: how to prompt, which tools to include, which middleware to include, etc. As we iterate, we want to have a robust evaluation set to benchmark our decisions against. We recently revamped our evaluation framework, and in this blog we will share how we think about evaluating Deep Agents.

Our recent evaluation work has centered around identifying a set of end-to-end evals. Previously, we had smaller, “unit”-style tests. We still have those, but as we’ve seen agent tasks get longer and longer running, we’ve moved to more end-to-end evals.

In order to accomplish this, we used Harbor：https://www.langchain.com/blog/unified-stack-for-evaluating-agents as an eval runner. Harbor is a popular open source framework for running agent evals, most well known for powering Terminal Bench (a leading coding benchmark).

To use Harbor, you provide three things:

Each dataset：https://www.harborframework.com/docs/core-concepts#dataset has tasks：https://www.harborframework.com/docs/core-concepts#task, which consist of:

Compared to simpler LLM evaluation, there are two main differences:

Today we run three benchmarks, each covering a distinct kind of agent work. Deep Agents is a general purpose harness, so we need to benchmark its capability across multiple different domains.

This is just the start - we will grow these sets of tasks over time.

We use these benchmarks to iterate with confidence. When we are making decisions, benchmarking changes allows to have confidence we are moving in the right direction.

A concrete example of this is how we used it to prepare for a 0.7 release of Deep Agents. As part of this release, we are looking to slim down the harness and remove prompting that may have once been necessary but no longer is. This pays off for someone running Deep Agents: fewer tokens per run and more of the model's attention on the instructions they wrote.

Two concrete changes we are considering: removing the todo-list middleware, and significantly slimming down the system prompt. We are using this benchmark to help decide whether those inclusions are still necessary for the agent harness or not.

LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.