Get notified about new product features, customer updates, and more.
We built 𝜏-bench in 2024：https://sierra.ai/blog/benchmarking-ai-agents to answer a question that felt novel at the time: Can a model act as a reliable customer service agent? That’s table stakes now. The harder question is, who’s building the agent in the first place? Increasingly, it’s the models themselves.
We’ve partnered closely with some of the world’s leading companies to launch their agents. In practice, the work is less like implementing a spec, and more like doing research. Requirements are scattered across handbooks, support, spreadsheets, and the minds of your best frontline reps — so you form a hypothesis, dig up evidence, and build and test to identify which levers actually move performance.
Today we’re open-sourcing hyper-𝜏-bench (published as 𝜏^𝜏-bench), a new long horizon agent evaluation that measures how well models can not only act as an agent, but construct one.
Hyper-𝜏-bench drops a developer agent into a sandboxed workspace with the records of a simulated business, plus a simulated client that it can message at any time. From there, the developer agent does the job end-to-end — it recovers the spec from the evidence, designs the architecture, and turns the business’s actions into tools — until it has a working customer-service agent. The client’s REST API may be subtly defective, so part of the job is figuring out whether a bug is in the spec or in the code. The finished agent has to serve from a fixed menu of models, within a cost budget per conversation. Once it’s handed off, we deploy it against simulated production traffic using fully verifiable 𝜏-bench-style tests the developer never saw while building.
Working alone, our best configuration — Claude Opus 5 (max reasoning) running in Claude Code — passes just 23.9% of the held-out evaluation tasks. Paired with an engineer with deep context, the same class of model reaches 82.2% on the same tasks.
We read through developer trajectories to see where their builds lost ground. Five patterns stood out:
Hyper-𝜏-bench sits alongside benchmarks like MLE-bench and RE-Bench, which measure research capability: designing experiments, weighing tradeoffs, and iterating toward a better system. Building an agent demands all of that — and adds a few problems of its own. The spec has to be recovered from documents and people. And because the system being built is an AI itself, the only way to know if a design works is to run it and read what it says to real users, who the developer never sees while building.
𝜏-bench asked whether models could be good agents. Hyper-𝜏-bench asks whether they can build them. As agents take on more of that work themselves, we’ll keep using hyper-𝜏-bench to track how well they’re doing it.
Paper：https://arxiv.org/abs/2609.04611 | Codebase：https://github.com/sierra-research/hyper-tau-bench | Leaderboard：https://sierra-research.github.io/hyper-tau-bench/
Find out how Sierra can help you deliver better outcomes with AI.
𝜏³-Bench is here. We've expanded agent evaluation to two new frontiers: knowledge retrieval and voice.
𝜏²-bench challenges AI agents not just to reason and act, but to coordinate, guide, and assist a user in achieving a shared objective. This leap from solo operation to co-ownership of a task pushes agents into a much more demanding space. And, critically, it reflects the kinds of tasks AI agents are increasingly being asked to perform in the real world.
We explore how Sierra’s 𝜏-bench is shaping the development and evaluation of agents—from academic research to industry applications and next-generation development.
Scan to open this article