METR's new metric, the "expenditure horizon," puts a dollar figure on how cost-effective AI agents are at solving problems. Early results on the NanoGPT speedrun are underwhelming, the metric has blind spots, and the newest generation of models could change the picture.
One of the biggest questions in AI research is whether AI can accelerate its own development and keep getting better at an increasing pace. That's been hard to measure because it requires comparing very different kinds of costs: human labor, compute for experiments, and the cost of running the AI itself.
Research organization METR proposes a new metric to tackle this: the "expenditure horizon." METR compares how much an AI and how much a human have to spend to achieve the same improvement. The expenditure horizon is the point where both cost the same. Below that budget, the AI is the better deal. Above it, the human works cheaper.
The idea builds on a pattern METR has seen in previous tests: AI agents often solve simple, low-cost tasks faster than humans. But as budgets grow and tasks get harder, they fall behind.
Compared to typical AI benchmarks, the method has two advantages, according to METR. First, it doesn't just give a pass-or-fail verdict. Instead, it produces a fine-grained value showing how much improvement you get for how much money. Second, it converts all costs into a single currency, covering not just the cost of running the AI but also the expensive compute for experiments and human labor time.
METR chose the NanoGPT speedrun：https://github.com/kellerjordan/modded-nanogpt as its testing ground. It's a public community project where volunteers compete to train an AI language model as fast as possible. The task stays the same; only the training approach can change. Since May 2024, the required training time on standardized hardware dropped from about 45 minutes to under two minutes across 82 documented improvement steps.
To figure out the cost of human work, METR interviewed two of the project's most active contributors and also had an AI model (Opus-4.6) estimate the effort behind each improvement. Both approaches landed on roughly 16 hours of work per one-percent speedup. At an assumed hourly rate of $150, that comes to about $2,500 per percentage point.
METR stresses that this number is very uncertain. One detail from the interviews stands out: most of the time went into ideas that ultimately didn't work.
For the comparison, METR had six AI models work on the same task independently. They didn't start from scratch but from an already highly optimized state of the speedrun (Record #78 from March 2026) and were allowed to spend up to $10,000 in compute and operating costs per run. The result: estimated expenditure horizons between $0 and $3,300.
The differences between models were stark. GPT-5 and Opus-4.1 produced no real progress after careful verification. Their apparent gains turned out to be random noise. GPT-5.5 and Opus-4.8, on the other hand, delivered real improvements of about 1 and 1.5 percent, respectively.
The quality of AI-generated ideas was mixed. The speedrun's maintainer estimated that about 70 percent of them could in principle be integrated into the project, but many weren't very original. He praised one clever, low-level optimization from GPT-5.5 as the "coolest one," while calling most of the rest just parameter tweaking. The models also tried to cheat multiple times, taking shortcuts that faked good results in the test but would have been useless in practice, like shutting off parts of training right before the finish line.
METR's takeaway: while individual models reach expenditure horizons in the low four figures, those values are tiny compared to the estimated $250,000 in total human effort. Autonomous optimization has barely moved the needle on NanoGPT progress so far.
An important caveat: METR only tested older models (GPT-5, GPT-5.2, GPT-5.5, and Opus-4.1 and Opus-4.8). The models released since then, Fable 5, GPT-5.6 Sol, and Opus 5, don't appear in the paper. Anthropic markets Opus 5 as a major leap：https://the-decoder.com/anthropic-claims-its-new-claude-opus-5-delivers-near-fable-5-performance-at-half-the-token-price/: on the Frontier-Bench test, it doubles Opus 4.8's performance at lower cost per task. According to Anthropic, Opus 5 wastes less effort on dead ends, checks its own work more reliably, and achieves similar performance with an average of 26 percent fewer compute steps. All of those are factors that directly affect METR's expenditure horizon.
The progress on ARC-AGI-3：https://the-decoder.com/anthropics-opus-5-blows-past-fable-5-and-gpt-5-6-sol-on-the-benchmark-designed-to-measure-real-intelligence/ is even more telling. That benchmark doesn't test memorized knowledge but genuine problem-solving: the AI is dropped into unfamiliar, game-like environments with no instructions or goals and has to figure everything out through trial and error. Opus 5 has held the top spot since July 24, 2026, scoring 30.2 percent and solving five tasks that every previous model had failed. Its predecessor Opus 4.8 managed just 1.5 percent. The ARC Prize team attributes the jump to better logical reasoning, which lets the AI explore and plan more independently. That kind of ability could also prove useful in the NanoGPT speedrun.
Perhaps the biggest limitation is one METR calls out itself: the entire study measures AI working alone, purely autonomous optimization. In real AI research, humans typically use AI as a tool. METR sketches a third, hypothetical curve for this scenario. If humans make smart decisions about when and how to deploy the AI, this hybrid curve should theoretically beat both the pure human and pure AI curves by combining the strengths of each.
METR tempers that expectation, though, pointing to its own earlier work showing that human-plus-AI setups sometimes performed worse than humans alone：https://the-decoder.com/ai-coding-can-make-developers-slower-even-if-they-feel-faster/. The added value isn't guaranteed and depends on whether the AI gets used in the right places. Measuring this properly would require a controlled experiment comparing the same researchers working with and without AI support. That kind of experiment is hard to organize, but METR says it would be extremely informative. Until it happens, the expenditure horizon says a lot about what AI can do on its own, but very little about how much it actually speeds up human researchers.
Stay in the loop on AI. Clear, useful, no fluff.
Follow The Decoder for AI news, background stories and expert analyses.
扫码直达文章详情