MirrorCode：AI 能独立完成的最大规模软件项目基准测试

近年来，AI在软件工程基准测试上取得了快速进展。然而，大多数此类基准测试往往专注于较短的任务，例如修复错误或实现单个功能。MirrorCode 是我们与 METR 共同开发的基准测试，用于测试 AI 模型在长周期编码任务中的表现。在 MirrorCode 任务中，AI 模型的任务是从头重实现整个程序，不得访问原始源代码。AI 生成的解决方案必须在端到端测试中（包括保留测试）完全匹配原程序的输出。MirrorCode 的 25 个目标程序涵盖不同的计算领域：Unix 工具、数据序列化与查询工具、生物信息学、解释器、静态分析、密码学及压缩。

关键是，我们提供了足够大的推理预算，以便对 MirrorCode 任务进行认真尝试。许多现有的软件工程基准测试将推理开支限制在约 1–10 美元，即使任务对人类来说需要几周时间才能完成。例如，MirrorCode 中最大的一项任务单次运行成本为 2,600 美元，AI 连续工作了 19 天，没有人工干预。

从头重实现整个程序对人类软件工程师来说极具挑战。我们相信，即使没有 AI 的帮助，人类工程师也需要数月时间才能完成最复杂的 MirrorCode 任务。然而，MirrorCode 任务也是可行的；我们知道这类任务的信息量足够，使其具有公平性。

我们对 AI 模型进行沙箱管理，要求它们在进行工作时无法访问互联网、无法访问原始代码库，且无法在任务中作弊。还有一些模型在开发代码时永远不会看到的端到端测试，因此它们不能简单地创建查找表来模拟原程序的输出。

尽管困难重重，AI 已经能够解决长周期的 MirrorCode 任务。例如，Claude Opus 4.7 成功重实现了 gotree：一个包含约 16,000 行 Go 代码和 40 多条命令的生物信息学工具包。我们认为，此任务没有 AI 辅助的人类工程师需要 2–17 周的时间，而 Opus 4.7 在 14 小时内完成，成本为 251 美元。

这些结果的一个重要警告是数据污染。因为MirrorCode任务涉及重新实现开源程序，AI模型在预训练中很可能已经见过原始代码库。这可能导致基准测试结果被高估。然而，AI成功重新实现了一些通过我们记忆筛选的目标程序，而在记忆筛选显示有记忆证据的程序上失败了。这表明结果并非完全由记忆主导，但我们不能排除记忆对AI性能有贡献的可能性。总体而言，我们预计MirrorCode测量的能力会推广到未见过的代码库。我们在论文中对此进行了进一步讨论，并提供了更多结果和基准构建细节：https://arxiv.org/pdf/2606.30182。

MirrorCode并未完全解决。在我们定期更新的排行榜中，我们报告MirrorCode (ML, +Private, 2L)。这意味着我们运行来自中大型类别的15个目标程序，并丢弃小型类别的程序。每个目标程序在两种实现语言中进行评估（通常是Go和Ada），共30个任务。我们对每个任务运行三次，每次尝试的预算为100亿个token和7天。

我们发布了我们的框架和25个MirrorCode目标程序中的22个（涵盖六种支持的编程语言，总计132个任务实例）作为开源：https://github.com/epoch-research/MirrorCode，其他三个目标保留作为私有测试集。

这项工作与METR共同开发，并由METR资助。MirrorCode的作者是Tom Adamczewski、David Owen 和 David Rein。Florian Brand、Giles Edkins、Allen Hart和Daniel O’Connell贡献了额外的目标程序。Rasmus Faber-Espensen在关键基础设施改进和工程建议方面提供了支持。

得分最高的AI gotree实现通过了2000/2001个测试，但在一个针对操作日期注释的冷门命令的边缘案例测试中失败。因此，它们并未严格达到100%的任务完成度，但我们认为重新实现几乎完美，涵盖了基本上所有的功能范围。

请参阅论文中“建议命名规范”的定义。“+Private”表示包含私有测试集：这里指 private_M 和 private_L，即中等和大型类别中的私有目标。在 2L 映射下，目标程序通常使用 Go 和 Ada（一种主流语言和一种低资源语言），有两个例外。这些分数与论文中的分数不可直接比较，论文评估了所有 25 个目标程序，在 Small 和 Medium 类别中使用了所有 6 种代理实现语言，并为尝试分配了 10 亿个 token 的预算（大型目标除外），且没有时间限制。：#user-content-fnref-2

Epoch AI 的作品可以自由使用、分发和复制，但必须在署名的情况下遵循知识共享署名许可协议：https://creativecommons.org/licenses/by/4.0/。

在收件箱中获取 Epoch AI 的最新信息

有问题吗？发现了错误？请告诉我们。

如果您希望收到回复，请提供您的姓名和电子邮件地址。

MirrorCode 是 Epoch AI 的长期编码基准测试：AI 可以从头到尾重新实现整个程序，而无需访问原始源代码。

AI has made rapid progress on software engineering benchmarks in the past few years. However, most such benchmarks tend to focus on shorter tasks like fixing bugs or implementing individual features. MirrorCode is our benchmark, co-developed with METR, to test AI models on long-horizon coding tasks. In a MirrorCode task, AI models are tasked with reimplementing an entire program end-to-end, without access to the original source code. AI-generated solutions must match the original program’s output exactly on end-to-end tests, including held-out tests. MirrorCode’s 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression.

Crucially, we provide a large enough inference budget to make a serious attempt at MirrorCode tasks. Many existing software engineering benchmarks limit inference spending to around $1–10, even when the task would take weeks for a human to complete. For example, one of the largest MirrorCode tasks cost $2,600 for a single run and involved AI working for 19 days without human intervention.

Reimplementing entire programs is extremely challenging for human software engineers. We believe a human engineer without AI would take months to solve the most complex MirrorCode tasks. However, MirrorCode tasks are also feasible; we know that there is enough information for the tasks to be fair.

We sandbox AI models, requiring them to conduct their work without access to the internet, without access to the original codebase, and with no way to cheat on the task. There are end-to-end tests that models never see while developing their code, so they cannot simply create a lookup table to mimic the original program's outputs.

AI can already solve long-horizon MirrorCode tasks, despite their difficulty. For example, Claude Opus 4.7 reimplemented gotree: a bioinformatics toolkit with ~16,000 lines of Go and 40+ commands. 1：#user-content-fn-1 We believe this same task would take a human engineer without AI assistance 2–17 weeks. Opus 4.7 solved it in 14 hours, costing $251.

One important caveat to these results is data contamination. Because MirrorCode tasks involve reimplementing open-source programs, AI models are likely to have seen the original codebases in pretraining. This might lead to inflated performance on the benchmark. However, AI successfully reimplemented several target programs that passed our memorization screen, and failed to reimplement programs where the screen showed evidence of memorization. This suggests that the results were not dominated by memorization, but we cannot rule out the possibility that memorization contributes to AI performance. Overall, we expect that the capabilities measured by MirrorCode would generalize to an unseen codebase. We discuss this further, along with more results and details on benchmark construction, in the paper：https://arxiv.org/pdf/2606.30182.

MirrorCode is not fully solved. For our regularly updated leaderboard, we report MirrorCode (ML, +Private, 2L) . This means we run the 15 target programs from the Medium and Large buckets, and drop the Small bucket. Each target program is evaluated in two implementation languages (generally Go and Ada) giving 30 tasks. We run each task three times, with a budget of 10 billion tokens and 7 days per attempt. 2：#user-content-fn-2

We release our scaffold and 22 of the 25 MirrorCode target programs (totaling 132 task instances across the six supported programming languages) as open-source：https://github.com/epoch-research/MirrorCode, with the other three targets held out as a private test set.

This work was co-developed with METR and supported by a grant from METR. The authors of MirrorCode are Tom Adamczewski, David Owen, and David Rein. Florian Brand, Giles Edkins, Allen Hart, and Daniel O’Connell contributed additional target programs. Rasmus Faber-Espensen made crucial infrastructure improvements and gave advice on engineering

The best-scoring AI gotree implementations passed 2000/2001 tests, but failed a single edge-case test for a niche command to manipulate date annotations. Consequently, they do not strictly solve the task to 100% completion, but we consider the reimplementation near-perfect, covering essentially all scoped functionality. ：#user-content-fnref-1

See definitions in the “Suggested naming conventions” of the paper. “+Private” indicates that the private test set is included: here, private_M and private_L, the private targets in the Medium and Large buckets. Under the 2L mapping, target programs generally use Go and Ada (one mainstream language and one low-resource language), with two exceptions. These scores are not directly comparable with the paper, which evaluated all 25 target programs, used all 6 agent implementation languages on the Small and Medium buckets, and gave attempts a budget of 1 billion tokens except on Large targets, with no time limit. ：#user-content-fnref-2

Epoch AI’s work is free to use, distribute, and reproduce provided the source and authors are credited under the Creative Commons Attribution license：https://creativecommons.org/licenses/by/4.0/.

Get the latest from Epoch AI in your inbox

Have a question? Noticed something wrong? Let us know.

If you would like a reply, please include your name and email address.

MirrorCode is Epoch AI's benchmark for long-horizon coding: AI can reimplement entire programs end-to-end, with no access to the original source code.