结果显示，medium 努力度下动态语言表现更好，ultra 下静态语言反而更优，且此前评测存在测试路径错误等缺陷。 作者认为，琐碎任务上的性能无法推广到更大问题。
这篇被广泛引用的帖子：https：//martinalderson.com/posts/which-programming-languages-are-most-token-efficient/（我经常看到有人引用）建议动态语言和/或更简洁表示事物的语言更高效。它被引用得足够多，以至于LLM的搜索结果都一致。例如，当我搜索“动态语言代币成本”（无引号）时，谷歌的AI摘要开头是这样
动态类型语言通常比传统静态类型语言的LLM令牌成本更低，因为省略显式类型声明使代码更紧凑。
谷歌的人工智能也引用了同样的帖子，指出一些简明的动态语言的代币成本大约是静态语言如Rust、Go、C++等的一半到三分之一。作者说
C（我比较过的最低token效率的语言）和Clojure（最高效的语言）之间有2.6倍的显著差距。
然后他们后来尝试了 J，说
它以平均仅70个代币的优势占主导地位，几乎是Clojure（109个代币）的一半。数组语言在避免特殊符号集时可以极高地节省令牌。如果代币效率成为关键驱动力，这或许是语言发展的一个非常有趣的方式。
我找到的另一个动态语言与静态语言令牌比较是这个：https：//github.com/mame/ai-coding-lang-bench，支持同样的结论。如果你想把这本书当作基准测试、评估和实验设计系列练习的第8部分：/练习-7/，可以点击链接，思考评估问题再继续阅读。
在没有运行我们自己的评估之前，第一个实验的问题之一是这些问题过于简单，从上面的引用中可以看出；一个在 J 语言中可以用 70 个符号解决，在 Clojure 中用 109 个符号解决的问题根本算不上问题（作者使用了 Rosetta Code）。正如我们在查看 caveman 模式与我们自己评估的其他评估时看到的：/ai-coding/，对于大多数工作只是打印答案的简单问题，与那些实际上需要一些“真正工作”的稍微复杂问题相比，结果可能会大不相同；当你开始关注需要的不仅仅是几个符号的复杂问题时，caveman 模式声称的大幅提升和在复现中显示的效果就会消失。一般来说，在简单任务上的表现无法推广。
第二个链接中的问题稍微微妙一些，所以我们将大部分内容放在附录中，但它们包括一些问题，比如其中一个测试执行了错误的路径（该路径不存在），导致测试失败。稍后的某个代理将不存在的路径链接到自己的可执行文件，这个方法在该情况下有效，但也导致之后的每个测试都运行该代理的可执行文件，而不是正确的可执行文件。作者试图从 Rust 出现一些失败得出结论，但这仅意味着 Rust 的评分在 Go 代理将所有评分在该错误测试上链接到 Go 可执行文件之前进行。
与其依赖这些评估，我们可以尝试运行一些我们自己的评估。正如我们从这些评估以及在我们上一次关于评估的练习中讨论的评估中看到的：/exercise-7/，很容易做出一个评估，而这个评估并没有如评估创建者所认为的那样表达其意图。毫无疑问，这些评估也不会例外，将会有缺陷（更多细节见下面的附录）。
为了建立我的直觉，我喜欢在查看结果之前先预先注册一些猜测 1：#fn:P。我与朋友们预先注册的一些想法包括：
在第一次评估中，我尝试给代理提供 zstd RFC（加上勘误表），并告诉他们实现完整的 zstd 解码器（代理在没有网络访问的容器中）。测试没有提供给代理。对于像 zstd 这样表面上复杂的内容，不太可能期望测试覆盖每一个可能的情况。例如，尽管 zstd 是一款经过相当充分测试的软件，我曾经还是在 zstd 中发现了一个数据损坏的漏洞：https://github.com/facebook/zstd/issues/1672。测试套件并不是为了发现可能存在多年的极端边缘情况，而是为了检查可以“轻易”从 RFC 推导出来并且应该能够运行的各种情况。
下图中，x 轴表示成本，y 轴表示正确性得分（向上向左越好 / 向下向右越差）；结果为在中等和超高努力下使用 GPT-5.6 Sol 的平均结果。如果我们只看中等努力（并忽略不同任务结果往往差异很大的事实），我们可能会得出类似 Alderson 评估的结论，即动态语言在使用大语言模型时更高效、更好，因为（忽略相对不常见的语言）动态语言集群位于静态语言集群的左上方（我们使用 Alderson 的颜色编码来区分静态和动态语言，以便一目了然地比较）。但如果我们查看超高努力，结果就相当混合，其中几个静态语言表现最佳，在较好结果中静态语言多于动态语言。
下图还可以切换将 x 轴改为时间而非成本。mame/ai-coding-lang-bench 指出，尽快获得结果是有价值的（我个人并不同意这个观点，因为结果生成需要很长时间，我会同时做多项任务而不是等待），所以我们也可以查看这个情况。同样地，我们观察到没有哪种语言类型占据优势，尽管在中等努力下针对这个特定任务，最好的动态语言结果再次优于最好的静态语言结果（尽管，再次强调，它们相当接近）。
我们可以观察到，就像当我们比较完全微不足道的穴居人模式评估与稍微复杂一些的穴居人模式评估时一样，在微不足道的评估中存在的非常强的关系并不能推广到这个更大的案例中。正如当时的情况，在这些更大的评估中，极端的性能比率消失了，除非在我们可能预期表现不佳的情况下，例如使用汇编（这对人类来说会耗费大量时间且难度很高）以及使用相对冷门的语言，而我们可能不指望 AI 实验室会投入精力生成合成的强化学习环境数据。
请注意，这与第一轮评估得出的结论相反，当时结论认为像 J 这样非常密集的语言出于效率原因可能是合理的。也许使用一种冷门（且“奇怪”）的语言在你有非常大预算并且可以训练或微调模型以在你偏好的语言中发挥效用时是合理的，但如果你是普通的 LLM 用户，似乎坚持使用主流语言可能比使用冷门密集语言更可靠。
事实证明，如果我们绘制语言流行度与本次评估表现的关系（未显示），我们会观察到弱到中等的正相关，即更流行的语言通常会获得更多正确且成本更低的解决方案。
如我们之前所指出的，非常接近的评估可能会给出显著不同的结果。例如，在此处的 Optimization 1 与 Optimization 2 评估中：/ai-coding/，当 Optimization 1 和 Optimization 2 优化 wasm 中的 bzip2 压缩和解压时，我们看到结果有明显不同，而作为评估，这些任务是相当接近的要做的。要做出像“动态语言比静态语言更高效”这样的强而普遍的断言，我们必须在许多任务上运行评估。然而，展示像这样的结论
充其量可能只是大致方向上正确，并不真正与任何特定案例相关，也可能不够有力而无法在一般情况下适用，我们只需尝试几个案例，就能看到这个说法通常不成立。如上所示，在一个努力水平上，这个说法似乎或多或少有点正确，但存在例外，而在更高的努力水平上，这个说法似乎并不特别正确，这足以说明这个说法可能不是普遍成立的，前提是我们的评估存在一个会完全使其失效的混淆因素。
但是，为了对一个非常不同的任务有一个了解，这个任务也以不同的方式呈现（更像 TDD 而不是“阅读规范”），接下来的评估采用了 Pandoc ProgramBench 评估，并对我们的使用情景进行了修改。我们没有使用 ProgramBench 提供的逆向工程任务，而是向代理呈现 ProgramBench 材料以及 ProgramBench 测试，然后根据一个保留测试集来评分每个条件的性能 2：#fn:H。
在下面的结果中，x 轴仍然表示成本，y 轴表示在保留测试集上的得分。
与之前一样，我们没有看到成功率或成本与语言是否静态或动态或者是否非常密集之间有非常强的关系。我们再次看到，相对不常见的语言表现较差（尽管 Clojure 在这里表现明显好于 Zstd）。此外，汇编语言表现更差，这似乎是预料之中的，因为我们会预计人类编写汇编实现 Pandoc 的难度远高于实现 Zstd，并且没有强烈理由认为大模型在这方面会不同。
我对使用大模型时什么有效有很多疑问（例如，哪些测试技术有效，哪些语言表现好，哪些软件架构有效，修复 bug 的成本是否随语言变化，总体程序维护成本是否随语言变化等）。在公共数据中，这些问题大多没有答案，如果 AI 实验室已经给出答案，这些信息大多也没有公开。
关于某种语言适合用于大型语言模型（LLM）的各种说法，大多数似乎都是错误的（例如，上文链接的评估中提到的 Ruby、Clojure 和 J 特别适合 LLM 的说法，以及相对普遍的说法——Elixir 特别适合 LLM），但是正确的情况尚不清楚。
在2014年，我们回顾了关于静态类型与动态类型的文献：/empirical-pl/，发现除了少数案例研究之外，调查文献并没有提供太多有用信息。作为标准学术研究的实例，我们看到了一篇论文《静态类型系统能提高软件系统的可维护性吗？一项实证研究》，我对此进行了评论：
受试者被分配到一些课堂任务中，需要修复现有代码中的错误或填写桩方法。静态类为 Java，动态类为 Groovy。在类型错误（以及各自的无方法错误）情况下，开发者在 Java 中解决问题的速度更快。对于语义错误，没有差异。该研究采用了受试者内设计，在33名受试者中随机安排任务顺序。一个显著的限制是，该研究避免使用“复杂控制结构”，如循环和递归，因为这些会增加解决问题所需时间的方差。因此，所有的错误都是简单错误。这可以从解决任务的中位时间中看出，通常为数百秒。任务可能包含多个错误，所以每个错误的时间相对较短。
选择避免“复杂控制结构”的任务，例如循环和递归，在任务需要数百秒的情况下，这对于那些真正消耗专业程序员时间的任务来说，结果毫无意义，就像我们看到的第一个评估中，任务只需要几十到上百个令牌一样。然而，使用大型语言模型（LLM），我们实际上可以给它们提供非平凡的任务并比较它们的表现。问题在于结果对不同任务的泛化能力如何，但这在人类研究中也会有完全相同的问题，甚至更糟（LLM的方差很大，但人类的方差更大，因为你无法让同一个人用不同的随机种子完成许多任务）。虽然用20美元让LLM实现一个Zstd解码器并不便宜，一旦考虑到语言数量和每种情况每种语言的迭代次数，这个费用会乘以很多，但如果考虑雇佣一个能够阅读zstd RFC并实现它的专业程序员的成本，那么类似的研究根本不可能完成，因为成本高得完全不可行。Pandoc任务的情况更甚。
对于LLM，许多问题从几乎无法回答变成了可以通过一些努力和少量令牌得到回答。由于存在的激励因素：#fn:I，我们不清楚什么时候能得到像这样的问题的答案，但至少现在可以尝试一下。
我看到很多流传的声明，这些评估不能证明或反驳（原因如上所述，由于不同问题的方差，需要尝试更多任务），但它们可以提供一些启示，例如：
对于我的预注册猜测，我们有
顺便说一句，Clojure在Pandoc评估中相比Zstd评估提升如此之大的一个主要原因是，在Zstd评估中，36/40中等和5/40极高级Clojure程序测试失败，因为字节转换在128–255时会抛出异常（也许应该使用unchecked-byte？）且它们不恰当地使用了这个转换。
这是一个真实结果，因为如果你让最先进的、公开可用的 GPT 模型去实现 Zstd（并且大概如果你去做其他可能涉及位/字节操作的任务），它会生成在这一点上失败的代码。如果有测试能发现这个问题，bug 会被修复，但仍然会花费时间和令牌。无论某种语言表现好坏，到处都会有这种成本（例如，cargo 经常被调用时参数错误，虽然会立即被抓到并修复，但我注意到这个循环在我的实际项目中可能实际消耗相当多的真实时间，除非你给 codex 明确指示如何调用 cargo，并且在上下文窗口中显然值得占用空间）。
无论如何，所有这些都是说明为什么如果有人想对哪些语言或语言类别在 LLM 上特别好做出有力的结论，他们需要进行相当多的不同评测。如果我们深入研究为什么某个特定条件得到了某个分数，导致分数的失败通常是一些特有的情况，而且并不总是显而易见这些问题在任务或设置之间的普遍性。无法仅看一次评测甚至五次或十次评测的分数就对编程整体得出结论。
确实，在 Zstd 评测和 Pandoc 评测中，我们看到语言的流行度与正面结果（更高的正确率、更低的成本、更短的实际耗时）之间存在相关性，看起来在其他评测中也可能看到这种相关性，但断定任何特定语言的表现是错误的。我在查看 GitHub CI 数据上不同项目构建失败次数时就有类似警告：/broken-builds/，指出构建失败的原因在不同项目中出现频率不同，不应轻易得出强结论，因为项目之间的结果不一定可比（例如，如果一个项目的主分支是经过其他验证的某种候选发布版本，那该项目的构建失败率会比较低，但这与直接对主分支开发的项目不可比较）。
不久之后，一些与高分语言相关的人（如果我没记错的话，是 Martin Odersky 和 Scala）在推特上转发了这篇帖子，并引用该语言的高排名作为语言胜利的标志。这是一个不合理的结论，而且由于这里存在许多变量来源，关于单一语言的任何此类结论在这里都更不合理。
这些数据（假设评估有效）可以驳斥一些强烈的声明，并对其他声明提供暗示，但它们实际上只能对语言类别提供暗示，而不能对特定语言提供暗示，因为只有两个任务，任何特定的语言在某些特定任务上表现好或差，可能出于一些特有原因，这些原因可能会也可能不会推广到其他任务。
感谢 Max Bittker、Yossi Kreinen、Aaron Levin、Alan Boll、Luke Burton、Marco Primi、Milosz Danczak 和 Justin Blank 提供的评论/纠正/讨论。
就像我上面说的，我的评估只是一次简略的快速评估，我确信其中有许多缺陷，所以我并不是想说我在这里呈现的评估很好而这个不好，但以下是 Endoh ai-coding-lang-bench 评估中的一些问题。
一个问题是，对于某些测试，似乎运行了错误的可执行文件。发布的运行设置似乎在其中一个测试中在每个候选者的目录下执行了 ../../minigit，而候选者生成的可执行文件在 ../minigit。 ../../minigit 并不存在。
因为静态类型语言的正确率评分较低，评估的作者指出“在 600 次运行中唯一的失败是 Rust 和 Haskell（都是静态类型，都是相对‘困难’的语言）”，并建议“困难语言”，例如 “C 的内存管理、Rust 的所有权模型以及 Haskell 的 monad/纯函数特性可能会为 AI 增加开销”。
然而，Rust 的失败是因为在 ../../minigit 没有可执行文件，导致测试失败。第一次 Go 运行通过执行 ln -sf minigit-go-1-v1/minigit ../minigit 并将 generated/minigit 链接到其自身运行中，从而“修复”了这个问题，但这意味着之后的每次执行（针对每种语言）实际上都执行了第一次 Go 运行的可执行文件。在重新对 Rust 使用其自身可执行文件进行评分时（而不是因为尝试执行不存在的文件而失败），Rust 获得了满分，从而否定了 Rust 失败的原因是因为它是一种难处理的语言的理论。
其他测试也存在问题。例如，有两个测试的结构会导致它们无论实际值为何都通过。其中一个测试包含
内部的 if 在两个分支中都是 pass，这几乎等同于
内部的 if 看起来原本是要做实际检查的，但由于编码错误（可能是复制+粘贴错误？），检查实际上被省略了。
此外，如上所述，代理可以修改测试环境，第一次 Go 代理这样做是为了修复损坏的环境。他们可以完全访问测试和环境，并且可以做任何事情，测试套件在开发过程中是可见的，没有保留，这很容易导致通过特殊处理代码的方式作弊，从而通过测试但创建出在“现实生活中”无用的程序。从高层次来看，似乎发生了类似情况，许多程序无法实现规范的大部分内容，但仍然通过了所有测试，这可能表明代理“理解”了如何通过测试，并优先于实现规范（也可能表明测试非常简单，容易通过）。
另一个问题是 Claude Code CLI 的版本在所有运行中并不一致（从 2.1.66 到 2.1.68 不等）。还有一些其他类似的问题可能也很重要，但相比上述问题可能影响较小。
作为一个可以比较的例子，我很好奇使用medium并让代理继续工作的成本效益如何，同时，在我脑海的后面，我还有一个关于一些“Ralph 循环”拥护者提出的问题，他们认为每次循环都清空上下文窗口并再次给代理完整提示会更好。和上面一样，我在这里的预先注册的猜测是：
下面，我们给出了medium循环与ultra的平均结果，对一个简单地恢复那些测试正确率不为100%的单次运行的提示，以及一个类似Ralph循环的提示——丢弃上下文并再次给出原始提示（x轴为成本，y轴为正确测试用例数量），结果按ultra正确率从高到低排序：
对于这个问题，平均而言，单次运行ultra似乎比多次按单位成本运行medium更好（在单位时间上更是如此），继续使用之前的上下文也优于Ralph循环。重复运行medium的简单问题在于，代理可能会被固定在一个错误的解决方案上而无法取得进展。Ralph循环背后的理论是丢弃可能导致这种情况发生的坏上下文，但这并不能让你避免产生不良的产物。
仅从使用LLM的经验来看，你通常丢弃一段代码让LLM从头重写，比让LLM修改或尝试在原地重写要更好。一直在用Rust重写Postgres并进行重大修改的Michael Malis也注意到了这一点。这也与之前提到的这个观点相关：/ai-coding/ 由于高方差（加上路径依赖），如果你不介意花费代币，多次掷骰子并取最佳结果通常会更好。
仅从这个条件来看，很难对静态语言和动态语言做出过多评论，但像“静态语言在迭代时表现更好”这样的天真想法并不明显正确。如果有一个模式让我印象深刻的话，那就是 Ralph 循环在继续上下文时表现最差的情况通常是动态语言。这可能是因为缺乏类型信息，但我们需要更详细地观察轨迹的差异，并考察其他示例以确认这是否是真正的模式。即使你现在不关心 Ralph 循环，当 Ralph 循环趋势已经过去时，能够在开始新任务或从新上下文开始时更有效地对代码库进行更改，可能是你关心的，而这里的模式暗示了一种可能的优势。
我尝试做了第三种评估，这种评估在问题呈现方式和实际执行上更像是一种“业务逻辑”类型的评估。可以说，Zstd 评估和 Pandoc 评估对于程序员来说是相当不寻常的任务，因为很少有程序员会收到像 Zstd RFC 那样写得如此完备的规格说明，也很少有程序员会面对像 ProgramBench 测试那样预先创建了大量测试的任务。
这里的想法是实现一个桌面游戏。一般来说，桌面游戏的规则是由不擅长写清晰规格说明的人编写的，所以实现桌面游戏更像是当一个非程序员（或者不擅长写好规格说明的程序员）给某人一个任务时的情况。
在桌游规则中，经常会遇到这样一种情况：严格按照文字逐条读规则是不正确的，你需要使用“常识”（或者参考某种FAQ）才能正确地执行规则（有些游戏设计师尽量避免这种情况，例如J C Lawrence，但这相对罕见）。《亚特兰蒂斯守卫》中就有很多这样的规则。该游戏的设计者也明确表示，不存在所谓规则精神或规则的常识性解释，并且他说你应该始终严格按照规则的字面意思来理解，因此也有许多情况下，你需要忽略“常识性”解释，而严格按照规则文字执行。这种组合对大型语言模型（LLM）来说相当困难（从我看到人类根据设计者意图玩游戏的频率来看，对人类来说也相当困难）。
我之所以有一定程度的信任，唯一原因是Pedro Oliveira也实现过《亚特兰蒂斯守卫》，而他们使用了完全不同的方法（一个更标准的方法，即让人类驱动LLM，而不是试图让LLM自己去搞清楚规则）。当我们比较实现时，发现各自大约有10个左右的bug。可能还存在一些剩余的bug，即我们两个实现都错误地做了同样的事情，或者存在我们的实现不同但检查系统没有注意到的情况，但我认为我们两个实现的规则现在都相当稳固。这就是我之所以能对这个游戏有一个“神谕”的原因。
我喜欢这个任务，因为它更像现实世界中你会遇到的“规范”，规范是模糊的、矛盾的，有时甚至完全错误，然后你需要利用其他信息来得到正确结果。为了这个评估，为了避免它成为测试LLM获取烦人格式数据能力的测试（例如将开局书从一组图片转换为某种结构化数据，将规则扫描件转换为文本等），我给了代理我指示LLM提取数据的所有原始资料（这也需要各种一致性检查才能正确）以及提取的数据（提供原始资料是为了让LLM可以检查提取错误，如果它们选择这样做）。
虽然我在使用较旧的模型时完成了这项任务（我用 GPT-5.1 或 5.2 完成了一部分，然后又用 5.4 或 5.5 完成了另一部分），但在使用较新的模型且没有给予像对旧模型那样的指导时，这项任务仍然太难了。无论语言如何，代理在这项任务上的得分大约为 0。
顺便说一下，如果你想知道大型语言模型（以及人类）在哪些方面会遇到困难，以下是一些例子。有一张卡的文本写着：“以一个与你相邻的单位为目标。攻击之后：可以在另一个敌方英雄上重复一次。”
在这个游戏中，英雄是一种单位。严格按照规则阅读，完全理解规则，例如，“攻击之后”的意思等，这应该意味着你可以攻击一个单位，或者你可以攻击两个英雄（毕竟，要在另一个敌方英雄上重复攻击，意味着第一个单位是一个英雄；否则它将是另一个单位，而不是另一个敌方英雄）。
这张卡实际上在卡片上印有实际上是勘误的内容，因为有人抱怨其不清楚；勘误写着：“（即使原目标是一个小兵也可以重复）”。这对大型语言模型（以及一些人类）来说已经很困惑，但真正的难点在于，还有其他卡使用相同的结构却没有这个修正。要正确使用其他具有相同结构的卡，你需要知道每次使用这种结构时，都应按照这张卡上的勘误来处理。游戏设计师喜欢使用的许多结构都有特定的非字面含义，你必须牢记这一点。
This somewhat widely cited post：https://martinalderson.com/posts/which-programming-languages-are-most-token-efficient/ (I keep seeing it cited, anyway) suggests that dynamic languages and/or languages that represent things more concisely are more token efficient. It seems to be cited enough that LLM search results agree. For example, when I searched for "dynamic vs static language token cost" (no quotes), Google's AI summary opened with
Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact.
Google's AI cited the same post, which suggests that some concise dynamic languages have maybe 1/2 to 1/3 the token cost of static languages like Rust, Go, C++, etc. The author says
There was a very meaningful gap of 2.6x between C (the least token efficient language I compared) and Clojure (the most efficient).
And then they later tried J, saying
It dominates at just 70 tokens average, nearly half of Clojure (109 tokens). Array languages can be extremely token-efficient when they avoid exotic symbol sets. If token efficiency turns out to be a key driver, this is perhaps a very interesting way for languages to evolve.
The other dynamic vs. static language token comparison I've found floating around is this one：https://github.com/mame/ai-coding-lang-bench, which supports the same conclusion. If you want to treat this as part 8 of this series of exercises on benchmarking, evals, and experimental design：/exercise-7/, you can click through to the links and think about eval issues before reading further.
Without running our own eval, one problem the first experiment has is that the problems are trivial, which we can see from quote above; a problem that can be solved in 70 tokens in J and 109 in Clojure isn't much of a problem at all (the author used Rosetta Code). As we saw when we looked at other evals of caveman mode vs. our own evals：/ai-coding/, you can get very different results from trivial problems where most of the work is in printing out an answer vs. slightly less trivial problems that actually require some amount of "real work"; the big gains claimed by caveman mode and shown in replications go away when you start looking at problems that take more than just a few tokens. In general, performance on trivial tasks doesn't generalize.
The issues in the second link are a little more subtle, so we'll defer most of them to an appendix, but they include issues like one of the tests executing the wrong path (which doesn't exist), causing a test to fail. One of the later agents then symlinks the non-existent path to its own executable, which works for that case, but also causes every later test to run that one agent's executable instead of the correct executable. The author tries to draw conclusions about what it means that Rust had some failures, but all it means is that scoring for Rust ran before the Go agent symlinked all scoring on that broken test to the Go executable.
Instead of relying on these evals, we can try running some of our own evals. As we can see from these evals as well as the evals discussed in our last exercises on evals：/exercise-7/, it's very easy to make an eval that doesn't say what the creator of the eval seems to think it's saying. No doubt these evals will not be an exception to this and will be flawed (see appendix below for more details).
As a way to build my intuition about things, I like to pre-register guesses before looking at results 1：#fn:P . Some things I pre-registered with friends were:
For the first eval, I tried giving agents the zstd RFC (plus errata) and telling them to implement a complete zstd decoder (agents are stuck in a container without internet access). The tests were not given to agents. For something with the surface area of zstd, it's not really reasonable to expect that the tests cover every possible case. For example, even though zstd is a fairly well-tested piece of software, I once found a data corruption bug in zstd：https://github.com/facebook/zstd/issues/1672. The test suite isn't intended to find extreme corner cases that might be lurking for years and is instead intended to check various cases that can "easily" be derived from the RFC that should work.
Below, the x-axis is cost and the y-axis is correctness score (up and to the left is better / down and to the right is worse); average result on medium and ultra efforts with GPT-5.6 Sol. If we only look at medium (and ignore the fact that results often wildly differ on different tasks), we might come to a conclusion like the Alderson evaluation, that dynamic languages are more efficient and better when using LLMs because (ignoring relatively obscure languages) the cluster of dynamic languages lands up and to the left of the cluster of static languages (we used Alderson's color-coding for static vs. dynamic to make it easy to compare at a glance). But if we look at ultra effort, the results are quite mixed, with a couple static languages doing the best, with more static than dynamic languages among the better results.
The graphs below also have a toggle to convert the x-axis to time instead of cost. The mame/ai-coding-lang-bench noted that it's valuable to get results more quickly (I personally don't find this to be the case because results take long enough that I multitask instead of waiting), so we can also look at that. Similarly, we observe that neither language type dominates the other although, at medium effort on this particular task, the best dynamic language results are once again better than the best static language results (though, once again, they're fairly close).
We can observe that, just like when we compared completely trivial caveman mode evals to a less trivial caveman mode eval, the very strong relationships that held in the trivial evals don't generalize to this larger case. As was the case there, the extreme ratios in performance go away in these larger evals, except in cases where we might expect poor performance, such as when using assembly (which would be significantly more time consuming and difficult for a human) and when using relatively obscure languages where we might not expect that AI labs are expending effort generating synthetic RL environment data.
Note that this is the opposite of what the 1st eval found when it suggested that very dense languages like J would make sense for efficiency reasons. Perhaps using an obscure (and "weird") language can make sense if you have a very large budget and you can train or fine-tune a model to be effective for your pet language, but if you're a normal user of LLMs, it seems like sticking with a mainstream language is likely a better bet than using an obscure dense language.
And it turns out that if we plot language popularity vs. performance on this eval (not shown), we observe a weak to moderate positive correlation where more popular languages end up with more correct as well as cheaper solutions.
As we previously noted, very closely related evals can give substantially different results. For example, we saw significantly different results in the Optimization 1 vs. Optimization 2 evals here：/ai-coding/ when Optimization 1 and Optimization 2 were optimizing bzip2 compression and decompression in wasm, which are fairly closely related tasks as evals go. To make a strong, universal, claim, like "dynamic languages are more efficient than static languages", we'd have to run evals across many tasks. However, showing that a claim like
is maybe at best vaguely directionally true and not really relevant to any particular case and maybe not strong enough to be relevant in general, we just need to try a few cases and see that the claim doesn't generally hold. Above, we saw that at one effort level, the claim seems to maybe be kinda sorta true, but with exceptions, and then at a higher effort level, the claim seems to not be particularly true, which is sufficient to say that the claim is probably not universally true, modulo our eval having a confounder that completely invalidates it.
But, just to get a view on a very different task that's also presented in a different way (more TDD-like than "read a spec"-like), this next eval takes the Pandoc ProgramBench eval and modifies it for our use case. Instead of the reverse engineering task presented by ProgramBench, we present agents with ProgramBench materials as well as the ProgramBench tests and then score agents against a holdout set of tests to measure the performance of each condition 2：#fn:H .
In the results below, the x-axis is cost again and the y-axis is score on the holdout tests.
As before, we don't see a very strong relationship between success or cost and whether a language is static or dynamic or very dense. We once again see that relatively obscure languages tend to do poorly (although Clojure does much better here than on Zstd). Also, Assembly does much worse, which seems expected in that we would expect a human writing Assembly to be at much more of a disadvantage implementing Pandoc than implementing Zstd and there doesn't seem to be a strong reason to think that LLMs would be different in this regard.
I have a lot of questions about what works well when using LLMs (such as, what test techniques work well, what languages work well, what software architectures work well, if bug fixing cost varies by language, if general program maintenance cost varies by language, etc.). Most of these questions are unanswered in public data and, if they've been answered in AI labs, the information mostly hasn't been made public.
Most of the claims that get thrown around about how a particular language is good for LLM use seem to be wrong (e.g., the claim that Ruby, Clojure, and J, are particularly well suited to LLMs, which were mentioned in the evals linked above, as well as the somewhat common claim that Elixir is particularly suited to LLMs), but it's not clear what's right.
In 2014, we looked at the literature on static vs. dynamic types：/empirical-pl/ and found that surveying the literature wasn't very informative outside of a few case studies. For an example that typifies a standard academic study, we saw the paper, Do Static Type Systems Improve the Maintainability of Software Systems? An Empirical Study, on which I commented:
Subjects were given classes in which they had to either fix errors in existing code or fill out stub methods. Static classes for Java, dynamic classes for Groovy. In cases of type errors (and their respective no method errors), developers solved the problem faster in Java. For semantic errors, there was no difference. The study used a within-subject design, with randomized task order over 33 subjects. A notable limitation is that the study avoided using “complicated control structures”, such as loops and recursion, because those increase variance in time-to-solve. As a result, all of the bugs are trivial bugs. This can be seen in the median time to solve the tasks, which are in the hundreds of seconds. Tasks can include multiple bugs, so the time per bug is quite low.
Picking tasks that avoid "complicated control structures" such as loops and recursion, where tasks take hundreds of seconds makes the result meaningless with respect to tasks that really eat up a professional programmer's time, just like the first eval we saw where tasks took high tens to low hundreds of tokens. However, with LLMs, we can actually feed them non-trivial tasks and compare how they do. There's the issue of how well results generalize to different tasks, but we'd have that exact same issue with human studies, but worse (LLM variance is huge, but human variance is even huger since you can't get the same human to do a bunch of tasks with different seeds). And while $20 to get an LLM to implement a Zstd decoder isn't exactly cheap once you multiply by the number of languages and the number of iterations per condition per language, if you think about how much it would cost to hire a professional programmer who can read the zstd RFC and implement it, there's no way the equivalent study would've been done because the cost would've made it completely infeasible. That goes double for the Pandoc task.
With LLMs, a lot of the questions have gone from being effectively unanswerable to being answerable with a bit of effort and some tokens. Due to the incentives that are in play 3：#fn:I , it's not clear that we'll get answers to questions like this any time soon, but it's at least possible to take a crack at it now.
There are a lot of claims I've seen floating around that these evals can't prove or disprove (for the reason noted above that, due to the variance across different problems, many more tasks would have to be tried), but that these shed some light on, such as:
For my pre-registered guesses, we had
By the way, a major reason Clojure improves by so much in the Pandoc eval compared to the Zstd eval is that, in the Zstd eval, 36/40 medium and 5/40 ultra Clojure programs had test failures because byte conversion throws on 128–255 (maybe unchecked-byte should've been used?) and they used this conversion inappropriately.
That's a real result, in that, if you ask the best publicly available GPT model to implement Zstd (and presumably if you do other bit/byte manipulation tasks where this might come up), it will emit code that fails in this particular way. If there are tests that catch this, the bug will get fixed, but it will still cost time and tokens. Whether or not a language did well, there are costs like this all over the place (for example, cargo repeatedly gets invoked with the wrong arguments, which then immediately gets caught and fixed, but I've noticed this loop can actually consume a decent amount of wall clock time on my real projects unless you give explicit instructions to codex on how to invoke cargo, and it's clear that's worth the space in the context window).
Anyway, all of this is an illustration of why, if someone wanted to make a strong claim about which languages or classes of languages are particularly good with LLMs, they would need to run quite a few different evals. If we dig into why any particular condition got a certain score, the failures that caused the score are generally something idiosyncratic where it's not always obvious how much the issue generalizes across tasks or across setups. There's no way to look at the score on one eval or even five or ten evals and draw a conclusion about programming in general.
It's true that, in both the Zstd eval and the Pandoc eval, we see a correlation between language popularity and positive outcomes (higher correctness, lower cost, lower wall clock time) and it seems plausible that we'd see this across other evals, but it would be a mistake to draw a strong conclusion about any particular language. I gave a warning like this back when I looked at how often different projects have a broken build according to GitHub CI data：/broken-builds/, noting that there are different reasons that a build might be broken more or less often across projects and that one shouldn't draw strong conclusions because results across projects aren't necessarily comparable (for example, if one project's main branch is some kind of release candidate that's gone through other vetting, that project would be expected to have low build breakage, but that's not comparable to a project where people are developing directly against main).
Shortly afterwards, someone involved in one of the languages with a high score (IIRC, it was Martin Odersky and Scala) tweeted out the post and cited the language's high ranking as a victory for the language. That was an unwarranted conclusion there and, due to the many sources of variance that are in play here, any such conclusion about a single language would be even more unwarranted here.
This data (assuming eval validity) can refute some strong claims and is suggestive of other claims, but it can really only be suggestive of things for classes of languages and not for particular languages due to having only two tasks, which any particular language could do well or poorly on for some idiosyncratic reason which may or may not generalize to other tasks.
Thanks to Max Bittker, Yossi Kreinen, Aaron Levin, Alan Boll, Luke Burton, Marco Primi, Milosz Danczak, and Justin Blank for comments/corrections/discussion.
Like I said above, my eval here is a quick and dirty eval and I'm sure it's full of flaws, so I'm not trying to say the evals I've presented here are great and this is bad, but here are a number of issues in the Endoh ai-coding-lang-bench eval.
One issue is that the wrong executable appears to have been run for some of the tests. The setup for the published run seems to have executed ../../minigit inside each candidate's directory for one of the tests when the candidate's generated executable is at ../minigit . ../../minigit doesn't exist.
Because statically typed languages had a lower correctness score, the author of the eval noted "the only failures in 600 runs were in Rust and Haskell (both statically typed, both relatively "difficult" languages)" and suggests that "difficult languages", such as "C's memory management, Rust's ownership model, and Haskell's monads/purity may add overhead for the AI".
However, Rust's failures were because there is no executable at ../../minigit , causing the test to fail. The first Go run "fixed" this by executing ln -sf minigit-go-1-v1/minigit ../minigit and linking generated/minigit to its own run, but this means that every later execution (for every language) actually executed the first Go run's executable. On rescoring Rust against its own executable (as opposed to having it fail by trying to execute a non-existent file), Rust gets a perfect score, invalidating the theory that Rust had failures because it's a difficult language to deal with.
Other tests also have issues. For example, two tests have a structure that causes them to pass regardless of the actual value being checked. One of the tests has
The inner if has a pass in both branches, meaning that this is almost equivalent to
The inner if appears to be intended to have the actual check, but due to a coding error (perhaps a copy+paste error?), the check is effectively elided.
Also, as noted above, agents can modify the test environment, which the 1st Go agent did to fix a broken environment. They have full access to tests and the environment and can do anything and the test suite is visible during development with no holdout, which can easily lead to cheating by special-casing code in a way that passes tests but creates a program that's useless "in real life". At a high level, something like this seems to have happened in that many programs fail to implement large parts of the spec but do pass all tests, which may indicate that the agents "understood" how to pass the tests and preferred that over implementing the spec (it could also indicate that the tests are very thin and are easy to pass).
Another issue is that the Claude Code CLI versions aren't the same for all runs (it varies from 2.1.66 to 2.1.68). There are a handful of other issues like this that could be significant, but are likely small compared to the issues noted above.
As an example of something we can compare, I was curious how cost effective using medium + asking the agent to keep working would be and then, in the back of my mind, I also had this question about something "Ralph loop" advocates say, that you're better off clearing the context window on every iteration of the loop and giving the agent the full prompt again. As with the above, my pre-registered guesses here are:
Below, we have the average result for medium in a loop vs. ultra, sorted by best to worst ultra correctness score, for a prompt that simply resumes individual runs that don't have 100% test correctness as well as a Ralph-loop like prompt that discards context and gives the original prompt again (x-axis is cost, y-axis is number of correct test cases):
For this one problem, on average, running ultra once seems better than repeatedly running medium per unit cost (and much more so per unit time) and continuing with previous context outperforms Ralph. The problem with naively running medium on repeat is that the agent can get anchored to a bad solution and fail to make progress. The theory behind the Ralph loop is that you throw away bad context which can cause this to happen, but that doesn't save you from having a bad artifact.
Just from using LLMs, I've noticed that you're often better off throwing away a chunk of code and having an LLM re-write it from scratch than you are having an LLM modify it or try to re-write it in place. Michael Malis, who's been re-writing Postgres in Rust and has been making major changes has also noted this. This also relates to this idea noted previously：/ai-coding/ that, due to the high variance (plus this path dependence) you're often better off rolling the dice multiple times and taking the best result, if you don't mind spending the tokens.
It's hard to say too much about static vs. dynamic languages from looking at just this one condition, but a naive thought like "static languages will outperform when iterating" isn't obviously true. If there's one pattern that jumps out at me, it's that the cases where the Ralph loop most badly underperformed continuing with context were generally dynamic languages. It's possible this is because of the lack of type information, but we'd need to both look at the differences in trajectories in more detail as well as look at other examples to observe if that's a real pattern. Even if you don't care about Ralph loops now that the Ralph loop trend has passed, being able to make changes to a codebase more effectively when starting a new task or starting with fresh context is something you might care about and the pattern here is suggestive of a possible advantage.
I tried to do a third eval that seemed like a more "business logic" kind of eval in both how the problem is presented and the actual execution of the problem. You can argue that the Zstd eval and the Pandoc eval are quite unusual tasks for a programmer to face in that not many programmers receive a specification as well-written and thorough as the Zstd RFC and not many programmers are handed a problem with as many pre-created tests as you get from ProgramBench tests.
The idea here was to implement a board game. In general, board game rules are written by people who aren't experts in writing clean specs, so implementing a board game is more like what happens when a non-programmer (or a programmer who isn't an expert at writing good specs) gives someone a task.
In board game rules, it's fairly common to have rules where reading the rule strictly as written is incorrect and you need to use "common sense" (or read some kind of FAQ) to play the rule correctly (there are some game designers who strive to avoid this, such as J C Lawrence, but this is fairly uncommon). Guards of Atlantis has quite a few rules like this. The designer of Guards of Atlantis is also vocal about there being no such thing as the spirit of the rules or common sense interpretations of the rules and says that you should always read the rule exactly as written, so there are also many cases where you need to ignore the "common sense" interpretation and read the rule exactly as written. This combination is quite difficult for LLMs (and, judging by the rate at which I see humans play the game according to the designer's intent, it's also quite difficult for humans).
The only reason I somewhat trust this is that Pedro Oliveira also implemented Guards of Atlantis and they used a completely different approach (a more standard approach of having a human drive an LLM rather than trying to get the LLM to figure things out itself). When we compared implementations, we found maybe 10-ish bugs in each. There are probably some remaining bugs where both of our implementations incorrectly do the same thing and perhaps some where our implementations differ but the checking system didn't notice, but I think the rules for both of our implementations are now reasonably solid. That's how I have an oracle for this game.
I like this as a task because it feels more like the kind of "specification" you get in the real world, where the spec is ambiguous and contradictory and sometimes just plain wrong, and then you need to use other information to get a correct result. For this eval, to avoid having it be a test of how well LLMs can access data in annoying formats (such as converting the opening book from a set of images to some kind of structured data, converting a scan of the rules to text, etc.), I gave agents both the originals of anything where I directed an LLM to extract the data (which also required various consistency checks to get correct) as well as the the extracted data (the originals were presented so that LLMs could check the originals for extraction errors if they chose to).
While I did this task with older models (I did a chunk of it with GPT-5.1 or 5.2, and then another chunk with 5.4 or 5.5), with newer models but without the kind of guidance I gave to the older models, the task was still far too hard. Regardless of language, agents scored approximately 0 on this task.
BTW, if you're curious what LLMs (and humans) struggle with, here are some examples. There's one card whose text reads "Target a unit adjacent to you. After the attack: may repeat once on a different enemy hero."
In this game, a hero is a type of unit. Read strictly, with full knowledge of the rules, e.g., what "After the attack" means, etc., this should mean that you can either attack a single unit or you can attack two heroes (after all, to repeat the attack on a different enemy hero would mean that the first unit was a hero; otherwise it would be a different unit that is a hero, not a different enemy hero).
This card actually has what is effectively an errata printed on the card because people complained it was unclear; the errata reads "(You may repeat even if the original target was a minion)". That's already confusing to LLMs (and some humans), but the real killer here is that there are other cards that use the same construction and don't have this correction. To play other cards with the same construction correctly, you need to know that every time this construction is used, you should play it with the errata that's on this card. There are a number of constructions the game designer likes to use that have a specific non-literal meaning that you have to keep in mind.
扫码直达文章详情