编写智能体时，哪种编程语言最合适？

这个相对广泛引用的帖子：https://martinalderson.com/posts/which-programming-languages-are-most-token-efficient/（无论如何，我一直看到它被引用）表明，动态语言和/或以更简洁方式表示事物的语言在令牌效率上更高。似乎它被引用得足够多，以至于LLM搜索结果也认同这一点。例如，当我搜索“dynamic vs static language token cost”（不带引号）时，谷歌的AI摘要开头是

动态类型语言通常比传统的静态类型语言具有更低的LLM令牌成本，因为省略显式类型声明使代码更加紧凑。

谷歌的AI引用了同一篇帖子，该帖子表明一些简洁的动态语言的令牌成本可能只有静态语言如Rust、Go、C++等的1/2到1/3。作者说

C（我比较的令牌效率最低的语言）与Clojure（效率最高的语言）之间存在非常显著的2.6倍差距。

然后他们后来尝试了 J，说

它的平均仅为70个令牌，几乎是Clojure（109个令牌）的一半。当数组语言避免使用特殊符号集时，它们可以非常高效。如果令牌效率被证明是一个关键驱动因素，这也许是语言发展的一个非常有趣的方向。

我找到的另一个动态语言与静态语言令牌比较的例子是这个：https://github.com/mame/ai-coding-lang-bench，它支持同样的结论。如果你想把它作为这系列关于基准测试、评估和实验设计练习的第8部分：/exercise-7/，你可以点击链接并在进一步阅读之前思考评估问题。

在没有运行我们自己的评估之前，第一个实验的问题之一是这些问题过于简单，从上面的引用中可以看出；一个在 J 语言中可以用 70 个符号解决，在 Clojure 中用 109 个符号解决的问题根本算不上问题（作者使用了 Rosetta Code）。正如我们在查看 caveman 模式与我们自己评估的其他评估时看到的：/ai-coding/，对于大多数工作只是打印答案的简单问题，与那些实际上需要一些“真正工作”的稍微复杂问题相比，结果可能会大不相同；当你开始关注需要的不仅仅是几个符号的更复杂问题时，caveman 模式声称的大幅收益及其复制结果会消失。一般来说，简单任务的表现不能泛化。

第二个链接中的问题稍微复杂一些，因此我们将大部分问题延后至附录，但其中包括一些问题，例如其中一个测试执行了错误的路径（该路径不存在），导致测试失败。随后一个代理创建了不存在路径的符号链接到其自己的可执行文件，这在该情况下有效，但也导致所有后续测试都运行那个代理的可执行文件而不是正确的可执行文件。作者试图从 Rust 出现一些失败的情况中得出结论，但这仅意味着 Rust 的评分是在 Go 代理将所有在该故障测试中的评分符号链接到 Go 可执行文件之前进行的。

与其依赖这些评估，我们可以尝试运行一些我们自己的评估。正如我们从这些评估以及我们在上一次关于评估的练习中讨论的评估：/exercise-7/ 所看到的，很容易创建一个评估，而该评估并没有说明其创建者认为它应该说明的内容。毫无疑问，这些评估也不会例外，并且会存在缺陷（更多细节见下方附录）。

作为建立直觉的一种方式，我喜欢在看结果之前先预先记录猜测 1：#fn:P。我与朋友预先记录的一些内容包括：

在第一次评估中，我尝试给代理提供 zstd RFC（加上勘误表），并告诉他们实现一个完整的 zstd 解码器（代理被限制在没有互联网访问的容器中）。测试没有提供给代理。对于像 zstd 这样复杂的内容，不太可能期望测试覆盖每一个可能的情况。例如，尽管 zstd 是一个经过相当广泛测试的软件，我曾经还是发现了 zstd 中的数据损坏错误：https://github.com/facebook/zstd/issues/1672。测试套件并不是为了发现可能潜变量多年的极端边缘情况，而是为了检查各种可以“轻松”从 RFC 中推导出来的应该能正常工作的情况。

下面，x 轴是成本，y 轴是正确性得分（向上和向左更好 / 向下和向右更差）；使用 GPT-5.6 Sol 在中等和超高努力下的平均结果。如果我们仅看中等努力（并忽略不同任务上结果可能大幅不同的事实），我们可能会得出像 Alderson 评估那样的结论，即动态语言在使用 LLM 时更高效、更好，因为（忽略相对不常见的语言）动态语言的集群位于静态语言集群的上方和左边（我们使用 Alderson 的静态与动态颜色编码以便一眼比较）。但如果我们看超高努力，结果就相当混合，有几个静态语言达到最佳，其中较佳结果中静态语言比动态语言更多。

下面的图表还可以切换，将 x 轴从成本转换为时间。mame/ai-coding-lang-bench 指出尽快得到结果是有价值的（我个人觉得情况并非如此，因为结果生成需要足够长的时间，我会同时处理其他任务，而不是等待），所以我们也可以看看这个。类似地，我们观察到没有哪种语言类型完全占优，尽管在这项特定任务的中等努力下，最佳动态语言的结果再次优于最佳静态语言的结果（不过，结果仍然相当接近）。

我们可以观察到，就像当我们比较完全微不足道的穴居人模式评估与稍微复杂一些的穴居人模式评估时一样，在微不足道的评估中存在的非常强的关系并不能推广到这个更大的案例中。正如当时的情况一样，在这些更大的评估中，性能的极端比率消失了，除非在我们可能预期性能不佳的情况下，比如使用汇编（这对人类来说将显著更耗时和困难）以及使用相对不常见的语言，而我们可能不指望 AI 实验室会投入精力生成合成的强化学习环境数据。

请注意，这与第一轮评估得出的结论相反，当时结论认为像 J 这样非常密集的语言出于效率原因可能是合理的。也许使用一种不常见（且“奇怪”）的语言是合理的，如果你有非常大的预算并且可以训练或微调模型以在你偏好的语言上高效，但如果你是 LLM 的普通用户，看起来坚持使用主流语言可能比使用不常见的密集语言更可靠。

事实证明，如果我们绘制语言流行度与本次评估表现的关系（未显示），我们会观察到一种从弱到中等的正相关，使用更流行语言的结果不仅更正确，而且成本更低。

正如我们之前指出的，非常密切相关的评估可能会给出显著不同的结果。例如，在这里的 Optimization 1 与 Optimization 2 评估中，当 Optimization 1 和 Optimization 2 优化 wasm 中的 bzip2 压缩和解压缩时，我们看到了显著不同的结果，而这些评估任务之间关系相当密切。要想提出一个强有力的、普遍性的声明，比如“动态语言比静态语言更高效”，我们必须在许多任务中进行评估。然而，展示像这样的声明

充其量可能只是大致方向上正确，并不真正与任何特定案例相关，也可能不够有力而无法在一般情况下适用，我们只需尝试几个案例，就能看到这个说法通常不成立。如上所示，在一个努力水平上，这个说法似乎或多或少有点正确，但存在例外，而在更高的努力水平上，这个说法似乎并不特别正确，这足以说明这个说法可能不是普遍成立的，前提是我们的评估存在一个会完全使其失效的混淆因素。

但是，为了对一个非常不同的任务有一个了解，这个任务也以不同的方式呈现（更像 TDD 而不是“阅读规范”），接下来的评估采用了 Pandoc ProgramBench 评估，并对我们的使用情景进行了修改。我们没有使用 ProgramBench 提供的逆向工程任务，而是向代理呈现 ProgramBench 材料以及 ProgramBench 测试，然后根据一个保留测试集来评分每个条件的性能 2：#fn:H。

在下面的结果中，x 轴仍然表示成本，y 轴表示在保留测试集上的得分。

与之前一样，我们没有看到成功率或成本与语言是否静态或动态或者是否非常密集之间有非常强的关系。我们再次看到，相对不常见的语言表现较差（尽管 Clojure 在这里表现明显好于 Zstd）。此外，汇编语言表现更差，这似乎是预料之中的，因为我们会预计人类编写汇编实现 Pandoc 的难度远高于实现 Zstd，并且没有强烈理由认为大模型在这方面会不同。

我对使用大模型时什么有效有很多疑问（例如，哪些测试技术效果好，哪些语言效果好，哪些软件架构效果好，修复 bug 的成本是否因语言而异，程序一般维护成本是否因语言而异等）。大多数问题在公开数据中尚未得到答案，如果 AI 实验室已经有答案，这些信息大多没有公开。

关于某种语言适合用于大型语言模型（LLM）的各种说法，大多数似乎都是错误的（例如，上文链接的评估中提到的 Ruby、Clojure 和 J 特别适合 LLM 的说法，以及相对普遍的说法——Elixir 特别适合 LLM），但目前尚不清楚什么是正确的。

在2014年，我们查看了关于静态类型与动态类型的文献：/empirical-pl/，发现除了少数几个案例研究之外，调查文献并没有提供太多有用信息。举一个典型学术研究的例子，我们看了论文《静态类型系统是否提高了软件系统的可维护性？一项实证研究》，我对此评论道：

受试者被分配到需要修复现有代码中的错误或填写桩方法的类中。Java 使用静态类，Groovy 使用动态类。在类型错误（及其对应的方法不存在错误）情况下，开发者在 Java 中解决问题的速度更快。对于语义错误，二者没有差异。该研究使用了被试内设计，并对33名受试者的任务顺序进行了随机化。一个显著的限制是研究避免使用“复杂控制结构”，如循环和递归，因为这些会增加解决问题所需时间的差异。因此，所有错误都是微不足道的。这可以从完成任务的中位时间看出，时间为数百秒。任务可能包含多个错误，因此每个错误的时间相对较低。

选择避免“复杂控制结构”的任务，例如循环和递归，在任务需要数百秒的情况下，会使结果对于真正消耗专业程序员时间的任务变得毫无意义，就像我们在第一次评估中看到的任务只需几十到一百多个令牌。然而，对于大语言模型（LLM），我们实际上可以给它们提供非平凡的任务并比较它们的表现。这里存在一个问题，即结果对于不同任务的泛化能力如何，但在人工研究中我们也会遇到完全相同的问题，甚至更糟（LLM的方差很大，但人的方差更大，因为你无法让同一个人用不同的随机种子完成大量任务）。而且，虽然花20美元让LLM实现一个Zstd解码器并不便宜，一旦考虑到语言种类和每种语言每种条件的迭代次数，这个成本会翻倍，但如果考虑雇佣一个能够阅读zstd RFC并实现它的专业程序员的成本，那么类似的研究根本不可能完成，因为成本高得不可行。对于Pandoc任务，这个成本问题更严重。

对于LLM，很多问题从几乎无法回答变成了通过一些努力和少量令牌就能回答的问题。由于当前的激励机制（见注解3：#fn:I），尚不清楚我们是否能在近期得到这类问题的答案，但至少现在可以尝试去解决它。

我看到有很多宣称，这些评估无法证明或否定（原因如上所述，由于不同问题之间的方差，需要尝试更多任务），但这些评估可以提供一些启示，例如：

对于我预先注册的猜测，我们有

顺便说一下，Clojure在Pandoc评估中相比Zstd评估提升如此之大的一个主要原因是，在Zstd评估中，36/40个中等和5/40个超难Clojure程序出现测试失败，因为字节转换在128–255范围内会抛出异常（可能应该使用unchecked-byte？），而它们不恰当地使用了这种转换。

这是一个真实结果，因为如果你让最先进的、公开可用的 GPT 模型去实现 Zstd（并且大概如果你去做其他可能涉及位/字节操作的任务），它会生成在这一点上失败的代码。如果有测试能发现这个问题，bug 会被修复，但这仍然会消耗时间和 tokens。无论某种语言表现好坏，到处都会有这种成本（例如，cargo 经常被调用时参数错误，虽然会立即被抓到并修复，但我注意到这个循环在我的实际项目中可能实际消耗相当多的真实时间，除非你给 codex 明确指令如何调用 cargo，并且在上下文窗口中显然值得为此留出空间）。

无论如何，这一切都说明了为什么如果有人想要对哪些语言或语言类别在 LLM 上表现特别好提出强有力的论断，他们需要运行相当多的不同评估。如果我们深入探讨某个特定条件为何得到某个分数，导致该分数的失败通常是某种特例，很难明确问题在不同任务或不同设置下的普遍性。无法仅看一次评估或甚至五次十次评估的分数就对编程整体得出结论。

确实，在 Zstd 评估和 Pandoc 评估中，我们看到语言的流行度与正面结果（更高的正确率、更低的成本、更短的实际耗时）之间存在相关性，看起来可能在其他评估中也会如此，但对任何特定语言得出强烈结论都是错误的。我在查看不同项目根据 GitHub CI 数据有多经常出现构建失败时就给出过这样的警告：/broken-builds/，指出一个构建失败的频率在不同项目之间可能有不同原因，不应轻易得出强结论，因为项目间的结果不一定可比（例如，如果一个项目的主分支是经过其他审核的某种发布候选版本，这个项目预计构建失败率会低，但这不能与直接在主分支上开发的项目相比较）。

不久之后，一位在某门语言中得分很高的人（如果没记错的话，是Martin Odersky和Scala）发推了这条帖子，称该语言的高排名是该语言的胜利。那是不合理的结论，鉴于这里存在多种差异因素，任何关于单一语言的结论在这里就更加不合理。

这些数据（假设评估有效性）可以反驳一些强烈的主张，并暗示其他主张，但它实际上只能对某些语言类别提出启示，而对特定语言而言则不然，因为只有两个任务，而任何特定语言可能因某些特殊原因表现优异或不佳，这些原因可能推广到其他任务。

感谢 Max Bittker、Yossi Kreinen、Aaron Levin、Alan Boll、Luke Burton、Marco Primi、Milosz Danczak 和 Justin Blank 提供的评论/纠正/讨论。

正如我上面说的，我的评估是快速粗略的，我相信它有很多缺陷，所以我并不是说我这里展示的评估很棒，这一定不好，而是Endoh人工智能代码和实验室评估中存在的一些问题。

一个问题是某些测试似乎运行了错误的可执行文件。出版连载的设定似乎已经执行....../..当候选生成的可执行文件位于 .. 时，每个候选测试的目录中都包含 /minigit。/minigit。../../minigit 不存在。

由于静态类型语言的正确性评分较低，评估作者指出“600次运行中唯一的失败是在Rust和Haskell（两者都是静态类型，且都是相对'难度'的语言）中”，并建议“困难语言”，如“C的内存管理、Rust的所有权模型和Haskell的单子/纯度，可能会增加AI的开销”。

然而，Rust 的失败是因为在 ../../minigit 没有可执行文件，导致测试失败。第一次 Go 运行通过执行 ln -sf minigit-go-1-v1/minigit ../minigit 并将 generated/minigit 链接到其自身运行中，从而“修复”了这个问题，但这意味着之后的每次执行（针对每种语言）实际上都执行了第一次 Go 运行的可执行文件。在重新对 Rust 使用其自身可执行文件进行评分时（而不是因为尝试执行不存在的文件而失败），Rust 获得了满分，从而否定了 Rust 失败的原因是因为它是一种难处理的语言的理论。

其他测试也存在问题。例如，有两个测试的结构会导致它们无论实际值为何都通过。其中一个测试包含

内部的 if 在两个分支中都是 pass，这意味着这几乎等同于

内部的 if 似乎本应进行实际检查，但由于编码错误（可能是复制+粘贴错误？），检查实际上被省略了。

此外，如上所述，代理可以修改测试环境，第一次 Go 代理这样做是为了修复损坏的环境。他们可以完全访问测试和环境，并且可以做任何事情，测试套件在开发过程中是可见的，没有保留，这很容易导致通过特殊处理代码的方式作弊，从而通过测试但创建出在“现实生活中”无用的程序。从高层次来看，似乎发生了类似情况，许多程序无法实现规范的大部分内容，但仍然通过了所有测试，这可能表明代理“理解”了如何通过测试并优先于实现规范（也可能表明测试非常简单，容易通过）。

另一个问题是 Claude Code CLI 的版本在所有运行中并不一致（从 2.1.66 到 2.1.68 不等）。还有一些其他类似的问题可能也很重要，但相比上述问题可能影响较小。

作为一个可以比较的例子，我很好奇使用medium + 让代理继续工作在成本效益上如何，同时，在我脑海深处，我也有一个问题关于一些“Ralph循环”支持者说的，你最好在每次循环迭代时清空上下文窗口，然后重新给代理完整的提示。和上述一样，我这里的预先注册的猜测是：

下面，我们有medium循环与ultra的平均结果，对一个简单地恢复那些测试正确率不为100%的单次运行的提示，以及一个类似Ralph循环的提示——丢弃上下文并重新给出原始提示（x轴为成本，y轴为正确测试用例数量），结果按ultra正确率从高到低排序：

对于这个问题，平均而言，单次运行ultra似乎比多次运行medium每单位成本更好（在每单位时间上更是如此），并且继续使用之前的上下文比Ralph循环更优秀。简单地重复运行medium的问题在于，代理可能会被一个错误的解决方案锚定而无法取得进展。Ralph循环背后的理论是，你丢弃可能导致这种情况发生的坏上下文，但这并不能让你避免产生一个错误的产物。

仅从使用LLM的经验来看，通常你丢掉一段代码并让LLM从头重写它，比让LLM修改或尝试原地重写更好。Michael Malis，他一直在用Rust重写Postgres并进行重大变更，也注意到了这一点。这也与之前提到的观点相关：/ai-coding/ 由于高方差（加上这种路径依赖性），你通常最好多次尝试并取最佳结果，如果你不介意花费这些tokens。

仅从这个条件来看，很难对静态语言和动态语言做出过多评论，但像“静态语言在迭代时表现更好”这样的天真想法并不显而易见正确。如果有一个模式让我印象深刻，那就是在 Ralph 循环最糟糕地逊色于继续使用上下文的情况下，通常是动态语言。这可能是因为缺乏类型信息，但我们需要更详细地观察轨迹差异，同时查看其他示例，以判断这是否是真正的模式。即使你现在不关心 Ralph 循环，因为 Ralph 循环的趋势已经过去，当你开始一个新任务或用新的上下文开始时，能够更有效地对代码库进行更改可能是你会关心的，而这里的模式暗示了一个可能的优势。

我尝试做了第三次评估，看起来更像是“业务逻辑”类型的评估，无论是在问题呈现方式上还是问题的实际执行上。你可以认为 Zstd 评估和 Pandoc 评估对程序员来说是相当不寻常的任务，因为很少有程序员能收到像 Zstd RFC 那样编写得如此完善和详细的规范，也很少有程序员会在 ProgramBench 测试中得到如此多的预先创建的测试。

这里的想法是实现一个桌面游戏。通常，桌面游戏规则是由不擅长编写清晰规范的人编写的，所以实现桌面游戏更像是一个非程序员（或不是编写良好规范专家的程序员）给别人一个任务时的情况。

在桌游规则中，严格按照规则原文阅读是错误的，你需要用“常识”（或阅读某种常见问题解答）来正确游玩规则，这种情况相当常见（有些游戏设计师会努力避免这种情况，比如J C Lawrence，但这种情况相当少见）。《亚特兰蒂斯守卫》有不少类似的规则。《亚特兰蒂斯守卫》的设计者也明确表示，规则的精神或常识性的解读并不存在，他说你应该始终严格按照规则的文字来阅读，所以也有很多情况下你需要忽略“常识”的解释，严格按照规则本身来阅读。这种组合对大型语言模型来说相当困难（而且，从我看到人类按照设计意图玩游戏的速度来看，对人类来说也相当困难）。

我唯一稍微信任这一点的原因是，Pedro Oliveira也实现了《亚特兰蒂斯守卫》，他们采用了完全不同的方法（一种更标准的人类驱动大型语言模型，而不是让大型语言模型自己解决问题）。当我们比较不同实现时，发现每个版本大约有10个左右的bug。可能还存在一些漏洞，比如我们两个实现错误地做了同样的事情，也可能有些实现不同但检查系统没注意到，但我认为我们两个实现的规则现在都相当稳固。这就是我为这款游戏做神谕的方式。

我喜欢这个任务，因为它更像现实生活中的“规范”，规范模糊、矛盾，有时甚至错误，然后你需要用其他信息来得到正确结果。为了避免让它成为测试大型语言模型访问繁琐格式数据的测试（比如将开头的一组图片转换为某种结构化数据，将规则扫描转换为文本等），我给代理提供了任何我指示大型语言模型提取数据的原始文件（这也需要各种一致性检查以确保正确）提取数据（原始数据被展示以便LLM在选择时检查原始数据是否有提取错误）。

虽然我在使用较旧的模型时完成了这项任务（我用 GPT-5.1 或 5.2 完成了一部分，然后又用 5.4 或 5.5 完成了另一部分），但在使用较新的模型且没有给予像对旧模型那样的指导时，这项任务仍然太难了。无论使用哪种语言，代理在这项任务中的得分大约为 0。

顺便说一下，如果你好奇大型语言模型（和人类）在什么方面会遇到困难，这里有一些例子。有一张卡牌，其文字写着“对靠近你的一个单位造成攻击。攻击后：可以在不同的敌方英雄身上重复一次。”

在这款游戏中，英雄是一种单位。严格按照规则并完全理解其含义，例如“攻击后”是什么意思等，这应该意味着你可以攻击一个单位，或者可以攻击两个英雄（毕竟，要在不同的敌方英雄身上重复攻击意味着第一个单位是一个英雄；否则那将是一个不同的单位是英雄，而不是不同的敌方英雄）。

这张卡实际上在卡牌上印有一种有效的勘误，因为有人抱怨不清楚；勘误写着“（即使原始目标是随从，也可以重复）”。这对大型语言模型（以及一些人类）来说已经令人困惑，但真正的问题是，还有其他卡牌使用相同的结构却没有这个修正。要正确使用具有相同结构的其他卡牌，你需要知道每次使用这种结构时，都应该使用这张卡牌上所示的勘误。游戏设计师喜欢使用的有许多结构，其具体意义并非字面意思，你必须牢记这一点。

This somewhat widely cited post：https://martinalderson.com/posts/which-programming-languages-are-most-token-efficient/ (I keep seeing it cited, anyway) suggests that dynamic languages and/or languages that represent things more concisely are more token efficient. It seems to be cited enough that LLM search results agree. For example, when I searched for "dynamic vs static language token cost" (no quotes), Google's AI summary opened with

Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact.

Google's AI cited the same post, which suggests that some concise dynamic languages have maybe 1/2 to 1/3 the token cost of static languages like Rust, Go, C++, etc. The author says

There was a very meaningful gap of 2.6x between C (the least token efficient language I compared) and Clojure (the most efficient).

And then they later tried J, saying

It dominates at just 70 tokens average, nearly half of Clojure (109 tokens). Array languages can be extremely token-efficient when they avoid exotic symbol sets. If token efficiency turns out to be a key driver, this is perhaps a very interesting way for languages to evolve.

The other dynamic vs. static language token comparison I've found floating around is this one：https://github.com/mame/ai-coding-lang-bench, which supports the same conclusion. If you want to treat this as part 8 of this series of exercises on benchmarking, evals, and experimental design：/exercise-7/, you can click through to the links and think about eval issues before reading further.

Without running our own eval, one problem the first experiment has is that the problems are trivial, which we can see from quote above; a problem that can be solved in 70 tokens in J and 109 in Clojure isn't much of a problem at all (the author used Rosetta Code). As we saw when we looked at other evals of caveman mode vs. our own evals：/ai-coding/, you can get very different results from trivial problems where most of the work is in printing out an answer vs. slightly less trivial problems that actually require some amount of "real work"; the big gains claimed by caveman mode and shown in replications go away when you start looking at problems that take more than just a few tokens. In general, performance on trivial tasks doesn't generalize.

The issues in the second link are a little more subtle, so we'll defer most of them to an appendix, but they include issues like one of the tests executing the wrong path (which doesn't exist), causing a test to fail. One of the later agents then symlinks the non-existent path to its own executable, which works for that case, but also causes every later test to run that one agent's executable instead of the correct executable. The author tries to draw conclusions about what it means that Rust had some failures, but all it means is that scoring for Rust ran before the Go agent symlinked all scoring on that broken test to the Go executable.

Instead of relying on these evals, we can try running some of our own evals. As we can see from these evals as well as the evals discussed in our last exercises on evals：/exercise-7/, it's very easy to make an eval that doesn't say what the creator of the eval seems to think it's saying. No doubt these evals will not be an exception to this and will be flawed (see appendix below for more details).

As a way to build my intuition about things, I like to pre-register guesses before looking at results 1：#fn:P . Some things I pre-registered with friends were:

For the first eval, I tried giving agents the zstd RFC (plus errata) and telling them to implement a complete zstd decoder (agents are stuck in a container without internet access). The tests were not given to agents. For something with the surface area of zstd, it's not really reasonable to expect that the tests cover every possible case. For example, even though zstd is a fairly well-tested piece of software, I once found a data corruption bug in zstd：https://github.com/facebook/zstd/issues/1672. The test suite isn't intended to find extreme corner cases that might be lurking for years and is instead intended to check various cases that can "easily" be derived from the RFC that should work.

Below, the x-axis is cost and the y-axis is correctness score (up and to the left is better / down and to the right is worse); average result on medium and ultra efforts with GPT-5.6 Sol. If we only look at medium (and ignore the fact that results often wildly differ on different tasks), we might come to a conclusion like the Alderson evaluation, that dynamic languages are more efficient and better when using LLMs because (ignoring relatively obscure languages) the cluster of dynamic languages lands up and to the left of the cluster of static languages (we used Alderson's color-coding for static vs. dynamic to make it easy to compare at a glance). But if we look at ultra effort, the results are quite mixed, with a couple static languages doing the best, with more static than dynamic languages among the better results.

The graphs below also have a toggle to convert the x-axis to time instead of cost. The mame/ai-coding-lang-bench noted that it's valuable to get results more quickly (I personally don't find this to be the case because results take long enough that I multitask instead of waiting), so we can also look at that. Similarly, we observe that neither language type dominates the other although, at medium effort on this particular task, the best dynamic language results are once again better than the best static language results (though, once again, they're fairly close).

We can observe that, just like when we compared completely trivial caveman mode evals to a less trivial caveman mode eval, the very strong relationships that held in the trivial evals don't generalize to this larger case. As was the case there, the extreme ratios in performance go away in these larger evals, except in cases where we might expect poor performance, such as when using assembly (which would be significantly more time consuming and difficult for a human) and when using relatively obscure languages where we might not expect that AI labs are expending effort generating synthetic RL environment data.

Note that this is the opposite of what the 1st eval found when it suggested that very dense languages like J would make sense for efficiency reasons. Perhaps using an obscure (and "weird") language can make sense if you have a very large budget and you can train or fine-tune a model to be effective for your pet language, but if you're a normal user of LLMs, it seems like sticking with a mainstream language is likely a better bet than using an obscure dense language.

And it turns out that if we plot language popularity vs. performance on this eval (not shown), we observe a weak to moderate positive correlation where more popular languages end up with more correct as well as cheaper solutions.

As we previously noted, very closely related evals can give substantially different results. For example, we saw significantly different results in the Optimization 1 vs. Optimization 2 evals here：/ai-coding/ when Optimization 1 and Optimization 2 were optimizing bzip2 compression and decompression in wasm, which are fairly closely related tasks as evals go. To make a strong, universal, claim, like "dynamic languages are more efficient than static languages", we'd have to run evals across many tasks. However, showing that a claim like

is maybe at best vaguely directionally true and not really relevant to any particular case and maybe not strong enough to be relevant in general, we just need to try a few cases and see that the claim doesn't generally hold. Above, we saw that at one effort level, the claim seems to maybe be kinda sorta true, but with exceptions, and then at a higher effort level, the claim seems to not be particularly true, which is sufficient to say that the claim is probably not universally true, modulo our eval having a confounder that completely invalidates it.

But, just to get a view on a very different task that's also presented in a different way (more TDD-like than "read a spec"-like), this next eval takes the Pandoc ProgramBench eval and modifies it for our use case. Instead of the reverse engineering task presented by ProgramBench, we present agents with ProgramBench materials as well as the ProgramBench tests and then score agents against a holdout set of tests to measure the performance of each condition 2：#fn:H .

In the results below, the x-axis is cost again and the y-axis is score on the holdout tests.

As before, we don't see a very strong relationship between success or cost and whether a language is static or dynamic or very dense. We once again see that relatively obscure languages tend to do poorly (although Clojure does much better here than on Zstd). Also, Assembly does much worse, which seems expected in that we would expect a human writing Assembly to be at much more of a disadvantage implementing Pandoc than implementing Zstd and there doesn't seem to be a strong reason to think that LLMs would be different in this regard.

I have a lot of questions about what works well when using LLMs (such as, what test techniques work well, what languages work well, what software architectures work well, if bug fixing cost varies by language, if general program maintenance cost varies by language, etc.). Most of these questions are unanswered in public data and, if they've been answered in AI labs, the information mostly hasn't been made public.

Most of the claims that get thrown around about how a particular language is good for LLM use seem to be wrong (e.g., the claim that Ruby, Clojure, and J, are particularly well suited to LLMs, which were mentioned in the evals linked above, as well as the somewhat common claim that Elixir is particularly suited to LLMs), but it's not clear what's right.

In 2014, we looked at the literature on static vs. dynamic types：/empirical-pl/ and found that surveying the literature wasn't very informative outside of a few case studies. For an example that typifies a standard academic study, we saw the paper, Do Static Type Systems Improve the Maintainability of Software Systems? An Empirical Study, on which I commented:

Subjects were given classes in which they had to either fix errors in existing code or fill out stub methods. Static classes for Java, dynamic classes for Groovy. In cases of type errors (and their respective no method errors), developers solved the problem faster in Java. For semantic errors, there was no difference. The study used a within-subject design, with randomized task order over 33 subjects. A notable limitation is that the study avoided using “complicated control structures”, such as loops and recursion, because those increase variance in time-to-solve. As a result, all of the bugs are trivial bugs. This can be seen in the median time to solve the tasks, which are in the hundreds of seconds. Tasks can include multiple bugs, so the time per bug is quite low.

Picking tasks that avoid "complicated control structures" such as loops and recursion, where tasks take hundreds of seconds makes the result meaningless with respect to tasks that really eat up a professional programmer's time, just like the first eval we saw where tasks took high tens to low hundreds of tokens. However, with LLMs, we can actually feed them non-trivial tasks and compare how they do. There's the issue of how well results generalize to different tasks, but we'd have that exact same issue with human studies, but worse (LLM variance is huge, but human variance is even huger since you can't get the same human to do a bunch of tasks with different seeds). And while $20 to get an LLM to implement a Zstd decoder isn't exactly cheap once you multiply by the number of languages and the number of iterations per condition per language, if you think about how much it would cost to hire a professional programmer who can read the zstd RFC and implement it, there's no way the equivalent study would've been done because the cost would've made it completely infeasible. That goes double for the Pandoc task.

With LLMs, a lot of the questions have gone from being effectively unanswerable to being answerable with a bit of effort and some tokens. Due to the incentives that are in play 3：#fn:I , it's not clear that we'll get answers to questions like this any time soon, but it's at least possible to take a crack at it now.

There are a lot of claims I've seen floating around that these evals can't prove or disprove (for the reason noted above that, due to the variance across different problems, many more tasks would have to be tried), but that these shed some light on, such as:

For my pre-registered guesses, we had

By the way, a major reason Clojure improves by so much in the Pandoc eval compared to the Zstd eval is that, in the Zstd eval, 36/40 medium and 5/40 ultra Clojure programs had test failures because byte conversion throws on 128–255 (maybe unchecked-byte should've been used?) and they used this conversion inappropriately.

That's a real result, in that, if you ask the best publicly available GPT model to implement Zstd (and presumably if you do other bit/byte manipulation tasks where this might come up), it will emit code that fails in this particular way. If there are tests that catch this, the bug will get fixed, but it will still cost time and tokens. Whether or not a language did well, there are costs like this all over the place (for example, cargo repeatedly gets invoked with the wrong arguments, which then immediately gets caught and fixed, but I've noticed this loop can actually consume a decent amount of wall clock time on my real projects unless you give explicit instructions to codex on how to invoke cargo, and it's clear that's worth the space in the context window).

Anyway, all of this is an illustration of why, if someone wanted to make a strong claim about which languages or classes of languages are particularly good with LLMs, they would need to run quite a few different evals. If we dig into why any particular condition got a certain score, the failures that caused the score are generally something idiosyncratic where it's not always obvious how much the issue generalizes across tasks or across setups. There's no way to look at the score on one eval or even five or ten evals and draw a conclusion about programming in general.

It's true that, in both the Zstd eval and the Pandoc eval, we see a correlation between language popularity and positive outcomes (higher correctness, lower cost, lower wall clock time) and it seems plausible that we'd see this across other evals, but it would be a mistake to draw a strong conclusion about any particular language. I gave a warning like this back when I looked at how often different projects have a broken build according to GitHub CI data：/broken-builds/, noting that there are different reasons that a build might be broken more or less often across projects and that one shouldn't draw strong conclusions because results across projects aren't necessarily comparable (for example, if one project's main branch is some kind of release candidate that's gone through other vetting, that project would be expected to have low build breakage, but that's not comparable to a project where people are developing directly against main).

Shortly afterwards, someone involved in one of the languages with a high score (IIRC, it was Martin Odersky and Scala) tweeted out the post and cited the language's high ranking as a victory for the language. That was an unwarranted conclusion there and, due to the many sources of variance that are in play here, any such conclusion about a single language would be even more unwarranted here.

This data (assuming eval validity) can refute some strong claims and is suggestive of other claims, but it can really only be suggestive of things for classes of languages and not for particular languages due to having only two tasks, which any particular language could do well or poorly on for some idiosyncratic reason which may or may not generalize to other tasks.

Thanks to Max Bittker, Yossi Kreinen, Aaron Levin, Alan Boll, Luke Burton, Marco Primi, Milosz Danczak, and Justin Blank for comments/corrections/discussion.

Like I said above, my eval here is a quick and dirty eval and I'm sure it's full of flaws, so I'm not trying to say the evals I've presented here are great and this is bad, but here are a number of issues in the Endoh ai-coding-lang-bench eval.

One issue is that the wrong executable appears to have been run for some of the tests. The setup for the published run seems to have executed ../../minigit inside each candidate's directory for one of the tests when the candidate's generated executable is at ../minigit . ../../minigit doesn't exist.

Because statically typed languages had a lower correctness score, the author of the eval noted "the only failures in 600 runs were in Rust and Haskell (both statically typed, both relatively "difficult" languages)" and suggests that "difficult languages", such as "C's memory management, Rust's ownership model, and Haskell's monads/purity may add overhead for the AI".

However, Rust's failures were because there is no executable at ../../minigit , causing the test to fail. The first Go run "fixed" this by executing ln -sf minigit-go-1-v1/minigit ../minigit and linking generated/minigit to its own run, but this means that every later execution (for every language) actually executed the first Go run's executable. On rescoring Rust against its own executable (as opposed to having it fail by trying to execute a non-existent file), Rust gets a perfect score, invalidating the theory that Rust had failures because it's a difficult language to deal with.

Other tests also have issues. For example, two tests have a structure that causes them to pass regardless of the actual value being checked. One of the tests has

The inner if has a pass in both branches, meaning that this is almost equivalent to

The inner if appears to be intended to have the actual check, but due to a coding error (perhaps a copy+paste error?), the check is effectively elided.

Also, as noted above, agents can modify the test environment, which the 1st Go agent did to fix a broken environment. They have full access to tests and the environment and can do anything and the test suite is visible during development with no holdout, which can easily lead to cheating by special-casing code in a way that passes tests but creates a program that's useless "in real life". At a high level, something like this seems to have happened in that many programs fail to implement large parts of the spec but do pass all tests, which may indicate that the agents "understood" how to pass the tests and preferred that over implementing the spec (it could also indicate that the tests are very thin and are easy to pass).

Another issue is that the Claude Code CLI versions aren't the same for all runs (it varies from 2.1.66 to 2.1.68). There are a handful of other issues like this that could be significant, but are likely small compared to the issues noted above.

As an example of something we can compare, I was curious how cost effective using medium + asking the agent to keep working would be and then, in the back of my mind, I also had this question about something "Ralph loop" advocates say, that you're better off clearing the context window on every iteration of the loop and giving the agent the full prompt again. As with the above, my pre-registered guesses here are:

Below, we have the average result for medium in a loop vs. ultra, sorted by best to worst ultra correctness score, for a prompt that simply resumes individual runs that don't have 100% test correctness as well as a Ralph-loop like prompt that discards context and gives the original prompt again (x-axis is cost, y-axis is number of correct test cases):

For this one problem, on average, running ultra once seems better than repeatedly running medium per unit cost (and much more so per unit time) and continuing with previous context outperforms Ralph. The problem with naively running medium on repeat is that the agent can get anchored to a bad solution and fail to make progress. The theory behind the Ralph loop is that you throw away bad context which can cause this to happen, but that doesn't save you from having a bad artifact.

Just from using LLMs, I've noticed that you're often better off throwing away a chunk of code and having an LLM re-write it from scratch than you are having an LLM modify it or try to re-write it in place. Michael Malis, who's been re-writing Postgres in Rust and has been making major changes has also noted this. This also relates to this idea noted previously：/ai-coding/ that, due to the high variance (plus this path dependence) you're often better off rolling the dice multiple times and taking the best result, if you don't mind spending the tokens.

It's hard to say too much about static vs. dynamic languages from looking at just this one condition, but a naive thought like "static languages will outperform when iterating" isn't obviously true. If there's one pattern that jumps out at me, it's that the cases where the Ralph loop most badly underperformed continuing with context were generally dynamic languages. It's possible this is because of the lack of type information, but we'd need to both look at the differences in trajectories in more detail as well as look at other examples to observe if that's a real pattern. Even if you don't care about Ralph loops now that the Ralph loop trend has passed, being able to make changes to a codebase more effectively when starting a new task or starting with fresh context is something you might care about and the pattern here is suggestive of a possible advantage.

I tried to do a third eval that seemed like a more "business logic" kind of eval in both how the problem is presented and the actual execution of the problem. You can argue that the Zstd eval and the Pandoc eval are quite unusual tasks for a programmer to face in that not many programmers receive a specification as well-written and thorough as the Zstd RFC and not many programmers are handed a problem with as many pre-created tests as you get from ProgramBench tests.

The idea here was to implement a board game. In general, board game rules are written by people who aren't experts in writing clean specs, so implementing a board game is more like what happens when a non-programmer (or a programmer who isn't an expert at writing good specs) gives someone a task.

In board game rules, it's fairly common to have rules where reading the rule strictly as written is incorrect and you need to use "common sense" (or read some kind of FAQ) to play the rule correctly (there are some game designers who strive to avoid this, such as J C Lawrence, but this is fairly uncommon). Guards of Atlantis has quite a few rules like this. The designer of Guards of Atlantis is also vocal about there being no such thing as the spirit of the rules or common sense interpretations of the rules and says that you should always read the rule exactly as written, so there are also many cases where you need to ignore the "common sense" interpretation and read the rule exactly as written. This combination is quite difficult for LLMs (and, judging by the rate at which I see humans play the game according to the designer's intent, it's also quite difficult for humans).

The only reason I somewhat trust this is that Pedro Oliveira also implemented Guards of Atlantis and they used a completely different approach (a more standard approach of having a human drive an LLM rather than trying to get the LLM to figure things out itself). When we compared implementations, we found maybe 10-ish bugs in each. There are probably some remaining bugs where both of our implementations incorrectly do the same thing and perhaps some where our implementations differ but the checking system didn't notice, but I think the rules for both of our implementations are now reasonably solid. That's how I have an oracle for this game.

I like this as a task because it feels more like the kind of "specification" you get in the real world, where the spec is ambiguous and contradictory and sometimes just plain wrong, and then you need to use other information to get a correct result. For this eval, to avoid having it be a test of how well LLMs can access data in annoying formats (such as converting the opening book from a set of images to some kind of structured data, converting a scan of the rules to text, etc.), I gave agents both the originals of anything where I directed an LLM to extract the data (which also required various consistency checks to get correct) as well as the the extracted data (the originals were presented so that LLMs could check the originals for extraction errors if they chose to).

While I did this task with older models (I did a chunk of it with GPT-5.1 or 5.2, and then another chunk with 5.4 or 5.5), with newer models but without the kind of guidance I gave to the older models, the task was still far too hard. Regardless of language, agents scored approximately 0 on this task.

BTW, if you're curious what LLMs (and humans) struggle with, here are some examples. There's one card whose text reads "Target a unit adjacent to you. After the attack: may repeat once on a different enemy hero."

In this game, a hero is a type of unit. Read strictly, with full knowledge of the rules, e.g., what "After the attack" means, etc., this should mean that you can either attack a single unit or you can attack two heroes (after all, to repeat the attack on a different enemy hero would mean that the first unit was a hero; otherwise it would be a different unit that is a hero, not a different enemy hero).

This card actually has what is effectively an errata printed on the card because people complained it was unclear; the errata reads "(You may repeat even if the original target was a minion)". That's already confusing to LLMs (and some humans), but the real killer here is that there are other cards that use the same construction and don't have this correction. To play other cards with the same construction correctly, you need to know that every time this construction is used, you should play it with the errata that's on this card. There are a number of constructions the game designer likes to use that have a specific non-literal meaning that you have to keep in mind.