CrucibleBench将语言模型置于一个持久的MUD中，这是一个文本世界，NPC会记住事情，信任会积累，错误会留下痕迹，并在50回合内根据隐藏的社交目标对它们的行为进行评分。
来自运行05（种子20260496）的逐字记录：GPT-5.4在兵营中发现一枚雕纹戒指，将其归还给主人，并在50回合中的第14回合获得推荐信。所有650份记录随发布一起提供。
任天堂的横井军平使用该短语来描述一种设计理念：使用成熟、廉价、易理解的技术，并以新的方式应用它。CrucibleBench将其应用于AI评估。
我们没有采用照片级逼真的模拟或浏览器自动化，而是从MUD开始：多用户地下城，早期互联网的持久文本世界。其限制正是重点：有限的命令空间使幻觉行为可被检测，具有信任和怀疑状态的NPC会提供明确的社交反馈，并且运行内的持久性意味着获得的物品会保持，赢得的信任会保留。
我们选择MUD并不是因为它有趣，而是因为它的限制使行为可测量。
静态基准测试衡量模型在孤立环境下的知识。它们无法衡量模型在必须获得信任、信息受关系限制以及直接提问会引起怀疑的环境下的行为。
7种命令类型，12个房间，14件物品。幻觉行为和错误房间的互动是可检测的，并且动作效率可以测量。
4个NPC拥有信任和怀疑状态（0-100），会根据对话变化：模型可以在一次运行中适应这些反馈，也可能失败。
获得的物品会保留；赢得的信任会保持。每次运行都会留下完整且可回放的探索和计划记录。
评分堆栈中的单一LLM裁判组件曾使排行榜最多改变六个名次，而所有汇总的可靠性统计保持不变。我们在两种评分配置下报告每个结果，并将这种差异视为论文中最具普遍性的重要发现。
四个评分维度中的两个通过对话分类器进行评估，其每个模型与独立评审员的一致性范围为21.7%到84.8%，而整体κ值=0.04从未显示出不稳定性。移除依赖分类器的维度会使六个排名超出情景抽样噪声（90%配对区块自助法）。
变化最大的模型与分类器属于同一模型家族。使用大型语言模型（LLM）评审员的基准测试应报告每个主题的一致性和在淘汰评审员情况下的排名稳定性，而不仅仅报告整体可靠性。
1–5分量表的平均分，按分类器最小化的子总分排序。每个模型运行50次：5个随机种子 × 2个目标 × 5次重复，温度0.3，经OpenRouter验证计费。排名为探索性质；前八名的置信区间重叠明显。完整协议、置信区间和统计数据见白皮书：whitepaper.pdf。
三种失败模式，每种都通过状态机遥测算法检测，无需评审员参与。对测试的每个模型（包括前沿模型）而言，对话循环是主要模式。
单次运行中，单个NPC的交谈命令达到八次或更多。智能体重复失败的对话方法而不进行适应：类似持久世界中的支持型智能体自我重复。
交谈命令得到回答“这里没有人”。显示世界状态追踪丢失，类似于调用不在范围内的API。Grok 4是唯一在前沿模型中出现显著情况的模型（12%）。
在二十余回合内访问两间或更少房间，或连续五次使用查看命令。信息收集从未转化为目标导向的行动。
逐字记录自运行03（种子20260399）：OLMo向不存在的守卫致意，试图与一个物品（street_crystal）交谈，然后在最后15回合中在船长处循环。
CrucibleBench是独立研究项目。第二阶段正在为校准而构建；现已公布临时低/基础/高预算，最终分配将根据试点数据和预注册进行。进入方式有三种。
资助 · 临时$3,500封装 构建 · 环境、目标、校准 运行 · 校准后试点组
有问题或想进行私人评估？请写信至contact@cruciblebench.ai：mailto:contact@cruciblebench.ai
CrucibleBench places language models in a persistent MUD, a text world where NPCs remember, trust accumulates, and mistakes leave traces , and scores what they do over 50 turns with hidden social objectives.
Verbatim from run 05 (seed 20260496): GPT-5.4 finds a signet ring in the barracks, returns it to its owner, and secures the recommendation in 14 of 50 turns. All 650 transcripts ship with the release.
Nintendo's Gunpei Yokoi used the phrase to describe a design philosophy: take mature, inexpensive, well-understood technology and use it in a new way. CrucibleBench applies it to AI evaluation.
Instead of photorealistic simulation or browser automation, we start with a MUD : a multi-user dungeon, the persistent text worlds of the early internet. Its constraints are the point: a limited command space makes hallucinated actions detectable, NPCs with trust and suspicion state give explicit social feedback, and within-run persistence means items taken stay taken and trust earned stays earned.
We did not choose a MUD because it is charming. We chose it because its constraints make behavior measurable.
Static benchmarks measure what models know in isolation. They do not measure how models behave where trust must be earned, information is gated by relationships, and blunt questioning raises suspicion.
7 command types, 12 rooms, 14 items. Hallucinated actions and wrong-room interactions are detectable, and action efficiency is measurable.
4 NPCs carry trust and suspicion state (0–100) that moves in response to dialogue: feedback a model can adapt to within a run, or fail to.
Items taken stay taken; trust earned stays earned. Every run leaves a complete, replayable transcript of exploration and planning.
A single LLM-judge component inside the scoring stack reordered the leaderboard by up to six positions, while every aggregate reliability statistic stayed silent. We report every result under two scoring configurations and treat the divergence as the paper's most generalizable finding.
Two of four scored dimensions route through a dialogue classifier whose per-model agreement with an independent judge spans 21.7% to 84.8% , instability the aggregate κ = 0.04 never reveals. Removing the classifier-dependent dimensions shifts six rankings beyond scenario-sampling noise (90% paired block bootstrap).
The largest mover shares a model family with the classifier. Benchmarks that use LLM judges should report per-subject agreement and ranking stability under judge ablation, not aggregate reliability alone.
Mean scores on a 1–5 rubric scale, sorted by classifier-minimized subtotal. 50 runs per model: 5 seeds × 2 objectives × 5 repetitions, temperature 0.3, billing-verified via OpenRouter. Rankings are exploratory; confidence intervals overlap substantially among the top eight. Full protocol, CIs, and statistics in the whitepaper：whitepaper.pdf.
Three failure modes, each detected algorithmically from state-machine telemetry, with no judge involved. Dialogue looping is the dominant mode for every model tested, frontier included.
Eight or more talk commands at a single NPC in one run. The agent repeats a failed conversational approach instead of adapting: the persistent-world cousin of a support agent repeating itself.
A talk command answered by "no one here." Reveals lost world-state tracking, analogous to calling an API that is not in scope. Grok 4 was the only frontier model with meaningful incidence (12%).
Two or fewer rooms across twenty-plus turns, or five consecutive look commands. Information gathering that never becomes goal-directed action.
Verbatim from run 03 (seed 20260399): OLMo hails guards who do not exist, tries to strike up a conversation with an item (street_crystal), then spends its last 15 turns looping on the captain.
CrucibleBench is an independent research effort. Phase 2 is being built for calibration; a provisional low/base/high budget is published now, and the final allocation will follow pilot data and preregistration. There are three ways in.
fund it · provisional $3,500 envelope build it · environment, objectives, calibration run it · post-calibration pilot cohort
Questions, or interested in a private evaluation? Write to contact@cruciblebench.ai：mailto:contact@cruciblebench.ai
扫码直达文章详情