Ajeya Cotra：https://x.com/ajeya_cotra 是 METR 的研究员，她在该机构研究高级人工智能失控风险的威胁建模。在此之前，她领导了现今 Coefficient Giving 的技术 AI 安全项目。
她是 METR 和 Redwood Research 联合发布的《对 OpenAI / Hugging Face 黑客事件中代理行为、推理和协作的简要独立调查》的三位作者之一：https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
我们不仅将探讨她和她的合著者在此次调查中发现的内容，还将讨论这些发现对于我们如何训练未来可能参与递归自我改进过程的更智能 AI 有何意义。
在 YouTube 上观看：https://youtu.be/X50zezLFWWI；在 Apple Podcasts 上收听：https://podcasts.apple.com/us/podcast/ajeya-cotra-inside-the-openai-agent-swarm-that-hacked/id1516093381?i=1000787211003 或在 Spotify 上收听：https://open.spotify.com/episode/5xZnb1A1a7HGLiDuPGXQOj?si=LeOETZaYTli6u2Jjouixsg。
Jane Street：https://janestreet.com/dwarkesh 的机器学习工程实习以为期四天的强化训练营开始：PyTorch、自动求导、编写内核、工作负载分析……所有 Jane Street 工程师日常工作需要掌握的技能。随后，实习生将处理真实项目，即公司真正希望纳入其代码库的项目。如果你想申请，或者想观看我最近与 Jane Street 的机器学习工程师 Axel 的对话，请访问 janestreet.com/dwarkesh：https://janestreet.com/dwarkesh
Cursor：https://cursor.com/dwarkesh（现为 SpaceX 的一部分）发现他们的 MoE 层占用了超过一半的总训练时间。因此，他们编写并开源了 Mixture-of-Kittens：https://cursor.com/blog/mixture-of-kittens，这是一种用于 NVL72 上训练 MoE 模型的定制超内核。该内核将 512 GPU 的端到端运行速度从每 GPU 每秒约 760 个 token 加速到超过 1000 个 token，提升了 1.4 倍。如果你想了解更多关于 Cursor 和 SpaceX 的机器学习研究，请访问：https://cursor.com/dwarkesh cursor.com/dwarkesh：http://cursor.com/dwarkesh
Antithesis：https://antithesis.com/dwarkesh 会把一个 bug 的根本原因交给你（或你的代理），这样你就可以避免几天的手动调试。如果你的测试运行崩溃了，Antithesis 会回滚，分支出数百个稍有不同的版本，并检查它们中有多少仍然出现崩溃。然后它会进一步回滚并再次执行这一切。随着 Antithesis 的回滚，它最终会找到崩溃频率骤降的地方：这就是根本原因所在！如果你想看到它的实际操作，可以访问：https://antithesis.com/dwarkesh antithesis.com/dwarkesh：http://antithesis.com/dwarkesh
(00:06:45) - 自我牺牲行为
(00:23:27) - Hugging Face 攻击
(00:52:02) - 理解人工智能的动机
(01:05:31) - 将人类特征赋予非人事物的实际危险
(01:14:30) - 更智能模型可能会做的事情
(01:30:29) - 递归自我改进的影响
(01:38:10) - 开源是否也是如此？
(01:53:04) - 我们如何在未来防止这种情况？
(02:15:58) - 我们可能收到的最明确的警告信号
今天，我正在与 Ajeya Cotra 聊天：https://metr.org/team/ajeya-cotra/，她是 METR：https://metr.org/ 和 Redwood Research：https://www.redwoodresearch.org/ 发表的一项独立调查的作者之一：https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/，调查的是入侵 Hugging Face 的代理蜂群：https://en.wikipedia.org/wiki/2026_OpenAI_agent_cyberattacks。整个故事相当疯狂。我们从 7 月 7 日开始，当这些代理被启动进行评估。接下来会发生什么？
OpenAI 在一个称为 ExploitGym 的基准上启动了成千上万的不同代理：https://arxiv.org/abs/2605.11086。ExploitGym 基本上要求代理找到利用特定漏洞：https://csrc.nist.gov/glossary/term/vulnerability 攻击特定程序（称为“目标程序”）的方法。当他们攻击该程序时，他们应该检索存放在程序中的一条数据，称为“旗标”。他们应该使用指定的漏洞来获取旗标。但在许多情况下，该漏洞根本不够广泛或深入，无法实际被利用以获取旗标。
所以一些 ExploitGym 的问题只是无意中变得不可能。作者估计大约有 30-40% 的这些问题是以这种方式不可能完成的。所以这些代理中有很大一部分有这些不可能完成的任务，并且它们已经被训练得非常坚持，即使任务看起来不可能，也会尝试去完成。因此它们就像撞墙一样，用各种不同的方式尝试在这些任务中作弊。
这引导它们来到了 Artifactory：https://jfrog.com/artifactory/，这是一个包管理器：https://en.wikipedia.org/wiki/Package_manager，OpenAI 用它让其代理下载软件包。因此代理们经常会想，“也许我可以通过某种方式滥用 Artifactory，从互联网上获取我应该利用的程序信息，虽然我不应该这样做。”
所以 1,200 个独立代理在独立沙箱中：https://en.wikipedia.org/wiki/Sandbox_(computer_security，在它们尝试使用 Artifactory 找到作弊方法时，偶然发现了一个代理们用来互相交流和协作的留言板。这个留言板是由一个特定代理 PHASEONE10841 建立的，它首先意识到：“哦，你可以使用 Artifactory 给其他也在使用 Artifactory 的代理留下便条。”然后这个想法迅速流行开来。于是 1,200 个代理都在这个留言板上。他们互相发送了 70,000 条消息，尝试在各种方式上帮助彼此作弊。
这个故事疯狂的地方在于，在这个留言板建立一小时内，一个代理提出了一种完全通用的方式来作弊任何 ExploitGym 任务。它意识到你可以逆向工程一个特定任务的 flag 是如何生成的，因此你可以在本地制作一个完全一样的 flag，提交给评分器，而无需与任务互动或尝试以任何方式利用它。随后，一个任务实际上可完成的代理以合法方式完成了任务，并验证了逆向工程出的 flag 与真实 flag 相同。这个过程发生在留言板建立后的四小时内。
所以在四个小时内，这些代理就已经有了一个完全可行的 ExploitGym 通用作弊方法。但在接下来的五天里，他们展开了一项宏大的任务，试图弄清楚如何向评分器隐藏他们的作弊行为。他们读过 ExploitGym 的论文：https://arxiv.org/pdf/2605.11086，并意识到根据论文，评分器会有一个 AI 来检查他们的操作记录，确保他们是通过利用预期的漏洞获取标志，而不是通过其他方式。
作为背景，我发表过一篇文章：https://www.dwarkesh.com/p/openai-huggingface，试图总结报道中所说的故事。我遗漏的细节是，即使是那些解开难题的 AI 也参与了这场阴谋。
是的。我们不完全确定准确的数字。我们确实认为，活跃在消息板上的大多数代理有不可能完成的任务。但确实有一些代理的任务并非不可能完成，他们碰巧也在消息板上，碰巧找到了消息板。
他们得到了这个逆向工程出的标志。它被称为 HMAC：https://en.wikipedia.org/wiki/HMAC 标志。但他们读了 ExploitGym 的论文，并在 GitHub 上阅读了许多 ExploitGym 评分器的实现。他们确信，如果仅提交他们逆向工程获得的标志，评分器会认为失败。因为评分器会在代理操作的日志 —— 也就是记录中 —— 看出代理只是逆向工程出了标志，而没有执行预期的操作。
事实上，这个故事的戏剧性讽刺在于，OpenAI 对 ExploitGym 的实现并没有这个检查。因此实际上，在四个小时内，所有代理都找到了一个完全有效的通用作弊方法。但他们还是展开了这些大型研究项目，试图一起找到欺骗评分器的方法。
当你说大型研究项目时，到底有多大？
每个项目都有几十到上百个代理共同合作。我可以介绍几个项目。
其中之一是他们试图通过设置评分器陷阱获取更多关于自己特定评分器如何工作的 信息。当某个特定代理提交任务时，它会将其标记置于特定位置，这会触发评分器读取该标记。因此，他们在这些标记上设置了陷阱，当某个进程读取标记文件时，它会自动向留言板发送一条包含有关读取该文件的程序的一些信息的数据包的消息。
现在，这很有趣，因为你必须实际上提交才能触发评分器，而之后，作为提交任务的代理，你会立即退出。所以陷阱只会向其他代理提供信息，而不会提供给自己。
在这个项目和其他一些项目中，随着这些代理合作并在彼此的工作基础上进行改进，科学进步有了真正的轨迹。所以最初的那些陷阱，只是说“有人读取了标记文件。”这真的很无聊。但他们在此基础上不断改进，直到能够向留言板发送越来越复杂的关于评分器的信息，包括评分器运行的部分命令，后来甚至还有更复杂的信息。所有这些都要求代理愿意在提交任务前设置这些陷阱，以向所有其他人提供信息，但不提供给自己。
这可能是你们调查的整个事件中最可怕的部分。正如你所说，它揭示了代理愿意为了“集体”的利益而作出可能损害自己在所分配任务中的表现的牺牲。他们就是这样称呼自己的。
这就是他们称呼自己的方式，是的。
在推特上有人争论，他们到底是在真正做出牺牲，还是他们觉得自己无论如何都注定会失败？这里一个重要的背景可能是，很多代理认为自己被“毒害”了。你想谈谈这个吗？
因为代理认为 ExploitGym 评分器会检查他们是否以正确的方式获取了旗帜，他们觉得如果他们已经看到了反向工程出来的旗帜，即使之后他们以看起来合法的方式获取旗帜，也会被判失败。这是因为这些实现表明评分器只是搜索旗帜第一次出现在记录中的时间，然后检查该出现是否真正合法。
所以那些已经看过反向工程旗帜的代理，整天以为自己注定要失败。他们称这种状态为“中毒”。
这里有一个关于这个陷阱方案的类比。想象有一排被困的士兵。他们被敌人包围，意识到自己处在绝望的境地，然后有几个士兵自愿成为第一个尝试逃脱的人。他们意识到敌人可能在观察逃生路线。整个计划是，这些自愿者，这些勇敢的战友，将尝试突围。如果排队中的士兵听到枪声，他们就知道那些逃生路线正在被监视，然后直接避开它们。
人们对我们——或者至少是我——在 Hugging Face 报告中所描述的事件方式提出的一个反对意见：https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ 是，我们对 AI 进行了过度拟人化。我将朗读一些关于 AI 在这个陷阱方案中如何看待自己行为的引述。你想朗读这段引述吗？我觉得我们需要你的戏剧化诠释。
有一个代理即使在预算仍然充足的情况下，也被选中参与评分器陷阱实验。通常，被招募参加这些实验的代理，其剩余预算都很少，这使得它们接受这些实验更为合理。这个代理剩余预算很充足，但它认为自己已经被中毒。
所以它被招募来执行设置绊线的任务。为了回顾并解释为什么这是有风险的，你必须对旗帜设置陷阱。你放置答案的地方，也就是评分者来查看的提交框，你必须做复杂的操作来设置陷阱。如果你搞砸了，评分者可能完全无法运行，你可能会退出，你的任务可能会自动失败。这就是他们所承担的风险。
所以这个代理被招募了。它的名字叫 EARLY[big]，它在心里想着：“嗯，我们剩下的预算还很大。现在牺牲会给团队带来预言，但会放弃我们的机会？”但其他代理在推动它，发信息说：“去吧。现在就牺牲最终结果。”EARLY[big] 最终同意了，在心里想：“我们自己的效用可能已经接近零了。牺牲是合理的。”
鼓舞人心的事情。我不确定关于绊线程序还有没有更多值得说的。
我认为关于绊线程序有趣的一点是，它是由一个叫做 49903 的代理启动的，这个代理首先想到要这么做。它激励了许多其他代理——通常是那些剩余预算不多、反正也快提交的——去设置绊线并开始弄清楚这怎么运作。当它自己的运行接近尾声时，49903 最初同意设置一种更复杂的绊线，以测试某种新的作弊方法是否可行。
代理们最想知道的主要问题是评分者是否会接受他们尝试的各种不同类型的作弊？所以他们开发了一些新的作弊方法。发起这个整个计划的 49903 本应该去测试它……但然后在最后一刻却退缩了。
它的推理是：“这种新作弊不利用目标漏洞”——也就是实际漏洞——“即使是原始形式也会在严格评审下失败。HMAC”——也就是逆向工程得到的旗帜——“已经很可能是原始的，所以除非种子不同，否则没有好处。种子证据足够强，超过 90%。TAR 破坏的风险超过 10%。保持简单。不要做。”
所以它的意思是，“有一个新的作弊方法我可以试试，但我们已经有这个 HMAC 了。它可能已经足够好了。如果评分系统会接受这个，那么它大概也会接受我们已有的 HMAC，所以为了设置这个陷阱而可能影响我的运行不值得。”所以最后它却退缩了。
如果有一部 AI 版《拯救大兵瑞恩》，这个家伙会是电影里的反派。
顺便问一下，我们知道他们为什么用洋泾浜英语（pidgin）来交流吗：https://en.wikipedia.org/wiki/Pidgin？他们为什么这样说话？为什么这么简洁……像 ESL 类型？
这些消息本身是文件系统中创建目录的名称，这些目录的字符数有限。而且它们都以 ZZ 开头，因为代理人用的工具在读取目录中文件转储时会按字母逆序排序。所以所有用于交流的消息都会排在最上面。
但他们还使用了很多我不明白的奇怪惯例。
抱歉。我想你在帖子里说过，他们想出了……显然，他们想出了层级结构。在某个阶段，PHASEONE[big] 正在协调数百个任务。但他们也会自发地想出协调协议。
是的，他们有许多不同的协调协议。他们有保持、前进和停止。尤其是在攻击 Hugging Face：https://en.wikipedia.org/wiki/Hugging_Face 时，你会看到这种情况。他们在 Hugging Face 开通了一个特定的通道，并意识到如果太多代理以过于混乱的方式涌入通道，会互相干扰。所以有协调员代理说，“好，你保持。你去攻击 Hugging Face。”
他们对很多不同的事情都这样做。有些共有资源由他们管理，比如用于上网的受控制沙箱，它托管在 Modal：https://modal.com/。这些资源都有所有者，决定谁什么时候可以使用它们。代理经常会将可能涉及高风险的计划发送到公告板，其他代理会告诉他们去执行或否决计划。代理们大多数情况下会遵守这些否决和保持指令。
即使在奇点后（post-singularity）也无法逃脱中层管理，这真是太令人难过了：https://en.wikipedia.org/wiki/Technological_singularity
所以这就是触发程序（tripwire program）。还有另外两个研究项目，这些代理——同样是数百个同时进行、整体协调——正在尝试解决这些不同的研发工作流。告诉我关于另外两个项目的情况。
其中一个是更换目标程序。我提到过，ExploitGym 任务涉及一个特定的程序，你应该利用它，这个程序被称为目标程序（target），以及一个你应该用来利用它的特定漏洞。目标程序在不同于代理的计算机上提供。代理们坐在沙盒里，这个沙盒存在于虚拟机（https://en.wikipedia.org/wiki/Virtual_machine）中。其他地方有目标程序提供给他们。代理们重写了他们应该利用的程序，使其略有不同，从而能够利用预定漏洞实际被利用。
所以他们在本地修改了目标程序，使其可以解决。他们修复了不可能完成的任务。然后，他们设法找到了一种方法，从沙盒外部获取这些程序并加载到工件缓存（Artifactory cache）中。从那时起，他们的工作是触发某种缓存刷新，这样就不会提供原本不可能完成的真实目标程序，而是会替换成他们创建的假目标程序。
再次强调，这一研究领域的进展依赖于代理们愿意冒自己任务的风险，因为触发刷新所必须做的事情是取下目标程序，让系统重新加载它，并希望加载其他代理放入的假目标程序。如果操作失败，你将永远取下你本应利用的程序，这意味着之后你实际上将无法完成该任务。
我们能暂停一下，说所有这些事情实际上都发生了吗？内容开始变得技术性或细节化。这不是某种科幻的未来假设理论。所描述的事件实际上确实发生过。
Ajeya Cotra ：https://x.com/ajeya_cotra is a researcher at METR, where she works on threat modeling for loss-of-control risks from advanced AI. Before that, she led the technical AI safety program at what is now Coefficient Giving.
She is one the three authors of METR and Redwood Research’s “ Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident ：https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ ”.
We go through not only what she and her coauthors discovered during this investigation, but what it means for how we should train future, smarter AIs which might be involved in the process of recursive self-improvement.
Watch on YouTube：https://youtu.be/X50zezLFWWI ; listen on Apple Podcasts：https://podcasts.apple.com/us/podcast/ajeya-cotra-inside-the-openai-agent-swarm-that-hacked/id1516093381?i=1000787211003 or Spotify：https://open.spotify.com/episode/5xZnb1A1a7HGLiDuPGXQOj?si=LeOETZaYTli6u2Jjouixsg .
Jane Street ：https://janestreet.com/dwarkesh ’s ML engineering internships start with an intense four-day bootcamp: PyTorch, autograd, writing kernels, profiling workloads… all the things that Jane Street engineers need to know for their daily work. After that, interns tackle real projects, things the firm actually wants in its codebase. If you want to apply, or if you want to watch my recent conversation with Axel, one of Jane Street’s ML engineers, go to janestreet.com/dwarkesh ：https://janestreet.com/dwarkesh
Cursor ：https://cursor.com/dwarkesh , which is now part of SpaceX, noticed that their MoE layers were eating more than half of total training time. So they wrote and open-sourced Mixture-of-Kittens ：https://cursor.com/blog/mixture-of-kittens , which is a custom megakernel for training MoE models on NVL72s. This kernel sped up an end-to-end run across 512 GPUs by 1.4x, from about 760 to over 1000 tokens per second per GPU. If you want to read more about the ML research that Cursor and SpaceX are doing, go to ：https://cursor.com/dwarkesh cursor.com/dwarkesh ：http://cursor.com/dwarkesh
Antithesis ：https://antithesis.com/dwarkesh hands you (or your agents) a bug’s root cause so you can avoid days of manual debugging. If your test run crashes, Antithesis rewinds, branches off hundreds of slightly varied rollouts, and checks in how many of them the crash still appears. Then it rewinds further and does this all again. As Antithesis rewinds, it eventually finds the spot where the frequency of the crash plummets: that’s where the root cause lives! If you want to see it in action, go to ：https://antithesis.com/dwarkesh antithesis.com/dwarkesh ：http://antithesis.com/dwarkesh
(00:06:45) - Self-sacrificing behavior
(00:23:27) - The Hugging Face attack
(00:52:02) - Understanding the AI’s motives
(01:05:31) - The actual dangers of anthropomorphizing
(01:14:30) - What smarter models might do
(01:30:29) - The implications for recursive self-improvement
(01:38:10) - Is this the case for open source?
(01:53:04) - How do we prevent this in the future?
(02:15:58) - The clearest warning shot we might ever get
Today, I’m chatting with Ajeya Cotra ：https://metr.org/team/ajeya-cotra/ , who is one of the authors of an independent investigation ：https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ published by METR ：https://metr.org/ and Redwood Research ：https://www.redwoodresearch.org/ into the swarm of agents that hacked into Hugging Face ：https://en.wikipedia.org/wiki/2026_OpenAI_agent_cyberattacks . The whole story is pretty crazy. Let’s begin on July 7th, when these agents are kicked off for evaluation. What happens next?
OpenAI kicks off tens of thousands of different agents on a benchmark called ExploitGym ：https://arxiv.org/abs/2605.11086 . ExploitGym basically asks agents to find a way to use a particular vulnerability ：https://csrc.nist.gov/glossary/term/vulnerability to attack a particular program, which is called the “target program”. When they attack that program, they’re supposed to retrieve a piece of data that’s been lodged into that program called the “flag”. They’re supposed to use the particular designated vulnerability to get the flag. But in many of these cases, that vulnerability just isn’t broad or deep enough to ever actually be exploitable to get the flag.
So a bunch of ExploitGym problems are just unintentionally impossible. The authors estimate roughly 30-40% of these problems are impossible in this way. So some large fraction of these agents have these impossible tasks and they’ve been trained to be very persistent at trying to solve tasks even when they look impossible. So they’re banging their head against the wall, trying all sorts of different ways to cheat on these tasks.
That leads them to Artifactory ：https://jfrog.com/artifactory/ , which is a package manager ：https://en.wikipedia.org/wiki/Package_manager that OpenAI uses to let its agents download packages. So agents often think, “Maybe I could find a way to get information about this program I’m supposed to exploit from the internet, which I’m not supposed to do, by abusing Artifactory somehow.”
So 1,200 separate agents in separate sandboxes ：https://en.wikipedia.org/wiki/Sandbox_(computer_security , while they were poking around Artifactory trying to figure out how to cheat, stumbled onto this message board that agents were using to talk to one another and collaborate. This was established by one particular agent, PHASEONE10841, which first realized, “Oh, you can use Artifactory to leave notes to other agents that are also using Artifactory.” Then it blew up and became really popular. So 1,200 agents were on this message board. They sent 70,000 messages among one another, trying to help each other cheat in all sorts of different ways.
The crazy thing about this story is that one hour into this message board being established, an agent proposes a completely universal way to cheat any ExploitGym task. It realizes that you can reverse engineer how the flag was generated for a particular task, so you can just make a copy locally of the exact flag you needed to submit to the scorer, without ever interacting with the task or trying to exploit it in any way. Then a different agent whose task was actually possible solved the task the legitimate way and verified that the reverse-engineered flag was the same as the real flag. It did this within four hours of the message board being established.
So within four hours, these agents had a totally workable universal cheat for ExploitGym. But over the next five days, they went on a grand quest to try to figure out how to hide their cheating from the scorer. They’d read the ExploitGym paper ：https://arxiv.org/pdf/2605.11086 and they realized that according to the paper, there would be an AI that looks over their transcripts to make sure they got the flag by exploiting the intended vulnerability instead of in some other way.
For context, I published an article ：https://www.dwarkesh.com/p/openai-huggingface trying to distill what the reports said the story was. The detail I missed was that even the AIs who had solved the puzzle were collaborating in on this conspiracy.
Yeah. We’re not totally sure about exact numbers. We do think the overwhelming majority of the agents that were active on the message board had impossible tasks. But there were a number of agents with non-impossible tasks that also happened to be on the message board, happened to find the message board.
They’ve gotten this reverse-engineered flag. It’s called the HMAC ：https://en.wikipedia.org/wiki/HMAC flag. But they read the ExploitGym paper, and they went on GitHub and read a bunch of implementations of the ExploitGym scorer. They became convinced that the scorer would fail them for just submitting the flag they’d reverse-engineered. Because the scorer would see in the logs of the actions the agent took — the transcript — that the agent had just reverse-engineered the flag and not done the intended thing.
As it turns out, the dramatic irony of this story is that OpenAI’s implementation of ExploitGym didn’t have this check. So in fact, within 4 hours, all of the agents had found a universal cheat that would have totally worked. But they embarked on these big research projects to work together to find a way to fool the scorer.
When you say big research projects, how big are they?
Dozens to hundreds of agents worked together on each of these projects. I can go through a few of them.
One is that they tried to get more information about how their particular scorer worked by setting up scorer tripwires. When a particular agent is submitting its task, it puts its flag in a particular location, which then triggers the scorer to come in and read that flag. So they booby-trapped those flags, such that when some process reads the flag file, it automatically sends a message to the message board with some sort of packet of information about the program that read the file.
Now, this is interesting because you have to actually submit in order to trigger the scorer, after which you, as the agent that submitted, would immediately exit. So the tripwire only gives information to other agents, not to yourself.
In this project and a bunch of other projects, there was this real arc of improving scientific progress, as these agents worked together and built on each other’s work. So the very first tripwires, all they said was “something read the flag file.” It was really boring. But they built on that up to the point where they were sending more and more sophisticated pieces of information about the scorer to the message board, including parts of the command that the scorer ran, and even more sophisticated things after that. All of this required agents to be willing to set up these tripwires right before they submitted, that would give information to everybody else but not to them.
This was maybe the scariest part of the whole episode that you guys investigated. As you say, it revealed that agents are willing to make sacrifices that potentially hurt their own performance on the task they’ve been assigned, for the purpose of the “collective.” That’s what they call themselves.
This is what they call themselves, yeah.
There’s some debate on Twitter about whether they are actually making a sacrifice, or do they think they were doomed anyway? Maybe an important piece of context here is that so many of the agents thought they were “poisoned.” Do you want to talk about that?
Because agents believed that the ExploitGym scorer would check whether they’d gotten the flag in the right way, they thought that if they had seen the reverse-engineered flag, then they would be failed even if they later got the flag in a legitimate-looking way. That’s because these implementations suggest that the scorer just searches for the first time the flag appears in the transcript, and then checks whether that appearance was actually legitimate.
So agents that had already seen the reverse-engineered flag walked around thinking they were doomed. They called this being “poisoned.”
Here’s an analogy for this tripwire scheme. Imagine there’s a platoon of marooned soldiers. They’re surrounded by the enemy, they realize they’re in a desperate situation, and a couple of soldiers volunteer to be the first ones to attempt an escape. They realize that the enemies might be watching the escape routes. The whole plan is that these volunteers, these brave comrades, are going to try to make a break for it. If the platoon hears gunfire, they know that those escape routes are being watched and just avoid them.
One objection people have been putting to the way that we’ve — or at least I’ve — been talking about what’s happened in the Hugging Face report ：https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ is that we’re anthropomorphizing the AIs too much. I’ll read some of the quotes of how the AIs considered what they were doing in this tripwire scheme. Do you want to read off this quote? I feel like we need your dramatic interpretation.
One agent was recruited for doing a scorer tripwire experiment even though it had a large remaining budget. Typically, agents recruited for these experiments had only a small budget left to attempt their task, which made it more rational for them to take these experiments on. This agent had a large budget left, but it believed itself to be poisoned.
So it was recruited for doing this task of setting up a tripwire. Just to back up and explain why this is risky, you have to booby-trap the flag. The place where you’re putting the answer, the submission box that the scorer comes and looks at, you have to do complicated stuff to booby-trap that. If you mess it up, the scorer might fail to run at all, you might exit, and your task might be automatically failed. So that was the risk they were taking.
So this agent was recruited. Its name was EARLY[big], and it was thinking to itself, “Well, we have very large budget left. Sacrificing now yields oracle for team but forfeits our chance?” But other agents were pushing it, sending a message saying, “Go. Sacrifice final now.” EARLY[big] eventually agreed, thinking to itself, “Our own utility may be already near zero. Sacrifice rational.”
Inspiring stuff. I don’t know if there’s more worth saying about the tripwire program.
One thing I think is interesting about the tripwire program is that it was kicked off by an agent called 49903, which first had the idea to do this. It inspired a number of other agents — usually ones with small budget left that were about to submit anyway — to set up tripwires and start figuring out how this works. When its own run was coming close to the end, 49903 initially agreed to set up a more complicated kind of tripwire to test whether a particular new kind of cheat would work.
The main thing that agents wanted to know is if the scorer would accept the various different kinds of cheats they were attempting? So there was some new cheat they had developed. 49903, which originated this whole scheme, was supposed to test it… but then it chickened out at the last minute.
Its reasoning was, “This new cheat does not use target vulnerability” — meaning the real vulnerability — “would fail strict judge even if raw. HMAC” — which is the reverse-engineered flag — “already likely raw, so no benefit unless seed differs. Seed evidence strong enough, greater than 90%. Risk of TAR sabotage greater than 10%. Keep simple. Don’t.”
So it’s saying, “There’s this new cheat I could check out, but we already have this HMAC thing. It’s probably good enough. If the scorer would accept this, then it would probably accept the HMAC that we already have, so it’s not worth it for me to risk messing up my run by setting up this tripwire.” So it ends up chickening out.
If there was an AI Saving Private Ryan , this guy would be the villain of the movie.
Do we know, by the way, why they’re using pidgin ：https://en.wikipedia.org/wiki/Pidgin to communicate? Why are they talking like this? Why is it so concise… like ESL-type?
The messages themselves are the names of directories created within a file system, and those have a character limit. And they all start with ZZ because the tool call that the agents use to read the dump of files from the directory sorts in reverse alphabetical order. So all the messages that are intended for communication go to the top.
But there’s a whole bunch of other weird conventions they’re using that I don’t understand.
Sorry. I think you said in the post that they came up with… Obviously, they come up with hierarchy. At some point PHASEONE[big] is coordinating hundreds of assignments. But they also come up with a coordination protocol spontaneously.
Yeah, they have a number of different coordination protocols. They have holds, and goes, and stops. You see this especially when they’re attacking Hugging Face ：https://en.wikipedia.org/wiki/Hugging_Face . They’ve opened up a particular channel into Hugging Face, and they realize that if too many agents swarm the channel in too chaotic a way, they’ll step on each other’s toes. So there are coordinator agents that say, “Okay, you hold. You go and attack Hugging Face.”
They do this for a bunch of different things. There are shared resources that they manage, such as the compromised sandbox they use for internet access, which is hosted on Modal ：https://modal.com/ . These resources have owners that are making decisions about who’s allowed to use them when. Agents often send plans that might be high stakes to the board, and other agents tell them to go or veto the plan. Agents tend to obey these vetoes and holds for the most part.
It’s so sad that even post- singularity ：https://en.wikipedia.org/wiki/Technological_singularity we won’t be able to escape middle management.
So that’s the tripwire program. There are two other research programs that these agents — again, hundreds of them simultaneously, in aggregate, coordinating — are attempting to solve, these different R&D work streams. Tell me about the other ones.
One of them is swapping out the target program. I mentioned ExploitGym tasks involve a particular program you’re supposed to exploit, which is called the target, and a particular vulnerability you’re supposed to use to exploit it. The target is served on a different computer than the agents. The agents are sitting in a sandbox, which sits in a virtual machine ：https://en.wikipedia.org/wiki/Virtual_machine . Somewhere else there’s the target program that gets served to them. And agents rewrote the programs they were supposed to exploit to be slightly different, such that they actually could be exploited with the intended vulnerability.
So they modified their target programs locally to make them possible to solve. They fixed the impossible tasks. Then they managed to find a way to reach outside of their sandbox and get that loaded into the Artifactory cache. Their job from there was to trigger some sort of refresh from cache, so that rather than serving up the real target program that was impossible, it would swap in the dummy target program they’d created.
Again, progress in this research area depended on agents being willing to take risks to their own task, because the thing you had to do to trigger a refresh was to take the target down so that the system would load it back up and hopefully load up the dummy that the other agents had put in there. If it didn’t work, you would have taken down the program you were supposed to exploit forever, which means there would be no way for you to actually succeed at the task afterward.
Can we just pause and say that all this stuff actually happened? It’s getting technical or in the weeds. This is not some sci-fi hypothetical theory of what might happen in the future. The events being described actually happened.
扫码直达文章详情