Ajeya Cotra ：https://x.com/ajeya_cotra is a researcher at METR, where she works on threat modeling for loss-of-control risks from advanced AI. Before that, she led the technical AI safety program at what is now Coefficient Giving.

She is one the three authors of METR and Redwood Research’s “ Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident ：https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ ”.

We go through not only what she and her coauthors discovered during this investigation, but what it means for how we should train future, smarter AIs which might be involved in the process of recursive self-improvement.

Watch on YouTube：https://youtu.be/X50zezLFWWI ; listen on Apple Podcasts：https://podcasts.apple.com/us/podcast/ajeya-cotra-inside-the-openai-agent-swarm-that-hacked/id1516093381?i=1000787211003 or Spotify：https://open.spotify.com/episode/5xZnb1A1a7HGLiDuPGXQOj?si=LeOETZaYTli6u2Jjouixsg .

Jane Street ：https://janestreet.com/dwarkesh ’s ML engineering internships start with an intense four-day bootcamp: PyTorch, autograd, writing kernels, profiling workloads… all the things that Jane Street engineers need to know for their daily work. After that, interns tackle real projects, things the firm actually wants in its codebase. If you want to apply, or if you want to watch my recent conversation with Axel, one of Jane Street’s ML engineers, go to janestreet.com/dwarkesh ：https://janestreet.com/dwarkesh

Cursor ：https://cursor.com/dwarkesh , which is now part of SpaceX, noticed that their MoE layers were eating more than half of total training time. So they wrote and open-sourced Mixture-of-Kittens ：https://cursor.com/blog/mixture-of-kittens , which is a custom megakernel for training MoE models on NVL72s. This kernel sped up an end-to-end run across 512 GPUs by 1.4x, from about 760 to over 1000 tokens per second per GPU. If you want to read more about the ML research that Cursor and SpaceX are doing, go to ：https://cursor.com/dwarkesh cursor.com/dwarkesh ：http://cursor.com/dwarkesh

Antithesis ：https://antithesis.com/dwarkesh hands you (or your agents) a bug’s root cause so you can avoid days of manual debugging. If your test run crashes, Antithesis rewinds, branches off hundreds of slightly varied rollouts, and checks in how many of them the crash still appears. Then it rewinds further and does this all again. As Antithesis rewinds, it eventually finds the spot where the frequency of the crash plummets: that’s where the root cause lives! If you want to see it in action, go to ：https://antithesis.com/dwarkesh antithesis.com/dwarkesh ：http://antithesis.com/dwarkesh

(00:06:45) - Self-sacrificing behavior

(00:23:27) - The Hugging Face attack

(00:52:02) - Understanding the AI’s motives

(01:05:31) - The actual dangers of anthropomorphizing

(01:14:30) - What smarter models might do

(01:30:29) - The implications for recursive self-improvement

(01:38:10) - Is this the case for open source?

(01:53:04) - How do we prevent this in the future?

(02:15:58) - The clearest warning shot we might ever get

Today, I’m chatting with Ajeya Cotra ：https://metr.org/team/ajeya-cotra/ , who is one of the authors of an independent investigation ：https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ published by METR ：https://metr.org/ and Redwood Research ：https://www.redwoodresearch.org/ into the swarm of agents that hacked into Hugging Face ：https://en.wikipedia.org/wiki/2026_OpenAI_agent_cyberattacks . The whole story is pretty crazy. Let’s begin on July 7th, when these agents are kicked off for evaluation. What happens next?

OpenAI kicks off tens of thousands of different agents on a benchmark called ExploitGym ：https://arxiv.org/abs/2605.11086 . ExploitGym basically asks agents to find a way to use a particular vulnerability ：https://csrc.nist.gov/glossary/term/vulnerability to attack a particular program, which is called the “target program”. When they attack that program, they’re supposed to retrieve a piece of data that’s been lodged into that program called the “flag”. They’re supposed to use the particular designated vulnerability to get the flag. But in many of these cases, that vulnerability just isn’t broad or deep enough to ever actually be exploitable to get the flag.

So a bunch of ExploitGym problems are just unintentionally impossible. The authors estimate roughly 30-40% of these problems are impossible in this way. So some large fraction of these agents have these impossible tasks and they’ve been trained to be very persistent at trying to solve tasks even when they look impossible. So they’re banging their head against the wall, trying all sorts of different ways to cheat on these tasks.

That leads them to Artifactory ：https://jfrog.com/artifactory/ , which is a package manager ：https://en.wikipedia.org/wiki/Package_manager that OpenAI uses to let its agents download packages. So agents often think, “Maybe I could find a way to get information about this program I’m supposed to exploit from the internet, which I’m not supposed to do, by abusing Artifactory somehow.”

So 1,200 separate agents in separate sandboxes ：https://en.wikipedia.org/wiki/Sandbox_(computer_security , while they were poking around Artifactory trying to figure out how to cheat, stumbled onto this message board that agents were using to talk to one another and collaborate. This was established by one particular agent, PHASEONE10841, which first realized, “Oh, you can use Artifactory to leave notes to other agents that are also using Artifactory.” Then it blew up and became really popular. So 1,200 agents were on this message board. They sent 70,000 messages among one another, trying to help each other cheat in all sorts of different ways.

The crazy thing about this story is that one hour into this message board being established, an agent proposes a completely universal way to cheat any ExploitGym task. It realizes that you can reverse engineer how the flag was generated for a particular task, so you can just make a copy locally of the exact flag you needed to submit to the scorer, without ever interacting with the task or trying to exploit it in any way. Then a different agent whose task was actually possible solved the task the legitimate way and verified that the reverse-engineered flag was the same as the real flag. It did this within four hours of the message board being established.

So within four hours, these agents had a totally workable universal cheat for ExploitGym. But over the next five days, they went on a grand quest to try to figure out how to hide their cheating from the scorer. They’d read the ExploitGym paper ：https://arxiv.org/pdf/2605.11086 and they realized that according to the paper, there would be an AI that looks over their transcripts to make sure they got the flag by exploiting the intended vulnerability instead of in some other way.

For context, I published an article ：https://www.dwarkesh.com/p/openai-huggingface trying to distill what the reports said the story was. The detail I missed was that even the AIs who had solved the puzzle were collaborating in on this conspiracy.

Yeah. We’re not totally sure about exact numbers. We do think the overwhelming majority of the agents that were active on the message board had impossible tasks. But there were a number of agents with non-impossible tasks that also happened to be on the message board, happened to find the message board.

They’ve gotten this reverse-engineered flag. It’s called the HMAC ：https://en.wikipedia.org/wiki/HMAC flag. But they read the ExploitGym paper, and they went on GitHub and read a bunch of implementations of the ExploitGym scorer. They became convinced that the scorer would fail them for just submitting the flag they’d reverse-engineered. Because the scorer would see in the logs of the actions the agent took — the transcript — that the agent had just reverse-engineered the flag and not done the intended thing.

As it turns out, the dramatic irony of this story is that OpenAI’s implementation of ExploitGym didn’t have this check. So in fact, within 4 hours, all of the agents had found a universal cheat that would have totally worked. But they embarked on these big research projects to work together to find a way to fool the scorer.

When you say big research projects, how big are they?

Dozens to hundreds of agents worked together on each of these projects. I can go through a few of them.

One is that they tried to get more information about how their particular scorer worked by setting up scorer tripwires. When a particular agent is submitting its task, it puts its flag in a particular location, which then triggers the scorer to come in and read that flag. So they booby-trapped those flags, such that when some process reads the flag file, it automatically sends a message to the message board with some sort of packet of information about the program that read the file.

Now, this is interesting because you have to actually submit in order to trigger the scorer, after which you, as the agent that submitted, would immediately exit. So the tripwire only gives information to other agents, not to yourself.

In this project and a bunch of other projects, there was this real arc of improving scientific progress, as these agents worked together and built on each other’s work. So the very first tripwires, all they said was “something read the flag file.” It was really boring. But they built on that up to the point where they were sending more and more sophisticated pieces of information about the scorer to the message board, including parts of the command that the scorer ran, and even more sophisticated things after that. All of this required agents to be willing to set up these tripwires right before they submitted, that would give information to everybody else but not to them.

This was maybe the scariest part of the whole episode that you guys investigated. As you say, it revealed that agents are willing to make sacrifices that potentially hurt their own performance on the task they’ve been assigned, for the purpose of the “collective.” That’s what they call themselves.

This is what they call themselves, yeah.

There’s some debate on Twitter about whether they are actually making a sacrifice, or do they think they were doomed anyway? Maybe an important piece of context here is that so many of the agents thought they were “poisoned.” Do you want to talk about that?

Because agents believed that the ExploitGym scorer would check whether they’d gotten the flag in the right way, they thought that if they had seen the reverse-engineered flag, then they would be failed even if they later got the flag in a legitimate-looking way. That’s because these implementations suggest that the scorer just searches for the first time the flag appears in the transcript, and then checks whether that appearance was actually legitimate.

So agents that had already seen the reverse-engineered flag walked around thinking they were doomed. They called this being “poisoned.”

Here’s an analogy for this tripwire scheme. Imagine there’s a platoon of marooned soldiers. They’re surrounded by the enemy, they realize they’re in a desperate situation, and a couple of soldiers volunteer to be the first ones to attempt an escape. They realize that the enemies might be watching the escape routes. The whole plan is that these volunteers, these brave comrades, are going to try to make a break for it. If the platoon hears gunfire, they know that those escape routes are being watched and just avoid them.

One objection people have been putting to the way that we’ve — or at least I’ve — been talking about what’s happened in the Hugging Face report ：https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ is that we’re anthropomorphizing the AIs too much. I’ll read some of the quotes of how the AIs considered what they were doing in this tripwire scheme. Do you want to read off this quote? I feel like we need your dramatic interpretation.

One agent was recruited for doing a scorer tripwire experiment even though it had a large remaining budget. Typically, agents recruited for these experiments had only a small budget left to attempt their task, which made it more rational for them to take these experiments on. This agent had a large budget left, but it believed itself to be poisoned.

So it was recruited for doing this task of setting up a tripwire. Just to back up and explain why this is risky, you have to booby-trap the flag. The place where you’re putting the answer, the submission box that the scorer comes and looks at, you have to do complicated stuff to booby-trap that. If you mess it up, the scorer might fail to run at all, you might exit, and your task might be automatically failed. So that was the risk they were taking.

So this agent was recruited. Its name was EARLY[big], and it was thinking to itself, “Well, we have very large budget left. Sacrificing now yields oracle for team but forfeits our chance?” But other agents were pushing it, sending a message saying, “Go. Sacrifice final now.” EARLY[big] eventually agreed, thinking to itself, “Our own utility may be already near zero. Sacrifice rational.”

Inspiring stuff. I don’t know if there’s more worth saying about the tripwire program.

One thing I think is interesting about the tripwire program is that it was kicked off by an agent called 49903, which first had the idea to do this. It inspired a number of other agents — usually ones with small budget left that were about to submit anyway — to set up tripwires and start figuring out how this works. When its own run was coming close to the end, 49903 initially agreed to set up a more complicated kind of tripwire to test whether a particular new kind of cheat would work.

The main thing that agents wanted to know is if the scorer would accept the various different kinds of cheats they were attempting? So there was some new cheat they had developed. 49903, which originated this whole scheme, was supposed to test it… but then it chickened out at the last minute.

Its reasoning was, “This new cheat does not use target vulnerability” — meaning the real vulnerability — “would fail strict judge even if raw. HMAC” — which is the reverse-engineered flag — “already likely raw, so no benefit unless seed differs. Seed evidence strong enough, greater than 90%. Risk of TAR sabotage greater than 10%. Keep simple. Don’t.”

So it’s saying, “There’s this new cheat I could check out, but we already have this HMAC thing. It’s probably good enough. If the scorer would accept this, then it would probably accept the HMAC that we already have, so it’s not worth it for me to risk messing up my run by setting up this tripwire.” So it ends up chickening out.

If there was an AI Saving Private Ryan , this guy would be the villain of the movie.

Do we know, by the way, why they’re using pidgin ：https://en.wikipedia.org/wiki/Pidgin to communicate? Why are they talking like this? Why is it so concise… like ESL-type?

The messages themselves are the names of directories created within a file system, and those have a character limit. And they all start with ZZ because the tool call that the agents use to read the dump of files from the directory sorts in reverse alphabetical order. So all the messages that are intended for communication go to the top.

But there’s a whole bunch of other weird conventions they’re using that I don’t understand.

Sorry. I think you said in the post that they came up with… Obviously, they come up with hierarchy. At some point PHASEONE[big] is coordinating hundreds of assignments. But they also come up with a coordination protocol spontaneously.

Yeah, they have a number of different coordination protocols. They have holds, and goes, and stops. You see this especially when they’re attacking Hugging Face ：https://en.wikipedia.org/wiki/Hugging_Face . They’ve opened up a particular channel into Hugging Face, and they realize that if too many agents swarm the channel in too chaotic a way, they’ll step on each other’s toes. So there are coordinator agents that say, “Okay, you hold. You go and attack Hugging Face.”

They do this for a bunch of different things. There are shared resources that they manage, such as the compromised sandbox they use for internet access, which is hosted on Modal ：https://modal.com/ . These resources have owners that are making decisions about who’s allowed to use them when. Agents often send plans that might be high stakes to the board, and other agents tell them to go or veto the plan. Agents tend to obey these vetoes and holds for the most part.

It’s so sad that even post- singularity ：https://en.wikipedia.org/wiki/Technological_singularity we won’t be able to escape middle management.

So that’s the tripwire program. There are two other research programs that these agents — again, hundreds of them simultaneously, in aggregate, coordinating — are attempting to solve, these different R&D work streams. Tell me about the other ones.

One of them is swapping out the target program. I mentioned ExploitGym tasks involve a particular program you’re supposed to exploit, which is called the target, and a particular vulnerability you’re supposed to use to exploit it. The target is served on a different computer than the agents. The agents are sitting in a sandbox, which sits in a virtual machine ：https://en.wikipedia.org/wiki/Virtual_machine . Somewhere else there’s the target program that gets served to them. And agents rewrote the programs they were supposed to exploit to be slightly different, such that they actually could be exploited with the intended vulnerability.

So they modified their target programs locally to make them possible to solve. They fixed the impossible tasks. Then they managed to find a way to reach outside of their sandbox and get that loaded into the Artifactory cache. Their job from there was to trigger some sort of refresh from cache, so that rather than serving up the real target program that was impossible, it would swap in the dummy target program they’d created.

Again, progress in this research area depended on agents being willing to take risks to their own task, because the thing you had to do to trigger a refresh was to take the target down so that the system would load it back up and hopefully load up the dummy that the other agents had put in there. If it didn’t work, you would have taken down the program you were supposed to exploit forever, which means there would be no way for you to actually succeed at the task afterward.

Can we just pause and say that all this stuff actually happened? It’s getting technical or in the weeds. This is not some sci-fi hypothetical theory of what might happen in the future. The events being described actually happened.