We talk about multi-agent, Navier-Stokes, and what the current explosion of maths progress tells us about what happens once you automate AI research.

And we also discuss how we will know if the models are actually aligned before we kick off RSI.

Watch on YouTube：https://youtu.be/6AgOfiZOWiY ; listen on Apple Podcasts：https://podcasts.apple.com/us/podcast/noam-brown-agent-swarms-alignment-recursive-self-improvement/id1516093381?i=1000790373289 or Spotify：https://open.spotify.com/episode/3ngDaNm2UVDH0BsMUYVwcG?si=E_7EYGCoTn-rJbMUJLR7CQ .

Jane Street ：https://janestreet.com/dwarkesh has been interested in AI for a lot longer than you’d think, and not just for trading. In 2011, a full year before AlexNet and over a decade before ChatGPT launched, they hosted the first FOOM Debate between Eliezer Yudkowsky and Robin Hanson on whether AI would lead to an intelligence explosion. Now Jane Street is revisiting the question with a new panel: Daniel Kokotajlo, Ege Erdil, Ryan Greenblatt, and Jaime Sevilla, hosted by Ron Minsky in San Francisco this October. I expect it to be a truly excellent conversation. Register at ：https://janestreet.com/dwarkesh janestreet.com/dwarkesh ：http://janestreet.com/dwarkesh

Grok Bot ：https://x.ai/bot has made handing off work super easy. It runs on its own cloud computer, where it installs the tools it needs to handle tasks end-to-end. For the podcast, we use Grok Bot to help produce our videos. You may have noticed that our ads feature animations of real websites. Getting these pixel-perfect used to mean running a convoluted, multi-step workflow ourselves. Now we just let Grok Bot handle it. Best of all, Grok Bot has learned all of our specs and preferences, so we don’t have to redescribe the task each time! Try Grok Bot for yourself at ：https://x.ai/bot x.ai/bot ：http://x.ai/bot

Antithesis ：https://antithesis.com/dwarkesh gives you the confidence of a giant test suite without actually having to write one. Say you’re doing a major backend refactor: building enough tests to trust it could take weeks. Antithesis solves this by running your software through countless simulated worlds, injecting faults and hunting for failures. On any PR, you can turn a dial to decide exactly how much testing you want. And because every run is fully deterministic, agents can branch off the moment a bug appears, rewind it, inspect memory, and replay it, all while the original test keeps running. Learn more at ：https://antithesis.com/dwarkesh antithesis.com/dwarkesh ：http://antithesis.com/dwarkesh

(00:00:00) – Multi-agent and Navier-Stokes

(00:15:28) – How will AI firms work?

(00:22:02) – What math progress tells us about recursive self improvement

(00:40:22) – Hugging Face and alignment

(01:01:18) – The internal/external model gap

(01:08:34) – Chain of thought is degrading

(01:14:12) – How will we know when alignment is solved?

Today, I’m chatting with Noam Brown ：https://x.com/polynoamial , who is a researcher at OpenAI. He was one of the foundational contributors to what became o1 ：https://en.wikipedia.org/wiki/OpenAI_o1 and the reasoning models. Now he’s working on multi-agent systems ：https://en.wikipedia.org/wiki/Multi-agent_system . Speaking of which, you guys announced last week ：https://openai.com/index/navier-stokes-solution/ that you solved one of the Millennium Prize Problems ：https://en.wikipedia.org/wiki/Millennium_Prize_Problems with a system of 10,000 different AI agents ：https://www.ibm.com/think/topics/ai-agents that spent 130 billion tokens over 88 hours.

One of the reasons I’m interested in talking to you is that you were among the first people, maybe two or three years ago, who were thinking about how the reasoning models would allow us to see into the future. Because if you scale up inference ：https://cloud.google.com/discover/what-is-ai-inference compute, you can see what the base capabilities of the models will be a few years in the future.

I feel like you’re in a similar position now to help us understand what future capabilities will look like, given the enormous scaling of agent sizes that we can do right now.

The way I think about it, when you plot the performance of these reasoning models with test-time compute ：https://openai.com/index/learning-to-reason-with-llms/ on the x-axis and performance on basically any reasoning benchmark on the y-axis, you see a very clear pattern where the longer these models take to think about their answer, the better they do. This is a very natural thing. It’s the same thing with people. If you’re taking the SATs and you have five minutes to go through the entire exam, you’re not going to do very well. If you have five hours, you’re probably going to do a lot better.

The AI models are pretty similar. They’ll spend that time doing this monologue to themselves, figuring things out, going through different cases, ruling out different possibilities, building on some of their previous discoveries.

The problem is that as you push that further and further, you hit a latency bottleneck. You don’t want to sit around for three years waiting for a response. So what you can do is what a lot of people do. They parallelize. They just get a team of people. If you’re going to found a company, you want to get a group of people together so you can go faster. It’s the same thing with these AI models. It helps to just have multiple agents working on something because they can go faster.

So multi-agent is a way of scaling test-time compute in parallel instead of purely serially. It is less efficient, because it’s not like a single agent has all the context to itself. But it is a very effective way of scaling test-time compute if it’s done well.

I’m going to ask a bunch of naive questions. This is an unreleased model, so we haven’t publicly seen how these systems work. I just have a bunch of ways in which I’m confused about what the qualitative properties of such systems are.

I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time. Think about what 130 billion tokens ：https://blogs.nvidia.com/blog/ai-tokens-explained/ are. If it were a single human thinking as a full-time job, stretched back to back, 130 billion tokens would be a human thinking for 4,000 years. Eight hours a day, working a normal work week. Starting from ancient Sumeria up till today, a single sequential human thinking that long, concentrated in 88 hours.

I feel like qualitatively, that is a super important consideration. I’m surprised that there isn’t a bigger parallelization penalty. You can just have 10,000 agents collaborate. Maybe because the agents are better at collaborating than humans might be, they’re going much faster. They can actually productively collaborate at such a big scale. Or maybe there is a big parallelization penalty.

Let’s talk about the parallelization penalty, and then we can talk about the qualitative stuff. The truth is that we don’t have very good science on multi-agent scaling up to this kind of scale. When we released 5.6 ：https://openai.com/index/previewing-gpt-5-6-sol/ , I think that was the first time that we had a proper multi-agent system in our models. We actually did show some plots in the blog post of the scaling performance of multi-agent systems, because we have it as an option. It’s Ultra Mode. The default is four agents, but you can set that higher.

In the plot, we show what the performance looks like on some benchmarks for one agent, for four agents working together, for 16 agents working together. It depends on the benchmark, but for some of the benchmarks, what you see is that if you have four agents working on the problem, it is done twice as fast. Because there are four agents working for half as long, you’re paying 2x more to get an answer twice as quickly. If you go to 16 agents, you see a similar pattern. It’s a little less efficient, but you continue to see that performance.

Is it a linear serial time speedup or a sublinear speedup as you increase the number of parallel agents?

It’s slightly sublinear, though it does depend a lot on the problem. Math, for example, is quite parallelizable. It’s not the most parallelizable thing, but it is very parallelizable. Web search, things like doing a Deep Research ：https://openai.com/index/introducing-deep-research/ report where you have to look through a bunch of sources, is extremely parallelizable. I suspect that something like writing a novel would be very unparallelizable. You would probably not see a big benefit from having 10,000 agents working on a novel together, in the same way that you’d probably not get a big benefit from having 10,000 people work on a novel together.

So the performance does depend on the domain. We do measure it up to 16 or so agents in our published blog posts. The problem is that it’s very hard to push that science to 10,000 agents because it’s just so expensive.

You guys just did it over a weekend.

But that’s one data point. We don’t know how long it would take a single agent to solve Navier-Stokes ：https://en.wikipedia.org/wiki/Navier%E2%80%93Stokes_existence_and_smoothness , because we haven’t done that experiment yet. Maybe we will, but that’s also only one data point.

If we want to do a thorough ablation ：https://en.wikipedia.org/wiki/Ablation_(artificial_intelligence) , the experiments are just too expensive at that scale. So we have to do some kind of methodical science about what happens when you go to 64, 128, 256 or something and get a sense of the behavior. But it’s going to be very hard to push that all the way to 10,000 and know for sure what the benefit was that we actually got from using 10,000 agents versus 1,000.

There’s one thing I want to make clear. The effort to solve a Millennium Prize Problem, this was not due to multi-agent. I wouldn’t even attribute 10% of the credit to multi-agent. The reality is that OpenAI has trained a very powerful model. We can get that model to operate over very long horizons. We can get it to think in parallel.

But at its core, the reason why we’re able to do this is because we just have a general-purpose, very strong model. Things like multi-agent are flashy and new, and that probably gets disproportionate credit for that reason. But the core reason is this is just a very powerful model.

The generalization is quite shocking to me. I don’t know how these systems were trained, but presumably they were trained how RL training ：https://en.wikipedia.org/wiki/Reinforcement_learning happens. You have a bunch of checkable synthetic problems and you do a bunch of RL against them. Nowhere in the training process, I’m guessing, was the model solving anything as ambitious as a Millennium Prize Problem. But the generalization was strong enough that you could have these much easier verifiable problems generalize to this much parallel effort on such a hard problem.

I think that is true. First of all, we do train the model on very hard problems. There is definitely a gap. We see that if we train on some kinds of tasks, it’s able to do tasks that are more ambitious than that.

There is an interesting challenge that as the models become smarter and smarter, a lot of the kinds of questions we can ask them are just too easy. It’s hard to challenge the model. I do think that’s going to be interesting. If I had to make an argument for why you might not see AIs like LLMs ：https://en.wikipedia.org/wiki/Large_language_model go the same path as AlphaGo ：https://en.wikipedia.org/wiki/AlphaGo and AlphaZero ：https://en.wikipedia.org/wiki/AlphaZero and all these kinds of game-playing AIs, it might be this kind of problem.

In things like AlphaZero, where you have self-play ：https://en.wikipedia.org/wiki/Self-play_(reinforcement_learning_technique) , you have an infinite curriculum ：https://en.wikipedia.org/wiki/Curriculum_learning . You’re always playing against an AI that’s equally strong. Whereas for things like training an LLM with reinforcement learning, at least the ways that are out there right now, you give the model a problem and you ask it to solve it. If the problem is so easy that it can just solve it in a second, it’s not really learning anything.

If we run out of problems to challenge it, then that is a plausible scenario where it becomes much harder to make progress. Now, I do think there are ways around that. We haven’t really hit that as a wall yet. I think that if it ever became a serious problem, there would be ways around it. But it is a plausible scenario.

Just for the audience, when you’re referring to AlphaGo or AlphaZero, you’re talking about getting superhuman relatively fast after achieving human-level performance.

If you look at the trajectory of game-playing AIs, like Go ：https://en.wikipedia.org/wiki/Go_(game) , within a span of a year they went from beating a European champion ：https://en.wikipedia.org/wiki/AlphaGo_versus_Fan_Hui — something like number 50 in the world — to beating the world champion ：https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol , to being unimaginably, orders of magnitude stronger than any human alive. It’s possible that in domains like math we see a similar trajectory, but I think there is a very plausible scenario where that doesn’t happen.

I want to understand, if in six months people will have access to multi-agent systems, how should one model what it is like to collaborate with or hire a multi-agent system?

I should start by talking about how these multi-agent systems actually work, which I think is a very different way than a lot of multi-agent systems in other AIs. A lot of people that have approached multi-agents for things like LLMs tend to take this very scaffolded approach. For example, there might be a coordinator agent that delegates work to a bunch of children and gives them a task. The children work on it and then return their answer.

This seems like a very sensible setup, a very sensible scaffold. It definitely helps, but there are a bunch of limitations with these kinds of setups. For example, if in this setup you have a coordinator that’s sending tasks to children, and the children work on it and then return their answers, what happens if two children are given similar tasks? Can they talk to each other? Usually the answer is no. That’s very inefficient.

If you’re given a task and it’s actually really helpful to talk to somebody that might know an answer to a question that you’re working on — or part of something that you’re working on — it’d be really helpful for you to just be able to ping them and say, “Hey, can you help me out with this thing?” But a lot of systems don’t have that setup. Adding it significantly increases the complexity of the scaffold that you have.

Another thing is, what if the child doesn’t really understand or has a clarification question? Then it has to choose between, “Okay, do I just return and ask the question instead of solving the problem?” or “Do I solve the problem, make an assumption about what the parent wanted me to do, and just solve it that way?” In any scaffold that people come up with, there are always limitations involved. The approach that we wanted to take was to just go toward the extreme end of baking in as little structure as we could and give the agents very primitive tools to use, and they figure out for themselves how to use them effectively. So we give the agents the ability to message another agent, and when it messages another agent, it is inserted into the context. It can do a few other similar things, but that’s basically the core of it. It can just send a message whenever it wants — just a tool call — and it can send that to other agents.

They figure out for themselves the best way to coordinate around that. It turns out that if this is done well, you get very sophisticated behavior. To me, it looks a lot like how human collaborators work over something like Slack, for example.

When we were working on this project, it was really exciting when we finally got it working to see these agents working on problems together. I remember one example. We give the agents a problem, and then one agent says, “I think I’ve got the answer.” Then another agent says, “Actually, I got a different answer.” Then they have this whole discussion about, “Well, how did you arrive at that answer? Can you explain it to me?” Going back and forth and trying to clarify what could’ve been wrong in each other’s reasoning.

Then they finally converge on, “Oh, yeah. Okay, that seems right.” Then it just broadcasts to the other agents, “Actually, I’ve changed my answer. I think he’s right.” It just felt like a very natural conversation.

It felt like when you see chain of thought ：https://research.google/blog/language-models-perform-reasoning-via-chain-of-thought/ for the first time that’s trained through reinforcement learning, and you’re like, “Oh, this is just kind of like what a person would think if they were writing down their thoughts as they’re thinking them.” It felt like that. It is really cool to see this kind of behavior. Collaborating with these things, honestly, feels a lot like collaborating with a person. It’s just a very natural flow.

Except one qualitative difference that might become salient in the future is that these systems will be thinking maybe more than 10x as fast, if you just look at how many tokens per second they output versus how fast a human talks. They’re working all the time. They’re not sleeping. They’re collaborating with each other at a much more intense pace than humans have the capacity to collaborate with other humans.

I’m trying to think of what to qualitatively expect in a year. Is it like a shadow organization that is moving 100x faster in my company than the human level is? What would take a human organization a year to do is happening within a week within this shadow organization?

Will it feel foreign? I don’t know. I’ve actually found that it’s surprisingly natural to work with these things right now. I think that could change. For example, we have these ultra-fast modes that enable sampling to be 10-15x faster or whatever. Then it’s going to be pretty hard to keep up with these things. The idea is that these agents, when they’re communicating with each other, can go super fast. But they also understand when they’re talking to an agent versus when they’re talking to a person, and their behavior will be different in those situations.

The main example that we have publicly of sophisticated multi-agent systems is unfortunately the Hugging Face one ：https://www.dwarkesh.com/p/openai-huggingface . A lot of things I found concerning there, obviously. But the thing I found interesting there is the spontaneous emergence of hierarchy, of middle management. It sounds like you’re saying this level of organization emerges spontaneously from training?

The details are spontaneous. But while we’re giving a lot of flexibility to the agents to decide how to communicate with each other in the optimal way, we are still giving them a starting point. We’re giving them a prior about what reasonable communication might look like. They’re also trained on a lot of human text. They have an understanding of how humans organize and coordinate, so that’s all baked in.

I think it is surprising the way they’re able to polish this. If you look at what it starts out at, it’s not very sophisticated behavior. In fact, it’s actually very difficult to get these agents to coordinate in a productive way, because it’s very tempting for them to just collapse to, “Oh, we’re all just going to solve the problem independently.” That is a local minimum ：https://en.wikipedia.org/wiki/Local_optimum that you can get stuck in. But if it’s done well, they can end up coordinating very effectively in these kinds of very structured ways.

I wrote this essay a couple of years ago about what automated firms will look like ：https://www.dwarkesh.com/p/ai-firm . I was thinking about, if you had fully automated firms of, let’s say, human-level intelligences, what is different about the nature of AI minds that would make the organizations AIs form different? There are a couple of very important differences. For example, AIs can share context much more seamlessly than humans can. They can merge their knowledge much more seamlessly. Also, you can spin up or spin down an arbitrary number of instances which have the right knowledge.

So if you want to hire more people, it’s not all the schlep of finding the right talent or whatever. Your best talent, you can just make infinite copies of them. Or if you don’t need them for the task anymore, you can spin them down. You can replicate the most effective parts of your organization, or replicate whole organizations together which are effective. Where do you see these multi-agent systems going a year from now or two years from now?

It’s a great question: how do these things actually differ from working with a human coworker? You highlighted some. One really interesting thing is that if you have a person and you want two copies of them, you can’t just clone the person. But with AIs, it’s actually really easy to just say, “Okay, just fork yourself,” and then have both copies work on this thing and then merge back together. We already have this, I think, in multi-agent for Astra ：https://en.wikipedia.org/wiki/GPT-6_Astra and 5.6 Sol, where when they spin up sub-agents, the context is just forked. So it has all the context that’s relevant.

There are other interesting ways where the agents will differ from people. Like, what are some reasons why startups disrupt incumbents ：https://en.wikipedia.org/wiki/Disruptive_innovation ? There are a few factors. One is that they’re willing to take more risks. But another major factor is, as organizations grow in size, you see increasing misalignment between the individuals in the organization.

If you have a startup with five people and each person has a 20% share in the company, they’re all highly aligned to the company succeeding. If you have a massive company with 10,000 people, you see a lot more instances where people are territorial, or just care about getting a lot of headcount for their project or their team, building their fiefdoms, getting a lot of resources so that they can publish cool work or whatever and get promoted. This is actually a real detriment. I think this explains a lot of why startups are able to disrupt incumbents.