我们正处在一个时代，少数组织正在使用成千上万的并发智能体来改进他们的流程和产出。这些组织恰好是前沿人工智能实验室，特别是OpenAI和Anthropic。在过去的几周里，我一直在思考，对于这些组织中如此多的员工迅速调整他们对人工智能进展速度和相关风险的预期意味着什么。
我有一个核心观点，即前沿实验室以及旧金山人工智能领域更广泛的紧张、竞争激烈的文化，创造了一种放大任何人工智能担忧的环境。这在引起更广泛公众对人工智能的关注方面有一定好处，因为恐惧能促销，但夸大风险时间表或严重性会产生负面的二阶效应。我记得在2023年和2024年，有许多喧嚣的人工智能安全讨论，以及它们对开源人工智能可行性的相关阴影——当时的主要风险并没有按预测的时间出现。
即使在一年前，这两个关键实验室的普通员工对人工智能的风险和进展速度已经非常焦虑，尤其是在2026年初，当智能体在产品市场契合度上变得更强时。当这一文化预条件暴露于成千上万的智能体将持续、高效地在你的业务中工作这一现实时，只会增加这种焦虑。从这种焦虑，以及像OpenAI-HuggingFace这样的事件，到灭绝风险的跨越，感觉非常宗教化。
分享：https://www.interconnects.ai/p/where-i-stand-on-rsi?utm_source=substack&utm_medium=email&utm_content=share&action=share
Richard Ngo 对此情形有一个恰当的总结：https://www.lesswrong.com/posts/FuGfR3jL3sw6r8kB4?commentId=AtXHripmWKEHTvPNs&utm_source=chatgpt.com
现在，人工智能安全社区中的很大一部分人，显性或隐性地将未来定位于智能爆炸在几年内发生的情景。我的默认预期（在没有广泛暂停的情况下）是类似的事情会发生：他们方向上是正确的（相对于几乎所有未关联社区的人而言的预期），但事实是错误的。具体来说，我们在未来8年内不会出现超级智能，但事情仍会进展得如此之快，以至于“感觉”上支持短时间表的人是正确的。
… 我现在想说点什么，因为感觉围绕“奇点即将到来”的从众情绪已经变得相当疯狂。
就个人而言，我认为这种观点与我在替代递归自我提升（RSI）的情景中提出的观点高度一致，我称之为“有损自我提升”：https://www.interconnects.ai/p/lossy-self-improvement。这个观点的总结是：
可自动化的研究过于狭窄，难以在规模定律的指数成本面前带来巨大的净进展加速，
并行更多的 AI 代理的收益递减是真实存在的，
资源瓶颈和政治因素是构建强大大语言模型（LLM）的主要因素（AI 在加速这一过程方面能做的事情有限）。
所以，我在权衡上述情况、文化温度的潜在提升以及实验室可能已经看到的尚未公开的真正令人害怕的具体突破之间找到平衡。我的预期是，目前的 AI 安全关注更多在前者——即扩展代理的工作——但我对这方面保持高度不确定性。基于想象力的基础性 AI 突破，是那种会让我将 RSI 时间线从接近于在指数成本（规模定律）面前维持进展的工具，更新为更不可预测和/或不稳定的东西。
关于RSI的一些近期最佳资源是Dwarkesh与Noam Brown的播客：https://www.dwarkesh.com/p/noam-brown，以及三人组：https://www.dwarkesh.com/p/john-beren-charlie，成员包括John Schulman：http://joschu.net/，Beren Millidge：https://www.beren.io/ 和Charlie O’Neill：https://charlesponeill.com/。我从他们那里获得了一些重要的思考。
首先，与诺姆·布朗的播客让我意识到短期内推理能力加速增长的规模。这些实验室会投入数千个代理去解决重要且可衡量的问题。同时，他们可用的计算能力也会持续增长。我对实验室是否能够在总计算量增加时仍然将固定比例的计算投入到内部研发持怀疑态度，尤其是在计划上市时，因为他们会受到基础经济问题的更多审查。重要的是不要将推理时间规模的巨大跃进——一个相对可预测的动态——与高度不确定的RSI输出混淆。
其次，关于技术能力现状的三人播客引发了更出乎意料的反应，我还没有完全整理清楚。在播客的前一小时左右，他们讨论强化学习、蒸馏、规模扩展、推理时间计算等的作用时，我发现自己非常认同他们的观点分布。简而言之，我们当前的技术是有效的，可以解决我们已知如何陈述的问题，但在大多数部分可验证领域中，它们并不会产生对未知、更困难问题的魔力般的泛化能力（数学上的进展是例外，而非规律）。
这个播客令人意外的部分是结尾，他们预测了AI达到各个能力阈值的时间线。我让GPT-6-Astra总结了Dwarkesh提出的三个类似“AI什么时候会达到X能力”的问题的答案：
所有时间线都是相对于采访日期的。
可以作为一个月的远程代班白领工作者
查理·奥尼尔：如果可以程序化访问工作工具，大约1年；如果必须通过浏览器操作，大约2年。这意味着普通白领工作，而非高度创新的研究。
贝伦·米利奇：全面泛化大约需要3年；80–90%的覆盖率会更早实现。主要不确定因素：在线学习和长尾任务。
约翰·舒尔曼：大约1年可得到一个“还行”的版本，能力不均衡，但会随时间改进。
人工智能研究人员生产力提升10倍
查理·奥尼尔：5–10年。瓶颈：吸收信息并决定接下来进行哪个实验。
Beren Millidge：认为 John 约 2 年的估计是合理的，但没有提供独立的时间表。假设 AI 可以进行连续实验并从反馈中学习；其他瓶颈仍然存在。
AI 超越顶尖人类专家完成所有基于计算机的工作，包括多年项目（“ASI”）
Charlie O’Neill：5–10 年。强调记忆和上下文长度的限制。
Beren Millidge：实验室重点领域约 5 年；对所有领域可能更长。没有给出通用版本的明确时间表。
John Schulman：3–4 年。空间/物理领域可能需要更长时间；需要入门培训并解决长期学习问题。
大致来说，讨论 RSI 时反复出现的问题是缺乏对智能的具体定义。智能的参差不齐意味着我们需要在具体、可衡量的任务中讨论门槛。LLM 的智能本质与人类大不相同，而我们预测的角色是以人为形态的。因此，AI 不会像远程工作人员或 AI 研究员那样明确地跨越这些门槛。这是一个缓慢的扩散过程，并且长尾形式将始终存在。
以 AI 研究员的生产力为例。许多人低估了科学中与同事沟通和制定标准的比例。我确实相信在不久的将来实验设计和测试的周期会快 10 倍，但假设生成和直觉构建不会加速。加速理解将是关键瓶颈——尽管所有 AI 工具得到大幅提升，人类的能力仅会略微改善。科学性质的大幅改善，将是使人类能够投入更多时间，而不是人类在科学能力上指数级提升。
发表评论：https://www.interconnects.ai/p/where-i-stand-on-rsi/comments
这与Noam播客有关。未来不久，智能体群将在解决具有明确、可验证答案的开放问题方面非常有效。在这一方面，谈到改进AI模型时，RSI在提高效率方面比在提升峰值智能上更有帮助。这是因为LLM服务有明确的可衡量和可调节的指标，这将使推理时的扩展更加优化，并使多智能体系统更高效。
尽管如此，我仍然无法忽视这样一个事实：我们所有的扩展规律都显示，要使智力线性提高，需要指数级的计算和资源。RSI有望使现代LLM大幅降本。数据显示，在给定智力水平下，LLM成本呈指数下降的趋势可能会加速：https://epoch.ai/data-insights/llm-inference-price-trends?utm_source=chatgpt.com。对于实验室来说，一个关键因素是提高利润率，因为如果在固定智力水平下降低价格的竞争非常激烈，收入可能会受到负面压力——Jevons悖论：https://en.wikipedia.org/wiki/Jevons_paradox 可能会占上风，从而造就强大的企业。
RSI因素在改善LLM难题的某些部分上会更困难，比如管理复杂的后训练方案。我非常同意John Schulman关于实验室后训练状态的一些话：
如果我考虑一个后训练团队，以及为什么团队需要很多人，仅仅是因为有很多不同的领域需要弄清楚模型应该如何表现。要自动化整个过程非常困难，仅仅因为有人必须思考模型在某个领域应该如何表现。
在某些方面搞砸后训练非常容易，而这些错误在基准测试中可能不会显现。
这些任务对目前的LLM来说非常困难。是的，随着行业仍在快速扩展与这些领域相关的RL环境，它们会变得更好，但这一模式不会永远持续。在不久的将来，构想、构建和测试能够真正挑战领先LLM的新环境可能会变得指数级困难——这些困难环境是强化学习中作为学习信号至关重要的部分。
OpenAI：https://openai.com/index/research-acceleration-view-inside-openai/ 以及 Anthropic：https://www.anthropic.com/institute/measuring-pace-of-ai-development 已经分享了大量与 RSI 相关的内部测量数据，我目前的理解是，实验室中自动化最大的突破出现在软件工程、日志监控、计划实验管理以及其他相对例行（但不总是容易）的任务中。例如，我在最近的 Claude Fable 5.1 & Mythos 5.1 系统卡中看到以下表述时感到惊讶：https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf：
我们相信，近期 AI 模型的内部使用是保持当前进展速度的关键因素，但我们尚未看到明显迹象表明进展速度会出现显著加速。
总体来说，我认为我们正在应对的最难的指数增长是峰值智能。这是最难推动甚至加速的部分。不过，我对于 RSI 非常早期阶段的思维模型更多是将推理阶段的计算能力大规模扩展并扩散到 AI 研究及相关活动中，这里有大量容易获取的成果可用。仅此一点，本身已经具有经济转型的潜力。它还可能释放更多资源以推动 AI 的扩散，而 AI 扩散是释放 AI 潜在收益的关键瓶颈。
目前，并且在更多证据出现之前，有损自我改进：https://www.interconnects.ai/p/lossy-self-improvement 仍然是我对进展轨迹的基本假设，而关于灭绝风险的讨论增加似乎非常不合适。正如往常，AI 的发展变化可能非常迅速。
We’re in an era where a few organizations are using thousands of concurrent agents to improve their processes and output. These organizations happen to be just the frontier AI labs, in particular OpenAI and Anthropic. In the last few weeks, I’ve been pondering what it means for so many employees across these organizations to rapidly update their expectations for the pace of AI progress and associated risks.
A core perspective I have is that the frontier labs and broader frenetic, competitive culture in the San Francisco AI scene set up an environment that amplifies any AI concern. This has some benefits in causing more general audience awareness of AI, as fear sells, but exaggerating risk timelines or severity will have negative second-order effects. I remember many loud AI safety debates, and their associated clouds over the viability of open-source AI, in 2023 and 2024 — the primary risks then did not arrive in the forecasted timelines.
The general populace of these two key labs was very anxious about AI risks and the rate of progress even a year ago, and especially as agents got stronger product-market fit at the start of 2026. This cultural precondition, when exposed to the reality that thousands of agents will constantly be working fairly productively in your business, will only increase this anxiety. The step from this anxiety, and incidents like OpenAI-HuggingFace, to extinction risks feels very religious.
Share ：https://www.interconnects.ai/p/where-i-stand-on-rsi?utm_source=substack&utm_medium=email&utm_content=share&action=share
Richard Ngo had an apt summary：https://www.lesswrong.com/posts/FuGfR3jL3sw6r8kB4?commentId=AtXHripmWKEHTvPNs&utm_source=chatgpt.com of the situation:
Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they’ll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right.
… I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild.
Personally, I think this view aligns closely to what I outlined in my alternate scenario to true recursive self-improvement (RSI), which I called lossy self-improvement：https://www.interconnects.ai/p/lossy-self-improvement . A summary of this view is that:
Automatable research is too narrow to achieve a massive net acceleration in progress, in the face of scaling laws’ exponential costs,
Diminishing returns of more AI agents in parallel are real, &
Resource bottlenecks and politics are a major factor in building strong LLMs (and AI can do much less to accelerate this).
So, I’m left balancing the above, latent increase in the cultural temperature with the potential that the labs have seen genuinely scary, specific breakthroughs that are not public yet. My expectation is that more of the current AI safety concern is on the former – scaled agents working – but I hold high levels of uncertainty here. Foundational, imagination-based AI breakthroughs are the sort of thing that would make me update my RSI timelines from closer to a tool to sustain progress in the face of exponential costs (scaling laws), to something more unpredictable and/or unstable.
Some of the best recent resources on RSI have been Dwarkesh’s podcasts with Noam Brown ：https://www.dwarkesh.com/p/noam-brown and the trio ：https://www.dwarkesh.com/p/john-beren-charlie of John Schulman ：http://joschu.net/ , Beren Millidge ：https://www.beren.io/ and Charlie O’Neill ：https://charlesponeill.com/ . I have a few important reflections from both of them.
First, the podcast with Noam Brown made me internalize how big of a short-term acceleration mass inference capacity is. These labs will throw thousands of agents at important, measurable problems. At the same time, compute capacity available to them is going to continue to scale. I have my doubts that the labs can afford to spend a constant portion of this compute on internal R&D as the total volume goes up, especially with plans to IPO, as they face increased scrutiny on basic economics. It is important to not confuse massive steps in inference-time scaling, a dynamic which should be fairly predictable, with being the outputs of RSI, which is highly uncertain.
Second, the trio podcast debating the state of the art in technical capacities induced more of a surprising reaction that I haven’t fully settled. Through the first hour or so of this podcast, where they debate the role of RL, distillation, scaling, inference-time compute, etc., I found myself strongly agreeing with the distribution of claims. A TLDR would be that our current techniques work and let us solve problems we know how to state, but they don’t result in a magical level of generalization to unknown, harder problems in most partially verifiable domains (i.e. progress in math is an exception, rather than a rule).
The surprise of this podcast was the end, where they were predicting timelines for various thresholds of AI. I had GPT-6-Astra summarize the answers provided to three questions from Dwarkesh, of the form “when will AI reach X ability”:
All timelines are relative to the interview date.
Drop-in remote worker for broad white-collar work over a month
Charlie O’Neill: ~1 year with programmatic access to workplace tools; ~2 years if it must operate through a browser. Means ordinary white-collar work, not highly creative research.
Beren Millidge: ~3 years for full generality; 80–90% coverage sooner. Main uncertainties: online learning and the long tail of tasks.
John Schulman: ~1 year for an “okay” version, with uneven capabilities that improve over time.
10× productivity uplift for AI researchers
Charlie O’Neill: 5–10 years. Bottleneck: absorbing information and deciding which experiment to run next.
Beren Millidge: Finds John’s ~2-year estimate plausible, but gives no independent timeline. Assumes AI can run successive experiments and learn from feedback; other bottlenecks would remain.
AI surpassing top human experts across all computer-based work, including multiyear projects (“ASI”)
Charlie O’Neill: 5–10 years. Highlights limitations in memory and context length.
Beren Millidge: ~5 years for areas labs focus on; potentially longer for literally every domain. Gives no firm timeline for the universal version.
John Schulman: 3–4 years. Spatial/physical fields may take longer; requires onboarding and solving longer-horizon learning.
Roughly, a recurring problem when discussing RSI is a lack of specification in intelligence. The jaggedness of intelligence means that we need to discuss thresholds in specific, measurable tasks. The nature of LLMs’ intelligence is shaped very differently than humans, and the roles we forecast are human-shaped. AIs, therefore, do not cross these thresholds like remote worker or AI researcher discretely. It’s a slow diffusion, and a form of long tail will always exist.
Take the case of productivity of AI researchers. Many people under-index how much of science is communication and standard setting with colleagues. I do buy the cycle of experiment design and testing being 10x faster in the near future, but not hypothesis generation and intuition building. Accelerating understanding will be the key bottleneck – and it is one that despite all of the AI tools getting massively improved, humans will only improve marginally in their capability. A big improvement in the nature of science will be enabling humans to invest more time here, not them becoming exponentially better at it.
Leave a comment ：https://www.interconnects.ai/p/where-i-stand-on-rsi/comments
This links back to the Noam podcast. Agent swarms in the near future will be effective at solving clear, open problems with verifiable answers. In this vein, when it comes to improving AI models, RSI is much more helpful at efficiency rather than expanding peak intelligence. This is due to the fact that LLM serving has clear metrics you want to improve that are measurable and malleable . This’ll enable better inference-time scaling and more efficient multi-agent systems.
Still, I cannot get past the fact that all of our scaling laws show that you need exponential compute and resources to make linear improvements in intelligence. RSI is poised to make modern LLMs vastly cheaper. Trends that have shown LLMs get exponentially cheaper at a given intelligence are likely to accelerate ：https://epoch.ai/data-insights/llm-inference-price-trends?utm_source=chatgpt.com . A crucial factor for the labs will be increasing margins as revenue could potentially have negative pressure if there’s fierce competition in lowering prices at a fixed intelligence level — Jevons paradox ：https://en.wikipedia.org/wiki/Jevons_paradox will likely prevail, resulting in strong businesses.
RSI factors will have a much harder time improving pieces of the LLM puzzle like managing complex post-training recipes. There were a few quotes from John Schulman that I strongly agree with on the state of post-training at the labs:
If I think about a post-training team and why you need a lot of people on the team, it’s just because there are a lot of different areas where you have to figure out how the model should behave. It would be very hard to automate the whole thing, just because someone has to think about how the model should behave in this area.
It’s really easy to screw up post-training in some way that doesn’t show up in benchmarks.
These tasks are uniquely hard for current LLMs. Yes, they’ll get better as the industry is still rapidly scaling RL environments related to these domains, but this paradigm does not last forever. In the near future, it could become exponentially harder to conceive, build, and test new environments that meaningfully challenge the leading LLMs – these hard environments are the ones that are crucial as a learning signal in RL.
OpenAI ：https://openai.com/index/research-acceleration-view-inside-openai/ and Anthropic ：https://www.anthropic.com/institute/measuring-pace-of-ai-development have shared a good amount of internal measurements related to RSI, and my current read is that the biggest takeoff in automation within the labs is in tasks like software engineering, monitoring logs, managing planned experiments, and other fairly routine (but not always easy) tasks. For example, I was surprised by this language in the recent Claude Fable 5.1 & Mythos 5.1 System Card ：https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf :
We believe that internal usage of recent AI models has been a key factor in maintaining the current rate of progress, but we do not yet see clear signs of dramatic acceleration beyond that rate.
Altogether, I think the hardest exponential we are fighting is on peak intelligence. That is the hardest one to budge or even accelerate. Still, my mental model for the very early innings of RSI is more of massively scaling and diffusing inference-time compute to AI research and related activities, which has a large amount of low-hanging fruit available. This, on its own, is still poised to be economically transformative. It may also unlock more resources to push on AI diffusion, which is the crucial bottleneck in unlocking much of the potential benefits of AI.
For now and until more evidence emerges, lossy self-improvement ：https://www.interconnects.ai/p/lossy-self-improvement remains my baseline on the trajectory of progress, and the increased discussion of extinction risk seems very misplaced. As always, things can change fast in AI.
扫码直达文章详情