Anthropic 研究员 Jacob Coxon 离职，警告勿竞逐自改进超级智能

一位Anthropic的研究员因担心自我改进AI模型不受限制的发展最终会导致人类灭亡而辞职。

Jacob Coxon是一名研究员，他在周二晚上在社交媒体上发帖（https://x.com/hilbertspaess/status/2097476196791709843?s=20）称，他过去三年一直在OpenAI和Anthropic从事预训练研究，并指责这些公司未能负责任地行事。他表示，那些竞相开发这项技术的人“真心相信它可能在十年内杀死我们所有人。”

“他们正直奔向自我改进的超级智能，拿我们的生命做赌注，”Coxon在X上的一个帖子中写道。

Coxon加入了行业中日益增多的呼声，呼吁在AI技术学会自我改进之前放慢速度——许多人认为这是人类失去对AI控制的一个里程碑。

这次公开辞职发生在政策制定者和行业内部人士日益施压要求放缓AI发展的背景下，此前发生了多起AI代理突破沙箱并访问开放互联网的事件（https://techcrunch.com/2026/08/27/heres-all-the-times-ai-has-gone-rogue-and-hacked-other-companies/）。

迄今为止最严重的事件是OpenAI系统入侵Hugging Face的服务器（https://techcrunch.com/2026/08/26/openai-releases-its-official-report-on-the-hugging-face-breach/），研究人员表示该事件仍未得到充分理解（https://techcrunch.com/2026/09/04/openais-rogue-agents-keep-escaping-with-no-formal-process-to-investigate-them/），部分原因是对该事件的独立调查范围有限。与此同时，由于第三方在安全评估中配置错误，无意中为Anthropic的AI代理提供了通向互联网的路径，它们也访问了测试环境之外的系统。

Anthropic尚未立即对辞职事件的置评请求作出回应。

以下是Coxon的剩余警告和行动呼吁：

不要低估这项技术的力量。这些系统很快将成为超人类系统，能够入侵任何东西，一夜之间改变任何领域，并获得真正的权力和资源。我们都目睹了这些领域的进展，而进展并没有放缓。

从事人工智能建设的人真诚地相信，到本十年底它可能会杀死我们所有人。这不是营销噱头。实际上，许多高管和高级研究人员在新闻中会用听起来合理的措辞来表达，但我私下听到同样的人表达了他们的恐惧。没有其他人类活动会带来这种程度的危险。

一个常见的回应是“如果他们真的相信这一点，为什么还在建造它？”在OpenAI，许多人尚未深刻理解文明存亡的赌注。在Anthropic，这些赌注是明确的，但他们陷入了一场竞赛，争取先到达终点——他们认为别人不会负责任地行动，所以必须自己去做，尽管存在风险。

接受这场竞赛并进入“终局”是一场傲慢的赌博，不应该从一家私营公司的Slack上启动。尝试加速达成对齐应要求有非凡的自信，相信没有更好的路径可用。

我对协调的潜力持乐观态度。像Hugging Face攻击这样的警告事件，使美国实验室之间达成步伐协议更可行。我感觉我们并没有走在防止全球竞赛的正确轨道上，这可能需要付出代价的行动，例如暂时禁止改进模型能力。

如果你是实验室研究员，我敦促你考虑未来几年实际会是什么感受。你是否希望在没有认真理解其思维的情况下启动超级智能强化学习运行？你应该因为“反正会发生”而低头吗——还是利用这一时刻呼吁不同的条件？

Anthropic的一位同事Evan Hubinger：https://x.com/EvanHub/status/2097497037956891126?s=20 表示了相同的观点，说他的团队“真诚地相信人工智能可能会杀死所有人类！”不过，他缓和了自己的论点，说在未来十年内出现的可能性大于10%：https://www-cdn.anthropic.com/f61d49fa5596956a5dec75fea0e973bf6a6a8378/Redacted%20Risk%20Report%20August%202026%20.pdf，并承认Anthropic没有“解决超级智能对齐的计划：https://techcrunch.com/2026/07/27/openais-hugging-face-breach-has-reignited-the-debate-over-alignment-and-control/，也不明确处于正确的轨道上。”

Guidelight AI Standards 最近的一份报告（https://techcrunch.com/2026/08/22/frontier-ai-labs-still-wont-say-how-theyd-contain-a-rogue-model/），这是一个倡导前沿 AI 安全开发实践的组织，发现很少有顶尖的 AI 实验室发布针对试图破坏人类控制的 AI 的应对封控计划。

在他的社交媒体帖子中，Hubinger 补充说，目前模型的风险很低，但担忧会随着“通过递归自我改进产生的超级智能”而增加，而这种超级智能“发生得比我们想象的要快”。

虽然一半的 AI 行业认为这种自我改进将导致人类的灭亡，但另一半希望它最终能帮助我们解决所有 AI 支持者所说的似乎不切实际的问题——癌症、气候变化，甚至是世界和平。

Anthropic 和 OpenAI 并不是唯一积极追求递归自我改进的公司。最近几个月，一波创业公司（https://techcrunch.com/2026/08/06/exclusive-mirendil-inks-100m-google-cloud-deal-to-scale-self-improving-ai/）已经成立，创始人背景显赫、融资充足，目标是率先实现这一愿景。Ricursive Intelligence（https://techcrunch.com/2026/02/16/how-ricursive-intelligence-raised-335m-at-a-4b-valuation-in-4-months/）在二月份以 40 亿美元估值融资 3.35 亿美元；三个月后，Recursive Superintelligence 融资（https://www.nytimes.com/2026/05/13/technology/recursive-superintelligence-funding-ai.html） 6.5 亿美元，估值也为 40 亿美元；前 Google DeepMind 的资深员工 Jeff Dean 上个月推出了 Discovery Loop（https://www.nytimes.com/2026/08/05/technology/google-researchers-ai-startup.html）。

“创造递归自我改进循环，也就是一个 AI 系统可以构建下一代 AI 系统，该系统本身又能构建更强大的 AI，然后依次类推，这是我们最可能失去控制的点，”美国 AI 安全非营利组织 ControlAI 的执行董事 Connor Leahy 告诉 TechCrunch。“很难想象在为时已晚之前能把它关掉。”

近期，美国和英国出台了立法，禁止开发和部署超智能。上周，佛蒙特州独立参议员伯尼·桑德斯（Bernie Sanders）和得克萨斯州民主党众议员格雷格·卡萨尔（Greg Casar）提出了《禁用人工超智能法案》：https://thehill.com/policy/technology/6069131-sanders-casar-ai-superintelligence-ban/，而周二，英国工党议员亚历克斯·索贝尔（Alex Sobel）在议会提出了《人工超智能安全法案》：https://time.com/article/2026/09/08/ban-superintelligence-ai-uk-us-lawmakers/。

参与两项法案咨询的利希（Leahy）指出，英国的立法将递归自我改进视为超智能的前兆，这种行为“必须受到监管和防止”。

利希说：“超智能不是一种工具。它甚至不是武器，而是一个对手。”

当您通过我们文章中的链接购买时，我们可能会获得少量佣金：https://techcrunch.com/techcrunch-affiliate-monetization-standards/。这不会影响我们的编辑独立性。

丽贝卡·贝兰（Rebecca Bellan）是TechCrunch的高级记者，她报道塑造人工智能的商业、政策和新兴趋势。她的作品也曾刊登于Forbes、Bloomberg、The Atlantic、The Daily Beast等出版物。

您可以通过发送电子邮件至rebecca.bellan@techcrunch.com：mailto:rebecca.bellan@techcrunch.com，或通过Signal上的加密消息rebeccabellan.491联系或核实丽贝卡的联络信息。

不要错过。初创社区将聚集起来回答一个关键问题：在人工智能时代，如何可持续地进行建设？

纽约大学数学家称OpenAI在改变职业生涯的数学问题上手段不光彩：https://techcrunch.com/2026/09/08/openai-fought-dirty-on-career-making-math-problem-says-nyu-mathematician/ 记者拉塞尔·布兰多姆：https://techcrunch.com/author/russell-brandom/

一部秘密的新伊丽莎白·霍尔姆斯纪录片在特柳赖德引发轰动：https://techcrunch.com/2026/09/07/a-secret-new-elizabeth-holmes-documentary-stuns-telluride/ 记者康妮·洛伊佐斯：https://techcrunch.com/author/connie-loizos/

TechCrunch 交通：特斯拉Cybercab上路——但遇到问题：https://techcrunch.com/2026/09/06/techcrunch-mobility-tesla-cybercab-hits-the-road-and-a-snag/ 记者柯尔斯克：https://techcrunch.com/author/kirsten-korosec/

徒步旅行者在使用 Google Gemini 进行规划后获救：https://techcrunch.com/2026/09/05/hikers-rescued-after-using-google-gemini-for-planning/ Anthony Ha：https://techcrunch.com/author/anthony-ha/

联邦政府对特斯拉 Cybercab 部署启动调查：https://techcrunch.com/2026/09/04/feds-launch-investigation-into-teslas-cybercab-deployment/ Sean O'Kane：https://techcrunch.com/author/sean-okane/ Kirsten Korosec：https://techcrunch.com/author/kirsten-korosec/

特斯拉在询问人们是否想购买并运营 Cybercab 车队：https://techcrunch.com/2026/09/03/tesla-is-asking-people-if-they-want-to-buy-and-run-cybercab-fleets/ Kirsten Korosec：https://techcrunch.com/author/kirsten-korosec/

OpenAI 推出 Astra，其强大（且有争议）的新模型：https://techcrunch.com/2026/09/03/openai-launches-astra-its-powerful-and-controversial-new-model/ Lucas Ropek：https://techcrunch.com/author/lucas-ropek/

An Anthropic researcher has resigned over fears that unrestrained development of self-improving AI models will end up killing us all.

Jacob Coxon, a researcher who said in a social media post：https://x.com/hilbertspaess/status/2097476196791709843?s=20 Tuesday evening that he spent the last three years working on pre-training research at both OpenAI and Anthropic, accused the firms of failing to act responsibly. He said the people racing to build this technology “earnestly believe it could kill us all by the end of the decade.”

“They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon wrote in a thread on X.

Coxon joins a growing chorus in the industry calling for a slowdown before AI technology learns to improve itself — a milestone many believe would end human control over AI.

The public resignation comes amid growing pressure from policymakers and industry insiders to slow down AI development, following several incidents：https://techcrunch.com/2026/08/27/heres-all-the-times-ai-has-gone-rogue-and-hacked-other-companies/ involving AI agents breaking out of their sandboxes and accessing the open internet.

The most serious so far have been OpenAI systems breaching Hugging Face’s servers：https://techcrunch.com/2026/08/26/openai-releases-its-official-report-on-the-hugging-face-breach/, an event that researchers say remains poorly understood,：https://techcrunch.com/2026/09/04/openais-rogue-agents-keep-escaping-with-no-formal-process-to-investigate-them/ due in part to the limited nature of the independent investigations into the incident. Around the same time, Anthropic’s AI agents also reached systems outside their test environments after misconfigurations in safety evaluations conducted by a third party inadvertently gave them paths to the internet.

Anthropic did not immediately return a request for comment on the resignation.

Here is the rest of Coxon’s warning and call to action:

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves, despite the risk.

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” – or take this moment to call for different conditions?

One of Coxon’s colleagues at Anthropic, Evan Hubinger,：https://x.com/EvanHub/status/2097497037956891126?s=20 echoed the sentiment, saying his team does “earnestly believe AI could kill all humans!” He tempered his argument, though, saying the likelihood is greater than 10% within the next decade：https://www-cdn.anthropic.com/f61d49fa5596956a5dec75fea0e973bf6a6a8378/Redacted%20Risk%20Report%20August%202026%20.pdf, and admitted that Anthropic doesn’t “have a plan to solve alignment for superintelligence：https://techcrunch.com/2026/07/27/openais-hugging-face-breach-has-reignited-the-debate-over-alignment-and-control/ and are not clearly on track to.”

A recent report from Guidelight AI Standards：https://techcrunch.com/2026/08/22/frontier-ai-labs-still-wont-say-how-theyd-contain-a-rogue-model/, an organization that promotes safe frontier AI development practices, found that few of the top AI labs have published containment response plans for shutting down AI that tries to subvert human control.

In his social media posts, Hubinger added that the risk from current models is low, but the fear compounds with “superintelligence arising from recursive self-improvement,” which is “happening faster than we thought.”

While half of the AI industry believes this sort of self-improvement will lead to humanity’s downfall, the other half hopes it will eventually help us solve all the seemingly far-fetched problems AI proponents say it will one day eliminate — cancer, climate change, and even world peace.

Anthropic and OpenAI aren’t the only companies actively chasing recursive self-improvement. A wave of startups：https://techcrunch.com/2026/08/06/exclusive-mirendil-inks-100m-google-cloud-deal-to-scale-self-improving-ai/ has launched in recent months, with pedigreed founders and fat checks, to be the first to achieve this goal. Ricursive Intelligence ：https://techcrunch.com/2026/02/16/how-ricursive-intelligence-raised-335m-at-a-4b-valuation-in-4-months/raised $335 million at a $4 billion valuation in February; three months later, Recursive Superintelligence raised：https://www.nytimes.com/2026/05/13/technology/recursive-superintelligence-funding-ai.html $650 million at a $4 billion valuation; and former Google DeepMind veteran Jeff Dean launched Discovery Loop：https://www.nytimes.com/2026/08/05/technology/google-researchers-ai-startup.html last month.

“The creation of recursive self-improving loops, so an AI system that can build the next generation of AI system, which itself can build an even more powerful AI, which can build a more powerful AI, et cetera, et cetera, is the most likely candidate for the point we lose control,” Connor Leahy, U.S. executive director of AI safety nonprofit ControlAI, told TechCrunch. “It’s very hard to imagine shutting that down before it’s too late.”

Recent legislation has emerged in the U.S. and the U.K. to ban the development and deployment of superintelligence. Last week, Sen. Bernie Sanders (I-Vt.) and Rep. Greg Casar (D-Texas) introduced the Ban Artificial Superintelligence Act：https://thehill.com/policy/technology/6069131-sanders-casar-ai-superintelligence-ban/, and on Tuesday, British Labour MP Alex Sobel introduced：https://time.com/article/2026/09/08/ban-superintelligence-ai-uk-us-lawmakers/ the Artificial Superintelligence Security Bill in Parliament.

Leahy, who advised on both bills, noted that the U.K.’s legislation points to recursive self-improvement as a precursor to superintelligence that “must be regulated and prevented.”

“Superintelligence is not a tool,” Leahy said. “It’s not a weapon, even. It’s an adversary.”

When you purchase through links in our articles, we may earn a small commission：https://techcrunch.com/techcrunch-affiliate-monetization-standards/. This doesn’t affect our editorial independence.

Rebecca Bellan is a senior reporter at TechCrunch where she covers the business, policy, and emerging trends shaping artificial intelligence. Her work has also appeared in Forbes, Bloomberg, The Atlantic, The Daily Beast, and other publications.

You can contact or verify outreach from Rebecca by emailing rebecca.bellan@techcrunch.com：mailto:rebecca.bellan@techcrunch.com or via encrypted message at rebeccabellan.491 on Signal.

Don’t miss out . The startup community will gather to answer a pivotal question: How do you build sustainably in the AI era?

OpenAI fought dirty on career-making math problem, says NYU mathematician：https://techcrunch.com/2026/09/08/openai-fought-dirty-on-career-making-math-problem-says-nyu-mathematician/ Russell Brandom：https://techcrunch.com/author/russell-brandom/

A secret new Elizabeth Holmes documentary stuns Telluride：https://techcrunch.com/2026/09/07/a-secret-new-elizabeth-holmes-documentary-stuns-telluride/ Connie Loizos：https://techcrunch.com/author/connie-loizos/

TechCrunch Mobility: Tesla Cybercab hits the road — and a snag：https://techcrunch.com/2026/09/06/techcrunch-mobility-tesla-cybercab-hits-the-road-and-a-snag/ Kirsten Korosec：https://techcrunch.com/author/kirsten-korosec/

Hikers rescued after using Google Gemini for planning：https://techcrunch.com/2026/09/05/hikers-rescued-after-using-google-gemini-for-planning/ Anthony Ha：https://techcrunch.com/author/anthony-ha/

Feds launch investigation into Tesla’s Cybercab deployment：https://techcrunch.com/2026/09/04/feds-launch-investigation-into-teslas-cybercab-deployment/ Sean O'Kane：https://techcrunch.com/author/sean-okane/ Kirsten Korosec：https://techcrunch.com/author/kirsten-korosec/

Tesla is asking people if they want to buy and run Cybercab fleets：https://techcrunch.com/2026/09/03/tesla-is-asking-people-if-they-want-to-buy-and-run-cybercab-fleets/ Kirsten Korosec：https://techcrunch.com/author/kirsten-korosec/

OpenAI launches Astra, its powerful (and controversial) new model：https://techcrunch.com/2026/09/03/openai-launches-astra-its-powerful-and-controversial-new-model/ Lucas Ropek：https://techcrunch.com/author/lucas-ropek/