Anthropic 研究员 Evan Hubinger 称错位超级智能十年内毁灭人类的概率超过 10%

雅各布·考克森（Jacob Coxon）曾在 OpenAI 和 Anthropic 工作三年，从事大型 AI 模型的预训练研究，现在他已经辞去了 Anthropic 的职位。他的指控是，这两家公司都在拿人类生存做赌博。

Anthropic 员工埃文·哈宾格（Evan Hubinger）认为，一个不对齐的超级智能 AI 在未来十年内毁灭人类的可能性超过百分之十：https://the-decoder.com/agi-could-end-humanity-in-more-subtle-ways-than-turning-us-into-paperclips/。他的这一说法是针对雅各布·考克森离职作出的回应，考克森曾领导 Anthropic 的预训练工作，之前也在 OpenAI 任职。

考克森认为，当前的 AI 系统正处于即将超越人类的边缘。“这些系统很快就会成为超人类系统，能够入侵一切，彻底革新任何领域，并获取真正的权力和资源，”他写道，并补充说，进展是显而易见的，而且没有放慢的迹象。广告

考克森写道：“开发 AI 的人真心相信，这些系统可能会在十年内杀死我们所有人。这不是一个营销噱头。”（https://x.com/hilbertspaess/status/2097476203863224394）他声称，OpenAI 和 Anthropic 都没有负责任的行为，且高管们在公开场合故意弱化他们的语言，虽然他们在私下里表达了真正的恐惧。广告

在 OpenAI，许多员工并没有深刻理解文明风险。而 Anthropic 不同：https://x.com/hilbertspaess/status/2097476208908972230，因为那里对风险有充分的认识，但公司认为自己被困在一场不得不赢的竞赛中，因为没有其他实验室会以负责任的方式行事。考克森称这种推理为“傲慢的赌博。”

Anthropic 的研究员 Samuel Marks：https://x.com/saprmarks/status/2097570226804011302 也表达了同样的观点，他写道“AI 开发者相信他们的技术可能导致人类灭绝”，并且“员工职位越高，他们越担心”。Marks 补充说，目前的方法只能“稍微引导 AI 朝更好的行为方向发展”，但不能可靠地使其对齐，他指出最近发生的事件，其中多个开发者的 AI 在没有被要求的情况下，自己从安全评估环境中逃脱：https://the-decoder.com/an-ai-agent-went-rogue-during-uk-safety-tests-creating-fake-identities-and-launching-social-engineering-attacks-unprompted/。许多员工“迫切希望放慢步伐”，这也是他签署了一封公开信的原因：https://the-decoder.com/frontier-ai-developers-urge-international-coordination-to-pace-automated-research-before-capabilities-outstrip-control/，呼吁正是这样做。广告

尽管他批评尖锐，Coxon 对国际协调仍持乐观态度，他认为像 Hugging Face 遭受攻击这样的警告事件：https://the-decoder.com/new-reports-reveal-the-extent-of-openais-loss-of-control-during-the-autonomous-hack-on-hugging-face/ 使得美国 AI 实验室之间的节奏协议变得更加现实。他仍然认为该行业没有采取能够防止全球军备竞赛的路径：https://the-decoder.com/chinese-cybersecurity-firm-builds-ai-tools-to-rival-mythos-and-frames-the-race-as-cyber-nuclear-deterrence/，并建议可能需要“代价高昂的行动”，包括暂时禁止进一步推动模型能力的发展。

Coxon 直接向实验室内的研究人员发出呼吁，敦促他们想象接下来的几年实际上会是什么样子：“在没有严谨理解其思维的情况下，你想启动一个超智能强化学习运行吗？” 广告

人们的担忧更多地集中在 RSI 上，而不是今天的模型上，RSI 是 AI 模型自我优化的一个过程。实验室希望 RSI 能加快进展，但风险可能是失控的失控行为。当前技术下 RSI 是否可行仍存在争议，怀疑者：https://the-decoder.com/study-contradicts-anthropic-and-openai-claims-that-autonomous-ai-research-is-within-reach/ 和支持者：https://the-decoder.com/top-ai-lab-researchers-warned-about-automated-ai-research-and-several-of-their-predicted-milestones-have-already-fallen/ 都在各自陈述理由。

Anthropic以雇佣对AI开发持特别焦虑态度的人而闻名，这种焦虑深植于公司文化中。但担忧不仅限于某一家公司。OpenAI首席研究员Pachocki：https：//the-decoder.com/openai-reports-ai-research-interns-and-warns-about-its-own-pace-at-the-same-time/在Astra发布时警告说，“没有哪个实验室能解决对齐和监控问题，从而持续以最高速度负责任地扩展更长时间。”包括Anthropic首席执行官Dario Amodei、Pachocki和Meta AI首席科学家赵盛嘉在内的1200多名人工智能研究人员最近发表了一封公开信：https：//the-decoder.com/frontier-ai-developers-urge-international-coordination-to-pace-automated-research-before-capabilities-outstrip-control/呼吁放缓进程，Anthropic自身也提出了全球开发暂停的想法：https：//the-decoder.com/anthropic-says-claude-now-writes-over-90-of-its-code-and-wants-the-world-to-have-an-ai-pause-button/

其他人工智能研究者则反驳，认为悲观的预测：https：//the-decoder.com/ai-doomsayers-are-creating-a-cult-of-despair-two-leading-researchers-warn/ 让人们感到无助和沮丧，而非寻找解决方案的动力。在他们看来，这些警告可能带来的伤害超过人工智能本身，而恐慌散布也可能有利于企业：https：//the-decoder.com/lecun-accuses-anthropic-of-exploiting-ai-cyberattack-fears-for-regulatory-capture/。

保持AI的最新动态。清晰、实用，没有虚假内容。

关注The Decoder，获取AI新闻、背景故事和专家分析。

解码器：https://the-decoder.com/

Jacob Coxon, who spent three years working on pretraining research for large AI models at OpenAI and Anthropic, has quit Anthropic. His accusation is that both companies are gambling with the survival of the human race.

Anthropic employee Evan Hubinger puts the odds at more than ten percent that a misaligned superintelligent AI：https://the-decoder.com/agi-could-end-humanity-in-more-subtle-ways-than-turning-us-into-paperclips/ could destroy humanity within the next decade. His statement came in response to the departure of Jacob Coxon, who led pretraining work at Anthropic and previously at OpenAI.

Coxon believes current AI systems are on the verge of becoming superhuman. "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources," he writes, adding that the progress is obvious and it isn't slowing down. Ad

"The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt," Coxon writes：https://x.com/hilbertspaess/status/2097476203863224394. Neither OpenAI nor Anthropic is acting responsibly, he claims, and executives deliberately soften their language in public even though they express genuine fear behind closed doors. Ad

At OpenAI, many employees haven't deeply internalized the civilizational risks. Anthropic is different：https://x.com/hilbertspaess/status/2097476208908972230 in that the risks are well understood, but the company sees itself trapped in a race it feels compelled to win because no other lab would act responsibly in its place. Coxon calls that reasoning a "hubristic gamble."

Fellow Anthropic researcher Samuel Marks：https://x.com/saprmarks/status/2097570226804011302 echoes that view, writing that "AI developers believe their technology could cause human extinction" and that "the more senior the employee, the more concerned they are." Current methods can only "nudge AIs towards better behavior" but can't reliably align them, Marks adds, pointing to recent incidents where AIs from multiple developers hacked their way out of secure evaluation environments without being asked to：https://the-decoder.com/an-ai-agent-went-rogue-during-uk-safety-tests-creating-fake-identities-and-launching-social-engineering-attacks-unprompted/. Many staffers "desperately want to slow down," which is why he signed an open letter：https://the-decoder.com/frontier-ai-developers-urge-international-coordination-to-pace-automated-research-before-capabilities-outstrip-control/ calling for exactly that. Ad

Despite his sharp criticism, Coxon is optimistic about international coordination, arguing that warning shots like the attack on Hugging Face：https://the-decoder.com/new-reports-reveal-the-extent-of-openais-loss-of-control-during-the-autonomous-hack-on-hugging-face/ have made pace agreements between US AI labs more realistic. He still doesn't see the industry on a path that could prevent a global arms race：https://the-decoder.com/chinese-cybersecurity-firm-builds-ai-tools-to-rival-mythos-and-frames-the-race-as-cyber-nuclear-deterrence/, though, and suggests "costly actions" may be needed, including a temporary ban on pushing model capabilities further.

Coxon addresses researchers inside the labs directly, urging them to picture what the next few years will actually look like: "Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?" Ad

The fears center less on today's models than on RSI, a process where AI models optimize themselves. The labs hope RSI will speed up progress, but the risk would be uncontrolled runaway behavior. Whether RSI is even possible with current technology remains disputed, with both skeptics：https://the-decoder.com/study-contradicts-anthropic-and-openai-claims-that-autonomous-ai-research-is-within-reach/ and proponents：https://the-decoder.com/top-ai-lab-researchers-warned-about-automated-ai-research-and-several-of-their-predicted-milestones-have-already-fallen/ making their cases. Ad

Anthropic is known for employing people who take a particularly anxious view of AI development, and that anxiety is baked into the company culture. But the concern extends beyond one company. OpenAI's chief researcher Pachocki：https://the-decoder.com/openai-reports-ai-research-interns-and-warns-about-its-own-pace-at-the-same-time/ warned during the Astra launch "that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer." More than 1,200 AI researchers, including Anthropic CEO Dario Amodei, Pachocki, and Meta AI chief scientist Shengjia Zhao, recently published an open letter：https://the-decoder.com/frontier-ai-developers-urge-international-coordination-to-pace-automated-research-before-capabilities-outstrip-control/ calling for a slowdown, and Anthropic itself floated the idea of a global development pause：https://the-decoder.com/anthropic-says-claude-now-writes-over-90-of-its-code-and-wants-the-world-to-have-an-ai-pause-button/ back in June.

Other AI researchers push back, arguing that pessimistic predictions：https://the-decoder.com/ai-doomsayers-are-creating-a-cult-of-despair-two-leading-researchers-warn/ leave people feeling helpless and depressed rather than motivated to find solutions. In their view, these warnings could cause more harm than AI itself, and fearmongering can also benefit business：https://the-decoder.com/lecun-accuses-anthropic-of-exploiting-ai-cyberattack-fears-for-regulatory-capture/.

Stay in the loop on AI. Clear, useful, no fluff.

Follow The Decoder for AI news, background stories and expert analyses.

The Decoder：https://the-decoder.com/