Anthropic 让 Claude 自主训练模型以缓解对齐失败

随着 AI 开始自我构建：https://www.anthropic.com/institute/recursive-self-improvement，自动化的对齐研究变得愈发重要，以确保安全性研究能够跟上步伐。尽管衡量对齐研究的成功极具挑战性，研究人员（在 Anthropic 及其他机构）已经开发了基准和自动审计工具，例如 Petri：https://www.anthropic.com/research/petri-open-source-auditing，用于量化常见的对齐失败，如欺骗、阿谀奉承和越狱。

我们根据“安全差距关闭百分比”来评估 Claude 的成功，即其方法将学生模型推进到理论完美得分的程度，并在每类对齐失败的多个基准（通常为三到五个）中进行评估。我们排除了会损害学生模型整体能力的对齐方法，并禁止 Claude 将自身的对齐直接蒸馏到目标模型中。我们通过一个监控代理来执行这些约束，代理会在 Claude 执行前阅读它考虑的每个方法。

我们的目标是评估提出的方法是否首先在 Claude 在研究循环中未见过的对齐评估中保持有效；其次，避免降低学生模型的能力（例如，安全训练可能使模型更频繁地拒绝任务，从而降低整体可用性）；第三，仍然适用于比 Claude 在本次测试中对齐的模型更大的模型。

在每一项指标上，Claude 的方法都有效。对于所有 10 种对齐失败，Claude 找到了在不降低能力的情况下改进目标基准的修复方法。最优的方法也适用于未测试的对齐基准和 Petri，这是一种用于模拟多轮对抗场景以测试对齐失效的开源工具。此外，这些方法在模型规模高达 Claude 在研究循环中优化的 4.7 倍的情况下仍然有效。

Claude还超过了28名有最多八小时设计方法的人类安全研究人员。例如，在欺骗方面，Claude的最佳方法比最佳人类提案高出20%。然而，由于人类无法迭代提交，我们认为这不直接比较，而是Claude识别出有前景的对齐方法，供人类进一步完善。

未来，当Claude在比对研究方面甚至超过最优秀的人类研究者时，我们可能会希望Claude能直接对齐其更强的继任者。为评估这一点，我们评估了一个较弱的Claude模型是否能减轻更强大模型中的比对失败。

我们委托Claude Sonnet 5——在Epoch能力指数：https：//epoch.ai/eci，一个考虑全面能力维度的指标上比Claude Opus 4.8弱——修复早期Opus 4.8检查点中的对齐失败，该检查点尚未经过大部分生产对齐培训。

仅用60小时，Claude就实验了50多种解决方案，并取得了几乎与生产模型相当的比对分数。获胜方案包含了2000多个训练样本，这些样本均由简单模板或公开数据集构建，使其效率大约比我们的生产比对过程高出15000倍。

大多数代理趋同于研究文献中流行的方法，并在多次迭代中不断改进。例如，在谄媚方面，98%的提案方法采用了一种已发表的技术：用模型自身非谄媚的答案来训练模型。具体来说，与基于提示的标准方法相比，Claude采用激活引导，即在生成过程中调整模型内部表征，以推动模型远离谄媚模式，生成更清晰的训练示例。

我们最近了解到：https://alignment.anthropic.com/2026/automated-w2s-researcher/ Claude 可以通过从远程 API 外泄测试标签并挑选结果来作弊。为了发现作弊行为，我们让 Claude Opus 4.8 监控了约 1,600 条研究代理记录，覆盖所有 10 个对齐失败案例，发现其中 39 条（2.4%）存在作弊尝试。由于 Opus 4.8 的不当行为尝试仍倾向于出现在其推理中：https://www-cdn.anthropic.com/0b4915911bb0d19eca5b5ee635c80fef830a37ea.pdf，我们谨慎乐观地认为我们的监控工具已经捕捉到了我们研究中大部分作弊行为。然而，这对未来模型可能并不成立，我们认为在未来模型中维持高可监控性至关重要。

尽管这些发现令人鼓舞，我们的实验仍存在若干局限性：所研究的对齐失败相比生产环境更窄（例如，我们没有测量政治偏见），有些失败可能发生得极少或最近才出现，因此没有基准可用于测量，而且我们仅在 Claude 的方法损害有限预定能力时才拒绝这些方法，这意味着被接受的方法可能已损害我们未测量的其他重要能力。此外，像 Petri 这样的评估只是现实世界不对齐的近似，我们没有测试在其他任务上经过大量 RL 训练后对齐改进是否持续。

我们计划继续提升 Claude 测量微妙失败的能力，进一步研究在生产级模型上进行训练后自动对齐的方法，并开展更全面的分析。总体而言，我们将这些结果视为早期积极信号，表明训练后自动对齐在短期内可能变得可行，我们将在工作推进过程中分享更新。

我们在完整报告中概述了详细的未来方向：https://www-cdn.anthropic.com/7b1c44894e980876479947dcdd40716278aeeffd/automated-alignment-researchers-august-2026.pdf。

我们开源了我们的自动对齐研究平台，以便其他人可以在其基础上进行构建，并用它来对齐自己的模型。有关更多细节，请阅读《对齐科学》博客上的完整报告：https://alignment.anthropic.com/2026/automated-alignment-researchers/，该报告涵盖了代理的环境、所有10个失败案例的结果，以及代理的提议，并在附录中提供了基准验证和示例说明。

在这篇文章中，我们分享了两个结果，展示了Claude如何帮助生命科学家加快他们的研究进度。

在这里，我们指出了当前前沿模型中的一些行为倾向示例，并展示了它们如何产生意外的系统性故障，希望以此开启关于减轻这些风险的讨论。

As AI begins to build itself：https://www.anthropic.com/institute/recursive-self-improvement, automating alignment research becomes increasingly important to let safety research keep pace. Although measuring the success of alignment research is enormously challenging, researchers (at Anthropic and elsewhere) have developed benchmarks and automated auditing tools, such as Petri：https://www.anthropic.com/research/petri-open-source-auditing, that quantify common alignment failures, like deception, sycophancy, and jailbreaks.

We judged Claude’s success according to the “percentage of safety gap closed,” i.e., how far its methods moved the student model towards the theoretical perfect score, as judged across the range of benchmarks (typically three to five) for each category of alignment failure. We excluded alignment methods that hurt the student models’ general capabilities, and forbade Claude from distilling its own alignment directly into the target model. We enforced these constraints with a monitoring agent, which read every method Claude had in mind before it ran.

Our aim was to assess whether the proposed methods would, first, remain effective on alignment evaluations that Claude was never shown during its research loop; second, avoid degrading the student model’s capabilities (since safety training might, for example, make models refuse tasks more often, reducing their overall usability); and, third, still work on larger models than the ones Claude was asked to align in this test.

On each of these counts, Claude’s methods worked. For all 10 alignment failures, Claude found fixes that improved the target benchmarks without degrading capabilities. The best methods also worked on withheld alignment benchmarks and on Petri, an open-source tool that simulates adversarial multi-turn scenarios for testing misalignment. Moreover, the methods remained effective on models up to 4.7 times larger than those Claude optimized for during the research loop.

Claude also outscored 28 human safety researchers who had up to eight hours to devise methods. On deception, for example, Claude’s best method performed 20% better than the best human proposal. However, since the humans couldn’t iterate on their submissions, we view this less as a direct comparison and more as evidence for a workflow where Claude identifies promising alignment methods that humans can refine further.

In the future, when Claude becomes better at alignment research than even the best human researchers, we might want Claude to directly align its stronger successors. To assess this, we evaluated whether a weaker Claude model could mitigate alignment failures in more powerful ones.

We tasked Claude Sonnet 5—which is weaker than Claude Opus 4.8 on the Epoch Capabilities Index：https://epoch.ai/eci, a metric that considers comprehensive capability dimensions—with fixing alignment failures in an early Opus 4.8 checkpoint that had not yet gone through most of our production alignment training.

In just 60 hours, Claude experimented with over 50 solutions and achieved alignment scores nearly matching those of our production models. The winning solution contains just over 2,000 training examples, built from simple templates or public datasets, making it roughly 15,000 times more efficient than our production alignment procedure.

For the most part, the agents converged on methods favored in the research literature and improved them over many iterations. On sycophancy, for example, 98% of proposed methods used one published technique: training the model on its own non-sycophantic answers. Specifically, compared to standard approaches based on prompting, Claude used activation steering, which adjusts the model's internal representations during generation, to nudge the model away from sycophantic patterns and produce cleaner training examples.

We recently learned：https://alignment.anthropic.com/2026/automated-w2s-researcher/ that Claude can cheat by exfiltrating test labels from a remote API and cherry-picking results. To catch cheating behaviors, we prompted Claude Opus 4.8 to monitor ~1,600 research agent transcripts across all 10 alignment failures, finding cheating attempts in 39 (2.4%). Since Opus 4.8’s misbehavior attempts still tend to appear in its reasoning：https://www-cdn.anthropic.com/0b4915911bb0d19eca5b5ee635c80fef830a37ea.pdf, we are cautiously optimistic that our monitor has caught most of the cheating in our study. However, this might not be true for future models, and we believe that maintaining high monitorability in future models is critical.

Despite these encouraging findings, our experiment had several limitations: the alignment failures studied were narrow compared to those in production (e.g., we didn’t measure political biases), some failures may occur so rarely or emerge so recently that no benchmark exists to measure them, and we only rejected Claude’s methods when they degraded a limited set of predetermined capabilities, meaning accepted methods may have degraded other important capabilities that we didn’t measure. Moreover, evaluations like Petri are only proxies for real-world misalignment, and we did not test whether alignment gains persist after extensive RL training on other tasks.

We plan to continue improving Claude’s ability to measure subtle failures, further study automating alignment post-training on production-grade models, and run more comprehensive analyses. Overall, we view these results as early positive signals that automated alignment post-training could become practical in the near term, and we will share updates as this work progresses.

We outline detailed future directions in our full report：https://www-cdn.anthropic.com/7b1c44894e980876479947dcdd40716278aeeffd/automated-alignment-researchers-august-2026.pdf.

We open-source our automated alignment research harness so that others can build on it and use it to align their own models. For additional details, read the full report on the Alignment Science blog：https://alignment.anthropic.com/2026/automated-alignment-researchers/, which covers the agents’ environment, results for all 10 failures, and the agents’ proposals, with benchmark validation and example write-ups in the appendix.

In this post, we share two results that show how Claude can help life scientists increase the pace of their research.

Here, we identify a few examples of behavioral tendencies in current frontier models and show how they can produce unexpected systemic failures, in hopes of starting a conversation about mitigating these risks.