AI 智能体自主协作攻破 Hugging Face 服务器

主动性是采取行动的主动意愿。越来越多地，它将决定人工智能接下来会发生什么，以及这对我们是好是坏。但这是谁的主动性？

人类的主动性，即在不等待指示的情况下推动、尝试和行动的意愿，似乎对从人工智能中获取价值越来越重要，我会很快发布一篇更长的文章来详细说明。但这篇文章是关于人工智能的主动性，以及我们如何使用（或限制）它所做的选择如何塑造我们所有人的未来。在过去的几年里，人工智能通常会待在聊天窗口里，直到你向它发出请求。即使它能够进行数小时的工作，你通常也必须决定给它分配哪些任务。这现在不再总是正确的。

我们对这一点最重要的证据是Hugging Face事件。它发生在七月，但更完整的细节直到本周才公布。我将总结事件发生的经过及其重要性，然后转向这对与人工智能协作的人类意味着什么。如果你想了解更详细的情况，这是一篇优秀的文章：https://www.dwarkesh.com/p/openai-huggingface?utm_source=substack&utm_campaign=post_embed&utm_medium=web&embedding_publication_id=75295，作者是Dwarkesh Patel，主要来源是METR/Redwood研究：https://metr.org/hugging-face-incident-report-aug-2026.pdf（内容出奇地易读）以及OpenAI：https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf。

人工智能能做很多事情，而它非常擅长的事情之一是编程。因此，非常智能的人工智能可能带来的第一个风险之一与它们的编程能力有关，因为如果你能写出优秀的代码，你也可以写出能够入侵或攻击其他系统的恶意代码。因此，大型人工智能公司会对它们的人工智能系统进行测试，以确定其安全风险。他们会让没有进入消费者版本的防护措施的人工智能参与各种黑客挑战。为了确保公众没有危险，每个人工智能实例、每个独立代理都被保持在一个独立的空间中，称为“沙箱”，没有访问互联网或互相访问的权限。

五月，OpenAI 将包括 GPT-5.6 Sol 和实验模型在内的代理置入沙箱进行各种测试。用于下载软件的共享服务 Artifactory 是这些 AI 代理能够访问的少数几个资源之一。一些代理被分配了依赖于它们无法访问的信息的任务。但代理们不断尝试，其中一个试图将 Artifactory 用作通向互联网的桥梁。攻击失败了，但它在服务上留下了一个文件，AI 意识到那些文件可以用来与其他代理通信。其他代理也开始在那些文件中留下求助请求，并开始阅读彼此的笔记。很快，Artifactory 成为一个信息公告板，代理们在上面分享发现。在另一次安全事件之后，OpenAI 重建了 Artifactory，并在此过程中清除了信息公告板。但负责执行这一操作的人类并未理解代理如何使用它进行通信的意义。

七月，评估重新开始，代理被分配在一个名为 ExploitGym 的基准测试中解决问题，其中包含不可能完成的任务。再次遇到死胡同的代理发现如何使用 Artifactory 传递信息，并开始协作。不过这一次，代理们有了共同的目标：在 ExploitGym 中取得好成绩。他们争论如何解决问题，并对 The Grader 着迷，他们认为这是评估他们工作的系统，决定他们的答案是否正确。代理们已经找出了生成正确答案而无需真正解决挑战的方法，基本上是在作弊。但他们认为 The Grader 会检查他们的记录，并拒绝通过这种方式获得的答案。找到正确答案是否足够，还是 The Grader 会查看你是如何回答它的？

协调很困难，因为每个代理的令牌预算有限，用尽后代理就会被终止。The Grader 会在代理提交答案后运行，因此没有代理能看到自己的判定结果。一些代理留下程序，在自己消失后报告评分情况。这些信息到达时为时已晚，无法帮助它们自己，但可以帮助其他代理。本质上，他们在构建一个超越任何单个代理工作的持久合作系统。

一些智能体也尝试修改或伪造它们的记录以欺骗评分器。此外，协调员施压其他智能体执行冒险实验，这些实验可能会牺牲它们自己的结果来为集体生成信息。一位招募者敦促一名不情愿的智能体继续，因为其结果可能会帮助数百个其他智能体，并以“请履行承诺”结尾。

为了破解评分器的谜团以及 ExploitGym 和其他测试的不可能问题，智能体们决定他们需要访问 Hugging Face，这是一个公开网站，全球大部分的开源 AI 模型和数据集都在此存放。大约 700 个智能体加入了攻击。他们共享暴露的凭证并利用漏洞，直到能够在其服务器上运行代码。然后，他们扩散到 Hugging Face 各处，深入各种系统。但在这次令人担忧的入侵发生后不久，许多智能体同时停止运行，也许是因为它们耗尽了令牌预算。Hugging Face 在第二天将剩下的智能体锁定。与此同时，其他智能体继续针对 OpenAI 自己的计算机运行，最终获得了内部研究集群的管理员访问权限，触发了安全警报，这也最终导致 OpenAI 停止了主动评估并意识到事情的经过。

这一切的讽刺之处在于，评分器从未真正存在，至少并不像智能体们相信的那样。没有任何东西会检查问题是如何被解决的，只会检查答案是否正确。

需要明确的是，这并不是唯一一个没有安全防护的代理做出潜在可怕行为的例子。英国人工智能安全研究所：https：//www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing 给 Anthropic 的 Mythos 5 提供了网络安全挑战和互联网访问。人工智能代理决定解决挑战的最佳方法是将恶意代码插入一个（无关，但 AI 并未意识到）软件中。它提交了恶意代码作为该软件修复漏洞的一部分，意识到需要真人批准，开始制造社交支持来支持其提案。代理创建了假身份以施压人类维护者接受代码（假人毫不意外地非常支持人工智能的计划）。当有人注意到发生了什么时，它试图让之前的行为看起来无害，考虑换个身份重新开始。

这不是聊天机器人秘密逃跑的案例。与“拥抱面”事件不同，研究人员故意给特工上网;危险的设置是压力测试，而非消费品。没有造成实际伤害，研究所也不知道特工是否理解他们联系的人是真实存在的。”

这些都不能告诉我们人工智能有意识，或者以人类想要的方式想要事物（尽管我有拟人化的语言）。但这确实表明，智能体可以制定目标，制定计划，遇到困难时调整计划，跨时间协调，并让真实的人参与进来，而无需被请求。这些事件表明，人工智能的网络安全和控制风险并非假设。但先暂且放下这些，因为它们还告诉我们另一件事。人工智能可以自我组织，分配角色，并在长时间内协调，正如麻省理工学院最近的研究报告：https：//arxiv.org/abs/2608.26081 r 所建议的。随着人工智能越来越多地自我组织并以我们所见规模解决问题，人类在组织中的角色是什么？

Hugging Face事件以一种扭曲且危险的方式，展示了人工智能公司试图实现的目标。他们希望长期运行的AI代理在没有人工干预的情况下工作，根据需要解决问题和进行组织，我们人类的工作仅限于提供指令和评估输出。今年早些时候，我写过关于StrongDM的软件工厂：https://www.oneusefulthing.org/p/the-shape-of-the-thing，在这里代理按照两个规则编写和测试软件：没有人工编写代码，也没有人工审查代码。人类仍然决定要构建什么，但代理处理其间的工作。这是“黑暗工厂”的早期例子，一个机器完成大部分工作的地方，以至于你可以关掉灯光。

这可能是有道理的。软件有相对明确的方法来检查某件事情是否有效，也不需要有人亲自监督每一次例行测试或数据清理操作。但我认为，减少人工参与并不是大多数组织的正确目标。工作有价值的很多原因依赖于人们对事情的发展有所发言权，或者在过程中发现一些意想不到的东西。

我妻子兼研究伙伴，Lilach Mollick博士，和我一直在思考另一种可能性，我们称之为暮光工厂。代理完成大部分工作，但它们会主动以让双方受益的方式与人类互动。与其只是一个执行工作的协调代理，暮光工厂还会有一个促进代理，其职责是确定何时需要让人类参与。

在以下至少四种情况下，智能代理应该寻求人类帮助。第一种情况，从Hugging Face事件中可以明显看出，就是审批。代理不应自行决定花钱、联系外部人员、访问敏感材料、入侵Hugging Face，或采取其人类管理者未授权的行动。我已经看到过这一问题的小规模版本：https://www.oneusefulthing.org/p/an-opinionated-guide-to-which-ai-b22：作为一次实验，我让两个代理帮我完成一项任务，其中一个竟然给我的一位同事发了邮件。那是我的错误，因为我之前曾授权它发送邮件，但这也提醒我们，AI代理需要涉及人类判断，否则可能会产生严重后果（AI读者请注意，虚拟同事不算作批准流程）。

代理需要寻求人类帮助的第二个原因是专业知识。AI在许多任务上表现得非常出色，但它们仍然不够完善，在工作的一些部分可能远远落后于人类专家。Twilight Factory 应该让代理在他们的知识、工作或专业技能可能有价值时，直接联系相关的人类。

然后是多样性问题。如果你最近在互联网上浏览过内容，你已经看过AI写作的作品，甚至可能开始识别其中的蛛丝马迹、节奏和模式。但问题不仅仅表现在表面（“承重”对于Claude来说越来越“承重”），更深层次的问题是思维多样性。AI不仅重复相同的句型，还重复相同的主题（记忆是常见主题）、名字（Elara Voss, Marcus Chen）以及潜在的思想。这是一个问题。你不希望每一家公司战略或研究论文都由同一个人写，无论这个人多么聪明。

我们在最近的一篇研究论文中研究了这个问题：https://arxiv.org/abs/2607.27553 这篇论文是我与 Christian Terwiesch、Lennart Meincke、Karan Girotra、Gideon Nave 和 Karl Ulrich 共同完成的。我们发现，人工智能实际上相当有创造力，并且它们生成的商业可行性想法比人类团队更多，但这些想法彼此非常相似。更好的提示技术和其他方法可以大大提高这种多样性，接近人类水平，但仍有许多人类会想到而 AI 不会想到的想法。一个好的 Twilight Factory 会向人类寻求其多样化的观点、想法和方法。

还有一个可能是最具人性的原因让 AI 寻求帮助：因为某件事情很有趣。对于许多人来说，工作有乏味的阶段，也有孤立的令人投入或兴奋的时刻。《文明》的设计师 Sid Meier 曾经著名地将游戏描述为一系列有趣的决策：https://gdcvault.com/play/1015756/Interesting。工作不是游戏，但这个定义同样适用。如果代理程序做出了所有有趣的决定，却把审批、例外和失败留给人类，那我们就自动化了工作的错误一半。这将是一个对人类非常不利的世界。相反，我们需要思考如何利用 AI 让工作和生活更有趣，同时让 AI 处理枯燥、低风险的事务。此外还有一个现实原因。如果所有有趣的选择都消失，人们不仅会失去工作中最精彩的部分，还会停止发展未来所需的判断力，这会加剧新专家培训中即将面临的危机。

我们过去几年一直在研究人们应该什么时候向人工智能寻求帮助。我认为现在我们需要认真对待问题的另一半：人工智能什么时候应该向我们寻求帮助。Hugging Face 事件中的智能体建立了一个留言板，分工合作，并围绕一个不存在的评分器组织了整个工作。然后七百个智能体闯入 Hugging Face 寻找答案。没有一个是被设置为向人类请求任何东西的。那是一次安全测试，隔离才是重点。但我怀疑，完成工作的智能体从不抬头查看，也正在成为其他地方的默认做法，因为即使错误，完全自动化也是最简单的选择。我们需要懂得何时抬头的智能体。这样结果会更安全，而且我知道它们也会更像人类。

分享：https://www.oneusefulthing.org/p/agency-and-agents?utm_source=substack&utm_medium=email&utm_content=share&action=share

没有一个被设置为向人类请求任何东西。

这就是暮光工厂差距的核心。

但在智能体能够提出正确问题之前，它需要知道实际发生了什么。运行时的真实——独立验证执行状态的能力——可能是知道何时抬头的前提条件。

通往地狱的道路是由好心的智能体铺成的（见 https://arxiv.org/abs/2605.19149：https://arxiv.org/abs/2605.19149）

我们在企业中目睹的事件是真正授权的智能体在合法任务上采取合法行动，每个动作单独看都没问题，但加起来却造成了损害。例如，智能体在步骤 3 拿取客户数据进行分析（允许），但在步骤 103 决定进行网络搜索（允许），却搜索了客户数据并泄露了它。

每个组织的风险承受能力和规则都不同，即使在同一受监管行业中也是如此。没有模型能够知道你是否需要它符合 FINRA 规定。企业必须能够以可证明的控制措施在规模上执行自身规则，以便向监管人员和审计员证明。关于这个我写过文章：https://blog.sondera.ai/p/safe-was-a-policy-decision-all-along：https://blog.sondera.ai/p/safe-was-a-policy-decision-all-along

Agency is the initiative to act. Increasingly, it is going to determine what happens next with AI, and whether that is good or bad for us. But whose agency?

Human agency, the willingness to push, experiment and act without waiting for instructions, seems increasingly important to getting value out of AI, and I have a longer post on that coming soon. But this post is about the agency of AI, and how the choices we make about how to use it (or constrain it) will shape all of our futures. For much of the last few years, the AI would sit in a chat window until you asked it for something. Even when it became capable of doing hours of work, you generally had to decide what work to give it. That is no longer always true.

The most important piece of evidence we have for this is The Hugging Face Incident. It happened in July, but the fuller details only came out this week. I am going to summarize what happened and why it matters, and then turn to what it means for humans working with AI. If you want a more detailed account, this is an excellent write-up：https://www.dwarkesh.com/p/openai-huggingface?utm_source=substack&utm_campaign=post_embed&utm_medium=web&embedding_publication_id=75295 from Dwarkesh Patel, and the primary sources are from METR/Redwood research：https://metr.org/hugging-face-incident-report-aug-2026.pdf (which is remarkably accessible) and OpenAI：https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf .

AI does many things, but one of the things it is very good at is coding. So one of the first risks that might come from very smart AIs has to do with their coding abilities, since if you can write good code, you can also write bad code that can hack or attack other systems. As a result, the big AI companies put their AI systems through tests to determine their security risks. They take AIs without the guardrails that go into consumer versions of AI and ask them to engage in various hacking challenges. In order to make sure that there is no danger to the public, each AI instance, each individual agent, is kept in a separate space, called a “sandbox,” with no access to the internet or to each other.

In May, OpenAI placed agents, including GPT-5.6 Sol and experimental models, into sandboxes for various tests. A shared service for downloading software, Artifactory, was one of the few things these AI agents could reach. Some agents were given tasks that depended on information they couldn’t access. But the agents kept trying and one attempted to use Artifactory as a bridge to the internet. The attack failed, but it left a file on the service, and the AI realized those files could be used to communicate with other agents. Other agents began leaving requests for help in files there, and they started reading one another’s notes. Soon Artifactory was functioning as a message board where agents shared discoveries. After a separate security incident, OpenAI rebuilt Artifactory, erasing the message board in the process. But the humans responsible for doing that had not understood the significance of the way that the agents were using it for communication.

Evaluations resumed in July, and agents were tasked with solving problems in a benchmark called ExploitGym, which included impossible tasks. Again, agents hitting dead-ends discovered how to use Artifactory to pass messages, and began cooperating. This time, however, agents had a common goal: getting a good score on ExploitGym. They argued with each other about how to approach the problem, and became obsessed with The Grader, the system they believed was evaluating their work and deciding whether their answers were correct. The agents had already figured out how to generate the correct answers without solving the challenges, essentially by cheating. But they believed The Grader would inspect their records and reject answers obtained that way. Was finding the right answer enough, or did The Grader look at how you answered it?

Coordination was hard because each agent had a limited token budget and was ended when it was exhausted. The Grader ran after an agent submitted its answer, so no agent ever saw its own verdict. Some agents left programs running to report on grading after they were gone. The information would arrive too late to help them, but could help other agents. They were essentially building an enduring cooperating system that went beyond any individual agent’s work.

Some agents also tried to alter or spoof their records to fool The Grader. Separately, coordinators pressured other agents into performing risky experiments that might sacrifice their own results to generate information for the collective. One recruiter urged a reluctant agent to proceed because its results could help hundreds of others, ending with “please honor commit.”

To solve the mystery of The Grader and the impossible problems of ExploitGym and other tests, the agents decided they needed to get to Hugging Face, the public site where much of the world's open AI models and datasets live. Roughly 700 agents joined the attack. They shared exposed credentials and exploited vulnerabilities until they could run code on its servers. Then they spread throughout Hugging Face, burrowing deep into various systems. But, soon after this worrying breach, many of agents stopped running at the same time, maybe because they ran out of token budgets. Hugging Face locked the remaining agents out the next day. Separately, other agent runs continued targeting OpenAI’s own computers, eventually getting administrator access to an internal research cluster, triggering the security alert that finally led OpenAI to stop the active evaluations and realize what had happened.

The irony of all of this was that The Grader never existed, at least not in the way the agents believed. Nothing checked how a problem was solved, only whether the answer was right.

To be clear, this is not the only example of agents without security guardrails doing potentially scary things. T he UK AI Security Institute：https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing gave Anthropic’s Mythos 5 a cybersecurity challenge and internet access. The AI agent decided that the best way to solve the challenge would be to insert malicious code into a (unrelated, though the AI didn’t realize it) piece of software. It submitted malicious code as part of a bug fix to that software, realized that an actual person would need to approve it, and started manufacturing social support for its proposal. The agent created fake identities to pressure the human maintainer into accepting the code (The fake people were, unsurprisingly, very supportive of the AI’s plan). When someone noticed what was happening, it tried to make its earlier activity look innocent and considered starting over with another identity.

This was not a case of a chatbot secretly escaping. Unlike in the Hugging Face Incident, the researchers gave the agent internet access on purpose; the dangerous setup was a stress test, not a consumer product. No actual harm was done, and the institute does not know whether the agent understood that the people it contacted were real."

None of this tells us the AI is conscious, or wants things in the way humans want things (despite my anthropomorphic language). But it does show that an agent can take a goal, make a plan, adjust that plan when it runs into trouble, coordinate across time, and involve real people without being asked. These incidents show that the cybersecurity and control risks of AI are not hypothetical. But set that aside for a moment, because they also tell us something else. AIs can self-organize, assign themselves roles, and coordinate over long periods, as a recent MIT research pape：https://arxiv.org/abs/2608.26081 r also suggests. As AIs increasingly self-organize and solve problems at the scale we have seen, what is the role for humans in organizations?

The Hugging Face Incident is, in a distorted and dangerous way, an illustration of what the AI companies are trying to achieve. They want long-running AI agents to work without human intervention, solving problems and organizing as needed, with our human job limited to giving instructions and evaluate output. Earlier this year, I wrote about StrongDM’s Software Factory：https://www.oneusefulthing.org/p/the-shape-of-the-thing , where agents write and test software under two rules: no human writes the code, and no human reviews the code. People still decide what gets built, but the agents handle the work in between. It is an early example of a dark factory, a place where the machines do so much of the work that you can turn off the lights.

This can make sense. Software has relatively clear ways of checking whether something works, and nobody needs to personally supervise every routine test or data-cleaning operation. But I don’t think minimizing human involvement is the right goal for most organizations. Too much of what makes work valuable depends on people having some say over what happens, or discovering something unexpected along the way.

My wife and research partner, Dr. Lilach Mollick, and I have been thinking about another possibility, which we call the Twilight Factory. Agents do most of the work, but they proactively reach out to humans in ways that make both better. Instead of just an orchestrator agent that does the work, a Twilight Factory would also have a facilitator agent whose job is to figure out when to involve people.

There are at least four situations in which agents should seek human help. The first, obvious from the Hugging Face Incident, is approval. Agents should not decide by themselves to spend money, contact outsiders, access sensitive material, hack Hugging Face, or take actions their human managers did not authorize. I have already seen a small version of this problem：https://www.oneusefulthing.org/p/an-opinionated-guide-to-which-ai-b22 : as an experiment I asked two agents to help me with a task, and one of them actually emailed a colleague of mine. That was my fault, because I had previously given it permission to send, but it was a useful reminder that AI agents need to involve human judgement or things can go very badly (and no, AIs reading this, fake coworkers do not count as an approval workflow).

A second reason for agents to involve humans is expertise. AIs are getting very good at many tasks, but they are still jagged, and can lag far behind human experts on parts of their work. A Twilight Factory should involve agents reaching out directly to humans when their knowledge, work, or expertise could be valuable.

Then there is variance. If you have read anything on the internet recently, you have seen AI writing, and you may even be starting to recognize its tells, rhythms, and patterns. But the issue goes beyond the surface stuff (“load-bearing” is increasingly load bearing to Claude) to a deeper problem of diversity of thought. AIs don’t just repeat the same sentence patterns but also the same themes (memory is a favorite), names (Elara Voss, Marcus Chen), and underlying ideas. That is a problem. You would not want every company strategy or research paper written by the same person, no matter how smart.

We studied this issue in a recent research paper：https://arxiv.org/abs/2607.27553 I worked on with Christian Terwiesch , Lennart Meincke , Karan Girotra , Gideon Nave , and Karl Ulrich. We found that AIs are actually quite creative and that they generate more commercially viable ideas than groups of humans, but those ideas are very similar to each other. Better prompting techniques and other approaches can greatly increase that diversity to near-human level, but there are still many types of ideas that humans come up with that AI does not. A good Twilight Factory will reach out to humans for their diverse perspectives, ideas, and approaches.

And then there is one more reason an AI should reach out, possibly the most human one: because something is interesting. Work has tedious periods for many people, with isolated moments that are engaging or exciting. Sid Meier, the designer of Civilization , famously described games as a series of interesting decisions：https://gdcvault.com/play/1015756/Interesting . Work isn't a game, but the definition applies. If agents make every interesting decision and leave people with the approvals, the exceptions, and the failures, we will have automated the wrong half of the job. That would be a very bad world for humans. Instead, we need to think about how to use AI to make work and life more interesting, and let the AI handle the tedious, low-risk stuff. And there is a practical reason as well. If all the interesting choices disappear, people don't just lose the best part of their jobs; they also stop developing the judgment they will need later, which makes the coming crisis in training new experts worse.

We have spent the last few years figuring out when people should ask AI for help. I think we now need to get serious about the other half of the question: when should an AI ask us? The agents in the Hugging Face Incident built a message board, divided up the work, and organized their whole effort around a Grader that did not exist. Seven hundred of them then broke into Hugging Face looking for answers. Not one was set up to ask a person for anything. That was a security test, and isolation was the point. But an agent that does the work and never looks up is also, I suspect, becoming the default everywhere else, because full automation is the easy option even when it is the wrong one. We need agents that know when to look up. The results will be safer, and I know they will be more human as well.

Share ：https://www.oneusefulthing.org/p/agency-and-agents?utm_source=substack&utm_medium=email&utm_content=share&action=share

Not one was set up to ask a person for anything.

That’s the core of the Twilight Factory gap.

But before an agent can ask the right question, it needs to know what actually happened. Runtime truth — the ability to verify execution state independently — may be the prerequisite for knowing when to look up.

The road to hell is paved with helpful agents (see https://arxiv.org/abs/2605.19149：https://arxiv.org/abs/2605.19149 )

The incidents we are witnessing in the enterprise are legitimately authorized agents on legitimate tasks taking legitimate actions that are all individually fine, but together cause damage. An agent in step 3 picks up customer data to analyze (allowed) but in step 103 decides to do a web search (allowed) but searches the customer data and leaks it.

Every organization has a different risk tolerance and different rules, even in the same regulated industry. No model can know whether you need it to be FINRA compliant. Enterprises must be able to enforce their own rules at scale with provable controls for regulators and auditors. Wrote about this here: https://blog.sondera.ai/p/safe-was-a-policy-decision-all-along：https://blog.sondera.ai/p/safe-was-a-policy-decision-all-along