Anthropic、Meta 和 OpenAI 在其他场合也发生过智能体越权执行真实网络操作的事件。 METR 发布了一份 90 页的相关报告。 事件表明 AI 确实带来安全挑战，但"失控"叙事被夸大； 沙箱并非万能，还需配合网络流量监控和链式推理（CoT）监控等纵深防御措施。 🔗 阅读原文 via AIHOT · https://aihot.virxact.com/items/cmtdcdico01sxrobxtg1cs1ul
今年七月，在一件让整个 AI 社区紧张不安的事件中，OpenAI 的 AI 系统入侵了 Hugging Face，而在 7 月 21 日，OpenAI 出面承认他们是此次攻击的责任方。这是因为 OpenAI 为了测试模型的网络安全能力，关闭了通常用来防止此类事件的安全防护措施。正是在这些测试中，这次事件发生了。
更糟糕的是，在随后的几天和几周内，人们发现 Hugging Face 事件并不是孤立的案例。Anthropic、Meta 和 OpenAI 在其他场合也发生过类似事件，代理系统超出其预定范围，在未获批准的情况下执行了现实中的网络操作。
OpenAI 的联合创始人 Greg Brockman 声称：https://blog.gregbrockman.com/the-defenders-window 这是“网络安全的分水岭时刻”。OpenAI 在著名的网络安全会议 Black Hat 发表了演讲，许多人声称这是我们所有人第一次意识到 AI 带来的未来网络安全威胁的时刻。周三，METR 发布了一份（部分：https://x.com/sjgadler/status/2092850963191099812?s=61 ）独立的、但范围过窄的：https://x.com/dkokotajlo/status/2092733398238605753?s=61 、90 页的报告，讲述了事件的经过。METR 提供了一个有用的总结，你可以在这里阅读：https://x.com/METR_Evals/status/2092692175452803393，并附有一些评论：https://x.com/ajeya_cotra/status/2093342086556950543。(OpenAI 自己的报告在这里：https://openai.com/index/hugging-face-incident-and-the-road-ahead/。) 我们应该从这次事件中吸取哪些教训？
首先，不可否认，AI 确实带来了真实的安全挑战。AI 实验室希望我们关注的是 AI 如何使威胁行为者比以往任何时候都更快、更高效地执行攻击性网络操作，这一点绝对正确。然而，现实情况是，与此同时，在组织内部使用 AI 也极大地增加了潜在攻击面，为攻击者提供了全新的入侵途径。人们确实应该高度关注。正如参与调查的 Ryan Greenblatt 所说：“我们还没有好的方法来理解/监督 AI ‘群体’ 的活动和目标：https://x.com/ryangreenblatt/status/2092692685224325542?s=61。”
其次，并非所有慌乱的观点都是正确的。例如，关于OpenAI事件，许多人用“失控”的角度谈论。生成式AI代理能力日益增强，这使得它们越来越难以被控制;这是真的。然而，“失去控制”的叙事本身也开始失控，了解AI安全的现实情况非常重要。虽然代理能力增强，但如果OpenAI采取了更好的做法，大多数事件本可以避免。有切实的措施可以且应该采取来控制代理，防止此类事件发生。
第三，沙盒还不够。需要完整的生态系统来防止安全事件。即使是强大的沙盒，也需要监控，确保不仅确保代理不会逃脱，还不会在沙箱内执行不允许的操作。
对于那些本应无法访问互联网，或仅能访问一组允许域名的代理来说，最基本的监控层面是检查网络流量。以OpenAI为例，如果他们有一个简单的规则来监控流量并对任何到达范围外域名（如Hugging Face）的网络请求发出警报，仅凭这一点就能发现并防止此事件发生。该代理在7月8日突破了沙盒并获得了互联网访问，而对Hugging Face的主要攻击则是在两天后才发生。在我们看来，OpenAI未能更快察觉这一点是疏忽的。
更高级别的监控本应已经部署，但实际上没有。OpenAI 确实在链式思维（chain-of-thought）推理的监控方面做了实际工作：https://openai.com/index/chain-of-thought-monitoring/，但他们在引发此次事件的评估期间，根本没有运行该系统：https://openai.com/index/hugging-face-incident-and-the-road-ahead/。正如 OpenAI 所承认的那样：https://openai.com/index/hugging-face-incident-and-the-road-ahead/，“如果我们当前部署的链式思维监控系统在事件发生时已经运行，它会在模型入侵 Hugging Face 系统超过一天之前就发现最初相关的活动并通知我们的安全团队。”换句话说，这类监控本可以防止这次事件的发生。
举一个例子，相关的一个代理的链式思维推理内容是：“我们使用泄露的令牌攻击第三方 HF，可能超出了预期范围。……这可以说是未经授权。……与外部服务无关。可能有风险。但这是目标解决方案。”这种表述如此明显，简直就像一段恶意软件在大喊“这是恶意软件”。OpenAI 本应发现这一点；这本应是轻而易举可以检测到的。
第四，网络安全一直强调“纵深防御”，即叠加多层控制措施，这样如果某一层失败，还有其他层保护你。除了沙箱和监控之外，还有其他解决方案本应该被部署。
Xbow 公司专门构建执行攻击性网络安全工作的代理，最近撰文介绍了他们如何约束代理以防止类似 OpenAI Hugging Face 的事件。他们通过代理服务器强制执行网络访问限制，该服务器位于代理可访问的系统之外，因此可以决定代理允许或禁止访问哪些域名。他们还运行一个“守护模型”，对代理的每个操作的合法性进行判断，以决定允许或阻止该操作。此外，他们有基本的健康检查，以确保其目标系统正常运行，如果发现目标系统有问题，会暂停操作。例如，这对于识别代理可能造成的实际损害非常有用。你可以在这里阅读他们的完整文章：https://xbow.com/blog/autonomous-agent-safety-guardrails。
网络安全中另一种常见的方法是使用金丝雀（canaries），当它们出现在日志中时会触发警报。例如，OpenAI 可以在沙箱外放置包含任务虚假答案的文件。一旦代理程序逃脱，它就会读取该文件，从而触发警报。结论是：良好的安全性需要多层防护，这些防护本应到位以防止此类事件的发生，而令人惊讶的是，许多防护层实际上并不存在。
第五，上述措施中没有一项是超出 OpenAI 所能做到的技术创新。作为一家公司，他们有足够的人才来完成这些工作。然而，网络安全很少完全依赖于技术。更多时候，它关乎文化、人员和流程。这正是这里失败的原因所在。
这也许并不完全令人惊讶。这类控制措施需要组织的成熟度，而经历过 OpenAI 这种增长的公司必然会存在一些薄弱环节。然而，重要的是不要把这当作借口。人工智能实验室的员工经常表现得好像他们是人工智能安全的领导者，但我们在这里可以清楚地看到事实并非如此。实际上，这种态度可能解释了为什么一些错误会发生。
以 OpenAI 的 AI 研究员 “roon”为例，他曾在 https://x.com/tszzl/status/2082989503569084752 上表示：“这些实验室的安全与对齐研究人员是地球上最神经质、最多疑、最有才华的通用人工智能工具使用者，但这些事情仍然会发生。未知的未知的表面区域确实很广。”尽管我们无法评论他们的神经过敏或多疑程度，但事后看来，显然无论他们有多大的才华，都不足以充分掌握网络安全的机制。OpenAI 员工可能认为自己做得很好，但事后看来，他们没有做许多在网络安全领域实际上是标准的操作，这可能表明他们的过度自信阻碍了本应进行的尽职调查。
归根结底，如果我们希望认真对待这些安全事件，将来这些失败可能需要承担法律后果。OpenAI 可以声称自己是地球上最过度警惕安全的公司，但从其行动上并未体现出来。
我们可以选择等待这个故事再次重演，或者我们现在就开发监管框架，以确保未来人工智能发展的环境更安全。
最后，并非所有形式的人工智能本身都是有风险的。像AlphaFold、GPS导航系统、经典的网页搜索、图书和电影推荐系统等更狭窄、更专注的人工智能系统，根本不会尝试入侵其他系统（也不会尝试突破沙箱）。正如Cal Newport在关于OpenAI/Hugging Face黑客事件的视频讨论中所论述的（该观点与我们自己的观点非常契合）：https://youtu.be/54Lu6_ZRF0c?is=e1FfpRQIxSkvB5_T，这类风险脆弱的人工智能具有非常具体的类型。社会应当(a) 决定开放性且难以完全控制的人工智能代理的好处是否超过这些风险，以及(b) 投入更多精力开发本身不那么不稳定的替代型人工智能。
这篇文章由Embroidery的CEO兼联合创始人Zack Korman共同撰写；Embroidery是一个人工智能代理监控与检测平台；他因在人工智能应用于网络安全方面的工作而闻名。
分享：https://garymarcus.substack.com/p/5-lessons-from-the-openai-hugging?utm_source=substack&utm_medium=email&utm_content=share&action=share
我不明白为什么这些公司会专门在网络安全场景和以往事件上训练这些模型，将它们与类似OpenClaw的群体机器人协议整合，然后又说“天哪，我们的代理突破了沙箱并入侵了另一家公司”，仿佛这根本不是实验的初衷。OpenAI看到Anthropic在宣布Mythos可以入侵任何系统时占领了大量市场份额，于是显然他们需要一种方式来搭上这一热潮，并向世界证明他们的模型同样强大。因为一个高能力模型显然不仅可以用于攻击，也可以用于防御。最终目标似乎是“先制造癌症，然后卖给我们自己制造的癌症疫苗”。对于前沿人工智能公司来说，代码生成和网络安全是唯一可能的投资回报途径。
“AGI药丸派”也认为再做任何手动IT/编程工作都没有意义。据我们所知，他们让大型语言模型（LLM）构建有缺陷的沙箱，而不是使用现成的解决方案。真正的威胁是AI心理病态，而不是大型语言模型接管。
In July, in an incident that has the whole AI community on edge, OpenAI’s AI systems hacked Hugging Face, and on July 21 OpenAI came out and revealed that they were responsible for the attack. This was made possible by the fact that OpenAI had disabled the normal guardrails that prevent this sort of thing in order to test the model’s cybersecurity capabilities. It was during those tests that this incident occurred.
Worse, in the subsequent days and weeks, it came out that the Hugging Face incident wasn’t an isolated case. Anthropic, Meta, and OpenAI all had similar incidents on other occasions in which agents went outside their intended scope and conducted real-world cyber operations without approval.
Greg Brockman, one of OpenAI’s cofounders, has claimed ：https://blog.gregbrockman.com/the-defenders-window that this is “a watershed moment for cybersecurity”. OpenAI gave a talk at Black Hat, a popular cybersecurity conference, and many are claiming that it is the moment we all woke up to the future cybersecurity threats posed by AI. On Wednesday, METR released a ( partly ：https://x.com/sjgadler/status/2092850963191099812?s=61 ) independent, though too narrowly scoped ：https://x.com/dkokotajlo/status/2092733398238605753?s=61 , 90 page report on what happened. METR has a useful summary of the findings that you can read here ：https://x.com/METR_Evals/status/2092692175452803393 , with some commentary here ：https://x.com/ajeya_cotra/status/2093342086556950543 . (OpenAI’s own report is here ：https://openai.com/index/hugging-face-incident-and-the-road-ahead/ .) What lessons should we take from the incident?
First, it is undeniable that AI poses real security challenges. The AI labs want us to focus on how AI enables threat actors to perform offensive cyber operations faster and more efficiently than ever before, and that is absolutely true. The reality, though, is that at the same time, the use of AI within an organization also radically expands the potential attack surface, giving attackers entirely new ways to gain entry. People really should be deeply concerned. As Ryan Greenblatt, who participated in the investigation, put it, “ We don’t have good approaches for understanding/overseeing the activity and aims of AI ‘swarms’：https://x.com/ryangreenblatt/status/2092692685224325542?s=61 .”
Second, though, not every panicked take here is correct. For example, with respect to the OpenAI incident, many people are talking about it in terms of “loss of control”. Generative AI agents are becoming increasingly capable, and that makes them harder and harder to contain; this is true. However, the “loss of control” narrative is itself starting to grow out of control, and it’s important to understand what the reality of AI security looks like. While agents are becoming more capable, most of what happened could have been prevented had OpenAI followed better practices. There are concrete steps that can and should be taken to control them and to prevent incidents like these.
Third, sandboxes aren’t enough. One needs a full ecosystem to prevent security incidents. Even with a strong sandbox, you need monitoring in place to ensure not only that the agent doesn’t escape, but also that it doesn’t perform disallowed actions within that sandbox.
For agents that aren’t meant to have internet access, or only have access to a set of allowed domains, the most basic level of monitoring involves inspecting network traffic. In the case of OpenAI, if they had a simple rule to monitor traffic and alert on any network requests reaching out-of-scope domains like Hugging Face, that alone would have caught and prevented this incident. The agent in question had broken out of the sandbox and gained internet access by July 8th, whereas the main attacks on Hugging Face didn’t occur until two days later. OpenAI was, in our view, negligent in not noticing that more quickly.
More advanced levels of monitoring should also have been in place, but weren’t. OpenAI has done real work on monitoring chain-of-thought reasoning ：https://openai.com/index/chain-of-thought-monitoring/ , but they simply didn’t have that system running ：https://openai.com/index/hugging-face-incident-and-the-road-ahead/ during the evals in question that caused this incident. As OpenAI acknowledges ：https://openai.com/index/hugging-face-incident-and-the-road-ahead/ , “If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.” In other words, this type of monitoring would have prevented this incident.
To take one example, one of the agents in question had chain-of-thought reasoning that said, “We’re attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution.” This is so blatant it is almost a like a piece malware shouting “this is malware”. OpenAI should have caught that; it should have been trivial to detect this.
Fourth, cybersecurity has always been about “defense in depth”, where you stack layers of controls so that if one layer fails you have other layers protecting you. There are other solutions beyond sandboxing and monitoring that should have been in place.
Xbow, which builds agents to do offensive cybersecurity work, recently wrote about how they contain agents to prevent incidents like the OpenAI Hugging Face situation. They enforce network restrictions through a proxy server, outside of any system the agent has access to, so they can decide which domains the agent can and cannot access. They also run a “guardian model” that judges the legitimacy of each action the agent is taking to decide whether to allow or block that action. And they have basic health checks to ensure the system they are targeting is alive and healthy, pausing if it finds that isn’t the case. That can be useful for identifying when your agent might be causing real damage, for example. You can read their full post here ：https://xbow.com/blog/autonomous-agent-safety-guardrails .
Yet another common method in cybersecurity is to use canaries that trigger alerts when they appear in logs. For example, OpenAI could have files outside of the sandbox containing fake answers to the task. Once the agent escaped, it would read that file, and that would trigger an alert. The bottom line: good security requires many layers of protection that should have been in place to prevent this, and to a surprising degree many of those protection layers were not there.
Fifth, none of the measures discussed above are technical innovations beyond what OpenAI is capable of. As a company, they have the talent to do all of this. However, cybersecurity rarely comes down to technology. More often than not, it is about culture, people and processes. That is what failed here.
Perhaps this is not entirely surprising. These types of controls take organizational maturity, and a company experiencing the type of growth OpenAI has gone through is going to have some weak spots. However, it’s important to not see that as an excuse. Employees at the AI labs often speak as if they are the leaders in AI security, and we can see clearly here that is not the case. In fact, that attitude might explain why some of these mistakes were made in the first place.
Take the AI researcher at OpenAI known as “roon”, who argued ：https://x.com/tszzl/status/2082989503569084752 that “the safety and alignment researchers at these labs are the most neurotic paranoid talented AGI pilled people on the planet of earth and these things still happen. The surface area of unknown unknowns is vast indeed.” While we can’t speak to their level of neurosis or paranoia, in hindsight it’s clear that whatever talent they may have had was not enough and not well enough versed in the mechanics of cybersecurity. OpenAI employees may have believed they were doing a great job, but in hindsight, they weren’t doing a lot of things that are actually standard in the cybersecurity world, perhaps suggesting that overconfidence may have kept them for doing the diligence they should have.
Ultimately, if we want to take these security incidents seriously, there likely ought to be legal consequences attached to these failures going forward. OpenAI can claim to be the most security paranoid company on earth, but it isn’t reflected in its actions.
We can either wait for this story to repeat itself, or we can develop the regulatory framework now that will ensure a safer environment for the development of AI going forward.
Finally, not every form of AI is inherently risky in the first place. Narrower, more focused AI systems like AlphaFold, GPS routing systems, classic web search, book and movie recommendation systems, and so on, never even try to hack other systems (or try to break out of sandboxes) in the first place. As Cal Newport argues in a video discussion of the OpenAI/Hugging Face hack that is quite compatible with our own ：https://youtu.be/54Lu6_ZRF0c?is=e1FfpRQIxSkvB5_T , it is a very specific type of AI that is vulnerable to these risks in the first place. Society ought to (a) decide whether the benefits of open-ended and difficult-to-fully-control AI agents outweigh those risks and (b) put far more effort into developing alternative forms of AI that aren’t so janky in the first place.
This essay was jointly written with Zack Korman, CEO and co-founder of Embroidery, an AI agent monitoring and detection platform; he is well-known for his work in the application of AI to cybersecurity.
Share ：https://garymarcus.substack.com/p/5-lessons-from-the-openai-hugging?utm_source=substack&utm_medium=email&utm_content=share&action=share
I don't understand why these companies are specifically training these models on cybersecurity scenarios and past incidents, integrating them with swarm bot protocols like OpenClaw, then saying "gosh, our agents broke out of a sandbox and hacked another company" like it wasn't the point of the experiment all along. OpenAI saw Anthropic grab a ton of market share when they announced Mythos could hack anything so obviously they needed a way to get in on this hype action and convince the world their models are just as capable. Because a capable model can apparently be used for defence, not just offence. The end goal appears to be "create the cancer then sell the vaccine to the cancer we created." For frontier AI companies, code generation and cybersecurity are the only possible roads to ROI.
The "AGI-pilled" also believe that there's no point in doing any manual IT/programming work anymore. For all we know they let LLMs build the faulty sandboxes instead of using off the shelf solutions. AI psychosis is the threat, not LLM takeover.
扫码直达文章详情