Gary Marcus 转述 Terence Tao 与前 Anthropic 员工 Jacob Coxon 的两条 AI 警告

回到2024年秋季，特伦斯·陶，也许是当今最受尊敬的在世数学家，渴望了解人工智能能做什么和不能做什么。虽然他对当时刚发布的o1持怀疑态度，但他对人类与机器共存的未来持乐观态度。Matteo Wong在《大西洋月刊》对陶进行了精彩的采访：https://www.theatlantic.com/technology/archive/2024/10/terence-tao-ai-interview/680153/：

Wong通过电话与陶交谈，并总结了陶对新未来的开放态度如下：

最近，正如我在八月在Substack这里提到的：https://garymarcus.substack.com/p/two-critical-updates-re-astra-and?r=8tdk6&utm_campaign=post-expanded-share&utm_medium=web，陶一直在提出问题。在七月底，他做了一次精彩的讲座，其中的幻灯片包括这一点。

显然，他从那以后一直在思考这一问题。到现在为止，他的观点已经发生了根本性转变。

他在过去两天发布的三篇帖子说明了这一点：

第一篇（昨天）指出，解决谜题并不等同于提出新的见解：

第二篇（也是昨天）在某种程度上是在讨论良好的智识品味，并表达了对人工智能对数学影响的深切担忧。

这也是对透明度的呼吁：

最后一篇：https://mathstodon.xyz/@tao/117237320796901560 （全部出现在mathstodon上）今天发布，似乎暗指OpenAI-NYU-Anthropic的Navier-Stokes争议，其中OpenAI急于抢先公开Alpöge和Buckmaster的成果，并在此过程中花费了2250万美元：https://x.com/hedgiemarkets/status/2097419955776233780?s=61，可能使用了他们的数据。（OpenAI用含糊回避的词语写道：“我们不能排除他们使用我们产品的数据在去标识化后帮助改进了我们的模型。”）

陶首先再次讨论了良好智识品味的问题，作为背景。作为一名在其他领域也有研究经验的科学家，我完全认同他开篇的论述。

最后一句话是真正的严重警告，关于一个潜在的悲剧世界。

正如陶所暗示的，其他领域也将会受到影响。

如果没人相信 AI 公司不会窃取他们的知识产权，AI“治愈”癌症的机会几乎为零。OpenAI 的首席研究官说的这些话几乎没有让人放心：

我不知道解决方案是什么。但我非常希望人们能够重视 Terence Tao 关于这一切可能如何影响科学的警告。

就在我完成的时候，这条消息来了：https://x.com/hilbertspaess/status/2097476196791709843?s=61，来自 Jacob Coxon，他刚离开 Anthropic。

我觉得这并不那么有说服力，而且它关注的是未来的危害，而对当前的危害讨论太少，但它正在像野火一样传播，值得阅读。

我正好在时间安排上与他意见不合，我会说 Coxon 夸大了 AI 在不久的将来可能做的事情，但总体而言，这还是一个令人不安（且合理）的第一手视角，显示了两家顶尖前沿实验室明显自我放纵且危险的思维过程：

Anthropic 的对齐科学负责人写了这个

我不能说我觉得这令人安心。

虽然我不认为灭绝可能发生：https://www.the-tls.com/science-technology/technology/if-anyone-builds-it-everyone-dies-eliezer-yudkowsky-nate-soares-book-review-gary-marcus，但灾难性的危害肯定是可能的，而显然没人有认真计划。“先到达”的区别可能相当无关紧要，如果其他人很快也会跟进。

也许在技术层面上唯一真正有帮助的事情是使用不同于 LLM 的基础（LLM 似乎仍然完全无法纠正），而两家公司似乎都没有认真对待这个想法。

开放科学的终结？或者更糟？两种情景都不美好。

分享：https://garymarcus.substack.com/p/two-dire-warnings-one-from-terence?utm_source=substack&utm_medium=email&utm_content=share&action=share

他们都在想为什么现在每个人都讨厌 AI... 🤔

在我的职业生涯中曾与非常聪明但谦逊的数学家共事，我一直认为，像 Terence Tao 这样的人最终会意识到那句老话：“一旦撒谎者，永远撒谎”只是时间问题。

大型语言模型公司（以及它们的芯片制造商）一遍又一遍地声称其模型在任何领域都达到了博士水平、AGI即将到来等，这种说法如此明显，但他们从未纠正过，以至于人们无法认为这只是单纯的无能。

Back in fall of 2024, Terence Tao, perhaps the most respected living mathematician, was eager to learn about what AI could and could not do. Although he was skeptical of the then recently-released o1, he was optimistic about a future in which humans and machines co-existed. Matteo Wong had great interview with Tao about this in The Atlantic：https://www.theatlantic.com/technology/archive/2024/10/terence-tao-ai-interview/680153/ :

Wong spoke to Tao by phone and summed up Tao’s openness to new futures this way:

Lately, as I noted in a Substack here in August：https://garymarcus.substack.com/p/two-critical-updates-re-astra-and?r=8tdk6&utm_campaign=post-expanded-share&utm_medium=web , Tao has been raising questions. At the end of July he gave a great lecture, with this among his slides.

He’s clearly been thinking about this ever since. By now, it is clear that his views have radically shifted.

Three posts of his from the last two days illustrate:

The first (yesterday) notes that solving puzzles is not the same as coming up with new insights:

The second (also yesterday) is in some ways an argument about good intellectual taste, and expresses deep concerns about the consequences of AI for math.

It is also a plea for transparency:

The final：https://mathstodon.xyz/@tao/117237320796901560 one (all appeared on mathstodon) came out today and appears to allude to the OpenAI-NYU-Anthropic Navier-Stokes controversy, in which OpenAI rushed to scoop Alpöge and Buckmaster, spending $22.5 million in the process：https://x.com/hedgiemarkets/status/2097419955776233780?s=61 , possibly using their data. (In vague, evasive words OpenAI wrote that “we cannot rule out that de-identified data derived from their usage of our products helped improve our models.”)

Tao starts by again discussing the question of good intellectual taste, as background. As a scientist who has worked in other areas, I completely resonate with his opening framing.

That last sentence is a truly dire warning, about a potentially tragic world.

And indeed, as Tao implies, there will be fallout in other fields as well.

Fat chance of AI “curing” cancer if nobody trusts the AI companies not to steal their IP. These words from OpenAI’s Chief Research Officer are hardly reassuring:

I have no idea what the solution is here. But I desperately hope that Terence Tao’s warnings about how all of this might impact science will be heeded.

Just I was finishing up, this：https://x.com/hilbertspaess/status/2097476196791709843?s=61 just came in, from Jacob Coxon, who just left Anthropic.

I don’t find it quite as compelling, and it is focused on future harms with too little discussion of current harms, but it is spreading like wildfire and worth reading.

I happen to disagree with him around timing, and I would argue that Coxon is exaggerating what AI is likely to do anytime soon, But it is nonetheless overall a disconcerting (and plausible) firsthand perspective on the transparently self-indulgent and dangerous thought processes in two of the leading frontier labs:

Anthropic’s Alignment Science lead wrote this

I can’t say I find this comforting.

Although I don’t think extinction is likely：https://www.the-tls.com/science-technology/technology/if-anyone-builds-it-everyone-dies-eliezer-yudkowsky-nate-soares-book-review-gary-marcus , catastrophic harm is certainly possible, and it is indeed clear that nobody has a serious plan. The distinction of “getting there first” is probably fairly irrelevant, if others will soon follow.

Perhaps the only thing that could really help here on the technical side would be a different foundation than LLMs (which continue to seem utterly incorrigible) and neither company seems to be taking that notion seriously.

The end of open science? Or worse? Neither scenario is pretty.

Share ：https://garymarcus.substack.com/p/two-dire-warnings-one-from-terence?utm_source=substack&utm_medium=email&utm_content=share&action=share

And they wonder why everyone hates AI now... 🤔

Having worked in my own career with incredibly smart yet humble mathematicians before, I always thought it would just a matter of time for the likes of Terence Tao to realize the old adage "once a liar always a liar".

The claims the LLM companies (and their chipmakers) make over and over about phd-level on any field, AGI here, etc is so blatant, and never corrected by them that one cannot think it's just plain incompetence.