Dylan Castillo：https://dylancastillo.co
在过去几年里，Simon Willison：https://simonwillison.net/tags/pelican-riding-a-bicycle/ 使用同一个提示测试了每一个主要的大型语言模型发布版本：“生成一只骑自行车的鹈鹕的SVG。”
这个最初开玩笑式的基准测试已经成为人工智能中最著名的非正式基准测试之一。Simon 的骑自行车鹈鹕的结果通常是 Hacker News 上发布 AI 实验室新版本的帖子中点赞最多的评论之一。
这个基准现在已经足够有名，因此有大量的讨论：https://news.ycombinator.com/item?id=48994959 关于它的实用性以及 AI 实验室是否可能在上面进行“benchmaxxing”1：#fn1。当数十亿甚至数万亿美元的利益攸关，而一个强有力的结果可能帮助说服用户时，会不会很诱人去稍微让模型实现“pelicanmaxx”？
我想弄清楚，所以我组织了一个小实验。我在七个前沿模型中生成了1,008个SVG，用一个大型语言模型（LLM）进行评分，并使用 Claude Fable 5 进行分析。
本文呈现了实验结果。所有代码都可以在 Github 上获取：https://github.com/dylanjcastillo/blog/tree/main/_extras/pelicanmaxxing。
我建立了一个包含8种动物 × 6种交通工具 = 48个提示的网格，其中著名提示是其中的一个单元格：
每个提示几乎使用与 Simon 相同的措辞，只是更换了动物和交通工具。动物和交通工具的选择并没有采用非常严格的方法，但我尽量在与原始提示的相似性和难度上有所变化。火烈鸟和苍鹭与鹈鹕相当相似；猫、浣熊和水獭是简单案例；羚羊较难；而鲸鱼则是差异最大的一类。
我通过 OpenRouter：https://openrouter.ai/ 测试了七个模型：GPT-5.6 Terra、Claude Sonnet 5、Gemini 3.5 Flash、Grok 4.5、Qwen3.7-Max、GLM-5.2 和 DeepSeek V4 Pro。我为每个提示生成了3个样本，温度设为1.0，并要求每个模型使用相同的推理努力。这样总共生成了1,008个SVG。
然后我将每个图像通过一个三阶段管道处理：
我的假设是，如果实验室在该基准上进行了训练，它应该会在以下某种组合中表现出来：鹈鹕行的得分高于该动物应得的分数，自行车列的得分高于该交通工具应得的分数，或特定的鹈鹕-自行车单元格同时超出两者。
在任何评分之前，最简单的测试是你自己看看图片。选择一个实验室查看它绘制的所有内容，每张图片下都有裁判的评分（点击可以查看完整尺寸）：
在进行以下分析之前，我自己浏览了这些图像。没有什么特别突出。我找不到鹈鹕-自行车的图片明显好于该模型网格中其他图片的情况。也许在 GLM-5.2 的第一批样本中，它感觉略好于其他图片，但那批样本也生成了一只相当酷的滑板鹭，所以我不能确定。除此之外，它们看起来和每个模型绘制的其他内容一样，那些能画出优秀鹈鹕骑自行车的实验室，其他动物-交通工具组合也做得很好。
但这个测试难以复现，每个人的看法也不一样。所以我想要一些更量化的东西，这就是我选择上述方法的原因。
以下是按动物汇总、跨所有模型的动物平均评分：
鹈鹕在 8 种动物中排第 6，落后于猫、鲸鱼、浣熊、鹭和羚羊。如果 AI 实验室在基准上进行了训练，你会期望鹈鹕得分排在前列。而实际上它们位于下半部分。所有七个实验室画的猫、鲸鱼和浣熊都比画鹈鹕好。
当然，鹈鹕可能比猫更难画。一个实验室即使在鹈鹕上进行了训练，也可能无法让它们超过容易画的动物，所以仅凭这个排名不能排除这一点。我将在证据 #4 中调整难度因素。
自行车的表现更差。它们排倒数第二，几乎与飞机并列，而飞机排在最后：
如果实验室在基准上进行了训练，你会期望自行车在这个排名中位于靠前位置。但实际上并非如此。然而，同样的警告在此也适用。自行车比滑板更难画：它需要两个匹配的轮胎，一个延伸到两个车轴的车架，车把，座椅和踏板。裁判指出，在 2/3 的自行车图片中，其中一项要么缺失要么未连接。你可以在自行车图片上进行训练，但与较简单的交通工具相比，效果仍可能不佳。
不过关于飞机有一点需要说明：我本应该选择“airplane”而不是“plane”，因为模型通常从几何角度理解“plane”。它们把动物画在平面上，而不是画成驾驶飞机。飞机是唯一一个特征提取器有时根本找不到车辆的交通工具（168张图片中有25张，其他五种交通工具都是零），20%的飞机图片在车辆评分上得分为1或2，而自行车为5%，船、滑板车或滑板则完全没有低分。
综合来看，‘骑自行车的鹈鹕’最终排名靠后，在48个组合中排第42位：
不过，某些组合可能确实比其他组合更难画。
为了解决这个问题，我在所有1,008张图片上拟合了一个固定效应回归：score ~ lab + animal × vehicle，并加上每个实验室针对鹈鹕、自行车以及鹈鹕-自行车组合的交互项，并使用稳健标准误。animal × vehicle 项吸收了所有48种组合的固有难度。交互项测量每个实验室相对于平均实验室的专项加成，同时给出置信区间。
以下是每个实验室的完整估计结果。一个鹈鹕表现最优的实验室会在整行中看到点都在零线的右侧：
每个鹈鹕区间和每个单元格区间都包含零。恰好有一个不包含：Gemini 3.5 Flash在自行车列中。但是在21次检验中，随机出现约一个假阳性是正常的（21 × 0.05 ≈ 1.05），而这个结果正好符合预期。同时，它在多重比较校正下也不显著：21次检验的Bonferroni门槛为0.05/21 ≈ 0.002，而其p值为0.022。完整的估计值和p值表格在代码仓库：https://github.com/dylanjcastillo/blog/tree/main/_extras/pelicanmaxxing。
但这些区间较宽，平均约为 ±0.6 个评审分数。任何小于这个幅度的加成都无法通过此检验捕获。
有人建议骑自行车的鹈鹕看起来像是记忆中构图出来的，指出一些重复模式，例如鹈鹕总是面向右侧，或者重复出现的元素如太阳或围巾。因此我想知道这是否属实。
方向：所有21张鹈鹕-自行车图片，在七个实验室中都面向右，没有其他动物/交通工具组合有这种情况。
然而，面向右侧很常见：在所有1008张图片中，有60%是这样。具体有多普遍取决于动物和交通工具，而自行车是两种最明显的交通工具之一：
鹈鹕也是倾向于面向右侧的动物之一：
很难画出面向观众的鹈鹕或自行车，所以模型几乎总是从侧面画它们，面向左或右。这就是为什么它们的图像很少模糊不清。其他组合也几乎是一致的：滑板上的羚羊和滑板上的鹈鹕达到21张中的20张，自行车上的鹭鸟达到21张中的19张。所以21张全正确看起来并不奇怪。
场景元素：我让提取器命名它在图像中看到的任何元素。这些是统计数据：
一个记忆化场景会表现为同一套元素在每张图片中重复出现。我去寻找了这一点，发现确实有些组合每次都会生成相同的元素。每一张船上的火烈鸟图片里都有太阳。飞机上的水獭38%会戴围巾。自行车上的猫38%会有篮子。
自行车上的鹈鹕似乎并没有特别不同的地方。它只是有一些出现频率更高的元素，就像其他动物-交通工具组合一样。
抱歉，HN的反对者，但没有多少证据表明AI实验室在刻意强化鹈鹕。或者至少他们没有以明显的方式这么做。
鹈鹕并没有比其他动物画得更好。自行车也没有比其他交通工具画得更好。而且没有哪个实验室能比其现有的鹈鹕和自行车预测效果更好。GLM-5.2最接近：它在鹈鹕-自行车具体单元上提升最大，而且它的第一个鹈鹕骑自行车样本引起了我的注意。但这种效果很小且不显著，所以我不会把它看得太重。
另一件突出的事情是场景构图中的方向。所有21张鹈鹕骑自行车的图片都面向右侧，这是网格中唯一每张图片都一致的组合。但这看起来并不奇怪。在整个实验中，面向右侧是常态。其他三个组合的面向右侧比例达到90%或以上，而在48个组合中，我一点也不惊讶有一个达到了21张全对。
更可信的说法是，SVGmaxxing 就像 Google/DeepMind 那样进行。其他实验室可能在更低调地做这件事。遗憾的是，这个实验无法说明到底是谁在做。但至少你今晚可以安心睡觉，因为 AI 实验室并不是为了愚弄 Simon Willison 而生成成千上万的骑自行车的鹈鹕数据。
如果你想自己查看数据，完整的流程在仓库里：https://github.com/dylanjcastillo/blog/tree/main/_extras/pelicanmaxxing。
优化 AI 模型以在流行基准测试中取得高分的做法。↩︎：#fnref1
Dylan Castillo：https://dylancastillo.co
For the past few years, Simon Willison：https://simonwillison.net/tags/pelican-riding-a-bicycle/ has tested every major LLM release with the same prompt: “Generate an SVG of a pelican riding a bicycle”.
What began as a tongue-in-cheek benchmark has become one of the most famous informal benchmarks in AI. Simon’s pelican-on-a-bicycle results are often among the most upvoted comments on Hacker News threads announcing new releases from AI labs.
The benchmark is now famous enough that there’s plenty：https://news.ycombinator.com/item?id=48994959 of：https://news.ycombinator.com/item?id=48994217 discussion：https://news.ycombinator.com/item?id=48956159 about its usefulness and about whether AI labs might be benchmaxxing 1 ：#fn1 on it. When billions or even trillions of dollars are at stake, and a strong result could help persuade users, wouldn’t it be tempting to pelicanmaxx your model just a bit?
I wanted to find out, so I put together a small experiment. I generated 1,008 SVGs across seven frontier models, scored them with an LLM judge, and used Claude Fable 5 for the analysis.
This article presents the results. All the code is available on Github：https://github.com/dylanjcastillo/blog/tree/main/_extras/pelicanmaxxing.
I built a grid of 8 animals × 6 vehicles = 48 prompts, where the famous prompt is one cell:
Every prompt uses almost identical phrasing to Simon’s, only switching the animal and vehicle. The animal and vehicle selection wasn’t done in a very rigorous manner, but I tried to vary both similarity to the original prompt and difficulty. Flamingo and heron are quite similar to pelicans; cat, raccoon, and otter are easy cases; antelope is hard; and whale is as different as you can get.
I tested seven models through OpenRouter：https://openrouter.ai/: GPT-5.6 Terra , Claude Sonnet 5 , Gemini 3.5 Flash , Grok 4.5 , Qwen3.7-Max , GLM-5.2 , and DeepSeek V4 Pro . I generated 3 samples per prompt, at temperature 1.0, requesting the same reasoning effort from every model. That resulted in 1,008 SVGs.
Then I ran each image through a three-stage pipeline:
My hypothesis is that if a lab trained on the benchmark, it should show up in some combination of the pelican row scoring above what the animal deserves, the bicycle column scoring above what the vehicle deserves, or the specific pelican-bicycle cell beating both.
Before any scoring, the simplest test is to look at the images yourself. Pick a lab to see everything it drew, with the judge’s score under each image (click to open full size):
I looked through the images myself before running the analysis below. Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid. Maybe in GLM-5.2’s first sample it felt slightly better than the rest, but that batch also produced a pretty cool heron on a skateboard, so I cannot say for sure. Otherwise they look like the rest of what each model draws, and the labs that draw good pelicans on bicycles also do a good job drawing other animal-vehicle combinations.
But this test is hard to replicate, and everyone will have a different opinion. So I wanted something more quantitative, which is why I opted for the method detailed above.
Here’s the mean animal rating per animal, pooled across all models:
The pelican is 6th of 8, behind cat, whale, raccoon, heron, and antelope. If AI labs were training on the benchmark, you’d expect pelicans at the top. Instead they’re in the bottom half. All seven labs draw cats, whales, and raccoons better than pelicans.
Of course, a pelican may simply be harder to draw than a cat. A lab could train on pelicans and still not push them past the easy animals, so this ranking alone can’t rule that out. I’ll adjust for difficulty in Evidence #4.
Bicycles fare even worse. They sit second from last, in a near-tie with planes, which come in last:
If labs were training on the benchmark, you’d expect bicycles near the top of this ranking. They’re not. However, the same caveat applies here. A bicycle is harder to draw than a skateboard: it needs two matching wheels, a frame that reaches both axles, handlebars, a seat, and pedals. The judge flags a missing or disconnected one of those on 2/3 of the bicycle images. You can train on bicycle images and still not do a great job relative to simpler vehicles.
One note on the plane, though: I should’ve picked “airplane” instead of “plane” because models often read it geometrically. They drew the animal standing on a flat surface instead of flying an aircraft. The plane is the only vehicle where the feature extractor sometimes found no vehicle at all (25 of 168 images, against zero for the other five), and 20% of plane images scored a 1 or 2 on the vehicle rating, against 5% for bicycles and none at all for boats, scooters, or skateboards.
Put the two together and the “pelican on a bicycle” ends up near the bottom of the ranking, at #42 of 48:
But again, some combinations might be just harder to draw than others.
To account for that, I fit a fixed-effects regression on all 1,008 images: score ~ lab + animal × vehicle, plus per-lab interaction terms for pelican, bicycle, and the pelican-bicycle cell, with robust standard errors. The animal × vehicle terms absorb the inherent difficulty of all 48 combinations. The interactions measure each lab’s benchmark-specific boost relative to the average lab, with confidence intervals.
Here are the full per-lab estimates. A pelicanmaxxing lab would show dots to the right of the zero line across its whole row:
Every pelican interval and every cell interval contains zero. Exactly one doesn’t: Gemini 3.5 Flash in the bicycle column. But with 21 tests at p < 0.05, chance alone predicts about one false positive (21 × 0.05 ≈ 1.05), and one is exactly what came up. It also doesn’t survive a multiple-comparisons correction: the Bonferroni threshold across the 21 tests is 0.05/21 ≈ 0.002, and its p-value is 0.022. The full table of estimates and p-values is in the repo：https://github.com/dylanjcastillo/blog/tree/main/_extras/pelicanmaxxing.
But these intervals are wide, about ±0.6 judge points on average. Any boost smaller than that won’t be captured by this test.
Some have suggested that the pelican on a bicycle looks like a memorized composition, pointing to recurring patterns such as the pelican always facing right, or recurring elements like a sun or a scarf. So I wanted to know if this was true.
Direction : All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that.
However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest:
Pelicans are also among the animals that tend to face right:
It’s hard to draw a pelican or a bicycle facing the viewer, so models almost always draw them from the side, facing left or right. That’s why so few of their images are ambiguous. Other combinations also come close to unanimous: antelope on a scooter and pelican on a scooter land at 20 of 21, and heron on a bicycle at 19 of 21. So 21 out of 21 doesn’t seem like an outlier.
Scene elements : I let the extractor name any element it saw in the image. These are the counts:
A memorized scene would show up as the same set of elements recurring picture after picture. I went looking for that, and found some combinations do tend to produce the same elements every time. Every single flamingo on a boat has a sun in it. Otters on planes wear scarves 38% of the time. Cats on bicycles get a basket 38% of the time.
The pelican on a bicycle doesn’t seem to have anything particularly different about it. It just has some elements that appear more frequently, like every other animal-vehicle combination.
Sorry, HN haters, but there’s little evidence that AI labs are pelicanmaxxing. Or at least they’re not doing it in a plainly obvious manner.
Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict. GLM-5.2 comes closest: it has the largest boost on the exact pelican-bicycle cell, and and its first pelican-on-bicycle sample caught my eye. But the effect is small and not significant, so I wouldn’t put too much weight on it.
The other thing that stands out is direction in the scene composition. All 21 pelican-bicycle images face right, the only combination in the grid where every image agrees. But it doesn’t seem that strange. Facing right is the norm across the experiment. Three other combinations land at 90% or above, and with 48 of them, I’m not surprised one reached 21 out of 21.
The more plausible story is SVGmaxxing like Google/DeepMind does. Other labs might be doing it more quietly. Sadly, this experiment can’t say who’s doing it. But at least you can sleep tonight knowing that AI labs are not producing terabytes of pelicans on bicycles just to trick Simon Willison.
If you want to look at the data yourself, the full pipeline is in the repo：https://github.com/dylanjcastillo/blog/tree/main/_extras/pelicanmaxxing.
the practice of optimizing AI models to achieve high scores on popular benchmarks.↩︎：#fnref1
扫码直达文章详情