Moonshot AI 发布 PerceptionBench：多模态模型视觉感知能力诊断基准

在多模态大语言模型中评估原子视觉感知能力

我们发布了 PerceptionBench，这是一个将视觉感知孤立出来并以一组原子能力进行评估的基准——这些能力是从当今模型的失败中发现的，而非事先定义的。通过将前沿模型在40个基准上的失败归因于其最早的视觉原因，我们提炼出了10种感知能力和3,000个经过验证的问题，每个问题都可以通过观察回答，无需推理或外部知识。

其结果是精确的诊断，而不是又一个分数。我们评估的没有一个模型能达到60%的准确率，且整体分数几乎相同的模型可能表现出截然不同的感知优劣。更令人惊讶的是，大量正确答案在重复提问时未能维持——这表明当前模型常常是猜测而非真正感知。PerceptionBench旨在暴露感知断裂的具体位置，并推动多模态AI朝着忠实且稳定的视觉理解迈进。

每个原始基准都捕捉了感知错误的一个狭窄切片，而这些切片之间的重叠很弱（平均成对加权Jaccard指数0.20）。没有单一基准或少数几个基准能覆盖感知的整体，这促使我们提出以能力为中心的基准，将这些零散视角进行汇总和平衡。

该数据集包含3,000个高质量的经过验证的样本。其分布旨在从干扰因素中孤立出原子感知能力，并且通过三个核心设计原则自成特色：

质量说明：为了使基准成为可靠的黄金标准，所有样本都经过严格验证和难度平衡，仅保留真正的感知失败样本，并保证每个问题有唯一可验证的答案。

紫色的线连接到杯子的哪一部分？

出现了多少种不同颜色的帽子？

图片中有多少个红点？

u₁轴被分成多少个相等的子区间？

图中最右侧的音符接触了多少条五线谱线？

这些小立方体有多少个面直接接触地面？

右下角的方框内的数字是多少？

图片中有多少个动物剪影完全相同？

棋盘上有多少个黑色皇后？

有多少只猴子碰过方向盘罩？

单元格 x 位于多少个圆的交集中？

图中出现了多少个黄色空心环？

卡车里有多少人？

PerceptionBench 是一个简单但具有挑战性的基准测试，用于评估前沿模型的基本视觉感知能力。它衡量的是多模态模型实际看到的内容，而不是它们推断的内容，从而为当前和未来的多模态模型的感知能力提供了真实而细致的诊断。

Evaluating Atomic Visual Perception in Multimodal Large Language Models

We are releasing PerceptionBench , a benchmark that isolates visual perception and evaluates it as a set of atomic capabilities— discovered from how today's models fail, not defined in advance. By attributing frontier-model failures across 40+ benchmarks to their earliest visual cause, we distill 10 perceptual capabilities and 3,000 verified questions, each answerable by looking, with no reasoning or outside knowledge required.

The result is a sharp diagnosis rather than one more score. No model we evaluate clears 60% accuracy, and models with nearly identical overall scores can exhibit very different perceptual strengths and weaknesses. More strikingly, a large share of correct answers fail to survive a repeated ask—evidence that current models often guess rather than perceive. PerceptionBench is built to expose exactly where perception breaks, and to drive progress toward multimodal AI that sees faithfully and consistently.

Each source benchmark captures a narrow slice of perception errors, and these slices overlap only weakly (mean pairwise weighted Jaccard 0.20). No single benchmark—or small group of them—covers perception as a whole, which motivates a capability-centric benchmark that aggregates and rebalances these fragmented views.

The dataset consists of 3,000 high-quality verified samples. The distribution aims to isolate atomic perceptual capabilities from confounding factors, and distinguishes itself through three core design principles:

Note on Quality : To make the benchmark a reliable gold standard, all samples underwent rigorous verification and difficulty-balancing, keeping only genuinely perceptual failures with a single verifiable answer.

Which part of the mug does the purple line connect to?

How many different colors of hats appear?

How many red dots are there in the picture?

How many equal subintervals is the u₁ axis divided into?

How many staff lines does the rightmost note in the figure touch?

How many faces of these small cubes are in direct contact with the ground?

What is the digit in the lower-right box?

How many animal silhouettes in the picture are exactly the same?

How many black queens are on the chessboard?

How many monkeys have touched the wheel cover?

Cell x lies in the intersection of how many circles?

How many yellow hollow rings appear in the figure?

How many people are inside the truck?

PerceptionBench is a simple but challenging benchmark for evaluating atomic visual perception in frontier models. It measures what multimodal models actually see rather than what they infer, providing a faithful and fine-grained diagnosis of the perceptual capabilities of current and future multimodal models.