Evaluating Atomic Visual Perception in Multimodal Large Language Models
We are releasing PerceptionBench , a benchmark that isolates visual perception and evaluates it as a set of atomic capabilities— discovered from how today's models fail, not defined in advance. By attributing frontier-model failures across 40+ benchmarks to their earliest visual cause, we distill 10 perceptual capabilities and 3,000 verified questions, each answerable by looking, with no reasoning or outside knowledge required.
The result is a sharp diagnosis rather than one more score. No model we evaluate clears 60% accuracy, and models with nearly identical overall scores can exhibit very different perceptual strengths and weaknesses. More strikingly, a large share of correct answers fail to survive a repeated ask—evidence that current models often guess rather than perceive. PerceptionBench is built to expose exactly where perception breaks, and to drive progress toward multimodal AI that sees faithfully and consistently.
Each source benchmark captures a narrow slice of perception errors, and these slices overlap only weakly (mean pairwise weighted Jaccard 0.20). No single benchmark—or small group of them—covers perception as a whole, which motivates a capability-centric benchmark that aggregates and rebalances these fragmented views.
The dataset consists of 3,000 high-quality verified samples. The distribution aims to isolate atomic perceptual capabilities from confounding factors, and distinguishes itself through three core design principles:
Note on Quality : To make the benchmark a reliable gold standard, all samples underwent rigorous verification and difficulty-balancing, keeping only genuinely perceptual failures with a single verifiable answer.
Which part of the mug does the purple line connect to?
How many different colors of hats appear?
How many red dots are there in the picture?
How many equal subintervals is the u₁ axis divided into?
How many staff lines does the rightmost note in the figure touch?
How many faces of these small cubes are in direct contact with the ground?
What is the digit in the lower-right box?
How many animal silhouettes in the picture are exactly the same?
How many black queens are on the chessboard?
How many monkeys have touched the wheel cover?
Cell x lies in the intersection of how many circles?
How many yellow hollow rings appear in the figure?
How many people are inside the truck?
PerceptionBench is a simple but challenging benchmark for evaluating atomic visual perception in frontier models. It measures what multimodal models actually see rather than what they infer, providing a faithful and fine-grained diagnosis of the perceptual capabilities of current and future multimodal models.
Scan to open this article