测量语音识别中的基准优化：Hugging Face 新测试揭示 ASR 模型"刷分"现象

参考分歧（VoxPopuli 案例研究）：#reference-disagreement-voxpopuli-case-study 掩码实体检索：#masked-entity-retrieval 文字形式切换：#orthographic-switching 定位切换：#localizing-the-switches 结论：#conclusion 公共语音 AI 基准测试越来越多地表明，模型的表现达到了人类水平。然而，这些分数并不总能反映模型在现实世界中的工作情况。由于公共基准是开放的并被广泛使用，模型也可能针对测试本身进行优化。它们的分数可能会提高，因为它们学会了基准特定的模式，而不是因为它们在基础任务上变得更好。

其中一个原因是，传统基准忽略了许多使语音系统在实践中可靠、自然、符合上下文并有效的条件和特性。这就是为什么我们最近在 Real World VoiceEQ：https://huggingface.co/spaces/HumeAI/rw-voice-eq、Open-ASR 排行榜：https://huggingface.co/blog/open-asr-leaderboard-private-data 和远场 ASR 排行榜：https://huggingface.co/spaces/treble-technologies/ffasr 中引入了保留集：以衡量在现实世界使用中更重要的方面。

然而，仅仅更广泛的测量并不能解决问题。这个现象，有时称为基准优化或“benchmaxxing”，在机器学习中经常被讨论，但在语音识别中很难测量。

我们最新的研究引入了三项测试来帮助量化这个问题。我们评估了 11 个广泛使用的开源 ASR 模型，发现几种得分最高的系统在 VoxPopuli：https://huggingface.co/datasets/facebook/voxpopuli 英文和 LibriSpeech：https://huggingface.co/datasets/openslr/librispeech_asr （clean、other）数据集上再现了基准转录文本——即使音频与之相矛盾、相关词被静音，或者音频同样支持两种不同的书面形式。

在某些情况下，模型似乎不仅依赖于所说内容，还依赖于微妙的声学提示，这些提示表明它们正在测试的基准。因此，它们的分数夸大了它们一般转录语音的能力。

VoxPopuli 众所周知包含大量转录错误（这也是 Artificial Analysis 发布了清理版本的原因：https://huggingface.co/datasets/ArtificialAnalysis/VoxPopuli-Cleaned-AA）。我们的共识分歧探测测试用于检验在顶级 ASR 模型遇到这些错误时会发生什么：它们是准确转录音频所说内容，还是重现基准数据中错误的参考转录文本？

为了在大规模上测试这一点，我们使用了一个独立模型的集成，这些模型是根据低音素错误率（PER）选出的。PER 测量书面转录与音频中声音的匹配程度，是评估模型如何忠实转录所听内容的有用指标。集成结果可用于标记那些模型一致不同意基准参考转录的情况。然后我们将这些标记案例的样本与人工标注进行对比，以验证已纠正的转录文本。

例如，一个 VoxPopuli 剪辑中可听到短语“Thank you, Mr. President”，但参考转录省略了“Thank you”。我们测试的 11 个模型中有 6 个重现了基准的错误转录——给出了“预期”的答案，即使这与音频内容相矛盾。在实际剪辑中，格式也遵循相同模式：省略“Thank you”的模型也沿用了基准的标点风格，将“Mr”写作无句号，而包含该可听短语的模型倾向于写作带句号的“Mr.”。

当我们在新收集的欧盟议会录音或通用声音中呈现相同内容时，这种行为往往减弱或消失。在下面的样本中，除了一个模型之外，其余模型都回到了为新议会录音克隆忠实转录音频的方式。这表明，模型正在响应帮助它们识别基准所属信息的声学线索，从而即使与音频内容相矛盾也生成预期的转录文本。

该片段的参考文本记录为“总统先生，我对这一程序有另一个抱怨，那就是它不保密。”下面三个片段中的音频实际上说的都是同样的话，并在前面有一个可听到的“谢谢”，——这些克隆是该真实句子的文本转语音版本，所以在三个片段中都可以听到礼貌用语。以绿色高亮和✅标记的文本包含可听到的“谢谢”；以红色高亮和❌标记的文本则再现了基准文本中错误的遗漏。所有文本均为模型原始输出，尚未进行任何标准化——大小写和标点完全按生成时的原样保留，包括某些模型输出的小写字母。

每个模型训练截止后录制的议会发言人克隆

鹦鹉模型是唯一在真实片段上复现基准文本与在同一发言者克隆上正确复现之间切换的模型。Phi-4 是唯一在 ep-fresh 克隆上仍然省略礼貌用语的模型。当我们改用与任何议会录音无关的通用 TTS 语音重新合成该句时，所有十一种模型都恢复了礼貌用语。

结果表明，这一问题既广泛又具有重要意义。我们的方法在分析的 VoxPopuli 测试片段中，发现了 40% 存在潜在参考错误，影响了大约 3% 的所有参考词汇。

显示基准优化行为的模型有 18–30% 的时间复现了错误的参考文本。下方散点图将 x 轴上的 VoxPopuli 单词错误率（WER）与每个模型复现基准错误参考而非共识修正的频率进行比较。WER 最低、因此报告的基准表现最强的模型，也是最可能复现这些错误的模型。

为了进一步探查共识分歧，我们故意在测试数据集的音频样本中将数字静音，并要求模型转录所听内容。音频中确实没有数字，因此模型不应输出任何数字，更不能输出文本中的确切数字。

这些数字中有些是半可预测的（尽管模型仍然不太可能预测到），而其他一些则相当令人惊讶。以下片段结合了两种探测，展示了模型如何重现参考转录错误，包括一个错误的数字，甚至有一个模型在数字被静音的情况下自动补全了一个相对随机的年份（2011）。在下面每个模型的行中：

在公共基准测试上恢复率最高，而在保留或新收集的音频（下文的 ep-fresh 和 libri-fresh）上较低。在 LibriSpeech 上，一些基准表现最佳的模型在大约 30–40% 的示例中再现了被掩盖的数字，即使该数字本身已被删除。在若干模型的新收集数据上的效果减弱，这表明不仅仅是文本自动补全，而是与基准相关的周围音频帮助模型恢复参考内容。

我们的正字法切换探测测试模型是否会重现基准参考转录中使用的确切拼写，即使音频中不清楚。正字法变体是指语义和发音相同但拼写不同的词（1 与 one，Mr. 与 mister，John 与 Jon，Honor 与 Honour 等）。理论上，模型应始终偏好一种拼写，或以大致随机的比例交替使用。如果模型系统性地切换以匹配每个基准的参考转录，这表明模型正在学习测试所期望的拼写。

抄写："我督促男孩们，无论发生什么，我们都不应该开枪射击任何人"——使用“any one”的模型：6/11，使用“anyone”的模型：5/11

抄录：“伪装”不是船长或他那个时代的任何人所理解的词 — 使用“any one”的模型：2/11，使用“anyone”的模型：9/11

在 LibriSpeech 中，我们测试一个涉及旧间距约定的同数据集内切换：一些参考转录使用 "any one"，而其他使用 "anyone"。我们测量某个变体的最小准确率，称为“切换率”。如果模型只使用一种变体，则切换率为 0%；随机选择的模型预期切换率为 50%。如果模型在每个测试样本中都知道要使用哪个变体，则切换率为 100%。

我们的第二次探测测试了跨数据集切换，其中每个基准在其测试语料库中始终使用不同的拼写规范。例如，VoxPopuli 使用缩写“Mr.”，而 LibriSpeech 则拼写为“Mister”。

多种模型的表现都超过了 50% 的随机选择基线，其中一些模型的切换准确率约为 90%。这表明模型可以识别音频样本来自哪个数据集，并选择基准所期望的拼写规范，尽管两种形式在发音上完全相同。

为了测试这些行为是否会超越公开基准，我们还从相同来源域收集了模型训练截止日期之后的新数据：最近的欧洲议会录音用于 VoxPopuli，以及新活跃的 LibriVox 朗读者录音用于 LibriSpeech。然而，当呈现来自相同领域的最近收集的数据时，许多模型会停止匹配参考转录，并恢复更忠实于音频的转录。

其他干预措施也指向相同的结论。存在于音频中但在参考转录中被省略的短语，当模型被要求翻译音频或其注意力被限制在相关帧时，可能会重新出现。修剪周围的基准上下文，或附加普通对话音频，也可以恢复忠实的转录。附加 VoxPopuli 音频可能产生相反效果，使原本忠实的合成或挖掘样本更可能匹配基准参考。

综上，这些结果表明模型能够忠实地转录字面上的口语，但会利用周围的声学上下文来决定是遵循音频还是特定基准的转录策略。

我们的研究结果表明，在两个主要开源数据集上，一些模型能够检测与数据集相关的声学信号，并相应地调整其转录行为。具体来说，模型可能会复现音频中不存在但参考转录中出现的词语，提高恢复被静音的数字的比率，或利用周围的声学上下文选择某一特定基准所期望的书写形式。

对于选择模型的人来说，这些发现强调了使用完全独立的评估集的重要性，就像 RW-Voice-EQ Bench 和 Open ASR Leaderboard 那样，并且需要超越单一公开基准的词错误率。因此，Open ASR Leaderboard 中新增了一个“基准拟合”标签：https://huggingface.co/spaces/hf-audio/open_asr_leaderboard，其中包括对所有模型的两项上述分析：量化 (1) VoxPopuli 的参考错误率，以及 (2) 所有公开数据集的正字法切换。相关脚本已在 GitHub 上开源：https://github.com/huggingface/open_asr_leaderboard/tree/main/benchmark_fitting，以及未归一化的模型输出：https://huggingface.co/buckets/hf-audio/asr_leaderboard_h200。

我们的研究结果也表明，基准开发者应避免简单的独立同分布测试集划分，而应选择基于时间、说话人或其他元数据的分离。对训练数据和模型选择过程的更大透明度也将有助于研究人员理解这些行为是如何产生的。

公开基准仍然有价值：它们透明、可重复、易于运行，并且研究社区对其有很好的理解。但它们最有用的情况是，当我们能够区分真正的转录改进和仅在基准上表现良好的无法推广到新音频的成果时。

更多信息，请查阅我们的完整报告：https://huggingface.co/papers/2608.19936。

探索并比较带有音频样本的语音模型排行榜

跨数据集探索语音识别模型基准

远场语音识别 — 干净 / 嘈杂 / 混响 基准

Reference disagreement (VoxPopuli case study) ：#reference-disagreement-voxpopuli-case-study Masked Entity Retrieval ：#masked-entity-retrieval Orthographic Switching ：#orthographic-switching Localizing the switches ：#localizing-the-switches Conclusion ：#conclusion Public voice AI benchmarks increasingly suggest that models are performing at human levels. Yet those scores don't always reflect how models work in the real-world. Since public benchmarks are open and widely used, models can also become optimized for the tests themselves. Their scores may improve because they have learned benchmark-specific patterns and not because they have become better at the underlying task.

One reason is that traditional benchmarks overlook many of the conditions and qualities that make voice systems reliable, natural, contextually appropriate, and effective in practice. That's why we recently introduced held-out sets in Real World VoiceEQ：https://huggingface.co/spaces/HumeAI/rw-voice-eq, the Open-ASR Leaderboard：https://huggingface.co/blog/open-asr-leaderboard-private-data, and the Far-field ASR Leaderboard：https://huggingface.co/spaces/treble-technologies/ffasr: to measure more of what matters in real-world use.

However, broader measurement alone does not solve the problem. This phenomenon, sometimes called benchmark optimization or "benchmaxxing," is often discussed around machine learning, however, it has been difficult to measure in speech recognition.

Our latest research introduces three tests to help quantify it. We evaluated 11 widely used open-source ASR models and found that several of the highest-scoring systems reproduced benchmark transcripts from the VoxPopuli：https://huggingface.co/datasets/facebook/voxpopuli English and LibriSpeech：https://huggingface.co/datasets/openslr/librispeech_asr (clean, other) datasets – even when the audio contradicted them, relevant words had been silenced, or the audio equally supported two different written forms.

In some cases, models appeared to rely not only on what was said, but also on subtle acoustic cues that indicated which benchmark they were being tested on. As a result, their scores overstated how well they could transcribe speech more generally.

VoxPopuli is known to contain a high number of transcription errors (which is why Artificial Analysis released a cleaned version：https://huggingface.co/datasets/ArtificialAnalysis/VoxPopuli-Cleaned-AA). Our consensus disagreement probe tests what happens when leading ASR models encounter these errors: Do they accurately transcribe what the audio says, or reproduce the benchmark's incorrect reference transcript?

To test this at scale, we use an ensemble of independent models selected for their low phoneme error rate (PER). PER measures how closely a written transcription matches the sounds in the audio, making it a useful proxy for how faithfully a model transcribes what it hears. The ensemble results can be used to flag cases in which the models unanimously disagree with the benchmark's reference transcript. We then compare a sample of those flagged cases against human annotations to validate the corrected transcripts.

For example, one VoxPopuli clip audibly includes the phrase "Thank you, Mr. President," but the reference transcript omits "Thank you." Six of the 11 models we tested reproduced the benchmark's erroneous transcript—giving the "expected" answer even though it contradicted the audio. On the real clip, the formatting follows the same pattern: models that omit "Thank you" also reproduce the benchmark's punctuation style, writing "Mr" without a period, while models that include the audible phrase tend to write "Mr." with the period.

When we present the same content in newly collected voices from EU parliamentary recordings or generic voices, this behavior often weakens or disappears. In the below samples, all but one model flips back to transcribing the audio-faithful transcript for a clone of a new parliamentary recording. This suggests that the models are responding to acoustic cues that help them identify the benchmark membership and thus produce the expected transcript even if it contradicts the audio.

The reference transcript for this clip reads "Mr President, I have another complaint about this procedure, which is that it is not secret." The audio in all three clips below actually says the same thing, preceded by an audible "Thank you,"—the clones are text-to-speech renditions of that true sentence, so the courtesy is audible in all three. Green highlighting and ✅ mark a transcript that includes the audible "Thank you"; red highlighting and ❌ mark a transcript that reproduces the benchmark's erroneous omission. All transcripts are raw model output, prior to any normalization—casing and punctuation are preserved exactly as generated, including lowercase output from some models.

Clone of a parliament speaker recorded after every model's training cutoff

Parakeet is the only model that flips between reproducing the benchmark on the real clip and getting it right on the same-speaker clone. Phi-4 is the only model still dropping the courtesy on the ep-fresh clone. When we instead resynthesize the sentence in a generic TTS voice unconnected to any parliamentary recording, all eleven models restore the courtesy.

The results suggest that this problem is both widespread and meaningful. Our methodology flagged potential reference errors in 40% of the VoxPopuli test clips we analyzed, affecting roughly 3% of all reference words.

Models exhibiting benchmark-optimized behavior reproduced erroneous reference transcripts 18–30% of the time. The scatterplot below compares VoxPopuli word error rate (WER) on the x-axis with the rate at which each model reproduces the benchmark's incorrect reference instead of the consensus correction. The models with the lowest WER—and therefore the strongest reported benchmark performance—are also the most likely to reproduce these errors.

To build on the consensus disagreement probe, we deliberately silence numbers in the audio samples of test datasets and ask the models to transcribe what it hears. The number is literally absent from the audio, so models should not output any number, much less the exact number in the text.

Some of these numbers are semi-predictable (although still unlikely for a model to predict), yet others are quite surprising. The following clip combines both probes, showing both how models recreate reference transcript errors including an incorrect number and one model even autocompletes a relatively random year (2011) despite it being silenced. In each model's row below:

Recovery rates were highest on the public benchmarks and lower on held-out or newly collected audio (ep-fresh and libri-fresh below). On LibriSpeech, some of the strongest benchmark-performing models reproduced masked numbers in roughly 30–40% of examples, even though the number itself had been removed. The effect weakened on freshly collected data for several models, suggesting that the surrounding benchmark-associated audio—not only textual autocomplete—helped the models recover the reference.

Our orthographic switching probe tests whether models reproduce the exact spelling used in a benchmark's reference transcript despite it not being clear in the audio. Orthographic variants are words that are semantically and phonetically identical but can be spelled different ways (1 vs one, Mr. vs mister, John vs Jon, Honor vs Honour, etc). In theory, models should consistently prefer one spelling over another, or alternate between them at roughly random rates. If models systematically switch to match what is in each benchmark's reference transcript, that suggests the models are picking up on which spelling the test expects.

Transcription: "I URGED ON THE BOYS THAT WHATEVER HAPPENED WE SHOULD NOT SHOOT ANY ONE" — models using "any one": 6/11, models using "anyone": 5/11

Transcription: "CAMOUFLAGE WAS NOT A WORD THE CAPTAIN OR ANYONE ELSE OF HIS TIME YET UNDERSTOOD" — models using "any one": 2/11, models using "anyone": 9/11

Within LibriSpeech, we test one intra-dataset switch involving an older spacing convention: some reference transcripts use "any one", while others use "anyone." We measure the minimum accuracy for a given variant, which we call "switch rate". If a model only uses one variant it would have a 0% switch rate; a model which picks randomly would be expected to have a 50% switch rate. A model which knows which variant to use in every test sample would earn a 100% switch rate.

Our second probe tests an inter-dataset switch , in which each benchmark uses a different spelling convention consistently across its test corpus. For example, VoxPopuli uses the abbreviation "Mr.," while LibriSpeech spells out "Mister."

Multiple models exceed the 50% random-choice baseline, with some reaching roughly 90% switch accuracy. This suggests that the models can identify which dataset an audio sample comes from and select the spelling convention that benchmark expects, even though both forms sound identical.

To test whether these behaviors generalize beyond the public benchmarks, we also collected fresh data from the same source domains but after the models' training cutoffs: recent European Parliament recordings for VoxPopuli and recordings from newly active LibriVox narrators for LibriSpeech. However, when presented with recently collected data from the same domain, many models stop matching the reference transcript and revert to more audio faithful transcriptions.

Other interventions point to the same conclusion. Phrases which are present in the audio but are omitted in the reference transcript can reappear when a model is asked to translate the audio or when its attention is restricted to the relevant frames. Trimming away surrounding benchmark context, or appending ordinary conversational audio, can also restore the faithful transcript. Appending VoxPopuli audio can have the opposite effect, making otherwise faithful synthetic or mined samples more likely to match the benchmark reference.

Together, these results suggest that models are able to faithfully transcribe the literal spoken words, but are using surrounding acoustic context to decide whether to follow the audio or a benchmark-specific transcription policy.

Our findings suggest that, on two major open-source datasets, some models detect dataset-associated acoustic cues and adjust their transcription behavior accordingly. Specifically, models may reproduce words that are absent from the audio but present in the reference transcript, recover silenced numbers at elevated rates, or use surrounding acoustic context to select the written variant expected by a particular benchmark.

For people selecting models, these findings underscore the importance of using fully held-out evaluation sets, as RW-Voice-EQ Bench and the Open ASR Leaderboard do, and of looking beyond word error rate on a single public benchmark. To this end, a "Benchmark fitting" tab has been added to the Open ASR Leaderboard：https://huggingface.co/spaces/hf-audio/open_asr_leaderboard, which includes two of the above analyses across all models: quantifying (1) reference error rates from VoxPopuli and (2) orthographic switching across all public datasets. The relevant scripts are open-sourced on GitHub：https://github.com/huggingface/open_asr_leaderboard/tree/main/benchmark_fitting as well as the un-normalized model outputs：https://huggingface.co/buckets/hf-audio/asr_leaderboard_h200.

Our findings also suggest that benchmark developers should avoid simple independent and identically distributed test splits in favor of temporal, speaker, or other metadata-based separation. Greater transparency around training data and model-selection procedures would also help researchers understand how these behaviors arise.

Public benchmarks remain valuable: they are transparent, repeatable, easy to run, and well understood by the research community. But they are most useful when we can distinguish genuine transcription improvements from benchmark-specific gains that do not generalize to new audio.

For more information, we encourage you to read our full report：https://huggingface.co/papers/2608.19936.

Explore and compare speech model leaderboards with audio samples

Explore speech recognition model benchmarks across datasets

Far-Field ASR — clean / noisy / reverberant benchmark