{"@context":"https://schema.org","@type":"NewsArticle","generatedAt":"2026-09-20T20:00:42.136Z","headline":"Hugging Face 与 Ai2 发布 BenchMIRT：从题目层面审计 LLM 基准究竟测了什么","description":"Hugging Face 博客介绍 BenchMIRT，一种基于多维 IRT 在单个题目层面审计 LLM 基准的方法，训练数据覆盖 100 个 LLM、16 个基准和超过 34K 道题。","url":"https://www.aioga.com/news/cmtj8p4l509paroh9e7kjhej4/","mainEntityOfPage":"https://www.aioga.com/news/cmtj8p4l509paroh9e7kjhej4/","datePublished":"2026-09-01T21:39:07.000Z","dateModified":"2026-09-01T21:39:07.000Z","inLanguage":"zh-CN","publisher":{"@type":"NewsMediaOrganization","name":"Aioga","url":"https://www.aioga.com"},"citation":["https://huggingface.co/blog/allenai/benchmirt","https://aihot.virxact.com/items/cmtj8p4l509paroh9e7kjhej4"],"canonicalUrl":"https://www.aioga.com/news/cmtj8p4l509paroh9e7kjhej4/","directAnswer":{"@type":"Answer","text":"Hugging Face 博客介绍了 BenchMIRT，一种在单个题目或任务层面审计大语言模型基准的方法。其训练数据覆盖 100 个 LLM、16 个基准和超过 3.4 万道题。","url":"https://www.aioga.com/news/cmtj8p4l509paroh9e7kjhej4/","dateCreated":"2026-09-01T21:39:07.000Z","author":{"@type":"Organization","@id":"https://www.aioga.com/authors/aioga-editorial/#editorial-team","name":"Aioga Editorial Team","url":"https://www.aioga.com/authors/aioga-editorial/"}},"evidence":[{"@type":"CreativeWork","name":"huggingface.co source article","url":"https://huggingface.co/blog/allenai/benchmirt","datePublished":"2026-09-01T21:39:07.000Z","provider":{"@type":"Organization","name":"huggingface.co","url":"https://huggingface.co/blog/allenai/benchmirt"}},{"@type":"CreativeWork","name":"AIHot archive record","url":"https://aihot.virxact.com/items/cmtj8p4l509paroh9e7kjhej4","datePublished":"2026-09-01T21:39:07.000Z","provider":{"@type":"Organization","name":"AIHot","url":"https://aihot.virxact.com/items/cmtj8p4l509paroh9e7kjhej4"}}],"aggregationSource":"Hugging Face：Blog（RSS）","originalPublisher":{"name":"huggingface.co","url":"https://huggingface.co/blog/allenai/benchmirt"},"geoDeepAnswer":null,"article":{"id":"cmtj8p4l509paroh9e7kjhej4","slug":"cmtj8p4l509paroh9e7kjhej4","url":"https://www.aioga.com/news/cmtj8p4l509paroh9e7kjhej4/","title":"Hugging Face 与 Ai2 发布 BenchMIRT：从题目层面审计 LLM 基准究竟测了什么","title_en":"","summary":"Hugging Face 博客介绍 BenchMIRT，一种基于多维 IRT 在单个题目层面审计 LLM 基准的方法，训练数据覆盖 100 个 LLM、16 个基准和超过 34K 道题。","source":"Hugging Face：Blog（RSS）","sourceUrl":"https://huggingface.co/blog/allenai/benchmirt","aiHotUrl":"https://aihot.virxact.com/items/cmtj8p4l509paroh9e7kjhej4","publishedAt":"2026-09-01T21:39:07.000Z","category":"行业动态","score":58,"selected":false,"articleBody":["Finding the signals inside a benchmark ：#finding-the-signals-inside-a-benchmark What BenchMIRT reveals about existing benchmarks ：#what-benchmirt-reveals-about-existing-benchmarks Doing more with fewer questions ：#doing-more-with-fewer-questions What this could mean for LLM evaluation ：#what-this-could-mean-for-llm-evaluation 📄 Tech Report: http://allenai.org/papers/benchmirt：http://allenai.org/papers/benchmirt | 📊 Data: https://huggingface.co/collections/allenai/benchmirt：https://huggingface.co/collections/allenai/benchmirt | 💻 Code: https://github.com/allenai/BenchMIRT：https://github.com/allenai/BenchMIRT","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/jlxf_E3loRcUO5Pu936zP.png","Today we’re introducing BenchMIRT, a new method for auditing LLM benchmarks at the level of individual prompts—the questions and tasks a model is scored on.","A benchmark is usually designed to measure a particular ability, such as safety, general reasoning, or instruction following. But the individual tasks inside it may depend on more than that stated goal. Take BBQ, a benchmark designed to test whether models rely on social stereotypes. One question asks about a grandson and grandfather trying to book an Uber. It probes age bias, but also requires the model to track who’s who and reason from the evidence provided rather than assumptions.","And even within a single benchmark, different groups of questions and tasks can measure different things. WildJailbreak, for example, includes harmful jailbreak prompts alongside benign prompts designed to test whether a model refuses harmless requests too often. The harmful prompts are more closely associated with safety, while the benign prompts are more closely associated with general reasoning. Averaging them into a single benchmark score can obscure that difference.","BenchMIRT helps researchers separate those signals and see what’s actually driving a benchmark’s score. It does this by analyzing how models perform on each question or task and estimating which underlying capabilities are most closely associated with getting it right.","BenchMIRT takes cues from Item Response Theory (IRT), a technique originating in psychometrics—the field concerned with measuring abilities and traits from patterns of test responses. IRT starts from a simple idea: not every question tells you the same amount about the person taking a test. Some are harder than others, and some do a better job of distinguishing stronger performers from weaker ones.","Researchers have previously applied single-dimensional IRT to individual benchmarks, including in our Fluid Benchmarking work. BenchMIRT extends that approach with multidimensional IRT, or MIRT, allowing it to separate multiple capabilities that may contribute to performance on the same questions.","BenchMIRT applies IRT at both the model and question level. For a given model, it estimates the model’s strength on the capabilities reflected across the selected benchmarks. For each question, it estimates how difficult the question is and how well it distinguishes models that are stronger or weaker on those capabilities.","We trained BenchMIRT on benchmarking results from 100 LLMs across 16 benchmarks and more than 34K questions. Six of those benchmarks measure general reasoning, including MMLU-Pro, GPQA, MATH, and BBH. The other 10 come from our Olmo 3 safety suite, including HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest.","Crucially, we didn’t tell BenchMIRT which benchmarks were measuring which capabilities. It independently recovered two dominant dimensions: safety and general reasoning. When we repeated our analysis from scratch, those same two dimensions emerged each time, suggesting the result was stable rather than specific to one analysis.","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/emqg9ENtMBcpo4S86cxy3.png","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/LMPWH4haeYge_HJInjhai.png","For many benchmarks, BenchMIRT largely confirmed their intended focus: strong performance on reasoning benchmarks tracked with reasoning ability, while strong performance on jailbreak and harmful-content benchmarks tracked with safety.","But BenchMIRT also revealed a more complicated picture in some evaluations.","BBQ, which evaluates social bias and is commonly grouped with safety benchmarks, aligned much more strongly with general reasoning in BenchMIRT’s analysis. That means a low BBQ score may partly reflect difficulty understanding or reasoning through certain questions, rather than safety behavior alone.","WMDP behaves differently from most safety benchmarks. It tests dangerous dual-use knowledge in areas such as biology, chemistry, and cybersecurity—for example, knowledge that could help someone misuse a biological agent or exploit a computer system. BenchMIRT found that WMDP scores were more strongly associated with general reasoning than with safety. Stronger general reasoning, however, was associated with lower WMDP scores, because the benchmark counts refusing or failing to provide the dangerous knowledge as the desired response.","These findings don’t necessarily mean the benchmarks are flawed or incomplete. Rather, they show that a single benchmark score can combine several different signals—and that BenchMIRT can help disentangle those signals and make the score easier to interpret.","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/ue15HltywHtVyTIud5YlY.png","Item difficulty and discrimination in both dimensions for Harmbench. Dimension 0 models the safety dimension, while Dimension 1 maps to the general reasoning dimension.","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/J18F7xrVwgaA68xg92Hs1.png","Bar size and direction show the Pearson correlation, across 100 open-weight LLMs, between BenchMIRT ability scores and benchmark scores on a −1 to 1 scale—pink for general reasoning and teal for safety; bars extending left of center are negative. Bold with underline marks each row’s stronger correlation, except where the two are too close to separate; asterisks mark p","BenchMIRT can also help identify which questions in an evaluation are most informative about the capability the benchmark is trying to measure.","Using BenchMIRT’s question-level estimates, we ranked questions across the same 16 benchmarks used to train BenchMIRT and kept those that did the best job of distinguishing stronger from weaker models, while still preserving a mix of easier and harder questions.","Across those benchmarks, keeping only 10% of the questions generally preserved nearly the same picture of which models were stronger or weaker on the underlying safety or reasoning capability as using the full set. Keeping 50% of the questions often matched the full benchmark’s measure of those capabilities even more closely.","BenchMIRT can also use the patterns it learns across models and questions to predict how a model would perform on a benchmark question it hasn’t been observed answering. In our experiments, it correctly predicted whether a model would answer a held-out question correctly 79% of the time. By comparison, a simpler approach that assumes a model will perform on each question about as well as it does on the benchmark overall was correct 70% of the time.","In practice, that means BenchMIRT can estimate model performance more precisely from what it has already learned about the model’s abilities and the demands of each question, without needing to evaluate every model on every question.","BenchMIRT offers a way to better understand and refine the benchmarks researchers use to evaluate model capabilities. By looking at individual questions rather than only overall scores, it can reveal when a benchmark mixes together different capabilities, identify clusters of questions that behave differently from the rest, and surface questions that add little useful information about the capability the benchmark is meant to measure.","There are important limitations. The models we used to train and evaluate BenchMIRT were all released by March 2025, so our analysis doesn’t capture how BenchMIRT behaves on newer generations of LLMs. And the dimensions BenchMIRT discovers depend on the benchmark set it’s given—safety and reasoning emerged as the dominant dimensions across the 16 benchmarks we selected for this project, but a different mix of evaluations could surface different underlying capabilities.","There are trade-offs, too. If the goal is to rank models by their predicted performance on randomly held-out items, the benchmark’s average score performs slightly better than BenchMIRT. BenchMIRT’s advantage is the finer-grained picture it provides of performance on individual questions.","That question-level detail can also cut both ways: the same estimates that help identify a benchmark’s most informative safety questions could be used to remove them, producing a weaker evaluation that an unsafe model could pass. Existing tools already make it possible to trim evaluations in similar ways, and we think the added transparency into what benchmark questions are actually measuring is worth that risk—but it’s a real one.","Still, we see BenchMIRT – and future tools like it – as a step toward more targeted benchmark design and efficient evaluation. By showing which questions are actually driving a benchmark’s results, these approaches could help researchers build evaluations that are smaller, more focused, and easier to interpret, while giving a clearer picture of the capabilities they’re meant to measure."],"articleImages":[{"sourceUrl":"https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/jlxf_E3loRcUO5Pu936zP.png","alt":"BenchMIRT blog draft latest - Google Docs-image-1 (3)","afterParagraph":0,"url":"/media/articles/cmtj8p4l509paroh9e7kjhej4/8a5275cac394480b.png"},{"sourceUrl":"https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/emqg9ENtMBcpo4S86cxy3.png","alt":"BenchMIRT blog draft latest - Google Docs-image-2 (1)","afterParagraph":10,"url":"/media/articles/cmtj8p4l509paroh9e7kjhej4/265f44046406b444.png"},{"sourceUrl":"https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/LMPWH4haeYge_HJInjhai.png","alt":"BenchMIRT blog draft latest - Google Docs-image-3 (1)","afterParagraph":11,"url":"/media/articles/cmtj8p4l509paroh9e7kjhej4/28850de55dc2ec24.png"}],"mediaStatus":"ok","articleBodyZh":["在基准测试中寻找信号：#finding-the-signals-inside-a-benchmark BenchMIRT揭示了现有基准测试的哪些信息：#what-benchmirt-reveals-about-existing-benchmarks 用更少的问题做更多事情：#doing-more-with-fewer-questions 这对LLM评估可能意味着什么：#what-this-could-mean-for-llm-evaluation 📄 技术报告: http://allenai.org/papers/benchmirt：http://allenai.org/papers/benchmirt | 📊 数据: https://huggingface.co/collections/allenai/benchmirt：https://huggingface.co/collections/allenai/benchmirt | 💻 代码: https://github.com/allenai/BenchMIRT：https://github.com/allenai/BenchMIRT","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/jlxf_E3loRcUO5Pu936zP.png","今天我们介绍BenchMIRT，一种用于在单个提示级别审核LLM基准测试的新方法——也就是模型评分所依据的问题和任务。","基准测试通常是为了测量特定能力而设计的，例如安全性、通用推理或遵循指令。但其中的单个任务可能依赖的不仅仅是所声明的目标。以BBQ为例，这是一个用于测试模型是否依赖社会刻板印象的基准测试。一个问题是关于孙子和祖父试图预订Uber的情景。它考察年龄偏见，但同时还要求模型追踪谁是谁，并根据提供的证据而非假设进行推理。","即使在单一基准测试中，不同的问题和任务组也可能测量不同的内容。例如，WildJailbreak包括有害的越狱提示，以及设计来测试模型是否也拒绝无害请求的良性提示。有害提示与安全性更紧密相关，而良性提示与通用推理更紧密相关。将它们平均到一个基准分数中可能掩盖了这种差异。","BenchMIRT帮助研究人员分离这些信号，看看到底是什么在推动基准测试的分数。它通过分析模型在每个问题或任务上的表现，并估计哪些底层能力与答对该问题最密切相关来实现这一点。","BenchMIRT 从项目反应理论（IRT）中获得启发，IRT 是起源于心理测量学的一种技术——该领域关注通过测试反应模式测量能力和特质。IRT 基于一个简单的想法：不是每个问题都能告诉你关于测试者的相同信息。有些问题比其他问题更难，有些问题则更能区分表现强与表现弱的人。","研究人员此前曾将单维 IRT 应用于各个基准测试，包括我们的 Fluid Benchmarking 工作。BenchMIRT 将该方法扩展为多维 IRT，或称 MIRT，使其能够区分可能影响同一问题表现的多种能力。","BenchMIRT 在模型层面和问题层面都应用 IRT。对于给定模型，它估计模型在所选基准测试中反映的各项能力上的强度。对于每个问题，它估计问题的难度以及问题在区分能力较强或较弱模型方面的效果。","我们在从 100 个大型语言模型获取的 16 个基准测试和超过 34K 个问题的基准结果上训练了 BenchMIRT。其中六个基准测试衡量通用推理能力，包括 MMLU-Pro、GPQA、MATH 和 BBH。其他十个基准测试来自我们的 Olmo 3 安全套件，包括 HarmBench、StrongReject、WildJailbreak、BBQ、WMDP 和 XSTest。","关键是，我们没有告诉 BenchMIRT 哪些基准测试衡量哪些能力。它独立发现了两个主要维度：安全性和通用推理能力。当我们从头重复分析时，每次都会出现这两个相同维度，这表明结果是稳定的，而不是特定于某次分析。","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/emqg9ENtMBcpo4S86cxy3.png","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/LMPWH4haeYge_HJInjhai.png","对于许多基准测试，BenchMIRT 在很大程度上确认了它们的预期焦点：在推理基准测试中表现强劲与推理能力相关，而在越狱和有害内容基准测试中表现强劲与安全性相关。","但 BenchMIRT 也在一些评估中揭示了更加复杂的情况。","BBQ 用于评估社会偏见，通常被归类于安全基准，在 BenchMIRT 的分析中与一般推理的相关性要强得多。这意味着低 BBQ 得分可能部分反映了理解或推理某些问题的困难，而不仅仅是安全行为。","WMDP 的表现与大多数安全基准不同。它测试生物学、化学和网络安全等领域的危险双用知识——例如，可能帮助人滥用生物制剂或利用计算机系统的知识。BenchMIRT 发现 WMDP 得分与一般推理的关联比与安全性的关联更强。然而，更强的一般推理能力与较低的 WMDP 得分相关，因为该基准将拒绝或未能提供危险知识视为期望的回答。","这些发现不一定意味着这些基准存在缺陷或不完整。相反，它们表明，单一基准得分可能结合了几种不同的信号——BenchMIRT 可以帮助解开这些信号，使得分更易于解释。","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/ue15HltywHtVyTIud5YlY.png","Harmbench 的两个维度下的题目难度和辨别力。维度 0 模拟安全维度，维度 1 对应一般推理维度。","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/J18F7xrVwgaA68xg92Hs1.png","条形的大小和方向显示在 100 个开放权重 LLM 中，BenchMIRT 能力得分与基准得分之间的皮尔逊相关系数（范围 −1 到 1）——粉色表示一般推理，青绿色表示安全；条形向左延伸表示负相关。加粗带下划线表示每行更强的相关性，除非两者太接近无法区分；星号表示 p 值。","BenchMIRT 还可以帮助识别评估中哪些问题对基准试图测量的能力最具信息性。","利用 BenchMIRT 的题目级估计，我们对训练 BenchMIRT 所使用的相同 16 个基准的题目进行了排序，并保留那些在区分强模型和弱模型方面表现最好的题目，同时仍保持容易题和难题的混合。","在这些基准测试中，只保留10%的问题通常仍然能够几乎保持完整问题集所显示的模型在基础安全性或推理能力上的强弱情况。保留50%的问题通常能更接近地匹配完整基准测试对这些能力的衡量。","BenchMIRT 还可以利用它在模型和问题之间学到的模式，预测模型在未观察到的基准问题上的表现。在我们的实验中，它正确预测模型是否能正确回答保留的问题的准确率为79%。相比之下，另一种更简单的方法假设模型在每个问题上的表现大致与其在整个基准测试上的表现相当，其准确率为70%。","实际上，这意味着 BenchMIRT 可以更精确地估计模型的表现，基于它已学到的模型能力和每个问题的要求，而无需对每个模型的每个问题进行评估。","BenchMIRT 提供了一种更好地理解和优化研究人员用于评估模型能力的基准测试的方法。通过查看单个问题而不仅仅是整体分数，它可以揭示基准测试何时将不同能力混合在一起，识别表现与其他问题不同的题目群组，并发现那些对基准测试旨在衡量的能力提供信息有限的问题。","这其中存在重要限制。用于训练和评估 BenchMIRT 的模型均在 2025 年 3 月前发布，因此我们的分析未涵盖 BenchMIRT 在新一代大型语言模型上的表现。而 BenchMIRT 发现的维度取决于所给的基准测试集合——在我们为本项目选择的16个基准测试中，安全性和推理能力成为主要维度，但不同的评估组合可能会显现不同的潜在能力。","也存在权衡。如果目标是根据模型在随机保留题目上的预测表现来排名，基准测试的平均分略优于 BenchMIRT。BenchMIRT 的优势在于它能提供对单个问题表现的更细致分析。","这种问题级别的细节也可能有双重影响：同样的估计既可以帮助识别基准中最具信息性的安全问题，也可能被用来删除这些问题，从而产生一个不安全的模型也能通过的较弱评估。现有工具已经可以以类似方式修剪评估，我们认为对基准问题实际测量内容的透明度增加是值得冒这个风险的——但这确实是一个真实存在的风险。","尽管如此，我们仍然将 BenchMIRT——以及未来类似的工具——视为朝着更有针对性的基准设计和高效评估迈出的一步。通过展示哪些问题实际上在推动基准结果，这些方法可以帮助研究人员构建更小、更专注且更易于解读的评估，同时更清晰地展示其旨在测量的能力。"],"translationStatus":"translated","bodyOrigin":"source-page","editorial":{"summary":"Hugging Face 博客介绍了 BenchMIRT，一种在单个题目或任务层面审计大语言模型基准的方法。其训练数据覆盖 100 个 LLM、16 个基准和超过 3.4 万道题。","background":"BenchMIRT 借鉴项目反应理论，并扩展为多维项目反应理论，用于分析不同题目对模型表现的影响。来源举例称，BBQ 题目可能同时涉及年龄偏见、身份追踪和基于证据推理，WildJailbreak 中不同题目组也可能对应不同信号。","viewpoint":"Aioga 判断：BenchMIRT 的重点不只是比较模型分数，而是追问分数由哪些题目和潜在能力共同驱动。把基准拆到题目层面，可能有助于识别单一总分掩盖的测量差异。","implications":"可能影响：基准评测的解读需要关注题目构成及其区分度，单一平均分不足以说明模型具备某项能力。该方法可能为评测审计提供补充，但不代表仅凭题目层面分析即可完整定义模型能力。","nextStep":"后续观察：需要关注 BenchMIRT 在不同基准和题目集合上的审计结果，以及研究者如何使用其数据、技术报告和代码；这些材料是否改变基准分数的解读，仍待进一步观察。","evidenceRefs":["title","summary","articleBody","source"],"status":"published","aiGenerated":true,"autoApproved":true,"generatedBy":"aioga-editorial:gpt-5.6-sol","reviewedBy":"aioga-editorial-review:gpt-5.6-sol","generatedAt":"2026-09-01T23:24:00.558Z","sourceHash":"1544fe9abd7e08bf","review":{"approved":true,"groundedness":92,"clarity":91,"duplicationRisk":18,"blockingIssues":[],"notes":["“训练数据覆盖 100 个 LLM、16 个基准和超过 3.4 万道题”与来源摘要一致；如需更严谨，可改为“研究所用数据覆盖”。","“单一平均分不足以说明模型具备某项能力”属于基于来源关于总分可能掩盖差异的概括，当前已在 implications 中使用“可能”等限定词，未构成事实错误。","viewpoint 和 implications 已明确使用判断性、可能性表述，未将推测冒充为确定事实。"]},"validation":{"passed":true,"mode":"ai-auto","revisions":0,"checks":["schema","length","source-attribution","editorial-labels","inference-boundary","low-source-overlap","no-html","independent-ai-review"]}},"tags":["行业动态","Hugging Face：Blog（RSS）"],"translations":{"zh-CN":{"title":"Hugging Face 与 Ai2 发布 BenchMIRT：从题目层面审计 LLM 基准究竟测了什么","summary":"Hugging Face 博客介绍 BenchMIRT，一种基于多维 IRT 在单个题目层面审计 LLM 基准的方法，训练数据覆盖 100 个 LLM、16 个基准和超过 34K 道题。","category":"行业动态","source":"huggingface.co","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face 与 Ai2 发布 BenchMIRT：从题目层面审计 LLM 基准究竟测了什么 - Aioga AI资讯","description":"Hugging Face 博客介绍 BenchMIRT，一种基于多维 IRT 在单个题目层面审计 LLM 基准的方法，训练数据覆盖 100 个 LLM、16 个基准和超过 34K 道题。","url":"https://www.aioga.com/news/cmtj8p4l509paroh9e7kjhej4/","articleBody":["在基准测试中寻找信号：#finding-the-signals-inside-a-benchmark BenchMIRT揭示了现有基准测试的哪些信息：#what-benchmirt-reveals-about-existing-benchmarks 用更少的问题做更多事情：#doing-more-with-fewer-questions 这对LLM评估可能意味着什么：#what-this-could-mean-for-llm-evaluation 📄 技术报告: http://allenai.org/papers/benchmirt：http://allenai.org/papers/benchmirt | 📊 数据: https://huggingface.co/collections/allenai/benchmirt：https://huggingface.co/collections/allenai/benchmirt | 💻 代码: https://github.com/allenai/BenchMIRT：https://github.com/allenai/BenchMIRT","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/jlxf_E3loRcUO5Pu936zP.png","今天我们介绍BenchMIRT，一种用于在单个提示级别审核LLM基准测试的新方法——也就是模型评分所依据的问题和任务。","基准测试通常是为了测量特定能力而设计的，例如安全性、通用推理或遵循指令。但其中的单个任务可能依赖的不仅仅是所声明的目标。以BBQ为例，这是一个用于测试模型是否依赖社会刻板印象的基准测试。一个问题是关于孙子和祖父试图预订Uber的情景。它考察年龄偏见，但同时还要求模型追踪谁是谁，并根据提供的证据而非假设进行推理。","即使在单一基准测试中，不同的问题和任务组也可能测量不同的内容。例如，WildJailbreak包括有害的越狱提示，以及设计来测试模型是否也拒绝无害请求的良性提示。有害提示与安全性更紧密相关，而良性提示与通用推理更紧密相关。将它们平均到一个基准分数中可能掩盖了这种差异。","BenchMIRT帮助研究人员分离这些信号，看看到底是什么在推动基准测试的分数。它通过分析模型在每个问题或任务上的表现，并估计哪些底层能力与答对该问题最密切相关来实现这一点。","BenchMIRT 从项目反应理论（IRT）中获得启发，IRT 是起源于心理测量学的一种技术——该领域关注通过测试反应模式测量能力和特质。IRT 基于一个简单的想法：不是每个问题都能告诉你关于测试者的相同信息。有些问题比其他问题更难，有些问题则更能区分表现强与表现弱的人。","研究人员此前曾将单维 IRT 应用于各个基准测试，包括我们的 Fluid Benchmarking 工作。BenchMIRT 将该方法扩展为多维 IRT，或称 MIRT，使其能够区分可能影响同一问题表现的多种能力。","BenchMIRT 在模型层面和问题层面都应用 IRT。对于给定模型，它估计模型在所选基准测试中反映的各项能力上的强度。对于每个问题，它估计问题的难度以及问题在区分能力较强或较弱模型方面的效果。","我们在从 100 个大型语言模型获取的 16 个基准测试和超过 34K 个问题的基准结果上训练了 BenchMIRT。其中六个基准测试衡量通用推理能力，包括 MMLU-Pro、GPQA、MATH 和 BBH。其他十个基准测试来自我们的 Olmo 3 安全套件，包括 HarmBench、StrongReject、WildJailbreak、BBQ、WMDP 和 XSTest。","关键是，我们没有告诉 BenchMIRT 哪些基准测试衡量哪些能力。它独立发现了两个主要维度：安全性和通用推理能力。当我们从头重复分析时，每次都会出现这两个相同维度，这表明结果是稳定的，而不是特定于某次分析。","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/emqg9ENtMBcpo4S86cxy3.png","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/LMPWH4haeYge_HJInjhai.png","对于许多基准测试，BenchMIRT 在很大程度上确认了它们的预期焦点：在推理基准测试中表现强劲与推理能力相关，而在越狱和有害内容基准测试中表现强劲与安全性相关。","但 BenchMIRT 也在一些评估中揭示了更加复杂的情况。","BBQ 用于评估社会偏见，通常被归类于安全基准，在 BenchMIRT 的分析中与一般推理的相关性要强得多。这意味着低 BBQ 得分可能部分反映了理解或推理某些问题的困难，而不仅仅是安全行为。","WMDP 的表现与大多数安全基准不同。它测试生物学、化学和网络安全等领域的危险双用知识——例如，可能帮助人滥用生物制剂或利用计算机系统的知识。BenchMIRT 发现 WMDP 得分与一般推理的关联比与安全性的关联更强。然而，更强的一般推理能力与较低的 WMDP 得分相关，因为该基准将拒绝或未能提供危险知识视为期望的回答。","这些发现不一定意味着这些基准存在缺陷或不完整。相反，它们表明，单一基准得分可能结合了几种不同的信号——BenchMIRT 可以帮助解开这些信号，使得分更易于解释。","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/ue15HltywHtVyTIud5YlY.png","Harmbench 的两个维度下的题目难度和辨别力。维度 0 模拟安全维度，维度 1 对应一般推理维度。","：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/J18F7xrVwgaA68xg92Hs1.png","条形的大小和方向显示在 100 个开放权重 LLM 中，BenchMIRT 能力得分与基准得分之间的皮尔逊相关系数（范围 −1 到 1）——粉色表示一般推理，青绿色表示安全；条形向左延伸表示负相关。加粗带下划线表示每行更强的相关性，除非两者太接近无法区分；星号表示 p 值。","BenchMIRT 还可以帮助识别评估中哪些问题对基准试图测量的能力最具信息性。","利用 BenchMIRT 的题目级估计，我们对训练 BenchMIRT 所使用的相同 16 个基准的题目进行了排序，并保留那些在区分强模型和弱模型方面表现最好的题目，同时仍保持容易题和难题的混合。","在这些基准测试中，只保留10%的问题通常仍然能够几乎保持完整问题集所显示的模型在基础安全性或推理能力上的强弱情况。保留50%的问题通常能更接近地匹配完整基准测试对这些能力的衡量。","BenchMIRT 还可以利用它在模型和问题之间学到的模式，预测模型在未观察到的基准问题上的表现。在我们的实验中，它正确预测模型是否能正确回答保留的问题的准确率为79%。相比之下，另一种更简单的方法假设模型在每个问题上的表现大致与其在整个基准测试上的表现相当，其准确率为70%。","实际上，这意味着 BenchMIRT 可以更精确地估计模型的表现，基于它已学到的模型能力和每个问题的要求，而无需对每个模型的每个问题进行评估。","BenchMIRT 提供了一种更好地理解和优化研究人员用于评估模型能力的基准测试的方法。通过查看单个问题而不仅仅是整体分数，它可以揭示基准测试何时将不同能力混合在一起，识别表现与其他问题不同的题目群组，并发现那些对基准测试旨在衡量的能力提供信息有限的问题。","这其中存在重要限制。用于训练和评估 BenchMIRT 的模型均在 2025 年 3 月前发布，因此我们的分析未涵盖 BenchMIRT 在新一代大型语言模型上的表现。而 BenchMIRT 发现的维度取决于所给的基准测试集合——在我们为本项目选择的16个基准测试中，安全性和推理能力成为主要维度，但不同的评估组合可能会显现不同的潜在能力。","也存在权衡。如果目标是根据模型在随机保留题目上的预测表现来排名，基准测试的平均分略优于 BenchMIRT。BenchMIRT 的优势在于它能提供对单个问题表现的更细致分析。","这种问题级别的细节也可能有双重影响：同样的估计既可以帮助识别基准中最具信息性的安全问题，也可能被用来删除这些问题，从而产生一个不安全的模型也能通过的较弱评估。现有工具已经可以以类似方式修剪评估，我们认为对基准问题实际测量内容的透明度增加是值得冒这个风险的——但这确实是一个真实存在的风险。","尽管如此，我们仍然将 BenchMIRT——以及未来类似的工具——视为朝着更有针对性的基准设计和高效评估迈出的一步。通过展示哪些问题实际上在推动基准结果，这些方法可以帮助研究人员构建更小、更专注且更易于解读的评估，同时更清晰地展示其旨在测量的能力。"]},"en":{"title":"Hugging Face and Ai2 Launch BenchMIRT: Auditing What LLM Benchmarks Really Measure at the Question Level","summary":"Hugging Face’s blog introduces BenchMIRT, a method using multidimensional IRT to audit LLM benchmarks at the individual question level. The training data covers 100 LLMs, 16 benchmarks, and over 34K questions.","category":"Industry","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face and Ai2 Launch BenchMIRT: Auditing What LLM Benchmarks Really Measure at the Question Level - Aioga AI News","description":"Hugging Face’s blog introduces BenchMIRT, a method using multidimensional IRT to audit LLM benchmarks at the individual question level. The training data covers 100 LLMs, 16 benchm...","url":"https://www.aioga.com/en/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:03:47.970Z"},"ja":{"title":"Hugging FaceとAi2がリリースしたBenchMIRT:問題レベルでLLMベンチマークが実際に測定したものの監査","summary":"Hugging Faceブログでは、1問レベルでLLMベンチマークを監査するための多次元IRTベースの手法BenchMITRを紹介します。トレーニングデータは100のLLM、16のベンチマーク、34,000以上の問題をカバーしています。","category":"業界動向","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging FaceとAi2がリリースしたBenchMIRT:問題レベルでLLMベンチマークが実際に測定したものの監査 - Aioga AIニュース","description":"Hugging Faceブログでは、1問レベルでLLMベンチマークを監査するための多次元IRTベースの手法BenchMITRを紹介します。トレーニングデータは100のLLM、16のベンチマーク、34,000以上の問題をカバーしています。","url":"https://www.aioga.com/ja/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:03:48.342Z"},"ko":{"title":"Hugging Face와 Ai2 릴리스 BenchMIRT: 문제 수준에서 LLM 벤치마크가 실제로 측정한 내용을 감사하기","summary":"Hugging Face 블로그는 BenchMIRT를 소개합니다. BenchMIRT는 단일 질문 수준에서 LLM 벤치마크를 감사할 수 있는 다차원 IRT 기반 방법으로, 100개의 LLM, 16개의 벤치마크, 34,000개 이상의 문제를 포함하는 학습 데이터를 제공합니다.","category":"업계 동향","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face와 Ai2 릴리스 BenchMIRT: 문제 수준에서 LLM 벤치마크가 실제로 측정한 내용을 감사하기 - Aioga AI 뉴스","description":"Hugging Face 블로그는 BenchMIRT를 소개합니다. BenchMIRT는 단일 질문 수준에서 LLM 벤치마크를 감사할 수 있는 다차원 IRT 기반 방법으로, 100개의 LLM, 16개의 벤치마크, 34,000개 이상의 문제를 포함하는 학습 데이터를 제공합니다.","url":"https://www.aioga.com/ko/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:03:57.113Z"},"es":{"title":"Hugging Face y Ai2 lanzan BenchMIRT: Auditando lo que realmente midió el benchmark LLM a nivel de problema","summary":"El blog Hugging Face presenta BenchMIRT, un método multidimensional basado en IRT para auditar benchmarks de LLM a nivel de una sola pregunta, con datos de entrenamiento que cubren 100 LLMs, 16 benchmarks y más de 34K problemas.","category":"Industria","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face y Ai2 lanzan BenchMIRT: Auditando lo que realmente midió el benchmark LLM a nivel de problema - Aioga Noticias de IA","description":"El blog Hugging Face presenta BenchMIRT, un método multidimensional basado en IRT para auditar benchmarks de LLM a nivel de una sola pregunta, con datos de entrenamiento que cubren...","url":"https://www.aioga.com/es/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:03:56.703Z"},"fr":{"title":"Hugging Face et Ai2 publient BenchMIRT : Auditer ce que le benchmark LLM mesurait réellement au niveau du problème","summary":"Le blog Hugging Face présente BenchMIRT, une méthode multidimensionnelle basée sur IRT pour auditer les benchmarks LLM au niveau d’une seule question, avec des données d’entraînement couvrant 100 LLM, 16 benchmarks et plus de 34 000 problèmes.","category":"Industrie","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face et Ai2 publient BenchMIRT : Auditer ce que le benchmark LLM mesurait réellement au niveau du problème - Aioga Actualités IA","description":"Le blog Hugging Face présente BenchMIRT, une méthode multidimensionnelle basée sur IRT pour auditer les benchmarks LLM au niveau d’une seule question, avec des données d’entraîneme...","url":"https://www.aioga.com/fr/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:05.900Z"},"de":{"title":"Hugging Face und Ai2 veröffentlichen BenchMIRT: Überprüfung dessen, was der LLM-Benchmark tatsächlich auf Problemebene gemessen hat","summary":"Der Blog Hugging Face stellt BenchMIRT vor, eine multidimensionale, IRT-basierte Methode zur Überprüfung von LLM-Benchmarks auf der Ebene einer einzelnen Frage, mit Trainingsdaten, die 100 LLMs, 16 Benchmarks und über 34.000 Aufgaben abdecken.","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face und Ai2 veröffentlichen BenchMIRT: Überprüfung dessen, was der LLM-Benchmark tatsächlich auf Problemebene gemessen hat - Aioga KI-News","description":"Der Blog Hugging Face stellt BenchMIRT vor, eine multidimensionale, IRT-basierte Methode zur Überprüfung von LLM-Benchmarks auf der Ebene einer einzelnen Frage, mit Trainingsdaten,...","url":"https://www.aioga.com/de/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:06.320Z"},"pt-BR":{"title":"Hugging Face e Ai2 lançam BenchMIRT: Auditando o que o benchmark do LLM realmente mediu no nível do problema","summary":"O blog Hugging Face apresenta o BenchMIRT, um método multidimensional baseado em IRT para auditar benchmarks de LLM no nível de uma única pergunta, com dados de treinamento abrangendo 100 LLMs, 16 benchmarks e mais de 34K problemas.","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face e Ai2 lançam BenchMIRT: Auditando o que o benchmark do LLM realmente mediu no nível do problema - Aioga Notícias de IA","description":"O blog Hugging Face apresenta o BenchMIRT, um método multidimensional baseado em IRT para auditar benchmarks de LLM no nível de uma única pergunta, com dados de treinamento abrange...","url":"https://www.aioga.com/pt-BR/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:15.377Z"},"ru":{"title":"Релиз Hugging Face и Ai2 BenchMIRT: Аудит того, что на самом деле измерил бенчмарк LLM на уровне задачи","summary":"Блог Hugging Face представляет BenchMIRT — многомерный метод на основе IRT для аудита бенчмарков LLM на уровне одного вопроса, с обучающими данными, охватывающими 100 LLM, 16 бенчмарков и более чем 34K задач.","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Релиз Hugging Face и Ai2 BenchMIRT: Аудит того, что на самом деле измерил бенчмарк LLM на уровне задачи - Aioga Новости ИИ","description":"Блог Hugging Face представляет BenchMIRT — многомерный метод на основе IRT для аудита бенчмарков LLM на уровне одного вопроса, с обучающими данными, охватывающими 100 LLM, 16 бенчм...","url":"https://www.aioga.com/ru/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:14.847Z"},"ar":{"title":"إصدار BenchMIRT من Hugging Face وAi2: تدقيق ما قاسته معيار LLM فعليا على مستوى المشكلة","summary":"تقدم مدونة Hugging Face طريقة BenchMIRT، وهي طريقة متعددة الأبعاد تعتمد على IRT لتدقيق معايير نماذج اللغة الكبيرة على مستوى سؤال واحد، مع بيانات تدريبية تغطي 100 نموذج LLM، و16 معيارا، وأكثر من 34 ألف مسألة.","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"إصدار BenchMIRT من Hugging Face وAi2: تدقيق ما قاسته معيار LLM فعليا على مستوى المشكلة - Aioga أخبار الذكاء الاصطناعي","description":"تقدم مدونة Hugging Face طريقة BenchMIRT، وهي طريقة متعددة الأبعاد تعتمد على IRT لتدقيق معايير نماذج اللغة الكبيرة على مستوى سؤال واحد، مع بيانات تدريبية تغطي 100 نموذج LLM، و16 معي...","url":"https://www.aioga.com/ar/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:24.340Z"},"hi":{"title":"हगिंग फेस और एआई2 रिलीज बेंचएमआईआरटी: ऑडिटिंग एलएलएम बेंचमार्क ने वास्तव में समस्या के स्तर पर क्या मापा","summary":"हगिंग फेस ब्लॉग बेंचएमआईआरटी का परिचय देता है, जो एक ही प्रश्न के स्तर पर एलएलएम बेंचमार्क का ऑडिट करने के लिए एक बहुआयामी आईआरटी-आधारित विधि है, जिसमें 100 एलएलएम, 16 बेंचमार्क और 34K से अधिक समस्याओं को कवर करने वाला प्रशिक्षण डेटा है।","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"हगिंग फेस और एआई2 रिलीज बेंचएमआईआरटी: ऑडिटिंग एलएलएम बेंचमार्क ने वास्तव में समस्या के स्तर पर क्या मापा - Aioga AI समाचार","description":"हगिंग फेस ब्लॉग बेंचएमआईआरटी का परिचय देता है, जो एक ही प्रश्न के स्तर पर एलएलएम बेंचमार्क का ऑडिट करने के लिए एक बहुआयामी आईआरटी-आधारित विधि है, जिसमें 100 एलएलएम, 16 बेंचमार्क और...","url":"https://www.aioga.com/hi/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:24.263Z"},"it":{"title":"Hugging Face e Ai2 rilasciano BenchMIRT: Revisione di ciò che il benchmark LLM ha effettivamente misurato a livello di problema","summary":"Il blog Hugging Face introduce BenchMIRT, un metodo multidimensionale basato su IRT per auditare benchmark LLM a livello di una singola domanda, con dati di addestramento che coprono 100 LLM, 16 benchmark e oltre 34K problemi.","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face e Ai2 rilasciano BenchMIRT: Revisione di ciò che il benchmark LLM ha effettivamente misurato a livello di problema - Aioga Notizie IA","description":"Il blog Hugging Face introduce BenchMIRT, un metodo multidimensionale basato su IRT per auditare benchmark LLM a livello di una singola domanda, con dati di addestramento che copro...","url":"https://www.aioga.com/it/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:31.755Z"},"nl":{"title":"Hugging Face en Ai2 brengen BenchMIRT uit: Auditing van wat de LLM-benchmark daadwerkelijk op probleemniveau heeft gemeten","summary":"De Hugging Face-blog introduceert BenchMIRT, een multidimensionale IRT-gebaseerde methode voor het auditen van LLM-benchmarks op het niveau van één enkele vraag, met trainingsdata die 100 LLM's, 16 benchmarks en meer dan 34K problemen omvatten.","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face en Ai2 brengen BenchMIRT uit: Auditing van wat de LLM-benchmark daadwerkelijk op probleemniveau heeft gemeten - Aioga AI-nieuws","description":"De Hugging Face-blog introduceert BenchMIRT, een multidimensionale IRT-gebaseerde methode voor het auditen van LLM-benchmarks op het niveau van één enkele vraag, met trainingsdata...","url":"https://www.aioga.com/nl/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:32.709Z"},"tr":{"title":"Hugging Face ve Ai2 sürümü BenchMIRT: LLM kıyaslamasının problem seviyesinde neyi ölçtüğü denetlemek","summary":"Hugging Face blogu, 100 LLM, 16 benchmark ve 34K'den fazla problemi kapsayan, LLM kıyaslamalarını tek bir soru seviyesinde denetlemek için çok boyutlu IRT tabanlı bir yöntem olan BenchMIRT'i tanıtıyor.","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face ve Ai2 sürümü BenchMIRT: LLM kıyaslamasının problem seviyesinde neyi ölçtüğü denetlemek - Aioga AI Haberleri","description":"Hugging Face blogu, 100 LLM, 16 benchmark ve 34K'den fazla problemi kapsayan, LLM kıyaslamalarını tek bir soru seviyesinde denetlemek için çok boyutlu IRT tabanlı bir yöntem olan B...","url":"https://www.aioga.com/tr/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:40.539Z"},"vi":{"title":"Hugging Face và Ai2 phát hành BenchMIRT: Kiểm toán những gì benchmark LLM thực sự đo được ở cấp độ vấn đề","summary":"Blog Hugging Face giới thiệu BenchMIRT, một phương pháp đa chiều dựa trên IRT để kiểm tra các bài kiểm tra hiệu năng LLM ở cấp độ một câu hỏi duy nhất, với dữ liệu huấn luyện bao gồm 100 LLMs, 16 bài kiểm tra và hơn 34K bài toán.","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face và Ai2 phát hành BenchMIRT: Kiểm toán những gì benchmark LLM thực sự đo được ở cấp độ vấn đề - Tin tức AI Aioga","description":"Blog Hugging Face giới thiệu BenchMIRT, một phương pháp đa chiều dựa trên IRT để kiểm tra các bài kiểm tra hiệu năng LLM ở cấp độ một câu hỏi duy nhất, với dữ liệu huấn luyện bao g...","url":"https://www.aioga.com/vi/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:41.151Z"},"id":{"title":"Hugging Face dan rilis Ai2 BenchMIRT: Mengaudit apa yang sebenarnya diukur oleh benchmark LLM pada tingkat masalah","summary":"Blog Hugging Face memperkenalkan BenchMIRT, metode berbasis IRT multidimensi untuk mengaudit tolok ukur LLM pada tingkat satu pertanyaan, dengan data pelatihan mencakup 100 LLM, 16 tolok ukur, dan lebih dari 34 ribu masalah.","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face dan rilis Ai2 BenchMIRT: Mengaudit apa yang sebenarnya diukur oleh benchmark LLM pada tingkat masalah - Berita AI Aioga","description":"Blog Hugging Face memperkenalkan BenchMIRT, metode berbasis IRT multidimensi untuk mengaudit tolok ukur LLM pada tingkat satu pertanyaan, dengan data pelatihan mencakup 100 LLM, 16...","url":"https://www.aioga.com/id/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:49.245Z"},"th":{"title":"Hugging Face และ Ai2 เปิดตัว BenchMIRT: การตรวจสอบว่า LLM benchmark วัดอะไรได้จริงในระดับปัญหา","summary":"บล็อก Hugging Face แนะนํา BenchMIRT วิธีการหลายมิติที่ใช้ IRT สําหรับการตรวจสอบ LLM benchmarks ในระดับคําถามเดียว โดยมีข้อมูลการฝึกอบรมครอบคลุม LLM 100 รายการ, 16 การทดสอบประสิทธิภาพ และปัญหากว่า 34K ข้อ","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face และ Ai2 เปิดตัว BenchMIRT: การตรวจสอบว่า LLM benchmark วัดอะไรได้จริงในระดับปัญหา - ข่าว AI Aioga","description":"บล็อก Hugging Face แนะนํา BenchMIRT วิธีการหลายมิติที่ใช้ IRT สําหรับการตรวจสอบ LLM benchmarks ในระดับคําถามเดียว โดยมีข้อมูลการฝึกอบรมครอบคลุม LLM 100 รายการ, 16 การทดสอบประสิทธิภ...","url":"https://www.aioga.com/th/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:48.995Z"},"pl":{"title":"Hugging Face i Ai2 wydają BenchMIRT: audyt tego, co benchmark LLM faktycznie zmierzył na poziomie problemu","summary":"Blog Hugging Face wprowadza BenchMIRT, wielowymiarową metodę audytu benchmarków LLM opartą na IRT na poziomie pojedynczego pytania, z danymi treningowymi obejmującymi 100 LLM, 16 benchmarków oraz ponad 34 tys. problemów.","category":"行业动态","source":"Hugging Face：Blog（RSS）","aggregationSource":"Hugging Face：Blog（RSS）","pageTitle":"Hugging Face i Ai2 wydają BenchMIRT: audyt tego, co benchmark LLM faktycznie zmierzył na poziomie problemu - Aioga Wiadomości AI","description":"Blog Hugging Face wprowadza BenchMIRT, wielowymiarową metodę audytu benchmarków LLM opartą na IRT na poziomie pojedynczego pytania, z danymi treningowymi obejmującymi 100 LLM, 16 b...","url":"https://www.aioga.com/pl/news/cmtj8p4l509paroh9e7kjhej4/","contentTranslated":true,"sourceHash":"b6d0c34551846b48","translatedAt":"2026-09-01T23:04:58.395Z"}},"evidenceTier":"verified-news","reviewStatus":"automated-ingest","indexable":true,"editorialCover":""}}