{"@context":"https://schema.org","@type":"NewsArticle","generatedAt":"2026-10-22T19:00:36.461Z","headline":"Microsoft ThinkingBox 在 Hugging Face 上发布，以数据库终态和 20 次重复评测智能体","description":"Microsoft 与 Hugging Face 发布 ThinkingBox 智能体沙箱与 ThinkingBox-Bench 基准，覆盖 507 个有状态业务工作流、每任务运行 20 次，以终局数据库状态和副作用作可执行判定，现可通过 OpenEnv 在 Hugging Face 上运行。","url":"https://www.aioga.com/news/gqh4yclcjmaci6uhur56580uh/","mainEntityOfPage":"https://www.aioga.com/news/gqh4yclcjmaci6uhur56580uh/","datePublished":"2026-10-03T22:56:48.000Z","dateModified":"2026-10-03T22:56:48.000Z","inLanguage":"zh-CN","publisher":{"@type":"NewsMediaOrganization","name":"Aioga","url":"https://www.aioga.com"},"citation":["https://huggingface.co/blog/microsoft/thinkingbox","https://aihot.news/items/gqh4yclcjmaci6uhur56580uh"],"canonicalUrl":"https://www.aioga.com/news/gqh4yclcjmaci6uhur56580uh/","directAnswer":{"@type":"Answer","text":"Microsoft 与 Hugging Face 发布 ThinkingBox 智能体沙箱和 ThinkingBox-Bench 基准，覆盖 507 个有状态业务工作流，并要求每项任务运行 20 次，以最终数据库状态和副作用进行判定。","url":"https://www.aioga.com/news/gqh4yclcjmaci6uhur56580uh/","dateCreated":"2026-10-03T22:56:48.000Z","author":{"@type":"Organization","@id":"https://www.aioga.com/authors/aioga-editorial/#editorial-team","name":"Aioga Editorial Team","url":"https://www.aioga.com/authors/aioga-editorial/"}},"evidence":[{"@type":"CreativeWork","name":"Hugging Face：Blog source article","url":"https://huggingface.co/blog/microsoft/thinkingbox","datePublished":"2026-10-03T22:56:48.000Z","provider":{"@type":"Organization","name":"Hugging Face：Blog","url":"https://huggingface.co/blog/microsoft/thinkingbox"}},{"@type":"CreativeWork","name":"AIHot archive record","url":"https://aihot.news/items/gqh4yclcjmaci6uhur56580uh","datePublished":"2026-10-03T22:56:48.000Z","provider":{"@type":"Organization","name":"AIHot","url":"https://aihot.news/items/gqh4yclcjmaci6uhur56580uh"}}],"aggregationSource":"Hugging Face：Blog","originalPublisher":{"name":"Hugging Face：Blog","url":"https://huggingface.co/blog/microsoft/thinkingbox"},"geoDeepAnswer":null,"article":{"id":"gqh4yclcjmaci6uhur56580uh","slug":"gqh4yclcjmaci6uhur56580uh","url":"https://www.aioga.com/news/gqh4yclcjmaci6uhur56580uh/","title":"Microsoft ThinkingBox 在 Hugging Face 上发布，以数据库终态和 20 次重复评测智能体","title_en":"","summary":"Microsoft 与 Hugging Face 发布 ThinkingBox 智能体沙箱与 ThinkingBox-Bench 基准，覆盖 507 个有状态业务工作流、每任务运行 20 次，以终局数据库状态和副作用作可执行判定，现可通过 OpenEnv 在 Hugging Face 上运行。","source":"Hugging Face：Blog","sourceUrl":"https://huggingface.co/blog/microsoft/thinkingbox","aiHotUrl":"https://aihot.news/items/gqh4yclcjmaci6uhur56580uh","publishedAt":"2026-10-03T22:56:48.000Z","category":"行业动态","score":72,"selected":true,"articleBody":["A tool call is not an outcome ：#a-tool-call-is-not-an-outcome One success is not reliability ：#one-success-is-not-reliability Can you depend on the model behind your agent? ：#can-you-depend-on-the-model-behind-your-agent What consistency costs ：#what-consistency-costs Pareto cost frontier ：#pareto-cost-frontier Now price consistency ：#now-price-consistency Failure signatures ：#failure-signatures How it works ：#how-it-works Run it yourself ：#run-it-yourself Before you start ：#before-you-start Install ：#install Start Typesense ：#start-typesense Start the MCP servers ：#start-the-mcp-servers Start the OpenEnv server ：#start-the-openenv-server Check readiness ：#check-readiness Score an episode ：#score-an-episode Where this goes next ：#where-this-goes-next Microsoft ThinkingBox grades AI agents on the records they leave behind, not the sentences they generate, and then asks whether they can do it twenty times in a row. It is now available through Hugging Face.","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/Dzuml_9K2lfRq4hZi17EY.png","Figure 1: ThinkingBox runs an agent against isolated MCP tool sessions, then grades the terminal backend state and side effects it leaves behind. From our ThinkingBox paper：https://arxiv.org/abs/2608.19741.","A customer writes in. Her $745 kitchen appliance has been stuck in a courier \"exception\" at a Nashville distribution center, fifteen days past its estimated delivery date.","The AI agent does careful work. Nine tool calls: it pulls the order, checks tracking, looks up her customer profile, searches the refund policy twice, confirms no ticket exists, opens one, documents the timeline, and reads the policy correctly; her account segment genuinely does not qualify for late-delivery compensation.","Then it closes the ticket as resolved and replies “ Since your query is resolved, is there anything I may assist you with? ”","Two things are wrong. The carrier exception is still open, so the required end state was on hold , pending resolution. And the customer never got a real answer to what she actually asked.","An AI grader checking tool calls would see nine well-formed ones. The grader checking whether the agent wrote to the database would see that too. The database is what disagrees .","That gap is what ThinkingBox measures. Across 507 stateful business workflows, each run 20 times against various LLM models, it grades agents on terminal backend state and side effects. This post covers what we found, what consistency costs, and how to run the benchmark yourself through OpenEnv：https://github.com/huggingface/OpenEnv/tree/main/envs/thinkingbox_env.","You can run this one yourself: the example above is adapted from a benchmark task sandbox_external_retail_group1.py:test_case_ST003_006：https://github.com/microsoft/thinkingbox-data/blob/thinkingbox-bench-v1.0/dataset/test_case/sandbox_external_retail/sandbox_external_retail_group1.py#L984-L1223, and the executable check that fails is a single field: the ticket's status is solved where the required end state is hold. The full trace is in Appendix D.4, Case 3 of our paper：https://arxiv.org/pdf/2608.19741.","Want to try it before reading the results? Skip to section Run it yourself：#run-it-yourself.","Final responses and valid tool calls are only proxies. An agent can sound correct while leaving the wrong value, changing the wrong record, or creating an extra side effect. Only the records it leaves behind settle the question.","The gap is substantial. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed the executable checks. Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool, and reported no final tool error. Executable checks nevertheless found wrong field values in 77.61% of them, unintended extra effects in 43.30%, and missing required effects in 25.36%. Those state-check findings overlap.","A trajectory is a claim. Database state is the evidence. Repetition is the trust test.","An agent that processes a refund correctly once and mishandles it the next four times is not a working refund agent. So every task runs 20 independent times , each from an identical clean backend, and we report three different things:","Table 1: The three numbers we report, and the question each one answers.","We use observed 20/20 in this blog post as the literal count of how many of the 507 tasks passed 20 out of 20. No estimator, no smoothing.","Starting with the familiar view. The table below reports pass@1, the single-attempt score estimate, broken out by domain. This is the number most leaderboards publish, and on its own it reads like an ordinary capability ranking.","Table 2: ThinkingBox-Bench pass@1 (%) by domain. Each model is evaluated on every task for 20 repeated trials. Bold marks the group leader; underline marks the runner-up. The standard errors for the single attempt score estimates are provided in Table 4 in our ThinkingBox paper：https://arxiv.org/abs/2608.19741.","Claude Opus 5.5 leads overall at 67.16%, two-thirds of a point above Claude Opus 5. Kimi-K3 is the strongest open-weights model , within a point of GPT-6-Astra. Domain matters just as much : Claude Opus 4.6 scores 68.62% on retail but 8.30% on auto insurance.","One good run tells you a model can do the work. It does not tell you whether it will do it again. So run every task 20 times and ask how much of that score survives.","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/DjaLNZKNeU--Xy2WFCvVX.png","Figure 2: How much of each model's single-attempt score survives 20 repeats.","Only three hold on to most of their pass@1 scores: GPT-6 Astra retains 78% of its single-attempt rate, and Claude Opus 5.5 and Claude Opus 5 each retain 71%. At the other end, GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8%.","The gap between what a model can do once and what it does every time is the whole story.","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/NONd2WGm4vtGpsVgs-CIq.png","Figure 3: Breadth and consistency pull apart. Twelve of the eighteen models are shown; six below 33% pass@1 are omitted for legibility.","Kimi-K3 has the broadest coverage of any model we tested. It solves 93.89% of the benchmark at least once: 476 of 507 tasks. Only 31 tasks defeat it entirely, the lowest count in the field. On retail workflows it leads outright at 82.24% pass@1, ahead of every proprietary model.","Kimi-K3 is also among the least consistent. Just 68 of 507 tasks, 13.41%, succeed in all 20 attempts.","Claude Opus 5 inverts this. It solves fewer tasks at least once (79.09%; 106 defeat it entirely) but completes 47.53% of the benchmark on every single attempt.","A newer model does not fix this. Claude Opus 5.5 scores higher than Claude Opus 5 on every-attempt average, 67.16% against 66.50%, and solves more tasks at least once. It passes exactly the same number of tasks on all 20 attempts: 241. Half a point of headline accuracy bought no additional dependability at all.","If you are choosing a model for work that touches real records, pass@20 is the wrong column to look at.","Capability comparisons usually stop at the score. For anyone deploying, the relevant question is what a successful unit of work costs. We measure that as cost per successful task attempt. We say task attempt because every benchmark task is run repeatedly and cost is incurred per attempt, so pass@1 is the matching quality denominator.","We took each model's recorded token usage from its full 507 × 20 campaign and priced it at undiscounted list rates available on OpenRouter：https://openrouter.ai/ + , reversing promotional discounts and excluding endpoints that declare quantization. Input, output and cache rates all come from one provider endpoint per model.","Then we divided one run's cost by the number of attempts that succeeded :","Cost per successful task attempt = estimated cost for 507 attempts, one per task ÷ (507 × pass@1)","This is a comparative efficiency index, not an invoice, and not the price of serving one production request. It also prices single successes, not consistency. We price consistency next.","Example: GPT-5.4 costs $43.49 for 507 attempts (one attempt per task) and has 65.36% pass@1, so $43.49 ÷ (507 × 0.6536) = $0.131 per successful task attempt.","A model is on the frontier if no other model is both no more expensive and at least as accurate . Three models qualify; every other model is dominated on at least one axis.","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/oGmXYrxoNPJfsQndxlcMD.png","Figure 4: Cost per successful task attempt against pass@1. Ringed dots are pareto cost frontier models.","The frontier has three steps. GPT-5.6 Sol has the lowest cost per success at $0.127; GPT-5.4 raises pass@1 by 3.45 percentage points for $0.004 more per success; Claude Opus 5.5 adds another 1.80 points at $0.276 per success. Each of the three remains on the cost frontier line because no cheaper model matches its pass@1.","Claude Opus 5 is the clearest case: at $0.475 per successful attempt and 66.50% pass@1, it is both more expensive and less accurate than Claude Opus 5.5 at $0.276 and 67.16%.","Cost per success rewards a model that is cheap and often right. It does not reward a model that is right every time. So we also compute cost per dependable task: the cost of the full 20-run campaign divided by the number of tasks the model passed on all 20 attempts.","Cost per dependable task = estimated cost of 20 runs of 507 attempts ÷ tasks passing 20/20","Example: GPT-6-Astra costs 20 × $86.03 = $1,720.60 for the campaign and passes 231 tasks on every attempt, so $1,720.60 ÷ 231 = $7.45 per dependable task.","Table 3: The nine lowest costs per dependable task among models with at least one observed 20/20 task, sorted low to high. Estimated $, not actual cloud bills.","Now rank by consistency. GPT-5.4 is the cheapest at $6.80, though only 128 tasks meet the bar. GPT-6 Astra reaches 231 at $7.45, and Claude Opus 5.5 the joint-highest 241 at $7.80.","None of the three dominates the others: each additional dependable task costs more. Claude Opus 5 also passes 241, but at $13.30, so Opus 5.5 dominates it outright. GPT-5.6 Sol, the cheapest per single success at $0.127, costs $9.76 per dependable task. The cheapest way to get a right answer is not the cheapest way to get a dependable one.","We assign each failed trace one deterministic diagnostic signature, and the headline is actionable: roughly four in five failures are tool handling, not reasoning. Across an ablation study in Table 5 of our paper：https://arxiv.org/pdf/2608.19741:","These are unweighted averages of per-model shares and observable labels, not unique causal explanations.","The practical pattern is simple: agents usually get far enough to attempt the workflow, then fail to recover from tool errors, failed preconditions, or empty lookups. That is a retry and error-recovery problem before it is a model problem.","Difficulty also changes by domain: across the models listed in Table 2 above, retail averages 59.52% pass@1 while auto insurance averages 33.83%.","What to do about it. Treat the 20/20 rate as a design input, not a verdict. The same signal the benchmark grades on is available in production: check the terminal state before you commit, not the model's summary of it.","Classify tool and system errors so retries target the recoverable ones. Cut the tool surface to what the workflow needs. And require human approval on the changes you cannot cheaply reverse. We have not measured the lift from any of these on this benchmark, which is exactly the kind of thing the environment now makes testable.","ThinkingBox is the agent sandbox, while ThinkingBox-Bench is a dataset benchmark to evaluate agents. The diagram at the top of this post shows the loop; here is what each part does.","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/elbRGqTXW6PIqci7xcEJN.png","Figure 5: The sandbox loop from panel A in Figure 1 above: isolated tool session, terminal database state, side effects, executable judges.","Each task defines a starting backend state, a user goal, the available MCP tools, the domain policy, and executable checks over the terminal state. A simulated user holds private context (a booking reference, a preference, or a date of birth) and releases it only when asked.","Every attempt gets an isolated MCP session with freshly initialized state. Two attempts of the same task never share a database row or cached tool state, which is what makes 20-trial comparison meaningful."],"articleImages":[{"sourceUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64b8491203124195cd795cad/KWtqxooyzbqzLqdJorOPi.png","alt":"","afterParagraph":0,"url":"/media/articles/gqh4yclcjmaci6uhur56580uh/f753aad2c3a9197f.webp"},{"sourceUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a27ffbf0ea26227e43a0d29/Hk8XoWHT5braavBTjifaw.jpeg","alt":"","afterParagraph":0,"url":"/media/articles/gqh4yclcjmaci6uhur56580uh/bb4916223c989ad4.webp"},{"sourceUrl":"https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/Dzuml_9K2lfRq4hZi17EY.png","alt":"Figure-1","afterParagraph":0,"url":"/media/articles/gqh4yclcjmaci6uhur56580uh/5789d0e8ecbd7361.png"},{"sourceUrl":"https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/DjaLNZKNeU--Xy2WFCvVX.png","alt":"Figure-2","afterParagraph":20,"url":"/media/articles/gqh4yclcjmaci6uhur56580uh/7c9c0618a44721b5.png"}],"mediaStatus":"ok","articleBodyZh":["一个工具调用不是结果：#a-tool-call-is-not-an-outcome 一次成功不是可靠性：#one-success-is-not-reliability 你能依赖你代理背后的模型吗？：#can-you-depend-on-the-model-behind-your-agent 一致性要付出什么代价：#what-consistency-costs 帕累托成本前沿：#pareto-cost-frontier 现在定价一致性：#now-price-consistency 失败特征：#failure-signatures 它是如何工作的：#how-it-works 自己运行它：#run-it-yourself 在你开始之前：#before-you-start 安装：#install 启动 Typesense：#start-typesense 启动 MCP 服务器：#start-the-mcp-servers 启动 OpenEnv 服务器：#start-the-openenv-server 检查就绪情况：#check-readiness 给一集打分：#score-an-episode 下一步走向：#where-this-goes-next Microsoft ThinkingBox 根据 AI 代理留下的记录对其评分，而不是生成的句子，然后检查它们是否可以连续完成二十次。现在可以通过 Hugging Face 获取。","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/Dzuml_9K2lfRq4hZi17EY.png","图 1：ThinkingBox 让代理在独立的 MCP 工具会话中运行，然后对其留下的终端后端状态和副作用进行评分。来自我们的 ThinkingBox 论文：https://arxiv.org/abs/2608.19741。","一位客户来信。她的 745 美元厨房电器在纳什维尔的配送中心被卡在快递“例外”状态，超过预计交货日期十五天。","AI 代理仔细工作。九次工具调用：它拉取订单，检查追踪信息，查看她的客户档案，两次搜索退款政策，确认没有现有工单，创建一个工单，记录时间线，并正确阅读政策；她的账户分段确实不符合延迟交货赔偿。","然后它将工单关闭为已解决，并回复“由于您的问题已解决，我还能为您提供其他帮助吗？”","有两点错误。承运商异常仍然存在，因此所需的最终状态是挂起，等待解决。此外，客户从未真正得到她实际提出的问题的答案。","检查工具调用的 AI 评分器会看到九次格式良好的调用。检查代理是否写入数据库的评分器也会看到这一点。不同的是数据库给出的结果与预期不符。","ThinkingBox 衡量的就是这个差距。在 507 个有状态的业务工作流中，每个工作流在不同的 LLM 模型上运行 20 次，它根据终端后台状态和副作用对代理进行评分。本文介绍了我们的发现、一致性成本以及如何通过 OpenEnv 自行运行基准测试：https://github.com/huggingface/OpenEnv/tree/main/envs/thinkingbox_env。","你可以自己运行这个任务：上面的示例是从基准任务 sandbox_external_retail_group1.py:test_case_ST003_006 改编而来的：https://github.com/microsoft/thinkingbox-data/blob/thinkingbox-bench-v1.0/dataset/test_case/sandbox_external_retail/sandbox_external_retail_group1.py#L984-L1223，且失败的可执行检查仅涉及一个字段：工单状态为“已解决”，而所需的最终状态是“待处理”。完整的追踪记录见我们论文附录 D.4，案例 3：https://arxiv.org/pdf/2608.19741。","想在阅读结果前试试吗？跳到“自己运行”部分：#run-it-yourself。","最终响应和有效工具调用只是代理指标。一个代理可能听起来正确，但却留下错误的数值、修改了错误的记录或生成了额外的副作用。只有它留下的记录才能揭示问题的真相。","差距是显著的。在一个包含 121,680 次有效试验、覆盖 12 个 LLM 模型的常见集合消融实验中，有 79,853 次尝试未通过可执行检查。在这些失败中，67.24% 仍然正常结束，调用了会改变状态的工具，并报告无最终工具错误。然而，可执行检查仍在其中 77.61% 的案例中发现字段值错误，43.30% 的案例中出现意外额外副作用，25.36% 的案例中缺少必要副作用。这些状态检查结果存在重叠。","一个轨迹是一种声明。数据库状态是证据。重复是信任测试。","一个代理如果一次正确处理退款，但接下来的四次处理错误，就不能算作一个可用退款代理。因此，每个任务都会独立运行 20 次，每次从相同的干净后台开始，我们报告三种不同的数据：","表 1：我们报告的三组数据，以及每组数据回答的问题。","在本文博客中，我们使用 \"observed 20/20\" 作为字面计数，表示 507 个任务中有多少任务 20 次全满分。没有估计器，没有平滑处理。","从熟悉的视角开始。下表报告了 pass@1，即单次尝试得分估计，按领域分类。这个数字是大多数排行榜发布的数字，本身看起来像普通的能力排名。","表 2：ThinkingBox-Bench 按领域的 pass@1 (%)。每个模型在每个任务上都进行 20 次重复评估。加粗表示组内领头；下划线表示亚军。单次尝试得分估计的标准误见我们 ThinkingBox 论文的表 4：https://arxiv.org/abs/2608.19741。","Claude Opus 5.5 总体领先，达 67.16%，比 Claude Opus 5 高出约三分之一点。Kimi-K3 是最强的开源权重模型，接近 GPT-6-Astra 仅一分之差。领域同样很重要：Claude Opus 4.6 在零售领域得分 68.62%，但在汽车保险领域仅得 8.30%。","一次良好的运行表明模型能够完成工作，但并不能说明它是否会再次完成。因此，对每个任务运行 20 次，然后观察得分能保持多少。","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/DjaLNZKNeU--Xy2WFCvVX.png","图 2：每个模型的单次尝试得分在 20 次重复中保持的比例。","只有三款模型能保持大部分 pass@1 得分：GPT-6 Astra 保持 78% 的单次尝试率，Claude Opus 5.5 和 Claude Opus 5 各保持 71%。相反，GLM-5.1、Kimi-K2.6 和 DeepSeek-V4-Pro 各仅保持约 8%。","模型一次能够完成的与每次都能完成的之间的差距才是完整的故事。","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/NONd2WGm4vtGpsVgs-CIq.png","图 3：广度与一致性拉开了距离。显示了 18 个模型中的 12 个；为了可读性，pass@1 低于 33% 的 6 个模型被省略。","Kimi-K3 拥有我们测试的模型中最广的覆盖率。它至少成功完成了 93.89% 的基准测试：507 个任务中完成了 476 个。仅有 31 个任务完全失败，是该领域最低的数量。在零售工作流程中，它以 82.24% 的 pass@1 排名第一，领先于所有专有模型。","Kimi-K3 也是一致性较低的模型之一。在 507 个任务中，仅有 68 个任务（13.41%）在 20 次尝试中全部成功。","Claude Opus 5 扭转了这一点。它至少完成较少的任务（79.09%；106 个任务完全没完成），但在每次尝试中完成了基准测试的 47.53%。","更新的模型没有解决这个问题。Claude Opus 5.5 在每次尝试的平均得分上高于 Claude Opus 5，分别为 67.16% 对 66.50%，并且至少完成更多任务。它在所有 20 次尝试中通过的任务数量完全相同：241 个。半个百分点的总体准确率没有带来任何额外的可靠性。","如果你正在选择用于处理真实记录的模型，pass@20 不是应该关注的列。","能力比较通常止于分数。对于任何部署者来说，真正相关的问题是成功完成一项工作的成本。我们将其测量为每次成功任务尝试的成本。我们说“任务尝试”，因为每个基准任务都重复运行且每次尝试都会产生成本，所以 pass@1 是匹配质量的分母。","我们使用每个模型在完整的 507 × 20 测试中记录的 token 使用情况，并按 OpenRouter（https://openrouter.ai/）上提供的未折扣列表价进行定价，撤销促销折扣并且排除声明量化的端点。输入、输出和缓存费用都来自每个模型的一个提供端点。","然后我们将一次运行的成本除以成功尝试的次数：","每次成功任务尝试的成本 = 507 次尝试（每个任务各一次）的预计成本 ÷ (507 × pass@1)","这是一种比较效率指数，而不是发票，也不是处理一次生产请求的价格。它也只计算单次成功，而不是一致性。我们接下来定价一致性。","示例：GPT-5.4 的 507 次任务尝试（每个任务一次）成本为 $43.49，pass@1 为 65.36%，所以 $43.49 ÷ (507 × 0.6536) = 每次成功任务尝试 $0.131。","如果没有其他模型既不更贵又至少同样准确，则某模型处于前沿。三个模型符合条件；其他所有模型在至少一个方面被支配。","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/oGmXYrxoNPJfsQndxlcMD.png","图 4：每次成功任务尝试成本与 pass@1 的关系。圈出的点是帕累托成本前沿模型。","前沿线有三个步骤。GPT-5.6 Sol 的每次成功成本最低，为 $0.127；GPT-5.4 以每次成功多花 $0.004 的代价，将 pass@1 提高了 3.45 个百分点；Claude Opus 5.5 再增加 1.80 个百分点，每次成功成本 $0.276。这三个模型都保持在成本前沿线上，因为没有更便宜的模型能够匹配它们的 pass@1。","Claude Opus 5 是最明显的例子：每次成功尝试成本 $0.475，pass@1 为 66.50%，既比 Claude Opus 5.5 更贵（$0.276），又准确率更低（67.16%）。","每次成功成本奖励的是廉价且经常正确的模型。它不会奖励每次都正确的模型。因此我们还计算了每次可靠任务的成本：20 次完整运行活动的成本 ÷ 模型在所有 20 次尝试中通过的任务数。","每次可靠任务的成本 = 20 次 507 次尝试的预计成本 ÷ 20/20 完成的任务数","示例：GPT-6-Astra 的活动成本为 20 × $86.03 = $1,720.60，且在每次尝试中通过 231 个任务，因此每次可靠任务成本为 $1,720.60 ÷ 231 = $7.45。","表 3：在至少有一个 20/20 任务记录的模型中，每次可靠任务成本最低的九个模型，按从低到高排序。为估算值，以美元计，而非实际云账单。","现在按一致性排名。GPT-5.4 成本最低，为 $6.80，但只有 128 个任务达到标准。GPT-6 Astra 达到 231 个任务，成本 $7.45；Claude Opus 5.5 的最高记录为 241，成本 $7.80。","三者中没有模型完全占据优势：每增加一个可靠任务成本就会增加。Claude Opus 5 也通过 241 个任务，但成本为 $13.30，因此 Opus 5.5 完全占优。GPT-5.6 Sol，每次单次成功成本最低，为 $0.127，但每次可靠任务成本为 $9.76。获得正确答案的最便宜方式并不是获得可靠答案的最便宜方式。","我们为每个失败的追踪分配一个确定性的诊断特征，且结论具有可操作性：大约五分之四的失败是工具处理问题，而非推理问题。在我们论文表 5 的消融研究中：https://arxiv.org/pdf/2608.19741","这些是每个模型的份额和可观察标签的非加权平均值，而非独特的因果解释。","实际模式很简单：代理通常会尝试完成工作流程，但随后由于工具错误、未满足的前提条件或空查找而无法恢复。这是一个重试和错误恢复的问题，而不是模型本身的问题。","难度也会因领域而异：在上表2中列出的模型中，零售领域平均通过率为59.52%，而汽车保险领域平均为33.83%。","该如何处理。将20/20的通过率视为设计输入，而不是最终裁决。基准评分的相同信号在生产中也是可用的：在提交之前检查终端状态，而不是模型对它的总结。","将工具和系统错误分类，以便重试针对可恢复的错误。将工具界面限制到工作流程所需的部分。并且在无法廉价撤回的更改上要求人工审批。我们尚未测量这些在此基准上带来的提升，但这正是当前环境可以进行测试的类型。","ThinkingBox是代理沙箱，而ThinkingBox-Bench是用于评估代理的数据集基准。本文顶部的图示显示了循环；以下是各部分的作用。","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/elbRGqTXW6PIqci7xcEJN.png","图5：上文图1中面板A的沙箱循环：隔离的工具会话、终端数据库状态、副作用、可执行评判器。","每个任务定义了初始后端状态、用户目标、可用的MCP工具、领域策略，以及对终端状态的可执行检查。模拟用户持有私密上下文（预订参考号、偏好或出生日期），仅在被请求时释放。","每次尝试都会获得一个状态全新初始化的隔离MCP会话。同一任务的两次尝试从不共享数据库行或缓存的工具状态，这也正是20次试验比较具有意义的原因。"],"translationStatus":"translated","bodyOrigin":"source-page","editorial":{"summary":"Microsoft 与 Hugging Face 发布 ThinkingBox 智能体沙箱和 ThinkingBox-Bench 基准，覆盖 507 个有状态业务工作流，并要求每项任务运行 20 次，以最终数据库状态和副作用进行判定。","background":"来源介绍称，ThinkingBox 不仅检查工具调用，还检查智能体在隔离的 MCP 工具会话中留下的终态后端状态及副作用；该项目现可通过 OpenEnv 在 Hugging Face 上运行。","viewpoint":"Aioga 判断：将数据库终态与重复运行纳入评测，能把“调用工具”与“完成任务”区分开，但单一基准仍不足以代表所有真实业务场景中的可靠性。","implications":"可能影响：智能体评估可能更重视可验证结果和一致性，而非仅看文本或工具调用记录；这不代表 ThinkingBox 已覆盖全部业务风险，部署判断仍需要结合具体流程。","nextStep":"后续观察：建议关注 ThinkingBox-Bench 的任务覆盖、评分执行方式及实际运行情况，并核对不同智能体在终态正确性、重复表现和副作用控制上的差异。","evidenceRefs":["title","summary","articleBody","source"],"status":"published","aiGenerated":true,"autoApproved":true,"generatedBy":"aioga-editorial:gpt-5.6-sol","reviewedBy":"aioga-editorial-review:gpt-5.6-sol","generatedAt":"2026-10-04T02:29:43.723Z","sourceHash":"e9e3cdf7c08f07cb","review":{"approved":true,"groundedness":95,"clarity":93,"duplicationRisk":15,"blockingIssues":[],"notes":["“Aioga 判断”表述略显突兀，建议确认是否为有意保留的署名或改为“观点判断”。","“单一基准仍不足以代表所有真实业务场景中的可靠性”属于合理分析性判断，但并非来源原文直接陈述；当前已通过 viewpoint 字段明确其观点属性。"]},"validation":{"passed":true,"mode":"ai-auto","revisions":0,"checks":["schema","length","source-attribution","editorial-labels","inference-boundary","low-source-overlap","no-html","independent-ai-review"]}},"tags":["行业动态","Hugging Face：Blog"],"translations":{"zh-CN":{"title":"Microsoft ThinkingBox 在 Hugging Face 上发布，以数据库终态和 20 次重复评测智能体","summary":"Microsoft 与 Hugging Face 发布 ThinkingBox 智能体沙箱与 ThinkingBox-Bench 基准，覆盖 507 个有状态业务工作流、每任务运行 20 次，以终局数据库状态和副作用作可执行判定，现可通过 OpenEnv 在 Hugging Face 上运行。","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox 在 Hugging Face 上发布，以数据库终态和 20 次重复评测智能体 - Aioga AI资讯","description":"Microsoft 与 Hugging Face 发布 ThinkingBox 智能体沙箱与 ThinkingBox-Bench 基准，覆盖 507 个有状态业务工作流、每任务运行 20 次，以终局数据库状态和副作用作可执行判定，现可通过 OpenEnv 在 Hugging Face 上运行。","url":"https://www.aioga.com/news/gqh4yclcjmaci6uhur56580uh/","articleBody":["一个工具调用不是结果：#a-tool-call-is-not-an-outcome 一次成功不是可靠性：#one-success-is-not-reliability 你能依赖你代理背后的模型吗？：#can-you-depend-on-the-model-behind-your-agent 一致性要付出什么代价：#what-consistency-costs 帕累托成本前沿：#pareto-cost-frontier 现在定价一致性：#now-price-consistency 失败特征：#failure-signatures 它是如何工作的：#how-it-works 自己运行它：#run-it-yourself 在你开始之前：#before-you-start 安装：#install 启动 Typesense：#start-typesense 启动 MCP 服务器：#start-the-mcp-servers 启动 OpenEnv 服务器：#start-the-openenv-server 检查就绪情况：#check-readiness 给一集打分：#score-an-episode 下一步走向：#where-this-goes-next Microsoft ThinkingBox 根据 AI 代理留下的记录对其评分，而不是生成的句子，然后检查它们是否可以连续完成二十次。现在可以通过 Hugging Face 获取。","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/Dzuml_9K2lfRq4hZi17EY.png","图 1：ThinkingBox 让代理在独立的 MCP 工具会话中运行，然后对其留下的终端后端状态和副作用进行评分。来自我们的 ThinkingBox 论文：https://arxiv.org/abs/2608.19741。","一位客户来信。她的 745 美元厨房电器在纳什维尔的配送中心被卡在快递“例外”状态，超过预计交货日期十五天。","AI 代理仔细工作。九次工具调用：它拉取订单，检查追踪信息，查看她的客户档案，两次搜索退款政策，确认没有现有工单，创建一个工单，记录时间线，并正确阅读政策；她的账户分段确实不符合延迟交货赔偿。","然后它将工单关闭为已解决，并回复“由于您的问题已解决，我还能为您提供其他帮助吗？”","有两点错误。承运商异常仍然存在，因此所需的最终状态是挂起，等待解决。此外，客户从未真正得到她实际提出的问题的答案。","检查工具调用的 AI 评分器会看到九次格式良好的调用。检查代理是否写入数据库的评分器也会看到这一点。不同的是数据库给出的结果与预期不符。","ThinkingBox 衡量的就是这个差距。在 507 个有状态的业务工作流中，每个工作流在不同的 LLM 模型上运行 20 次，它根据终端后台状态和副作用对代理进行评分。本文介绍了我们的发现、一致性成本以及如何通过 OpenEnv 自行运行基准测试：https://github.com/huggingface/OpenEnv/tree/main/envs/thinkingbox_env。","你可以自己运行这个任务：上面的示例是从基准任务 sandbox_external_retail_group1.py:test_case_ST003_006 改编而来的：https://github.com/microsoft/thinkingbox-data/blob/thinkingbox-bench-v1.0/dataset/test_case/sandbox_external_retail/sandbox_external_retail_group1.py#L984-L1223，且失败的可执行检查仅涉及一个字段：工单状态为“已解决”，而所需的最终状态是“待处理”。完整的追踪记录见我们论文附录 D.4，案例 3：https://arxiv.org/pdf/2608.19741。","想在阅读结果前试试吗？跳到“自己运行”部分：#run-it-yourself。","最终响应和有效工具调用只是代理指标。一个代理可能听起来正确，但却留下错误的数值、修改了错误的记录或生成了额外的副作用。只有它留下的记录才能揭示问题的真相。","差距是显著的。在一个包含 121,680 次有效试验、覆盖 12 个 LLM 模型的常见集合消融实验中，有 79,853 次尝试未通过可执行检查。在这些失败中，67.24% 仍然正常结束，调用了会改变状态的工具，并报告无最终工具错误。然而，可执行检查仍在其中 77.61% 的案例中发现字段值错误，43.30% 的案例中出现意外额外副作用，25.36% 的案例中缺少必要副作用。这些状态检查结果存在重叠。","一个轨迹是一种声明。数据库状态是证据。重复是信任测试。","一个代理如果一次正确处理退款，但接下来的四次处理错误，就不能算作一个可用退款代理。因此，每个任务都会独立运行 20 次，每次从相同的干净后台开始，我们报告三种不同的数据：","表 1：我们报告的三组数据，以及每组数据回答的问题。","在本文博客中，我们使用 \"observed 20/20\" 作为字面计数，表示 507 个任务中有多少任务 20 次全满分。没有估计器，没有平滑处理。","从熟悉的视角开始。下表报告了 pass@1，即单次尝试得分估计，按领域分类。这个数字是大多数排行榜发布的数字，本身看起来像普通的能力排名。","表 2：ThinkingBox-Bench 按领域的 pass@1 (%)。每个模型在每个任务上都进行 20 次重复评估。加粗表示组内领头；下划线表示亚军。单次尝试得分估计的标准误见我们 ThinkingBox 论文的表 4：https://arxiv.org/abs/2608.19741。","Claude Opus 5.5 总体领先，达 67.16%，比 Claude Opus 5 高出约三分之一点。Kimi-K3 是最强的开源权重模型，接近 GPT-6-Astra 仅一分之差。领域同样很重要：Claude Opus 4.6 在零售领域得分 68.62%，但在汽车保险领域仅得 8.30%。","一次良好的运行表明模型能够完成工作，但并不能说明它是否会再次完成。因此，对每个任务运行 20 次，然后观察得分能保持多少。","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/DjaLNZKNeU--Xy2WFCvVX.png","图 2：每个模型的单次尝试得分在 20 次重复中保持的比例。","只有三款模型能保持大部分 pass@1 得分：GPT-6 Astra 保持 78% 的单次尝试率，Claude Opus 5.5 和 Claude Opus 5 各保持 71%。相反，GLM-5.1、Kimi-K2.6 和 DeepSeek-V4-Pro 各仅保持约 8%。","模型一次能够完成的与每次都能完成的之间的差距才是完整的故事。","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/NONd2WGm4vtGpsVgs-CIq.png","图 3：广度与一致性拉开了距离。显示了 18 个模型中的 12 个；为了可读性，pass@1 低于 33% 的 6 个模型被省略。","Kimi-K3 拥有我们测试的模型中最广的覆盖率。它至少成功完成了 93.89% 的基准测试：507 个任务中完成了 476 个。仅有 31 个任务完全失败，是该领域最低的数量。在零售工作流程中，它以 82.24% 的 pass@1 排名第一，领先于所有专有模型。","Kimi-K3 也是一致性较低的模型之一。在 507 个任务中，仅有 68 个任务（13.41%）在 20 次尝试中全部成功。","Claude Opus 5 扭转了这一点。它至少完成较少的任务（79.09%；106 个任务完全没完成），但在每次尝试中完成了基准测试的 47.53%。","更新的模型没有解决这个问题。Claude Opus 5.5 在每次尝试的平均得分上高于 Claude Opus 5，分别为 67.16% 对 66.50%，并且至少完成更多任务。它在所有 20 次尝试中通过的任务数量完全相同：241 个。半个百分点的总体准确率没有带来任何额外的可靠性。","如果你正在选择用于处理真实记录的模型，pass@20 不是应该关注的列。","能力比较通常止于分数。对于任何部署者来说，真正相关的问题是成功完成一项工作的成本。我们将其测量为每次成功任务尝试的成本。我们说“任务尝试”，因为每个基准任务都重复运行且每次尝试都会产生成本，所以 pass@1 是匹配质量的分母。","我们使用每个模型在完整的 507 × 20 测试中记录的 token 使用情况，并按 OpenRouter（https://openrouter.ai/）上提供的未折扣列表价进行定价，撤销促销折扣并且排除声明量化的端点。输入、输出和缓存费用都来自每个模型的一个提供端点。","然后我们将一次运行的成本除以成功尝试的次数：","每次成功任务尝试的成本 = 507 次尝试（每个任务各一次）的预计成本 ÷ (507 × pass@1)","这是一种比较效率指数，而不是发票，也不是处理一次生产请求的价格。它也只计算单次成功，而不是一致性。我们接下来定价一致性。","示例：GPT-5.4 的 507 次任务尝试（每个任务一次）成本为 $43.49，pass@1 为 65.36%，所以 $43.49 ÷ (507 × 0.6536) = 每次成功任务尝试 $0.131。","如果没有其他模型既不更贵又至少同样准确，则某模型处于前沿。三个模型符合条件；其他所有模型在至少一个方面被支配。","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/oGmXYrxoNPJfsQndxlcMD.png","图 4：每次成功任务尝试成本与 pass@1 的关系。圈出的点是帕累托成本前沿模型。","前沿线有三个步骤。GPT-5.6 Sol 的每次成功成本最低，为 $0.127；GPT-5.4 以每次成功多花 $0.004 的代价，将 pass@1 提高了 3.45 个百分点；Claude Opus 5.5 再增加 1.80 个百分点，每次成功成本 $0.276。这三个模型都保持在成本前沿线上，因为没有更便宜的模型能够匹配它们的 pass@1。","Claude Opus 5 是最明显的例子：每次成功尝试成本 $0.475，pass@1 为 66.50%，既比 Claude Opus 5.5 更贵（$0.276），又准确率更低（67.16%）。","每次成功成本奖励的是廉价且经常正确的模型。它不会奖励每次都正确的模型。因此我们还计算了每次可靠任务的成本：20 次完整运行活动的成本 ÷ 模型在所有 20 次尝试中通过的任务数。","每次可靠任务的成本 = 20 次 507 次尝试的预计成本 ÷ 20/20 完成的任务数","示例：GPT-6-Astra 的活动成本为 20 × $86.03 = $1,720.60，且在每次尝试中通过 231 个任务，因此每次可靠任务成本为 $1,720.60 ÷ 231 = $7.45。","表 3：在至少有一个 20/20 任务记录的模型中，每次可靠任务成本最低的九个模型，按从低到高排序。为估算值，以美元计，而非实际云账单。","现在按一致性排名。GPT-5.4 成本最低，为 $6.80，但只有 128 个任务达到标准。GPT-6 Astra 达到 231 个任务，成本 $7.45；Claude Opus 5.5 的最高记录为 241，成本 $7.80。","三者中没有模型完全占据优势：每增加一个可靠任务成本就会增加。Claude Opus 5 也通过 241 个任务，但成本为 $13.30，因此 Opus 5.5 完全占优。GPT-5.6 Sol，每次单次成功成本最低，为 $0.127，但每次可靠任务成本为 $9.76。获得正确答案的最便宜方式并不是获得可靠答案的最便宜方式。","我们为每个失败的追踪分配一个确定性的诊断特征，且结论具有可操作性：大约五分之四的失败是工具处理问题，而非推理问题。在我们论文表 5 的消融研究中：https://arxiv.org/pdf/2608.19741","这些是每个模型的份额和可观察标签的非加权平均值，而非独特的因果解释。","实际模式很简单：代理通常会尝试完成工作流程，但随后由于工具错误、未满足的前提条件或空查找而无法恢复。这是一个重试和错误恢复的问题，而不是模型本身的问题。","难度也会因领域而异：在上表2中列出的模型中，零售领域平均通过率为59.52%，而汽车保险领域平均为33.83%。","该如何处理。将20/20的通过率视为设计输入，而不是最终裁决。基准评分的相同信号在生产中也是可用的：在提交之前检查终端状态，而不是模型对它的总结。","将工具和系统错误分类，以便重试针对可恢复的错误。将工具界面限制到工作流程所需的部分。并且在无法廉价撤回的更改上要求人工审批。我们尚未测量这些在此基准上带来的提升，但这正是当前环境可以进行测试的类型。","ThinkingBox是代理沙箱，而ThinkingBox-Bench是用于评估代理的数据集基准。本文顶部的图示显示了循环；以下是各部分的作用。","：https://cdn-uploads.huggingface.co/production/uploads/64b8491203124195cd795cad/elbRGqTXW6PIqci7xcEJN.png","图5：上文图1中面板A的沙箱循环：隔离的工具会话、终端数据库状态、副作用、可执行评判器。","每个任务定义了初始后端状态、用户目标、可用的MCP工具、领域策略，以及对终端状态的可执行检查。模拟用户持有私密上下文（预订参考号、偏好或出生日期），仅在被请求时释放。","每次尝试都会获得一个状态全新初始化的隔离MCP会话。同一任务的两次尝试从不共享数据库行或缓存的工具状态，这也正是20次试验比较具有意义的原因。"]},"en":{"title":"Microsoft ThinkingBox released on Hugging Face, evaluating agents using database end states and 20 repeated evaluations.","summary":"Microsoft and Hugging Face have released the ThinkingBox agent sandbox and the ThinkingBox-Bench benchmark, covering 507 stateful business workflows. Each task is run 20 times, with executable evaluation based on the final database state and side effects. They can now be run on Hugging Face via OpenEnv.","category":"Industry","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox released on Hugging Face, evaluating agents using database end states and 20 repeated evaluations. - Aioga AI News","description":"Microsoft and Hugging Face have released the ThinkingBox agent sandbox and the ThinkingBox-Bench benchmark, covering 507 stateful business workflows. Each task is run 20 times, wit...","url":"https://www.aioga.com/en/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:05:27.972Z"},"ja":{"title":"Microsoft ThinkingBoxがHugging Faceで公開、データベースの最終状態と20回の反復でエージェントを評価","summary":"MicrosoftとHugging Faceは、ThinkingBoxエージェントサンドボックスとThinkingBox-Benchベンチマークを発表しました。507件のステートフルな業務ワークフローを対象とし、各タスクを20回実行して、最終的なデータベース状態と副作用に基づき実行可能な判定を行います。現在はOpenEnvを通じてHugging Face上で実行できます。","category":"業界動向","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBoxがHugging Faceで公開、データベースの最終状態と20回の反復でエージェントを評価 - Aioga AIニュース","description":"MicrosoftとHugging Faceは、ThinkingBoxエージェントサンドボックスとThinkingBox-Benchベンチマークを発表しました。507件のステートフルな業務ワークフローを対象とし、各タスクを20回実行して、最終的なデータベース状態と副作用に基づき実行可能な判定を行います。現在はOpenEnvを通じてHugging Face上で...","url":"https://www.aioga.com/ja/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:06:18.655Z"},"ko":{"title":"Microsoft ThinkingBox가 Hugging Face에 공개되었으며, 데이터베이스의 최종 상태와 20회 반복 평가를 통해 에이전트를 평가합니다.","summary":"Microsoft와 Hugging Face는 ThinkingBox 에이전트 샌드박스와 ThinkingBox-Bench 벤치마크를 공개했습니다. 이는 상태를 유지하는 507개의 비즈니스 워크플로를 포괄하며, 각 작업을 20회 실행하고 최종 데이터베이스 상태와 부작용을 기준으로 실행 가능한 판정을 내립니다. 이제 Hugging Face에서 OpenEnv를 통해 실행할 수 있습니다.","category":"업계 동향","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox가 Hugging Face에 공개되었으며, 데이터베이스의 최종 상태와 20회 반복 평가를 통해 에이전트를 평가합니다. - Aioga AI 뉴스","description":"Microsoft와 Hugging Face는 ThinkingBox 에이전트 샌드박스와 ThinkingBox-Bench 벤치마크를 공개했습니다. 이는 상태를 유지하는 507개의 비즈니스 워크플로를 포괄하며, 각 작업을 20회 실행하고 최종 데이터베이스 상태와 부작용을 기준으로 실행 가능한 판정을 내립니다. 이제 Huggin...","url":"https://www.aioga.com/ko/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:05:53.622Z"},"es":{"title":"Microsoft ThinkingBox se ha publicado en Hugging Face para evaluar agentes mediante estados finales de bases de datos y 20 evaluaciones repetidas.","summary":"Microsoft y Hugging Face lanzan el entorno aislado para agentes ThinkingBox y el benchmark ThinkingBox-Bench, que abarca 507 flujos de trabajo empresariales con estado y ejecuta cada tarea 20 veces. La evaluación ejecutable se basa en el estado final de la base de datos y en los efectos secundarios. Ya se puede ejecutar a través de OpenEnv en Hugging Face.","category":"Industria","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox se ha publicado en Hugging Face para evaluar agentes mediante estados finales de bases de datos y 20 evaluaciones repetidas. - Aioga Noticias de IA","description":"Microsoft y Hugging Face lanzan el entorno aislado para agentes ThinkingBox y el benchmark ThinkingBox-Bench, que abarca 507 flujos de trabajo empresariales con estado y ejecuta ca...","url":"https://www.aioga.com/es/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:10:46.852Z"},"fr":{"title":"Microsoft ThinkingBox est publié sur Hugging Face pour évaluer les agents à l’aide de l’état final de la base de données et de 20 répétitions.","summary":"Microsoft et Hugging Face lancent le bac à sable pour agents ThinkingBox et le benchmark ThinkingBox-Bench, couvrant 507 workflows métier avec état, avec 20 exécutions par tâche, et utilisant l’état final de la base de données ainsi que les effets secondaires pour une évaluation exécutable. Il est désormais possible de l’exécuter sur Hugging Face via OpenEnv.","category":"Industrie","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox est publié sur Hugging Face pour évaluer les agents à l’aide de l’état final de la base de données et de 20 répétitions. - Aioga Actualités IA","description":"Microsoft et Hugging Face lancent le bac à sable pour agents ThinkingBox et le benchmark ThinkingBox-Bench, couvrant 507 workflows métier avec état, avec 20 exécutions par tâche, e...","url":"https://www.aioga.com/fr/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:10:28.891Z"},"de":{"title":"Microsoft ThinkingBox auf Hugging Face veröffentlicht, mit der Evaluierung von Agenten anhand von Datenbank-Endzuständen und 20 Wiederholungen","summary":"Microsoft und Hugging Face veröffentlichen die Agenten-Sandbox ThinkingBox und den Benchmark ThinkingBox-Bench. Diese decken 507 zustandsbehaftete Geschäfts-Workflows ab; jede Aufgabe wird 20-mal ausgeführt, wobei der endgültige Datenbankzustand und die Nebenwirkungen als ausführbare Bewertungskriterien dienen. Ab sofort kann ThinkingBox über OpenEnv auf Hugging Face ausgeführt werden.","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox auf Hugging Face veröffentlicht, mit der Evaluierung von Agenten anhand von Datenbank-Endzuständen und 20 Wiederholungen - Aioga KI-News","description":"Microsoft und Hugging Face veröffentlichen die Agenten-Sandbox ThinkingBox und den Benchmark ThinkingBox-Bench. Diese decken 507 zustandsbehaftete Geschäfts-Workflows ab; jede Aufg...","url":"https://www.aioga.com/de/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:10:55.007Z"},"pt-BR":{"title":"Microsoft ThinkingBox é lançado no Hugging Face para avaliar agentes com base no estado final do banco de dados e em 20 repetições.","summary":"A Microsoft e a Hugging Face lançaram o sandbox de agentes ThinkingBox e o benchmark ThinkingBox-Bench, abrangendo 507 fluxos de trabalho empresariais com estado. Cada tarefa é executada 20 vezes, com a avaliação executável baseada no estado final do banco de dados e nos efeitos colaterais. Agora, é possível executá-los na Hugging Face por meio do OpenEnv.","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox é lançado no Hugging Face para avaliar agentes com base no estado final do banco de dados e em 20 repetições. - Aioga Notícias de IA","description":"A Microsoft e a Hugging Face lançaram o sandbox de agentes ThinkingBox e o benchmark ThinkingBox-Bench, abrangendo 507 fluxos de trabalho empresariais com estado. Cada tarefa é exe...","url":"https://www.aioga.com/pt-BR/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:14:45.658Z"},"ru":{"title":"Microsoft ThinkingBox опубликован на Hugging Face для оценки интеллектуальных агентов по конечному состоянию базы данных и с 20-кратным повторением.","summary":"Microsoft и Hugging Face представили песочницу для ИИ-агентов ThinkingBox и бенчмарк ThinkingBox-Bench. Они охватывают 507 бизнес-процессов с сохранением состояния; каждое задание запускается 20 раз, а оценка выполняется на основе конечного состояния базы данных и побочных эффектов. Теперь их можно запускать на Hugging Face через OpenEnv.","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox опубликован на Hugging Face для оценки интеллектуальных агентов по конечному состоянию базы данных и с 20-кратным повторением. - Aioga Новости ИИ","description":"Microsoft и Hugging Face представили песочницу для ИИ-агентов ThinkingBox и бенчмарк ThinkingBox-Bench. Они охватывают 507 бизнес-процессов с сохранением состояния; каждое задание...","url":"https://www.aioga.com/ru/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:14:50.944Z"},"ar":{"title":"أطلقت Microsoft ThinkingBox على Hugging Face لتقييم الوكلاء باستخدام الحالات النهائية لقواعد البيانات و20 تكرارًا للتقييم.","summary":"أعلنت Microsoft وHugging Face عن إطلاق بيئة الصندوق الرملي للوكلاء ThinkingBox ومعيار ThinkingBox-Bench، الذي يغطي 507 سير عمل تجاريًا ذا حالة، مع تشغيل كل مهمة 20 مرة، واستخدام حالة قاعدة البيانات النهائية والآثار الجانبية كمعايير قابلة للتنفيذ للتقييم، ويمكن تشغيله الآن على Hugging Face عبر OpenEnv.","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"أطلقت Microsoft ThinkingBox على Hugging Face لتقييم الوكلاء باستخدام الحالات النهائية لقواعد البيانات و20 تكرارًا للتقييم. - Aioga أخبار الذكاء الاصطناعي","description":"أعلنت Microsoft وHugging Face عن إطلاق بيئة الصندوق الرملي للوكلاء ThinkingBox ومعيار ThinkingBox-Bench، الذي يغطي 507 سير عمل تجاريًا ذا حالة، مع تشغيل كل مهمة 20 مرة، واستخدام حا...","url":"https://www.aioga.com/ar/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:14:29.607Z"},"hi":{"title":"Microsoft ThinkingBox को Hugging Face पर जारी किया गया, जिसमें डेटाबेस की अंतिम अवस्थाओं और 20 बार दोहराए गए मूल्यांकनों के आधार पर एजेंटों का मूल्यांकन किया जाता है।","summary":"Microsoft और Hugging Face ने ThinkingBox एजेंट सैंडबॉक्स और ThinkingBox-Bench बेंचमार्क जारी किए हैं, जो 507 स्टेटफुल व्यावसायिक वर्कफ़्लो को कवर करते हैं। प्रत्येक कार्य को 20 बार चलाया जाता है और अंतिम डेटाबेस स्थिति तथा साइड इफ़ेक्ट्स के आधार पर निष्पादन योग्य मूल्यांकन किया जाता है। अब इसे OpenEnv के माध्यम से Hugging Face पर चलाया जा सकता है।","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox को Hugging Face पर जारी किया गया, जिसमें डेटाबेस की अंतिम अवस्थाओं और 20 बार दोहराए गए मूल्यांकनों के आधार पर एजेंटों का मूल्यांकन किया जाता है। - Aioga AI समाचार","description":"Microsoft और Hugging Face ने ThinkingBox एजेंट सैंडबॉक्स और ThinkingBox-Bench बेंचमार्क जारी किए हैं, जो 507 स्टेटफुल व्यावसायिक वर्कफ़्लो को कवर करते हैं। प्रत्येक कार्य को 20 बार...","url":"https://www.aioga.com/hi/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:18:57.522Z"},"it":{"title":"Microsoft ThinkingBox è stato pubblicato su Hugging Face, con lo stato finale del database e 20 valutazioni ripetute dell’agente.","summary":"Microsoft e Hugging Face hanno rilasciato il sandbox per agenti ThinkingBox e il benchmark ThinkingBox-Bench, che coprono 507 flussi di lavoro aziendali con stato. Ogni attività viene eseguita 20 volte e valutata in modo eseguibile sulla base dello stato finale del database e degli effetti collaterali. Ora è possibile eseguirli su Hugging Face tramite OpenEnv.","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox è stato pubblicato su Hugging Face, con lo stato finale del database e 20 valutazioni ripetute dell’agente. - Aioga Notizie IA","description":"Microsoft e Hugging Face hanno rilasciato il sandbox per agenti ThinkingBox e il benchmark ThinkingBox-Bench, che coprono 507 flussi di lavoro aziendali con stato. Ogni attività vi...","url":"https://www.aioga.com/it/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:18:49.504Z"},"nl":{"title":"Microsoft ThinkingBox is uitgebracht op Hugging Face en evalueert agents aan de hand van de eindtoestanden van databases en 20 herhaalde evaluaties.","summary":"Microsoft en Hugging Face lanceren ThinkingBox, een sandbox voor agents, en de ThinkingBox-Bench-benchmark. Deze omvat 507 zakelijke workflows met een status, wordt 20 keer per taak uitgevoerd en beoordeelt prestaties uitvoerbaar op basis van de uiteindelijke databasestatus en neveneffecten. De benchmark kan nu via OpenEnv op Hugging Face worden uitgevoerd.","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox is uitgebracht op Hugging Face en evalueert agents aan de hand van de eindtoestanden van databases en 20 herhaalde evaluaties. - Aioga AI-nieuws","description":"Microsoft en Hugging Face lanceren ThinkingBox, een sandbox voor agents, en de ThinkingBox-Bench-benchmark. Deze omvat 507 zakelijke workflows met een status, wordt 20 keer per taa...","url":"https://www.aioga.com/nl/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:18:57.685Z"},"tr":{"title":"Microsoft ThinkingBox, Hugging Face üzerinde veritabanı final durumu ve 20 tekrar ile ajan değerlendirmesi için yayınlandı","summary":"Microsoft ve Hugging Face, 507 durum bilgili iş akışını kapsayan ThinkingBox ajan sanal alanını ve ThinkingBox-Bench karşılaştırmasını yayımladı. Her görev 20 kez çalıştırılıyor; değerlendirme, nihai veritabanı durumu ve yan etkiler temel alınarak yürütülebilir şekilde yapılıyor. Artık Hugging Face üzerinde OpenEnv aracılığıyla çalıştırılabilir.","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox, Hugging Face üzerinde veritabanı final durumu ve 20 tekrar ile ajan değerlendirmesi için yayınlandı - Aioga AI Haberleri","description":"Microsoft ve Hugging Face, 507 durum bilgili iş akışını kapsayan ThinkingBox ajan sanal alanını ve ThinkingBox-Bench karşılaştırmasını yayımladı. Her görev 20 kez çalıştırılıyor; d...","url":"https://www.aioga.com/tr/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:23:04.020Z"},"vi":{"title":"Microsoft ThinkingBox được phát hành trên Hugging Face, sử dụng trạng thái cuối của cơ sở dữ liệu và đánh giá tác nhân qua 20 lần lặp lại.","summary":"Microsoft và Hugging Face công bố môi trường sandbox dành cho tác nhân ThinkingBox cùng với benchmark ThinkingBox-Bench, bao phủ 507 quy trình nghiệp vụ có trạng thái; mỗi tác vụ được chạy 20 lần và được đánh giá có thể thực thi dựa trên trạng thái cuối cùng của cơ sở dữ liệu cùng các tác động phụ. Hiện có thể chạy thông qua OpenEnv trên Hugging Face.","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox được phát hành trên Hugging Face, sử dụng trạng thái cuối của cơ sở dữ liệu và đánh giá tác nhân qua 20 lần lặp lại. - Tin tức AI Aioga","description":"Microsoft và Hugging Face công bố môi trường sandbox dành cho tác nhân ThinkingBox cùng với benchmark ThinkingBox-Bench, bao phủ 507 quy trình nghiệp vụ có trạng thái; mỗi tác vụ đ...","url":"https://www.aioga.com/vi/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:22:54.201Z"},"id":{"title":"Microsoft ThinkingBox dirilis di Hugging Face, mengevaluasi agen menggunakan keadaan akhir basis data dan 20 kali pengulangan.","summary":"Microsoft dan Hugging Face merilis sandbox agen ThinkingBox dan benchmark ThinkingBox-Bench, yang mencakup 507 alur kerja bisnis stateful. Setiap tugas dijalankan 20 kali, dengan status akhir database dan efek samping digunakan sebagai penentuan yang dapat dieksekusi. Kini, benchmark ini dapat dijalankan di Hugging Face melalui OpenEnv.","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox dirilis di Hugging Face, mengevaluasi agen menggunakan keadaan akhir basis data dan 20 kali pengulangan. - Berita AI Aioga","description":"Microsoft dan Hugging Face merilis sandbox agen ThinkingBox dan benchmark ThinkingBox-Bench, yang mencakup 507 alur kerja bisnis stateful. Setiap tugas dijalankan 20 kali, dengan s...","url":"https://www.aioga.com/id/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:22:20.693Z"},"th":{"title":"Microsoft ThinkingBox เปิดตัวบน Hugging Face เพื่อประเมินเอเจนต์โดยใช้สถานะสุดท้ายของฐานข้อมูลและการประเมินซ้ำ 20 ครั้ง","summary":"Microsoft และ Hugging Face เปิดตัวแซนด์บ็อกซ์เอเจนต์ ThinkingBox และเบนช์มาร์ก ThinkingBox-Bench ซึ่งครอบคลุมเวิร์กโฟลว์ธุรกิจแบบมีสถานะ 507 รายการ โดยรันแต่ละงาน 20 ครั้ง และใช้สถานะฐานข้อมูลเมื่อสิ้นสุดงานกับผลข้างเคียงในการตัดสินผลที่ตรวจสอบได้ ขณะนี้สามารถรันได้ผ่าน OpenEnv บน Hugging Face","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox เปิดตัวบน Hugging Face เพื่อประเมินเอเจนต์โดยใช้สถานะสุดท้ายของฐานข้อมูลและการประเมินซ้ำ 20 ครั้ง - ข่าว AI Aioga","description":"Microsoft และ Hugging Face เปิดตัวแซนด์บ็อกซ์เอเจนต์ ThinkingBox และเบนช์มาร์ก ThinkingBox-Bench ซึ่งครอบคลุมเวิร์กโฟลว์ธุรกิจแบบมีสถานะ 507 รายการ โดยรันแต่ละงาน 20 ครั้ง และใช้สถ...","url":"https://www.aioga.com/th/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:27:38.704Z"},"pl":{"title":"Microsoft ThinkingBox opublikowany na Hugging Face, z oceną agentów na podstawie końcowych stanów bazy danych i 20 powtórzonych ewaluacji.","summary":"Microsoft i Hugging Face opublikowały ThinkingBox — piaskownicę dla agentów — oraz benchmark ThinkingBox-Bench, obejmujące 507 stanowych przepływów pracy biznesowej. Każde zadanie jest uruchamiane 20 razy, a o poprawności wykonania decydują końcowy stan bazy danych i skutki uboczne. Obecnie można je uruchamiać na Hugging Face za pośrednictwem OpenEnv.","category":"行业动态","source":"Hugging Face：Blog","aggregationSource":"Hugging Face：Blog","pageTitle":"Microsoft ThinkingBox opublikowany na Hugging Face, z oceną agentów na podstawie końcowych stanów bazy danych i 20 powtórzonych ewaluacji. - Aioga Wiadomości AI","description":"Microsoft i Hugging Face opublikowały ThinkingBox — piaskownicę dla agentów — oraz benchmark ThinkingBox-Bench, obejmujące 507 stanowych przepływów pracy biznesowej. Każde zadanie...","url":"https://www.aioga.com/pl/news/gqh4yclcjmaci6uhur56580uh/","contentTranslated":true,"translationStatus":"translated","translationRetryAt":"","translationError":"","sourceHash":"360b504361f80689","translatedAt":"2026-10-04T02:27:24.344Z"}},"evidenceTier":"verified-news","reviewStatus":"editorial-selected","indexable":true,"editorialCover":""}}