{"@context":"https://schema.org","@type":"NewsArticle","generatedAt":"2026-08-11T09:21:12.743Z","headline":"构建一个先进的代理式测试框架","description":"文章介绍如何构建一个先进的代理式测试框架（Agentic Harness），用于系统化评估和测试AI智能体。框架设计聚焦于可重复、可扩展的测试流程，以验证智能体在复杂任务中的表现。文中讨论了框架的核心架构、关键组件以及实际应用中的考量，为开发者提供了一套完整的测试解决方案参考。","url":"https://www.aioga.com/news/cmsgrdx2s0ehgro5qo5xp6ttm/","mainEntityOfPage":"https://www.aioga.com/news/cmsgrdx2s0ehgro5qo5xp6ttm/","datePublished":"2026-08-05T23:51:32.802Z","dateModified":"2026-08-05T23:51:32.802Z","inLanguage":"zh-CN","publisher":{"@type":"NewsMediaOrganization","name":"Aioga","url":"https://www.aioga.com"},"citation":["https://data4sci.com/blog/building-an-advanced-agentic-harness","https://aihot.virxact.com/items/cmsgrdx2s0ehgro5qo5xp6ttm"],"canonicalUrl":"https://www.aioga.com/news/cmsgrdx2s0ehgro5qo5xp6ttm/","directAnswer":{"@type":"Answer","text":"Aioga 编辑摘要：文章介绍如何构建一个先进的代理式测试框架（Agentic Harness），用于系统化评估和测试AI智能体。 Aioga 将其归入「技巧观点」方向，重点关注它对真实使用和行业竞争的影响。","url":"https://www.aioga.com/news/cmsgrdx2s0ehgro5qo5xp6ttm/","dateCreated":"2026-08-05T23:51:32.802Z","author":{"@type":"Organization","@id":"https://www.aioga.com/authors/aioga-editorial/#editorial-team","name":"Aioga Editorial Team","url":"https://www.aioga.com/authors/aioga-editorial/"}},"evidence":[{"@type":"CreativeWork","name":"data4sci.com source article","url":"https://data4sci.com/blog/building-an-advanced-agentic-harness","datePublished":"2026-08-05T23:51:32.802Z","provider":{"@type":"Organization","name":"data4sci.com","url":"https://data4sci.com/blog/building-an-advanced-agentic-harness"}},{"@type":"CreativeWork","name":"AIHot archive record","url":"https://aihot.virxact.com/items/cmsgrdx2s0ehgro5qo5xp6ttm","datePublished":"2026-08-05T23:51:32.802Z","provider":{"@type":"Organization","name":"AIHot","url":"https://aihot.virxact.com/items/cmsgrdx2s0ehgro5qo5xp6ttm"}}],"aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","originalPublisher":{"name":"data4sci.com","url":"https://data4sci.com/blog/building-an-advanced-agentic-harness"},"geoDeepAnswer":null,"article":{"id":"cmsgrdx2s0ehgro5qo5xp6ttm","slug":"cmsgrdx2s0ehgro5qo5xp6ttm","url":"https://www.aioga.com/news/cmsgrdx2s0ehgro5qo5xp6ttm/","title":"构建一个先进的代理式测试框架","title_en":"","summary":"文章介绍如何构建一个先进的代理式测试框架（Agentic Harness），用于系统化评估和测试AI智能体。框架设计聚焦于可重复、可扩展的测试流程，以验证智能体在复杂任务中的表现。文中讨论了框架的核心架构、关键组件以及实际应用中的考量，为开发者提供了一套完整的测试解决方案参考。","source":"Hacker News 热门（buzzing.cc 中文翻译）","sourceUrl":"https://data4sci.com/blog/building-an-advanced-agentic-harness","aiHotUrl":"https://aihot.virxact.com/items/cmsgrdx2s0ehgro5qo5xp6ttm","publishedAt":"2026-08-05T23:51:32.802Z","category":"技巧观点","score":25,"selected":false,"articleBody":["That Basic Harness loop is correct, but naive . A lone pilot in a well-built jet might win a dogfight, but nobody runs an air campaign that way. Real operations：https://amzn.to/4vqYj0o add mission planners who decide what sorties to fly before anyone takes off, squadrons that fly independent sorties in parallel, fuel budgets and bingo calls that force a return to base before the tanks run dry, flight recorders that make every mission reconstructible after the fact, and after-action reviews that decide whether the mission actually succeeded. None of these replace the pilot. They wrap the pilot in structure so that the whole system stays fast, safe, debuggable, and measurable.","Claude Code, Devin, Cursor, Hermes, and other production agents do exactly the same thing to the basic loop. In this post we upgrade every piece of our basic harness toward that production shape, without hiding any of the mechanics behind a framework. The guiding question for the whole exercise is a simple one:","How do you turn a single LLM call into a reliable system that can plan, act, recover, and prove it did the right thing?","Our answer is composition. We build small, testable primitives: typed tools, a plan DAG：https://en.wikipedia.org/wiki/Directed_acyclic_graph, tiered memory, a verification hierarchy, budgets, and a tracer, and wire them together with a deliberately thin orchestrator. Each primitive exists because naive agents fail in a specific, predictable way. LLMs invent invalid tool arguments, so we add typed tools with Pydantic：https://pydantic.dev/docs/validation/latest/get-started/ validation. Everything runs sequentially, so we add a dependency graph and parallel execution. The context window fills with junk, so we add multi-tier memory under a retrieval budget. Bad outputs propagate silently, so we add a verification hierarchy. One prompt tries to do everything, so we split it into Planner , Worker , and Critic roles. Costs run away, so we add multi-dimensional budgeting with graceful degradation.","Proving the harness usually works, with an eval suite, retrieval benchmarks, and specialized worker pools will get a full fledge treatment in a future post.","Throughout the post we build a city comparison agent: given a list of cities, it produces a report comparing them on population, timezone, and a short narrative summary of each. The task looks almost insultingly simple, but it was chosen carefully. Each city-attribute lookup is independent of every other one, which means a three-city request naturally decomposes into nine tool calls that could all run at the same time. The final report, on the other hand, depends on all of the lookups finishing first, so we’re. well beyond a flat list of steps. We can programmatically check that every requested city actually appears in the report to verify the results. And the tools have wildly different costs: population and timezone lookups are in-memory dictionary reads, while the per-city summaries and the final aggregation each call the LLM, which gives us realistic budget pressure to manage.","For the sake of reproducibility, lookup tools read from a small mocked dictionary, CITY_FACTS , so the notebook is fully reproducible without network access. The LLM-backed pieces can run against either a real Anthropic model or a deterministic mock, which brings us to the first primitive.","Every component we are about to build eventually calls an LLM: the planner, the summarizer, the aggregator, the critic. If that call is hard-wired to one SDK, the entire harness becomes untestable and vendor-locked.","So before anything else, we define a base class that provides an abstraction over the details of the various LLM calling APIs","We also implement a MockProvider for testing and debugging purposes that returns deterministic, role-aware responses: a canonical plan when asked to plan, a templated one-line summary when asked to summarize, a rule-based pass/fail verdict when asked to judge. This allows us to separate “is my orchestration wrong?” from “is the model planning badly?” during development, and it is the reason every experiment in this post is reproducible on any machine.","In the basic harness：https://data4sci.substack.com/p/building-a-basic-agentic-harness we validated tool arguments by hand, an approach collapses quickly: every new tool duplicates validation logic, the LLM never sees a formal schema and just guesses argument shapes, and the resulting errors are ad hoc strings the model can’t self-correct from.","The upgrade is to declare each tool’s arguments as a Pydantic model and let one definition drive everything:","This approach gets us runtime validation, a JSON Schema in exactly the shape that the Anthropic and OpenAI tool-use APIs expect, documentation (each Field( …, description =… ) becomes part of the catalog the planner reads), and a hook for cost accounting via cost_hint. Failing before execution allows us to avoid expensive tool calls with potential side effects. A bad plan should fail fast , at the validation layer, and not deep inside a database query. This approach is similar to what full fledge frameworks like LangChain tools, Anthropic tool use, and OpenAI function calling all converge on.","Our registry holds four tools with three cost tiers: get_population and get_timezone() are essentially free dictionary lookups ( cost_hint =0.1 ), summarize_city() makes one LLM call per city ( cost_hint =1.0 ), and aggregate_report() makes the token-heavy synthesis call that produces the final markdown ( cost_hint =2.0 ). Note that the last two are tools that call the LLM internally. LLMs are just like any other tool. The worker sees a uniform tool interface, but some tools are wrappers around sub-prompts, which means you can cache, rate-limit, or swap the inner model independently of the harness.","The basic harness executed one action per turn. That works when steps are strictly sequential, but our task has nine independent lookups feeding a single aggregation:","：https://substackcdn.com/image/fetch/$s_!bqrk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdda72665-ee70-4a2e-9194-eed00992aaad_1057x778.png","A while-loop runs these one at a time. A Directed Acyclic Graph：https://en.wikipedia.org/wiki/Directed_acyclic_graph expresses the dependencies explicitly and lets an executor run everything that is ready right now, concurrently. So instead of asking the LLM for one action at a time, we ask the Planner for the whole graph up front. The LLM declares the structure before we execute anything. Since the planner is an LLM, it can hallucinate structure too: dependencies on node IDs that don’t exist, or circular dependencies that can never complete. So the very first thing we do with a plan is to validate it before possibly wasting tokens trying to execute a broken plan.","ready_nodes() is the heart of the scheduler: at any moment, it returns the set of nodes whose dependencies are all satisfied. For our three-city goal, the planner emits ten nodes: nine fetches with empty dependency lists, all eligible to run in parallel, and one aggregate_report capstone that depends on all nine.","The executor is a level-synchronous DAG walker: compute the ready set, launch every ready node concurrently with asyncio.gather , mark each one done or failed, and repeat until nothing is left or no forward progress is possible.","Two small decisions carry most of the weight here. First, asyncio.to_thread runs our synchronous tool functions in a thread pool, which means we never have to rewrite tools as async def or couple the harness to async-native SDKs. Second, the semaphore caps concurrency, because without it a fifty-node plan would spawn fifty simultaneous LLM calls and promptly hit rate limits or a cost spike. This is deliberately not a full dynamic scheduler with work-stealing and priority queues. For agent workloads like ours, where each node is an API call lasting hundreds of milliseconds to seconds, level-synchronous parallelism captures most of the win. Sequentially, wall time is roughly the sum of the fetch latencies; in parallel, it is roughly the maximum of them plus the aggregation step.","Naive agents dump everything into the prompt: the full chat history, every tool output, every prior task. That fails twice over — you pay for tokens you’ll never use, and models measurably degrade when irrelevant text dilutes the goal. Production agents instead use tiered memory, loosely inspired by cognitive science. Working memory is the always-in-context scratchpad: the current goal, a plan summary, and the last few results. Episodic memory stores the outcomes of past runs and is retrieved when a past task looks similar to the current one. Semantic memory holds background facts, retrieved the same way but not tied to any particular run.","We never inject everything; we pull the top-k memories by similarity to the current goal and then assemble the context under a hard character budget:","Episodic memories get priority over semantic ones because past mistakes on similar tasks are usually more actionable than generic facts, and when the budget runs out, truncation is explicit rather than silent. Context should be actively assembled , not passively accumulated .","For the similarity function itself, the store supports two backends. Jaccard similarity：https://en.wikipedia.org/wiki/Jaccard_index costs nothing and is fine for teaching, but it fails on paraphrase: “famous landmarks in France” shares almost no words with “Paris is known for the Eiffel Tower.” Real sentence embeddings using 384 dimensional vectors generated by all-MiniLM-L6-v2：https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 map paraphrases to nearby vectors. Our MemoryStore tries embeddings first with Jaccard as a backup if the model isn’t available. Quantifying the effect of this upgrade requires a proper benchmark, which we’ll run in a future post.","Agents produce fluent, confident, and wrong output. Without verification, a report that silently dropped a city ships to the user, and regressions go unnoticed until a human happens to read the output. But not all checks cost the same, so we arrange them as a hierarchy: deterministic structural checks that are essentially free, and an LLM judge for subjective quality that costs real tokens. The rule is to always run the cheap tier first and only escalate survivors.","Feed this a deliberately incomplete report (say, Paris only, when three cities were requested) and it will fail at the deterministic tier with reason= ”Missing cities: [’Tokyo’, ‘New York’]”. Zero tokens were spent on judging, and the reason string is actionable enough that a replanner step (or a human) can see exactly what went wrong. This two-tier gate is a robust pattern behind most production eval pipelines: cheap filters first, expensive judges on survivors only. And just as important as the hierarchy is the separation of concerns behind it: the Worker produces and the Critic evaluates, so the generator is never grading its own homework.","A single prompt that plans, executes, summarizes, and self-critiques tends to confuse its objectives (planning constraints bleed into writing style) and is unable to isolate “the planning part” complicating both testing and swapping.","We split the work into narrow agents, each with a short system prompt and a single contract. The Planner receives the goal plus the tool schemas and returns DAG JSON, which we validate before running. The Worker receives the DAG and simply executes it. The Critic receives the goal and the finished report and returns a verdict. The Planner ’s system prompt has the live tool catalog spliced directly into it, so it can only reference tools that actually exist:","With a real model behind the planner, plans come back structurally valid but stylistically varied, and pinning down the one node the orchestrator needs to find later is much cheaper in the prompt than in cleanup code.","：https://substackcdn.com/image/fetch/$s_!5A9N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6b4fb9de-22ea-419c-a112-1d77ffbb4611_1131x1711.png","This mirrors what production systems like AutoGPT-style planners, SWE-agent workers, or LLM-as-judge evaluators actually do while staying minimal enough to read in one sitting. And because all three roles go through the same LLMProvider.complete( …, role =… ) interface, swapping the planner model or mocking the critic is a one-line change.","The basic harness had a single max_steps counter, which hides the real constraints: you can have steps left but no tokens left, be under the token budget but rate-limited on tool calls, or have a hung network call burn wall-clock time without incrementing any counter at all. BudgetMulti tracks tokens, tool calls, wall time, and estimated dollars simultaneously, and a run stops when any dimension is exhausted. Its most useful output is a single scalar:","Pressure is the maximum utilization across all dimensions. You are limited by whichever resource runs out first, exactly like real billing. That one number drives graceful degradation: below 0.7 the full pipeline runs, including the LLM judge; above 0.9 the orchestrator skips the expensive critic and falls back to deterministic checks only; at 1.0 the run halts with partial results. Production agents use the same kind of signal to switch to cheaper models, reduce retrieval depth, or ask the user for confirmation.","The other half of operational sanity is recognizing that not every error deserves the same response. A rate limit or timeout is transient with exponential back off (with jitter, so a fleet of agents doesn’t retry in lockstep) and retry. A validation error is tool misuse and we feed the structured error back to the LLM so it can correct its own arguments. An unknown entity is missing information so retrying is actively harmful, because retrying a hallucinated city name will fail identically every time; the right move is to re-plan without it. And a policy violation is fatal , we should halt immediately. A small classify_error() function maps error strings into these four classes, and the recovery policy follows from the class rather than from blind retries.","When an agent fails, we need an append-only log of structured events that can answer what happened in what order, how long each step took, which role consumed the tokens, and whether budget pressure was rising before things went wrong. Every event in our Tracer captures identity (a step ID and a parent ID linking worker steps back to the plan that spawned them), semantics (the role and the action taken), economics (latency, tokens, cost, and a snapshot of budget pressure at write time), and, for critic events, the verdict. The schema is flat and boring: a list of JSON-serializable dicts you can dump to a file, ship to OpenTelemetry or LangSmith, or plot directly in matplotlib. You don’t need a proprietary format to get real observability, just a sufficient schema.","Per-step latency, colored by role, shows immediately that the summarize_city() and aggregate_report() nodes dominate wall time while the lookups are flat. LLM calls are where the seconds go, which is exactly why running them in parallel matters.","：https://substackcdn.com/image/fetch/$s_!xL9O!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc684dfe-280c-4e3a-aaf6-bbb28f58f366_5844x2844.pngPer-step latency by role","Budget pressure over time rises monotonically with a sharp jump at the aggregator, and if it crosses the 0.9 degradation threshold before the critic runs, the trace itself explains why the LLM judge was skipped.","：https://substackcdn.com/image/fetch/$s_!7_R8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff1bb4ea-8569-4e51-a81d-fc411115565f_3684x2484.png","Finally, tokens by role shows whether planning or execution is consuming the budget.This task it skews heavily toward the worker, with its many summarize calls.","：https://substackcdn.com/image/fetch/$s_!5U5q!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ef2844d-5bb2-4053-9dc7-9b1c2cbc2a6f_3684x2484.pngToken usage by role. Worker dominates on this task.","With the primitives in place, the Orchestrator becomes almost boring. It builds context from memory, asks the Planner for a DAG, hands the DAG to the Worker with a callback that records a trace event and charges the budget on every tool call, checks for failures, verifies the result with pressure-aware degradation, stores the outcome in episodic memory for future runs, and returns a RunResult bundling the report, verdict, DAG, trace, and budget. The only new behavior it adds is the re-planning loop. If execution fails and the error classifies as missing information, it sends the failure context back to the Planner instead of retrying blindly, up to a configurable max_replans .","The heart of the method fits in a few lines:","The mock planner always names the capstone node aggregate , and an early version of the orchestrator simply looked it up by that id. Switch to a real model and that assumption quietly breaks. The planner often mirrors the tool name aggregate_report or invents its own id, producing a plan that is structurally valid but not what the mock trained us to expect. The fix is twofold: the DAG now resolves the capstone by tool rather than by id dag.aggregate_node() returns the unique aggregate_report node, (and refuses plans that contain more than one), and the planner prompt gained the explicit capstone instructions we saw earlier. Never take the LLM at its word, not even about node names.","Between the Basic Harness：https://data4sci.substack.com/p/building-a-basic-agentic-harness and this Advanced one, we covered practically all the main concepts and ideas you need to build a successful custom harness. We took the ~35-line loop from the previous post and layered on seven production-shaped primitives: typed tools that give validation and LLM introspection from one schema, a DAG executor that buys parallelism without rewriting a single tool, tiered memory that assembles relevant context under a hard budget, a verification hierarchy that spends tokens only on outputs that survive the cheap checks, narrow Planner/Worker/Critic roles that can be tested and swapped independently, multi-dimensional budgets that degrade gracefully instead of crashing, and a tracer that turns runs into comparable experiments instead of anecdotes.","None of these pieces requires the others, but they compose. Add a tool to the registry and the planner automatically sees its schema. Tighten verify_report() and every future run is held to the new bar. Swap the entire LLM backend and the harness reruns unchanged. Composability is the difference between a PoC and an extensible harness, and is why the orchestrator stayed thin.","A few caveats are in order: memory here is in-process, while production systems persist embeddings to Chroma, Weaviate, or pgvector; tool outputs are trusted as instructions, where production systems must sandbox them as data against prompt injection; irreversible actions should require human approval; and our token accounting estimates from character counts where real systems read usage metadata from the SDK. Each of those is one more compositional layer away — which is precisely the point.","There is also one deliberate omission. A single successful demo proves the harness can work; nothing in this post proves it actually works in most cases. That is the job of an eval harness, and it is where we pick up in a future follow up post"],"articleImages":[{"sourceUrl":"https://substackcdn.com/image/fetch/$s_!bqrk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdda72665-ee70-4a2e-9194-eed00992aaad_1057x778.png","alt":"","afterParagraph":14,"url":"/media/articles/cmsgrdx2s0ehgro5qo5xp6ttm/a8b05feca78054f5.png"},{"sourceUrl":"https://substackcdn.com/image/fetch/$s_!5A9N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6b4fb9de-22ea-419c-a112-1d77ffbb4611_1131x1711.png","alt":"","afterParagraph":28,"url":"/media/articles/cmsgrdx2s0ehgro5qo5xp6ttm/572a908463a36a3d.png"},{"sourceUrl":"https://substackcdn.com/image/fetch/$s_!xL9O!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc684dfe-280c-4e3a-aaf6-bbb28f58f366_5844x2844.png","alt":"","afterParagraph":35,"url":"/media/articles/cmsgrdx2s0ehgro5qo5xp6ttm/4215fca4bb153d49.jpg"},{"sourceUrl":"https://substackcdn.com/image/fetch/$s_!7_R8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff1bb4ea-8569-4e51-a81d-fc411115565f_3684x2484.png","alt":"","afterParagraph":37,"url":"/media/articles/cmsgrdx2s0ehgro5qo5xp6ttm/c7db010a36923128.jpg"},{"sourceUrl":"https://substackcdn.com/image/fetch/$s_!5U5q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ef2844d-5bb2-4053-9dc7-9b1c2cbc2a6f_3684x2484.png","alt":"","afterParagraph":39,"url":"/media/articles/cmsgrdx2s0ehgro5qo5xp6ttm/c5e405c26cdce430.png"}],"mediaStatus":"ok","articleBodyZh":["那个基础安全带循环是正确的，但太天真了。一个孤身驾驶坚固喷气机的飞行员或许能赢得空战，但没人会用这种方式进行空中战役。真实行动：https：//amzn.to/4vqYj0o 新增任务规划员，负责决定起飞前执行哪些出击，中队独立执行独立出击，燃料预算和呼叫，迫使燃料耗尽前返回基地，飞行记录员让每个任务事后可重建，以及事后审查，决定任务是否成功。这些都不能取代试播集。他们将试点包裹在结构中，确保整个系统保持快速、安全、可调试和可衡量。","Claude Code、Devin、Cursor、Hermes 以及其他生产代理对基本循环做的事情完全相同。在这篇文章中，我们将将所有基本线束部件升级到那个生产形状，同时不隐藏任何机制背后。整个活动的核心问题很简单：","如何将一个单一的LLM调用变成一个可靠的系统，能够规划、行动、恢复并证明自己做了正确的事？","我们的答案是作曲。我们构建了小型、可测试的原语：类型工具、计划DAG：https：//en.wikipedia.org/wiki/Directed_acyclic_graph、分层内存、验证层级、预算和追踪器，并用刻意设计的编排器将其连接起来。每个原元素的存在，是因为天真的智能体以特定且可预测的方式失败。LLM会发明无效的工具参数，所以我们用Pydantic添加类型工具：https：//pydantic.dev/docs/validation/latest/get-started/ validation。所有程序都是顺序运行的，所以我们添加了依赖图和并行执行。上下文窗口会被垃圾填充，所以我们在有限的检索预算下添加了多层内存。坏输出会悄无声息地传播，所以我们会添加验证层级。一个提示试图完成所有事情，所以我们把它分成了规划者、工作者和批评者三个角色。成本飞涨，因此我们采用多维度预算并优雅地递减。","验证这种背带通常有效，包括评估套件、取样基准和专业工人池，将在后续文章中全面处理。","在整篇文章中，我们构建了一个城市比较代理：给定一个城市列表，它会生成一份报告，比较这些城市的人口、时区以及每个城市的简短叙述性总结。这个任务看起来几乎简单到令人发笑，但它是经过仔细挑选的。每个城市-属性查询都是独立的，这意味着一个三城市的请求自然会分解成九个可以同时运行的工具调用。另一方面，最终报告依赖于所有查询结果先完成，因此我们已经远远超出了一个简单步骤列表。我们可以通过程序检查，确保每个请求的城市确实出现在报告中，以验证结果。而且各工具的成本差异巨大：人口和时区查询是内存字典读取，而每个城市的摘要和最终聚合都调用了大语言模型，这给我们带来了现实的预算管理压力。","为了可重现性，查询工具从一个小型模拟字典 CITY_FACTS 中读取，因此该笔记本在没有网络访问的情况下也能完全复现。基于大语言模型的部分可以对真实的 Anthropic 模型或确定性模拟进行运行，这引出了第一个基本原语。","我们即将构建的每个组件最终都要调用大语言模型：计划器、摘要器、聚合器、评论者。如果这个调用被硬性绑定到一个 SDK，那么整个框架就无法测试，并且会被供应商锁定。","因此，在进行其他操作之前，我们定义了一个基类，对各种大语言模型调用 API 的细节提供抽象。","我们还实现了一个 MockProvider，用于测试和调试，其返回确定性的、角色感知的响应：在被要求制定计划时返回标准计划，在被要求总结时返回模板化的一行摘要，在被要求判断时返回基于规则的通过/失败判定。这使我们在开发过程中能够区分“我的编排错了吗？”与“模型计划得不好吗？”，这也是本文中每个实验能够在任何机器上复现的原因。","在基础框架中：https://data4sci.substack.com/p/building-a-basic-agentic-harness 我们手动验证了工具参数，这种方法迅速失效：每增加一个新工具就重复验证逻辑，LLM 从未看到正式的模式，只能猜测参数形状，产生的错误是模型无法自我纠正的临时字符串。","升级的方法是将每个工具的参数声明为 Pydantic 模型，并让一个定义驱动一切：","这种方法为我们提供了运行时验证、符合 Anthropic 和 OpenAI 工具使用 API 所期望形状的 JSON Schema、文档（每个 Field( …, description =… ) 成为计划器读取的目录的一部分）以及通过 cost_hint 进行成本核算的钩子。在执行前失败可以避免具有潜在副作用的昂贵工具调用。一个不好的计划应在验证层快速失败，而不是在数据库查询深处失败。这种方法类似于像 LangChain 工具、Anthropic 工具使用和 OpenAI 函数调用等成熟框架的做法。","我们的注册表包含四个工具，分为三个成本等级：get_population 和 get_timezone() 基本上是免费的字典查找（cost_hint =0.1），summarize_city() 每个城市调用一次 LLM（cost_hint =1.0），aggregate_report() 做一次产生最终 markdown 的大量 token 合成调用（cost_hint =2.0）。注意最后两个是内部调用 LLM 的工具。LLM 就像其他任何工具一样。工作器看到统一的工具接口，但有些工具是子提示的包装器，这意味着你可以独立于框架缓存、限速或替换内部模型。","基础框架每轮执行一个操作。当步骤严格顺序执行时这有效，但我们的任务有九个独立查找汇总成一个聚合：","：https://substackcdn.com/image/fetch/$s_!bqrk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdda72665-ee70-4a2e-9194-eed00992aaad_1057x778.png","while循环一次运行这些任务。一个有向无环图（DAG）：https://en.wikipedia.org/wiki/Directed_acyclic_graph 明确表达了依赖关系，并允许执行器并发运行当前准备好的所有任务。所以我们不是一次向LLM请求一个操作，而是一次性向规划器请求整个图。LLM在执行任何操作之前声明结构。由于规划器本身是LLM，它也可能产生虚构结构：依赖不存在的节点ID，或者产生永远无法完成的循环依赖。因此，我们对计划进行的第一件事就是在尝试执行损坏的计划浪费令牌之前先验证它。","ready_nodes() 是调度器的核心：在任何时刻，它返回所有依赖都已满足的节点集合。对于我们的三城市目标，规划器输出十个节点：九个依赖列表为空的 fetch 节点，全部可以并行运行，还有一个依赖所有九个节点的 aggregate_report capstone 节点。","执行器是一个层次同步的 DAG 遍历器：计算 ready 集合，使用 asyncio.gather 并发启动每个就绪节点，标记每个节点为完成或失败，然后重复直到没有剩余节点或无法进一步前进。","两项小决策在这里起到最大作用。首先，asyncio.to_thread 在线程池中运行我们的同步工具函数，这意味着我们不必将工具重写为 async def，也不必将加载器与异步原生 SDK 耦合。其次，信号量限制并发，因为没有它，一个五十节点的计划会同时生成五十次 LLM 调用，并立刻触发速率限制或成本激增。这显然不是完整的动态调度器，没有工作窃取和优先队列。对于像我们这样的代理工作负载，每个节点都是持续数百毫秒到几秒的 API 调用，层次同步并行可以获得大部分收益。顺序执行时，实际耗时大约是 fetch 延迟的总和；并行执行时，实际耗时大约是最大延迟加上聚合步骤。","天真的代理会把所有内容都扔进提示中：完整的聊天记录、每个工具输出、每个先前的任务。这种做法失败的原因有两个——你会为永远不会使用的令牌付费，而且当无关文本稀释目标时，模型的性能会明显下降。生产环境中的代理则使用分层记忆，灵感部分来自认知科学。工作记忆是始终在上下文中的便笺：当前目标、计划摘要以及最近的几个结果。情节记忆存储过去任务的结果，当过去的任务与当前任务相似时会被调用。语义记忆保存背景事实，也通过相同方式调用，但不与任何特定运行绑定。","我们从不注入所有内容；我们会根据与当前目标的相似度提取 top-k 记忆，然后在严格的字符预算下组装上下文：","情节记忆优先于语义记忆，因为在类似任务上的过去错误通常比通用事实更具可操作性，并且当预算耗尽时，截断是明确的而非默默进行。上下文应主动组装，而不是被动累积。","对于相似度函数本身，存储支持两种后端。Jaccard 相似度：https://en.wikipedia.org/wiki/Jaccard_index 不产生成本，非常适合教学，但在处理同义改写时会失败：“法国著名地标”几乎没有与“巴黎以埃菲尔铁塔闻名”共享单词。通过 all-MiniLM-L6-v2：https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 生成的 384 维向量的真实句子嵌入可以将同义改写映射到邻近向量。我们的 MemoryStore 首先尝试使用嵌入，如果模型不可用，则备用使用 Jaccard。量化此升级效果需要一个合适的基准测试，我们将在未来的文章中进行。","代理会生成流畅、自信但错误的输出。如果不进行验证，默默丢掉一个城市的报告会直接发送给用户，并且回归问题会一直被忽略，直到有人偶然查看输出为止。但并非所有检查的成本都相同，因此我们将它们排列成层级：确定性的结构检查基本免费，而用于主观质量的 LLM 判定则需要真实令牌。规则是始终先运行低成本层，只有通过此层的内容才会升级到下一层。","给它一个故意不完整的报告（比如只给巴黎，而实际上要求的是三个城市），它将在确定性层面失败，原因是“缺失城市：[‘东京’, ‘纽约’]”。在判断上没有消耗任何令牌，并且原因字符串足够可操作，一个重规划步骤（或人工）可以确切看到出了什么问题。这种两层门控是大多数生产评估管道背后的稳健模式：先进行低成本过滤，然后只对幸存者执行高成本评判。和层级结构一样重要的，是背后的关注点分离：Worker（执行者）生成，Critic（评审者）评价，因此生成器永远不会评自己的作业。","一个同时计划、执行、总结并自我批评的单一提示往往会使目标混淆（计划约束会影响写作风格），且无法隔离“规划部分”，这会使测试和替换更加复杂。","我们将工作拆分成狭窄的代理，每个代理都有简短的系统提示和单一的合同。Planner（规划者）接收目标和工具模式并返回DAG JSON，我们在运行前对其进行验证。Worker（执行者）接收DAG并简单执行。Critic（评审者）接收目标和完成的报告并返回判决。Planner的系统提示中直接嵌入了实时工具目录，因此它只能引用实际存在的工具：","在真正的模型支持下，规划器返回的计划在结构上有效但风格多样，并且在提示中锁定协调者以后需要找到的单个节点，比在后处理代码中完成要便宜得多。","：https://substackcdn.com/image/fetch/$s_!5A9N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6b4fb9de-22ea-419c-a112-1d77ffbb4611_1131x1711.png","这与生产系统如AutoGPT风格的规划器、SWE-agent执行者或LLM作为评判者的评估做法相似，同时仍保持精简到可一口气阅读的程度。而且因为所有三个角色都通过相同的LLMProvider.complete( …, role =… )接口，所以更换规划器模型或模拟评审者只需一行代码即可完成。","基本的 harness 只有一个 max_steps 计数器，这掩盖了真实的约束条件：你可能还有剩余步骤但没有剩余的 tokens，或者在 token 预算范围内但工具调用受到速率限制，或者网络挂起调用消耗了实际时间却完全不增加任何计数器。BudgetMulti 同时跟踪 tokens、工具调用、墙钟时间和预估费用，而运行会在任一维度耗尽时停止。它最有用的输出是一个标量：","Pressure 是所有维度中最大利用率。你会受到首先耗尽的资源限制，就像真实计费一样。这个数字驱动了优雅降级：低于 0.7 时，全流程运行，包括 LLM 评判；高于 0.9 时，调度程序会跳过昂贵的评论程序，仅回落到确定性检查；达到 1.0 时，运行会在部分结果下停止。生产环境的代理使用相同的信号来切换到更便宜的模型，减少检索深度，或向用户请求确认。","操作理智的另一半是认识到不是每个错误都值得同样处理。速率限制或超时是短暂的，可采用指数回退（带抖动，这样一组代理不会同步重试）并重试。验证错误是工具误用，我们将结构化错误反馈给 LLM，使其可以纠正自己的论证。未知实体是信息缺失，所以重试实际上有害，因为重试一个捏造的城市名字每次都会失败；正确做法是重新规划而不使用该信息。策略违规是致命的，我们应立即停止。一个小型 classify_error() 函数将错误字符串映射到这四类中，恢复策略依据类别而非盲目重试。","当一个代理失败时，我们需要一个仅追加的结构化事件日志，它可以回答事情发生的顺序、每个步骤花费的时间、哪个角色消耗了令牌，以及在事情出错之前预算压力是否在上升。我们的 Tracer 中的每个事件都捕捉了身份信息（步骤 ID 和一个将工作步骤链接回生成它们的计划的父 ID）、语义信息（角色和所采取的动作）、经济信息（延迟、令牌、成本以及写入时的预算压力快照），对于评论事件还包括判决。模式是平的且简单无趣：一列可以序列化为 JSON 的字典，你可以将它们写入文件、发送到 OpenTelemetry 或 LangSmith，或者直接用 matplotlib 绘制。你不需要专有格式就能获得真实的可观测性，只需要足够的模式。","按步骤延迟，并按角色着色，能立刻显示 summarize_city() 和 aggregate_report() 节点占据了大多数墙钟时间，而查找操作延迟平缓。LLM 调用才是消耗秒数的地方，因此并行运行它们非常重要。","按角色的每步延迟","随着时间的推移，预算压力单调上升，在聚合器处急剧跳升，如果它在评论执行前超过 0.9 的降级阈值，追踪日志本身就能解释为什么 LLM 审判被跳过。","：https://substackcdn.com/image/fetch/$s_!7_R8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff1bb4ea-8569-4e51-a81d-fc411115565f_3684x2484.png","最后，按角色分配的令牌显示计划或执行是否在消耗预算。在这项任务中，它严重倾向于工作者，因为其调用了许多 summarize。","：https://substackcdn.com/image/fetch/$s_!5U5q!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ef2844d-5bb2-4053-9dc7-9b1c2cbc2a6f_3684x2484.png 按角色的令牌使用情况。工作者在此任务中占主导地位。","有了这些原语之后，协调器（Orchestrator）几乎变得无聊。它从记忆中构建上下文，向规划器（Planner）请求一个有向无环图（DAG），将 DAG 交给工作器（Worker）并附上一个在每次工具调用时记录跟踪事件并计入预算的回调，检查失败情况，使用压力感知降级验证结果，将结果存储在情节记忆中以备将来运行，并返回一个 RunResult，捆绑报告、裁决、DAG、跟踪和预算。它唯一新增的行为是重新规划循环。如果执行失败且错误被分类为信息缺失，它会将失败上下文发送回规划器，而不是盲目重试，最多可配置为 max_replans。","该方法的核心可以用几行代码概括：","模拟规划器（mock planner）总是将顶点节点命名为 aggregate，而协调器的早期版本只是通过该 id 查找它。切换到真实模型后，这个假设悄然失效。规划器经常镜像工具名称 aggregate_report 或发明自己的 id，生成一个结构上有效但不是模拟训练时我们预期的计划。解决方法有两个方面：DAG 现在通过工具而非 id 解析顶点节点 dag.aggregate_node() 返回唯一的 aggregate_report 节点，（并拒绝包含多个节点的计划），规划器提示也增加了我们之前看到的明确顶点指令。永远不要完全相信 LLM，包括节点名称。","在基础框架（Basic Harness：https://data4sci.substack.com/p/building-a-basic-agentic-harness）与这个高级框架之间，我们几乎涵盖了构建一个成功自定义框架所需的所有主要概念和思想。我们从前一篇文章中的约 35 行循环出发，并叠加了七个生产级原语：根据一种模式提供验证和 LLM 内省的类型化工具（typed tools）、无需重写任何工具即可实现并行的 DAG 执行器、在严格预算下组装相关上下文的分层记忆、只在通过廉价检查的输出上花费 token 的验证层次结构、可以独立测试和替换的狭窄规划器/工作器/批评者角色、多维预算可以优雅降级而非崩溃，以及将运行转化为可比较实验而非轶事的跟踪器。","这些组件彼此不依赖，但可以组合在一起。向注册表添加一个工具，规划器会自动看到它的模式。收紧 verify_report() 后，每一次未来的运行都必须达到新的标准。更换整个 LLM 后端，测试平台会在不改变的情况下重新运行。可组合性是概念验证与可扩展测试平台的区别所在，也是编排器保持轻量的原因。","有几个注意事项：这里的内存是在进程内，而生产系统会将嵌入持久化到 Chroma、Weaviate 或 pgvector；工具输出被信任为指令，而生产系统必须将其作为数据进行沙箱处理以防提示注入；不可逆操作应需要人工批准；我们的令牌计算是根据字符数估算的，而实际系统从 SDK 读取使用元数据。每一项都是进一步的一层可组合性——这正是重点所在。","还有一个刻意省略的内容。一次成功的演示证明测试平台可以工作；本文没有证明它在大多数情况下确实可行。这是评估测试平台的工作，也是我们将在未来的后续文章中展开的内容。"],"translationStatus":"translated","bodyOrigin":"source-page","editorial":{"summary":"Aioga 编辑摘要：文章介绍如何构建一个先进的代理式测试框架（Agentic Harness），用于系统化评估和测试AI智能体。 Aioga 将其归入「技巧观点」方向，重点关注它对真实使用和行业竞争的影响。","background":"背景分析：实践类内容的价值在于是否能被复现、是否有明确边界，以及它能否转化为稳定的开发或工作流方法。","viewpoint":"Aioga 判断：这条动态更适合作为行业观察信号，当前信息足以建立线索，但不足以推导长期结论。","implications":"影响分析：对相关团队而言，短期应先核对来源、可用范围和实际成本，再判断是否值得接入或跟进。","nextStep":"后续观察：继续观察示例是否可复现、工具版本变化、社区反馈和实际成本。","evidenceRefs":["title","summary","articleBody"],"confidence":"medium","status":"published","aiGenerated":false,"autoApproved":true,"generatedBy":"rule-safe-fallback","generatedAt":"2026-08-11T09:23:27.651Z","sourceHash":"a9d3fa20a5df0cb9","validation":{"passed":true,"mode":"rule-safe-fallback","checks":["schema","length","source-attribution","no-html"]}},"tags":["技巧观点","Hacker News 热门（buzzing.cc 中文翻译）"],"translations":{"zh-CN":{"title":"构建一个先进的代理式测试框架","summary":"文章介绍如何构建一个先进的代理式测试框架（Agentic Harness），用于系统化评估和测试AI智能体。框架设计聚焦于可重复、可扩展的测试流程，以验证智能体在复杂任务中的表现。文中讨论了框架的核心架构、关键组件以及实际应用中的考量，为开发者提供了一套完整的测试解决方案参考。","category":"技巧观点","source":"data4sci.com","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"构建一个先进的代理式测试框架 - Aioga AI资讯","description":"文章介绍如何构建一个先进的代理式测试框架（Agentic Harness），用于系统化评估和测试AI智能体。框架设计聚焦于可重复、可扩展的测试流程，以验证智能体在复杂任务中的表现。文中讨论了框架的核心架构、关键组件以及实际应用中的考量，为开发者提供了一套完整的测试解决方案参考。","url":"https://www.aioga.com/news/cmsgrdx2s0ehgro5qo5xp6ttm/","articleBody":["那个基础安全带循环是正确的，但太天真了。一个孤身驾驶坚固喷气机的飞行员或许能赢得空战，但没人会用这种方式进行空中战役。真实行动：https：//amzn.to/4vqYj0o 新增任务规划员，负责决定起飞前执行哪些出击，中队独立执行独立出击，燃料预算和呼叫，迫使燃料耗尽前返回基地，飞行记录员让每个任务事后可重建，以及事后审查，决定任务是否成功。这些都不能取代试播集。他们将试点包裹在结构中，确保整个系统保持快速、安全、可调试和可衡量。","Claude Code、Devin、Cursor、Hermes 以及其他生产代理对基本循环做的事情完全相同。在这篇文章中，我们将将所有基本线束部件升级到那个生产形状，同时不隐藏任何机制背后。整个活动的核心问题很简单：","如何将一个单一的LLM调用变成一个可靠的系统，能够规划、行动、恢复并证明自己做了正确的事？","我们的答案是作曲。我们构建了小型、可测试的原语：类型工具、计划DAG：https：//en.wikipedia.org/wiki/Directed_acyclic_graph、分层内存、验证层级、预算和追踪器，并用刻意设计的编排器将其连接起来。每个原元素的存在，是因为天真的智能体以特定且可预测的方式失败。LLM会发明无效的工具参数，所以我们用Pydantic添加类型工具：https：//pydantic.dev/docs/validation/latest/get-started/ validation。所有程序都是顺序运行的，所以我们添加了依赖图和并行执行。上下文窗口会被垃圾填充，所以我们在有限的检索预算下添加了多层内存。坏输出会悄无声息地传播，所以我们会添加验证层级。一个提示试图完成所有事情，所以我们把它分成了规划者、工作者和批评者三个角色。成本飞涨，因此我们采用多维度预算并优雅地递减。","验证这种背带通常有效，包括评估套件、取样基准和专业工人池，将在后续文章中全面处理。","在整篇文章中，我们构建了一个城市比较代理：给定一个城市列表，它会生成一份报告，比较这些城市的人口、时区以及每个城市的简短叙述性总结。这个任务看起来几乎简单到令人发笑，但它是经过仔细挑选的。每个城市-属性查询都是独立的，这意味着一个三城市的请求自然会分解成九个可以同时运行的工具调用。另一方面，最终报告依赖于所有查询结果先完成，因此我们已经远远超出了一个简单步骤列表。我们可以通过程序检查，确保每个请求的城市确实出现在报告中，以验证结果。而且各工具的成本差异巨大：人口和时区查询是内存字典读取，而每个城市的摘要和最终聚合都调用了大语言模型，这给我们带来了现实的预算管理压力。","为了可重现性，查询工具从一个小型模拟字典 CITY_FACTS 中读取，因此该笔记本在没有网络访问的情况下也能完全复现。基于大语言模型的部分可以对真实的 Anthropic 模型或确定性模拟进行运行，这引出了第一个基本原语。","我们即将构建的每个组件最终都要调用大语言模型：计划器、摘要器、聚合器、评论者。如果这个调用被硬性绑定到一个 SDK，那么整个框架就无法测试，并且会被供应商锁定。","因此，在进行其他操作之前，我们定义了一个基类，对各种大语言模型调用 API 的细节提供抽象。","我们还实现了一个 MockProvider，用于测试和调试，其返回确定性的、角色感知的响应：在被要求制定计划时返回标准计划，在被要求总结时返回模板化的一行摘要，在被要求判断时返回基于规则的通过/失败判定。这使我们在开发过程中能够区分“我的编排错了吗？”与“模型计划得不好吗？”，这也是本文中每个实验能够在任何机器上复现的原因。","在基础框架中：https://data4sci.substack.com/p/building-a-basic-agentic-harness 我们手动验证了工具参数，这种方法迅速失效：每增加一个新工具就重复验证逻辑，LLM 从未看到正式的模式，只能猜测参数形状，产生的错误是模型无法自我纠正的临时字符串。","升级的方法是将每个工具的参数声明为 Pydantic 模型，并让一个定义驱动一切：","这种方法为我们提供了运行时验证、符合 Anthropic 和 OpenAI 工具使用 API 所期望形状的 JSON Schema、文档（每个 Field( …, description =… ) 成为计划器读取的目录的一部分）以及通过 cost_hint 进行成本核算的钩子。在执行前失败可以避免具有潜在副作用的昂贵工具调用。一个不好的计划应在验证层快速失败，而不是在数据库查询深处失败。这种方法类似于像 LangChain 工具、Anthropic 工具使用和 OpenAI 函数调用等成熟框架的做法。","我们的注册表包含四个工具，分为三个成本等级：get_population 和 get_timezone() 基本上是免费的字典查找（cost_hint =0.1），summarize_city() 每个城市调用一次 LLM（cost_hint =1.0），aggregate_report() 做一次产生最终 markdown 的大量 token 合成调用（cost_hint =2.0）。注意最后两个是内部调用 LLM 的工具。LLM 就像其他任何工具一样。工作器看到统一的工具接口，但有些工具是子提示的包装器，这意味着你可以独立于框架缓存、限速或替换内部模型。","基础框架每轮执行一个操作。当步骤严格顺序执行时这有效，但我们的任务有九个独立查找汇总成一个聚合：","：https://substackcdn.com/image/fetch/$s_!bqrk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdda72665-ee70-4a2e-9194-eed00992aaad_1057x778.png","while循环一次运行这些任务。一个有向无环图（DAG）：https://en.wikipedia.org/wiki/Directed_acyclic_graph 明确表达了依赖关系，并允许执行器并发运行当前准备好的所有任务。所以我们不是一次向LLM请求一个操作，而是一次性向规划器请求整个图。LLM在执行任何操作之前声明结构。由于规划器本身是LLM，它也可能产生虚构结构：依赖不存在的节点ID，或者产生永远无法完成的循环依赖。因此，我们对计划进行的第一件事就是在尝试执行损坏的计划浪费令牌之前先验证它。","ready_nodes() 是调度器的核心：在任何时刻，它返回所有依赖都已满足的节点集合。对于我们的三城市目标，规划器输出十个节点：九个依赖列表为空的 fetch 节点，全部可以并行运行，还有一个依赖所有九个节点的 aggregate_report capstone 节点。","执行器是一个层次同步的 DAG 遍历器：计算 ready 集合，使用 asyncio.gather 并发启动每个就绪节点，标记每个节点为完成或失败，然后重复直到没有剩余节点或无法进一步前进。","两项小决策在这里起到最大作用。首先，asyncio.to_thread 在线程池中运行我们的同步工具函数，这意味着我们不必将工具重写为 async def，也不必将加载器与异步原生 SDK 耦合。其次，信号量限制并发，因为没有它，一个五十节点的计划会同时生成五十次 LLM 调用，并立刻触发速率限制或成本激增。这显然不是完整的动态调度器，没有工作窃取和优先队列。对于像我们这样的代理工作负载，每个节点都是持续数百毫秒到几秒的 API 调用，层次同步并行可以获得大部分收益。顺序执行时，实际耗时大约是 fetch 延迟的总和；并行执行时，实际耗时大约是最大延迟加上聚合步骤。","天真的代理会把所有内容都扔进提示中：完整的聊天记录、每个工具输出、每个先前的任务。这种做法失败的原因有两个——你会为永远不会使用的令牌付费，而且当无关文本稀释目标时，模型的性能会明显下降。生产环境中的代理则使用分层记忆，灵感部分来自认知科学。工作记忆是始终在上下文中的便笺：当前目标、计划摘要以及最近的几个结果。情节记忆存储过去任务的结果，当过去的任务与当前任务相似时会被调用。语义记忆保存背景事实，也通过相同方式调用，但不与任何特定运行绑定。","我们从不注入所有内容；我们会根据与当前目标的相似度提取 top-k 记忆，然后在严格的字符预算下组装上下文：","情节记忆优先于语义记忆，因为在类似任务上的过去错误通常比通用事实更具可操作性，并且当预算耗尽时，截断是明确的而非默默进行。上下文应主动组装，而不是被动累积。","对于相似度函数本身，存储支持两种后端。Jaccard 相似度：https://en.wikipedia.org/wiki/Jaccard_index 不产生成本，非常适合教学，但在处理同义改写时会失败：“法国著名地标”几乎没有与“巴黎以埃菲尔铁塔闻名”共享单词。通过 all-MiniLM-L6-v2：https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 生成的 384 维向量的真实句子嵌入可以将同义改写映射到邻近向量。我们的 MemoryStore 首先尝试使用嵌入，如果模型不可用，则备用使用 Jaccard。量化此升级效果需要一个合适的基准测试，我们将在未来的文章中进行。","代理会生成流畅、自信但错误的输出。如果不进行验证，默默丢掉一个城市的报告会直接发送给用户，并且回归问题会一直被忽略，直到有人偶然查看输出为止。但并非所有检查的成本都相同，因此我们将它们排列成层级：确定性的结构检查基本免费，而用于主观质量的 LLM 判定则需要真实令牌。规则是始终先运行低成本层，只有通过此层的内容才会升级到下一层。","给它一个故意不完整的报告（比如只给巴黎，而实际上要求的是三个城市），它将在确定性层面失败，原因是“缺失城市：[‘东京’, ‘纽约’]”。在判断上没有消耗任何令牌，并且原因字符串足够可操作，一个重规划步骤（或人工）可以确切看到出了什么问题。这种两层门控是大多数生产评估管道背后的稳健模式：先进行低成本过滤，然后只对幸存者执行高成本评判。和层级结构一样重要的，是背后的关注点分离：Worker（执行者）生成，Critic（评审者）评价，因此生成器永远不会评自己的作业。","一个同时计划、执行、总结并自我批评的单一提示往往会使目标混淆（计划约束会影响写作风格），且无法隔离“规划部分”，这会使测试和替换更加复杂。","我们将工作拆分成狭窄的代理，每个代理都有简短的系统提示和单一的合同。Planner（规划者）接收目标和工具模式并返回DAG JSON，我们在运行前对其进行验证。Worker（执行者）接收DAG并简单执行。Critic（评审者）接收目标和完成的报告并返回判决。Planner的系统提示中直接嵌入了实时工具目录，因此它只能引用实际存在的工具：","在真正的模型支持下，规划器返回的计划在结构上有效但风格多样，并且在提示中锁定协调者以后需要找到的单个节点，比在后处理代码中完成要便宜得多。","：https://substackcdn.com/image/fetch/$s_!5A9N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6b4fb9de-22ea-419c-a112-1d77ffbb4611_1131x1711.png","这与生产系统如AutoGPT风格的规划器、SWE-agent执行者或LLM作为评判者的评估做法相似，同时仍保持精简到可一口气阅读的程度。而且因为所有三个角色都通过相同的LLMProvider.complete( …, role =… )接口，所以更换规划器模型或模拟评审者只需一行代码即可完成。","基本的 harness 只有一个 max_steps 计数器，这掩盖了真实的约束条件：你可能还有剩余步骤但没有剩余的 tokens，或者在 token 预算范围内但工具调用受到速率限制，或者网络挂起调用消耗了实际时间却完全不增加任何计数器。BudgetMulti 同时跟踪 tokens、工具调用、墙钟时间和预估费用，而运行会在任一维度耗尽时停止。它最有用的输出是一个标量：","Pressure 是所有维度中最大利用率。你会受到首先耗尽的资源限制，就像真实计费一样。这个数字驱动了优雅降级：低于 0.7 时，全流程运行，包括 LLM 评判；高于 0.9 时，调度程序会跳过昂贵的评论程序，仅回落到确定性检查；达到 1.0 时，运行会在部分结果下停止。生产环境的代理使用相同的信号来切换到更便宜的模型，减少检索深度，或向用户请求确认。","操作理智的另一半是认识到不是每个错误都值得同样处理。速率限制或超时是短暂的，可采用指数回退（带抖动，这样一组代理不会同步重试）并重试。验证错误是工具误用，我们将结构化错误反馈给 LLM，使其可以纠正自己的论证。未知实体是信息缺失，所以重试实际上有害，因为重试一个捏造的城市名字每次都会失败；正确做法是重新规划而不使用该信息。策略违规是致命的，我们应立即停止。一个小型 classify_error() 函数将错误字符串映射到这四类中，恢复策略依据类别而非盲目重试。","当一个代理失败时，我们需要一个仅追加的结构化事件日志，它可以回答事情发生的顺序、每个步骤花费的时间、哪个角色消耗了令牌，以及在事情出错之前预算压力是否在上升。我们的 Tracer 中的每个事件都捕捉了身份信息（步骤 ID 和一个将工作步骤链接回生成它们的计划的父 ID）、语义信息（角色和所采取的动作）、经济信息（延迟、令牌、成本以及写入时的预算压力快照），对于评论事件还包括判决。模式是平的且简单无趣：一列可以序列化为 JSON 的字典，你可以将它们写入文件、发送到 OpenTelemetry 或 LangSmith，或者直接用 matplotlib 绘制。你不需要专有格式就能获得真实的可观测性，只需要足够的模式。","按步骤延迟，并按角色着色，能立刻显示 summarize_city() 和 aggregate_report() 节点占据了大多数墙钟时间，而查找操作延迟平缓。LLM 调用才是消耗秒数的地方，因此并行运行它们非常重要。","按角色的每步延迟","随着时间的推移，预算压力单调上升，在聚合器处急剧跳升，如果它在评论执行前超过 0.9 的降级阈值，追踪日志本身就能解释为什么 LLM 审判被跳过。","：https://substackcdn.com/image/fetch/$s_!7_R8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fff1bb4ea-8569-4e51-a81d-fc411115565f_3684x2484.png","最后，按角色分配的令牌显示计划或执行是否在消耗预算。在这项任务中，它严重倾向于工作者，因为其调用了许多 summarize。","：https://substackcdn.com/image/fetch/$s_!5U5q!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ef2844d-5bb2-4053-9dc7-9b1c2cbc2a6f_3684x2484.png 按角色的令牌使用情况。工作者在此任务中占主导地位。","有了这些原语之后，协调器（Orchestrator）几乎变得无聊。它从记忆中构建上下文，向规划器（Planner）请求一个有向无环图（DAG），将 DAG 交给工作器（Worker）并附上一个在每次工具调用时记录跟踪事件并计入预算的回调，检查失败情况，使用压力感知降级验证结果，将结果存储在情节记忆中以备将来运行，并返回一个 RunResult，捆绑报告、裁决、DAG、跟踪和预算。它唯一新增的行为是重新规划循环。如果执行失败且错误被分类为信息缺失，它会将失败上下文发送回规划器，而不是盲目重试，最多可配置为 max_replans。","该方法的核心可以用几行代码概括：","模拟规划器（mock planner）总是将顶点节点命名为 aggregate，而协调器的早期版本只是通过该 id 查找它。切换到真实模型后，这个假设悄然失效。规划器经常镜像工具名称 aggregate_report 或发明自己的 id，生成一个结构上有效但不是模拟训练时我们预期的计划。解决方法有两个方面：DAG 现在通过工具而非 id 解析顶点节点 dag.aggregate_node() 返回唯一的 aggregate_report 节点，（并拒绝包含多个节点的计划），规划器提示也增加了我们之前看到的明确顶点指令。永远不要完全相信 LLM，包括节点名称。","在基础框架（Basic Harness：https://data4sci.substack.com/p/building-a-basic-agentic-harness）与这个高级框架之间，我们几乎涵盖了构建一个成功自定义框架所需的所有主要概念和思想。我们从前一篇文章中的约 35 行循环出发，并叠加了七个生产级原语：根据一种模式提供验证和 LLM 内省的类型化工具（typed tools）、无需重写任何工具即可实现并行的 DAG 执行器、在严格预算下组装相关上下文的分层记忆、只在通过廉价检查的输出上花费 token 的验证层次结构、可以独立测试和替换的狭窄规划器/工作器/批评者角色、多维预算可以优雅降级而非崩溃，以及将运行转化为可比较实验而非轶事的跟踪器。","这些组件彼此不依赖，但可以组合在一起。向注册表添加一个工具，规划器会自动看到它的模式。收紧 verify_report() 后，每一次未来的运行都必须达到新的标准。更换整个 LLM 后端，测试平台会在不改变的情况下重新运行。可组合性是概念验证与可扩展测试平台的区别所在，也是编排器保持轻量的原因。","有几个注意事项：这里的内存是在进程内，而生产系统会将嵌入持久化到 Chroma、Weaviate 或 pgvector；工具输出被信任为指令，而生产系统必须将其作为数据进行沙箱处理以防提示注入；不可逆操作应需要人工批准；我们的令牌计算是根据字符数估算的，而实际系统从 SDK 读取使用元数据。每一项都是进一步的一层可组合性——这正是重点所在。","还有一个刻意省略的内容。一次成功的演示证明测试平台可以工作；本文没有证明它在大多数情况下确实可行。这是评估测试平台的工作，也是我们将在未来的后续文章中展开的内容。"]},"en":{"title":"Building an Advanced Agentic Testing Framework","summary":"The article explains how to build an advanced agentic testing framework (Agentic Harness) for systematically evaluating and testing AI agents. The framework design focuses on reproducible and scalable testing processes to verify agent performance in complex tasks. It discusses the framework's core architecture, key components, and practical considerations, providing developers with a complete reference for AI testing solutions.","category":"Insights","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Building an Advanced Agentic Testing Framework - Aioga AI News","description":"The article explains how to build an advanced agentic testing framework (Agentic Harness) for systematically evaluating and testing AI agents. The framework design focuses on repro...","url":"https://www.aioga.com/en/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:26:27.060Z"},"ja":{"title":"高度なプロキシテストフレームワークを構築する","summary":"本記事では、AIエージェントを体系的に評価・テストするための高度なエージェントテストフレームワーク(Agentic Harness)を構築する方法を紹介します。 フレームワーク設計は、複雑なタスクにおけるエージェントのパフォーマンスを検証するための、繰り返しかつスケーラブルなテストプロセスに焦点を当てています。 この記事では、フレームワークのコアアーキテクチャ、主要なコンポーネント、そして実用的な応用点について論じ、開発者に完全なテストソリューションを提供しています。","category":"ヒントと視点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"高度なプロキシテストフレームワークを構築する - Aioga AIニュース","description":"本記事では、AIエージェントを体系的に評価・テストするための高度なエージェントテストフレームワーク(Agentic Harness)を構築する方法を紹介します。 フレームワーク設計は、複雑なタスクにおけるエージェントのパフォーマンスを検証するための、繰り返しかつスケーラブルなテストプロセスに焦点を当てています。 この記事では、フレームワークのコアアーキテクチ...","url":"https://www.aioga.com/ja/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:26:28.857Z"},"ko":{"title":"고급 프록시 테스트 프레임워크를 구축하세요","summary":"이 글에서는 AI 에이전트를 체계적으로 평가하고 테스트하기 위한 고급 에이전트 테스트 프레임워크(에이전트 하네스)를 구축하는 방법을 소개합니다. 프레임워크 설계는 복잡한 작업에서 에이전트의 성능을 검증하기 위한 반복 가능하고 확장 가능한 테스트 프로세스에 중점을 둡니다. 이 글은 프레임워크의 핵심 아키텍처, 핵심 구성 요소, 그리고 실용적인 적용 사항에 대해 논의하며, 개발자들에게 완전한 테스트 솔루션 세트를 제공합니다.","category":"인사이트","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"고급 프록시 테스트 프레임워크를 구축하세요 - Aioga AI 뉴스","description":"이 글에서는 AI 에이전트를 체계적으로 평가하고 테스트하기 위한 고급 에이전트 테스트 프레임워크(에이전트 하네스)를 구축하는 방법을 소개합니다. 프레임워크 설계는 복잡한 작업에서 에이전트의 성능을 검증하기 위한 반복 가능하고 확장 가능한 테스트 프로세스에 중점을 둡니다. 이 글은 프레임워크의 핵심 아키텍처, 핵심 구성 요...","url":"https://www.aioga.com/ko/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:26:37.609Z"},"es":{"title":"Construir un marco avanzado de pruebas de proxy","summary":"Este artículo introduce cómo construir un avanzado Marco de Pruebas Agenticas (Agentic Harness) para evaluar y probar sistemáticamente agentes de IA. El diseño del framework se centra en procesos de prueba repetibles y escalables para verificar el rendimiento de los agentes en tareas complejas. El artículo analiza la arquitectura central, los componentes clave y las consideraciones prácticas de aplicación del framework, proporcionando a los desarrolladores un conjunto completo de soluciones de prueba.","category":"Ideas","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Construir un marco avanzado de pruebas de proxy - Aioga Noticias de IA","description":"Este artículo introduce cómo construir un avanzado Marco de Pruebas Agenticas (Agentic Harness) para evaluar y probar sistemáticamente agentes de IA. El diseño del framework se cen...","url":"https://www.aioga.com/es/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:26:37.525Z"},"fr":{"title":"Construire un cadre avancé de test de proxy","summary":"Cet article présente comment construire un cadre avancé de test agent (Agentic Harness) pour évaluer et tester systématiquement les agents d’IA. La conception du cadre se concentre sur des processus de test répétables et évolutifs afin de vérifier la performance de l’agent sur des tâches complexes. L’article aborde l’architecture de base, les composants clés et les considérations pratiques d’application du cadre, offrant aux développeurs un ensemble complet de solutions de test.","category":"Analyses","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Construire un cadre avancé de test de proxy - Aioga Actualités IA","description":"Cet article présente comment construire un cadre avancé de test agent (Agentic Harness) pour évaluer et tester systématiquement les agents d’IA. La conception du cadre se concentre...","url":"https://www.aioga.com/fr/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:26:45.825Z"},"de":{"title":"Aufbau eines fortschrittlichen agentenbasierten Testframeworks","summary":"Der Artikel beschreibt, wie man ein fortschrittliches agentenbasiertes Testframework (Agentic Harness) erstellt, um AI-Agenten systematisch zu evaluieren und zu testen. Das Framework ist auf wiederholbare und skalierbare Testprozesse ausgerichtet, um die Leistung von Agenten in komplexen Aufgaben zu überprüfen. Es werden die Kernarchitektur, wichtige Komponenten und praktische Überlegungen behandelt und Entwicklern eine vollständige Referenzlösung für Tests bereitgestellt.","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Aufbau eines fortschrittlichen agentenbasierten Testframeworks - Aioga KI-News","description":"Der Artikel beschreibt, wie man ein fortschrittliches agentenbasiertes Testframework (Agentic Harness) erstellt, um AI-Agenten systematisch zu evaluieren und zu testen. Das Framewo...","url":"https://www.aioga.com/de/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:26:44.527Z"},"pt-BR":{"title":"Construa uma estrutura avançada de testes de proxy","summary":"Este artigo apresenta como construir uma Framework avançada de Teste Agentico (Agentic Harness) para avaliar e testar sistematicamente agentes de IA. O design do framework foca em processos de teste repetíveis e escaláveis para verificar o desempenho do agente em tarefas complexas. O artigo discute a arquitetura central, os principais componentes e considerações práticas de aplicação do framework, oferecendo aos desenvolvedores um conjunto completo de soluções de teste.","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Construa uma estrutura avançada de testes de proxy - Aioga Notícias de IA","description":"Este artigo apresenta como construir uma Framework avançada de Teste Agentico (Agentic Harness) para avaliar e testar sistematicamente agentes de IA. O design do framework foca em...","url":"https://www.aioga.com/pt-BR/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:26:54.383Z"},"ru":{"title":"Создайте продвинутый фреймворк для тестирования прокси","summary":"В этой статье представлены способы создания продвинутого фреймворка агентного тестирования (Agentic Harness) для систематической оценки и тестирования агентов ИИ. Дизайн фреймворка сосредоточен на повторяемых, масштабируемых процессах тестирования для проверки эффективности агента в сложных задачах. В статье рассматривается основная архитектура, ключевые компоненты и практические аспекты применения фреймворка, предоставляя разработчикам полный набор тестовых решений.","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Создайте продвинутый фреймворк для тестирования прокси - Aioga Новости ИИ","description":"В этой статье представлены способы создания продвинутого фреймворка агентного тестирования (Agentic Harness) для систематической оценки и тестирования агентов ИИ. Дизайн фреймворка...","url":"https://www.aioga.com/ru/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:26:53.091Z"},"ar":{"title":"بناء إطار عمل متقدم لاختبار البروكسي","summary":"تقدم هذه المقالة كيفية بناء إطار عمل متقدم لاختبار الوكلاء (Agentic Harness) لتقييم واختبار وكلاء الذكاء الاصطناعي بشكل منهجي. يركز تصميم الإطار على عمليات اختبار قابلة للتكرار والتوسع للتحقق من أداء الوكيل في المهام المعقدة. تناقش المقالة البنية الأساسية، والمكونات الرئيسية، والاعتبارات العملية للتطبيق في الإطار، مما يوفر للمطورين مجموعة كاملة من حلول الاختبار.","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"بناء إطار عمل متقدم لاختبار البروكسي - Aioga أخبار الذكاء الاصطناعي","description":"تقدم هذه المقالة كيفية بناء إطار عمل متقدم لاختبار الوكلاء (Agentic Harness) لتقييم واختبار وكلاء الذكاء الاصطناعي بشكل منهجي. يركز تصميم الإطار على عمليات اختبار قابلة للتكرار وال...","url":"https://www.aioga.com/ar/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:27:02.637Z"},"hi":{"title":"एक उन्नत प्रॉक्सी परीक्षण ढांचा बनाएं","summary":"यह लेख एआई एजेंटों के व्यवस्थित मूल्यांकन और परीक्षण के लिए एक उन्नत एजेंटिक टेस्टिंग फ्रेमवर्क (एजेंटिक हार्नेस) बनाने का तरीका बताता है। फ्रेमवर्क डिज़ाइन जटिल कार्यों पर एजेंट के प्रदर्शन को सत्यापित करने के लिए दोहराने योग्य, स्केलेबल परीक्षण प्रक्रियाओं पर केंद्रित है। लेख ढांचे के मुख्य वास्तुकला, प्रमुख घटकों और व्यावहारिक अनुप्रयोग विचारों पर चर्चा करता है, जो डेवलपर्स को परीक्षण समाधानों का एक पूरा सेट प्रदान करता है।","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"एक उन्नत प्रॉक्सी परीक्षण ढांचा बनाएं - Aioga AI समाचार","description":"यह लेख एआई एजेंटों के व्यवस्थित मूल्यांकन और परीक्षण के लिए एक उन्नत एजेंटिक टेस्टिंग फ्रेमवर्क (एजेंटिक हार्नेस) बनाने का तरीका बताता है। फ्रेमवर्क डिज़ाइन जटिल कार्यों पर एजेंट क...","url":"https://www.aioga.com/hi/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:27:02.837Z"},"it":{"title":"Costruire un framework avanzato di proxy testing","summary":"Questo articolo introduce come costruire un avanzato Agentic Testing Framework (Agentic Harness) per valutare e testare sistematicamente gli agenti IA. Il design del framework si concentra su processi di test ripetibili e scalabili per verificare le prestazioni degli agenti su compiti complessi. L'articolo discute l'architettura di base, i componenti chiave e le considerazioni pratiche applicative del framework, fornendo agli sviluppatori un set completo di soluzioni di testing.","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Costruire un framework avanzato di proxy testing - Aioga Notizie IA","description":"Questo articolo introduce come costruire un avanzato Agentic Testing Framework (Agentic Harness) per valutare e testare sistematicamente gli agenti IA. Il design del framework si c...","url":"https://www.aioga.com/it/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:27:09.932Z"},"nl":{"title":"Een geavanceerd agent-based testframework opbouwen","summary":"Het artikel beschrijft hoe een geavanceerd agent-based testframework (Agentic Harness) kan worden opgebouwd voor systematische evaluatie en testing van AI-agents. Het frameworkontwerp richt zich op herhaalbare en schaalbare testprocessen om de prestaties van agents bij complexe taken te verifiëren. Het artikel behandelt de kernarchitectuur, belangrijke componenten en praktische overwegingen bij gebruik, en biedt ontwikkelaars een volledige referentie voor testoplossingen.","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Een geavanceerd agent-based testframework opbouwen - Aioga AI-nieuws","description":"Het artikel beschrijft hoe een geavanceerd agent-based testframework (Agentic Harness) kan worden opgebouwd voor systematische evaluatie en testing van AI-agents. Het frameworkontw...","url":"https://www.aioga.com/nl/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:27:09.186Z"},"tr":{"title":"Gelişmiş bir proxy test çerçevesi oluşturun","summary":"Bu makale, yapay zeka ajanlarını sistematik olarak değerlendirmek ve test etmek için gelişmiş bir Ajanik Test Çerçevesi (Ajanik Harness) nasıl oluşturulacağını tanıtır. Çerçeve tasarımı, karmaşık görevlerde ajan performansını doğrulamak için tekrarlanabilir, ölçeklenebilir test süreçlerine odaklanır. Makale, çerçevenin temel mimarisi, temel bileşenleri ve pratik uygulama konularını tartışarak geliştiricilere tam bir test çözümleri seti sunar.","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Gelişmiş bir proxy test çerçevesi oluşturun - Aioga AI Haberleri","description":"Bu makale, yapay zeka ajanlarını sistematik olarak değerlendirmek ve test etmek için gelişmiş bir Ajanik Test Çerçevesi (Ajanik Harness) nasıl oluşturulacağını tanıtır. Çerçeve tas...","url":"https://www.aioga.com/tr/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:27:18.456Z"},"vi":{"title":"Xây dựng khung kiểm thử proxy tiên tiến","summary":"Bài viết này giới thiệu cách xây dựng Khung Kiểm thử Tác nhân tiên tiến (Agentic Harness) để đánh giá và kiểm thử các tác nhân AI một cách hệ thống. Thiết kế khung tập trung vào các quy trình kiểm thử có thể lặp lại, mở rộng để xác minh hiệu suất của agent trên các tác vụ phức tạp. Bài viết thảo luận về kiến trúc cốt lõi, các thành phần chính và các cân nhắc ứng dụng thực tiễn của framework, cung cấp cho các nhà phát triển một bộ giải pháp kiểm thử hoàn chỉnh.","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Xây dựng khung kiểm thử proxy tiên tiến - Tin tức AI Aioga","description":"Bài viết này giới thiệu cách xây dựng Khung Kiểm thử Tác nhân tiên tiến (Agentic Harness) để đánh giá và kiểm thử các tác nhân AI một cách hệ thống. Thiết kế khung tập trung vào cá...","url":"https://www.aioga.com/vi/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:27:18.405Z"},"id":{"title":"Bangun kerangka kerja pengujian proxy canggih","summary":"Artikel ini memperkenalkan cara membangun Kerangka Pengujian Agen canggih (Agentic Harness) untuk mengevaluasi dan menguji agen AI secara sistematis. Desain kerangka kerja ini berfokus pada proses pengujian yang dapat diulang dan dapat diskalakan untuk memverifikasi kinerja agen pada tugas-tugas kompleks. Artikel ini membahas arsitektur inti, komponen utama, dan pertimbangan aplikasi praktis dari kerangka kerja, memberikan pengembang satu set solusi pengujian yang lengkap.","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Bangun kerangka kerja pengujian proxy canggih - Berita AI Aioga","description":"Artikel ini memperkenalkan cara membangun Kerangka Pengujian Agen canggih (Agentic Harness) untuk mengevaluasi dan menguji agen AI secara sistematis. Desain kerangka kerja ini berf...","url":"https://www.aioga.com/id/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:27:26.912Z"},"th":{"title":"สร้างกรอบการทดสอบพร็อกซีขั้นสูง","summary":"บทความนี้แนะนําวิธีการสร้างกรอบการทดสอบเอเจนต์ขั้นสูง (Agentic Harness) สําหรับการประเมินและทดสอบเอเจนต์ AI อย่างเป็นระบบ การออกแบบเฟรมเวิร์กเน้นกระบวนการทดสอบที่ทําซ้ําได้และขยายขนาดได้ เพื่อยืนยันประสิทธิภาพของตัวแทนในงานที่ซับซ้อน บทความนี้กล่าวถึงสถาปัตยกรรมหลัก ส่วนประกอบสําคัญ และข้อพิจารณาด้านการใช้งานจริงของเฟรมเวิร์ก เพื่อให้นักพัฒนาได้รับชุดโซลูชันการทดสอบที่ครบถ้วน","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"สร้างกรอบการทดสอบพร็อกซีขั้นสูง - ข่าว AI Aioga","description":"บทความนี้แนะนําวิธีการสร้างกรอบการทดสอบเอเจนต์ขั้นสูง (Agentic Harness) สําหรับการประเมินและทดสอบเอเจนต์ AI อย่างเป็นระบบ การออกแบบเฟรมเวิร์กเน้นกระบวนการทดสอบที่ทําซ้ําได้และขยายข...","url":"https://www.aioga.com/th/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:27:26.279Z"},"pl":{"title":"Zbuduj zaawansowane ramy testowania proxy","summary":"Ten artykuł przedstawia, jak zbudować zaawansowane ramy testowania agentycznego (Agentic Harness) do systematycznej oceny i testowania agentów AI. Projekt ramy koncentruje się na powtarzalnych, skalowalnych procesach testowania w celu weryfikacji wydajności agentów w złożonych zadaniach. Artykuł omawia podstawową architekturę, kluczowe komponenty oraz praktyczne zastosowania frameworka, oferując deweloperom pełen zestaw rozwiązań testowych.","category":"技巧观点","source":"Hacker News 热门（buzzing.cc 中文翻译）","aggregationSource":"Hacker News 热门（buzzing.cc 中文翻译）","pageTitle":"Zbuduj zaawansowane ramy testowania proxy - Aioga Wiadomości AI","description":"Ten artykuł przedstawia, jak zbudować zaawansowane ramy testowania agentycznego (Agentic Harness) do systematycznej oceny i testowania agentów AI. Projekt ramy koncentruje się na p...","url":"https://www.aioga.com/pl/news/cmsgrdx2s0ehgro5qo5xp6ttm/","contentTranslated":true,"sourceHash":"aec8b7bdaba9d3a9","translatedAt":"2026-08-06T00:27:35.401Z"}}}}