作为我在谷歌工作的一部分，我们在 GitHub 上发布了一套针对谷歌产品和技术的代理技能：https://github.com/google/skills。这些代理技能：https://agentskills.io/home 旨在帮助 AI 代理与我们的技术进行交互。但你如何测试这些技能是否有用并按预期工作呢？我在开发者关系团队一直专注于这个问题，因为拥有关于它们性能的可靠信号对于帮助我们随着时间改进这些技能至关重要。
正如你不会在没有编写单元测试的情况下部署生产 API 一样，你也应该对 AI 代理应用同样的标准。正如 Joe Spiro 在“设计 AI 评估”系列文章中展示的：https://dev.to/googleai/designing-ai-evals-clarity-now-and-visualization-next-4eii，扩展 AI 工具意味着要超越在终端中“凭感觉测试”。相反，你应该建立一个结构化的、自动化的评估管道来基准测试你的集成。评估（evals）是你要求代理执行的动作，这些动作使用评分器（例如评分标准）评分，以判断代理是否成功。本文将重点介绍评估，并在下一篇文章中讨论评分标准的技巧。
在编写评估之前，你需要了解所选框架的设置和限制。这包括 Harbor、Inspect AI 等系统，或在开发工具中的集成，如代理开发工具包中的集成：https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/agent-evaluation。它是否使用临时沙箱？有哪些可用工具？如何捕获输出？
As part of my work at Google, we are publishing a suite of Agent Skills for Google products and technologies on GitHub：https://github.com/google/skills. These agent skills：https://agentskills.io/home are designed to help AI agents interact with our technologies. But how do you test that these skills are useful and work as expected? My team in Developer Relations has been focused on this question, because having reliable signals on their performance is critical to help us improve them over time.
Just as you wouldn't deploy a production API without writing unit tests, you should apply the same standard to your AI agents. As Joe Spiro showed in the Designing AI Evals ：https://dev.to/googleai/designing-ai-evals-clarity-now-and-visualization-next-4eii post series, scaling AI tools means moving beyond "vibe testing" in a terminal. Instead, you should set up a structured, automated evaluation pipeline to benchmark your integration. The evaluations (evals) are the actions you asked the agent to perform, which are graded using scorers (for example rubrics) that assert whether the agent succeeded. We'll focus on evaluations in this post and tackle tips for scoring rubrics in the next post.
However, AI evaluations cost real tokens. You need to make sure that you use these tokens as efficiently as possible. They need to provide real value that helps you build better tools. Writing good evaluations is critical. Poor evaluations provide false signals, waste your token budget, and create noise in your metrics.
Here are five rules we learned to design better evaluations you can trust. Follow them to ensure that every token you spend produces a useful metric.
Before writing evaluations, you need to understand the setup and limitations of your chosen framework. This includes systems like Harbor, Inspect AI, or integrations in development tools like in the Agent Development Kit：https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/agent-evaluation. Does it use an ephemeral sandbox? What tools are available? How is the output captured
If your evaluations show a high baseline accuracy (i.e., without your agent tool), it might not prove its value, or the evaluation prompts are too easy.
You cannot grade an agent on something you did not explicitly ask it to do. Your evaluation prompts and graders should be complementary. This means that they should only test for things included in the prompt.
Agents possess inherent model knowledge and might skip your custom tools entirely to arrive at the correct answer. (That on its own is some useful feedback!)
A strong evaluation suite tests real, diverse use cases. But testing the same capability repeatedly causes overfitting and creates noisy metrics.
You cannot improve AI tools if you can't measure them accurately. If you treat AI evaluations with the same focus as traditional unit tests, you improve the quality of your metrics and get more robust signals.
By applying these five rules, you eliminate false signals that waste your token budget. Instead of generating noise, your test suite gives you actionable feedback you can use to guide your engineering decisions and improve your tools.
Figuring out what to test is only the first step. A well-designed evaluation is only useful if the scorer grading answers is reliable and returns meaningful results. In my next post, we will look at how to test. You will learn how to write lean, atomic rubrics that minimize ambiguity for an LLM grader and make every token count.
Photo by William Warby：https://unsplash.com/@wwarby on Unsplash：https://unsplash.com/photos/gray-and-yellow-measures-WahfNoqbYnM
Templates let you quickly answer FAQs or store snippets for re-use.
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink：#.
For further actions, you may consider blocking this person and/or reporting abuse：/report-abuse
Google AI Studio is the fastest way to start building with Gemini. Ready to build?
DEV Community：/ — A space to discuss and keep up software development and manage your software career
Built on Forem：https://www.forem.com — the open source：https://dev.to/t/opensource software that powers DEV：https://dev.to and other inclusive communities.
扫码直达文章详情