Follow up to Part 1: How to Design AI Evaluations You Can Actually Trust：https://dev.to/googleai/how-to-design-ai-evaluations-you-can-actually-trust-41c3
At Google, we are publishing a suite of Agent Skills for Google products and technologies on GitHub：https://github.com/google/skills. My team is interested in measuring their performance to understand how they perform. Deterministic tests, like checking if generated code compiles, are ideal. Unfortunately they can't be easily created at scale for nuanced, generative responses, such as answers to open-ended questions or information retrieval tasks.
In my previous post we looked at what you test：https://dev.to/googleai/how-to-design-ai-evaluations-you-can-actually-trust-41c3, which means the evaluations that are the actions you ask the agent to perform. The next step is to look at how you assert whether the agent succeeded. This means creating reliable and accurate assessments of the agent's response.
To assess complex outputs at scale, especially when topics cover broad areas with nuanced parts, we use an "LLM-as-a-judge" approach. Responses are assessed against a structured rubric using a model based grader. The judge evaluates each response using a set of true/false questions. Aggregated, these answers provide an accuracy score for a response.
Giving an LLM a vague prompt or subjective questions leads to ambiguity in its response. This ambiguity introduces noisy data and causes inconsistent evaluations. Ultimately, it wastes your token budget on metrics that aren't useful.
To make these assessments more reliable, you must treat your rubrics like formal specifications. By constraining the judge to evaluate strict, objective boolean truths, you reduce the chance of hallucinations. Because evaluating strict boolean truths is a less complex task, you can even use smaller, faster models for grading.
Here are four lessons we learned to help you write robust rubric questions for your LLM-as-a-judge scorer.
Evaluating multiple requirements in a single question, such as "Does the response contain a metadata property and format the output as JSON?" , forces the LLM judge to guess which clause is more important. This ambiguity leads to inconsistent grading and wasted tokens.
The rubric-based approach exists because giving an LLM judge a full prose prompt to evaluate a complex response leads to inconsistent numbers. If you ask the judge subjective questions like "Was this a comprehensive answer?" or ask it to explain "Why did the agent do this?" , you introduce ambiguity that creates noisy, unrepeatable data.
When building rubrics, it is easy to accidentally evaluate an agent on requirements that were never stated in the prompt. Doing this creates false negatives and reduces the accuracy of your measurement.
Even if you follow these rules and write perfectly atomic, objective questions, your LLM judge might still misinterpret your scoring instructions and rubrics. To guarantee your pipeline generates consistent ratings and reliable signals, you must prove that the judge's scoring aligns with how a human subject matter expert would evaluate the exact same response through calibration .
Once you have this reliable data, the next step is making it visible. In AI Evals at a Glance: Heatmaps for Stakeholders：https://dev.to/googleai/ai-evals-at-a-glance-heatmaps-for-stakeholders-2mki Joe Spiro explains how to take these raw measurements and visualize evaluations.
When building our agent skills, we learned that ambiguous evaluation rubrics do not provide useful signals and feedback. Forcing your LLM judge to evaluate strict, boolean facts eliminates this noise. It makes your tests repeatable, optimizes your token spend, and lets you confidently measure whether your AI tool is actually improving.
Photo by William Warby：https://unsplash.com/@wwarby on Unsplash：https://unsplash.com/photos/gray-and-yellow-measures-WahfNoqbYnM
Templates let you quickly answer FAQs or store snippets for re-use.
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink：#.
For further actions, you may consider blocking this person and/or reporting abuse：/report-abuse
Google AI Studio is the fastest way to start building with Gemini. Ready to build?
DEV Community：/ — A space to discuss and keep up software development and manage your software career
Built on Forem：https://www.forem.com — the open source：https://dev.to/t/opensource software that powers DEV：https://dev.to and other inclusive communities.
Made with love and Ruby on Rails：https://dev.to/t/rails. DEV Community © 2016 - 2026.
We're a place where coders share, stay up-to-date and grow their careers.
Scan to open this article