Building voice agents is hard because a good one must feel natural to talk to, be capable of resolving the user’s issue, and deliver the business outcome it was designed for.
A call might include long pauses from the agent that sound awkward but still resolve the user’s issue. Or, an agent could follow its instructions exactly yet fail the customer because it lacks the context required to answer the customer’s request.
Both scenarios reveal room for improvement. Evaluations help teams identify those weaknesses and measure whether changes actually make the agent better.
That’s why we recommend evaluating voice agents across three dimensions:
The dimensions are related, but they are not interchangeable. And to evaluate them well, you need more than a transcript. In LangSmith, you can trace the full interaction, score it with multiple evaluators, inspect the recording and tool activity, and compare changes over time.
Execution measures whether the agent adhered to its design.
Execution covers both the final response and the path the agent took to produce it. A voice agent might eventually give the right answer while calling unnecessary tools, skipping a required confirmation, or accessing information it should not have used. For a single invocation of the agent this might be ok, but with repeated interactions these mistakes can create bad user experiences.
Some execution requirements can be evaluated with straightforward rules. For an appointment-scheduling agent, you can check whether:
These checks work well when the correct behavior can be defined explicitly. They are also fast and inexpensive because they do not require an additional model call.
In LangSmith, this is where code evaluators：https://docs.langchain.com/langsmith/evaluation-types#code-evaluators are especially useful. You can trace the call and then check deterministic rules against the trace.
Other requirements depend on meaning rather than exact values. An LLM judge can evaluate whether the agent:
LLM judges work best when the task is narrow and the criteria are explicit. A broad question such as “Was this response good?” will produce noisy results. A specific rubric is more repeatable:
In LangSmith, this is where an LLM judge：https://docs.langchain.com/langsmith/evaluation-types#llm-as-a-judge can score a conversation against a precise rubric. For example, you can score whether the agent asked for the missing timezone before proceeding, or whether it handled a policy-sensitive request correctly.
LLM judges work well because narrow judging tasks are easier than generating the original response, especially when the judge receives an explicit rubric and the complete trace.
Suppose a scheduling agent is instructed to collect a date and time, check availability, and book an appointment. It completes all three steps. But the workflow never tells it to confirm the caller’s timezone, so the appointment is booked for the wrong time. In this case, the agent followed its instructions accurately, but still failed the user. This distinction separates instruction adherence from outcome effectiveness, and is critically important for assessing whether a voice agent is working.
An LLM judge can help determine whether the interaction achieved its intended goal:
These evaluations point to the appropriate intervention. A failure might require updating the knowledge base, adding a tool, clarifying the instructions, or defining behavior for a scenario the developers did not anticipate.
For example, in LangSmith you could score a scheduling conversation on whether a user who called the agent to book an appointment ended the call with a scheduled booking.
Production calls can also expose failures indirectly. A caller might say, “I called earlier, but the appointment was booked for the wrong time.” That signal can reveal an outcome failure that was not obvious in the original conversation.
Whenever possible, measure the downstream result instead of inferring success from the conversation alone. A scheduling agent should be evaluated against the appointment record, a support agent against resolution and ticket reopen data, and a transfer agent against whether the caller reached the correct destination.
Useful outcome metrics may include booking success rate, resolution rate, escalation rate, transfer success rate, reopened issue rate, conversion rate, or abandonment rate. The right metric depends on the workflow.
In LangSmith, you can connect these business signals back to the original trace and score the conversation against the actual outcome. That makes it easier to see whether a new prompt improved the flow while also improving the real business result.
A voice agent represents your business to its customers. Callers experience not only what the agent says, but also how quickly it responds, how clearly it speaks, and how naturally it participates in the conversation. A correct and effective agent can still be a poor voice agent.
Latency is one of the most noticeable characteristics of a voice interaction. The primary user-perceived measure is often end-of-turn latency: the time from the end of the caller’s turn to the beginning of the agent’s response.
A typical voice pipeline includes voice activity detection, speech-to-text, model inference, tool calls, and text-to-speech. Measuring each component separately helps identify whether a delay comes from transcription, reasoning, an external system, or audio generation.
The same awkward pause can have different causes. The model might be slow, a tool might be blocking, or speech generation might wait until the entire response has been produced. LangSmith traces help identify which component contributed to the delay instead of treating latency as a single black-box number.
Naturalness includes more than whether the generated voice sounds human. It can include:
These properties can be scored by an audio-capable model. A transcript-only judge can evaluate whether the wording was clear or friendly, but it cannot reliably determine whether the agent sounded clear or friendly. Claims about vocal delivery require access to the audio.
In LangSmith, this is where audio-aware LLM judges：https://docs.langchain.com/langsmith/evaluation-types#llm-as-a-judge become useful. For example, you might score whether the agent’s pronunciation was understandable, whether it interrupted too aggressively, or whether the pacing felt natural for a support call.
The structure of a conversation can reveal usability problems. Useful signals include:
These signals need context. A clarifying question can be good behavior when a request is ambiguous. The problem is unnecessary clarification, repeated clarification, or failure to use information the caller already provided.
Similarly, an interruption is not automatically bad. Callers expect to be able to interrupt a voice agent. The more useful question is whether the agent stopped speaking, preserved the relevant context, and responded appropriately.
In LangSmith, you can turn these into practical evaluators too. For example, you can score whether a call had repeated clarification loops, excessive silence, or failed recovery after an interruption.
No single evaluation method works for every dimension. Choose the method based on the evidence required:
MethodBest forExampleCode evaluatorsExplicit, deterministic, machine-verifiable behaviorTool order, required arguments, latency thresholdsLLM judgesNarrow semantic criteriaWhether the caller’s request was resolvedAudio-aware LLM judgesProperties present in the recordingPronunciation, pacing, overlapping speechBusiness-system checksReal-world resultsReopened cases, completed transfers, attended meetings
Human review remains important for ambiguous and high-stakes behavior. Reviewers can determine whether a written policy is clear, whether an automated evaluator is applying it correctly, and whether an unusual conversation represents a legitimate exception.
Human labels also help calibrate LLM judges. If reviewers frequently disagree with the evaluator, its rubric or prompt may need work. If reviewers disagree with one another, the evaluation criterion itself may be underspecified.
In LangSmith, human review is supported with annotation queues：https://docs.langchain.com/langsmith/annotation-queues: use them to validate the evaluator, sharpen the rubric, and improve the next version of the trace-based workflow.
In LangSmith, you can turn representative conversations into a dataset and define expected behavior for each scenario. A useful dataset might include:
You can then run experiments to compare prompts, models, tools, or workflow changes against the same dataset.
Because execution, outcome, and experience remain separate, an experiment can reveal that a new prompt improves instruction adherence while reducing resolution rate. It might show that a workflow change increases successful bookings while introducing more policy violations, or that a faster model lowers latency but handles interruptions less reliably. These tradeoffs disappear inside a single aggregate quality score.
Production evaluation can also reveal patterns that individual call reviews miss. Evaluators can attach structured feedback to traces, including intent, resolution status, failure reason, sentiment, or conversational-friction indicators. These labels make it possible to identify common intents, recurring failures, and unmet customer needs across many conversations.
A practical workflow looks like this:
With LangSmith, you can score voice agents consistently across frameworks and platforms, so you can compare behavior even when the underlying stack changes.
A voice agent is simultaneously a software system, a goal-directed workflow, and a customer experience. To understand whether the agent actually worked beyond simply knowing that the call completed requires evaluating all three.
LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.
Scan to open this article