构建语音代理很困难，因为一个好的代理必须在对话时感觉自然，能够解决用户的问题，并实现其设计的业务目标。
一次通话可能包括代理方的长时间停顿，听起来很尴尬，但仍能解决用户的问题。或者，代理可能完全按照指示操作，却因为缺乏回答用户请求所需的上下文而让客户失望。
这两种情景都显示了改进的空间。评估帮助团队识别这些弱点，并衡量更改是否真正提高了代理的表现。
这就是为什么我们建议从三个维度评估语音代理的原因：
这些维度是相关的，但不可互换。要进行良好评估，仅有文本记录是不够的。在 LangSmith 中，你可以追踪完整的交互、通过多名评估者打分、检查录音和工具使用活动，并比较不同时间的变化。
执行力衡量代理是否遵循其设计。
执行力涵盖最终响应以及代理生成响应的过程。一名语音代理可能最终给出了正确答案，同时调用了不必要的工具、跳过了必要的确认，或访问了不应使用的信息。对于单次调用，这可能没问题，但在多次交互中，这些错误可能导致不良的用户体验。
一些执行要求可以通过简单规则进行评估。对于一个预约调度代理，你可以检查是否：
当正确行为可以明确定义时，这些检查效果很好。它们还快速且成本低，因为不需要额外调用模型。
在 LangSmith 中，这就是代码评估器特别有用的地方：https://docs.langchain.com/langsmith/evaluation-types#code-evaluators。你可以跟踪调用，然后根据跟踪检查确定性规则。
其他要求依赖于意义而非具体值。LLM 评审可以评估代理是否：
当任务范围窄、评判标准明确时，LLM 评审效果最佳。一个广泛的问题如“这个回答是否好？”会产生噪声结果。具体的评分标准更易重复操作：
在 LangSmith 中，这就是 LLM 裁判可以根据精确的评估标准对对话进行评分的地方：https://docs.langchain.com/langsmith/evaluation-types#llm-as-a-judge。例如，你可以评分代理是否在继续之前询问了缺失的时区，或是否正确处理了涉及政策敏感的请求。
LLM 裁判之所以有效，是因为狭窄的评估任务比生成原始回复更容易，尤其是在裁判收到明确的评分标准和完整的追踪记录时。
假设一个日程安排代理被指示收集日期和时间、检查可用性并预约。当它完成所有三步，但工作流程从未告知它确认呼叫者的时区，因此预约被安排在错误的时间。在这种情况下，代理准确地遵循了指示，但仍未满足用户需求。这种区别将指令遵循与结果有效性区分开来，对于评估语音代理是否有效至关重要。
LLM 裁判可以帮助判断互动是否达到了预期的目标：
这些评估指向了适当的干预措施。失败可能需要更新知识库、添加工具、澄清指令，或为开发者未预料到的场景定义行为。
例如，在 LangSmith 中，你可以对一次日程安排对话进行评分，以判断用户在拨打代理电话预约后是否完成了预约。
生产中的呼叫也可以间接揭示失败。呼叫者可能会说：“我之前打过电话，但预约定错时间了。” 这一信号可以揭示原始对话中未显而易见的结果失败。
尽可能地，应衡量下游结果，而不仅仅从对话中推断成功。日程安排代理应根据预约记录进行评估，支持代理应根据解决情况和票据重新开启数据进行评估，转接代理应根据呼叫者是否到达正确的目的地进行评估。
有用的结果指标可能包括预约成功率、解决率、升级率、转接成功率、问题重新开启率、转化率或放弃率。正确的指标取决于工作流程。
在 LangSmith 中，您可以将这些业务信号连接回原始追踪，并根据实际结果对对话进行评分。这使得更容易判断新的提示是否改善了对话流程，同时也提升了实际的业务成果。
语音代理代表您的企业与客户进行交流。来电者不仅体验代理所说的话，还会感受到它的响应速度、发音清晰度以及在对话中的自然程度。一个正确有效的代理仍然可能是一个糟糕的语音代理。
延迟是语音交互中最显著的特征之一。主要的用户感知指标通常是回合结束延迟：从来电者回合结束到代理开始响应的时间。
典型的语音处理流程包括语音活动检测、语音转文本、模型推理、工具调用和文本转语音。分别测量每个组件有助于识别延迟来自转录、推理、外部系统还是音频生成。
相同的尴尬停顿可能有不同的原因。模型可能运行缓慢，工具可能被阻塞，或者语音生成可能要等到整个响应生成完毕才开始。LangSmith 追踪可以帮助识别哪个组件导致了延迟，而不是将延迟视为一个单一的黑箱数字。
自然性不仅包括生成的语音是否听起来自然。它可以包括：
这些属性可以由支持音频的模型评分。仅依赖文字记录的评判者可以评估措辞是否清晰或友好，但无法可靠判断代理的语音是否清晰或友好。关于语音表达的评价需要访问音频。
在 LangSmith 中，这正是具备音频感知能力的 LLM 评审者（audio-aware LLM judges）发挥用处的地方：https://docs.langchain.com/langsmith/evaluation-types#llm-as-a-judge。例如，您可以评分代理的发音是否易懂，是否打断过于频繁，或语速是否对支持电话来说感觉自然。
对话结构可以揭示可用性问题。有用的信号包括：
这些信号需要上下文。当请求模糊时，提出澄清性问题可以被视为良好行为。问题在于不必要的澄清、重复澄清，或未使用呼叫者已提供的信息。
同样，打断并不自动意味着不好。呼叫者希望能够打断语音代理。更有意义的问题是代理是否停止了讲话、保留了相关上下文，并做出了恰当的回应。
在LangSmith中，你也可以将这些转化为实际的评估工具。例如，你可以评分通话中是否出现了重复澄清循环、过长的停顿，或打断后恢复失败的情况。
没有单一的评估方法适用于所有维度。请选择基于所需证据的方法：
方法 最适合 示例 代码评估器 明确、确定性、可机器验证的行为 工具顺序、必需参数、延迟阈值 LLM评审 狭义语义标准 呼叫者的请求是否得到解决 音频感知LLM评审 录音中存在的属性 发音、语速、重叠讲话 业务系统检查 现实世界结果 重新开启的工单、完成的转接、参加的会议
人工审核对于模糊和高风险行为仍然很重要。审核者可以判断书面政策是否清晰，自动评估工具是否正确应用，以及异常对话是否属于合理例外。
人工标签还可以帮助校准LLM评审。如果审核者经常与评估器意见不一致，说明其评分标准或提示可能需要调整。如果审核者之间互相意见不一致，说明评估标准本身可能定义不够明确。
在LangSmith中，人工审核通过注释队列得到支持：https://docs.langchain.com/langsmith/annotation-queues：使用它们来验证评估器、优化评分标准，并改进下一版本基于追踪的工作流。
在LangSmith中，你可以将代表性对话转化为数据集，并为每种场景定义预期行为。一个有用的数据集可能包括：
然后，你可以通过实验来比较提示、模型、工具或工作流的变化在同一数据集上的表现。
由于执行、结果和体验保持分离，实验可以显示出一个新的提示词在提高指令遵循度的同时降低解决率。它可能表明工作流程的变化增加了成功预订的数量，同时引入了更多的政策违规，或者更快的模型降低了延迟，但处理中断的可靠性较差。这些权衡在单一的综合质量评分中消失。
生产评估还可以揭示单个通话审查可能遗漏的模式。评估者可以将结构化反馈附加到跟踪记录上，包括意图、解决状态、失败原因、情绪或对话摩擦指标。这些标签使得有可能在许多对话中识别常见意图、反复出现的失败和未满足的客户需求。
一个实用的工作流程看起来是这样的：
使用 LangSmith，您可以在不同框架和平台上始终如一地对语音代理进行评分，因此即使底层堆栈发生变化，也能比较其行为。
语音代理同时是一个软件系统、一个目标导向的工作流程和一种客户体验。要了解代理是否真正有效，而不仅仅是知道通话完成了，需要对三者进行评估。
我们的代理工程平台 LangSmith 帮助开发者调试每一个代理决策，评估更改，并一键部署。
Building voice agents is hard because a good one must feel natural to talk to, be capable of resolving the user’s issue, and deliver the business outcome it was designed for.
A call might include long pauses from the agent that sound awkward but still resolve the user’s issue. Or, an agent could follow its instructions exactly yet fail the customer because it lacks the context required to answer the customer’s request.
Both scenarios reveal room for improvement. Evaluations help teams identify those weaknesses and measure whether changes actually make the agent better.
That’s why we recommend evaluating voice agents across three dimensions:
The dimensions are related, but they are not interchangeable. And to evaluate them well, you need more than a transcript. In LangSmith, you can trace the full interaction, score it with multiple evaluators, inspect the recording and tool activity, and compare changes over time.
Execution measures whether the agent adhered to its design.
Execution covers both the final response and the path the agent took to produce it. A voice agent might eventually give the right answer while calling unnecessary tools, skipping a required confirmation, or accessing information it should not have used. For a single invocation of the agent this might be ok, but with repeated interactions these mistakes can create bad user experiences.
Some execution requirements can be evaluated with straightforward rules. For an appointment-scheduling agent, you can check whether:
These checks work well when the correct behavior can be defined explicitly. They are also fast and inexpensive because they do not require an additional model call.
In LangSmith, this is where code evaluators：https://docs.langchain.com/langsmith/evaluation-types#code-evaluators are especially useful. You can trace the call and then check deterministic rules against the trace.
Other requirements depend on meaning rather than exact values. An LLM judge can evaluate whether the agent:
LLM judges work best when the task is narrow and the criteria are explicit. A broad question such as “Was this response good?” will produce noisy results. A specific rubric is more repeatable:
In LangSmith, this is where an LLM judge：https://docs.langchain.com/langsmith/evaluation-types#llm-as-a-judge can score a conversation against a precise rubric. For example, you can score whether the agent asked for the missing timezone before proceeding, or whether it handled a policy-sensitive request correctly.
LLM judges work well because narrow judging tasks are easier than generating the original response, especially when the judge receives an explicit rubric and the complete trace.
Suppose a scheduling agent is instructed to collect a date and time, check availability, and book an appointment. It completes all three steps. But the workflow never tells it to confirm the caller’s timezone, so the appointment is booked for the wrong time. In this case, the agent followed its instructions accurately, but still failed the user. This distinction separates instruction adherence from outcome effectiveness, and is critically important for assessing whether a voice agent is working.
An LLM judge can help determine whether the interaction achieved its intended goal:
These evaluations point to the appropriate intervention. A failure might require updating the knowledge base, adding a tool, clarifying the instructions, or defining behavior for a scenario the developers did not anticipate.
For example, in LangSmith you could score a scheduling conversation on whether a user who called the agent to book an appointment ended the call with a scheduled booking.
Production calls can also expose failures indirectly. A caller might say, “I called earlier, but the appointment was booked for the wrong time.” That signal can reveal an outcome failure that was not obvious in the original conversation.
Whenever possible, measure the downstream result instead of inferring success from the conversation alone. A scheduling agent should be evaluated against the appointment record, a support agent against resolution and ticket reopen data, and a transfer agent against whether the caller reached the correct destination.
Useful outcome metrics may include booking success rate, resolution rate, escalation rate, transfer success rate, reopened issue rate, conversion rate, or abandonment rate. The right metric depends on the workflow.
In LangSmith, you can connect these business signals back to the original trace and score the conversation against the actual outcome. That makes it easier to see whether a new prompt improved the flow while also improving the real business result.
A voice agent represents your business to its customers. Callers experience not only what the agent says, but also how quickly it responds, how clearly it speaks, and how naturally it participates in the conversation. A correct and effective agent can still be a poor voice agent.
Latency is one of the most noticeable characteristics of a voice interaction. The primary user-perceived measure is often end-of-turn latency: the time from the end of the caller’s turn to the beginning of the agent’s response.
A typical voice pipeline includes voice activity detection, speech-to-text, model inference, tool calls, and text-to-speech. Measuring each component separately helps identify whether a delay comes from transcription, reasoning, an external system, or audio generation.
The same awkward pause can have different causes. The model might be slow, a tool might be blocking, or speech generation might wait until the entire response has been produced. LangSmith traces help identify which component contributed to the delay instead of treating latency as a single black-box number.
Naturalness includes more than whether the generated voice sounds human. It can include:
These properties can be scored by an audio-capable model. A transcript-only judge can evaluate whether the wording was clear or friendly, but it cannot reliably determine whether the agent sounded clear or friendly. Claims about vocal delivery require access to the audio.
In LangSmith, this is where audio-aware LLM judges：https://docs.langchain.com/langsmith/evaluation-types#llm-as-a-judge become useful. For example, you might score whether the agent’s pronunciation was understandable, whether it interrupted too aggressively, or whether the pacing felt natural for a support call.
The structure of a conversation can reveal usability problems. Useful signals include:
These signals need context. A clarifying question can be good behavior when a request is ambiguous. The problem is unnecessary clarification, repeated clarification, or failure to use information the caller already provided.
Similarly, an interruption is not automatically bad. Callers expect to be able to interrupt a voice agent. The more useful question is whether the agent stopped speaking, preserved the relevant context, and responded appropriately.
In LangSmith, you can turn these into practical evaluators too. For example, you can score whether a call had repeated clarification loops, excessive silence, or failed recovery after an interruption.
No single evaluation method works for every dimension. Choose the method based on the evidence required:
MethodBest forExampleCode evaluatorsExplicit, deterministic, machine-verifiable behaviorTool order, required arguments, latency thresholdsLLM judgesNarrow semantic criteriaWhether the caller’s request was resolvedAudio-aware LLM judgesProperties present in the recordingPronunciation, pacing, overlapping speechBusiness-system checksReal-world resultsReopened cases, completed transfers, attended meetings
Human review remains important for ambiguous and high-stakes behavior. Reviewers can determine whether a written policy is clear, whether an automated evaluator is applying it correctly, and whether an unusual conversation represents a legitimate exception.
Human labels also help calibrate LLM judges. If reviewers frequently disagree with the evaluator, its rubric or prompt may need work. If reviewers disagree with one another, the evaluation criterion itself may be underspecified.
In LangSmith, human review is supported with annotation queues：https://docs.langchain.com/langsmith/annotation-queues: use them to validate the evaluator, sharpen the rubric, and improve the next version of the trace-based workflow.
In LangSmith, you can turn representative conversations into a dataset and define expected behavior for each scenario. A useful dataset might include:
You can then run experiments to compare prompts, models, tools, or workflow changes against the same dataset.
Because execution, outcome, and experience remain separate, an experiment can reveal that a new prompt improves instruction adherence while reducing resolution rate. It might show that a workflow change increases successful bookings while introducing more policy violations, or that a faster model lowers latency but handles interruptions less reliably. These tradeoffs disappear inside a single aggregate quality score.
Production evaluation can also reveal patterns that individual call reviews miss. Evaluators can attach structured feedback to traces, including intent, resolution status, failure reason, sentiment, or conversational-friction indicators. These labels make it possible to identify common intents, recurring failures, and unmet customer needs across many conversations.
A practical workflow looks like this:
With LangSmith, you can score voice agents consistently across frameworks and platforms, so you can compare behavior even when the underlying stack changes.
A voice agent is simultaneously a software system, a goal-directed workflow, and a customer experience. To understand whether the agent actually worked beyond simply knowing that the call completed requires evaluating all three.
LangSmith, our agent engineering platform, helps developers debug every agent decision, eval changes, and deploy in one click.
扫码直达文章详情