Lyft、Vodafone、LATAM Airlines 的 CX 智能体生产实践与经验教训

客户体验已成为代理商中发展最快的类别之一，部分原因是其投资回报率相对容易衡量。更快的响应可以提高转化率，更少的升级可以降低每次联系的成本，而更多成功的解决方案有助于留住客户。

随着客户体验（CX）代理进入生产阶段，挑战已从构建它们转向改善它们的运行方式。团队正在从真实互动中学习，优化代理行为，并决定何时将对话转化为结构化流程。越来越多地，他们也在利用这些互动来改善更广泛的客户体验。

进展最远的团队将代理视为需要持续测试、部署、监控和迭代的生产系统。本文将探讨这种方法在三家公司中的形成情况：

借助来自Cisco和Podium的额外示例，我们将探索在客户体验中出现的用例、团队在生产中遇到的技术和运营挑战，以及LangSmith、Deep Agents和LangGraph如何在整个代理开发生命周期中支持持续改进：https://www.langchain.com/blog/the-agent-development-lifecycle。

面向消费者的自助代理通常是最显眼的起点。它们通过聊天或语音直接与客户互动，帮助处理账单、账户访问、索赔和预约等任务。它们的价值相对容易衡量：更快的响应可以提高转化率，而更多成功的解决方案可以减少升级并降低支持成本。例如，Podium的AI员工为汽车经销商、暖通空调承包商和其他本地企业的潜在客户提供响应。对于这些公司来说，五分钟内的响应产生的潜在客户转化率比一小时内的响应高46%。

前线和代表副驾驶可能是更高杠杆的用例。这些代理不是直接与客户交谈，而是与人工代表并肩工作，并提供下一步最佳行动。Cisco的客户体验组织在网络工程师中使用这种方法。其系统将成千上万的潜在发现缩小到最重要的几项，因此即使是像“帮助”这样模糊的请求，也可以被引导到正确的问题上。

当工程团队无法再构建每一个代理时，自助平台便应运而生。Lyft 的平台允许运营团队和产品经理创建提示和配置文件，然后在不涉及机器学习工程师的情况下启动新的支持代理。Podium 在其内部使用的相同基本元素上构建了类似的系统。这让一个底层架构支持广泛的用例，从汽车销售到暖通空调（HVAC）保修支持。

当客户请求不完整或模糊时，语义路由和分流变得至关重要。LATAM 航空在使用 Concierge 时就遇到了这种情况。起初，有 13% 的消息被归类为不在处理范围内。经过对这些对话的审查，团队发现其中 95% 是代理尚未设计处理的合法乘客需求，包括办理登机手续和行李问题。增加一名客户关怀专员将不在处理范围内的比例从 13% 降至 1%。

评估（Evals）成为技术团队和领域团队之间的共同语言。随着越来越多的人参与构建代理，团队需要一种一致的方式来定义良好行为的标准，并确定代理是否准备好投入使用。评估将领域专业知识转化为具体、可测试的标准，工程师、产品经理和运营团队可以用其来审查性能并指导改进。

Lyft 在开放代理开发给非工程师后遇到了这一点。平台不再是主要限制因素；提示和评估质量才是。团队引入了结构化的提示编写框架和自动检查，以在对话到达生产环境前捕捉矛盾的指令和不完整的对话路径。

总体而言，这些模式展示了 CX 代理投入生产后工作的变化。以下三个团队说明了组织如何在大规模下设计、评估和改进这些系统。

Lyft 的 AI Assist 支持骑手和司机解决账户访问、损害索赔、费用审核和收入争议等问题。Lyft 促成的出行量需要一个代理制支持系统。Lyft 每月促成 7900 万次出行，而 AI Assist 每月处理大约 27 万次交互，涉及七个或多个生产代理。该系统已实现 65% 的转移率和 35% 的 AI 解决率。

Lyft为解决问题设定了故意较高的标准，要求客服代理端到端地解决问题，而不仅仅是阻止客户联系人工客服。对于复杂流程，如司机损坏索赔，这可能包括收集信息和照片、通过工具检索数据、应用防欺诈信号、做出决策，并向司机解释结果（所有步骤需在15分钟内完成）。

Lyft当前的系统使用基于LangGraph构建的路由器式多代理架构。一个元代理会对每个传入请求进行分类，并将其路由到专门的子代理，骑手和司机有各自独立的路径。每个子代理自身也是一个完整的LangGraph状态图，并注册为子图节点。

当意图代理在对话中途确定某个请求需要更专业的处理者（例如，从通用司机意图代理切换到损坏索赔代理）时，它会将控制权返回给元代理以重新路由。这可以防止对话被迫走向错误的路径。

Lyft 将其代理分为两类：

这种方法将开发代理所需的时间从Lyft首个司机代理的大约六个月缩短到新可配置代理的大约两周。

随着平台变得更易使用，提示和评估质量开始成为瓶颈。

Lyft建立了连接开发与生产的评估飞轮。在发布前，团队会运行模拟的多轮对话，由大型语言模型扮演客户，与代理进行角色扮演。每次模拟围绕任务、用户角色和环境定义，这些都是反映代理在生产中可能遇到的情况。生成的轨迹可以通过代码断言和LLM评判的组合进行评估，包括代理是否给予了正确的让步、是否适当升级，或是否在预期轮次内解决了问题。

这些离线场景的多样性非常重要。Lyft使用离线评估作为发布门槛，允许团队快速推进，而不将真实客户作为测试对象。代理只有在满足所需质量标准时，才会向生产环境推进。

团队很早就意识到，通用评估指标是不够的。最初的一些衡量标准，如响应有用性、对话自然性、工具使用的适当性以及对话完整性，能够产生分数，但无法告诉团队需要改变什么。

相反，Lyft 与运营和质量专家合作，建立了基于支持交互应如何实际展开的狭义、行为特定的评分细则。团队还从广泛的标量评分转向了更简单的通过或不通过的结果。

例如，教育评分细则会检查代理是否在能够解决问题时提供有用的教育内容，但一旦明确无法解决问题就会升级。若代理重复提供相同的教育内容过多次、在合理尝试帮助之前就升级，或者包含事实错误，则判定为失败。

另一个升级评分细则定义了用户请求人工服务时的预期行为。代理应先推迟一次，然后在重复请求后升级。若代理立刻升级、在第二次请求后拒绝升级、在提供必要信息之前就升级，或在明显无法帮助后仍继续多轮对话，则判定为失败。

这些评分细则比通用质量分数更有用，因为每一次失败都指向具体的产品、提示或工作流程的改动。

Lyft 还会将其大语言模型（LLM）评审人员与人工评审者进行校准。团队收集人工标签，并对每位评审人员反复迭代，直到其达成足够高的一致率。这让团队有信心，自动化评分反映了运营和质量团队自己会应用的标准。

模拟用户也需要相同程度的校准。Lyft 的首批 LLM 生成的客户过于表达清楚、耐心且合作，在离线测试中通过率超过 90%，这并未反映实际生产环境的行为。真实用户常常以片段方式写作、省略上下文、重复自己，或带着特定目标，例如申请退款或绕过代理。

为了使离线评估更真实，Lyft 在真实客户原话的基础上微调了其模拟用户，并引入了退款寻求者、人工智能怀疑者以及决心联系人工客服的用户等角色。使模拟客户不那么完美增加了评估的难度，但也使离线结果更能预测生产性能。

一旦代理上线，同样的评估循环会在在线环境中继续。每一次调用都会在 LangSmith 中追踪，包括开发、预发布和生产环境，追踪内容包括代理的推理过程、它检索的教育内容以及调用的工具。这使团队能够识别失败是由路由、上下文、工具执行还是最终响应引起的。

LangSmith 还使评估过程对机器学习团队以外的人可用。产品经理和运营专家可以定义合格/不合格标准、编写评分细则并直接配置 LLM 裁判。这样能让最了解支持体验的人参与评估过程，而不需要工程师为他们翻译每一项需求。

Lyft 已配置了自动化，将失败的生产追踪发送到标注队列：https://docs.langchain.com/langsmith/annotation-queues。然后产品经理和质量审查员用自由形式语言标注失败模式，将单个糟糕互动转化为结构化的产品洞察。这些发现会反馈到提示、工作流、数据集和未来的离线测试中。

团队目前正致力于构建更标准化的评估工具。如今，许多离线测试仍然从一次性脚本或笔记本开始。Lyft 希望用版本化的原语（如任务、数据集、角色和评分器）替代这些，让团队可以共享并自动运行。

这将使得对每次提示更改进行回归测试、在相同场景下比较模型并维护可轻松扩展的评估集成为可能。

随着时间推移，Lyft 也看到这些追踪数据将不仅仅用于评估。成功的轨迹可以成为监督微调的示例。长期目标是让生产反馈不仅改善模型周围的提示和工作流，还能提升模型本身。

从 Lyft 得出的更广泛的经验教训是，将代理开发向更多人开放并不能消除严格性的需求，但它会将这种严格性转移到围绕提示创建、评估和生产反馈的系统中。自助平台使代理构建更快，而评估飞轮则使它们可以安全发布且随时间不断改进。

了解更多：Lyft 用户故事（博客：https://www.langchain.com/blog/lyft-built-a-self-serve-ai-agent-platform-for-customer-support-with-langgraph-and-langsmith），Lyft Interrupt 讲座（YouTube：https://www.youtube.com/watch?v=UVeeNW_z068）

Fastweb + Vodafone 是 Swisscom 集团的一部分，为意大利数百万电信客户提供服务。如此大规模的客户服务涉及广泛的需求，从账单和漫游到服务激活和技术支持，客户通常希望在一次互动中解决问题。

其现有聊天机器人 TOBi 可处理简单的请求，但更复杂的情况需要更深入的上下文、访问多个系统以及跨多个步骤的协调。呼叫中心顾问在内部也面临类似的挑战：他们需要快速了解客户历史，识别问题，并在多个系统和知识来源中确定正确的下一步操作。

Fastweb + Vodafone 着手支持体验的双方：一个面向客户的代理，能够端到端解决更复杂的请求；以及一个内部代理，可以帮助顾问更快、更一致地工作。

Fastweb + Vodafone 选择 LangGraph 和 LangChain 作为其 AI 转型的基础，因为其客户服务流程自然映射到基于图的决策流程。他们的实施围绕两个旗舰项目展开：Super TOBi 和 Super Agent。

Super TOBi 是 Fastweb + Vodafone 现有聊天机器人的代理演进版本。它现在在客户伴侣应用和语音频道为近 950 万客户提供服务，处理的用例包括成本控制、活动优惠、漫游、销售和账单。

该系统已实现 90% 的正确率、82% 的解决率，以及 7 分满分中 5.2 分的客户努力得分，有助于减少响应时间和向人工操作员的转接。

它的架构围绕两种类型的LangGraph代理组织：一个主管和一组专门的用例代理。

主管作为每个请求的入口点。它应用安全措施，验证并整理输入，并处理常见场景，如问候、对话结束以及将请求交给人工操作员。然后，它将请求路由到适当的用例代理，或者在意图不清时提出澄清问题。

每个用例代理负责特定类别的客户需求，并可以访问一组定义好的API。遵循LLM编译器模式，它可以确定调用哪些API，协调多步骤计划，并生成针对客户上下文的响应。

一些用例代理也可以返回结构化操作标签，而不仅仅是自然语言响应。这些标签使聊天机器人能够在对话中直接完成交易，例如激活优惠、停用服务或更新支付方式。

这使得Super TOBi能够超越仅回答问题。它可以计划并执行解决请求所需的步骤，在同一次交互中结合对话、数据检索、API调用和交易操作。

Super Agent是Fastweb + Vodafone面向呼叫中心顾问的内部AI系统。与Super TOBi不同，它不直接与客户互动。相反，它为顾问提供即时诊断、符合政策的指导、源支持的解释以及建议的下一步操作。这种方法帮助推动一次通话解决率超过86%。

该系统将LangChain的可组合工具与LangGraph的编排结合起来，并在Neo4j中以活跃图的形式存储操作知识。

业务专家首先在结构化模板中记录故障排除和信息流程，定义相关步骤、条件和操作。然后，使用LangGraph和任务专用代理构建的自动化管道解析这些文档，识别验证每个步骤所需的API，检查流程的一致性，并优化定义。

生成的内容存储在 Neo4j 中作为知识图谱，其中程序步骤与其条件、操作和支持的 API 相连接。CI/CD 管道负责验证和部署，使更新后的流程能够在数小时内无停机地进入生产。

当顾问提交请求时，LangGraph 主管首先确定请求是匹配结构化故障排除流程，还是需要开放式回答。此阶段会注入 CRM 数据，以便系统能够识别正确的客户并根据其上下文定制响应。

对于故障排除和故障隔离请求，主管会激活程序子图。系统从 Neo4j 检索相关流程，然后逐步执行。在每个阶段，它调用所需的 API 来测试相关条件。一旦满足条件，系统会识别问题并使用规定的操作以及沿途收集的客户上下文生成响应。如果没有条件满足，则继续到下一步，直到找到可能的问题及其解决方案。

关于公司知识的开放式问题则走不同的路径。这些问题会被路由到一个混合检索管道，该管道将向量存储与 Neo4j 知识图谱相结合。向量存储检索一组广泛相关的段落，而知识图谱则将答案绑定在正确的业务上下文中，添加来源引用，并帮助确保响应符合公司政策。

Fastweb + Vodafone 从开发的第一天起就实施了 LangSmith，认识到在生产 AI 系统中监控和评估的重要性。

“如果没有深入的可观察性，你不能在生产中运行自主系统。LangSmith 为我们提供了对 LangGraph 工作流的端到端可视性，让我们可以看到它们如何推理、路由和执行，把原本是黑箱的系统变成可以持续改进的操作系统。” — Pietro Capra，Fastweb + Vodafone 聊天工程章节负责人

团队已经开发了复杂的评估流程，这些流程每天运行，自动分类聊天机器人响应，并提供结构化反馈以进行持续改进：

该自动化评估系统使业务相关方能够审查每日绩效指标、提供战略性意见，并与技术团队沟通，以便及时调整，从而保持90%的正确率目标。自动监控与人工监督相结合，确保Super TOBi持续为客户创造价值，同时识别需要改进的领域。

正如Fastweb + Vodafone AI客户渠道负责人Lucia Barbieri所解释的，“自动化评估对有效扩展至关重要，使我们能够迅速识别改进领域并提升体验，推动持续增长和优化。”

Fastweb + Vodafone继续扩展Super TOBi和Super Agent的能力，同时保持其核心价值主张：通过智能自动化提供卓越的客户体验。展望未来，Fastweb + Vodafone计划利用其在LangGraph和LangSmith上的早期成功，探索在其电信业务中构建更多AI应用的可能性。

Customer experience has become one of the fastest-moving categories for agents, partly because the ROI is relatively easy to measure. Faster responses can improve conversion, fewer escalations can reduce the cost per contact, and more successful resolutions can help retain customers.

As CX agents move into production, the challenge shifts from building them to improving how they operate. Teams are learning from real interactions, refining agent behavior, and deciding when a conversation should become a structured workflow. Increasingly, they are also using those interactions to improve the broader customer experience.

The teams furthest along treat agents as production systems that require continuous testing, deployment, monitoring, and iteration. This piece looks at how that approach is taking shape across three companies:

Drawing on additional examples from Cisco and Podium, we’ll explore the use cases emerging across customer experience, the technical and operational challenges teams encounter in production, and how LangSmith, Deep Agents, and LangGraph support continuous improvement throughout the Agent Development Lifecycle：https://www.langchain.com/blog/the-agent-development-lifecycle.

Consumer-facing self-service agents are often the most visible starting point. They interact directly with customers through chat or voice, helping with tasks such as billing, account access, claims, and appointment scheduling. Their value is relatively easy to measure: faster responses can improve conversion, while more successful resolutions can reduce escalations and lower support costs. Podium’s AI Employee, for example, responds to inbound leads for car dealerships, HVAC contractors, and other local businesses. For these companies, responding within five minutes produces a 46% higher lead-conversion rate than responding within an hour.

Frontline and rep copilots can be an even higher-leverage use case. Rather than speaking directly to customers, these agents work alongside human representatives and surface the next best action. Cisco’s CX organization uses this approach for network engineers. Its system narrows thousands of potential findings down to the handful that matter most, so even a vague request like “help” can be routed toward the right issue.

Self-serve platforms emerge when engineering can no longer build every agent. Lyft’s platform allows operations teams and product managers to create a prompt and configuration file, then launch a new support agent without involving a machine learning engineer. Podium built a similar system around the same primitives it uses internally. This lets one underlying architecture support a wide range of use cases, from automotive sales to HVAC warranty support.

Semantic routing and triage become critical when customer requests are incomplete or ambiguous. LATAM Airlines saw this with Concierge. Initially, 13% of messages were classified as out of scope. After reviewing the conversations, the team found that 95% were legitimate passenger needs the agent had not yet been designed to handle, including check-in and baggage questions. Adding a customer-care specialist reduced the out-of-scope rate from 13% to 1%.

Evals become a shared language across technical and domain teams . As more people contribute to building agents, teams need a consistent way to define what good behavior looks like and determine whether an agent is ready to ship. Evals turn domain expertise into concrete, testable criteria that engineers, product managers, and operations teams can use to review performance and guide improvements.

Lyft encountered this after opening agent development to non-engineers. The platform was no longer the primary constraint; prompt and evaluation quality were. The team introduced a structured prompt-writing framework and automated checks to catch contradictory instructions and incomplete conversation paths before they reached production.

Together, these patterns show how the work changes once CX agents reach production. The following three teams illustrate how organizations are designing, evaluating, and improving these systems at scale.

Lyft’s AI Assist supports riders and drivers across issues such as account access, damage claims, charge reviews, and earnings disputes. The volume of trips Lyft facilitates necessitates an agentic system for support. Lyft facilitates 79 million trips each month, while AI Assist handles roughly 270,000 monthly interactions across seven or more production agents. The system has achieved a 65% deflection rate and a 35% AI resolution rate.

Lyft sets an intentionally high bar for resolution, requiring the agent to solve an issue end to end rather than simply prevent the customer from reaching a human. For complex workflows such as driver damage claims, that can include collecting information and photos, retrieving data through tools, applying fraud signals, making a decision, and explaining the outcome to the driver (all within 15 minutes).

Lyft’s current system uses a router-based, multi-agent architecture built on LangGraph . A meta-agent classifies each incoming request and routes it to a specialized subagent, with separate paths for riders and drivers. Each subagent is itself a complete LangGraph state graph registered as a subgraph node.

When an intent agent determines mid-conversation that a request requires a more specialized handler (e.g. moving from a general driver-intent agent to a damage-claim agent), it returns control to the meta-agent for rerouting. This prevents the conversation from being forced down the wrong path.

Lyft divides its agents into two categories:

This approach reduced the time required to develop an agent from roughly six months for Lyft’s first driver agent to about two weeks for a new configurable agent.

As the platform became easier to use, prompt and evaluation quality started to become bottlenecks.

Lyft built an evaluation flywheel that connects development and production. Before launch, the team runs simulated, multi-turn conversations in which an LLM role-plays the customer against the agent. Each simulation is defined around a task, user persona, and environment that reflects what the agent is likely to encounter in production. The resulting trajectory can be evaluated using a combination of code-based assertions and LLM judges, including whether the agent granted the correct concession, escalated appropriately, or resolved the issue within the expected number of turns.

The diversity of those offline scenarios is important. Lyft uses offline evaluation as a launch gate, allowing the team to move quickly without treating real customers as test cases. An agent only progresses toward production when it meets the required quality threshold.

The team learned early on that generic evaluation metrics weren’t enough. Initial measures such as response helpfulness, conversation naturalness, tool-use appropriateness, and conversation completeness produced scores, but didn’t tell the team what to change.

Lyft instead worked with operations and quality experts to build narrow, behavior-specific rubrics based on how support interactions should actually unfold. The team also moved from broad scalar scores to simpler pass-or-fail outcomes.

For example, an education rubric checks whether the agent provides useful educational content when it can solve the issue, but escalates once it becomes clear that it cannot. The agent fails if it repeats the same education too many times, escalates before making a reasonable attempt to help, or includes a factual error.

A separate escalation rubric defines the expected behavior when a user asks for a human. The agent should push back once, then escalate after a repeated request. It fails if it escalates immediately, refuses to escalate after the second request, escalates before providing necessary information, or continues for several turns after it is clear that it cannot help.

These rubrics are more useful than generic quality scores because each failure points toward a specific product, prompt, or workflow change.

Lyft also calibrates its LLM judges against human reviewers. The team collects human labels and iterates on each judge until it achieves a sufficiently high agreement rate. This gives the team confidence that automated scores reflect the standards its operations and quality teams would apply themselves.

The simulated user requires the same level of calibration. Lyft’s first LLM-generated customers were too articulate, patient, and cooperative, producing offline pass rates above 90% that did not reflect production behavior. Real users often write in fragments, omit context, repeat themselves, or arrive with a specific goal such as securing a refund or bypassing the agent.

To make offline evaluation more realistic, Lyft fine-tuned its simulated user on real customer verbatims and introduced personas such as refund seekers, AI skeptics, and users determined to reach a human. Making the simulated customer less polished made the evaluation harder, but also made offline results more predictive of production performance.

Once an agent launches, the same evaluation loop continues online. Every invocation is traced in LangSmith across development, staging, and production, including the agent’s reasoning, the educational content it retrieved, and the tools it called. This allows the team to identify whether a failure came from routing, context, tool execution, or the final response.

LangSmith also makes the evaluation process accessible beyond the machine learning team. Product managers and operations specialists can define pass-or-fail criteria, write rubrics, and configure LLM judges directly. That brings the people who understand the support experience most deeply into the evaluation process rather than requiring engineers to translate every requirement for them.

Lyft has configured automations that send failed production traces into an annotation queue：https://docs.langchain.com/langsmith/annotation-queues. Product managers and quality reviewers then label the failure mode in free-form language, turning individual bad interactions into structured product insights. Those findings feed back into prompts, workflows, datasets, and future offline tests.

The team is now working toward a more standardized evaluation harness. Today, many offline tests still begin as one-off scripts or notebooks. Lyft wants to replace those with versioned primitives (e.g. tasks, datasets, personas, and scorers) that teams can share and run automatically.

That would make it possible to regression-test every prompt change, compare models on the same scenarios, and maintain an evaluation set that grows easily.

Over time, Lyft also sees these traces becoming more than evaluation data. Successful trajectories can become supervised fine-tuning examples. The longer-term goal is for production feedback to improve not only the prompts and workflows around the model, but also the model itself.

The broader lesson from Lyft is that opening agent development to more people does not eliminate the need for rigor, but it shifts that rigor into the systems surrounding prompt creation, evaluation, and production feedback. The self-serve platform makes agents faster to build, while the eval flywheel makes them safe to ship and steadily better over time.

Learn more: Lyft User Story (Blog：https://www.langchain.com/blog/lyft-built-a-self-serve-ai-agent-platform-for-customer-support-with-langgraph-and-langsmith), Lyft Interrupt Talk (YouTube：https://www.youtube.com/watch?v=UVeeNW_z068)

Fastweb + Vodafone, part of the Swisscom Group, serves millions of telecommunications customers across Italy. Customer service at that scale involves a wide range of needs, from billing and roaming to service activation and technical support, often with customers expecting resolution in a single interaction.

Its existing chatbot, TOBi, could handle straightforward requests, but more complex cases required deeper context, access to multiple systems, and coordination across several steps. Call-center consultants faced a similar challenge internally: they needed to quickly understand a customer’s history, identify the issue, and determine the right next action across multiple systems and knowledge sources.

Fastweb + Vodafone set out to support both sides of the experience: a customer-facing agent capable of resolving more complex requests end to end, and an internal agent that could help consultants work more quickly and consistently.

Fastweb + Vodafone chose LangGraph and LangChain as the foundation for their AI transformation because their customer service process naturally mapped to a graph-based decision-making flow. Their implementation centers around two flagship projects: Super TOBi and Super Agent.

Super TOBi is the agentic evolution of Fastweb + Vodafone’s existing chatbot. It now serves nearly 9.5 million customers across the Customer Companion App and voice channels, handling use cases such as cost control, active offers, roaming, sales, and billing.

The system has achieved a 90% correctness rate, an 82% resolution rate, and a Customer Effort Score of 5.2 out of 7, helping reduce response times and transfers to human operators.

Its architecture is organized around two types of LangGraph agents: a Supervisor and a set of specialized Use Case agents.

The Supervisor acts as the entry point for every request. It applies guardrails, validates and shapes the input, and handles common scenarios such as greetings, conversation endings, and handoffs to human operators. It then routes the request to the appropriate Use Case agent or asks a clarifying question when the intent is unclear.

Each Use Case agent is responsible for a specific category of customer need and has access to a defined set of APIs. Following the LLM Compiler pattern, it can determine which APIs to call, coordinate a multistep plan, and generate a response tailored to the customer’s context.

Some Use Case agents can also return structured action tags rather than only natural-language responses. These tags allow the chatbot to complete transactions directly in the conversation, such as activating an offer, disabling a service, or updating a payment method.

This allows Super TOBi to move beyond answering questions. It can plan and execute the steps required to resolve a request, combining dialogue, data retrieval, API calls, and transactional actions within the same interaction.

Super Agent is Fastweb + Vodafone’s internally facing AI system for call-center consultants. Unlike Super TOBi, it does not interact directly with customers. Instead, it gives consultants instant diagnostics, policy-compliant guidance, source-backed explanations, and a recommended next step. This approach has helped drive One-Call Resolution rates above 86%.

The system combines LangChain’s composable tools with LangGraph’s orchestration and stores operational knowledge in a living graph in Neo4j.

Business specialists begin by documenting troubleshooting and informational procedures in structured templates, defining the relevant steps, conditions, and actions. An automated pipeline built with LangGraph and task-specific agents then parses those documents, identifies the APIs needed to verify each step, checks the procedures for consistency, and refines the definitions.

The resulting content is stored in Neo4j as a knowledge graph, where procedural steps are linked to their conditions, actions, and supporting APIs. A CI/CD pipeline handles validation and deployment, allowing updated procedures to reach production within hours and without downtime.

When a consultant submits a request, a LangGraph Supervisor first determines whether it matches a structured troubleshooting procedure or requires an open-ended answer. CRM data is injected at this stage so the system can identify the correct customer and tailor the response to their context.

For troubleshooting and fault-isolation requests, the Supervisor activates a procedural subgraph. The system retrieves the relevant procedure from Neo4j, then moves through it step by step. At each stage, it calls the required APIs to test the associated conditions. Once a condition is met, the system identifies the issue and generates a response using the prescribed action and the customer context gathered along the way. If no condition is met, it proceeds to the next step until it finds the likely problem and resolution.

Open-ended questions about company knowledge follow a different path. These are routed to a hybrid retrieval pipeline that combines a vector store with the Neo4j knowledge graph. The vector store retrieves a broad set of relevant passages, while the knowledge graph grounds the answer in the correct business context, adds source citations, and helps ensure the response follows company policy.

Fastweb + Vodafone implemented LangSmith from day one of development, recognizing the critical importance of monitoring and evaluation in production AI systems.

“You can’t run agentic systems in production without deep observability. LangSmith gave us end-to-end visibility into how our LangGraph workflows reason, route, and act, turning what would otherwise be a black box into an operational system we can continuously improve.” — Pietro Capra, Chat Engineering Chapter Lead, Fastweb + Vodafone

The team has developed sophisticated evaluation processes that run daily, automatically classifying chatbot responses and providing structured feedback for continuous improvement:

This automated evaluation system enables business stakeholders to review daily performance metrics, provide strategic input, and communicate with the technical team to make prompt adjustments to maintain the 90% correctness rate target. The combination of automated monitoring and human oversight ensures Super TOBi consistently delivers value to customers while identifying areas for improvement.

As Lucia Barbieri, Fastweb + Vodafone AI Customer Channels Lead, explains, “Automated evaluation has been crucial to scaling effectively, enabling us to quickly identify improvement areas and enhance experience, driving continuous growth and refinement.”

Fastweb + Vodafone continues expanding both Super TOBi and Super Agent capabilities while maintaining its core value proposition: delivering exceptional customer experiences through intelligent automation. Looking ahead, Fastweb + Vodafone plans to leverage its early success with LangGraph and LangSmith to explore building additional AI applications across its telecommunications operations.