Customer experience has become one of the fastest-moving categories for agents, partly because the ROI is relatively easy to measure. Faster responses can improve conversion, fewer escalations can reduce the cost per contact, and more successful resolutions can help retain customers.

As CX agents move into production, the challenge shifts from building them to improving how they operate. Teams are learning from real interactions, refining agent behavior, and deciding when a conversation should become a structured workflow. Increasingly, they are also using those interactions to improve the broader customer experience.

The teams furthest along treat agents as production systems that require continuous testing, deployment, monitoring, and iteration. This piece looks at how that approach is taking shape across three companies:

Drawing on additional examples from Cisco and Podium, we’ll explore the use cases emerging across customer experience, the technical and operational challenges teams encounter in production, and how LangSmith, Deep Agents, and LangGraph support continuous improvement throughout the Agent Development Lifecycle：https://www.langchain.com/blog/the-agent-development-lifecycle.

Consumer-facing self-service agents are often the most visible starting point. They interact directly with customers through chat or voice, helping with tasks such as billing, account access, claims, and appointment scheduling. Their value is relatively easy to measure: faster responses can improve conversion, while more successful resolutions can reduce escalations and lower support costs. Podium’s AI Employee, for example, responds to inbound leads for car dealerships, HVAC contractors, and other local businesses. For these companies, responding within five minutes produces a 46% higher lead-conversion rate than responding within an hour.

Frontline and rep copilots can be an even higher-leverage use case. Rather than speaking directly to customers, these agents work alongside human representatives and surface the next best action. Cisco’s CX organization uses this approach for network engineers. Its system narrows thousands of potential findings down to the handful that matter most, so even a vague request like “help” can be routed toward the right issue.

Self-serve platforms emerge when engineering can no longer build every agent. Lyft’s platform allows operations teams and product managers to create a prompt and configuration file, then launch a new support agent without involving a machine learning engineer. Podium built a similar system around the same primitives it uses internally. This lets one underlying architecture support a wide range of use cases, from automotive sales to HVAC warranty support.

Semantic routing and triage become critical when customer requests are incomplete or ambiguous. LATAM Airlines saw this with Concierge. Initially, 13% of messages were classified as out of scope. After reviewing the conversations, the team found that 95% were legitimate passenger needs the agent had not yet been designed to handle, including check-in and baggage questions. Adding a customer-care specialist reduced the out-of-scope rate from 13% to 1%.

Evals become a shared language across technical and domain teams . As more people contribute to building agents, teams need a consistent way to define what good behavior looks like and determine whether an agent is ready to ship. Evals turn domain expertise into concrete, testable criteria that engineers, product managers, and operations teams can use to review performance and guide improvements.

Lyft encountered this after opening agent development to non-engineers. The platform was no longer the primary constraint; prompt and evaluation quality were. The team introduced a structured prompt-writing framework and automated checks to catch contradictory instructions and incomplete conversation paths before they reached production.

Together, these patterns show how the work changes once CX agents reach production. The following three teams illustrate how organizations are designing, evaluating, and improving these systems at scale.

Lyft’s AI Assist supports riders and drivers across issues such as account access, damage claims, charge reviews, and earnings disputes. The volume of trips Lyft facilitates necessitates an agentic system for support. Lyft facilitates 79 million trips each month, while AI Assist handles roughly 270,000 monthly interactions across seven or more production agents. The system has achieved a 65% deflection rate and a 35% AI resolution rate.

Lyft sets an intentionally high bar for resolution, requiring the agent to solve an issue end to end rather than simply prevent the customer from reaching a human. For complex workflows such as driver damage claims, that can include collecting information and photos, retrieving data through tools, applying fraud signals, making a decision, and explaining the outcome to the driver (all within 15 minutes).

Lyft’s current system uses a router-based, multi-agent architecture built on LangGraph . A meta-agent classifies each incoming request and routes it to a specialized subagent, with separate paths for riders and drivers. Each subagent is itself a complete LangGraph state graph registered as a subgraph node.

When an intent agent determines mid-conversation that a request requires a more specialized handler (e.g. moving from a general driver-intent agent to a damage-claim agent), it returns control to the meta-agent for rerouting. This prevents the conversation from being forced down the wrong path.

Lyft divides its agents into two categories:

This approach reduced the time required to develop an agent from roughly six months for Lyft’s first driver agent to about two weeks for a new configurable agent.

As the platform became easier to use, prompt and evaluation quality started to become bottlenecks.

Lyft built an evaluation flywheel that connects development and production. Before launch, the team runs simulated, multi-turn conversations in which an LLM role-plays the customer against the agent. Each simulation is defined around a task, user persona, and environment that reflects what the agent is likely to encounter in production. The resulting trajectory can be evaluated using a combination of code-based assertions and LLM judges, including whether the agent granted the correct concession, escalated appropriately, or resolved the issue within the expected number of turns.

The diversity of those offline scenarios is important. Lyft uses offline evaluation as a launch gate, allowing the team to move quickly without treating real customers as test cases. An agent only progresses toward production when it meets the required quality threshold.

The team learned early on that generic evaluation metrics weren’t enough. Initial measures such as response helpfulness, conversation naturalness, tool-use appropriateness, and conversation completeness produced scores, but didn’t tell the team what to change.

Lyft instead worked with operations and quality experts to build narrow, behavior-specific rubrics based on how support interactions should actually unfold. The team also moved from broad scalar scores to simpler pass-or-fail outcomes.

For example, an education rubric checks whether the agent provides useful educational content when it can solve the issue, but escalates once it becomes clear that it cannot. The agent fails if it repeats the same education too many times, escalates before making a reasonable attempt to help, or includes a factual error.

A separate escalation rubric defines the expected behavior when a user asks for a human. The agent should push back once, then escalate after a repeated request. It fails if it escalates immediately, refuses to escalate after the second request, escalates before providing necessary information, or continues for several turns after it is clear that it cannot help.

These rubrics are more useful than generic quality scores because each failure points toward a specific product, prompt, or workflow change.

Lyft also calibrates its LLM judges against human reviewers. The team collects human labels and iterates on each judge until it achieves a sufficiently high agreement rate. This gives the team confidence that automated scores reflect the standards its operations and quality teams would apply themselves.

The simulated user requires the same level of calibration. Lyft’s first LLM-generated customers were too articulate, patient, and cooperative, producing offline pass rates above 90% that did not reflect production behavior. Real users often write in fragments, omit context, repeat themselves, or arrive with a specific goal such as securing a refund or bypassing the agent.

To make offline evaluation more realistic, Lyft fine-tuned its simulated user on real customer verbatims and introduced personas such as refund seekers, AI skeptics, and users determined to reach a human. Making the simulated customer less polished made the evaluation harder, but also made offline results more predictive of production performance.

Once an agent launches, the same evaluation loop continues online. Every invocation is traced in LangSmith across development, staging, and production, including the agent’s reasoning, the educational content it retrieved, and the tools it called. This allows the team to identify whether a failure came from routing, context, tool execution, or the final response.

LangSmith also makes the evaluation process accessible beyond the machine learning team. Product managers and operations specialists can define pass-or-fail criteria, write rubrics, and configure LLM judges directly. That brings the people who understand the support experience most deeply into the evaluation process rather than requiring engineers to translate every requirement for them.

Lyft has configured automations that send failed production traces into an annotation queue：https://docs.langchain.com/langsmith/annotation-queues. Product managers and quality reviewers then label the failure mode in free-form language, turning individual bad interactions into structured product insights. Those findings feed back into prompts, workflows, datasets, and future offline tests.

The team is now working toward a more standardized evaluation harness. Today, many offline tests still begin as one-off scripts or notebooks. Lyft wants to replace those with versioned primitives (e.g. tasks, datasets, personas, and scorers) that teams can share and run automatically.

That would make it possible to regression-test every prompt change, compare models on the same scenarios, and maintain an evaluation set that grows easily.

Over time, Lyft also sees these traces becoming more than evaluation data. Successful trajectories can become supervised fine-tuning examples. The longer-term goal is for production feedback to improve not only the prompts and workflows around the model, but also the model itself.

The broader lesson from Lyft is that opening agent development to more people does not eliminate the need for rigor, but it shifts that rigor into the systems surrounding prompt creation, evaluation, and production feedback. The self-serve platform makes agents faster to build, while the eval flywheel makes them safe to ship and steadily better over time.

Learn more: Lyft User Story (Blog：https://www.langchain.com/blog/lyft-built-a-self-serve-ai-agent-platform-for-customer-support-with-langgraph-and-langsmith), Lyft Interrupt Talk (YouTube：https://www.youtube.com/watch?v=UVeeNW_z068)

Fastweb + Vodafone, part of the Swisscom Group, serves millions of telecommunications customers across Italy. Customer service at that scale involves a wide range of needs, from billing and roaming to service activation and technical support, often with customers expecting resolution in a single interaction.

Its existing chatbot, TOBi, could handle straightforward requests, but more complex cases required deeper context, access to multiple systems, and coordination across several steps. Call-center consultants faced a similar challenge internally: they needed to quickly understand a customer’s history, identify the issue, and determine the right next action across multiple systems and knowledge sources.

Fastweb + Vodafone set out to support both sides of the experience: a customer-facing agent capable of resolving more complex requests end to end, and an internal agent that could help consultants work more quickly and consistently.

Fastweb + Vodafone chose LangGraph and LangChain as the foundation for their AI transformation because their customer service process naturally mapped to a graph-based decision-making flow. Their implementation centers around two flagship projects: Super TOBi and Super Agent.

Super TOBi is the agentic evolution of Fastweb + Vodafone’s existing chatbot. It now serves nearly 9.5 million customers across the Customer Companion App and voice channels, handling use cases such as cost control, active offers, roaming, sales, and billing.

The system has achieved a 90% correctness rate, an 82% resolution rate, and a Customer Effort Score of 5.2 out of 7, helping reduce response times and transfers to human operators.

Its architecture is organized around two types of LangGraph agents: a Supervisor and a set of specialized Use Case agents.

The Supervisor acts as the entry point for every request. It applies guardrails, validates and shapes the input, and handles common scenarios such as greetings, conversation endings, and handoffs to human operators. It then routes the request to the appropriate Use Case agent or asks a clarifying question when the intent is unclear.

Each Use Case agent is responsible for a specific category of customer need and has access to a defined set of APIs. Following the LLM Compiler pattern, it can determine which APIs to call, coordinate a multistep plan, and generate a response tailored to the customer’s context.

Some Use Case agents can also return structured action tags rather than only natural-language responses. These tags allow the chatbot to complete transactions directly in the conversation, such as activating an offer, disabling a service, or updating a payment method.

This allows Super TOBi to move beyond answering questions. It can plan and execute the steps required to resolve a request, combining dialogue, data retrieval, API calls, and transactional actions within the same interaction.

Super Agent is Fastweb + Vodafone’s internally facing AI system for call-center consultants. Unlike Super TOBi, it does not interact directly with customers. Instead, it gives consultants instant diagnostics, policy-compliant guidance, source-backed explanations, and a recommended next step. This approach has helped drive One-Call Resolution rates above 86%.

The system combines LangChain’s composable tools with LangGraph’s orchestration and stores operational knowledge in a living graph in Neo4j.

Business specialists begin by documenting troubleshooting and informational procedures in structured templates, defining the relevant steps, conditions, and actions. An automated pipeline built with LangGraph and task-specific agents then parses those documents, identifies the APIs needed to verify each step, checks the procedures for consistency, and refines the definitions.

The resulting content is stored in Neo4j as a knowledge graph, where procedural steps are linked to their conditions, actions, and supporting APIs. A CI/CD pipeline handles validation and deployment, allowing updated procedures to reach production within hours and without downtime.

When a consultant submits a request, a LangGraph Supervisor first determines whether it matches a structured troubleshooting procedure or requires an open-ended answer. CRM data is injected at this stage so the system can identify the correct customer and tailor the response to their context.

For troubleshooting and fault-isolation requests, the Supervisor activates a procedural subgraph. The system retrieves the relevant procedure from Neo4j, then moves through it step by step. At each stage, it calls the required APIs to test the associated conditions. Once a condition is met, the system identifies the issue and generates a response using the prescribed action and the customer context gathered along the way. If no condition is met, it proceeds to the next step until it finds the likely problem and resolution.

Open-ended questions about company knowledge follow a different path. These are routed to a hybrid retrieval pipeline that combines a vector store with the Neo4j knowledge graph. The vector store retrieves a broad set of relevant passages, while the knowledge graph grounds the answer in the correct business context, adds source citations, and helps ensure the response follows company policy.

Fastweb + Vodafone implemented LangSmith from day one of development, recognizing the critical importance of monitoring and evaluation in production AI systems.

“You can’t run agentic systems in production without deep observability. LangSmith gave us end-to-end visibility into how our LangGraph workflows reason, route, and act, turning what would otherwise be a black box into an operational system we can continuously improve.” — Pietro Capra, Chat Engineering Chapter Lead, Fastweb + Vodafone

The team has developed sophisticated evaluation processes that run daily, automatically classifying chatbot responses and providing structured feedback for continuous improvement:

This automated evaluation system enables business stakeholders to review daily performance metrics, provide strategic input, and communicate with the technical team to make prompt adjustments to maintain the 90% correctness rate target. The combination of automated monitoring and human oversight ensures Super TOBi consistently delivers value to customers while identifying areas for improvement.

As Lucia Barbieri, Fastweb + Vodafone AI Customer Channels Lead, explains, “Automated evaluation has been crucial to scaling effectively, enabling us to quickly identify improvement areas and enhance experience, driving continuous growth and refinement.”

Fastweb + Vodafone continues expanding both Super TOBi and Super Agent capabilities while maintaining its core value proposition: delivering exceptional customer experiences through intelligent automation. Looking ahead, Fastweb + Vodafone plans to leverage its early success with LangGraph and LangSmith to explore building additional AI applications across its telecommunications operations.