Build Low-Latency Multilingual Voice Agents： Open Weights & Full Deployment Control with NVIDIA Magpie TTS

语音 AI 正在默认支持多语言：#voice-ai-is-becoming-multilingual-by-default 一个开放模型，支持十二种语言：#one-open-model-twelve-languages 用户实际感知的延迟：#the-latency-your-users-actually-notice 针对实时语音生成优化：#optimized-for-real-time-speech-generation 更快无意义，如果听起来不自然：#faster-doesnt-matter-if-it-doesnt-sound-natural 为什么开放权重很重要：#why-open-weights-matter 构建完整的语音代理——不仅仅是更好的语音：#build-complete-voice-agents--not-just-better-speech 开始使用：#get-started：https://cdn-uploads.huggingface.co/production/uploads/68127936de624fb6f57d9989/6ORCRF3ze_mI5BYb6hh9u.png

每一次语音交互都有一个延迟预算。

当用户听到你的应用响应时，你已经花费了宝贵的毫秒来捕捉音频、转录语音、运行大型语言模型 (LLM)、检索上下文以及生成响应。文本转语音 (TTS) 是最后一步——也是用户最注意到的一步。如果语音生成速度慢，整个体验就显得缓慢。

你自己能够运行和调优的管道越多，你就能从延迟预算中获得更多回报。

语音 AI 正在快速发展。集成语音模型提供了简便性——一次 API 调用，输入音频，输出音频——但它们牺牲了在你的领域微调每个组件的能力、在模型发布时替换更好的模型、强制数据驻留以及精确了解延迟来源。为了获得更多控制，级联架构——专用的 ASR、TTS 和 LLM 组件一起运行——可以保持每一层独立可调，并可部署在你自己的基础设施上。

NVIDIA Magpie 多语言 TTS：https://huggingface.co/nvidia/magpie_tts_multilingual_357m 就是为此而构建的。通过开放权重、生产就绪的 NVIDIA NIM：https://build.nvidia.com/nvidia/magpie-tts-multilingual，以及对 12 种语言的支持，你可以在自己的基础设施内部署多语言语音，为你的工作负载优化延迟，并为你的领域定制模型——从头到尾，完全在你自己的环境中。

最新版本通过加入现代标准阿拉伯语、韩语和巴西葡萄牙语来扩展多语言覆盖，同时通过更新的训练数据和模型改进提高了许多现有语言的质量。

无论您是在构建客户支持代理、医疗助理、企业辅助工具、翻译系统还是会话式 AI 应用，Magpie 都为生产级语音 AI 提供了开放的基础。

今天的语音应用不只服务单一语言。

全球客户支持、企业助手、医疗文档、零售自动化和翻译工作流越来越多地需要跨多语言的自然对话——同时保持低延迟。

支持更多语言只是挑战的一部分。开发者还需要具备以下能力：

开放模型改变了在每一种应用中可能实现的功能。

Magpie TTS 多语言模型是一个 3.64 亿参数的开放权重模型，支持：

英语 · 西班牙语 · 法语 · 德语 · 意大利语 · 越南语 · 普通话 · 印地语 · 日语 · 现代标准阿拉伯语（新增） · 韩语（新增） · 巴西葡萄牙语（新增）

每种语言都通过共享的多语言说话人表示提供男性和女性的语音。

此版本还通过支持印度语和日语的扩展代码切换，提升了多语言灵活性。这通过 IPA 字母-音素处理和自定义发音词典实现，使准确发音名称、技术术语和混合语言内容变得更容易。

开发者无需为不同地区维护独立的 TTS 模型，可以在单一开放基础上构建多语言应用。

在会话式 AI 中，文本转语音是用户听到响应前的最后阶段。这使得首次音频时间（TTFA）——从语音生成开始到第一段音频到达用户的延迟——成为语音管道中最重要的延迟指标之一。

由于 Magpie TTS 可以部署在您自己的环境中，您测量的延迟就是您实际可控的服务器端延迟，无需通过托管服务的往返。

来源：NVIDIA TTS NIM 性能文档：https://docs.nvidia.com/nim/speech/26.07.0/reference/performances/tts/performance.html（v26.07），三次试验平均结果，本地部署。TTFA = 首个音频延迟；RTFX = 实时倍数的吞吐量。

在 B200 上以 32ms 测得，Magpie 的 TTFA 将剩余的延迟预算留给 ASR 和 LLM 处理 —— 保持端到端总延迟在自然对话所需的 200ms 以下。跨 NVIDIA GPU，Magpie 在单条流上可在 32–79ms 内生成首个音频。在 64 条并发流下，B200 的 TTFA 达到 239ms，同时吞吐量为实时的 320 倍 —— 即使在并发负载下，也能生成音频的速度超过播放速度 300 倍以上。

上表显示 Magpie 作为 NVIDIA NIM 提供服务，并在本地测量 —— 即在您自己的 GPU 上运行的优化容器。开源的 Hugging Face 检查点是相同的模型，也是您进行研究和微调的路径；而 NIM 是调优的服务堆栈，可产生这些生产环境的延迟。两者都运行在您控制的硬件上。

因为模型运行在您自己的基础设施上，您可以直接进行性能基准测试，为部署进行调优，并根据工作负载进行扩展。对于实时语音代理，这决定了对话是感觉响应及时还是延迟感明显。

低延迟不是偶然的。Magpie 引入了两个互补的架构改进，以减少推理时间，同时保持语音质量。

帧堆叠。解码器在每个解码步骤中预测两个音频帧，而不是一个。这将解码器迭代次数减少一半，缩短生成时间并提高吞吐量。

局部 Transformer。单独的帧堆叠会通过在同时生成的码本 token 之间引入依赖关系而降低音频质量。局部 Transformer 对这些依赖关系进行建模并优化生成的音频，恢复帧堆叠可能牺牲的音质。

这些技术结合起来既能加快生成速度，又能实现自然语音合成。该架构在《Frame-Stacked Local Transformers for Efficient Multi-Codebook Speech Generation》中有详细描述：https://arxiv.org/abs/2509.19592（ICASSP 2026）。

此次发布不仅增加了语言支持，还提升了多种已有语言的合成质量。与上一个版本相比，Magpie 在多种语言上的字符错误率（CER）降低，讲话者相似度（SSIM）提高，其中法语和西班牙语的提升最为明显：

来源：Magpie TTS 多语言模型卡：https://huggingface.co/nvidia/magpie_tts_multilingual_357m。CER 越低越好；SSIM 越高越好。

新增加的阿拉伯语（1.62% CER）、韩语（2.69%）和巴西葡萄牙语（2.91%）模型为未来的改进建立了基线质量。

虽然客观指标有助于衡量进展，但语音质量最终取决于感知。你可以在 NVIDIA Build 上亲自感受差异：https://build.nvidia.com/nvidia/magpie-tts-multilingual 或 Hugging Face 演示：https://huggingface.co/spaces/nvidia/magpie_tts_multilingual_demo。

可测量的延迟很有用。可控制的延迟更佳。

开放权重赋予开发者来自部署拥有的能力。使用 Magpie，你可以：

对于建设生产语音 AI 的企业而言，能够控制部署、性能和定制化通常是最重要的。

生产中的语音 AI 是一个由多个模型组成的系统，而非单一模型。Magpie TTS 是 NVIDIA Nemotron Voice Agent 开发示例的一部分：https://build.nvidia.com/nvidia/nemotron-voice-agent，这是一个参考实现，展示了专用语音、语言和推理模型如何协调工作——因此你可以构建全天候语音代理，而不仅仅是更好听的语音。

Nemotron Voice Agent 开发示例提供了一个端到端的参考实现，开发者可以克隆、定制并在数小时内部署。它包含用于生产的模式：

开发者无需从零组装各个组件，可以从完整的参考架构出发，并根据自己的应用进行适配。

推荐的推理配置：

使用 MagpieTTS 的 Nvidia 文本转语音。

Voice AI Is Becoming Multilingual by Default ：#voice-ai-is-becoming-multilingual-by-default One Open Model, Twelve Languages ：#one-open-model-twelve-languages The Latency Your Users Actually Notice ：#the-latency-your-users-actually-notice Optimized for Real-Time Speech Generation ：#optimized-for-real-time-speech-generation Faster Doesn't Matter If It Doesn't Sound Natural ：#faster-doesnt-matter-if-it-doesnt-sound-natural Why Open Weights Matter ：#why-open-weights-matter Build Complete Voice Agents — Not Just Better Speech ：#build-complete-voice-agents--not-just-better-speech Get Started ：#get-started ：https://cdn-uploads.huggingface.co/production/uploads/68127936de624fb6f57d9989/6ORCRF3ze_mI5BYb6hh9u.png

Every voice interaction has a latency budget.

By the time a user hears your application respond, you've already spent precious milliseconds capturing audio, transcribing speech, running an LLM, retrieving context, and generating a response. Text-to-speech (TTS) is the final step — and the one users notice most. If speech generation is slow, the whole experience feels slow.

The more of that pipeline you can run and tune yourself, the more of the latency budget you get back.

Voice AI is moving fast. Integrated speech models offer simplicity — one API call, audio in, audio out — but they trade the ability to fine-tune each component for your domain, swap in better models as they ship, enforce data residency, and understand exactly where latency is coming from. For more control, a cascaded architecture — purpose-built ASR, TTS, and LLM components running together — keeps each layer independently tunable and deployable on infrastructure you own.

NVIDIA Magpie Multilingual TTS：https://huggingface.co/nvidia/magpie_tts_multilingual_357m is built for that. With open weights, production-ready NVIDIA NIM：https://build.nvidia.com/nvidia/magpie-tts-multilingual, and support for 12 languages, you can deploy multilingual speech inside your own infrastructure, optimize latency for your workload, and customize the model for your domain — end to end, in your own environment.

The latest release expands multilingual coverage with Modern Standard Arabic, Korean, and Brazilian Portuguese, while improving quality across many existing languages through updated training data and model improvements.

Whether you're building customer support agents, healthcare assistants, enterprise copilots, translation systems, or conversational AI applications, Magpie provides an open foundation for production voice AI.

Today's voice applications don't serve a single language.

Global customer support, enterprise assistants, healthcare documentation, retail automation, and translation workflows increasingly require natural conversations across multiple languages — all while maintaining low latency.

Supporting more languages is only part of the challenge. Developers also need the ability to:

Open models change what's possible on every one of these.

Magpie TTS Multilingual is a 364M-parameter open-weights model supporting:

English · Spanish · French · German · Italian · Vietnamese · Mandarin · Hindi · Japanese · Modern Standard Arabic (new) · Korean (new) · Brazilian Portuguese (new)

Each language includes male and female speaker voices through a shared multilingual speaker representation.

This release also improves multilingual flexibility with expanded code-switching support for Hindi and Japanese, enabled through IPA grapheme-to-phoneme processing and custom pronunciation dictionaries — making it easier to accurately pronounce names, technical terminology, and mixed-language content.

Instead of maintaining separate TTS models for different regions, developers can build multilingual applications on a single open foundation.

In conversational AI, text-to-speech is the final stage before users hear a response. That makes Time to First Audio (TTFA) — the delay between speech generation beginning and the first audio reaching the user — one of the most important latency metrics in a voice pipeline.

Because Magpie TTS can be deployed inside your own environment, the latency you measure is the server-side latency you actually control, with no managed-service round-trip in the number.

Source: NVIDIA TTS NIM Performance documentation：https://docs.nvidia.com/nim/speech/26.07.0/reference/performances/tts/performance.html (v26.07), average of three trials, on-prem. TTFA = latency to first audio; RTFX = throughput as a multiple of real time.

At 32ms on B200, Magpie's TTFA leaves the rest of the latency budget for ASR and LLM processing — keeping total end-to-end latency within the sub-200ms window natural conversation requires. Across NVIDIA GPUs, Magpie delivers first audio in 32–79ms on a single stream. At 64 concurrent streams, B200 reaches 239ms TTFA while delivering throughput at 320× real time — generating audio more than 300 times faster than it plays back, even under concurrent load.

The table above shows Magpie served as the NVIDIA NIM, measured on-prem — the optimized container running on your own GPU. The open Hugging Face checkpoint is the same model and your path for research and fine-tuning; the NIM is the tuned serving stack that produces these production latencies. Both run on hardware you control.

Because the model runs on your own infrastructure, you can benchmark performance directly, tune it for your deployment, and scale according to your workload. For real-time voice agents, that's the difference between conversations that feel responsive and conversations that feel delayed.

Low latency isn't accidental. Magpie introduces two complementary architectural improvements that reduce inference time while maintaining speech quality.

Frame stacking. The decoder predicts two audio frames during each decoding step rather than one. This cuts the number of decoder iterations in half, shortening generation time and improving throughput.

Local transformer. Frame stacking alone would reduce audio quality by introducing dependencies between simultaneously generated codebook tokens. The local transformer models those dependencies and refines the generated audio, recovering the quality that frame stacking would otherwise sacrifice.

Together, these techniques deliver both faster generation and natural speech synthesis. The architecture is described in Frame-Stacked Local Transformers for Efficient Multi-Codebook Speech Generation：https://arxiv.org/abs/2509.19592 (ICASSP 2026).

This release doesn't only add languages — it also improves synthesis quality across many existing ones. Compared to the previous release, Magpie shows reduced character error rates (CER) and higher speaker similarity (SSIM) on several languages, with the clearest gains on French and Spanish:

Source: Magpie TTS Multilingual model card：https://huggingface.co/nvidia/magpie_tts_multilingual_357m. CER lower is better; SSIM higher is better.

The newly added Arabic (1.62% CER), Korean (2.69%), and Brazilian Portuguese (2.91%) models establish baseline quality for future improvements.

While objective metrics help measure progress, speech quality is ultimately perceptual. You can hear the difference yourself on NVIDIA Build：https://build.nvidia.com/nvidia/magpie-tts-multilingual or the Hugging Face demo：https://huggingface.co/spaces/nvidia/magpie_tts_multilingual_demo.

Latency you can measure is useful. Latency you can control is even better.

Open weights give developers capabilities that come from owning the deployment. With Magpie you can:

For enterprises building production voice AI, this control over deployment, performance, and customization is often what matters most.

Voice AI in production is a system of models, not a single one. Magpie TTS is part of the NVIDIA Nemotron Voice Agent Developer Example：https://build.nvidia.com/nvidia/nemotron-voice-agent, a reference implementation showing how purpose-built speech, language, and reasoning models work together as a coordinated system — so you can build always-on voice agents, not just better-sounding speech.

The Nemotron Voice Agent developer example provides an end-to-end reference implementation that developers can clone, customize, and deploy in hours. It includes production patterns for:

Rather than assembling individual components from scratch, developers can start from a complete reference architecture and adapt it to their own applications.

Recommended inference configuration:

Nvidia Text-to-Speech with MagpieTTS.