NVIDIA 发布 NemotronLabs VoiceChat 11B：开源全双工语音模型，支持约 450 毫秒轮换与实时工具调用

NVIDIA 发布了 NemotronLabs VoiceChat 11B：https://huggingface.co/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B，这是一个开源的 11B 端到端语音到语音模型，用于实时全双工对话。它不再是将 ASR、LLM 和 TTS 链接在一起，而是在一个统一网络中执行流式语音理解和语音生成。这消除了级联堆栈所需的多模型协调和 API 交接，并降低了端到端延迟：在 Full-Duplex-Bench 1.0 上测量的平滑轮换延迟为 448 毫秒：https://arxiv.org/abs/2503.04721。模型在说话时可以监听，因此用户可以在对话中途插入而代理会让位，接管率在 480 毫秒时为 1.00。它还是第一个在对话保持流畅的同时支持调用工具的开源全双工模型，使用单独的输出通道传递脚本，并配备操作员定义的“占位”语句填补 API 运行期间的空隙。

部分可用——今天可用于试点部署，但不适用于生产环境。权重和容器均公开，且许可宽松。但 NVIDIA 团队表示，该检查点“仅适用于研究目的”，并且仓库记录了实际的失败模式：音频上下文上限为两分钟，多轮对话后可能退化为不可恢复的乱码，轮次结束后自言自语失控，以及用户转录中可能丢词。

该模型是 Mamba/Transformer 混合体，由三个现有 NVIDIA 组件和一个新输出路径组装而成：

输出包括代理音频、代理文本以及正在生成的用户转录。训练使用了大约 550k 小时的真实与合成语料音频，基于 SALM-Duplex：https://arxiv.org/abs/2505.15670 和 Audio Flamingo 3：https://arxiv.org/abs/2507.08128。

工具调用通过侧通道以块形式发出；你的代码返回结果也以块呈现。值得注意的是占位消息：每个工具由操作员定义一条线路，模型生成触发调用的文本时，代理立即说出该线路，以便 API 运行时对话不会陷入沉默。

约束明确。NVIDIA 建议每个会话最多使用五个工具，模型无法可靠地同时调用多个工具，并且用户无法在工具执行过程中打断代理。系统提示和工具响应必须为仅 ASCII 且适合 TTS 的内容。

在 Full-Duplex-Bench 1.0：https://arxiv.org/abs/2503.04721：平滑轮流说话 TOR 0.82 在 448 毫秒，用户打断 TOR 1.00 在 480 毫秒，暂停处理 TOR 为 0.153（合成数据）和 0.255（Candor），值越低越好。

在 AU Harness：https://github.com/ServiceNow/AU-Harness BFCL-v3 语音工具调用：58.5% 简单，62.5% 多个，42.5% 并行，27.5% 并行-多个，89.6% 无关，56.1% 平均。在 Full-Duplex-Bench v3：https://arxiv.org/abs/2604.04847：82.5% 工具选择，44.2% 参数准确率，33% pass@1。

NVIDIA 报告称该模型在 VoiceBench：https://arxiv.org/abs/2410.17196 的开源全双工模型中排名第 #2，在 Full-Duplex-Bench 1.0 的开源模型中也排名第 #2。

需要与我们合作推广您的 GitHub 仓库或 Hugging Face 页面或产品发布或网络研讨会等吗？请联系我们：https://forms.gle/wbash1wF6efRj8G58

Asif Razzaq 是 Marktechpost Media Inc. 的首席执行官。作为一位有远见的企业家和工程师，Asif 致力于利用人工智能的潜力造福社会。他最近的努力是推出一个人工智能媒体平台 Marktechpost，该平台以其对机器学习和深度学习新闻的深入报道而脱颖而出，这些报道既技术可靠，又易于广泛受众理解。该平台每月浏览量超过 200 万次，显示了其在受众中的受欢迎程度。

NVIDIA has released NemotronLabs VoiceChat 11B：https://huggingface.co/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B, an open 11B end-to-end speech-to-speech model for real-time, full-duplex conversation. Instead of chaining ASR, an LLM, and TTS, it performs streaming speech understanding and speech generation in one unified network. That removes the multi-model orchestration and API handoffs a cascaded stack requires, and cuts end-to-end latency: measured smooth turn-taking latency is 448 ms on Full-Duplex-Bench 1.0：https://arxiv.org/abs/2503.04721. The model listens while it speaks, so a user can barge in mid-turn and the agent yields, with a take-over rate of 1.00 at 480 ms. It is also first open full-duplex model to support tool calling while conversation keeps flowing, using a separate output channel for scripts along with operator-defined “on-hold” lines that fill the gap while an API runs.

PARTIAL — deployable today for pilots, not for production. Weights and container are both public, and the license is permissive. But NVIDIA team states the checkpoint is ‘ready for research purposes only,’ and the repo documents real failure modes: a two-minute audio context ceiling, degradation into non-recoverable gibberish after several turns, runaway self-talk after a turn ends, and dropped words in user transcription.

The model is a hybrid Mamba/Transformer, assembled from three existing NVIDIA components along with one new output path:

Outputs include agent audio, agent text, and a running user transcription. Training used roughly 550k hours of audio across real and synthetic corpora, building on SALM-Duplex：https://arxiv.org/abs/2505.15670 and Audio Flamingo 3：https://arxiv.org/abs/2507.08128.

Tool calls are emitted on the side channel as a block; your code returns results in a block. The notable piece is the on-hold message : per tool, an operator defines a line the agent speaks the moment the model generates the text triggering the call, so the conversation does not fall silent while an API runs.

Constraints are explicit. NVIDIA recommends a maximum of five tools per session, the model cannot reliably call multiple tools simultaneously, and the user cannot interrupt the agent during tool execution. System prompts and tool responses must be ASCII-only and TTS-friendly.

On Full-Duplex-Bench 1.0：https://arxiv.org/abs/2503.04721: smooth turn-taking TOR 0.82 at 448 ms, user-interruption TOR 1.00 at 480 ms, and pause-handling TOR of 0.153 (synthetic) and 0.255 (Candor), where lower is better.

On AU Harness：https://github.com/ServiceNow/AU-Harness BFCL-v3 spoken tool calling: 58.5% simple, 62.5% multiple, 42.5% parallel, 27.5% parallel-multiple, 89.6% irrelevance, 56.1% average . On Full-Duplex-Bench v3：https://arxiv.org/abs/2604.04847: 82.5% tool selection, 44.2% argument accuracy, 33% pass@1.

NVIDIA reports the model ranks #2 among open full-duplex models on VoiceBench：https://arxiv.org/abs/2410.17196 and #2 among open models on Full-Duplex-Bench 1.0.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us ：https://forms.gle/wbash1wF6efRj8G58

Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.