Prime Intellect 发布 Prime Inference：面向开源前沿模型的 Serverless 与预留推理服务

Prime Intellect 已推出 Prime Inference：https://www.primeintellect.ai/blog/prime-inference，这是一个面向前沿开源模型的服务平台。它提供无服务器端点和 Prime 自有 GPU 的预留容量，覆盖多个数据中心。在公开发布前，它每天内部处理了近一万亿个 token。该流量来自 RL 回滚、合成数据生成、评估以及长时间运行的编码代理。

Prime Inference 是 Prime Intellect 开放训练堆栈的服务层。公司已经提供训练后工具，如 prime-rl、验证器和沙盒。服务层完成了这一闭环：部署的模型生成的生产轨迹可以反馈回训练环节。Prime 报告其 GLM-5.3：https://huggingface.co/zai-org/GLM-5.3 端点在 OpenRouter 上排名最快之一。它还提到几乎为零的工具调用错误率以及自发布以来 100% 的正常运行时间。

该堆栈结合了 NVIDIA Dynamo：https://github.com/ai-dynamo/dynamo、vLLM：https://github.com/vllm-project/vllm、Mooncake：https://github.com/kvcache-ai/Mooncake 和 FlashInfer：https://github.com/flashinfer-ai/flashinfer。它是与 Inferact 和 NVIDIA 一起构建的，修复内容也贡献给了上游项目。

目标工作负载是代理驱动型。一个典型的代理回合会向 140K token 的提示中增加约 6K token。Prime 通过 SemiAnalysis AgentX：https://inferencex.semianalysis.com/agentx，以及注入的冷启动请求来基准测试这种混合负载。

预填充/解码拆分：预填充和解码在不同的 GPU 组上运行。Dynamo 负责路由，vLLM 在每个组上运行模型。解码器通过 NIXL：https://github.com/ai-dynamo/nixl 拉取计算好的 KV。Prime 报告其测试中的 p90 token 间延迟降低了近 40%。

缓存感知路由：Dynamo 的 KV 感知路由器会权衡缓存前缀重叠与排队任务。会话在回合之间保持在同一解码器上。Mooncake 在主机 DRAM 中增加了第二级 KV。

互动性目标是每位用户每秒 100 个端到端 token。在该指标下，1:4 的预填充/解码比率服务了最多用户。它在每个预填充组达到 66 个会话，每位用户 101 tok/s，每个 GPU 输出 100 tok/s。

当工具调用使用错误的名称或损坏的参数时，代理会失败。Prime Intellect 的团队为 Dynamo 的 GLM 工具格式贡献了一个结构标签构建器。vLLM 然后使用 xgrammar：https://github.com/mlc-ai/xgrammar 来屏蔽违反工具模式的标记。团队还修复了解析错误，包括被解码到内部代码中。

竞争对手价格已于 2026 年 10 月 2 日验证。跟踪数据来自 ComputePrices，一家第三方价格跟踪工具。

需要与我们合作以推广您的 GitHub 仓库、Hugging Face 页面、产品发布或网络研讨会等吗？请联系：https://forms.gle/MJjjVDPS7whH8Ngs6

Datalab 推出了 OmniExtractBench，以解决提取基准中的偏差和不透明问题：https://pxllnk.co/hxplrq

面向从业者的 AI/ML 新闻和分析，每月有超过 100 万开发者和研究人员阅读。

Prime Intellect has launched Prime Inference：https://www.primeintellect.ai/blog/prime-inference, a serving platform for frontier open-source models. It offers serverless endpoints and reserved capacity on Prime’s own GPUs across multiple datacenters. Before public release, it processed nearly a trillion tokens per day internally. That traffic came from RL rollouts, synthetic data generation, evaluations and long-running coding agents.

Prime Inference is the serving layer of Prime Intellect’s open training stack. The company already ships post-training tools such as prime-rl, verifiers and sandboxes. Serving closes that loop: deployed models generate production traces that can feed back into training. Prime reports its GLM-5.3：https://huggingface.co/zai-org/GLM-5.3 endpoint ranks among the fastest on OpenRouter. It also cites a near-zero tool-call error rate and 100% uptime since launch.

The stack combines NVIDIA Dynamo：https://github.com/ai-dynamo/dynamo, vLLM：https://github.com/vllm-project/vllm, Mooncake：https://github.com/kvcache-ai/Mooncake and FlashInfer：https://github.com/flashinfer-ai/flashinfer. It was built with Inferact and NVIDIA, and fixes are contributed upstream.

The target workload is agentic. A typical agent turn adds about 6K tokens to a 140K-token prompt. Prime benchmarks this mix with SemiAnalysis AgentX：https://inferencex.semianalysis.com/agentx, and injected cold arrivals.

Prefill/decode disaggregation : Prefill and decode run on separate GPU groups. Dynamo handles routing, and vLLM runs the model on each group. Decoders pull computed KV through NIXL：https://github.com/ai-dynamo/nixl. Prime reports nearly 40% lower p90 inter-token latency in its tests.

Cache-aware routing :Dynamo’s KV-aware router weighs cached prefix overlap against queued work. Sessions stay on the same decoder between turns. Mooncake adds a second KV tier in host DRAM.

The interactivity target was 100 end-to-end tokens per second per user. At that bar, a 1:4 prefill/decode ratio served the most users. It reached 66 sessions per prefill group at 101 tok/s per user and 100 output tok/s per GPU.

Agents fail when tool calls carry wrong names or broken arguments. Prime Intellect’s team contributed a structural-tag builder to Dynamo for GLM’s tool format. vLLM then uses xgrammar：https://github.com/mlc-ai/xgrammar to mask tokens that violate the tool schema. The team also fixed parsing bugs, including being decoded into inside code.

Competitor prices verified October 2, 2026. Tracker figures come from ComputePrices, a third-party price tracker.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us ：https://forms.gle/MJjjVDPS7whH8Ngs6

Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks ：https://pxllnk.co/hxplrq

Practitioner-first AI/ML news and analysis, read by 1M+ developers and researchers every month.