Prime Intellect 发布推理平台 Prime Inference，已上线 GLM-5.3 端点

Prime 的使命是构建前沿开放模型和开放超级智能堆栈，以持续改进智能体。我们已经提供了端到端的训练后基础设施，从 prime-rl 和验证器到沙盒和强化学习环境。但持续学习循环只有在训练好的模型能够服务真实用户、生成新经验并将这些生产痕迹反馈到训练中时才算完整。

我们今天很高兴发布 Prime 推理（Prime Inference）。它涵盖无服务器端点和保留容量，并在我们的多数据中心 GPU 基础设施上稳定提供前沿开源模型的服务。

Prime 推理最初是作为我们自己所需的服务平台。早在公开发布之前，它就支持大规模强化学习部署、合成数据生成、评估和长期运行的编码智能体，在内部每天处理近一万亿个令牌。除了我们自己的工作负载，我们自今年一月起也在生产环境中为大规模客户部署提供服务。这种规模推动我们优化持续性能、质量和可靠性，而不仅仅是基准速度。

我们的首次公开部署 GLM-5.3 于 9 月 22 日在 OpenRouter 上上线。它目前在 OpenRouter 上排名最快的 GLM-5.3 端点之一，工具调用错误率接近于零，自上线以来保持 100% 正常运行时间。

或将任何 OpenAI SDK 指向 https://api.pinference.ai/api/v1。查看文档：https://docs.primeintellect.ai/inference/overview 获取完整 API 参考。

Prime 推理将公共 API 与模型群组分离，因此容量可以移动、失败或扩展而不更改客户端端点。

我们的共享断路器使每个网关副本能够一致地应对故障，而基于租约的准入控制可以防止过载，并在进程消失时自动恢复容量。

在软件层之下，每个集群都持续接受监控，健康检查覆盖到 NVLink 和 InfiniBand。警报发送给全天候值班团队，因此 GPU 故障能够迅速被发现和修复，而不是逐渐降低服务质量。由于 Prime 保持了大量备用容量，我们可以在需要更多容量时重新路由流量并启动新部署。

这些保护措施共同保证了服务的可用性。接下来的章节将通过我们在 GB200 NVL72 上提供 GLM-5.3 的工作，深入了解生产部署。

一个典型的代理轮次会向一个 140K 令牌的提示中添加约 6K 令牌，同时重用大部分对话历史。在高负载下，这些返回的会话会与具有长且未缓存提示的新请求并行运行。

我们使用 SemiAnalysis 的 AgentX：https://inferencex.semianalysis.com/agentx 对这种混合情况进行基准测试，该工具可重放多轮代理会话。我们的基准测试框架还会注入带有长提示的冷启动请求。我们测量每位用户的端到端每秒令牌数以评估交互性，以及每块 GPU 的每秒输出令牌数以评估效率。

在共享 GPU 上，处理长提示可能会中断现有会话的令牌生成。分块预填可以限制这些中断，但两种工作负载仍然需要争夺 GPU 时间。

我们在独立的 GPU 组上运行预填和解码。NVIDIA Dynamo 处理路由和调度，而 vLLM 在每个组上运行模型。一旦预填完成，解码器通过 NIXL 拉取计算好的 KV 并将请求加入其批次。

通过 Dynamo 协调独立的预填和解码池，我们在测试中将 p90 令牌间延迟降低了近 40%。

Dynamo 的 KV 感知路由器会基于提示缓存的数量以及该工作队列的工作量选择预填工作器。工作器发布缓存更新，使路由器能够跟踪前缀的位置。我们还在轮次之间将会话保留在同一个解码器上，以支持 KV 重用。

Mooncake 在宿主机 DRAM 中提供第二层缓存。从 GPU 内存卸载的前缀可以被检索而不用重新计算，从而允许我们保留更多的对话历史。

在建立此架构后，我们对 GB200 NVL72 上的 GLM-5.3 进行了调整，以同时实现三个目标：交互速度、模型质量和并发性。

长上下文代理服务既是缓存管理问题，也是计算问题。因此，性能取决于保留历史记录、及时调度新任务以及在不中断正在生成的情况下移动缓存状态。

我们的交互目标是每位用户每秒 100 个端到端令牌。我们根据可以在该速度下支持的并发会话数量进行调整。

我们分别优化了这些路径：预填充拓扑和调度以减少首个 token 的时间；压缩 KV 和融合注意力内核以支持低延迟解码；以及便于传输的缓存布局以减少在工作节点之间移动 KV 的开销。

在每用户 100 tok/s 的标准下，1:4 的 P/D 比例服务最多：每个预填充组 66 个会话，101 tok/s/用户，每个 GPU 输出 100 个 token。

对于希望了解工程细节的读者，本篇文章的其余部分会深入讲解每项优化，并介绍我们在可靠工具调用方面的工作。

下面的章节涵盖了我们在拓扑、调度、压缩注意力机制以及 KV 传输方面的工作：

首个 token 的时间不仅取决于处理提示词。一个请求可能需要检索缓存历史、等待准入、计算新 token，并将 KV 传输到解码器。我们调查了这一路径上的延迟，从缓存容量和调度开始。

我们选择了 DEP8：八个数据并行注意力等级，并在组内采用专家并行。DEP8 在八个注意力工作节点之间分配请求，同时共享组内模型的专家。利用我们使用的 MLA 缓存布局，TEP8 会在所有八个等级中复制每个请求的 KV，而 DEP8 允许各等级缓存不同请求。即使考虑到这需要额外的权重内存，与 TEP8 这种拓扑相比，我们在相同硬件上大约拥有五倍的可用前缀缓存容量，同时也进行了基准测试。

DEP8 的缺点是 DP 等级的同步：每个 DP 等级处理不同请求，但在每个 MoE 层都参与相同的全连接通信，因此即使是空闲等级也可能需要运行前向计算以跟上其他等级的进度。

随着 EP 扩展，调度开销会进一步增加。我们发现单个 DEP16 比两个 DEP8 组慢 17.9%，其中合并和最终处理的增长最明显。

缓存容量并不能消除调度延迟。我们发现请求的缓存 KV 可能已经可用，但请求仍在等待进入运行批次。

工作人员在前向传递之间检查已完成的负载。结果然后经过批处理队列和调度器，在那里，准备就绪的请求可能会错过当前的决策并等待下一步。在负载下，已在处理中的请求可以填充下一个预填充批次，即使缓存的历史记录已就绪，也会延迟入队。这会产生调度器气泡。

为了减少这种气泡，我们将每个预填充步骤处理中提示令牌的数量减半，从每个 GPU 处理 8K 降到 4K。较短的步骤使等待的请求可以更早开始。在我们的配置下，中位队列等待时间从550毫秒降至110毫秒，首次生成令牌的中位时间大约减少了20%。

较小的步骤会为长且未缓存的提示增加开销，但这种权衡对我们的工作负载有效，因为大多数对话轮次都重用了已有前缀。

在我们目标负载下，TP4 在我们测试的配置中提供了最低的令牌间延迟。在相同的 GPU 预算下，我们可以运行更多的解码引擎，每个引擎运行的会话更少。

然而，TP 拓扑会带来 KV 容量的权衡。TP 将模型权重分布在 GPU 之间，但根据我们使用的 MLA 缓存布局，每个 rank 仍然持有请求的完整潜在 KV 副本，因此 TP 存储了 KV 副本，这也增加了 NIXL 从预填充到解码传输的数据量。TP4 存储的 KV 副本比 TP8 少，但每个 GPU 持有更多的模型权重。我们希望在保持 TP4 延迟优势的同时，将更多 KV 适配到每个 GPU 可用的内存中。

DEP 通过将请求分配给独立的注意力 rank 来避免这种复制，因而有更多可用的 KV 容量，但在测试中，我们发现 MoE 分发前 DP rank 之间的同步增加了解码延迟。

我们还测试了解码上下文并行（DCP），它在保留模型张量并行的同时，将缓存序列分配到不同的计算级别。这减少了KV的重复，但引入了通信，用于收集查询、合并选定候选项以及组合部分注意力输出。对于稀疏注意力，每个查询最多读取2048个缓存的token，因此一开始要跨级别拆分的注意力工作就不多。在我们的测试中，将这项工作分布到不同级别所节省的计算量不足以抵消额外通信的开销。因此，我们保持了TP4，并寻找增加KV容量的方法。

我们使用NVFP4压缩了512值的MLA潜变量：每个值四位，每16个值一组使用FP8比例因子。64值的位置组件保持FP8。包括比例因子在内，每行MLA缓存从576字节缩小到352字节。

在索引器和其他状态保持不变的情况下，总缓存容量约增加50%，在相同内存预算下，每个解码器的缓存token从109万增加到163万。

我们的初始实现是在调用现有注意力内核之前，将选定的行解包到GPU内存的临时FP8缓冲区中。这给我们带来了容量的好处，但每一层都需要为转换以及通过GPU内存的额外写入和读取付出代价。

我们构建了原生稀疏MLA内核以去除中间缓冲区。它使用稀疏索引器选定的位置，加载压缩的行，并在注意力需要时在芯片上解包它们。NVFP4是存储格式；注意力计算使用FP16操作数，并采用FP32累加。

对于每个查询token，一组协作的线程块将选定的行分配任务。在GB200配置中，每个线程块群覆盖三到八个SM。各块通过分布式共享内存合并部分结果，无需第二次内核启动。

在每个SM内部，单独的warp组解包行、计算注意力分数和softmax，并累加输出。

三个变化特别有用：

在每次启动15个查询token时，优化后的内核在GB200上大约耗时12.0 μs，而分阶段NVFP4路径为17.7 μs，FP8注意力为13.7 μs。这些结果适用于该工作负载，并不表示在每个批次大小下都比FP8更快。

在32个并发会话中，注意力占解码步骤的约8%，因此更大的实际收益来自于保留更多的KV。NVFP4在启用解码前缀缓存的情况下，为我们提供了大约50%的KV缓存容量，同时保持了相当的用户速度。

我们运行了严格的评估套件，特别强调长上下文任务，以确保NVFP4 KV压缩不降低准确性。

我们把原生内核贡献给 FlashInfer 作为实验操作。

压缩提高了译码能力，但拆分又带来了另一个瓶颈：将KV高效地从预填充转移到解码。

我们使用 vLLM 的 NIXL 连接器，让每个解码器通过多节点 NVLink 从预填充工作者中拉取 KV。早期对比中，NVLink 配置相较于 InfiniBand 增加了大约 292 毫秒的首个令牌时间。更快的互连并未转化为更快的服务。

瓶颈是传输碎片化。在我们分析的路径中，每个传输描述符都变成了小型设备间副本。单个20万令牌请求到达一个TP4等级时，触发了32K副本。提交和处理这么多小操作限制了传输，才能有效利用NVLink的带宽。

最初的层主 KV 布局 LBHNC 是分别存储每一层缓存。因此，跨模型传输逻辑块需要在不同层为其数据设置不同的描述符。

增加块大小有所帮助：使用1,024令牌块减少了传输描述符数量。然而，更大的块仍会使数据在多层间分散，且可能通过部分填充的块浪费更多空间来降低KV缓存的利用率。

随后我们测试了 BLHNC，这是一种由 vLLM 社区最近推出的块主布局。它将多个层块的数据集中存储在内存中，使 NIXL 能够以更少且更大的复制操作转移这些连续区域。我们感谢 vLLM 社区开发并分享了这一布局。

在一次单独的 TP8 比较中，按层主序布局（LBHNC）使用了 1,024 令牌块，而按块主序布局（BLHNC）使用了 64 令牌块。尽管 BLHNC 使用的块小 16 倍，但它将描述符数量从 19,559 降至约 1,940 —— 大约少了 10 倍 —— 并将平均传输时间从 146 毫秒降至 78 毫秒，减少约 47%。BLHNC 布局使我们能够使用粒度更细的 KV 块，同时减少传输开销。

性能只是生产代理服务的一部分。代理还需要可靠的工具调用来读取文件、运行命令和进行编辑。如果模型调用了错误的工具或生成了不可用的参数，代理必须重试或停止。

在我们的配置中，Dynamo 将模型生成的工具调用转换为代理接收的 API 响应。在大规模提供 GLM-5.3 服务时，我们发现了几种反复出现的失败模式。最初，我们发现系统可能会静默地丢弃对未声明工具的调用并返回正常停止，使代理没有可执行的操作或可恢复的错误。

其他调用存在缺失的参数或类型错误，即使客户端明确要求模型遵循工具的输入格式。模型收到了这些定义，但我们的 GLM 服务路径在生成时并未强制执行它们。

Dynamo 已经有一个机制来处理此问题，但不支持 GLM 格式的结构标签。我们向 Dynamo 提供了一个结构标签生成器，将工具定义转化为限制模型可生成名称和参数的规则。

vLLM 使用 xgrammar 在解码时强制执行这些规则，屏蔽那些会违反工具调用语法的令牌。我们还修复了解析过程中引入的错误：字面字符串如 被转换为，改变了代码或文件内容，同时一些模式引用和可空参数类型被错误解释。

我们测试了最终的 API 响应，内容包括：

我们维护自己的工具调用测试套件，涵盖这些情况，验证最终 API 响应，并在更新服务栈时检查回归。

Prime's mission is to build frontier open models and the open superintelligence stack for continuously improving agents. We already provide end-to-end post-training infrastructure, from prime-rl and verifiers to sandboxes and RL environments. But the continual learning loop is not complete until a trained model can serve real users, generate new experience, and feed those production traces back into training.

We're excited to release Prime Inference today. It covers both serverless endpoints and reserved capacity and offers resilient serving of frontier open-source models on our GPU infrastructure across multiple datacenters.

Prime Inference began as the serving platform we needed ourselves. Long before public release, it powered large-scale RL rollouts, synthetic data generation, evaluations, and long-running coding agents, processing nearly a trillion tokens every day just internally. Beyond our own workloads, we've also been serving large-scale customer deployments in production since January. This scale pushed us to optimize for sustained performance, quality, and reliability, rather than benchmark speed alone.

Our first public deployment, GLM-5.3, went live on OpenRouter on September 22. It currently ranks among the fastest GLM-5.3 endpoints on OpenRouter, with a near-zero tool-call error rate and 100% uptime since launch.

Or point any OpenAI SDK at https://api.pinference.ai/api/v1 . See the docs：https://docs.primeintellect.ai/inference/overview for the full API reference.

Prime Inference separates the public API from the model fleet, so capacity can move, fail, or scale without changing the client endpoint.

Our shared circuit breakers let every gateway replica react to failures consistently, while lease-based admission control prevents overload and automatically recovers capacity when a process disappears.

Below the software layer, every cluster is continuously monitored, with health checks that reach all the way down to NVLink and InfiniBand. Alerts go to an on-call team staffed 24/7, so GPU failures are caught and repaired quickly instead of slowly degrading service. And because Prime maintains significant overflow capacity, we can reroute traffic and bring up new deployments whenever more capacity is needed.

Together, these safeguards keep the service available. The next sections look inside a production deployment through our work serving GLM-5.3 on GB200 NVL72.

A typical agent turn adds about 6K tokens to a 140K-token prompt, reusing most of the conversation history. Under load, these returning sessions run alongside new requests with long, uncached prompts.

We benchmark this mix with AgentX：https://inferencex.semianalysis.com/agentx from SemiAnalysis, which replays multi-turn agent sessions. Our benchmark harness also injects cold arrivals with long prompts. We measure end-to-end tokens per second per user for interactivity and output tokens per second per GPU for efficiency.

On shared GPUs, processing a long prompt can interrupt token generation for existing sessions. Chunked prefill limits these interruptions, but both workloads still compete for GPU time.

We run prefill and decode on separate GPU groups. NVIDIA Dynamo handles routing and orchestration, while vLLM runs the model on each group. Once prefill finishes, the decoder pulls the computed KV through NIXL and adds the request to its batch.

With Dynamo coordinating separate prefill and decode pools, we reduced p90 inter-token latency by nearly 40% in our tests.

Dynamo's KV-aware router chooses a prefill worker based on how much of the prompt it already has cached and how much work is queued there. Workers publish cache updates so the router can track where prefixes are available. We also keep sessions on the same decoder between turns to support KV reuse.

Mooncake provides a second cache tier in host DRAM. Prefixes offloaded from GPU memory can be retrieved instead of recomputed, allowing us to retain more conversation history.

With this architecture in place, we tuned GLM-5.3 on GB200 NVL72 for three goals at once: interactive speed, model quality, and concurrency.

Long-context agentic serving is as much a cache-management problem as a compute problem. Performance therefore depends on retaining that history, scheduling new work promptly, and moving cached state without interrupting ongoing generation.

Our interactivity target was 100 end-to-end tokens per second per user. We tune for the number of concurrent sessions we can support at that speed.

We optimized these paths separately: prefill topology and scheduling to reduce time to first token; compressed KV and a fused attention kernel to support low-latency decoding; and a transfer-friendly cache layout to reduce the overhead of moving KV between workers.

At the 100 tok/s/user bar, a 1:4 P/D ratio serves the most: 66 sessions per prefill group at 101 tok/s/user and 100 output tok/s per GPU.

For readers who want the engineering details, the rest of this post walks through each optimization in depth, followed by our work on reliable tool calls.

The sections below cover our work on topology, scheduling, compressed attention, and KV transfer:

Time to first token depends on more than processing the prompt. A request may need to retrieve cached history, wait for admission, compute new tokens, and transfer KV to a decoder. We investigated delays across this path, starting with cache capacity and scheduling.

We chose DEP8: eight data-parallel attention ranks with expert parallelism across the group. DEP8 distributes requests across eight attention workers while sharing the model's experts across the group. With the MLA cache layout we used, TEP8 replicated each request's KV across all eight ranks, while DEP8 let the ranks cache different requests. Even after accounting for the extra weight memory this requires, we had roughly five times more usable prefix-cache capacity on the same hardware compared to a topology like TEP8, which was also benchmarked.

The downside of DEP8 is the DP rank synchronization: each DP rank processes different requests but joins the same all-to-all communication at every MoE layer, so even an idle rank may need to run forward passes to keep up with its peers.

The dispatch overhead would further increase as EP goes wider. We found 1 DEP16 took 17.9% longer than two DEP8 groups, with combine and finalization growing the most.

Cache capacity does not eliminate scheduling delays. We found that a request's cached KV could already be available while the request still waited to enter the running batch.

Workers check for completed loads between forward passes. Results then pass through a batch queue and scheduler, where a ready request can miss the current decision and wait another step. Under load, requests already in progress can fill the next prefill batch, delaying admission even when the cached history is ready. This creates a scheduler bubble.

To reduce this bubble, we halved the number of prompt tokens processed in each prefill step, from 8K to 4K per GPU. Shorter steps let waiting requests start sooner. On our configuration, median queue wait fell from 550ms to 110ms, reducing median time to first token by roughly 20%.

Smaller steps add overhead for long, uncached prompts, but the tradeoff worked for our workload because most turns reused an existing prefix.

At our target load, TP4 gave us the lowest inter-token latency among the configurations we tested. With the same GPU budget, we could run more decode engines with fewer sessions running on each one.

However, TP topology comes with a KV capacity tradeoff. TP distributes model weights across GPUs, but with the MLA cache layout we used, each rank still holds a full copy of a request's latent KV, so TP therefore stores KV copies, which also increases the amount of data NIXL transfers from prefill to decode. TP4 stores fewer KV copies than TP8, but each GPU also holds a larger share of the model weights. We wanted to keep the TP4 latency advantage while fitting more KV into the memory available per GPU.

DEP avoids this replication by assigning requests to separate attention ranks, giving it more usable KV capacity, but when testing, we found that the synchronization between DP ranks before MoE dispatch increased decode latency.

We also tested decode context parallelism (DCP), which distributes the cached sequence across ranks while retaining tensor parallelism for the model. This reduced KV duplication, but introduced communication to gather queries, merge selected candidates, and combine partial attention outputs. With sparse attention, each query reads at most 2,048 cached tokens, so there wasn't much attention work to split across ranks in the first place. In our tests, distributing that work across ranks saved too little computation to offset the added communication. We therefore kept TP4 and looked for a way to fit more KV.

We compressed the 512-value MLA latent using NVFP4: four bits per value, with an FP8 scale for each group of 16 values. The 64-value positional component remained FP8. Including scales, each MLA cache row shrank from 576 to 352 bytes.

With the indexer and other state unchanged, total cache capacity increased by roughly 50%, from 1.09 million to 1.63 million cached tokens per decoder at the same memory budget.

Our first implementation unpacked the selected rows into temporary FP8 buffers in GPU memory before calling the existing attention kernel. This gave us the capacity benefit, but every layer paid for conversion and an extra write-and-read through GPU memory.

We built a native sparse-MLA kernel to remove that intermediate buffer. It consumes the positions selected by the sparse indexer, loads the compressed rows, and unpacks them on-chip as attention needs them. NVFP4 is the storage format; the attention computation uses FP16 operands with FP32 accumulation.

For each query token, a cluster of cooperating thread blocks divides the selected rows. In the GB200 configuration, each cluster spans three to eight SMs. The blocks combine their partial results through distributed shared memory, without a second kernel launch.

Inside each SM, separate groups of warps unpack rows, compute attention scores and softmax, and accumulate the output.

Three changes were particularly useful:

At 15 query tokens per launch, the optimized kernel took approximately 12.0 μs on GB200, compared with 17.7 μs for the staged NVFP4 path and 13.7 μs for FP8 attention. These are results for that workload, not a speed advantage over FP8 at every batch size.

Attention accounted for about 8% of a decode step at 32 concurrent sessions, so the larger practical benefit came from retaining more KV. NVFP4 gave us roughly 50% more KV-cache capacity while maintaining comparable per-user speed with decode prefix caching enabled.

We ran a rigorous evaluation suite, with particular emphasis on long-context tasks, to ensure that NVFP4 KV compression does not degrade accuracy.

We are contributing the native kernel to FlashInfer as an experimental operation.

Compression increased decode capacity, but disaggregation introduced another bottleneck: moving KV efficiently from prefill to decode.

We use vLLM's NIXL connector to let each decoder pull KV from the prefill workers over multi-node NVLink. In an early comparison, the NVLink configuration added roughly 292 ms to time to first token relative to InfiniBand. The faster interconnect was not translating into faster serving.

The bottleneck was transfer fragmentation. In the path we profiled, each transfer descriptor became a small device-to-device copy. A single 200K-token request arriving at one TP4 rank triggered 32K copies. Submitting and processing that many small operations limited the transfer before we could make effective use of NVLink's bandwidth.

The original layer-major KV layout, LBHNC, stored each layer's cache separately. Transferring a logical block across the model therefore required separate descriptors for its data in different layers.

Increasing the block size helped: using 1,024-token blocks reduced the number of transfer descriptors. However, larger blocks still left data fragmented across layers and could reduce KV-cache utilization by wasting more space in partially filled blocks.

We then tested BLHNC, a block-major layout recently introduced by the vLLM community. It places a block's data from multiple layers together in memory, allowing NIXL to transfer those contiguous regions with fewer, larger copy operations. We're grateful to the vLLM community for developing and sharing this layout.

In a separate TP8 comparison, the layer-major layout (LBHNC) used 1,024-token blocks , while the block-major layout (BLHNC) used 64-token blocks . Despite using 16× smaller blocks, BLHNC reduced the descriptor count from 19,559 to roughly 1,940 — about 10× fewer — and lowered mean transfer time from 146 ms to 78 ms, a reduction of roughly 47%. The BLHNC layout let us use finer-grained KV blocks while also reducing transfer overhead.

Performance is only part of production agent serving. Agents also need reliable tool calls to read files, run commands, and make edits. If the model calls the wrong tool or produces unusable arguments, the agent must retry or stop.

In our configuration, Dynamo converts the model's generated tool calls into the API response the agent receives. When serving GLM-5.3 at scale, we found several recurring failure patterns. Initially, we found the system could silently discard calls to undeclared tools and return a normal stop, leaving the agent with no action to execute or error to recover from.

Other calls had missing arguments or incorrect types, even when the client explicitly required the model to follow the tool's input format. The model received these definitions, but our GLM serving path wasn't enforcing them during generation.

Dynamo already had a mechanism for this, but lacked structural-tag support for GLM's format. We contributed a structural-tag builder to Dynamo to translate tool definitions into rules that restrict the names and arguments the model can generate.

vLLM uses xgrammar to enforce those rules during decoding, masking tokens that would violate the tool-call grammar. We also fixed errors introduced during parsing: literal strings such as were being converted into , altering code or file contents, while some schema references and nullable argument types were interpreted incorrectly.

We tested the final API responses for:

We maintain our own tool-call test suite covering these cases, validating the final API responses and checking for regressions as we update the serving stack.