Prime Intellect has launched Prime Inference：https://www.primeintellect.ai/blog/prime-inference, a serving platform for frontier open-source models. It offers serverless endpoints and reserved capacity on Prime’s own GPUs across multiple datacenters. Before public release, it processed nearly a trillion tokens per day internally. That traffic came from RL rollouts, synthetic data generation, evaluations and long-running coding agents.
Prime Inference is the serving layer of Prime Intellect’s open training stack. The company already ships post-training tools such as prime-rl, verifiers and sandboxes. Serving closes that loop: deployed models generate production traces that can feed back into training. Prime reports its GLM-5.3：https://huggingface.co/zai-org/GLM-5.3 endpoint ranks among the fastest on OpenRouter. It also cites a near-zero tool-call error rate and 100% uptime since launch.
The stack combines NVIDIA Dynamo：https://github.com/ai-dynamo/dynamo, vLLM：https://github.com/vllm-project/vllm, Mooncake：https://github.com/kvcache-ai/Mooncake and FlashInfer：https://github.com/flashinfer-ai/flashinfer. It was built with Inferact and NVIDIA, and fixes are contributed upstream.
The target workload is agentic. A typical agent turn adds about 6K tokens to a 140K-token prompt. Prime benchmarks this mix with SemiAnalysis AgentX：https://inferencex.semianalysis.com/agentx, and injected cold arrivals.
Prefill/decode disaggregation : Prefill and decode run on separate GPU groups. Dynamo handles routing, and vLLM runs the model on each group. Decoders pull computed KV through NIXL：https://github.com/ai-dynamo/nixl. Prime reports nearly 40% lower p90 inter-token latency in its tests.
Cache-aware routing :Dynamo’s KV-aware router weighs cached prefix overlap against queued work. Sessions stay on the same decoder between turns. Mooncake adds a second KV tier in host DRAM.
The interactivity target was 100 end-to-end tokens per second per user. At that bar, a 1:4 prefill/decode ratio served the most users. It reached 66 sessions per prefill group at 101 tok/s per user and 100 output tok/s per GPU.
Agents fail when tool calls carry wrong names or broken arguments. Prime Intellect’s team contributed a structural-tag builder to Dynamo for GLM’s tool format. vLLM then uses xgrammar：https://github.com/mlc-ai/xgrammar to mask tokens that violate the tool schema. The team also fixed parsing bugs, including being decoded into inside code.
Competitor prices verified October 2, 2026. Tracker figures come from ComputePrices, a third-party price tracker.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us ：https://forms.gle/MJjjVDPS7whH8Ngs6
Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks ：https://pxllnk.co/hxplrq
Practitioner-first AI/ML news and analysis, read by 1M+ developers and researchers every month.
Scan to open this article