Nowadays, State-of-the-Art (SOTA) models are getting much bigger and reloading the model service after a crash is very expensive. Therefore, we are introducing the Weight Cache Daemon , a persistent GPU process that holds post-quantized model weights in GPU memory and serves them to new SGLang engine instances via CUDA IPC zero-copy mapping. This reduces weight loading from minutes to seconds.
The Weight Cache Daemon is the first phase of our Fast Engine Recovery Framework , which targets < 10 second cold restarts and < 1 second warm standby switches for production LLM serving.
As LLM models grow larger — Qwen3-235B, Ling-2.6-1T, and the newly released 2.8T Kimi K3 — the cold-start time of serving engines has become a critical bottleneck for production efficiency. A Ling-2.6-1T FP8 instance on 8×H20-3e GPUs takes ~8.52 minutes just to become ready to serve, weights stay in 3.5T NVME SSD. In production, this means:
Where does the time go? We profiled a complete SGLang engine startup for Ling-2.6-1T FP8:
The bottleneck is clear: weight loading from disk accounts for 93.2% of startup time . For Ling-2.6-1T FP8 model, each TP rank reads ~120GB of safetensors from disk, deserializes, applies TP sharding, and runs post-quantization transforms (FP8 quantization, weight repacking). This work is repeated identically on every restart , even though the resulting GPU tensors are deterministic and often already present in GPU memory.
Can we avoid reloading from disk every time? The answer is yes — by keeping weights in GPU memory across engine restarts.
The Weight Cache Daemon is a persistent GPU process that holds post-quantized, TP-sharded weights in GPU memory. On engine restart, the new engine process maps weights from the daemon via CUDA IPC zero-copy — no disk I/O, no deserialization, no quantization.
Each GPU runs one daemon process for its TP rank. The daemon:
The engine connects to the daemon, validates config compatibility, and maps weights directly into its address space — the engine and daemon share the same physical GPU memory via CUDA IPC.
The key to sub-second loading is zero-copy : the engine's param.data pointer is set directly to the IPC-mapped GPU tensor. No data is copied.
To achieve this, the engine initializes the model on the meta device (no GPU/CPU memory allocation), then replaces each parameter's data pointer with the IPC-mapped tensor.
Post-quantization parameters (e.g., weight_scale from FP8 quantization) that were created by process_weights_after_loading() are also cached by the daemon and mapped directly — no re-quantization needed.
Any mismatch between the engine's config and the daemon's cached config triggers a full disk reload , ensuring correctness:
The last two fields form an environment stamp : a daemon and a client that ran different post-processing branches (different compute capability or torch/kernel version) can produce weights that map cleanly over IPC yet serve garbage — stamping the environment into CacheConfig turns that into a clean mismatch.
This is critical for production safety: if an operator changes the model or quantization config, the engine will detect the mismatch and fall back to disk loading rather than mapping incompatible weights.
On top of config validation, quantization methods are gated by an IPC allowlist . CUDA IPC zero-copy exports only raw tensor data, so it is correct only when the entire effect of process_weights_after_loading() is captured by that data. Methods that stamp Python-side metadata or repack/transpose weights (per-tensor FP8, Marlin, AWQ/GPTQ) would silently serve wrong numerics — they raise a hard error instead. Currently verified: unquantized and block-wise FP8 ( weight_block_size set); more methods will be added after end-to-end verification.
In daemon mode, the engine spawns daemon processes during startup and waits for them to load weights from disk. The first start is still slow (daemons must load from disk), but subsequent restarts are instant.
In client mode, the engine connects to already-running daemons. This is the fast-restart path — the daemon was started earlier and already holds weights in GPU memory.
The Weight Cache Daemon is designed to be non-intrusive and safe :
The Weight Cache Daemon unlocks production patterns that are impractical with traditional disk-based loading:
A single daemon per GPU holds weights in memory; multiple engine instances (e.g., independent services) map to the same IPC handles via zero-copy. Weights are loaded from disk and quantized exactly once per GPU , regardless of how many instances consume them.
Run a high-priority online service and a low-priority batch job on the same GPU, backed by the same weight cache daemon. The low-priority instance can be evicted and re-spawned in sub-second time without reloading weights from disk — enabling flexible GPU time-sharing without the usual startup penalty.
Deploy a standby engine alongside the primary, both backed by the same weight cache daemon. The standby maps weights via zero-copy and stays warm. When the primary fails, the standby takes over in < 1 second — no weight loading, no disk I/O.
This achieves near-zero-downtime failover without dedicating a full set of GPUs to an idle replica , avoiding the expensive GPU resource waste of traditional hot-standby deployments.
One command launches all TP rank daemons:
Wait for daemons to become ready (they write a .ready file per rank):
In a multi-node deployment, each node runs its own daemon for its local TP ranks. All daemons join the same distributed group, so --nnodes , --node-rank , and --dist-init-method must be consistent across nodes, with $MASTER_ADDR pointing at node 0:
Once every node reports its daemons ready, start the engine clients. They use a separate rendezvous port ( 29600 ) from the daemons ( 29500 ):
The Weight Cache Daemon is Phase 1 of a broader Fast Recovery Framework targeting < 10s cold restarts and < 1s warm standby switches :
Support for more models is also on the way.
The Weight Cache Daemon is just the first step — there is still a lot to build, and we are excited about the road ahead. Phase 1 today covers TP + PP, single- and multi-node launch, per-GPU zero-copy CUDA IPC, and unquantized plus block-wise FP8. Beyond that, many high-impact directions remain open:
This is very much a community effort. The full plan is tracked publicly in sgl-project/sglang#33522：https://github.com/sgl-project/sglang/issues/33522 — contributions and feedback are very welcome , and there is plenty of impactful work to pick up.
Ant Ling Infra Team, Ant Group : Michael Qiu：https://github.com/QiuMike qiudayu.qdy@antgroup.com：mailto:qiudayu.qdy@antgroup.com
扫码直达文章详情