{"@context":"https://schema.org","@type":"NewsArticle","generatedAt":"2026-08-21T20:00:48.567Z","headline":"Ling-3.0-flash 在 4 块 Blackwell GPU 上如何将批处理 1 解码延迟降低 54%","description":"蚂蚁 Ling Infra 团队与 RadixArk SGLang 团队将 Ling-3.0-flash 混合线性注意力 MoE 模型的单请求解码速度从 288 tok/s 提升至 606 tok/s，平均 TPOT 从 3.33 ms 降至 1.53 ms。 🔗 阅读原文 via AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","url":"https://www.aioga.com/news/cmt393qov0kfhro6tuwhxhubl/","mainEntityOfPage":"https://www.aioga.com/news/cmt393qov0kfhro6tuwhxhubl/","datePublished":"2026-08-21T17:56:25.000Z","dateModified":"2026-08-21T17:56:25.000Z","inLanguage":"zh-CN","publisher":{"@type":"NewsMediaOrganization","name":"Aioga","url":"https://www.aioga.com"},"citation":["https://www.lmsys.org/blog/2026-08-21-ling3-flash-spec-decode-blackwell","https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl"],"canonicalUrl":"https://www.aioga.com/news/cmt393qov0kfhro6tuwhxhubl/","directAnswer":{"@type":"Answer","text":"LMSYS 博客称，蚂蚁 Ling Infra 与 RadixArk SGLang 团队在 4 块 NVIDIA Blackwell GPU 上优化 Ling-3.0-flash，将单请求解码速度从 288 tok/s 提升至 606 tok/s，平均 TPOT 从 3.33 ms 降至 1.53 ms。","url":"https://www.aioga.com/news/cmt393qov0kfhro6tuwhxhubl/","dateCreated":"2026-08-21T17:56:25.000Z","author":{"@type":"Organization","@id":"https://www.aioga.com/authors/aioga-editorial/#editorial-team","name":"Aioga Editorial Team","url":"https://www.aioga.com/authors/aioga-editorial/"}},"evidence":[{"@type":"CreativeWork","name":"LMSYS：Blog（Chatbot Arena 团队 source article","url":"https://www.lmsys.org/blog/2026-08-21-ling3-flash-spec-decode-blackwell","datePublished":"2026-08-21T17:56:25.000Z","provider":{"@type":"Organization","name":"LMSYS：Blog（Chatbot Arena 团队","url":"https://www.lmsys.org/blog/2026-08-21-ling3-flash-spec-decode-blackwell"}},{"@type":"CreativeWork","name":"AIHot archive record","url":"https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","datePublished":"2026-08-21T17:56:25.000Z","provider":{"@type":"Organization","name":"AIHot","url":"https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl"}}],"aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","originalPublisher":{"name":"LMSYS：Blog（Chatbot Arena 团队","url":"https://www.lmsys.org/blog/2026-08-21-ling3-flash-spec-decode-blackwell"},"geoDeepAnswer":null,"article":{"id":"cmt393qov0kfhro6tuwhxhubl","slug":"cmt393qov0kfhro6tuwhxhubl","url":"https://www.aioga.com/news/cmt393qov0kfhro6tuwhxhubl/","title":"Ling-3.0-flash 在 4 块 Blackwell GPU 上如何将批处理 1 解码延迟降低 54%","title_en":"","summary":"蚂蚁 Ling Infra 团队与 RadixArk SGLang 团队将 Ling-3.0-flash 混合线性注意力 MoE 模型的单请求解码速度从 288 tok/s 提升至 606 tok/s，平均 TPOT 从 3.33 ms 降至 1.53 ms。 🔗 阅读原文 via AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","source":"LMSYS：Blog（Chatbot Arena 团队","sourceUrl":"https://www.lmsys.org/blog/2026-08-21-ling3-flash-spec-decode-blackwell","aiHotUrl":"https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","publishedAt":"2026-08-21T17:56:25.000Z","category":"行业动态","score":72,"selected":true,"articleBody":["Batch-1 decode keeps getting more important. Xiaomi MiMo, for example, announced MiMo-V2.5-Pro UltraSpeed in June：https://mimo.xiaomi.com/blog/mimo-tilert-1000tps, claiming 1,000 tok/s decode on a one-trillion-parameter MoE model.","Batch 1 gives an inference stack no room to hide overhead. There is no batch to amortize launch cost across, no concurrency to fill pipeline bubbles, and not enough arithmetic intensity for clever tiling to pay off. Every microsecond on the critical path is a microsecond the user waits.","This post is about pushing that floor down for Ling-3.0-flash, a hybrid linear-attention MoE model, on 4 NVIDIA Blackwell GPUs. It covers two speculative decoding paths. On the NEXTN/MTP path we moved single-request decode from 288 tok/s to 606 tok/s and mean TPOT from 3.33 ms to 1.53 ms. The second path is DSpark, a confidence-scheduled speculative decoder built on the same stack: a 1000-request run reaches 1120 tok/s at a mean TPOT of 0.78 ms and an accept length of 9.95. That last comparison is the controlled one: NEXTN and DSpark were measured with the same command on the same machine, and mean TPOT is 1.9x lower (1.53 ms to 0.78 ms). The rest of the post is where that time was going and what it took to get it back.","Figure 1. Headline results across the four configurations.","On GSM8K, the same stack scored: accuracy 0.889, invalid 0.000, latency 341.5 s, output throughput 511.1 tok/s.","All runs use Ling-3.0-flash on 4 Blackwell GPUs, TP4, bf16, concurrency 1, greedy decoding, and the same fixed 8192-input / 1024-output random workload. From left to right, the columns show the initial NEXTN baseline, NEXTN after the draft-extend graph fix, final tuned NEXTN, and DSpark. The first two are shorter campaign checkpoints; the last two are the controlled comparison, each measured over the same 1000 requests on the same machine. Peak throughput is compared only between the last two runs because it is the maximum over fixed one-second windows.","Two definitions matter here, because together they explain why output throughput is not simply the reciprocal of mean TPOT even at concurrency 1: SGLang's TPOT excludes TTFT, while output throughput divides total output tokens by total benchmark wall time (see the bench_serving guide：https://github.com/sgl-project/sglang/blob/main/docs/developer_guide/bench_serving.md). All headline benchmark runs in this post use a synthetic random workload; accept length in particular depends on the prompt and output distribution, so 9.95 is this workload's accept length rather than the model's.","Figure 2. Ling-3.0-Flash architecture: 42 layers interleaving 35 KDA linear-attention layers with 7 MLA full-attention layers over a 512-expert MoE.","Ling-3.0-flash is a hybrid-attention MoE model ( BailingMoeV3 ), and most of what follows comes from that word hybrid .","Five of every six attention layers are KDA. That is why MLA attention costs only 244 µs per step at 8k context in the final profile, and why this model is a good batch-1 target in the first place: with attention cheap and the batch tiny, what remains on the critical path is weight bandwidth and launch latency, which is exactly the regime this post is about.","We decode with NEXTN speculative decoding at steps=5, topk=1, draft_tokens=6 . One decode step is three CUDA graphs in a relay.","Figure 3. Three graphs per step. The draft model proposes a 6-token chain, the target model scores all six in a single forward, and the extend graph replays the accepted prefix with the target's real hidden states to produce the next round's seed. The verdict itself ( eagle_sample ) happens inside the verify graph; the host learns how many tokens were accepted one step late.","The draft is a single-layer NEXTN model run autoregressively: five steps but only four forwards, because the first candidate comes from the previous round's seed and the fifth is read off the fourth forward's top-k. Verify is one forward of the full 42-layer target over all six chain positions. Extend fixes up the draft's KV cache, which only ever saw the draft's own guesses, and hands back the seed for the next round.","What crosses between the three graphs on the CPU is nothing. Fixed shapes plus padding make every accept-dependent count a GPU index rather than a host value; persistent buffers let producer graphs write straight into consumer buffers; and the decisions that genuinely need values on the CPU (EOS, stop strings, detokenization) go through a side-stream D2H and a copy_done event consumed one step late. Everything below rests on that property.","When we started, the GPU was busy about two-thirds of the step. Idle time at batch 1 comes in two flavors, and they need separate diagnoses because the fixes have nothing in common:","Figure 4. Two shapes of idle. Top: the host loop is longer than the GPU's work, so the holes are few and wide and land in the seams between graphs. Bottom: once the host is hidden, what remains is several hundred 1.5-6 µs kernel nodes whose launch floor rivals their arithmetic, plus the weight read itself.","These two kinds of idle describe the step-time side of TPOT. The other lever is how many tokens each step commits: mean TPOT ≈ step time / mean accept length. The rest of the post follows those levers. Host run-ahead and seam work remove host-mode idle; PDL, dtype changes, fusion, and retuning shorten the GPU critical path; speculation tuning and DSpark increase the tokens committed per target step. DSpark later revisits the first category when a blocking D2H read reintroduces a host pin.","Three properties of the measurement setup shape every number below.","The profiler inflates host-side events. CUPTI adds overhead to each host event it records. On the same configuration, a profiled step measures 5.2 ms while the real step, back-computed from TPOT × accept length on an unprofiled run, is 4.9 ms. That 0.3 ms gap is the same order as the host-side effects we wanted to reason about, so a profiled trace can show cross-rank waits that do not exist off the profiler. GPU kernel durations come from hardware timestamps and are more trustworthy than host-side timings, but not immune: tracing still perturbs launch timing, concurrency, cache state, and CUDA graph execution, and Nsight Systems documents potentially significant overhead for CUDA and graph-node tracing (user guide：https://docs.nvidia.com/nsight-systems/UserGuide/index.html). So every host-side conclusion here got a profiled-vs-unprofiled calibration first.","Microbenchmarks run optimistic for cold-weight kernels. A loop calling one kernel repeatedly keeps its 2.6 MB gate weight resident in L2, while the real model flushes L2 with about 94 MB of expert traffic between consecutive calls to the same layer. Hot 7 µs, cold 11 µs: enough to reverse a ranking against the library GEMV.","Peak throughput is a single-window statistic. The benchmark's peak number is the maximum over a fixed 1-second grid, so it carries roughly a ±5% phase band: shifted TTFT/TPOT re-slices the grid, and a change that improves mean throughput by 2.3% can print as a drop from 909 to 858. Both readings reproduce exactly under a fixed seed, so reproducibility does not separate signal from phase. A/B decisions here are made on mean TPOT × mean accept length. That product is a derived estimate of step time rather than a measured one (the product of two aggregates is not the aggregate of the product), but it is stable across runs and insensitive to accept-length drift in these runs, which is what an A/B criterion needs. We report peak but never optimized against it.","Correctness had its own gate, applied to every change before it stayed: byte-exact comparison of a 256-token greedy generation, accept length unchanged within 0.05, and a greedy re-run after interleaving temperature-sampled requests to catch state pollution. Changes that legitimately alter rounding (the bf16 gate, the single-rounding combine) said so in their commit message and were validated on accept and task metrics instead of bit parity.","This is the structural change the rest of the campaign rests on, and it is a host-mode idle fix.","Figure 5. Lockstep to deep pipelining. Before: every step the host blocks in resolve_seq_lens_cpu waiting for the previous verify graph to finish on the GPU, so run-ahead resets to zero and each host prep segment becomes a GPU bubble. After: the queue is a full step deep, the launch of verify k+1 leads its own execution by an entire step, and the only remaining synchronization is a copy_done event consumed one step late.","cudaGraphLaunch has always been asynchronous, and the draft → verify → extend ordering on the GPU is free: same stream, FIFO. So the question was never whether verify waits for draft. It was whether the host is pinned to GPU progress every step.","It was. Under spec-v2, the scheduler does not know the accept length, so FutureMap.resolve_seq_lens_cpu() pulls new_seq_lens back from the GPU while building the next batch: gated on a publish event, copied on a private stream, then synchronize() d. The host was not waiting for a microsecond-scale copy. It was waiting for the previous verify graph to finish executing. Median cost: 485 µs per step, with the run-ahead depth reset to zero every single step.","The cause is the needs_cpu_seq_lens flag, OR-ed across every backend involved in spec-v2. trtllm_mla declares False in all three roles; the sibling linear-attention backends GDNAttnBackend and Mamba2AttnBackend both declare False explicitly. KDAAttnBackend never declared it and inherited the base-class default of True , even though it runs the same base-class metadata code as its two siblings.","Declaring needs_cpu_seq_lens = False collapsed the OR and removed the per-step synchronize. The correctness argument is pointwise: KDA's metadata never reads the CPU mirror, and replay padding comes from forward_batch.num_padding .","How does the host dare launch step k+1 without knowing what step k accepted? Because the values never touch the CPU. FutureMap is a GPU-resident relay: step k's graph writes output tokens, new_seq_lens , top-k probabilities, and hidden states into device buffers indexed by req_pool_idx , and step k+1's graph reads them by the same index. The host only handles indices, which it already knows.","Figure 6. Where the slack lives. Panel A: the host loop (~4.3 ms) fits under the GPU step (~4.9 ms), so it is fully hidden. Panel B: when jitter (a gloo broadcast or a GC pause) exceeds the slack, the host finishes late and the GPU waits at the next verify boundary, where the first collective in the graph absorbs the cross-rank skew.","Run-ahead also changes the shape of host cost. Instead of every rank paying its host time directly every step, only a rank that exhausts its queue slack pays. In one four-rank trace, exactly one rank was in that state: its scheduler segment ran 5-10x longer than its siblings, its draft graph launched 40-80 µs late, its draft→verify seam ran +165 µs above the others' median, and it showed periodic 400-750 µs spikes with a GC signature. The other three ranks spin-waited for it at every rendezvous. The diagnostic that generalizes: a kernel's duration is not its work. A 20 KB embedding all-reduce showing 150-480 µs is not a slow all-reduce; it is absorbing skew, and only cross-rank time alignment tells you which rank is late.","With the lockstep pin gone, the seams between graphs became worth shrinking. Before a CUDA graph replays, step-specific attention metadata (kv indices, block tables, mamba state slots) has to be rebuilt from the live req_to_token and seq_lens into the graph's captured static buffers. That refill runs eagerly every step and is most of what a seam contains. At batch 1 it is purely host-bound: each op costs 5-15 µs to dispatch and 1-4 µs to execute.","We attacked it at two levels. First, fuse the index chains: assign_extend_cache_locs_uniform computes end offsets inside the kernel (the uniform draft_token_num expansion makes the cross-row prefix sum unnecessary), and _fused_state_indices_kernel collapses a gather, a translate, a padding-sentinel write, and a copy_ into one launch, carefully preserving both side effects, including zeroing req_pool_indices on padded rows, which nothing in that function needs but other captured kernels in the graph depend on for in-bounds gathers.","Second, capture the refill itself into a small CUDA graph keyed by (bs, forward_mode) . This works because of a pointer-stability property the replay contract already guarantees: the replay ForwardBatch view hands the backend only runner-static buffers and pool-resident tensors, so the whole prep sequence has fixed addresses. Four safety mechanisms surround it: two eager warmups, so Triton JIT and autotune happen outside capture; a snapshot of each backend's forward_metadata object restored before every replay (the graph replays device ops, the snapshot restores Python pointers); permanent eager fallback with a warning if capture fails; and guards against padding, TBO, pdmux, and LoRA. It ships opt-in behind SGLANG_ENABLE_METADATA_GLUE_GRAPH and is force-disabled for DFLASH-family speculation, because that path rebuilds its attention plan on the host every step and capturing the refill would freeze the plan at capture time.","There is a hard boundary on what may be captured. The criterion: a refill made of pure device kernels writing persistent buffers is capturable; anything that goes through a FlashInfer-style plan() is not. The draft side fails it: the multi-step draft backend re- plan() s wrappers that the main EAGLE graph has already captured, and recording that re-plan into a secondary graph corrupts the wrappers' internal state on replay. A related requirement is that capture be idempotent. trtllm_mla 's _init_cuda_graph_metadata used to allocate fresh tensors and replace its decode_cuda_graph_metadata[bs] entry on every call, which leaves earlier graphs reading freed memory after a second capture.","A batch-1 step executes several hundred kernel nodes in a short window. At that size, launch and prologue cost about as much as the math. Programmatic Dependent Launch (PDL) lets a consumer kernel be scheduled onto SMs while its producer is still running: the consumer executes everything that does not depend on the producer's output and fences at gdc_wait() only before the dependent read.","Figure 7. PDL on the router path. Without PDL, each kernel starts only after the previous one fully retires, and the gate matvec's cold-HBM weight load sits on the critical path. With PDL, the weight tile load is producer-independent, so it is issued before gdc_wait() and a 2.6 MB cold read flies under the producer's tail; the router top-k prefetches its bias the same way.","We wired three chains: the MoE main chain ( moe_align → up-GEMM → activation → down-GEMM → combine → all-reduce), the router chain (norm → gate matvec → top-k), and the KDA chain ( conv1d_update → recurrent delta-rule → gated norm). Two design points matter.","Producer-independent loads go before the wait. That is the whole trick in the figure, and it is what makes PDL more than launch-overhead removal for latency-bound kernels.","Inductor kernels cannot carry PDL attributes. The small-M MoE combine was a torch.compile -generated kernel; joining the chain meant swapping it for the repo's Triton reduction plus GDC. That had a numerical side effect: fp32 sum × scale with a single final cast, where the old path rounded twice. The result is slightly more accurate but not bit-equal, which the commit message declares.","Later we upgraded the semantics after a finding in PTX griddepcontrol : launch_dependents only releases the launch of dependents, while a consumer's wait always fences on the producer grid's complete retirement. Moving the trigger from the end of the producer to immediately after the producer's own wait lets a consumer's prologue overlap more of the producer's body than just its tail, subject to one precondition: the consumer must still keep its own gdc_wait() between the early launch and every read of producer output. That is a property of each consumer, not a blanket guarantee, so we checked it kernel by kernel and converted six. What the early trigger buys is also not deterministic: the driver may launch a dependent grid early, and how much overlap materializes depends on scheduling and resource pressure at the time (CUDA programming guide：https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/programmatic-dependent-launch.html). fused_moe gates this behind an M ≤ 512 check: at prefill shapes, releasing a large consumer grid early steals SMs from the producer, while at decode shapes it is pure gain.","PDL is pure scheduling semantics. Changes whose accumulation order is unchanged stay bitwise identical; the gate matvec passed a 4-of-4 GDC-on/off bit comparison.","moe_align , on the pair axis. The Triton fused-MoE GEMM consumes tokens in block_size tiles where every row shares an expert, and moe_align_block_size builds that permutation. The generic path needs two kernel launches: no token can be placed until every expert's offset is final, those offsets come out of a grid-wide scan, and a device-wide barrier exists only at a kernel boundary. A single-launch variant exists, but it stages per-thread expert counters in shared memory, so it is limited to 64 experts or fewer; a 513-expert decode always paid two launches.","The replacement works on the pair axis: an [NP, NP] pairwise comparison gives every (token, slot) pair a stable rank within its bucket and its bucket population in one shot; the rank-0 representative of each bucket then derives padded counts, bucket-ordered exclusive offsets, the published total, and per-block expert ids. Nothing scales with the expert count, so the expert-count limit disappears. The obvious alternative, histogram and cumsum over the padded expert axis (up to 1024 buckets), is correct but puts about 3x more single-SM work on the critical path than the two kernels it replaces. That is what makes the pair axis load-bearing here.","Two deliberate deviations from the reference, both argued from consumer invariants: intra-bucket order is stable in pair index rather than atomic-scheduling order (every pair writes its own output row, so consumers are order-invariant), and the buffer tail beyond the published total is left unwritten (consumer CTAs early-exit before reading it). One cliff: the pairwise tensors are O(NP²). They live entirely in registers at NP=64 (about 4 µs, on par with the CUDA two-kernel path) and spill to local memory at NP=256, costing about 230 µs per launch. The dispatch gate is a hard numel ≤ 64 ; larger batches fall back to the CUDA path.","SwiGLU in the up-GEMM epilogue. Folding silu(gate) * up into the MoE up-GEMM epilogue removes a standalone activation kernel per MoE layer and the whole write-then-read of the intermediate buffer. The layout trick is a per-expert row interleave of w13 applied at weight load, which makes gate and up land in adjacent even/odd columns of the same output tile. Since each GEMM output column is an independent dot product, interleaving is bitwise neutral.","Bit parity is where the care went. The kernel being replaced is compiled with -use_fast_math , so the epilogue reproduces it instruction by instruction: mul + ex2.approx.ftz for __expf , div.approx.ftz , and a single final rounding on the product. The subtle part: FlashInfer instantiates the activation functor in float , so silu never lands in bf16 before the multiply. Round it there and the result double-rounds and diverges on a large fraction of inputs. That is invisible in the documentation and invisible to a tolerance check; it takes an element-wise bit comparison over the full input range.","KDA chain-verify tile economics. The fused conv1d + gating delta-rule verify kernel already existed; these commits retuned it. On the rotating-cold test with in-graph timing, the Blackwell curve at T=6 is monotone: BV=4 at 11.56 µs, 8 at 12.53, 16 at 12.83, 32 at 14.26, 64 at 20.7, 128 at 38. BV=4 wins by up to 19% per call because 256 CTAs is 1.7 waves over 148 SMs, and duplicating the q/k convolution 32 times is still cheaper than shortening the serial chain. Tiling the V dimension never touches the K-dimension reduction order, so at num_warps=4 every BV is bitwise identical to the baseline and the retune carries no numerical risk.","The largest single post-structural change was a dtype change. At batch 1, the router gate and the lm_head are pure bandwidth: every decode step cold-reads each MoE layer's gate weight (2.6 MB in bf16) and the vocab-parallel lm_head projection, and neither has arithmetic to hide the read behind. Running both in bf16 instead of fp32 halves those bytes; end to end it was worth roughly +10%, the largest gain of any single change after the host run-ahead fix. Like the other rounding changes above, this one was declared in its commit message and validated on accept length and task metrics rather than bit parity.","A rejected speculative token leaves a KV cache entry harmlessly stale, but it has already corrupted a recurrent state in place. Linear attention and speculation do not coexist for free.","The scheme that makes it work: during verify, the recurrence runs with state updates disabled and writes each chain position's post-state into an intermediate buffer; after the verdict, commit_mamba_states_after_verify copies the state belonging to the last accepted position into the persistent slot. Stage, then commit. This is also why the compact spec cache is restricted to topk=1 : with a chain, the accepted prefix is unique and states can be indexed by position; with a tree, the accepted path is one of many and states would have to be indexed by tree path.","Profiling shows KDA decode is bandwidth-bound, mostly HBM traffic on the K×V state, so beyond the fusion and the tile retune there is not much left there. We did not measure achieved bandwidth against the Blackwell peak, so read that as a shape observation rather than a roofline result.","Weight-bandwidth dominance has a counter-intuitive corollary: verifying more tokens is nearly free. Verifying 4 tokens and verifying 6 tokens read exactly the same weights. Deepening speculation at batch 1 costs one extra cheap draft forward per added step (the draft is a single layer) plus the incremental serial cost in the KDA chain-verify recurrence, and buys accept length.","We swept it rather than assuming it (this sweep predates the fusion bundle; the optimum moved afterwards, as noted below):","The step-time column is TPOT × accept length, a derived estimate rather than a direct measurement. Each added step costs about 4-9% of step time while the marginal accept gain decays geometrically (d5 → d6 adds only 0.08). The break-even condition is roughly Δaccept > 0.05 × accept . The optimum also moves: after the fusion bundle landed and step time dropped, (5, 6) became the better configuration; once fp8 weights shrink the fixed base further, it will need another sweep.","Tuning NEXTN's depth is a one-dimensional knob on a fixed-shape algorithm. The larger lever is changing the algorithm, and the second half of the campaign went into bringing DSpark onto the same target and giving it the same batch-1 treatment.","The DSpark algorithm itself is public. The work here is adapting that public recipe to Ling-3.0-flash, long-context online distillation, and the batch-1 Blackwell serving stack. Our adaptation differs in four ways.","Distribution-aligned data. We distill mainly on Ling-3.0-flash post-training data, so the draft trains on the distribution it will face at serving time. We also use multiple sampling settings during distillation to improve trajectory diversity and robustness under speculative decoding.","An ablation-driven draft design. Instead of directly inheriting the Ling-3.0-flash architecture, we ran systematic ablations over key draft choices, including whether to reuse the Ling-3 attention structure and which RoPE variant to use (partial or interleaved). We kept the design with the best acceptance-length/latency tradeoff.","A serving-coupled online training system. For long-context and large-scale online training we built SplitServe Trainer, a single-node 8-GPU framework that splits resources evenly between training and SGLang inference. During training, the inference side runs target forwards to produce supervision signals such as target hidden states for the draft. This keeps the generation-training loop local, cuts IO overhead, and improves training efficiency for long-context workloads."],"articleImages":[{"sourceUrl":"https://www.lmsys.org/images/blog/ling3-flash-batch1/00_headline.png","alt":"Headline results across the four configurations","afterParagraph":2,"url":"/media/articles/cmt393qov0kfhro6tuwhxhubl/d981ca5c62f5f386.png"},{"sourceUrl":"https://www.lmsys.org/images/blog/ling3-flash-batch1/01_model_architecture.png","alt":"Ling-3.0-Flash architecture: 42 layers interleaving 35 KDA linear-attention layers with 7 MLA full-attention layers over a 512-expert MoE","afterParagraph":6,"url":"/media/articles/cmt393qov0kfhro6tuwhxhubl/1bc2ff7fa89b6947.png"},{"sourceUrl":"https://www.lmsys.org/images/blog/ling3-flash-batch1/02_three_graph_relay.png","alt":"Three graphs per decode step","afterParagraph":10,"url":"/media/articles/cmt393qov0kfhro6tuwhxhubl/b8c07970a9377f37.png"},{"sourceUrl":"https://www.lmsys.org/images/blog/ling3-flash-batch1/03_two_idle_modes.png","alt":"Two shapes of idle time at batch 1","afterParagraph":14,"url":"/media/articles/cmt393qov0kfhro6tuwhxhubl/c6148a0fd6d7d780.png"},{"sourceUrl":"https://www.lmsys.org/images/blog/ling3-flash-batch1/04_host_run_ahead.png","alt":"From lockstep to deep pipelining","afterParagraph":22,"url":"/media/articles/cmt393qov0kfhro6tuwhxhubl/e0b4e4121b2f8776.png"},{"sourceUrl":"https://www.lmsys.org/images/blog/ling3-flash-batch1/05_run_ahead_slack.png","alt":"Where the run-ahead slack lives","afterParagraph":28,"url":"/media/articles/cmt393qov0kfhro6tuwhxhubl/6a9a438a4b25bd6b.png"}],"mediaStatus":"ok","articleBodyZh":["批量1解码变得越来越重要。例如，小米MiMo在6月发布了MiMo-V2.5-Pro UltraSpeed：https://mimo.xiaomi.com/blog/mimo-tilert-1000tps，声称在一个万亿参数的MoE模型上实现每秒1000个token的解码。","批量1给推理堆栈留下的开销空间非常有限。没有批量可以分摊启动成本，也没有并发来填充流水线空隙，并且算术强度不足以让巧妙的平铺获得收益。关键路径上的每一微秒都是用户等待的微秒。","本文讨论的是如何在4块NVIDIA Blackwell GPU上将Ling-3.0-flash（一个混合线性注意力MoE模型）的这一底线推低。它涵盖了两种推测解码路径。在NEXTN/MTP路径上，我们将单请求解码速度从288 tok/s提升至606 tok/s，平均TPOT从3.33 ms降低到1.53 ms。第二条路径是DSpark，这是一个基于相同堆栈构建的、使用置信度调度的推测解码器：1000请求运行达到1120 tok/s，平均TPOT为0.78 ms，接收长度为9.95。最后的比较是受控的：NEXTN和DSpark在同一台机器上使用相同命令测量，平均TPOT降低了1.9倍（从1.53 ms降到0.78 ms）。本文其余部分将介绍这些时间都花在了哪里，以及如何将其找回。","图1. 在四个配置下的主要结果。","在GSM8K上，相同堆栈得分为：准确率0.889，无效0.000，延迟341.5秒，输出吞吐量511.1 tok/s。","所有运行均使用4块Blackwell GPU上的Ling-3.0-flash，TP4，bf16，并发1，贪婪解码，以及相同的固定8192输入/1024输出随机工作负载。从左到右，列显示初始NEXTN基线、修复draft-extend图后的NEXTN、最终调优后的NEXTN以及DSpark。前两列为较短的实验检查点；后两列为受控比较，每列在同一台机器上测量相同的1000个请求。峰值吞吐量仅在最后两次运行中比较，因为它是固定一秒窗口中的最大值。","这里有两个定义很重要，因为它们一起解释了为什么输出吞吐量并不是平均 TPOT 的简单倒数，即使在并发为 1 时也如此：SGLang 的 TPOT 不包括 TTFT，而输出吞吐量则是将总输出令牌数除以基准测试总墙钟时间（参见 bench_serving 指南：https://github.com/sgl-project/sglang/blob/main/docs/developer_guide/bench_serving.md）。本文中的所有主要基准测试运行都使用了合成随机工作负载；accept length 特别依赖于提示和输出分布，因此 9.95 是此工作负载的 accept length，而不是模型的。","图 2. Ling-3.0-Flash 架构：42 层，交错排列 35 层 KDA 线性注意力层和 7 层 MLA 全注意力层，基于 512 专家 MoE。","Ling-3.0-flash 是一个混合注意力 MoE 模型（BailingMoeV3），以下大部分内容来自“混合”这个词。","每六层注意力层中有五层是 KDA。这就是为什么在最终分析中，MLA 注意力在 8k 上下文中的每步耗时仅为 244 微秒的原因，也是该模型一开始就是一个优秀的 batch-1 目标的原因：由于注意力开销小且批量非常小，剩余的关键路径就是权重带宽和启动延迟，这正是本文讨论的场景。","我们在 steps=5, topk=1, draft_tokens=6 的情况下使用 NEXTN 推测解码进行解码。一次解码步骤是在一个中继中进行的三个 CUDA 图。","图 3. 每步三个图。草稿模型提出一个 6 令牌链，目标模型在一次前向中评分全部六个，extend 图使用目标模型的真实隐藏状态重放已接受的前缀，以生成下一轮的种子。判定本身（eagle_sample）发生在 verify 图中；主机迟一步才知道接受了多少令牌。","草稿是一个单层 NEXTN 模型，自回归运行：五步但仅四次前向，因为第一个候选来自前一轮的种子，第五个从第四次前向的 top-k 中读取。Verify 是对全部六个链位置执行一次包含 42 层的目标模型前向。Extend 修复草稿的 KV 缓存，该缓存仅见过草稿自身的猜测，并将种子交回下一轮。","CPU上的三个图之间交叉的部分实际上是没有的。固定形状加上填充使每个依赖接受的计数成为GPU索引，而不是主机值；持久缓冲区让生产者图直接写入消费者缓冲区；而真正需要CPU值的决策（EOS、停止字符串、去标记化）则通过一个旁路D2H流和一个延迟一步消费的copy_done事件来处理。下面的一切都基于这个特性。","当我们开始时，GPU在每步中大约有三分之二的时间处于繁忙状态。在批量为1时的空闲时间有两种类型，它们需要分别诊断，因为解决方法完全不同：","图4. 两种类型的空闲。上图：主机循环比GPU的工作长，因此空隙少而宽，并落在图之间的缝隙中。下图：一旦主机被隐藏，剩下的是几百个运行时间为1.5-6 µs的核节点，它们的启动开销相当于计算时间，加上权重读取本身。","这两种空闲描述了TPOT的每步时间侧。另一个杠杆是每步提交的令牌数：平均TPOT≈步长时间/平均接受长度。本文其余部分都基于这些杠杆。主机运行预读和缝隙工作消除主机模式下的空闲；PDL、数据类型更改、融合和重新调优缩短GPU关键路径；投机调优和DSpark增加每个目标步骤提交的令牌数。DSpark之后在阻塞D2H读取重新引入主机固定时，会重新审视第一类空闲。","测量设置的三个特性塑造了下面的每一个数值。","分析器会夸大主机端事件。CUPTI 会为它记录的每个主机事件增加开销。在相同配置下，一个被分析的步骤测量为 5.2 毫秒，而实际步骤（通过在未分析运行中从 TPOT × 接受长度反向计算）为 4.9 毫秒。那 0.3 毫秒的差距与我们想要分析的主机端效应同数量级，因此分析轨迹可能显示跨节点等待，而在未分析情况下并不存在。GPU 内核的持续时间来源于硬件时间戳，比主机端计时更可靠，但也不是免疫的：追踪仍然会扰动启动时机、并发性、缓存状态以及 CUDA 图执行，而且 Nsight Systems 文档指出 CUDA 和图节点追踪可能产生显著开销（用户指南：https://docs.nvidia.com/nsight-systems/UserGuide/index.html）。因此这里每个主机端结论都先进行了分析与未分析的校准。","微基准测试对冷权重内核运行时过于乐观。一个循环重复调用同一个内核会保持其 2.6 MB 门控权重驻留在 L2 中，而真实模型在连续调用同一层之间会通过大约 94 MB 的专家流量刷新 L2。热态 7 微秒，冷态 11 微秒：足以反转与库 GEMV 的排名。","峰值吞吐量是单窗口统计。基准测试的峰值是固定 1 秒网格上的最大值，因此大致存在 ±5% 的相位带：TTFT/TPOT 偏移会重新切分网格，而一个提高平均吞吐量 2.3% 的变化可能显示为从 909 降到 858。两个读数在固定种子下完全可复现，因此可复现性并不能区分信号与相位。在这里的 A/B 决策是基于平均 TPOT × 平均接受长度。该乘积是步骤时间的派生估计，而非测量值（两个聚合的乘积并非乘积的聚合），但它在各次运行中稳定，并且对这些运行中的接受长度漂移不敏感，这正是 A/B 判定所需的。我们报告峰值，但从未针对其进行优化。","正确性有它自己的门控，应用于每一次更改，确保更改生效之前：对256标记的贪婪生成进行逐字节比较，接受长度在0.05以内不变，并在交错温度采样请求后进行贪婪重运行以捕获状态污染。合法地改变舍入的更改（如bf16门控、单舍入合并）会在提交信息中说明，并在接受和任务指标上进行验证，而不是通过位一致性。","这是整个活动其余部分依赖的结构性更改，同时它也是一个主机模式空闲修复。","图5。锁步到深度流水线。之前：每一步主机都在resolve_seq_lens_cpu中阻塞，等待前一个验证图在GPU上完成，因此提前执行的深度重置为零，每个主机准备段都变成GPU气泡。之后：队列完全加深，每次启动验证k+1时其执行领先一步，唯一剩下的同步是延迟一步消费的copy_done事件。","cudaGraphLaunch一直是异步的，并且GPU上的draft → verify → extend顺序是免费的：同一流，FIFO。因此问题从来不是验证是否等待草稿，而是主机是否每一步都被绑定到GPU进度上。","确实是这样。在spec-v2下，调度器不知道接受长度，因此FutureMap.resolve_seq_lens_cpu()在构建下一批次时从GPU拉回new_seq_lens：以发布事件为门控，在私有流上拷贝，然后进行同步()。主机不是在等待微秒级别的拷贝，而是在等待前一个验证图完成执行。中位成本：每步485 µs，并且每一步运行深度重置为零。","原因是needs_cpu_seq_lens标志，在spec-v2涉及的每个后端上通过OR进行组合。trtllm_mla在三种角色中都声明为False；兄弟线性注意力后端GDNAttnBackend和Mamba2AttnBackend都显式声明为False。KDAAttnBackend从未声明该标志，并继承基类默认值True，尽管它运行与两个兄弟相同的基类元数据代码。","声明needs_cpu_seq_lens = False会折叠OR并移除每步同步。正确性论点是逐点的：KDA的元数据从未读取CPU镜像，并且回放填充来源于forward_batch.num_padding。","主机怎么敢在不知道步骤 k 接受了什么的情况下启动步骤 k+1？因为这些值从未接触 CPU。FutureMap 是驻留在 GPU 上的中继：步骤 k 的图将输出 token、新的序列长度、top-k 概率和隐藏状态写入由 req_pool_idx 索引的设备缓冲区，而步骤 k+1 的图通过相同的索引读取它们。主机只处理索引，而这些索引它已经知道。","图 6。松弛所在的位置。面板 A：主机循环（约 4.3 毫秒）小于 GPU 步骤（约 4.9 毫秒），所以它完全被隐藏。面板 B：当抖动（一次 gloo 广播或一次垃圾回收暂停）超过松弛，主机结束得晚，而 GPU 在下一个验证边界等待，这里图中的第一个集体操作吸收了跨秩的偏差。","提前执行也改变了主机消耗的形态。不是每个秩在每一步都直接支付其主机时间，只有耗尽队列松弛的秩才会支付。在一个四秩的跟踪中，正好有一个秩处于这种状态：它的调度器段运行时间是其同伴的 5-10 倍，它的草稿图启动晚了 40-80 微秒，它的草稿→验证接口比其他秩的中位数多 +165 微秒，并显示周期性的 400-750 微秒峰值，带有垃圾回收特征。其他三个秩在每次汇合时都在自旋等待它。普遍适用的诊断：一个内核的持续时间并不是它的工作。一个显示 150-480 微秒的 20 KB embedding 全归约并不是慢的全归约；它在吸收偏差，只有跨秩的时间对齐才能告诉你哪个秩延迟了。","随着锁步绑定的取消，图之间的接口值得缩短。在 CUDA 图重放之前，特定步骤的注意力元数据（kv 索引、块表、mamba 状态槽）必须从实时的 req_to_token 和 seq_lens 重建到图的捕获静态缓冲区中。这种补充在每一步都会主动执行，并且占接口的大部分。在批次 1 时，它纯粹受主机约束：每个操作调度时间为 5-15 微秒，执行时间为 1-4 微秒。","我们从两个层面进行攻击。首先，融合索引链：assign_extend_cache_locs_uniform 在内核中计算结束偏移量（统一的 draft_token_num 扩展使得跨行前缀和不再必要），而 _fused_state_indices_kernel 将 gather、translate、padding-sentinel 写入和 copy_ 合并到一次启动中，仔细保留两者副作用，包括在填充行上将 req_pool_indices 置零，该函数本身不需要，但图中其他捕获的内核依赖它来进行边界内的 gather。","其次，将 refill 本身捕获到由 (bs, forward_mode) 键控的小型 CUDA 图中。这可行的原因是回放契约已经保证的指针稳定性属性：replay ForwardBatch 视图仅向后端提供静态缓冲区和驻池张量，因此整个准备序列具有固定地址。周围有四个安全机制：两个主动预热，使 Triton JIT 和自动调优在捕获之外发生；每次回放前恢复每个后端的 forward_metadata 对象快照（图回放设备操作，快照恢复 Python 指针）；捕获失败时的永久主动回退和警告；以及针对于 padding、TBO、pdmux 和 LoRA 的保护。此功能通过 SGLANG_ENABLE_METADATA_GLUE_GRAPH 选择性启用，并且对于 DFLASH 系列推测强制禁用，因为该路径每步都在主机上重建注意力计划，如果捕获 refill，将在捕获时冻结计划。","关于可捕获内容存在硬性边界。标准是：由纯设备内核写入持久缓冲区的 refill 可捕获；任何通过 FlashInfer 风格 plan() 的操作则不可捕获。draft 端失败的原因是：多步骤 draft 后端重新 plan() 主 EAGLE 图已捕获的 wrapper，将该重新 plan 记录到次级图中会在回放时破坏 wrapper 的内部状态。相关要求是捕获必须是幂等的。trtllm_mla 的 _init_cuda_graph_metadata 过去会在每次调用时分配新的张量并替换其 decode_cuda_graph_metadata[bs] 条目，这会导致第二次捕获后早期图读取已释放的内存。","批处理-1 步骤在短时间窗口内执行数百个内核节点。在这种规模下，启动和前奏的开销大约与计算量相当。程序化依赖启动（PDL）允许消费者内核在其生产者仍在运行时被调度到SM上：消费者执行所有不依赖于生产者输出的操作，并且只在依赖读取前在 gdc_wait() 处设置屏障。","图7. 路由器路径上的PDL。没有PDL时，每个内核只能在前一个完全完成后才开始，而门控矩阵向量乘的冷HBM权重加载会成为关键路径。应用PDL后，权重块加载与生产者无关，因此它在 gdc_wait() 之前就发出，2.6 MB 的冷读取可以在生产者尾部下方执行；路由器 top-k 也是以同样的方式预取其偏置。","我们连接了三个链：MoE 主链（moe_align→up-GEMM→激活→down-GEMM→combine→all-reduce）、路由器链（norm→gate matvec→top-k）和 KDA 链（conv1d_update→循环 delta-rule→门控范数）。有两个设计点很重要。","与生产者无关的加载应在等待之前完成。这就是图中的全部技巧，也是使 PDL 对延迟受限的内核而言不仅仅是去除启动开销的原因。","感应内核不能携带PDL属性。小型M的MoE combine是由torch.compile生成的内核；加入链意味着用仓库的Triton reduction加上GDC来替换它。这带来了数值上的副作用：fp32求和 × 缩放只进行一次最终类型转换，而旧路径则进行了两次四舍五入。结果略微更精确，但不逐位相等，提交信息中有说明。","后来，在 PTX 的 griddepcontrol 中发现了一个问题后，我们升级了语义：launch_dependents 仅释放依赖网格的启动，而消费者的等待始终会在生产者网格完全完成时起到栅栏作用。将触发器从生产者的末尾移动到生产者自身等待之后立即触发，使消费者的前序可以与生产者主体的更多部分重叠，而不仅仅是它的尾部，但有一个前提条件：消费者在提前启动和每次读取生产者输出之间，仍然必须保持自己的 gdc_wait()。这是每个消费者自身的属性，而不是普遍保证，因此我们对每个内核进行了检查，并转换了六个。提前触发获得的收益也是不确定的：驱动程序可能会提前启动依赖网格，实际重叠的程度取决于当时的调度和资源压力 (CUDA 编程指南：https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/programmatic-dependent-launch.html)。fused_moe 将其限制在 M ≤ 512 的检查下：在预填充形状时，提前释放大型消费者网格会从生产者偷取 SM，而在解码形状时完全是收益。","PDL 是纯调度语义。累积顺序未改变的更改保持位级一致；gate matvec 通过了 4/4 GDC 开启/关闭的比特比较。","moe_align，在 pair 轴上。Triton 融合 MoE 的 GEMM 在 block_size 瓦片中消费 token，每行共享一个专家，而 moe_align_block_size 构建该排列。通用路径需要两次内核启动：在每个专家的偏移量最终确定之前，不能放置任何 token，这些偏移量来源于全网格扫描，并且设备范围内的屏障仅存在于内核边界。存在单次启动的变体，但它会将每个线程的专家计数器存放在共享内存中，因此限制为 64 个或更少专家；对于 513 个专家的解码，总是需要两次启动。","替换操作基于配对轴：[NP, NP] 的成对比较能一次性为每个 (token, slot) 对在其桶内及桶的总体人口中分配一个稳定的排名；随后，每个桶的排名为0的代表派生出填充计数、桶顺序的专属偏移量、公布总数以及每块的专家ID。任何操作都不会随着专家数量的增加而扩展，因此专家数量限制消失。显而易见的替代方案是对填充的专家轴（最多1024个桶）进行直方图和累积和操作，这种方法是正确的，但会在关键路径上增加大约3倍的单SM工作量，相比它替换的两个内核。这也是配对轴在这里承担负载的原因。","从参考中有两个刻意的偏离，两者都是基于消费者不变量的论证：桶内顺序在配对索引中保持稳定，而不是在原子调度顺序中（每对写入自己的输出行，所以消费者对顺序不敏感）；发布总数之后的缓存尾部保持未写（消费者CTA在读取前提前退出）。一个悬崖点：成对张量是 O(NP²)。在 NP=64 时，它们完全驻留在寄存器中（约4 µs，与 CUDA 的两内核路径相当），在 NP=256 时溢出到本地内存，每次启动大约耗时 230 µs。分发门槛严格限制 numel ≤ 64；较大的批次则退回到 CUDA 路径。","SwiGLU 在 up-GEMM 尾部。将 silu(gate) * up 折叠到 MoE up-GEMM 尾部，可以去掉每个 MoE 层独立的激活内核，以及中间缓存的全部写后读操作。布局技巧是在加载权重时对 w13 进行按专家行交错，使 gate 和 up 在同一输出块的相邻偶/奇列中对齐。由于每个 GEMM 输出列是独立的点乘，因此交错在位级上是中性的。","位奇偶性是关注重点。被替换的内核使用 -use_fast_math 编译，因此尾部逐条指令复现它：mul + ex2.approx.ftz 用于 __expf，div.approx.ftz，还有对乘积的最终单次舍入。微妙之处：FlashInfer 将激活仿函数实例化为 float，所以 silu 在乘法前从未进入 bf16。若在此处舍入，则结果会双重舍入，并在大量输入上产生偏差。这在文档中是不可见的，对容差检查也是不可见的；必须对整个输入范围逐元素进行位级比较才能发现。","KDA 链式验证块经济学。融合的 conv1d + 门控 delta 规则的验证内核已经存在；这些提交重新调整了它。在带有图内计时的旋转冷测试中，T=6 时的 Blackwell 曲线是单调的：BV=4 时 11.56 µs，8 时 12.53，16 时 12.83，32 时 14.26，64 时 20.7，128 时 38。BV=4 每次调用最多赢 19%，因为 256 个 CTA 在 148 个 SM 上是 1.7 波次，并且将 q/k 卷积复制 32 次仍然比缩短串行链便宜。V 维度的平铺从不触碰 K 维度的归约顺序，因此在 num_warps=4 时，每个 BV 在位级上与基线完全相同，重新调整没有数值风险。","最大的单一后结构变化是 dtype 的变化。在 batch 1 时，路由器门和 lm_head 完全受带宽限制：每个解码步骤都会冷读取每个 MoE 层的门权重（bf16 下为 2.6 MB）和词汇并行的 lm_head 投影，并且两者都没有算术操作来掩盖读取。将两者改为 bf16 而不是 fp32 可以减少一半的字节；端到端来看，大约带来 +10% 的提升，是主机预运行修复后的最大单项增益。像上文提到的其他舍入变化一样，这个也在提交信息中声明，并通过 accept length 和任务指标验证，而不是位级一致性。","被拒绝的推测 token 会使 KV 缓存项无害地过时，但它已经原地破坏了循环状态。线性注意力与推测不能免费共存。","使其工作的方案：在验证期间，递归运行时禁用状态更新，并将每个链位置的后状态写入中间缓冲区；在得到结果后，commit_mamba_states_after_verify 将最后一个被接受位置的状态复制到持久槽中。先 staging，再提交。这也是为什么紧凑型 spec 缓存限制为 topk=1：有链时，被接受的前缀是唯一的，状态可以按位置索引；有树时，被接受的路径是众多路径之一，状态必须按树路径索引。","分析显示 KDA 解码受带宽限制，主要是 K×V 状态的 HBM 流量，因此除了融合和平铺重新调整之外，没有太多改进空间。我们没有对比实际带宽与 Blackwell 峰值，因此应将其视为形状观察而非 roofline 结果。","权重-带宽优势有一个反直觉的推论：验证更多代币几乎是免费的。验证4个令牌和验证6个令牌的权重完全相同。在第一批加深的投机中，每增加一步就多花一个廉价的draft（draft为单层），加上KDA链验证递发中的增量序列成本，以及购买接受长度。","我们选择了扫掠它而不是假设它（该扫频发生在聚变丛之前;最优点随后移动，如下所述）：","步进时间列是TPOT，×接受长度，这是一种推导估计，而非直接测量。每增加一步大约耗时4-9%，而边际接受增益呈几何衰减（d5→d6仅增加0.08）。盈亏平衡条件大致为 Δ 接受 > 0.05 ×接受。最优配置也会变化：聚变束落地后，步进时间减少，（5， 6）成为更好的配置;一旦FP8配重进一步压缩固定底座，就需要再扫一次。","调优NEXTN的深度是固定形状算法中的一个一维旋钮。更大的杠杆是改变算法，战役后半段则用来让DSpark对抗同一目标，并给予它同样的批次-1处理。","DSpark 算法本身是公开的。这里的工作是将该公开配方改编为Ling-3.0闪存、长上下文在线蒸馏和第一批Blackwell的服务栈。我们的适应有四个方面的不同。","分布对齐的数据。我们主要基于Ling-3.0闪存的训练后数据进行提炼，因此草图训练时将面临的分布。我们还在蒸馏过程中使用多重采样设置，以提升轨迹多样性和推测解码下的鲁棒性。","一种消融驱动的草图设计。我们没有直接继承Ling-3.0闪存架构，而是对关键草稿选择进行了系统性的消融分析，包括是否重用Ling-3注意力结构以及采用哪种RoPE变体（部分或交错）。我们保留了在接受长度和延迟之间最佳权衡的设计。","一个配套的在线训练系统。针对长上下文和大规模在线训练，我们构建了 SplitServe Trainer，这是一种单节点 8 GPU 框架，将资源在训练和 SGLang 推理之间平均分配。在训练过程中，推理端执行目标前向计算以生成诸如草稿的目标隐藏状态等监督信号。这保持了生成-训练循环在本地进行，减少了 IO 开销，并提高了长上下文工作负载的训练效率。"],"translationStatus":"translated","bodyOrigin":"source-page","editorial":{"summary":"LMSYS 博客称，蚂蚁 Ling Infra 与 RadixArk SGLang 团队在 4 块 NVIDIA Blackwell GPU 上优化 Ling-3.0-flash，将单请求解码速度从 288 tok/s 提升至 606 tok/s，平均 TPOT 从 3.33 ms 降至 1.53 ms。","background":"Ling-3.0-flash 是混合线性注意力 MoE 模型，架构包含 42 层，其中 35 层为 KDA 线性注意力层、7 层为 MLA 全注意力层，并使用 512 专家 MoE。测试采用 TP4、bf16、并发 1、贪心解码及固定长度随机负载。","viewpoint":"Aioga 判断，这项工作的重点不只是峰值速度，而是针对 Batch-1 场景压缩推理栈开销。博客同时比较了 NEXTN 与 DSpark，但两者的吞吐和延迟指标应结合各自配置及测试口径理解。","implications":"可能的行业影响是，单请求低延迟会继续成为模型服务优化的重要指标，推测式解码、图执行修复和底层调优的组合价值值得关注。不过，结果来自合成随机负载，不能直接代表所有真实应用场景。","nextStep":"后续评估应优先复现相同硬件、模型、输入输出长度和解码设置，并分别核对 TTFT、TPOT、总墙钟时间、接受长度与任务准确率，避免仅以 tok/s 判断服务收益。","evidenceRefs":["title","summary","articleBody","source"],"status":"published","aiGenerated":true,"autoApproved":true,"generatedBy":"aioga-editorial:gpt-5.6-sol","reviewedBy":"aioga-editorial-review:gpt-5.6-sol","generatedAt":"2026-08-21T18:44:18.592Z","sourceHash":"efe007fbfe4500b8","review":{"approved":true,"groundedness":96,"clarity":91,"duplicationRisk":18,"blockingIssues":[],"notes":["“固定长度随机负载”可进一步明确为固定 8192-token 输入、1024-token 输出的合成随机负载。","“两者的吞吐和延迟指标应结合各自配置及测试口径理解”属于合理的编辑性概括；来源明确指出，只有最终调优 NEXTN 与 DSpark 是在同一命令、同一机器及相同 1000 请求条件下进行的受控比较。"]},"validation":{"passed":true,"mode":"ai-auto","revisions":0,"checks":["schema","length","source-attribution","low-source-overlap","no-html","independent-ai-review"]}},"tags":["行业动态","LMSYS：Blog（Chatbot Arena 团队）"],"translations":{"zh-CN":{"title":"Ling-3.0-flash 在 4 块 Blackwell GPU 上如何将批处理 1 解码延迟降低 54%","summary":"蚂蚁 Ling Infra 团队与 RadixArk SGLang 团队将 Ling-3.0-flash 混合线性注意力 MoE 模型的单请求解码速度从 288 tok/s 提升至 606 tok/s，平均 TPOT 从 3.33 ms 降至 1.53 ms。 🔗 阅读原文 via AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Ling-3.0-flash 在 4 块 Blackwell GPU 上如何将批处理 1 解码延迟降低 54% - Aioga AI资讯","description":"蚂蚁 Ling Infra 团队与 RadixArk SGLang 团队将 Ling-3.0-flash 混合线性注意力 MoE 模型的单请求解码速度从 288 tok/s 提升至 606 tok/s，平均 TPOT 从 3.33 ms 降至 1.53 ms。 🔗 阅读原文 via AIHOT · https://aihot.virxact.com/item...","url":"https://www.aioga.com/news/cmt393qov0kfhro6tuwhxhubl/","articleBody":["批量1解码变得越来越重要。例如，小米MiMo在6月发布了MiMo-V2.5-Pro UltraSpeed：https://mimo.xiaomi.com/blog/mimo-tilert-1000tps，声称在一个万亿参数的MoE模型上实现每秒1000个token的解码。","批量1给推理堆栈留下的开销空间非常有限。没有批量可以分摊启动成本，也没有并发来填充流水线空隙，并且算术强度不足以让巧妙的平铺获得收益。关键路径上的每一微秒都是用户等待的微秒。","本文讨论的是如何在4块NVIDIA Blackwell GPU上将Ling-3.0-flash（一个混合线性注意力MoE模型）的这一底线推低。它涵盖了两种推测解码路径。在NEXTN/MTP路径上，我们将单请求解码速度从288 tok/s提升至606 tok/s，平均TPOT从3.33 ms降低到1.53 ms。第二条路径是DSpark，这是一个基于相同堆栈构建的、使用置信度调度的推测解码器：1000请求运行达到1120 tok/s，平均TPOT为0.78 ms，接收长度为9.95。最后的比较是受控的：NEXTN和DSpark在同一台机器上使用相同命令测量，平均TPOT降低了1.9倍（从1.53 ms降到0.78 ms）。本文其余部分将介绍这些时间都花在了哪里，以及如何将其找回。","图1. 在四个配置下的主要结果。","在GSM8K上，相同堆栈得分为：准确率0.889，无效0.000，延迟341.5秒，输出吞吐量511.1 tok/s。","所有运行均使用4块Blackwell GPU上的Ling-3.0-flash，TP4，bf16，并发1，贪婪解码，以及相同的固定8192输入/1024输出随机工作负载。从左到右，列显示初始NEXTN基线、修复draft-extend图后的NEXTN、最终调优后的NEXTN以及DSpark。前两列为较短的实验检查点；后两列为受控比较，每列在同一台机器上测量相同的1000个请求。峰值吞吐量仅在最后两次运行中比较，因为它是固定一秒窗口中的最大值。","这里有两个定义很重要，因为它们一起解释了为什么输出吞吐量并不是平均 TPOT 的简单倒数，即使在并发为 1 时也如此：SGLang 的 TPOT 不包括 TTFT，而输出吞吐量则是将总输出令牌数除以基准测试总墙钟时间（参见 bench_serving 指南：https://github.com/sgl-project/sglang/blob/main/docs/developer_guide/bench_serving.md）。本文中的所有主要基准测试运行都使用了合成随机工作负载；accept length 特别依赖于提示和输出分布，因此 9.95 是此工作负载的 accept length，而不是模型的。","图 2. Ling-3.0-Flash 架构：42 层，交错排列 35 层 KDA 线性注意力层和 7 层 MLA 全注意力层，基于 512 专家 MoE。","Ling-3.0-flash 是一个混合注意力 MoE 模型（BailingMoeV3），以下大部分内容来自“混合”这个词。","每六层注意力层中有五层是 KDA。这就是为什么在最终分析中，MLA 注意力在 8k 上下文中的每步耗时仅为 244 微秒的原因，也是该模型一开始就是一个优秀的 batch-1 目标的原因：由于注意力开销小且批量非常小，剩余的关键路径就是权重带宽和启动延迟，这正是本文讨论的场景。","我们在 steps=5, topk=1, draft_tokens=6 的情况下使用 NEXTN 推测解码进行解码。一次解码步骤是在一个中继中进行的三个 CUDA 图。","图 3. 每步三个图。草稿模型提出一个 6 令牌链，目标模型在一次前向中评分全部六个，extend 图使用目标模型的真实隐藏状态重放已接受的前缀，以生成下一轮的种子。判定本身（eagle_sample）发生在 verify 图中；主机迟一步才知道接受了多少令牌。","草稿是一个单层 NEXTN 模型，自回归运行：五步但仅四次前向，因为第一个候选来自前一轮的种子，第五个从第四次前向的 top-k 中读取。Verify 是对全部六个链位置执行一次包含 42 层的目标模型前向。Extend 修复草稿的 KV 缓存，该缓存仅见过草稿自身的猜测，并将种子交回下一轮。","CPU上的三个图之间交叉的部分实际上是没有的。固定形状加上填充使每个依赖接受的计数成为GPU索引，而不是主机值；持久缓冲区让生产者图直接写入消费者缓冲区；而真正需要CPU值的决策（EOS、停止字符串、去标记化）则通过一个旁路D2H流和一个延迟一步消费的copy_done事件来处理。下面的一切都基于这个特性。","当我们开始时，GPU在每步中大约有三分之二的时间处于繁忙状态。在批量为1时的空闲时间有两种类型，它们需要分别诊断，因为解决方法完全不同：","图4. 两种类型的空闲。上图：主机循环比GPU的工作长，因此空隙少而宽，并落在图之间的缝隙中。下图：一旦主机被隐藏，剩下的是几百个运行时间为1.5-6 µs的核节点，它们的启动开销相当于计算时间，加上权重读取本身。","这两种空闲描述了TPOT的每步时间侧。另一个杠杆是每步提交的令牌数：平均TPOT≈步长时间/平均接受长度。本文其余部分都基于这些杠杆。主机运行预读和缝隙工作消除主机模式下的空闲；PDL、数据类型更改、融合和重新调优缩短GPU关键路径；投机调优和DSpark增加每个目标步骤提交的令牌数。DSpark之后在阻塞D2H读取重新引入主机固定时，会重新审视第一类空闲。","测量设置的三个特性塑造了下面的每一个数值。","分析器会夸大主机端事件。CUPTI 会为它记录的每个主机事件增加开销。在相同配置下，一个被分析的步骤测量为 5.2 毫秒，而实际步骤（通过在未分析运行中从 TPOT × 接受长度反向计算）为 4.9 毫秒。那 0.3 毫秒的差距与我们想要分析的主机端效应同数量级，因此分析轨迹可能显示跨节点等待，而在未分析情况下并不存在。GPU 内核的持续时间来源于硬件时间戳，比主机端计时更可靠，但也不是免疫的：追踪仍然会扰动启动时机、并发性、缓存状态以及 CUDA 图执行，而且 Nsight Systems 文档指出 CUDA 和图节点追踪可能产生显著开销（用户指南：https://docs.nvidia.com/nsight-systems/UserGuide/index.html）。因此这里每个主机端结论都先进行了分析与未分析的校准。","微基准测试对冷权重内核运行时过于乐观。一个循环重复调用同一个内核会保持其 2.6 MB 门控权重驻留在 L2 中，而真实模型在连续调用同一层之间会通过大约 94 MB 的专家流量刷新 L2。热态 7 微秒，冷态 11 微秒：足以反转与库 GEMV 的排名。","峰值吞吐量是单窗口统计。基准测试的峰值是固定 1 秒网格上的最大值，因此大致存在 ±5% 的相位带：TTFT/TPOT 偏移会重新切分网格，而一个提高平均吞吐量 2.3% 的变化可能显示为从 909 降到 858。两个读数在固定种子下完全可复现，因此可复现性并不能区分信号与相位。在这里的 A/B 决策是基于平均 TPOT × 平均接受长度。该乘积是步骤时间的派生估计，而非测量值（两个聚合的乘积并非乘积的聚合），但它在各次运行中稳定，并且对这些运行中的接受长度漂移不敏感，这正是 A/B 判定所需的。我们报告峰值，但从未针对其进行优化。","正确性有它自己的门控，应用于每一次更改，确保更改生效之前：对256标记的贪婪生成进行逐字节比较，接受长度在0.05以内不变，并在交错温度采样请求后进行贪婪重运行以捕获状态污染。合法地改变舍入的更改（如bf16门控、单舍入合并）会在提交信息中说明，并在接受和任务指标上进行验证，而不是通过位一致性。","这是整个活动其余部分依赖的结构性更改，同时它也是一个主机模式空闲修复。","图5。锁步到深度流水线。之前：每一步主机都在resolve_seq_lens_cpu中阻塞，等待前一个验证图在GPU上完成，因此提前执行的深度重置为零，每个主机准备段都变成GPU气泡。之后：队列完全加深，每次启动验证k+1时其执行领先一步，唯一剩下的同步是延迟一步消费的copy_done事件。","cudaGraphLaunch一直是异步的，并且GPU上的draft → verify → extend顺序是免费的：同一流，FIFO。因此问题从来不是验证是否等待草稿，而是主机是否每一步都被绑定到GPU进度上。","确实是这样。在spec-v2下，调度器不知道接受长度，因此FutureMap.resolve_seq_lens_cpu()在构建下一批次时从GPU拉回new_seq_lens：以发布事件为门控，在私有流上拷贝，然后进行同步()。主机不是在等待微秒级别的拷贝，而是在等待前一个验证图完成执行。中位成本：每步485 µs，并且每一步运行深度重置为零。","原因是needs_cpu_seq_lens标志，在spec-v2涉及的每个后端上通过OR进行组合。trtllm_mla在三种角色中都声明为False；兄弟线性注意力后端GDNAttnBackend和Mamba2AttnBackend都显式声明为False。KDAAttnBackend从未声明该标志，并继承基类默认值True，尽管它运行与两个兄弟相同的基类元数据代码。","声明needs_cpu_seq_lens = False会折叠OR并移除每步同步。正确性论点是逐点的：KDA的元数据从未读取CPU镜像，并且回放填充来源于forward_batch.num_padding。","主机怎么敢在不知道步骤 k 接受了什么的情况下启动步骤 k+1？因为这些值从未接触 CPU。FutureMap 是驻留在 GPU 上的中继：步骤 k 的图将输出 token、新的序列长度、top-k 概率和隐藏状态写入由 req_pool_idx 索引的设备缓冲区，而步骤 k+1 的图通过相同的索引读取它们。主机只处理索引，而这些索引它已经知道。","图 6。松弛所在的位置。面板 A：主机循环（约 4.3 毫秒）小于 GPU 步骤（约 4.9 毫秒），所以它完全被隐藏。面板 B：当抖动（一次 gloo 广播或一次垃圾回收暂停）超过松弛，主机结束得晚，而 GPU 在下一个验证边界等待，这里图中的第一个集体操作吸收了跨秩的偏差。","提前执行也改变了主机消耗的形态。不是每个秩在每一步都直接支付其主机时间，只有耗尽队列松弛的秩才会支付。在一个四秩的跟踪中，正好有一个秩处于这种状态：它的调度器段运行时间是其同伴的 5-10 倍，它的草稿图启动晚了 40-80 微秒，它的草稿→验证接口比其他秩的中位数多 +165 微秒，并显示周期性的 400-750 微秒峰值，带有垃圾回收特征。其他三个秩在每次汇合时都在自旋等待它。普遍适用的诊断：一个内核的持续时间并不是它的工作。一个显示 150-480 微秒的 20 KB embedding 全归约并不是慢的全归约；它在吸收偏差，只有跨秩的时间对齐才能告诉你哪个秩延迟了。","随着锁步绑定的取消，图之间的接口值得缩短。在 CUDA 图重放之前，特定步骤的注意力元数据（kv 索引、块表、mamba 状态槽）必须从实时的 req_to_token 和 seq_lens 重建到图的捕获静态缓冲区中。这种补充在每一步都会主动执行，并且占接口的大部分。在批次 1 时，它纯粹受主机约束：每个操作调度时间为 5-15 微秒，执行时间为 1-4 微秒。","我们从两个层面进行攻击。首先，融合索引链：assign_extend_cache_locs_uniform 在内核中计算结束偏移量（统一的 draft_token_num 扩展使得跨行前缀和不再必要），而 _fused_state_indices_kernel 将 gather、translate、padding-sentinel 写入和 copy_ 合并到一次启动中，仔细保留两者副作用，包括在填充行上将 req_pool_indices 置零，该函数本身不需要，但图中其他捕获的内核依赖它来进行边界内的 gather。","其次，将 refill 本身捕获到由 (bs, forward_mode) 键控的小型 CUDA 图中。这可行的原因是回放契约已经保证的指针稳定性属性：replay ForwardBatch 视图仅向后端提供静态缓冲区和驻池张量，因此整个准备序列具有固定地址。周围有四个安全机制：两个主动预热，使 Triton JIT 和自动调优在捕获之外发生；每次回放前恢复每个后端的 forward_metadata 对象快照（图回放设备操作，快照恢复 Python 指针）；捕获失败时的永久主动回退和警告；以及针对于 padding、TBO、pdmux 和 LoRA 的保护。此功能通过 SGLANG_ENABLE_METADATA_GLUE_GRAPH 选择性启用，并且对于 DFLASH 系列推测强制禁用，因为该路径每步都在主机上重建注意力计划，如果捕获 refill，将在捕获时冻结计划。","关于可捕获内容存在硬性边界。标准是：由纯设备内核写入持久缓冲区的 refill 可捕获；任何通过 FlashInfer 风格 plan() 的操作则不可捕获。draft 端失败的原因是：多步骤 draft 后端重新 plan() 主 EAGLE 图已捕获的 wrapper，将该重新 plan 记录到次级图中会在回放时破坏 wrapper 的内部状态。相关要求是捕获必须是幂等的。trtllm_mla 的 _init_cuda_graph_metadata 过去会在每次调用时分配新的张量并替换其 decode_cuda_graph_metadata[bs] 条目，这会导致第二次捕获后早期图读取已释放的内存。","批处理-1 步骤在短时间窗口内执行数百个内核节点。在这种规模下，启动和前奏的开销大约与计算量相当。程序化依赖启动（PDL）允许消费者内核在其生产者仍在运行时被调度到SM上：消费者执行所有不依赖于生产者输出的操作，并且只在依赖读取前在 gdc_wait() 处设置屏障。","图7. 路由器路径上的PDL。没有PDL时，每个内核只能在前一个完全完成后才开始，而门控矩阵向量乘的冷HBM权重加载会成为关键路径。应用PDL后，权重块加载与生产者无关，因此它在 gdc_wait() 之前就发出，2.6 MB 的冷读取可以在生产者尾部下方执行；路由器 top-k 也是以同样的方式预取其偏置。","我们连接了三个链：MoE 主链（moe_align→up-GEMM→激活→down-GEMM→combine→all-reduce）、路由器链（norm→gate matvec→top-k）和 KDA 链（conv1d_update→循环 delta-rule→门控范数）。有两个设计点很重要。","与生产者无关的加载应在等待之前完成。这就是图中的全部技巧，也是使 PDL 对延迟受限的内核而言不仅仅是去除启动开销的原因。","感应内核不能携带PDL属性。小型M的MoE combine是由torch.compile生成的内核；加入链意味着用仓库的Triton reduction加上GDC来替换它。这带来了数值上的副作用：fp32求和 × 缩放只进行一次最终类型转换，而旧路径则进行了两次四舍五入。结果略微更精确，但不逐位相等，提交信息中有说明。","后来，在 PTX 的 griddepcontrol 中发现了一个问题后，我们升级了语义：launch_dependents 仅释放依赖网格的启动，而消费者的等待始终会在生产者网格完全完成时起到栅栏作用。将触发器从生产者的末尾移动到生产者自身等待之后立即触发，使消费者的前序可以与生产者主体的更多部分重叠，而不仅仅是它的尾部，但有一个前提条件：消费者在提前启动和每次读取生产者输出之间，仍然必须保持自己的 gdc_wait()。这是每个消费者自身的属性，而不是普遍保证，因此我们对每个内核进行了检查，并转换了六个。提前触发获得的收益也是不确定的：驱动程序可能会提前启动依赖网格，实际重叠的程度取决于当时的调度和资源压力 (CUDA 编程指南：https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/programmatic-dependent-launch.html)。fused_moe 将其限制在 M ≤ 512 的检查下：在预填充形状时，提前释放大型消费者网格会从生产者偷取 SM，而在解码形状时完全是收益。","PDL 是纯调度语义。累积顺序未改变的更改保持位级一致；gate matvec 通过了 4/4 GDC 开启/关闭的比特比较。","moe_align，在 pair 轴上。Triton 融合 MoE 的 GEMM 在 block_size 瓦片中消费 token，每行共享一个专家，而 moe_align_block_size 构建该排列。通用路径需要两次内核启动：在每个专家的偏移量最终确定之前，不能放置任何 token，这些偏移量来源于全网格扫描，并且设备范围内的屏障仅存在于内核边界。存在单次启动的变体，但它会将每个线程的专家计数器存放在共享内存中，因此限制为 64 个或更少专家；对于 513 个专家的解码，总是需要两次启动。","替换操作基于配对轴：[NP, NP] 的成对比较能一次性为每个 (token, slot) 对在其桶内及桶的总体人口中分配一个稳定的排名；随后，每个桶的排名为0的代表派生出填充计数、桶顺序的专属偏移量、公布总数以及每块的专家ID。任何操作都不会随着专家数量的增加而扩展，因此专家数量限制消失。显而易见的替代方案是对填充的专家轴（最多1024个桶）进行直方图和累积和操作，这种方法是正确的，但会在关键路径上增加大约3倍的单SM工作量，相比它替换的两个内核。这也是配对轴在这里承担负载的原因。","从参考中有两个刻意的偏离，两者都是基于消费者不变量的论证：桶内顺序在配对索引中保持稳定，而不是在原子调度顺序中（每对写入自己的输出行，所以消费者对顺序不敏感）；发布总数之后的缓存尾部保持未写（消费者CTA在读取前提前退出）。一个悬崖点：成对张量是 O(NP²)。在 NP=64 时，它们完全驻留在寄存器中（约4 µs，与 CUDA 的两内核路径相当），在 NP=256 时溢出到本地内存，每次启动大约耗时 230 µs。分发门槛严格限制 numel ≤ 64；较大的批次则退回到 CUDA 路径。","SwiGLU 在 up-GEMM 尾部。将 silu(gate) * up 折叠到 MoE up-GEMM 尾部，可以去掉每个 MoE 层独立的激活内核，以及中间缓存的全部写后读操作。布局技巧是在加载权重时对 w13 进行按专家行交错，使 gate 和 up 在同一输出块的相邻偶/奇列中对齐。由于每个 GEMM 输出列是独立的点乘，因此交错在位级上是中性的。","位奇偶性是关注重点。被替换的内核使用 -use_fast_math 编译，因此尾部逐条指令复现它：mul + ex2.approx.ftz 用于 __expf，div.approx.ftz，还有对乘积的最终单次舍入。微妙之处：FlashInfer 将激活仿函数实例化为 float，所以 silu 在乘法前从未进入 bf16。若在此处舍入，则结果会双重舍入，并在大量输入上产生偏差。这在文档中是不可见的，对容差检查也是不可见的；必须对整个输入范围逐元素进行位级比较才能发现。","KDA 链式验证块经济学。融合的 conv1d + 门控 delta 规则的验证内核已经存在；这些提交重新调整了它。在带有图内计时的旋转冷测试中，T=6 时的 Blackwell 曲线是单调的：BV=4 时 11.56 µs，8 时 12.53，16 时 12.83，32 时 14.26，64 时 20.7，128 时 38。BV=4 每次调用最多赢 19%，因为 256 个 CTA 在 148 个 SM 上是 1.7 波次，并且将 q/k 卷积复制 32 次仍然比缩短串行链便宜。V 维度的平铺从不触碰 K 维度的归约顺序，因此在 num_warps=4 时，每个 BV 在位级上与基线完全相同，重新调整没有数值风险。","最大的单一后结构变化是 dtype 的变化。在 batch 1 时，路由器门和 lm_head 完全受带宽限制：每个解码步骤都会冷读取每个 MoE 层的门权重（bf16 下为 2.6 MB）和词汇并行的 lm_head 投影，并且两者都没有算术操作来掩盖读取。将两者改为 bf16 而不是 fp32 可以减少一半的字节；端到端来看，大约带来 +10% 的提升，是主机预运行修复后的最大单项增益。像上文提到的其他舍入变化一样，这个也在提交信息中声明，并通过 accept length 和任务指标验证，而不是位级一致性。","被拒绝的推测 token 会使 KV 缓存项无害地过时，但它已经原地破坏了循环状态。线性注意力与推测不能免费共存。","使其工作的方案：在验证期间，递归运行时禁用状态更新，并将每个链位置的后状态写入中间缓冲区；在得到结果后，commit_mamba_states_after_verify 将最后一个被接受位置的状态复制到持久槽中。先 staging，再提交。这也是为什么紧凑型 spec 缓存限制为 topk=1：有链时，被接受的前缀是唯一的，状态可以按位置索引；有树时，被接受的路径是众多路径之一，状态必须按树路径索引。","分析显示 KDA 解码受带宽限制，主要是 K×V 状态的 HBM 流量，因此除了融合和平铺重新调整之外，没有太多改进空间。我们没有对比实际带宽与 Blackwell 峰值，因此应将其视为形状观察而非 roofline 结果。","权重-带宽优势有一个反直觉的推论：验证更多代币几乎是免费的。验证4个令牌和验证6个令牌的权重完全相同。在第一批加深的投机中，每增加一步就多花一个廉价的draft（draft为单层），加上KDA链验证递发中的增量序列成本，以及购买接受长度。","我们选择了扫掠它而不是假设它（该扫频发生在聚变丛之前;最优点随后移动，如下所述）：","步进时间列是TPOT，×接受长度，这是一种推导估计，而非直接测量。每增加一步大约耗时4-9%，而边际接受增益呈几何衰减（d5→d6仅增加0.08）。盈亏平衡条件大致为 Δ 接受 > 0.05 ×接受。最优配置也会变化：聚变束落地后，步进时间减少，（5， 6）成为更好的配置;一旦FP8配重进一步压缩固定底座，就需要再扫一次。","调优NEXTN的深度是固定形状算法中的一个一维旋钮。更大的杠杆是改变算法，战役后半段则用来让DSpark对抗同一目标，并给予它同样的批次-1处理。","DSpark 算法本身是公开的。这里的工作是将该公开配方改编为Ling-3.0闪存、长上下文在线蒸馏和第一批Blackwell的服务栈。我们的适应有四个方面的不同。","分布对齐的数据。我们主要基于Ling-3.0闪存的训练后数据进行提炼，因此草图训练时将面临的分布。我们还在蒸馏过程中使用多重采样设置，以提升轨迹多样性和推测解码下的鲁棒性。","一种消融驱动的草图设计。我们没有直接继承Ling-3.0闪存架构，而是对关键草稿选择进行了系统性的消融分析，包括是否重用Ling-3注意力结构以及采用哪种RoPE变体（部分或交错）。我们保留了在接受长度和延迟之间最佳权衡的设计。","一个配套的在线训练系统。针对长上下文和大规模在线训练，我们构建了 SplitServe Trainer，这是一种单节点 8 GPU 框架，将资源在训练和 SGLang 推理之间平均分配。在训练过程中，推理端执行目标前向计算以生成诸如草稿的目标隐藏状态等监督信号。这保持了生成-训练循环在本地进行，减少了 IO 开销，并提高了长上下文工作负载的训练效率。"]},"en":{"title":"How Ling-3.0-flash reduces batch 1 decoding latency by 54% on 4 Blackwell GPUs","summary":"The Ant Ling Infra team and the RadixArk SGLang team increased the single-request decoding speed of the Ling-3.0-flash hybrid linear attention MoE model from 288 tok/s to 606 tok/s, with the average TPOT dropping from 3.33 ms to 1.53 ms. 🔗 Read the original article via AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"Industry","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"How Ling-3.0-flash reduces batch 1 decoding latency by 54% on 4 Blackwell GPUs - Aioga AI News","description":"The Ant Ling Infra team and the RadixArk SGLang team increased the single-request decoding speed of the Ling-3.0-flash hybrid linear attention MoE model from 288 tok/s to 606 tok/s...","url":"https://www.aioga.com/en/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:23:55.730Z"},"ja":{"title":"Ling-3.0-flashが4つのBlackwell GPUでバッチ1のデコード遅延を54%削減する方法","summary":"Ant Ling InfraチームとRadixArk SGLangチームは、Ling-3.0フラッシュハイブリッドリニアアテンションMoEモデルの単一リクエスト復号速度を288 tok/sから606 tok/sに向上させ、平均TPOT速度は3.33 msから1.53 msに低下しました。 🔗 原文記事はAIHOTより読むことができます。 https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"業界動向","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Ling-3.0-flashが4つのBlackwell GPUでバッチ1のデコード遅延を54%削減する方法 - Aioga AIニュース","description":"Ant Ling InfraチームとRadixArk SGLangチームは、Ling-3.0フラッシュハイブリッドリニアアテンションMoEモデルの単一リクエスト復号速度を288 tok/sから606 tok/sに向上させ、平均TPOT速度は3.33 msから1.53 msに低下しました。 🔗 原文記事はAIHOTより読むことができます。 https://ai...","url":"https://www.aioga.com/ja/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:23:57.371Z"},"ko":{"title":"Ling-3.0-플래시가 4개의 Blackwell GPU에서 배치 1 디코딩 지연을 54% 줄이는 방법","summary":"Ant Ling 인프라 팀과 RadixArk SGLang 팀은 Ling-3.0-플래시 하이브리드 선형 주의 MoE 모델의 단일 요청 디코딩 속도를 288 tok/s에서 606 tok/s로 향상시켰으며, 평균 TPOT는 3.33ms에서 1.53ms로 감소했습니다. 🔗 원문 기사는 AIHOT를 통해 읽을 수 있습니다. https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"업계 동향","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Ling-3.0-플래시가 4개의 Blackwell GPU에서 배치 1 디코딩 지연을 54% 줄이는 방법 - Aioga AI 뉴스","description":"Ant Ling 인프라 팀과 RadixArk SGLang 팀은 Ling-3.0-플래시 하이브리드 선형 주의 MoE 모델의 단일 요청 디코딩 속도를 288 tok/s에서 606 tok/s로 향상시켰으며, 평균 TPOT는 3.33ms에서 1.53ms로 감소했습니다. 🔗 원문 기사는 AIHOT를 통해 읽을 수 있습니다. htt...","url":"https://www.aioga.com/ko/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:06.101Z"},"es":{"title":"Cómo Ling-3.0-flash reduce la latencia de decodificación del lote 1 en un 54% en 4 GPUs Blackwell","summary":"El equipo de Ant Ling Infra y el equipo RadixArk SGLang aumentaron la velocidad de decodificación de solicitud única del modelo MoE híbrido de atención lineal Ling-3.0-flash de 288 tok/s a 606 tok/s, con la TPOT media bajando de 3,33 ms a 1,53 ms. 🔗 Lee el artículo original a través de AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"Industria","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Cómo Ling-3.0-flash reduce la latencia de decodificación del lote 1 en un 54% en 4 GPUs Blackwell - Aioga Noticias de IA","description":"El equipo de Ant Ling Infra y el equipo RadixArk SGLang aumentaron la velocidad de decodificación de solicitud única del modelo MoE híbrido de atención lineal Ling-3.0-flash de 288...","url":"https://www.aioga.com/es/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:06.275Z"},"fr":{"title":"Comment le Ling-3.0-flash réduit la latence de décodage du premier lot de 54 % sur 4 GPU Blackwell","summary":"L’équipe Ant Ling Infra et l’équipe RadixArk SGLang ont augmenté la vitesse de décodage à requête unique du modèle MoE hybride à attention linéaire Ling-3.0-flash de 288 tok/s à 606 tok/s, la TPOT moyenne passant de 3,33 ms à 1,53 ms. 🔗 Lisez l’article original via AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"Industrie","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Comment le Ling-3.0-flash réduit la latence de décodage du premier lot de 54 % sur 4 GPU Blackwell - Aioga Actualités IA","description":"L’équipe Ant Ling Infra et l’équipe RadixArk SGLang ont augmenté la vitesse de décodage à requête unique du modèle MoE hybride à attention linéaire Ling-3.0-flash de 288 tok/s à 60...","url":"https://www.aioga.com/fr/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:14.019Z"},"de":{"title":"Wie Ling-3.0-Flash die Batch-1-Dekodierungslatenz bei 4 Blackwell-GPUs um 54 % reduziert","summary":"Das Ant Ling Infra-Team und das RadixArk SGLang-Team erhöhten die Single-Request-Dekodierungsgeschwindigkeit des Ling-3.0-Flash-hybriden linearen Aufmerksamkeits-MoE-Modells von 288 Tok/s auf 606 Tok/s, wobei der durchschnittliche TPOT-Wert von 3,33 ms auf 1,53 ms sank. 🔗 Lesen Sie den Originalartikel über AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Wie Ling-3.0-Flash die Batch-1-Dekodierungslatenz bei 4 Blackwell-GPUs um 54 % reduziert - Aioga KI-News","description":"Das Ant Ling Infra-Team und das RadixArk SGLang-Team erhöhten die Single-Request-Dekodierungsgeschwindigkeit des Ling-3.0-Flash-hybriden linearen Aufmerksamkeits-MoE-Modells von 28...","url":"https://www.aioga.com/de/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:14.255Z"},"pt-BR":{"title":"Como o Ling-3.0-flash reduz a latência de decodificação do lote 1 em 54% em 4 GPUs Blackwell","summary":"A equipe Ant Ling Infra e a equipe RadixArk SGLang aumentaram a velocidade de decodificação de solicitação única do modelo MoE híbrido de atenção linear Ling-3.0-flash de 288 tok/s para 606 tok/s, com a média do TPOT caindo de 3,33 ms para 1,53 ms. 🔗 Leia o artigo original via AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Como o Ling-3.0-flash reduz a latência de decodificação do lote 1 em 54% em 4 GPUs Blackwell - Aioga Notícias de IA","description":"A equipe Ant Ling Infra e a equipe RadixArk SGLang aumentaram a velocidade de decodificação de solicitação única do modelo MoE híbrido de atenção linear Ling-3.0-flash de 288 tok/s...","url":"https://www.aioga.com/pt-BR/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:23.009Z"},"ru":{"title":"Как flash Ling-3.0 снижает задержку декодирования партии 1 на 54% на 4 GPU Blackwell","summary":"Команда Ant Ling Infra и команда RadixArk SGLang увеличили скорость декодирования с одним запросом гибридной модели Ling-3.0 flash linear attention MoE с 288 ток/с до 606 ток/с, при этом средний TPOT снизился с 3,33 мс до 1,53 мс. 🔗 Прочитайте оригинальную статью на сайте AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Как flash Ling-3.0 снижает задержку декодирования партии 1 на 54% на 4 GPU Blackwell - Aioga Новости ИИ","description":"Команда Ant Ling Infra и команда RadixArk SGLang увеличили скорость декодирования с одним запросом гибридной модели Ling-3.0 flash linear attention MoE с 288 ток/с до 606 ток/с, пр...","url":"https://www.aioga.com/ru/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:22.180Z"},"ar":{"title":"كيف يقلل Ling-3.0-flash من تأخير فك التشفير في دفعة 1 بنسبة 54٪ على 4 وحدات معالجة رسومات Blackwell","summary":"قام فريق Ant Ling Infra وفريق RadixArk SGLang بزيادة سرعة فك التشفير بطلب واحد لنموذج Ling-3.0 فلاش الهجين للانتباه الخطي MoE من 288 توك/ثانية إلى 606 توك/ثانية، مع انخفاض متوسط TPOT من 3.33 مللي ثانية إلى 1.53 مللي ثانية. 🔗 اقرأ المقال الأصلي عبر AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"كيف يقلل Ling-3.0-flash من تأخير فك التشفير في دفعة 1 بنسبة 54٪ على 4 وحدات معالجة رسومات Blackwell - Aioga أخبار الذكاء الاصطناعي","description":"قام فريق Ant Ling Infra وفريق RadixArk SGLang بزيادة سرعة فك التشفير بطلب واحد لنموذج Ling-3.0 فلاش الهجين للانتباه الخطي MoE من 288 توك/ثانية إلى 606 توك/ثانية، مع انخفاض متوسط TP...","url":"https://www.aioga.com/ar/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:31.798Z"},"hi":{"title":"कैसे Ling-3.0-flash 1 ब्लैकवेल GPU पर बैच 4 डिकोडिंग विलंबता को 54% तक कम कर देता है","summary":"एंट लिंग इंफ्रा टीम और रेडिक्सआर्क एसजीलैंग टीम ने लिंग-3.0-फ्लैश हाइब्रिड लीनियर अटेंशन एमओई मॉडल की एकल-अनुरोध डिकोडिंग गति को 288 टोक/सेकेंड से बढ़ाकर 606 टोक/सेकेंड कर दिया, जिसमें औसत टीपीओटी 3.33 एमएस से घटकर 1.53 एमएस हो गया। 🔗 AIHOT के माध्यम से मूल लेख पढ़ें · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"कैसे Ling-3.0-flash 1 ब्लैकवेल GPU पर बैच 4 डिकोडिंग विलंबता को 54% तक कम कर देता है - Aioga AI समाचार","description":"एंट लिंग इंफ्रा टीम और रेडिक्सआर्क एसजीलैंग टीम ने लिंग-3.0-फ्लैश हाइब्रिड लीनियर अटेंशन एमओई मॉडल की एकल-अनुरोध डिकोडिंग गति को 288 टोक/सेकेंड से बढ़ाकर 606 टोक/सेकेंड कर दिया, जि...","url":"https://www.aioga.com/hi/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:31.276Z"},"it":{"title":"Come il Ling-3.0-flash riduce la latenza di decodifica del batch 1 del 54% su 4 GPU Blackwell","summary":"Il team Ant Ling Infra e il team RadixArk SGLang hanno aumentato la velocità di decodifica a richiesta singola del modello MoE ibrido Ling-3.0-flash con attenzione lineare da 288 tok/s a 606 tok/s, con la media del TPOT che scende da 3,33 ms a 1,53 ms. 🔗 Leggi l'articolo originale su AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Come il Ling-3.0-flash riduce la latenza di decodifica del batch 1 del 54% su 4 GPU Blackwell - Aioga Notizie IA","description":"Il team Ant Ling Infra e il team RadixArk SGLang hanno aumentato la velocità di decodifica a richiesta singola del modello MoE ibrido Ling-3.0-flash con attenzione lineare da 288 t...","url":"https://www.aioga.com/it/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:40.516Z"},"nl":{"title":"Hoe Ling-3.0-flash batch 1-decodeerlatentie met 54% vermindert op 4 Blackwell GPU's","summary":"Het Ant Ling Infra-team en het RadixArk SGLang-team verhoogden de single-request decodeersnelheid van het Ling-3.0-flash hybride lineaire aandacht MoE-model van 288 tok/s naar 606 tok/s, waarbij de gemiddelde TPOT daalde van 3,33 ms naar 1,53 ms. 🔗 Lees het originele artikel via AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Hoe Ling-3.0-flash batch 1-decodeerlatentie met 54% vermindert op 4 Blackwell GPU's - Aioga AI-nieuws","description":"Het Ant Ling Infra-team en het RadixArk SGLang-team verhoogden de single-request decodeersnelheid van het Ling-3.0-flash hybride lineaire aandacht MoE-model van 288 tok/s naar 606...","url":"https://www.aioga.com/nl/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:40.529Z"},"tr":{"title":"Ling-3.0-flash, 4 Blackwell GPU'da toplu 1 kod çözme gecikmesini %54 azaltıyor","summary":"Ant Ling Infra ekibi ve RadixArk SGLang ekibi, Ling-3.0-flash hibrit doğrusal dikkat MoE modelinin tek istek çözme hızını 288 tok/s'den 606 tok/s'ye çıkardı; ortalama TPOT ise 3.33 ms'den 1.53 ms'ye düştü. 🔗 Orijinal makaleyi AIHOT üzerinden okuyun · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Ling-3.0-flash, 4 Blackwell GPU'da toplu 1 kod çözme gecikmesini %54 azaltıyor - Aioga AI Haberleri","description":"Ant Ling Infra ekibi ve RadixArk SGLang ekibi, Ling-3.0-flash hibrit doğrusal dikkat MoE modelinin tek istek çözme hızını 288 tok/s'den 606 tok/s'ye çıkardı; ortalama TPOT ise 3.33...","url":"https://www.aioga.com/tr/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:49.211Z"},"vi":{"title":"Cách Ling-3.0-flash giảm độ trễ giải mã lô 1 xuống 54% trên 4 GPU Blackwell","summary":"Nhóm Ant Ling Infra và nhóm RadixArk SGLang đã tăng tốc độ giải mã một yêu cầu của mô hình MoE lai Ling-3.0-flash linear attention từ 288 tok/s lên 606 tok/s, với TPOT trung bình giảm từ 3,33 ms xuống 1,53 ms. 🔗 Đọc bài viết gốc qua AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Cách Ling-3.0-flash giảm độ trễ giải mã lô 1 xuống 54% trên 4 GPU Blackwell - Tin tức AI Aioga","description":"Nhóm Ant Ling Infra và nhóm RadixArk SGLang đã tăng tốc độ giải mã một yêu cầu của mô hình MoE lai Ling-3.0-flash linear attention từ 288 tok/s lên 606 tok/s, với TPOT trung bình g...","url":"https://www.aioga.com/vi/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:49.170Z"},"id":{"title":"Bagaimana Ling-3.0-flash mengurangi latensi decoding batch 1 sebesar 54% pada 4 GPU Blackwell","summary":"Tim Ant Ling Infra dan tim RadixArk SGLang meningkatkan kecepatan decoding single-request model MoE fokus linier hybrid Ling-3.0-flash dari 288 tok/s menjadi 606 tok/s, dengan rata-rata TPOT turun dari 3,33 ms menjadi 1,53 ms. 🔗 Baca artikel asli melalui AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Bagaimana Ling-3.0-flash mengurangi latensi decoding batch 1 sebesar 54% pada 4 GPU Blackwell - Berita AI Aioga","description":"Tim Ant Ling Infra dan tim RadixArk SGLang meningkatkan kecepatan decoding single-request model MoE fokus linier hybrid Ling-3.0-flash dari 288 tok/s menjadi 606 tok/s, dengan rata...","url":"https://www.aioga.com/id/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:57.249Z"},"th":{"title":"วิธีที่ Ling-3.0-flash ลดความหน่วงเวลาการถอดรหัสชุดที่ 1 ลง 54% บน GPU Blackwell 4 ตัว","summary":"ทีม Ant Ling Infra และทีม RadixArk SGLang เพิ่มความเร็วในการถอดรหัสคําขอเดียวของโมเดล MoE แบบไฮบริด Ling-3.0-flash linear attention จาก 288 tok/s เป็น 606 tok/s โดยค่าเฉลี่ย TPOT ลดลงจาก 3.33 ms เหลือ 1.53 ms 🔗 อ่านบทความต้นฉบับผ่าน AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"วิธีที่ Ling-3.0-flash ลดความหน่วงเวลาการถอดรหัสชุดที่ 1 ลง 54% บน GPU Blackwell 4 ตัว - ข่าว AI Aioga","description":"ทีม Ant Ling Infra และทีม RadixArk SGLang เพิ่มความเร็วในการถอดรหัสคําขอเดียวของโมเดล MoE แบบไฮบริด Ling-3.0-flash linear attention จาก 288 tok/s เป็น 606 tok/s โดยค่าเฉลี่ย TPOT ล...","url":"https://www.aioga.com/th/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:24:57.862Z"},"pl":{"title":"Jak Ling-3.0-flash zmniejsza opóźnienie dekodowania partii 1 o 54% na 4 GPU Blackwell","summary":"Zespół Ant Ling Infra oraz zespół RadixArk SGLang zwiększyli prędkość dekodowania pojedynczego żądania hybrydowego modelu Ling-3.0-flash linear attention MoE z 288 tok/s do 606 tok/s, przy czym średnia prędkość TPOT spadła z 3,33 ms do 1,53 ms. 🔗 Przeczytaj oryginalny artykuł za pośrednictwem AIHOT · https://aihot.virxact.com/items/cmt393qov0kfhro6tuwhxhubl","category":"行业动态","source":"LMSYS：Blog（Chatbot Arena 团队","aggregationSource":"LMSYS：Blog（Chatbot Arena 团队","pageTitle":"Jak Ling-3.0-flash zmniejsza opóźnienie dekodowania partii 1 o 54% na 4 GPU Blackwell - Aioga Wiadomości AI","description":"Zespół Ant Ling Infra oraz zespół RadixArk SGLang zwiększyli prędkość dekodowania pojedynczego żądania hybrydowego modelu Ling-3.0-flash linear attention MoE z 288 tok/s do 606 tok...","url":"https://www.aioga.com/pl/news/cmt393qov0kfhro6tuwhxhubl/","contentTranslated":true,"sourceHash":"0866ac8234369ce2","translatedAt":"2026-08-21T18:25:05.925Z"}},"evidenceTier":"verified-news","reviewStatus":"editorial-selected","indexable":true,"editorialCover":""}}