Miles 在 Blackwell 架构上实现端到端 MXFP8 与逐 token NVFP4 强化学习方案

TL;DR：我们在 Miles 中实现了两种 Blackwell 原生的 RL 配方：端到端 MXFP8 和针对 MoE 专家使用的每令牌 NVFP4。两者都支持在检查点转换、Megatron 训练、SGLang rollout 以及实时权重更新中的精细化精度控制。MXFP8 覆盖了 rollout、前向传播、权重-梯度 GEMM 和数据-梯度 GEMM。NVFP4 在其 MoE 专家路径中使用在线每令牌激活缩放，并且两种格式都支持高精度或去量化的反向模式。在 8x B200 上进行的 Qwen3-30B-A3B 配方消融实验中，BF16 和所有五种低精度配置的原始奖励曲线几乎完全重叠，同时 MXFP8 和 NVFP4 减少了 rollout 时间。

在低精度 RL 中，rollout、训练、检查点转换和实时权重更新必须遵循同一精度约定，否则采样器和训练器的策略将会偏离。Miles 和 SGLang RL 生态系统已经引入了低精度配方：LMSYS FP8 RL 博文（https://www.lmsys.org/blog/2025-11-25-fp8-rl）展示了在训练和采样中使用 FP8 可以减少与 BF16 训练配 FP8 rollout 的不匹配；INT4 QAT 博文（https://www.lmsys.org/blog/2026-01-26-int4-qat）表明训练期间的伪量化和 W4A16 rollout 可以使 INT4 可行。我们通过在 Miles 中实现 MXFP8 和 NVFP4 配方，并在 SGLang、TransformerEngine、FlashInfer、Megatron 以及 cuDNN 前端贡献支持组件，将这项工作扩展到 Blackwell 原生格式。公共路线图可在 miles#615（https://github.com/radixark/miles/issues/615）查看。

以往的低精度方法并非围绕 MXFP8 或 NVFP4 设计。现有的 Miles 路径遵循 DeepSeek-V3（https://arxiv.org/html/2412.19437v2）风格的块缩放 FP8 配方：权重使用 128x128 块缩放，激活使用 1x128瓦片缩放，并且缩放系数为每个瓦片或块在线计算。这是 Hopper 时代的强力配方，但在 Blackwell 上，其 FP32 缩放仍然是在 Tensor Core 路径的软件层应用，而非通过原生微缩放硬件实现。

INT4 QAT 解决了一个不同的问题。训练使用伪量化来让模型适应 INT4 权重，而推理使用 W4A16。虽然内存效率高，但计算路径实际上仍然使用 BF16 激活与去量化的 INT4 权重。下表将 NVIDIA 的 HGX 平台：https://www.nvidia.com/en-us/data-center/hgx/ 的稠密 Tensor Core 规格标准化为每 GPU 吞吐量：来自 8-GPU HGX 系统的 B200 和 B300，以及 HGX Rubin NVL8 表中的 Rubin。

对于 RL 系统，精度契约涵盖：

MXFP8 是一种微缩 FP8 格式。TransformerEngine 的 MXFP8 文档：https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/features/low_precision_training/mxfp8/mxfp8.html 将其描述为 Blackwell 原生的块状缩放方案：每 32 个连续的 E4M3 值共享一个局部的 E8M0 缩放因子，且块为一维。

因为 E8M0 缩放因子表示 2 的幂，解码后的缩放因子通常会向上取整，以避免块中的最大值被裁剪。

NVFP4 是 Blackwell 的原生 FP4 格式。如 NVIDIA 的 NVFP4 介绍中描述：https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/，它以每 16 个值一个 FP8 E4M3 缩放因子的方式存储 FP4 E2M1 值。由于 E4M3 比 UE8M0 分辨率更高，其缩放因子通常会四舍五入到可表示的最近值。标准 NVFP4 配方还会为更大张量作用域增加一个 FP32 缩放因子，形成两级层次结构：

FP32 缩放因子可以根据不同张量作用域选择。这个选择是方案决策而不是格式本身的属性，对于 RL 尤其重要。

MXFP8 配方是早期端到端 FP8 工作最直接的 Blackwell 原生扩展。推理、前向传播、权重梯度 GEMM 和数据梯度 GEMM 都使用 MXFP8，而选定张量则根据下面描述的精度控制规则保持 BF16。

TransformerEngine 和 Megatron 实现了 MXFP8 作为性能优化的、一级 Blackwell 训练路径，包括 deepseek-v3-gb200-optimization.md 中描述的 GB200 DeepSeek-V3 优化：https://github.com/NVIDIA/Megatron-LM/blob/eb0783b6d35607ef1953eaca60b37b886b1a25d0/docs/discussions/deepseek-v3-gb200-optimization/deepseek-v3-gb200-optimization.md。在我们的 Miles 集成中，我们将此路径用作端到端 MXFP8 强化学习的训练端基础。

与 DeepSeek-V3 FP8 配方：https://arxiv.org/html/2412.19437v2 的一个区别在于反向激活的表示方式。DeepSeek-V3 将前向激活存储在 1x128 FP8 块中，并在反向 GEMM 之前将其转换为反向方向。这种方法存储的 FP8 数据更少，但在反向 GEMM 之前引入了去量化加重新量化。TransformerEngine 的 MXFP8 文档指出，行向 1x32 块和列向 32x1 块在数值上不同，必须独立于全精度数据进行量化。因此，TransformerEngine 在量化过程中会生成行向和列向量化副本。虽然这使用了更多内存，但它避免了额外的重新量化步骤，并减少了反向路径中的额外量化误差。

这是 RL 的典型系统权衡。我们使用 TransformerEngine 路径以保持端到端 MXFP8 协议，而不增加其他不匹配的来源。

在 rollout 端，SGLang 使用 FlashInfer 和 Triton 提供的 Blackwell MXFP8 内核。我们实现并上游了跨 FlashInfer 和 SGLang 的 rollout 路径（flashinfer#2581：https://github.com/flashinfer-ai/flashinfer/pull/2581, sglang#17449：https://github.com/sgl-project/sglang/pull/17449, sglang#19537：https://github.com/sgl-project/sglang/pull/19537, sglang#21576：https://github.com/sgl-project/sglang/pull/21576, 以及 sglang#28459：https://github.com/sgl-project/sglang/pull/28459）。

几乎所有主要 GEMM 都可以量化为 MXFP8，包括注意力投影和 MoE 专家。主要的例外是显式控制的高精度层，例如下文描述的 BF16 MLA 投影。

NVFP4 比 MXFP8 更激进，因此我们有选择性地应用它。我们对 MoE 专家进行量化，因为它们主导了模型大小和推理时的内存流量，而模型的其余部分保持 BF16，除非明确配置为其他格式。

例如，DeepSeek-V3：https://arxiv.org/html/2412.19437v2 总参数约为 671B。其 MoE 专家占：

大约占模型的 97.8%。因此，针对 MoE 专家可以捕获大部分内存优势，而无需将每一层都强制转换为最激进的精度格式。

原始 NVFP4 预训练方案：https://arxiv.org/html/2509.25149v2 是为大规模预训练设计的，其目标是在使用 FP4 GEMM 的同时，保留对大量标记的粗略优化方向。它将 FP4 线性层 GEMM 与几种稳定器结合起来：选择的层保持在较高精度，权重缩放在前向和反向中保持一致，并且训练路径使用随机舍入 (SR) 和随机 Hadamard 变换 (RHT)。在论文中，SR 应用于梯度以减少量化偏差并产生无偏量化梯度，而 RHT 分散了大幅度块级异常值，尤其是针对权重-梯度 GEMM 输入。

这是一个很好的起点，但 RL 有不同的失败面：

NVFP4 RL 方案并未涵盖预训练方案的每一部分。我们针对 MoE 专家权重量化、每标记激活缩放、一致精度控制，以及可选择使用原始或去量化操作数的 BF16 反向 GEMM。

两级 NVFP4 层次结构非常强大，但 FP32 激活比例的范围必须谨慎选择。如 Cursor Composer 2 技术报告所讨论：https://arxiv.org/html/2603.24477v2，每张张量的 NVFP4 缩放可能导致训练批次变化，并且跨标记的比例共享可能将未来标记信息泄漏到过去标记表示中。如果一个标记与其他标记共享其比例，其量化表示将取决于批次组成。这对于 RL 尤为成问题，因为推理调度和序列长度是可变的。

因此，我们的配方计算每个代币在线计算一个FP32激活量表。这使激活异常值局限于一个令牌，去除静态激活校准伪影，使SGLang的部署和威震天训练使用相同的激活尺度示波器。

在推广端，每个令牌的FP32规模计算融合到了FlashInfer的激活量化内核路径中：同样的调用，同时也返回每个令牌的FP32尺度。因此，每个令牌激活的缩放不需要单独的校准尺度处理。

列子推断一致性还需要匹配并行性。如果在专家-张量-并行划分中计算每个代币的FP32尺度，SGLang和Megatron应使用相同的ETP大小。否则，双方可能看到不同的张量划分，并计算出不同的尺度。

SwiGLU MoE 层增加了另一个关键合同。SGLang和Megatron通常将门和上投影融合成一个GEMM，因此即使Hugging Face检查点分别存储，转换和实时权重更新时，两个张量也必须共享相同的FP32刻度。Miles 通过在 NVFP4 导出路径中将门/上行对量化来强制执行这一机制。

我们实现并上游了每个代币的NVFP4配方，覆盖整个栈：

在高精度后向的NVFP4变体中，前向通行和滚动时为MoE专家使用NVFP4，而后向GEMM则使用原始BF16操作数。

反向去量子化是第二个可选模式。后向GEMM仍运行于BF16，但使用BF16解量化，正向产生的精确低精度操作数，而非原始BF16值。

这两种模式都避免了低精度的向后GEMM，因此这些配置不使用原始NVFP4预训练配方中的RHT或随机四舍五入。它们以后向吞吐量换取更高精度的计算，但强化学习通常是向外部署的，长上下文的关注和通信进一步减少了端到端的影响。

相同的向后模式选择同样适用于MXFP8：

我们实现并上游了 NVTE_BACKWARD_OVERRIDE，作为一个可重用的 TransformerEngine 接口，用于选择高精度或反量化的反向操作数（TransformerEngine#2644：https://github.com/NVIDIA/TransformerEngine/pull/2644），然后通过 Miles 配方配置暴露了这两种模式。相关的文章 humans&：https://humansand.ai/blog/nvfp4-rl 介绍了算法动机和额外的反向模式消融实验。

反量化反向增加了训练端的反量化步骤。我们与 NVIDIA 合作，在 TransformerEngine#2865 中降低了其开销：https://github.com/NVIDIA/TransformerEngine/pull/2865。

高精度和反量化反向还可以相对于 TransformerEngine 默认的低精度反向路径减少峰值内存。这两种模式都不需要生成和保留上面 MXFP8 训练部分描述的低精度反向 GEMM 所使用的第二列量化副本。

在验证我们在 TransformerEngine#2644 中的 TransformerEngine 反向模式实现时，我们测量了以下内存数据：https://github.com/NVIDIA/TransformerEngine/pull/2644#issuecomment-4026583299。alloc 列报告分配内存，resrv 列报告保留内存，所有值单位为 MB。

MXFP8 线性内存，dtype=torch.bfloat16，input_shape=(2048, 2048)，out_features=8192：

NVFP4 线性内存，dtype=torch.bfloat16，input_shape=(2048, 2048)，out_features=8192：

在 RL 中，量化不匹配可能在权重更新中积累。如果训练端和运行端以不同方式量化张量，用于采样和学习的策略不再是相同的低精度模型。因此，我们需要在 RL 堆栈的双方之间建立明确的契约。

我们将 FlashInfer 和 TransformerEngine 的量化器调整为相同的 MXFP8 和 NVFP4 位级契约。我们的 FlashInfer 单元测试检查随机数据、量化边界数据、全零张量和最大值张量（对应 init_data = ["random", "boundary", "zeros", "maxes"]）中，与 TransformerEngine 风格参考的精确字节级一致性。我们在 flashinfer#3387 中实现并上游了这个量化器对齐：https://github.com/flashinfer-ai/flashinfer/pull/3387。

在服务推理和强化学习（RL）之间存在一个实际区别。对于仅服务的工作负载，FlashInfer 可能在 FP4 量化路径的部分环节使用快速数学计算以提升性能。这是一个合理的服务默认设置，但 RL 权重更新受益于与训练端量化器的精确一致性。对于这个方法，我们设置：

所有涉及 rollout 权重的后端应严格遵守此量化约定，或将近似行为设为可选。

在实际操作中，单一全局精度开关对低精度 RL 来说是不够的。一些张量应保持 BF16，但只选择它们只是问题的一部分：相同的决策必须在 Hugging Face 检查点转换、Megatron 训练、SGLang rollout 以及实时权重导出中得到执行。

我们通过基于计数和基于名称的 BF16 异常，在 Miles 中实现了这种张量级精度控制，涵盖检查点转换、训练、rollout 和实时导出（miles#614：https://github.com/radixark/miles/pull/614, miles#1054：https://github.com/radixark/miles/pull/1054, 以及 miles#1261：https://github.com/radixark/miles/pull/1261）。我们还为生成的混合精度检查点实现了 SGLang 支持（sglang#18742：https://github.com/sgl-project/sglang/pull/18742 和 sglang#20214：https://github.com/sgl-project/sglang/pull/20214）。具体而言，转换使用 --num-layers-at-start-in-bf16 和 --num-layers-at-end-in-bf16；Megatron 训练将这些数量与 --first-last-layers-bf16 结合使用；SGLang 提供生成的混合精度检查点服务。

如 NVIDIA NVFP4 预训练论文推荐（https://arxiv.org/html/2509.25149v2），我们保持少数最终层的更高精度。在我们的实验中，将最后 15% 的层保持为 BF16 能明显减少训练与推理的不匹配并稳定梯度。

在我们的实验中，为早期层启用 BF16 并未显著降低训练与推理的不匹配：

保持共享专家为高精度也可以减少训练与推理的不匹配，同时性能和内存开销几乎不增加。路由专家被稀疏选择，其输出通过高精度加权归约。共享专家始终处于激活状态，因此其精度误差会影响通过该模块的每个 token。

对于MLA模型，kv_b_proj是一个重要的MXFP8案例。吸收和非吸收的MLA模式可以使用不同的收缩轴，而MXFP8使用一维微缩块。因此，更改收缩轴可能会改变哪些元素共享一个刻度。同样的关注点也适用于其他一维格式，包括NVFP4。原始的DeepSeek-V3：https://arxiv.org/html/2412.19437v2 FP8方案没有这个确切的权重侧问题，因为它使用的是128x128的权重刻度块，而不是一维块。我们将这些投影张量保存在BF16中，以避免隐藏的重新量化并保持训练-推理一致性。

匹配的基于名称的转换时间和推演时间参数为：

对于DeepSeek-V3：https://arxiv.org/html/2412.19437v2风格的MLA，这个BF16例外很小。一个形状为32768 x 512的kv_b_proj张量，在61层中以BF16存储，占用约1.90625 GB。

为了进行一致的比较，所有实验都使用同步Qwen3-30B-A3B RL，采用8x B200上的默认Miles设置。固定工作负载使用GRPO风格的训练，在dapo-math-17k上，每个提示有8个推演样本，最大响应长度为8192个标记。这只是一个方案消融设置，而不是完全调优的训练或服务基准。KL路径用于诊断，但其系数为0.0，因此在此消融中KL不是优化惩罚。

硬件分配为4个GPU用于推演，4个GPU用于训练。

如预期的那样，两种低精度格式显示比BF16更高的训练-推理不匹配，而每种格式内的两个反向选择表现相似。对于此次消融，数值仍保持在合理范围内。

NVFP4开始时的参考KL高于BF16或MXFP8。Miles将该诊断与Megatron BF16参考模型进行计算，因此该指标包括每个低精度策略与BF16参考之间的差异，以及RL期间的策略演变。它不应被视为独立的优化惩罚。

尽管诊断不匹配更高，所有五条低精度奖励曲线仍与BF16奖励曲线紧密跟踪。

这是配方消融的关键结果：在这个 Qwen3-30B-A3B B200 设置中，Blackwell 原生的低精度保持了观察到的学习曲线，同时提高了 rollout 的效率。

与 BF16 相比，MXFP8 和 NVFP4 都减少了 rollout 时间：

对于 NVFP4 rollout，FlashInfer 直接在激活量化内核路径中计算每个 token 的在线 FP32 缩放，而不是作为单独的预处理步骤。因此报告的 rollout 性能包括在线缩放计算的开销。

在训练方面，MXFP8 变体比 BF16 更快，而在此实现中测量的 NVFP4 反向覆盖变体则更慢：

训练端的差距来自于此消融中使用的实现，而不是 FP4 Tensor Core 的固有限制。我们的 TransformerEngine 路径将每个 token 的 FP32 缩放作为单独的 PyTorch 操作（TransformerEngine#2931：https://github.com/NVIDIA/TransformerEngine/pull/2931）应用，而不是将缩放融合到内核尾部的原生每个 token 的 NVFP4 GEMM 路径。我们已经实现并上游了融合的 cuDNN 前端内核（cudnn-frontend#251：https://github.com/NVIDIA/cudnn-frontend/pull/251）；TransformerEngine 的集成仍在等待中。去量化的反向过程添加了上述去量化步骤。因为这个 RL 工作负载主要以 rollout 为主，即使训练路径尚未完全加速，rollout 加速仍然具有实际意义。

在此消融之外，humans&：https://humansand.ai/ 在生产中使用相同的配方家族和组件，用于大规模、长上下文、多智能体异步 RL 研究。

TL;DR: We implemented two Blackwell-native RL recipes in Miles: end-to-end MXFP8 and per-token NVFP4 for MoE experts. Both are supported by fine-grained precision control across checkpoint conversion, Megatron training, SGLang rollout, and live weight updates. MXFP8 covers rollout, forward propagation, weight-gradient GEMMs, and data-gradient GEMMs. NVFP4 uses online per-token activation scaling for its MoE expert path, and both formats support high-precision or dequantized backward modes. In a Qwen3-30B-A3B recipe ablation on 8x B200, BF16 and all five low-precision configurations have closely overlapping raw reward curves, while MXFP8 and NVFP4 reduce rollout time.

In low-precision RL, rollout, training, checkpoint conversion, and live weight updates must agree on one precision contract, or the sampler and trainer policies will diverge. Miles and the SGLang RL ecosystem have already incorporated low-precision recipes: the LMSYS FP8 RL post：https://www.lmsys.org/blog/2025-11-25-fp8-rl showed that using FP8 across training and sampling reduces mismatch relative to BF16 training with FP8 rollout; the INT4 QAT post：https://www.lmsys.org/blog/2026-01-26-int4-qat showed that fake quantization during training and W4A16 rollout can make INT4 practical. We extend that work to Blackwell-native formats by implementing MXFP8 and NVFP4 recipes in Miles and contributing the supporting components across SGLang, TransformerEngine, FlashInfer, Megatron, and cuDNN frontend. The public roadmap is tracked in miles#615：https://github.com/radixark/miles/issues/615.

Previous low-precision approaches were not designed around MXFP8 or NVFP4. The existing Miles path follows a DeepSeek-V3：https://arxiv.org/html/2412.19437v2-style block-scaled FP8 recipe: weights use 128x128 block scaling, activations use 1x128 tile scaling, and scales are computed online for each tile or block. This is a strong Hopper-era recipe, but on Blackwell its FP32 scales are still applied in software around the Tensor Core path rather than through native microscaling hardware.

INT4 QAT solves a different problem. Training uses fake quantization to adapt the model to INT4 weights, while rollout uses W4A16. Although memory efficient, the compute path still effectively uses BF16 activations with dequantized INT4 weights. The table below normalizes NVIDIA's HGX platform：https://www.nvidia.com/en-us/data-center/hgx/ dense Tensor Core specs to per-GPU throughput: B200 and B300 from 8-GPU HGX systems, Rubin from the HGX Rubin NVL8 table.

For RL systems, the precision contract spans:

MXFP8 is a microscaling FP8 format. TransformerEngine's MXFP8 documentation：https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/features/low_precision_training/mxfp8/mxfp8.html describes it as a Blackwell-native blockwise scaling recipe: every 32 consecutive E4M3 values share one local E8M0 scale, and the block is one-dimensional.

Because E8M0 scales represent powers of two, the decoded scale is usually rounded up so the maximum value in the block is not clipped.

NVFP4 is Blackwell's native FP4 format. As described in NVIDIA's NVFP4 introduction：https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/, it stores FP4 E2M1 values with one FP8 E4M3 scale per 16-value block. Because E4M3 has finer resolution than UE8M0, its scale is typically rounded to the nearest representable value. A standard NVFP4 recipe also adds one FP32 scale for the larger tensor scope, creating a two-level hierarchy:

The FP32 scale can be chosen over different tensor scopes. That choice is a recipe decision rather than a property of the format itself, and it becomes especially important for RL.

The MXFP8 recipe is the most direct Blackwell-native extension of the earlier end-to-end FP8 work. Rollout, forward propagation, weight-gradient GEMMs, and data-gradient GEMMs all use MXFP8, while selected tensors remain BF16 through the precision-control rules described below.

TransformerEngine and Megatron implement MXFP8 as a performance-optimized, first-class Blackwell training path, including the GB200 DeepSeek-V3 optimizations described in deepseek-v3-gb200-optimization.md：https://github.com/NVIDIA/Megatron-LM/blob/eb0783b6d35607ef1953eaca60b37b886b1a25d0/docs/discussions/deepseek-v3-gb200-optimization/deepseek-v3-gb200-optimization.md. In our Miles integration, we use this path as the training-side foundation for end-to-end MXFP8 RL.

One difference from the DeepSeek-V3 FP8 recipe：https://arxiv.org/html/2412.19437v2 is how backward activations are represented. DeepSeek-V3 stores forward activations in 1x128 FP8 tiles and converts them to the backward orientation before the backward GEMM. That approach stores less FP8 data, but it introduces dequantization plus requantization before the backward GEMM. TransformerEngine's MXFP8 docs note that rowwise 1x32 blocks and columnwise 32x1 blocks are numerically different and must be quantized independently from full-precision data. TransformerEngine therefore materializes both row-wise and column-wise quantized copies during quantization. This uses more memory, but it avoids an extra requantization step and reduces additional quantization error in the backward path.

This is a typical systems trade-off for RL. We use the TransformerEngine path to preserve one end-to-end MXFP8 contract without adding another source of mismatch.

On the rollout side, SGLang uses Blackwell MXFP8 kernels from FlashInfer and Triton. We implemented and upstreamed the rollout path across FlashInfer and SGLang (flashinfer#2581：https://github.com/flashinfer-ai/flashinfer/pull/2581, sglang#17449：https://github.com/sgl-project/sglang/pull/17449, sglang#19537：https://github.com/sgl-project/sglang/pull/19537, sglang#21576：https://github.com/sgl-project/sglang/pull/21576, and sglang#28459：https://github.com/sgl-project/sglang/pull/28459).

Almost all major GEMMs can be quantized to MXFP8, including attention projections and MoE experts. The main exceptions are explicitly controlled high-precision layers, such as the BF16 MLA projections described below.

NVFP4 is more aggressive than MXFP8, so we apply it selectively. We quantize MoE experts because they dominate model size and rollout memory traffic, while the rest of the model remains BF16 unless explicitly configured otherwise.

For example, DeepSeek-V3：https://arxiv.org/html/2412.19437v2 has about 671B total parameters. Its MoE experts account for:

That is about 97.8% of the model. Targeting MoE experts therefore captures most of the memory benefit without forcing every layer into the most aggressive precision format.

The original NVFP4 pretraining recipe：https://arxiv.org/html/2509.25149v2 is designed for large-scale pretraining, where the goal is preserving a coarse optimization direction over many tokens while still using FP4 GEMMs. It combines FP4 linear-layer GEMMs with several stabilizers: selected layers remain in higher precision, weight scaling is consistent across forward and backward, and the training path uses stochastic rounding (SR) and Random Hadamard Transforms (RHT). In the paper, SR is applied to gradients to reduce quantization bias and produce unbiased quantized gradients, while RHT disperses large-magnitude block-level outliers, especially for weight-gradient GEMM inputs.

That is a good starting point, but RL has a different failure surface:

The NVFP4 RL recipe does not incorporate every part of the pretraining recipe. We target MoE expert weight quantization, per-token activation scaling, consistent precision control, and BF16 backward GEMMs with selectable original or dequantized operands.

The two-level NVFP4 hierarchy is powerful, but the scope of the FP32 activation scale must be chosen carefully. As discussed in the Cursor Composer 2 technical report：https://arxiv.org/html/2603.24477v2, per-tensor NVFP4 scaling can make training batch-variant, and inter-token scale sharing can leak future-token information into past-token representations. If a token shares its scale with other tokens, its quantized representation depends on the batch composition. This is especially problematic for RL, where rollout scheduling and sequence lengths vary.

Our recipe therefore computes one FP32 activation scale per token online. This localizes activation outliers to one token, removes the static activation calibration artifact, and lets SGLang rollout and Megatron training use the same activation-scale scope.

On the rollout side, the per-token FP32 scale computation is fused into FlashInfer's activation quantization kernel path: the same call that emits packed FP4 activations and E4M3 block scales also returns the per-token FP32 scales. As a result, per-token activation scaling does not require a separate calibration-scale pass.

Train-inference consistency also requires matching parallelism. If the FP32 scale is computed per token within an expert-tensor-parallel partition, SGLang and Megatron should use the same ETP size. Otherwise, each side may see a different partition of the tensor and compute a different scale.

SwiGLU MoE layers add another key contract. SGLang and Megatron commonly fuse the gate and up projections into one GEMM, so both tensors must share the same FP32 scale during conversion and live weight update even when the Hugging Face checkpoint stores them separately. Miles enforces this by quantizing gate/up pairs together in the NVFP4 export path.

We implemented and upstreamed the per-token NVFP4 recipe across the entire stack:

In the high-precision-backward NVFP4 variant, the forward pass and rollout use NVFP4 for MoE experts, while the backward GEMMs use the original BF16 operands.

Dequantized backward is a second selectable mode. The backward GEMMs still run in BF16, but consume BF16 dequantizations of the exact low-precision operands produced in forward instead of the original BF16 values.

Both modes avoid low-precision backward GEMMs, so these configurations do not use RHT or stochastic rounding from the original NVFP4 pretraining recipe. They trade backward throughput for higher-precision computation, but RL is often rollout-bound, and long-context attention and communication further reduce the end-to-end impact.

The same backward-mode selection also applies to MXFP8:

We implemented and upstreamed NVTE_BACKWARD_OVERRIDE as a reusable TransformerEngine interface for selecting high-precision or dequantized backward operands (TransformerEngine#2644：https://github.com/NVIDIA/TransformerEngine/pull/2644), then exposed both modes through the Miles recipe configuration. The companion humans&：https://humansand.ai/blog/nvfp4-rl post covers the algorithmic motivation and additional backward-mode ablations.

Dequantized backward adds a training-side dequantization step. We worked with NVIDIA to reduce its overhead in TransformerEngine#2865：https://github.com/NVIDIA/TransformerEngine/pull/2865.

High-precision and dequantized backward can also reduce peak memory relative to TransformerEngine's default low-precision backward paths. Neither mode needs to generate and retain the second column-wise quantized copy used by the low-precision backward GEMMs described in the MXFP8 training section above.

We measured the memory data below while validating our TransformerEngine backward-mode implementation in TransformerEngine#2644：https://github.com/NVIDIA/TransformerEngine/pull/2644#issuecomment-4026583299. The alloc columns report allocated memory, the resrv columns report reserved memory, and all values are in MB.

MXFP8 linear memory, dtype=torch.bfloat16 , input_shape=(2048, 2048) , out_features=8192 :

NVFP4 linear memory, dtype=torch.bfloat16 , input_shape=(2048, 2048) , out_features=8192 :

In RL, quantization mismatch can accumulate across weight updates. If the training and rollout sides quantize a tensor differently, the policies used for sampling and learning are no longer the same low-precision model. We therefore need an explicit contract between both sides of the RL stack.

We align the FlashInfer and TransformerEngine quantizers to the same MXFP8 and NVFP4 bit-level contract. Our FlashInfer unit tests check exact byte-level agreement against a TransformerEngine-style reference across random data, quantization-boundary data, all-zero tensors, and maximum-value tensors, corresponding to init_data = ["random", "boundary", "zeros", "maxes"] . We implemented and upstreamed this quantizer alignment in flashinfer#3387：https://github.com/flashinfer-ai/flashinfer/pull/3387.

There is one practical distinction between serving and RL. For serving-only workloads, FlashInfer may use fast math in parts of the FP4 quantization path for performance. This is a reasonable serving default, but RL weight updates benefit from exact agreement with the training-side quantizer. For this recipe, we set:

Every backend that touches rollout weights should either implement this quantization contract exactly or make approximate behavior opt-in.

In practice, a single global precision switch is insufficient for low-precision RL. Some tensors should remain in BF16, but selecting them is only part of the problem: the same decision must be enforced across Hugging Face checkpoint conversion, Megatron training, SGLang rollout, and live weight export.

We implemented this tensor-level precision control in Miles through count-based and name-based BF16 exceptions across checkpoint conversion, training, rollout, and live export (miles#614：https://github.com/radixark/miles/pull/614, miles#1054：https://github.com/radixark/miles/pull/1054, and miles#1261：https://github.com/radixark/miles/pull/1261). We also implemented SGLang support for the resulting mixed-precision checkpoints (sglang#18742：https://github.com/sgl-project/sglang/pull/18742 and sglang#20214：https://github.com/sgl-project/sglang/pull/20214). Concretely, conversion uses --num-layers-at-start-in-bf16 and --num-layers-at-end-in-bf16 ; Megatron training combines those counts with --first-last-layers-bf16 ; and SGLang serves the resulting mixed-precision checkpoints.

As recommended in the NVIDIA NVFP4 pretraining paper：https://arxiv.org/html/2509.25149v2, we keep a small fraction of final layers in higher precision. In our experiments, keeping the last 15% of layers in BF16 meaningfully reduces train-inference mismatch and stabilizes gradients.

Turning on BF16 for early layers does not lead to meaningful train-inference mismatch reduction in our experiments:

Keeping shared experts in high precision also reduces train-inference mismatch with little performance or memory overhead. Routed experts are selected sparsely, and their outputs pass through a high-precision weighted reduction. Shared experts are always active, so their precision errors affect every token passing through the block.

For MLA models, kv_b_proj is an important MXFP8 case. Absorbed and non-absorbed MLA modes can use different contraction axes, while MXFP8 uses one-dimensional microscaling blocks. Changing the contraction axis can therefore change which elements share a scale. The same concern applies to other one-dimensional formats, including NVFP4. The original DeepSeek-V3：https://arxiv.org/html/2412.19437v2 FP8 recipe does not have this exact weight-side issue because it uses 128x128 weight-scale blocks rather than one-dimensional blocks. We keep these projection tensors in BF16 to avoid hidden requantization and preserve train-inference consistency.

The matching name-based conversion-time and rollout-time arguments are:

For DeepSeek-V3：https://arxiv.org/html/2412.19437v2-style MLA, this BF16 exception is small. A kv_b_proj tensor of shape 32768 x 512 , stored in BF16 across 61 layers, occupies about 1.90625 GB.

For consistent comparison, all experiments use synchronous Qwen3-30B-A3B RL with the default Miles setup on 8x B200. The fixed workload uses GRPO-style training on dapo-math-17k , with 8 rollout samples per prompt and a maximum response length of 8192 tokens. This is only a recipe ablation setup, not a fully tuned training or serving benchmark. The KL path is enabled for diagnostics, but its coefficient is 0.0, so KL is not an optimization penalty in this ablation.

The hardware split is 4 GPUs for rollout and 4 GPUs for training.

As expected, both low-precision formats show higher train-inference mismatch than BF16, while the two backward choices behave similarly within each format. The values remain in a reasonable range for this ablation.

NVFP4 begins with a higher reference KL than BF16 or MXFP8. Miles computes this diagnostic against a Megatron BF16 reference model, so the metric includes the difference between each low-precision policy and the BF16 reference in addition to policy evolution during RL. It should not be read as a standalone optimization penalty.

Despite the higher diagnostic mismatch, all five low-precision reward curves closely track the BF16 reward curve.

This is the key result of the recipe ablation: in this Qwen3-30B-A3B B200 setup, Blackwell-native low precision preserves the observed learning curve while improving rollout efficiency.

MXFP8 and NVFP4 both reduce rollout time compared with BF16:

For NVFP4 rollout, FlashInfer computes the online per-token FP32 scale directly inside the activation quantization kernel path rather than as a separate preprocessing step. The reported rollout performance therefore includes the cost of online scale computation.

On the training side, the MXFP8 variants are faster than BF16, while the NVFP4 backward-override variants are slower in the implementation measured here:

The training-side gap comes from the implementation used in this ablation, not an inherent FP4 Tensor Core limit. Our TransformerEngine path applies per-token FP32 scaling as a separate PyTorch operation (TransformerEngine#2931：https://github.com/NVIDIA/TransformerEngine/pull/2931) instead of a native per-token NVFP4 GEMM path with scaling fused into the kernel epilogue. We have implemented and upstreamed the fused cuDNN frontend kernels (cudnn-frontend#251：https://github.com/NVIDIA/cudnn-frontend/pull/251); TransformerEngine integration remains pending. Dequantized backward adds the dequantization step described above. Because this RL workload is rollout-heavy, the rollout speedup remains meaningful even before the training path is fully accelerated.

Beyond this ablation, humans&：https://humansand.ai/ uses the same recipe family and components in production for large-scale, long-context, multi-agent asynchronous RL research.