TL;DR: We implemented two Blackwell-native RL recipes in Miles: end-to-end MXFP8 and per-token NVFP4 for MoE experts. Both are supported by fine-grained precision control across checkpoint conversion, Megatron training, SGLang rollout, and live weight updates. MXFP8 covers rollout, forward propagation, weight-gradient GEMMs, and data-gradient GEMMs. NVFP4 uses online per-token activation scaling for its MoE expert path, and both formats support high-precision or dequantized backward modes. In a Qwen3-30B-A3B recipe ablation on 8x B200, BF16 and all five low-precision configurations have closely overlapping raw reward curves, while MXFP8 and NVFP4 reduce rollout time.
In low-precision RL, rollout, training, checkpoint conversion, and live weight updates must agree on one precision contract, or the sampler and trainer policies will diverge. Miles and the SGLang RL ecosystem have already incorporated low-precision recipes: the LMSYS FP8 RL post：https://www.lmsys.org/blog/2025-11-25-fp8-rl showed that using FP8 across training and sampling reduces mismatch relative to BF16 training with FP8 rollout; the INT4 QAT post：https://www.lmsys.org/blog/2026-01-26-int4-qat showed that fake quantization during training and W4A16 rollout can make INT4 practical. We extend that work to Blackwell-native formats by implementing MXFP8 and NVFP4 recipes in Miles and contributing the supporting components across SGLang, TransformerEngine, FlashInfer, Megatron, and cuDNN frontend. The public roadmap is tracked in miles#615：https://github.com/radixark/miles/issues/615.
Previous low-precision approaches were not designed around MXFP8 or NVFP4. The existing Miles path follows a DeepSeek-V3：https://arxiv.org/html/2412.19437v2-style block-scaled FP8 recipe: weights use 128x128 block scaling, activations use 1x128 tile scaling, and scales are computed online for each tile or block. This is a strong Hopper-era recipe, but on Blackwell its FP32 scales are still applied in software around the Tensor Core path rather than through native microscaling hardware.
INT4 QAT solves a different problem. Training uses fake quantization to adapt the model to INT4 weights, while rollout uses W4A16. Although memory efficient, the compute path still effectively uses BF16 activations with dequantized INT4 weights. The table below normalizes NVIDIA's HGX platform：https://www.nvidia.com/en-us/data-center/hgx/ dense Tensor Core specs to per-GPU throughput: B200 and B300 from 8-GPU HGX systems, Rubin from the HGX Rubin NVL8 table.
For RL systems, the precision contract spans:
MXFP8 is a microscaling FP8 format. TransformerEngine's MXFP8 documentation：https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/features/low_precision_training/mxfp8/mxfp8.html describes it as a Blackwell-native blockwise scaling recipe: every 32 consecutive E4M3 values share one local E8M0 scale, and the block is one-dimensional.
Because E8M0 scales represent powers of two, the decoded scale is usually rounded up so the maximum value in the block is not clipped.
NVFP4 is Blackwell's native FP4 format. As described in NVIDIA's NVFP4 introduction：https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/, it stores FP4 E2M1 values with one FP8 E4M3 scale per 16-value block. Because E4M3 has finer resolution than UE8M0, its scale is typically rounded to the nearest representable value. A standard NVFP4 recipe also adds one FP32 scale for the larger tensor scope, creating a two-level hierarchy:
The FP32 scale can be chosen over different tensor scopes. That choice is a recipe decision rather than a property of the format itself, and it becomes especially important for RL.
The MXFP8 recipe is the most direct Blackwell-native extension of the earlier end-to-end FP8 work. Rollout, forward propagation, weight-gradient GEMMs, and data-gradient GEMMs all use MXFP8, while selected tensors remain BF16 through the precision-control rules described below.
TransformerEngine and Megatron implement MXFP8 as a performance-optimized, first-class Blackwell training path, including the GB200 DeepSeek-V3 optimizations described in deepseek-v3-gb200-optimization.md：https://github.com/NVIDIA/Megatron-LM/blob/eb0783b6d35607ef1953eaca60b37b886b1a25d0/docs/discussions/deepseek-v3-gb200-optimization/deepseek-v3-gb200-optimization.md. In our Miles integration, we use this path as the training-side foundation for end-to-end MXFP8 RL.
One difference from the DeepSeek-V3 FP8 recipe：https://arxiv.org/html/2412.19437v2 is how backward activations are represented. DeepSeek-V3 stores forward activations in 1x128 FP8 tiles and converts them to the backward orientation before the backward GEMM. That approach stores less FP8 data, but it introduces dequantization plus requantization before the backward GEMM. TransformerEngine's MXFP8 docs note that rowwise 1x32 blocks and columnwise 32x1 blocks are numerically different and must be quantized independently from full-precision data. TransformerEngine therefore materializes both row-wise and column-wise quantized copies during quantization. This uses more memory, but it avoids an extra requantization step and reduces additional quantization error in the backward path.
This is a typical systems trade-off for RL. We use the TransformerEngine path to preserve one end-to-end MXFP8 contract without adding another source of mismatch.
On the rollout side, SGLang uses Blackwell MXFP8 kernels from FlashInfer and Triton. We implemented and upstreamed the rollout path across FlashInfer and SGLang (flashinfer#2581：https://github.com/flashinfer-ai/flashinfer/pull/2581, sglang#17449：https://github.com/sgl-project/sglang/pull/17449, sglang#19537：https://github.com/sgl-project/sglang/pull/19537, sglang#21576：https://github.com/sgl-project/sglang/pull/21576, and sglang#28459：https://github.com/sgl-project/sglang/pull/28459).
Almost all major GEMMs can be quantized to MXFP8, including attention projections and MoE experts. The main exceptions are explicitly controlled high-precision layers, such as the BF16 MLA projections described below.
NVFP4 is more aggressive than MXFP8, so we apply it selectively. We quantize MoE experts because they dominate model size and rollout memory traffic, while the rest of the model remains BF16 unless explicitly configured otherwise.
For example, DeepSeek-V3：https://arxiv.org/html/2412.19437v2 has about 671B total parameters. Its MoE experts account for:
That is about 97.8% of the model. Targeting MoE experts therefore captures most of the memory benefit without forcing every layer into the most aggressive precision format.
The original NVFP4 pretraining recipe：https://arxiv.org/html/2509.25149v2 is designed for large-scale pretraining, where the goal is preserving a coarse optimization direction over many tokens while still using FP4 GEMMs. It combines FP4 linear-layer GEMMs with several stabilizers: selected layers remain in higher precision, weight scaling is consistent across forward and backward, and the training path uses stochastic rounding (SR) and Random Hadamard Transforms (RHT). In the paper, SR is applied to gradients to reduce quantization bias and produce unbiased quantized gradients, while RHT disperses large-magnitude block-level outliers, especially for weight-gradient GEMM inputs.
That is a good starting point, but RL has a different failure surface:
The NVFP4 RL recipe does not incorporate every part of the pretraining recipe. We target MoE expert weight quantization, per-token activation scaling, consistent precision control, and BF16 backward GEMMs with selectable original or dequantized operands.
The two-level NVFP4 hierarchy is powerful, but the scope of the FP32 activation scale must be chosen carefully. As discussed in the Cursor Composer 2 technical report：https://arxiv.org/html/2603.24477v2, per-tensor NVFP4 scaling can make training batch-variant, and inter-token scale sharing can leak future-token information into past-token representations. If a token shares its scale with other tokens, its quantized representation depends on the batch composition. This is especially problematic for RL, where rollout scheduling and sequence lengths vary.
Our recipe therefore computes one FP32 activation scale per token online. This localizes activation outliers to one token, removes the static activation calibration artifact, and lets SGLang rollout and Megatron training use the same activation-scale scope.
On the rollout side, the per-token FP32 scale computation is fused into FlashInfer's activation quantization kernel path: the same call that emits packed FP4 activations and E4M3 block scales also returns the per-token FP32 scales. As a result, per-token activation scaling does not require a separate calibration-scale pass.
Train-inference consistency also requires matching parallelism. If the FP32 scale is computed per token within an expert-tensor-parallel partition, SGLang and Megatron should use the same ETP size. Otherwise, each side may see a different partition of the tensor and compute a different scale.
SwiGLU MoE layers add another key contract. SGLang and Megatron commonly fuse the gate and up projections into one GEMM, so both tensors must share the same FP32 scale during conversion and live weight update even when the Hugging Face checkpoint stores them separately. Miles enforces this by quantizing gate/up pairs together in the NVFP4 export path.
We implemented and upstreamed the per-token NVFP4 recipe across the entire stack:
In the high-precision-backward NVFP4 variant, the forward pass and rollout use NVFP4 for MoE experts, while the backward GEMMs use the original BF16 operands.
Dequantized backward is a second selectable mode. The backward GEMMs still run in BF16, but consume BF16 dequantizations of the exact low-precision operands produced in forward instead of the original BF16 values.
Both modes avoid low-precision backward GEMMs, so these configurations do not use RHT or stochastic rounding from the original NVFP4 pretraining recipe. They trade backward throughput for higher-precision computation, but RL is often rollout-bound, and long-context attention and communication further reduce the end-to-end impact.
The same backward-mode selection also applies to MXFP8:
We implemented and upstreamed NVTE_BACKWARD_OVERRIDE as a reusable TransformerEngine interface for selecting high-precision or dequantized backward operands (TransformerEngine#2644：https://github.com/NVIDIA/TransformerEngine/pull/2644), then exposed both modes through the Miles recipe configuration. The companion humans&：https://humansand.ai/blog/nvfp4-rl post covers the algorithmic motivation and additional backward-mode ablations.
Dequantized backward adds a training-side dequantization step. We worked with NVIDIA to reduce its overhead in TransformerEngine#2865：https://github.com/NVIDIA/TransformerEngine/pull/2865.
High-precision and dequantized backward can also reduce peak memory relative to TransformerEngine's default low-precision backward paths. Neither mode needs to generate and retain the second column-wise quantized copy used by the low-precision backward GEMMs described in the MXFP8 training section above.
We measured the memory data below while validating our TransformerEngine backward-mode implementation in TransformerEngine#2644：https://github.com/NVIDIA/TransformerEngine/pull/2644#issuecomment-4026583299. The alloc columns report allocated memory, the resrv columns report reserved memory, and all values are in MB.
MXFP8 linear memory, dtype=torch.bfloat16 , input_shape=(2048, 2048) , out_features=8192 :
NVFP4 linear memory, dtype=torch.bfloat16 , input_shape=(2048, 2048) , out_features=8192 :
In RL, quantization mismatch can accumulate across weight updates. If the training and rollout sides quantize a tensor differently, the policies used for sampling and learning are no longer the same low-precision model. We therefore need an explicit contract between both sides of the RL stack.
We align the FlashInfer and TransformerEngine quantizers to the same MXFP8 and NVFP4 bit-level contract. Our FlashInfer unit tests check exact byte-level agreement against a TransformerEngine-style reference across random data, quantization-boundary data, all-zero tensors, and maximum-value tensors, corresponding to init_data = ["random", "boundary", "zeros", "maxes"] . We implemented and upstreamed this quantizer alignment in flashinfer#3387：https://github.com/flashinfer-ai/flashinfer/pull/3387.
There is one practical distinction between serving and RL. For serving-only workloads, FlashInfer may use fast math in parts of the FP4 quantization path for performance. This is a reasonable serving default, but RL weight updates benefit from exact agreement with the training-side quantizer. For this recipe, we set:
Every backend that touches rollout weights should either implement this quantization contract exactly or make approximate behavior opt-in.
In practice, a single global precision switch is insufficient for low-precision RL. Some tensors should remain in BF16, but selecting them is only part of the problem: the same decision must be enforced across Hugging Face checkpoint conversion, Megatron training, SGLang rollout, and live weight export.
We implemented this tensor-level precision control in Miles through count-based and name-based BF16 exceptions across checkpoint conversion, training, rollout, and live export (miles#614：https://github.com/radixark/miles/pull/614, miles#1054：https://github.com/radixark/miles/pull/1054, and miles#1261：https://github.com/radixark/miles/pull/1261). We also implemented SGLang support for the resulting mixed-precision checkpoints (sglang#18742：https://github.com/sgl-project/sglang/pull/18742 and sglang#20214：https://github.com/sgl-project/sglang/pull/20214). Concretely, conversion uses --num-layers-at-start-in-bf16 and --num-layers-at-end-in-bf16 ; Megatron training combines those counts with --first-last-layers-bf16 ; and SGLang serves the resulting mixed-precision checkpoints.
As recommended in the NVIDIA NVFP4 pretraining paper：https://arxiv.org/html/2509.25149v2, we keep a small fraction of final layers in higher precision. In our experiments, keeping the last 15% of layers in BF16 meaningfully reduces train-inference mismatch and stabilizes gradients.
Turning on BF16 for early layers does not lead to meaningful train-inference mismatch reduction in our experiments:
Keeping shared experts in high precision also reduces train-inference mismatch with little performance or memory overhead. Routed experts are selected sparsely, and their outputs pass through a high-precision weighted reduction. Shared experts are always active, so their precision errors affect every token passing through the block.
For MLA models, kv_b_proj is an important MXFP8 case. Absorbed and non-absorbed MLA modes can use different contraction axes, while MXFP8 uses one-dimensional microscaling blocks. Changing the contraction axis can therefore change which elements share a scale. The same concern applies to other one-dimensional formats, including NVFP4. The original DeepSeek-V3：https://arxiv.org/html/2412.19437v2 FP8 recipe does not have this exact weight-side issue because it uses 128x128 weight-scale blocks rather than one-dimensional blocks. We keep these projection tensors in BF16 to avoid hidden requantization and preserve train-inference consistency.
The matching name-based conversion-time and rollout-time arguments are:
For DeepSeek-V3：https://arxiv.org/html/2412.19437v2-style MLA, this BF16 exception is small. A kv_b_proj tensor of shape 32768 x 512 , stored in BF16 across 61 layers, occupies about 1.90625 GB.
For consistent comparison, all experiments use synchronous Qwen3-30B-A3B RL with the default Miles setup on 8x B200. The fixed workload uses GRPO-style training on dapo-math-17k , with 8 rollout samples per prompt and a maximum response length of 8192 tokens. This is only a recipe ablation setup, not a fully tuned training or serving benchmark. The KL path is enabled for diagnostics, but its coefficient is 0.0, so KL is not an optimization penalty in this ablation.
The hardware split is 4 GPUs for rollout and 4 GPUs for training.
As expected, both low-precision formats show higher train-inference mismatch than BF16, while the two backward choices behave similarly within each format. The values remain in a reasonable range for this ablation.
NVFP4 begins with a higher reference KL than BF16 or MXFP8. Miles computes this diagnostic against a Megatron BF16 reference model, so the metric includes the difference between each low-precision policy and the BF16 reference in addition to policy evolution during RL. It should not be read as a standalone optimization penalty.
Despite the higher diagnostic mismatch, all five low-precision reward curves closely track the BF16 reward curve.
This is the key result of the recipe ablation: in this Qwen3-30B-A3B B200 setup, Blackwell-native low precision preserves the observed learning curve while improving rollout efficiency.
MXFP8 and NVFP4 both reduce rollout time compared with BF16:
For NVFP4 rollout, FlashInfer computes the online per-token FP32 scale directly inside the activation quantization kernel path rather than as a separate preprocessing step. The reported rollout performance therefore includes the cost of online scale computation.
On the training side, the MXFP8 variants are faster than BF16, while the NVFP4 backward-override variants are slower in the implementation measured here:
The training-side gap comes from the implementation used in this ablation, not an inherent FP4 Tensor Core limit. Our TransformerEngine path applies per-token FP32 scaling as a separate PyTorch operation (TransformerEngine#2931：https://github.com/NVIDIA/TransformerEngine/pull/2931) instead of a native per-token NVFP4 GEMM path with scaling fused into the kernel epilogue. We have implemented and upstreamed the fused cuDNN frontend kernels (cudnn-frontend#251：https://github.com/NVIDIA/cudnn-frontend/pull/251); TransformerEngine integration remains pending. Dequantized backward adds the dequantization step described above. Because this RL workload is rollout-heavy, the rollout speedup remains meaningful even before the training path is fully accelerated.
Beyond this ablation, humans&：https://humansand.ai/ uses the same recipe family and components in production for large-scale, long-context, multi-agent asynchronous RL research.
Scan to open this article