Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

大多数这些后端都是仅权重模式（weight-only）。这意味着它们以低精度存储权重，并在计算时将其反量化回高精度。这显著减少了内存使用，但通常不会加快推理速度，甚至可能增加少量延迟开销。

SVDQuant：https://arxiv.org/abs/2411.05007，流行的 Nunchaku 推理引擎：https://github.com/nunchaku-tech/nunchaku 背后的量化方法，采取了不同的方法。它在主要的 Transformer 层中使用 4 位权重和激活（W4A4）运行，既减少了内存占用，又加快了去噪循环。下面会详细介绍，但直到现在，使用这些检查点仍需要单独的推理库。

使用目前的 Diffusers，加载 Nunchaku 检查点就像调用 from_pretrained() 一样简单，并且由于 kernels：https://github.com/huggingface/kernels 包的帮助，无需本地 CUDA 编译。另外，辅助工具 diffuse-compressor：https://github.com/rootonchair/diffuse-compressor 让你可以自己量化新架构，并将其发布为常规 Diffusers 仓库。

首先，安装依赖。你需要使用较新版本的 Diffusers 和 Hugging Face kernels 包：

然后像加载其他 Diffusers 模型一样，加载预量化的 pipeline：

无需自定义 pipeline 类或单独的推理引擎，也无需在本地编译任何内容。NVFP4 内核会在首次使用时通过 Nunchaku Lite 内核页面：https://huggingface.co/kernels/rootonchair/nunchaku-lite-kernels 从 Hub 下载。该检查点将 Nunchaku NVFP4 Transformer 与 bitsandbytes NF4 文本编码器配对，在 RTX 5090 上生成 1024x1024 图像大约需要 1.7 秒，峰值内存使用约为 12 GB，而 BF16 pipeline 则大约需要 24 GB。有关 Nunchaku Lite 检查点格式的更多详细信息，请参阅官方 Diffusers 文档：https://huggingface.co/docs/diffusers/main/en/quantization/nunchaku。

NVFP4 检查点需要 NVIDIA Blackwell GPU（RTX 50 系列，RTX PRO 6000，B200）。对于早期型号，请使用 INT4 版本。有关详细信息，请参见下方的硬件支持：#hardware-support 表格。

SVDQuant 是 Nunchaku 背后的量化方法，也是其参考 CUDA 推理引擎。对于扩散变压器，标准的 4 位量化比较困难，因为权重和激活都包含大量异常值。SVDQuant 通过将激活异常值移动到权重中处理这一问题，用一个小的 16 位低秩分支表示每个权重矩阵中最复杂的部分，并将剩余的残差量化为 4 位。Nunchaku 通过为 4 位路径和低秩分支提供融合内核，使这一过程变得快速。

原始的 Nunchaku 引擎：https://github.com/nunchaku-ai/nunchaku 很大程度上的速度来自于针对模型的特定融合执行路径：#quantizing-models-with-structural-rewrites，例如融合的 QKV 投影和融合的 GELU/MLP 内核。这些优化与每个架构的模块布局和检查点格式相关，因此支持一个新的模型家族通常需要针对模型的集成工作。

Nunchaku Lite 是 Diffusers 中的新集成路径。有了它，Diffusers 可以在没有自定义管道或单独推理引擎的情况下加载 Nunchaku 风格的检查点。在底层，Nunchaku Lite 在加载检查点之前，用运行时 SVDQ/AWQ 线性层修补 stock Diffusers 模型的相关 nn.Linear 模块。CUDA 内核来自 Hub，通过 kernels 包获取。使用了两种内核系列：

权衡是，没有架构特定的融合内核和模块，Nunchaku Lite 无法达到原始 Nunchaku 引擎的加速效果。然而，这个精简实现仍然可以在保持相同显存减少水平的同时带来大约 30% 的加速。

如果你在 Diffusers 中使用过 bitsandbytes 或 torchao，这种机制会让你觉得熟悉。Nunchaku Lite 模型库是一个普通的 Diffusers 库。唯一特别的部分是在 transformer 的 config.json 中的 quantization_config 块：

该配置告诉 Diffusers 哪些模块已经量化，它们使用何种方案，以及实例化哪个 Nunchaku Lite 运行时层（SVDQW4A4Linear 或 AWQW4A16Linear）。

因为量化模型保留了稠密模型的精确模块结构，下游的一切（调度器、LoRA 加载钩子、卸载、torch.compile）都能像正常 Diffusers 模型一样工作。

Nunchaku Lite 会根据 GPU 世代和检查点精度使用不同的内核变体：

Volta 和 Hopper GPU 目前不支持 4 位内核。量化器会在加载时验证 GPU 的 CUDA 功能，并在发现不兼容时抛出明确错误，而不是生成错误输出。

Nunchaku Lite 可以与其他 Diffusers 内存和速度优化结合使用。

torch.compile。编译 transformer 可以将端到端加速从 1.35 倍提高到 1.8 倍：

量化的文本编码器。transformer 并不是唯一占用大量内存的组件。文本编码器如 T5 或 Qwen3 本身可以占用数 GB 内存。使用 bitsandbytes NF4 进一步量化文本编码器，在我们的基准测试中可将峰值显存减少约 22%。

卸载。Diffusers 的卸载辅助函数，例如 enable_model_cpu_offload() 和 enable_sequential_cpu_offload()，在需要将管道迁移到较小 GPU 上时仍可照常使用。

以下所有数据均在 NVIDIA RTX PRO 6000（Blackwell）上，分辨率为 1024x1024，使用工具包 rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder：https://huggingface.co/rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder 测量得到。

如上所示，Nunchaku 在减少峰值显存最多 50% 的同时，延迟仍大约提高了 30%。剩余的开销主要来自额外的内核调用，torch.compile 可以缓解这一问题，将完整管道时间降至 1.68 秒，比 BF16 基线快 1.8 倍。

Nunchaku Lite 在 Diffusers 中的支持与架构无关，diffuse-compressor：https://github.com/rootonchair/diffuse-compressor 工具包提供了一个针对 Diffusers 模型的端到端 SVDQuant 工作流程：校准、量化、打包和发布。

下面，我们以量化 FLUX.2 Klein 4B 为例进行演示。它涵盖了主要步骤：检查模型、校准并量化 transformer，将结果打包为 Diffusers 管道，然后验证并推送到 Hub。完整教程：https://github.com/rootonchair/diffuse-compressor/blob/main/docs/quantize_new_hf_model.md 详细介绍了每个参数。

通用扫描器会遍历模型并决定目标：重复的 transformer-block 堆栈中的兼容线性层会成为 SVDQ W4A4 的目标，识别出的调制线性层会成为 AWQ W4A16 的目标，其余的保持为密集层。

在量化之前务必阅读此报告。对于 FLUX.2 Klein 4B，预期结果为 100 个 SVDQ 目标、3 个 AWQ 目标和 6 个密集外部线性层，不应有缺失模式或重复名称。

以下命令在 transformer 上运行 SVDQuant，并将量化后的检查点写入 outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors：

将 --precision int4 替换为 nvfp4 以构建 Blackwell 原生权重。

转换器会将量化的 transformer 与基础管线的其他组件合并，将紧凑的 nunchaku_lite 配置写入 transformer/config.json，并可以选择将文本编码器转换为 NF4：

一旦输出看起来正常，运行 pipe.push_to_hub("your-name/your-model-nunchaku-lite-int4")。其他用户随后可以使用上面显示的相同 from_pretrained() 模式加载它。

请注意，通用路径假定该架构可以在不进行结构重写的情况下进行量化。为了进一步加速，原始 Nunchaku 引擎将 Diffusers 层组重写为融合模块。通用路径无法自行推断这些变化，例如将独立的 Q、K、V 投影合并为一个模块，或将融合投影拆分到多个模块中。

FLUX.1-dev 的 QKV 投影是一个具体示例。Diffusers 定义了三个独立模块：https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py#L313-L329：

Nunchaku FLUX 模块将这些层合并：https://github.com/nunchaku-ai/nunchaku/blob/main/nunchaku/models/transformers/transformer_flux_v2.py#L63-L79 为一个量化的 to_qkv 模块：

该分组模块是必须的，因为 Nunchaku 的融合运算符会同时处理 QKV 投影、Q/K 归一化和旋转嵌入。相比之下，默认的 Diffusers 路径：https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py#L45-L116 会单独执行这些操作：

Nunchaku路径：https://github.com/nunchaku-ai/nunchaku/blob/main/nunchaku/models/attention_processors/flux.py#L69-L93 提供了分组投影、归一化模块以及旋转嵌入到一个融合操作器中：

这是通用路径无法推断的结构性重写。Diffusers 有三个目标模块，分别带有 to_q、to_k 和 to_v 参数前缀，而 Nunchaku 在 to_qkv 下只有一个分组模块。特定模型的目标配置或适配器必须声明 Q、K 和 V 参数应按输出维度顺序连接，并加载到 to_qkv 中。

像这样的结构性重写在量化期间由特定模型的目标配置描述，并在加载检查点时由一个小型运行时适配器处理。FLUX.2 Klein 4B 量化脚本：https://github.com/rootonchair/diffuse-compressor/blob/main/examples/text_to_image/quantize_flux2_klein_4b.py 提供了一个生成结构性重写检查点的具体目标配置示例，而 rootonchair/nunchaku-lite：https://github.com/rootonchair/nunchaku-lite 提供了加载分组 QKV 张量、拆分融合投影以及其他融合操作所需的运行时适配器。要了解完整工作流程，可以查看 Adding A New Model：https://github.com/rootonchair/diffuse-compressor/blob/main/docs/adding_new_model.md 指南。

要立即开始，请查看以下仓库：

Nunchaku 的 SVDQuant 内核是高效在消费级硬件上运行扩散变换器的最有效方式之一，现在已在 Diffusers 中得到原生支持。预量化检查点可通过 from_pretrained() 加载，而 diffuse-compressor 工具包可以实现对新架构的量化，而无需等待引擎支持。通过对权重和激活同时量化，W4A4 路径降低了内存使用，同时提高了去噪延迟，使图像质量接近 BF16 原始版本。

如果你量化并发布了新模型，我们很想听到你的反馈。在 Hub 上分享并告知我们！如果你对该功能有任何疑问，欢迎加入我们的 Discord：https://discord.gg/G7tWnz98XR。

要了解更多，请查看以下资源：

感谢 Diffusers 的维护者在整个集成过程中提供的审查和指导，也感谢 MIT HAN 实验室 / Nunchaku 团队提供的原始 SVDQuant 工作。感谢 Marc Sun 对博客文章提供的反馈。感谢 Álvaro Somoza 试用 nunchaku-lite 并提供反馈。

rootonchair 也感谢 SilverAI 对这项工作的支持，以及提供了进行大部分开发的环境。

Most of these backends are weight-only . This means that they store the weights in low precision and dequantize them back to high precision at compute time. This reduces memory usage significantly, but it usually does not make inference faster, and can even add a small latency overhead.

SVDQuant：https://arxiv.org/abs/2411.05007, the quantization method behind the popular Nunchaku：https://github.com/nunchaku-tech/nunchaku inference engine, takes a different approach. It runs the main transformer layers with 4-bit weights and activations (W4A4), reducing memory while also speeding up the denoising loop. The details are covered below, but until now, using these checkpoints required a separate inference library.

With current Diffusers, loading a Nunchaku checkpoint is as simple as calling from_pretrained() , with no local CUDA compilation required thanks to the kernels ：https://github.com/huggingface/kernels package. In addition, the companion diffuse-compressor：https://github.com/rootonchair/diffuse-compressor toolkit lets you quantize new architectures yourself and publish them as regular Diffusers repositories.

First, install the requirements. You need a recent version of Diffusers and the Hugging Face kernels package:

Then load a pre-quantized pipeline like any other Diffusers model:

No custom pipeline class or separate inference engine is needed, and there is nothing to compile locally. The NVFP4 kernels are downloaded from the Hub through the Nunchaku Lite kernels page：https://huggingface.co/kernels/rootonchair/nunchaku-lite-kernels the first time they are used. This checkpoint pairs a Nunchaku NVFP4 transformer with a bitsandbytes NF4 text encoder, and generates a 1024x1024 image in about 1.7 seconds on an RTX 5090 with a peak memory usage of about 12 GB, compared with about 24 GB for the BF16 pipeline. You can find more details about the Nunchaku Lite checkpoint format in the official Diffusers documentation：https://huggingface.co/docs/diffusers/main/en/quantization/nunchaku.

NVFP4 checkpoints require an NVIDIA Blackwell GPU (RTX 50 series, RTX PRO 6000, B200). For earlier generations, use the INT4 variants. See the hardware support：#hardware-support table below for details.

SVDQuant is the quantization method behind Nunchaku , its reference CUDA inference engine. Standard 4-bit quantization is difficult for diffusion transformers because both weights and activations contain large outliers. SVDQuant handles this by moving activation outliers into the weights, representing the hardest part of each weight matrix with a small 16-bit low-rank branch, and quantizing the remaining residual to 4 bits. Nunchaku makes this fast with fused kernels for the 4-bit path and the low-rank branch.

The original Nunchaku engine：https://github.com/nunchaku-ai/nunchaku gets much of its speed from model-specific fused execution paths：#quantizing-models-with-structural-rewrites, such as fused QKV projections and fused GELU/MLP kernels. Those optimizations are tied to each architecture's module layout and checkpoint format, so supporting a new model family usually requires model-specific integration work.

Nunchaku Lite is the new integration path in Diffusers. With it, Diffusers can load Nunchaku-style checkpoints without a custom pipeline or a separate inference engine. Under the hood, Nunchaku Lite patches the relevant nn.Linear modules of a stock Diffusers model with runtime SVDQ/AWQ linear layers before the checkpoint is loaded. The CUDA kernels come from the Hub through the kernels package. Two kernel families are used:

The trade-off is that, without architecture-specific fused kernels and modules, Nunchaku Lite cannot match the speedup of the original Nunchaku engine. However, the bare-bones implementation still delivers around 30% speedup while retaining the same level of VRAM reduction .

If you have used bitsandbytes or torchao in Diffusers, the mechanics will feel familiar. A Nunchaku Lite model repository is an ordinary Diffusers repository. The only special part is a quantization_config block inside the transformer's config.json :

This config tells Diffusers which modules were quantized, which scheme they use, and which Nunchaku Lite runtime layer to instantiate ( SVDQW4A4Linear or AWQW4A16Linear ).

Because the quantized model keeps the exact module structure of the dense one, everything downstream (schedulers, LoRA loading hooks, offloading, torch.compile ) sees a normal Diffusers model.

Nunchaku Lite uses different kernel variants depending on the GPU generation and checkpoint precision:

Volta and Hopper GPUs are currently not supported by the 4-bit kernels. The quantizer validates the GPU's CUDA capability at load time and raises a clear error instead of producing incorrect outputs.

Nunchaku Lite can be combined with other Diffusers memory and speed optimizations.

torch.compile . Compiling the transformer improves the end-to-end speedup from 1.35x to 1.8x:

Quantized text encoders. The transformer is not the only component with a large memory footprint. Text encoders such as T5 or Qwen3 can occupy several gigabytes on their own. Further quantizing the text encoder with bitsandbytes NF4 reduces peak VRAM by about 22% in our benchmark.

Offloading. Diffusers offloading helpers such as enable_model_cpu_offload() and enable_sequential_cpu_offload() work as usual if you need to fit the pipeline onto a smaller GPU.

All numbers below were measured on an NVIDIA RTX PRO 6000 (Blackwell) at 1024x1024 using rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder：https://huggingface.co/rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder.

As shown above, Nunchaku reduces peak VRAM by up to 50% while still improving latency by roughly 30%. The remaining overhead comes largely from extra kernel launches, which torch.compile can mitigate, bringing the full pipeline down to 1.68 s, or 1.8x faster than the BF16 baseline.

Nunchaku Lite support in Diffusers is architecture-agnostic, and the diffuse-compressor：https://github.com/rootonchair/diffuse-compressor toolkit provides an end-to-end SVDQuant workflow for Diffusers models: calibrate, quantize, package, and publish.

Below, we walk through quantizing FLUX.2 Klein 4B as an example. It covers the main steps: inspect the model, calibrate and quantize the transformer, package the result as a Diffusers pipeline, then verify and push it to the Hub. The full tutorial：https://github.com/rootonchair/diffuse-compressor/blob/main/docs/quantize_new_hf_model.md covers every flag in detail.

The generic scanner walks the model and decides what to target: compatible linears inside the repeated transformer-block stack become SVDQ W4A4 targets, recognized modulation linears become AWQ W4A16 targets, and everything else stays dense.

Always read this report before quantizing. For FLUX.2 Klein 4B, the expected result is 100 SVDQ targets, 3 AWQ targets, and 6 dense outer linears, with no missing patterns or duplicate names.

The following command runs SVDQuant on the transformer and writes the quantized checkpoint to outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors :

Replace --precision int4 with nvfp4 to build Blackwell-native weights.

The converter combines the quantized transformer with the base pipeline's other components, writes the compact nunchaku_lite configuration into transformer/config.json , and can optionally convert text encoders to NF4:

Once the outputs look good, run pipe.push_to_hub("your-name/your-model-nunchaku-lite-int4") . Other users can then load it with the same from_pretrained() pattern shown above.

Note that the generic path assumes the architecture can be quantized without structural rewrites. For additional speedup, the original Nunchaku engine rewrites groups of Diffusers layers as fused modules. The generic path cannot infer these changes on its own, such as combining separate Q, K, and V projections into one module or splitting a fused projection across several modules.

FLUX.1-dev's QKV projection is a concrete example. Diffusers defines three separate modules：https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py#L313-L329:

The Nunchaku FLUX module combines those layers：https://github.com/nunchaku-ai/nunchaku/blob/main/nunchaku/models/transformers/transformer_flux_v2.py#L63-L79 into one quantized to_qkv module:

This grouped module is required because Nunchaku's fused operator consumes the QKV projection, Q/K normalization, and rotary embeddings together. By comparison, the default Diffusers path：https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py#L45-L116 executes them separately:

The Nunchaku path：https://github.com/nunchaku-ai/nunchaku/blob/main/nunchaku/models/attention_processors/flux.py#L69-L93 supplies the grouped projection, normalization modules, and rotary embeddings to one fused operator:

This is the structural rewrite that the generic path cannot infer. Diffusers has three destination modules with to_q , to_k , and to_v parameter prefixes, while Nunchaku has one grouped module under to_qkv . A model-specific target config or adapter must state that the Q, K, and V parameters should be concatenated along the output dimension, in that order, and loaded into to_qkv .

Structural rewrites like these are described by a model-specific target config during quantization and handled by a small runtime adapter when the checkpoint is loaded. The FLUX.2 Klein 4B quantization script：https://github.com/rootonchair/diffuse-compressor/blob/main/examples/text_to_image/quantize_flux2_klein_4b.py provides a concrete target-config example for producing a structurally rewritten checkpoint, while rootonchair/nunchaku-lite：https://github.com/rootonchair/nunchaku-lite provides the runtime adapters needed to load grouped QKV tensors, split fused projections, and other fused operations. For the complete workflow, you can check the Adding A New Model：https://github.com/rootonchair/diffuse-compressor/blob/main/docs/adding_new_model.md guide.

To get started right away, check out the following repositories:

Nunchaku's SVDQuant kernels are one of the most effective ways to run diffusion transformers efficiently on consumer hardware, and they are now natively supported in Diffusers. Pre-quantized checkpoints load with from_pretrained() , and the diffuse-compressor toolkit makes it possible to quantize new architectures without waiting for engine support. By quantizing both weights and activations, the W4A4 path lowers memory use while improving denoising latency, keeping image quality close to the BF16 original.

If you quantize and publish a new model, we would love to hear about it. Share it on the Hub and let us know! If you have any questions about this feature, feel free to join our Discord：https://discord.gg/G7tWnz98XR.

To learn more, check out the following resources:

Thanks to the Diffusers maintainers for reviews and guidance throughout the integration, and to the MIT HAN Lab / Nunchaku team for the original SVDQuant work. Thanks to Marc Sun for providing feedback on the blog post. Thanks to Álvaro Somoza for trying out nunchaku-lite and for providing feedback.

rootonchair is also grateful to SilverAI for supporting this work and providing the environment in which much of this development took place.