Unsloth vs Axolotl vs TRL vs LLaMA-Factory：四大开源微调框架速度、显存与多GPU对比

目前，四大开源项目主导着大型语言模型的微调。Unsloth：https：//github.com/unslothai/unsloth，Axolotl：https：//github.com/axolotl-ai-cloud/axolotl，TRL：https：//github.com/huggingface/trl，以及LLaMA-Factory：https：//github.com/hiyouga/LlamaFactory，都包裹了相同的底层PyTorch和Hugging Face堆叠。他们在工程投入方面存在分歧。

Unsloth 重写内核。Axolotl 编写并行策略。TRL定义了其他工具基于的培训器API。LLaMA-Factory 优化模型覆盖范围和零代码操作。

这次比较涵盖了工程师们实际涉及的三个方面：训练吞吐量、峰值显存和多GPU扩展。

Unsloth发布的基准测试：https：//unsloth.ai/docs/basics/unsloth-benchmarks，Llama 3.1 8B和Llama 3.3 70B的训练速度是2倍。该设置使用了羊驼数据集，批次尺寸为2，梯度累积为4。QLoRA在所有线性图层上都以32级运行。

MoE的结果：https：//unsloth.ai/docs/basics/faster-moe 更大。Unsloth 在 NVIDIA B200 上微调了 unsloth/gpt-oss-20b-BF16。在8K上下文下，每步运行712.33毫秒，而变形金刚v5为5,226.86毫秒。那是7.3倍的差距。4K时差距是4.82倍，1K时只有1.37倍。

趋势方向依赖于模型，Unsloth的文档将这一说法扩展到GPT-OSS。在该区域，加速速度随着序列长度增加而增加，这归功于Flex Attention和MoE核心。

B200上的Qwen3-30B-A3B则是相反方向。据报道，其加速从1000倍时的1.7倍下降到16千倍时的1.1倍。内存节省则相反，从约2%上升到15%。

H100上的Qwen3-30B-A3B信号可达1.77倍。RTX PRO 6000上的GLM-4.7闪存可达2.1倍。与AMD合作：https：//unsloth.ai/docs/basics/amd 测量了 Llama-3.1-8B LoRA SFT 每步 2.07 秒。TRL加上FlashAttention-2每步加速2.87秒，差距为1.39倍，损耗曲线相匹配。

Axolotl 于 2025 年 2 月为 LoRA 添加了自定义的 Triton 核和自进阶梯函数：https：//axolotlai.substack.com/p/accelerating-lora-fine-tuning-with，明确引用了 Unsloth 作为灵感来源。他们通过lora_mlp_kernel、lora_qkv_kernel和lora_o_kernel选择加入。

最新发布说明：https：//github.com/axolotl-ai-cloud/axolotl/releases 新增了对 SonicMoE LoRA 的支持。相比grouped_mm基线，它可实现最高1.45倍的加速和30%的内存削减。该数据适用于单台H100 SXM上的Qwen3.5-35B-A3B 8位LoRA。

Axolotl 还发布了 FlashAttention 2/3/4、xFormers、Flex Attention、SageAttention、Liger Kernel、Cut Cross Entropy 和 ScatterMoE。

TRL通常是单GPU原始吞吐量的参考点，而非最终的。它通过《减少内存使用》中的内存和速度控制措施来补偿：https：//huggingface.co/docs/trl/en/reducing_memory_usage 和《加快训练速度：https：//huggingface.co/docs/trl/en/speeding_up_training》。

这些工具包括打包、无填充批处理、截断、Liger Kernel：https：//huggingface.co/docs/trl/en/liger_kernel_integration，以及用于GRPO的vLLM睡眠模式。TRL还有第一方Unsloth集成：https：//huggingface.co/docs/trl/en/unsloth_integration，所以两者并不互斥。

LLaMA-Factory 不编写自己的内核。它通过配置标志暴露了别人的作品。

设置use_unsloth：真值会激活无懈修补丁。项目的更新日志显示该路径的相对速度为170%。Unsloth的长序列训练显示速度为117%，记忆为50%。它还支持 enable_liger_kernel： true 和通过 flash_attn： fa2 实现的 FlashAttention-2。

Unsloth 发布了一个显存需求表：https：//unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements，按参数数量排序。8B型号在4位QLoRA中列出6 GB，70B为41 GB。16位LoRA的费用分别是22GB，同型号分别是164GB。

LLaMA-Factory 的 README 硬件表：https：//github.com/hiyouga/LlamaFactory 也涵盖了 4 位 QLoRA 的相同流程。它列出的7B为6GB，30B为24GB，70B为48GB。70B的完整bf16微调显示为600GB。

两张表都描述了最低要求。批次大小、序列长度和优化器的选择会移动实数。

在固定上下文下的峰值显存不如给定显存预算允许的最大上下文有趣。Unsloth的上下文长度基准测试：https：//unsloth.ai/docs/basics/unsloth-benchmarks 针对Llama 3.1 8B QLoRA在32级、批次1时的基准测试非常明显。

Unsloth 将此归因于其梯度检查点算法结合 Apple 的 Cut Cross Entropy。在 80 GB A100 上，Llama 3.3 70B 报告了 89,389 个 token。FA2 基线达到 6,916 个。

MoE 训练是 2026 年内内存行为变化最大的地方。Unsloth 报告 gpt-oss-20b 微调只需 12.8 GB，而 Qwen3-30B-A3B 在 16 位 LoRA 下需要 63 GB。

其 B200 gpt-oss 运行在 8K 上下文使用了 47.43 GB，而 Transformers v5 使用了 73.80 GB。在 16K 上，Transformers v5 内存不足，而 Unsloth 使用了 55.13 GB。

其机制是一个 split-LoRA 公式。PEFT 会在 MoE 矩阵乘之前将 LoRA delta 实例化在所有专家上。Unsloth 则重新排序操作，数学上等效，但避免了实例化。

Axolotl 以不同方式解决同一问题。其 MoE 专家量化：https://docs.axolotl.ai/docs/expert_quantization.html 在模型加载期间量化专家权重，立即释放原始 bf16 张量。

原因是 Transformers v5 的变化。专家层从 nn.Linear 移动到融合的 nn.Parameter 3D 张量，bitsandbytes 无法在加载时量化它们。Axolotl 的文档报告 GLM-4.7-Flash QLoRA 在 quantize_moe_experts: true 时，保留内存从约 127 GiB 降至约 23 GiB。

这就是排名反转的地方。Unsloth 的单 GPU 领先优势不会延续。

Axolotl 的多 GPU 指南：https://docs.axolotl.ai/docs/multi-gpu.html 提供三种互斥的分片策略：DeepSpeed ZeRO 1 到 3 阶段、FSDP 和 DDP。推荐使用 FSDP2，FSDP1 已弃用。

在此基础上，其 N-D 并行指南：https://docs.axolotl.ai/docs/nd_parallelism.html 通过 PyTorch 的 DeviceMesh 组合数据、张量、上下文和专家并行。文档中列出的支持矩阵确认 FSDP+TP、HSDP+TP、FSDP+CP、FSDP+TP+CP 和 FSDP+EP。

有两种组合明确不支持。专家并行在 v1 中不能与 TP 或 CP 组合。纯 DDP 也不能与它们组合。

Axolotl的序列并行性：https：//docs.axolotl.ai/docs/sequence_parallelism.html 使用环闪注意力库。它发布的H100基准测试：https：//huggingface.co/blog/axolotl-ai-co/long-context-with-sequence-parallelism-in-axolotl 针对Llama 3.1 8B QLoRA支持这种权衡。

语境的规模几乎是线性的。吞吐量效率崩溃。在4090的SP度8上，同样的基准测试显示了0.88倍的加速。训练速度变慢，而上下文代币达到32,768个。

Axolotl还支持通过Torchrun和Ray进行多节点训练：https：//docs.axolotl.ai/docs/ray-integration.html。

TRL的分布式培训指南：https：//huggingface.co/docs/trl/en/distributing_training 给出了清晰的区分。上下文并行性意味着 FSDP2 上的 Ring Attention。序列并行性指的是DeepSpeed上的ALST/Ulysses。

Ring Attention 需要 Accelerate 1.11.0+，使用 cp_size 和 cp_backend=“torch”，目前仅支持 SDPA。FlashAttention在该路径上不被支持。序列必须除以 cp_size * 2。

ALST/Ulysses 需要 DeepSpeed 0.18.1+ 和 Accelerate 1.12.0+。它使用sp_size，sp_backend=“deepspeed”，并与FlashAttention-2配合使用。它受注意力人数限制，要求num_heads >= sp_size。

TRL的指导很具体。环形注意力适用于1M+令牌序列和有限的网络拓扑。Ulysses 适合 NVLink 或 InfiniBand 互连，并排序最多约 500k 代币。

TRL自家的Ring Attention基准测试对Qwen3-8B在1、2、4和8张H100显卡上进行了微调。在8块GPU上，超过30万个令牌的上下文长度变得可训练。

LLaMA-Factory的分布式培训文档：https：//llamafactory.readthedocs.io/en/latest/advanced/distributed.html 涵盖DDP、DeepSpeed和FSDP，包括FSDP2和Ray，适用于单节点和多节点运行。文档还介绍了DeepSpeed AutoTP，它结合了张量并行和ZeRO。

2025年最重要的新增功能是通过mcore_adapter实现了威震天核心训练后端：https：//github.com/alibaba/ROLL/tree/main/mcore_adapter。这开启了一条真正的大规模预培训路径。

FSDP+QLoRA路径用于在两块24GBGPU上微调70B型号。这是本次比较中通往70B的最便宜路径。

摩擦点在于接口。分布式配置存在于 YAML 和 CLI 中，而不在 LlamaBoard 中。采用 LLaMA-Factory 以零代码 UI 的开发团队，在 GPU 数量超过一块时，会失去这一特性。

Unsloth 的多 GPU 文档：https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth 指出多 GPU 通过 Accelerate 和 DeepSpeed 实现，可访问 FSDP 和 DDP。文档还指出，该过程复杂且需要手动设置，官方支持仍在宣布中。

实际操作方式是使用 accelerate launch train.py 或 torchrun --nproc_per_node N_GPUS train.py。对于单 GPU 容纳不下的模型，device_map = "balanced" 可将模型拆分到不同设备上。

Unsloth 的 PyPI 列表：https://pypi.org/project/unsloth/ 表明多 GPU 功能可用，但重大改进仍在进行中。Studio 更新日志：https://unsloth.ai/docs/new/changelog 描述了截至 2026 年 3 月，为推理和训练提供的初步自动多 GPU 分配功能。

综合来看，立场很明确。Unsloth 支持多 GPU，但尚未提供 Axolotl 和 TRL 文档中描述的可组合并行矩阵。

Asif Razzaq 是 Marktechpost Media Inc. 的 CEO。作为一名有远见的企业家和工程师，Asif 致力于利用人工智能的潜力造福社会。他最近的努力是推出人工智能媒体平台 Marktechpost，该平台因其对机器学习和深度学习新闻的深入报道而脱颖而出，内容既技术扎实，又易于广大受众理解。该平台每月访问量超过 200 万次，显示其受欢迎程度。

构建一个能动事件场馆运营者 [完整代码]：https://pxllnk.co/twdn5

谢谢！我们的团队将很快与您联系 🙌

Four open source projects dominate LLM fine-tuning today. Unsloth：https://github.com/unslothai/unsloth, Axolotl：https://github.com/axolotl-ai-cloud/axolotl, TRL：https://github.com/huggingface/trl, and LLaMA-Factory：https://github.com/hiyouga/LlamaFactory all wrap the same underlying PyTorch and Hugging Face stack. They diverge on where they spend engineering effort.

Unsloth rewrites kernels. Axolotl composes parallelism strategies. TRL defines the trainer APIs the others build on. LLaMA-Factory optimizes for breadth of model coverage and zero-code operation.

This comparison covers three axes engineers actually hit: training throughput, peak VRAM, and multi-GPU scaling.

Unsloth's published benchmarks：https://unsloth.ai/docs/basics/unsloth-benchmarks show 2x training speed for Llama 3.1 8B and Llama 3.3 70B. The setup used the Alpaca dataset, batch size 2, and gradient accumulation 4. QLoRA ran at rank 32 on all linear layers.

The MoE results：https://unsloth.ai/docs/basics/faster-moe are larger. Unsloth fine-tuned unsloth/gpt-oss-20b-BF16 on an NVIDIA B200. It reports 712.33 ms per step at 8K context, versus 5,226.86 ms for Transformers v5. That is a 7.3x gap. At 4K the gap is 4.82x, and at 1K it is only 1.37x.

The trend direction is model-dependent, and Unsloth's docs scope this claim to gpt-oss. There, the speedup grows with sequence length, credited to Flex Attention and the MoE kernels.

Qwen3-30B-A3B on B200 runs the other way. Its reported speedup falls from 1.7x at 1K to 1.1x at 16K. Memory savings move the opposite direction, rising from about 2% to 15%.

Qwen3-30B-A3B on H100 reaches up to 1.77x. GLM-4.7-Flash on RTX PRO 6000 reaches 2.1x. A collaboration with AMD：https://unsloth.ai/docs/basics/amd measured Llama-3.1-8B LoRA SFT at 2.07 s/step. TRL plus FlashAttention-2 took 2.87 s/step, a 1.39x gap with matching loss curves.

Axolotl added custom Triton kernels and autograd functions：https://axolotlai.substack.com/p/accelerating-lora-fine-tuning-with for LoRA in February 2025, explicitly citing Unsloth as inspiration. They are opt-in through lora_mlp_kernel , lora_qkv_kernel , and lora_o_kernel .

Recent release notes：https://github.com/axolotl-ai-cloud/axolotl/releases add SonicMoE LoRA support. It delivers up to 1.45x speedup and 30% memory reduction over a grouped_mm baseline. That figure is for Qwen3.5-35B-A3B 8-bit LoRA on a single H100 SXM.

Axolotl also ships FlashAttention 2/3/4, xFormers, Flex Attention, SageAttention, Liger Kernel, Cut Cross Entropy, and ScatterMoE.

TRL is usually the reference point rather than the winner on raw single-GPU throughput. It compensates with breadth of memory and speed levers documented in Reducing Memory Usage：https://huggingface.co/docs/trl/en/reducing_memory_usage and Speeding Up Training：https://huggingface.co/docs/trl/en/speeding_up_training.

Those levers include packing, padding-free batching, truncation, Liger Kernel：https://huggingface.co/docs/trl/en/liger_kernel_integration, and vLLM sleep mode for GRPO. TRL also has a first-party Unsloth integration：https://huggingface.co/docs/trl/en/unsloth_integration, so the two are not mutually exclusive.

LLaMA-Factory does not write its own kernels. It exposes other people's work through config flags.

Setting use_unsloth: true activates the Unsloth patch. The project's changelog reports 170% relative speed from that path. Unsloth's long-sequence training is listed at 117% speed and 50% memory. It also supports enable_liger_kernel: true and FlashAttention-2 via flash_attn: fa2 .

Unsloth publishes a VRAM requirements table：https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements sorted by parameter count. It lists 6 GB for an 8B model in 4-bit QLoRA and 41 GB for 70B. LoRA at 16-bit costs 22 GB and 164 GB for the same models.

LLaMA-Factory's README hardware table：https://github.com/hiyouga/LlamaFactory covers the same regime for 4-bit QLoRA. It lists 6 GB at 7B, 24 GB at 30B, and 48 GB at 70B. Full bf16 fine-tuning of 70B is listed at 600 GB.

Both tables describe minimums. Batch size, sequence length, and optimizer choice move the real number.

Peak VRAM at a fixed context is less interesting than the maximum context a given VRAM budget allows. Unsloth's context length benchmarks：https://unsloth.ai/docs/basics/unsloth-benchmarks for Llama 3.1 8B QLoRA at rank 32 and batch size 1 are stark.

Unsloth attributes this to its gradient checkpointing algorithm combined with Apple's Cut Cross Entropy. For Llama 3.3 70B on an 80 GB A100, it reports 89,389 tokens. The FA2 baseline reaches 6,916.

MoE training is where memory behavior has shifted most in 2026. Unsloth reports gpt-oss-20b fine-tuning inside 12.8 GB, while Qwen3-30B-A3B at 16-bit LoRA needs 63 GB.

Its B200 gpt-oss run used 47.43 GB at 8K context where Transformers v5 used 73.80 GB. At 16K, Transformers v5 went out of memory and Unsloth used 55.13 GB.

The mechanism is a split-LoRA formulation. PEFT materializes the LoRA delta across all experts before the MoE matmul. Unsloth reorders the operations instead, which is mathematically identical but avoids the materialization.

Axolotl attacks the same problem differently. Its MoE expert quantization：https://docs.axolotl.ai/docs/expert_quantization.html quantizes expert weights during model loading, freeing the original bf16 tensor immediately.

The reason is a Transformers v5 change. Expert layers moved from nn.Linear to fused nn.Parameter 3D tensors. bitsandbytes could no longer quantize them on load. Axolotl's docs report GLM-4.7-Flash QLoRA dropping from roughly 127 GiB to roughly 23 GiB reserved memory with quantize_moe_experts: true .

This is where the ranking inverts. Unsloth's single-GPU lead does not carry over.

Axolotl's multi-GPU guide：https://docs.axolotl.ai/docs/multi-gpu.html offers three mutually exclusive sharding strategies: DeepSpeed ZeRO stages 1 through 3, FSDP, and DDP. FSDP2 is the recommended path, and FSDP1 is deprecated.

On top of those, its N-D Parallelism guide：https://docs.axolotl.ai/docs/nd_parallelism.html composes data, tensor, context, and expert parallelism through PyTorch's DeviceMesh . The documented support matrix confirms FSDP+TP, HSDP+TP, FSDP+CP, FSDP+TP+CP, and FSDP+EP.

Two combinations are explicitly unsupported. Expert parallelism cannot compose with TP or CP in v1. Pure DDP cannot compose with them either.

Axolotl's sequence parallelism：https://docs.axolotl.ai/docs/sequence_parallelism.html uses the ring-flash-attention library. Its published H100 benchmark：https://huggingface.co/blog/axolotl-ai-co/long-context-with-sequence-parallelism-in-axolotl for Llama 3.1 8B QLoRA supports the tradeoff.

Context scales close to linearly. Throughput efficiency collapses. On 4090s at SP degree 8, the same benchmark records 0.88x speedup. Training got slower while context reached 32,768 tokens.

Axolotl also supports multi-node training through torchrun and Ray：https://docs.axolotl.ai/docs/ray-integration.html.

TRL's distributed training guide：https://huggingface.co/docs/trl/en/distributing_training draws a clean distinction. Context Parallelism means Ring Attention on FSDP2. Sequence Parallelism means ALST/Ulysses on DeepSpeed.

Ring Attention needs Accelerate 1.11.0+, uses cp_size with cp_backend="torch" , and currently supports SDPA only. FlashAttention is not supported on that path. Sequences must divide by cp_size * 2 .

ALST/Ulysses needs DeepSpeed 0.18.1+ and Accelerate 1.12.0+. It uses sp_size with sp_backend="deepspeed" and works with FlashAttention-2. It is bounded by attention head count, requiring num_heads >= sp_size .

TRL's guidance is specific. Ring Attention suits 1M+ token sequences and limited network topology. Ulysses suits NVLink or InfiniBand interconnects and sequences up to roughly 500k tokens.

TRL's own Ring Attention benchmark fine-tuned Qwen3-8B across 1, 2, 4, and 8 H100 GPUs. At 8 GPUs, context lengths above 300k tokens became trainable.

LLaMA-Factory's distributed training docs：https://llamafactory.readthedocs.io/en/latest/advanced/distributed.html cover DDP, DeepSpeed, and FSDP, including FSDP2 and Ray for single-node and multi-node runs. The docs also describe DeepSpeed AutoTP, which combines tensor parallelism with ZeRO.

Its most consequential 2025 addition was a Megatron-core training backend through mcore_adapter：https://github.com/alibaba/ROLL/tree/main/mcore_adapter. That opens a genuine large-scale pretraining path.

The FSDP+QLoRA path fine-tunes a 70B model on two 24 GB GPUs. That is the cheapest documented route to 70B in this comparison.

The friction point is the interface. Distributed configuration lives in YAML and CLI, not in LlamaBoard. Dev teams that adopted LLaMA-Factory for its zero-code UI lose that property when they scale past one GPU.

Unsloth's multi-GPU documentation：https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth states that multi-GPU works through Accelerate and DeepSpeed, giving access to FSDP and DDP. It also states the process is complex and requires manual setup, with official support still being announced.

The practical route is accelerate launch train.py or torchrun --nproc_per_node N_GPUS train.py . For models too large for one GPU, device_map = "balanced" splits the model across devices.

Unsloth's PyPI listing：https://pypi.org/project/unsloth/ marks multi-GPU as available with major improvements pending. The Studio changelog：https://unsloth.ai/docs/new/changelog describes preliminary automatic multi-GPU allocation for inference and training as of March 2026.

Read together, the position is clear. Unsloth supports multi-GPU. It does not yet offer the composable parallelism matrix that Axolotl and TRL document.

Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.

Build an Agentic Event Venue Operator [Full Codes]：https://pxllnk.co/twdn5

Thanks! Our team will contact you soon 🙌