Cursor 开源 Mixture-of-Kittens （MoK）：面向 GB300 NVL72 机架的确定性 MoE 训练 Megakernel

Cursor Research 已开源 Mixture-of-Kittens (MoK)：https://cursor.com/blog/mixture-of-kittens，这是其 Composer 背后的混合专家训练超核：https://cursor.com/blog/composer-2 模型。MoK 将每个 MoE 的通信和计算步骤融合为一个确定性核。Cursor 团队报告称，其吞吐量比最强的公开基线高出最多 2.37 倍。它已经为数万 GPU 的 Composer 训练提供了支持。

是的，但硬件门槛很高。MoK 在 GitHub：https://github.com/cursor/mixture-of-kittens 上以 Apache-2.0 协议开源。它需要 NVIDIA Blackwell SM100 或 SM103 GPU，这意味着 GB200 NVL72 或 GB300 NVL72 机架。此外，还需要 Python 3.12+、PyTorch 2.10+ 和 CUDA 工具包 13.0+。GPU 之间的缓冲区依赖于 PyTorch 对称内存。

这限制了现实中的采用者，仅限拥有或租用 NVL72 容量的组织。前沿实验室、获得资金的模型初创公司、GPU 新云以及国家计算中心符合条件。单节点团队和 8-GPU 小型团队则不适用。

应用范围狭窄但价值高。包括 DeepSeek-V3 风格 MoE 模型的预训练和后训练。确定性也使其在策略内 RL 后训练和内部消融实验中有用。相关行业包括 AI 模型开发、云 GPU 基础设施、代码生成工具以及量化研究。

Cursor 早期的工作覆盖了计算方面。研究团队编写了自己的 MXFP8 和 NVFP4 训练核心：https://cursor.com/blog/kernels，以及用于 MoE 推理的“warp decode”路径：https://cursor.com/blog/warp-decode。这些假设 GPU 之间的通信是单独处理的。

在生产中，通信成为限制因素。MoE 层可能消耗超过端到端训练时间的一半。迁移到 GB300 NVL72 改变了问题。一机架内有 72 个 GPU，位于同一 NVLink 域内，这允许细粒度的重叠。但与 GPU 相比，集成的 Grace CPU 速度较慢。因此必须积极最小化 CPU 与 GPU 的同步。

MoK 构建为一个巨内核（megakernel）：https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles，并且是完全确定性的。它支持 BF16 和 MXFP8 精度模式。调度通过 Blackwell 的 Cluster Launch Control 运行，因此机架间的 RDMA 不会在其后排队。路由器权重梯度使用 SonicMoE：https://arxiv.org/abs/2512.14080 风格的计算，融合到 SwiGLU 的反向传播中。

层基准测试在单个 NVL72 机架上以 EP 等级 64 运行。每个 GPU 在路由之前持有 2,048 个令牌。基线包括 NCCL+PyTorch、DeepEP+PyTorch、DeepEP+TransformerEngine 和 HybridEP+Megatron。覆盖的模型包括 Kimi K2.7 代码、GLM-5.2、Qwen3.5-397B-A17B 和 DeepSeek-V4-Pro。

相比最快的基准，MoK 的 MXFP8 正向运算速度最多提高 2.37 倍。其他数据为 MXFP8 反向运算 1.78 倍、BF16 正向运算 1.92 倍、BF16 反向运算 1.58 倍。端到端测试使用了跨多个 GB300 NVL72 机架的 512 个 GPU。每个 GPU 的每秒 token 数从 760.9 上升到 1,070.2，提升 1.41 倍。

需要与我们合作推广您的 GitHub 仓库、Hugging Face 页面、产品发布或网络研讨会等吗？请联系我们： https://forms.gle/wbash1wF6efRj8G58

Asif Razzaq 是 Marktechpost Media Inc. 的 CEO。作为一位有远见的企业家和工程师，Asif 致力于利用人工智能的潜力来造福社会。他最近的项目是推出人工智能媒体平台 Marktechpost，该平台以深入报道机器学习和深度学习新闻而闻名，内容既具有技术性，又容易被广大读者理解。该平台每月访问量超过 200 万，显示出其在受众中的受欢迎程度。

[免费指南] 保护 AI 代理、MCP 服务器及 LLM 应用：https://pxllnk.co/lxn88m

Cursor Research has open-sourced Mixture-of-Kittens (MoK)：https://cursor.com/blog/mixture-of-kittens, the mixture-of-experts training megakernel behind its Composer：https://cursor.com/blog/composer-2 models. MoK fuses every MoE communication and computation step into a single deterministic kernel. Cursor team reports up to 2.37x higher throughput than the strongest public baseline. It already powers Composer training across tens of thousands of GPUs.

Yes, but the hardware floor is high. MoK is on GitHub：https://github.com/cursor/mixture-of-kittens under Apache-2.0. It requires NVIDIA Blackwell SM100 or SM103 GPUs, which means GB200 NVL72 or GB300 NVL72 racks. It also needs Python 3.12+, PyTorch 2.10+, and CUDA toolkit 13.0+. Inter-GPU buffers rely on PyTorch symmetric memory.

That limits realistic adopters to organizations that own or rent NVL72 capacity. Frontier labs, funded model startups, GPU neoclouds, and national computing centers fit. Single-node teams and 8-GPU shops do not.

Applications are narrow but high-value. They include pretraining and post-training of DeepSeek-V3-style MoE models. Determinism also makes it useful for on-policy RL post-training and internal ablations. Relevant industries are AI model development, cloud GPU infrastructure, code-generation tooling, and quantitative research.

Cursor’s earlier work covered the compute side. The research team wrote its own MXFP8 and NVFP4 training kernels：https://cursor.com/blog/kernels and a ‘warp decode’：https://cursor.com/blog/warp-decode path for MoE inference. Those assumed inter-GPU communication was handled separately.

In production, communication became the limiting factor. The MoE layer can consume more than half of end-to-end training time. Moving to GB300 NVL72s changed the problem again. A rack is 72 GPUs inside one NVLink domain, which allows fine-grained overlap. But the integrated Grace CPUs are slow relative to the GPUs. CPU-GPU synchronization therefore has to be minimized aggressively.

MoK is built as a megakernel：https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles and is fully deterministic. It supports BF16 and MXFP8 precision modes. Scheduling runs through Blackwell’s Cluster Launch Control, so inter-rack RDMA does not serialize behind it. Router weight gradients use a SonicMoE：https://arxiv.org/abs/2512.14080-style calculation fused into the SwiGLU backward.

Layer benchmarks ran in a single NVL72 rack at EP degree 64. Each GPU held 2,048 tokens before routing. Baselines were NCCL+PyTorch, DeepEP+PyTorch, DeepEP+TransformerEngine, and HybridEP+Megatron. Shapes covered Kimi K2.7 Code, GLM-5.2, Qwen3.5-397B-A17B, and DeepSeek-V4-Pro.

Against the fastest baseline, MoK is up to 2.37x faster for MXFP8 forward. The other figures are 1.78x MXFP8 backward, 1.92x BF16 forward, and 1.58x BF16 backward. End-to-end testing used 512 GPUs across several GB300 NVL72 racks. Tokens per second per GPU rose from 760.9 to 1,070.2, a 1.41x gain.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us ：https://forms.gle/wbash1wF6efRj8G58

Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.

[FREE GUIDE] Securing AI Agents, MCP Servers & LLM Apps ：https://pxllnk.co/lxn88m