MetaRoCE：为 AI 规模以太网打造的全新 RDMA 传输协议

在 Meta，我们一直是推动行业达成越来越多共识的强大动力，即以太网应成为 AI 基础设施的首选网络。我们已经展示了 RoCE 可以在大规模分布式 AI 训练中发挥作用：https://engineering.fb.com/2024/08/05/data-center-engineering/roce-network-distributed-ai-training-at-scale/。现在，我们正在在这项工作基础上推出 MetaRoCE，这是一种从零设计的协议，旨在支持百万 GPU 规模的以太网。

我们已经将 GPU 集群扩大到数十万：https://engineering.fb.com/2025/09/29/data-infrastructure/metas-infrastructure-evolution-and-the-advent-of-ai/，分布在多个数据中心和地区。无论这些集群是在训练下一代前沿模型，还是在全球范围内提供推理服务，网络都是关键环节。

像 all-reduce 和 all-to-all 这样的集合操作在训练期间同步数千个加速器，而最慢的传输速度决定了整个任务的进度。在推理中，分布式模型分片之间的低延迟通信直接影响数亿用户的响应时间。即使是少量的网络阻力，也会直接导致大量计算资源闲置。

标准 RoCE 期望网络按顺序传递每一帧，依赖 PFC，并不鼓励在多平面和大规模网络中提供性能的分包传输。MetaRoCE 的设计目标是提供高吞吐量、低尾延迟和操作简便性，同时网络中的加速器数量和它们之间的距离增加时也能保持性能。

MetaRoCE 的核心理念很简单：网络看见的是数据包，而网卡看见的是意图。传统架构将智能集中在网络中，依赖交换机来执行无损传输并维持顺序。

通过将智能转移到端点，MetaRoCE 将网络分解为许多细粒度的逻辑路径，每条路径都有自己的实时遥测数据——每条路径的 RTT、ECN 状态和利用率。这种可见性解锁了传统 RDMA 难以实现的能力。

MetaRoce 将数据包分散传输到多条路径，因此它们按设计会乱序到达。传输层将乱序到达视为正常情况。每个数据包都携带自己的目的地，因此数据在到达时会直接写入其最终内存位置，没有重排序缓冲区，也没有队头阻塞。

写操作在每个数据包中都携带它们的目的地。发送操作则携带与已发布接收缓冲区的匹配，因此即使前面的消息尚未到达，发送操作也能正确落地，无需往返确认数据的去向。集体通信库可以在适合的情况下使用双向消息，而不必将所有操作都简化为写操作。

MetaRoCE 为每个连接提供一流的路径，并逐包分散传输。每条路径携带不同的 UDP 源端口作为其 ECMP 随机性标识，网卡可以随时更改以将流量从不良路径转移。在多平面网络中，平面选择完全由网卡决定，网络的使用效果取决于网卡的分发能力。由于每条路径维护自身的窗口和往返时延估计，传输层可以区分拥塞和故障并进行显式的流量重平衡，因此一个繁忙或损坏的链路只会减慢该路径，而不会阻塞整个连接。

MetaRoCE 将以太网网络视为有丢失的，并不要求它表现得无丢失——不使用 PFC，也不使用暂停帧。因为每条路径有自己的顺序序列，所以其 256 位选择性确认（SACK）位向量出现的间隙是丢包的证据，而不是乱序。在其他协议中，SACK 主要用于避免重传已到达的数据；在这里，它在间隙出现的瞬间精确触发丢失数据包的重传，并在发生丢失的路径上进行。

MetaRoCE 将传统的基于 ECN 的发送端驱动 AIMD 拥塞控制与接收端驱动的公平分配速率提示结合。窗口大小既按每条路径维护，也按连接维护，因此拥塞标记会修剪经历拥塞的路径，并将下一批数据包引导至畅通的路径。在每次确认中，接收端返回分配给该发送端的入站带宽份额，因此发送端可以直接接近正确速率，而无需搜索。在 Incast 场景下，可在一到两轮往返中解决，且公平性更好，尾延迟更低。

MetaRoCE 向网络结构请求每台交换机已经具备的两样东西：ECN 标记和 ECMP。它不需要数据包裁剪、网络内遥测、基于信用的流控，也不需要交换机端的喷射机制，而且在网络结构提供这些功能时也不会出错。相同的传输可以运行在 fat-tree、多平面、深缓冲和浅缓冲结构上，也可以运行在你无法控制其配置的厂商云上。不涉及任何专有技术，因此网络结构仍然可以自由优化成本和布线。

一个队列对（QP）既传递有序消息流，也传递带宽。传统 RDMA 通过打开更多的 QP（每对节点数十个）来获取更多任一项，每个 QP 拥有独立的拥塞窗口，不关心其他 QP，并在 NIC 上维持自己的状态。

MetaRoCE 将两者分开。单个连接在上层携带许多独立有序流，每个流对应一个通信器或集合操作，下层在一个拥塞控制器下使用多条路径。随着工作负载并行度的增加，连接状态不会增长。

应用层保持大部分不变——现有 RDMA Verbs API 和软件栈无需修改即可使用。通过扩展 API 支持多平面等增强功能。

为了加速硬件验证，我们与 AMD 合作，在其 Pensando 可编程 NIC 上实现了 MetaRoCE。

在一个 64 节点 AMD GPU 集群上运行 RCCL 集合操作时，我们直接将 MetaRoCE 与 RoCEv2 在 all-reduce 和 all-to-all 操作上进行了比较。结果与设计目标一致：

MetaRoCE 始终提供比 RoCEv2 更高的吞吐量和更低的流完成时间。

在会降低 RoCEv2 性能的数据包丢失条件下，MetaRoCE 在 1% 丢包率时仍维持约 86% 的吞吐量，即使在极端 10% 丢包率下也持续提供有效带宽——平滑收敛而不是崩溃。

在多平面验证中，针对 4 平面和 8 平面拓扑及多达 4,000 个并发连接，吞吐量随平面数量呈线性增长。

在模拟平面故障期间，协议表现出平滑的自动恢复能力——流量在无需应用介入或操作员干预的情况下重新分布。

这些结果支持了 MetaRoce 的核心设计选择。通过从一开始就针对丢包进行设计，并将智能推向边缘，你可以得到在理想条件下表现更好的传输，并在出现问题时平滑降级。

AI 基础设施受益于加速整个生态系统创新的共享标准。MetaRoCE 将开放多厂商理念扩展到传输层，这一理念来源于开放计算项目（OCP）的以太网可扩展统一网络（ESUN）：https://engineering.fb.com/2025/10/13/data-infrastructure/ocp-summit-2025-the-open-future-of-networking-hardware-for-ai/ 为底层网络建立的倡议。

通过 OCP 的开放规范：完整的协议规范已贡献给 OCP，任何供应商都可以实现并构建互操作硬件。

多种网卡实现：MetaRoCE 被设计为可在多种网卡架构上运行——无论是可编程还是固定功能的。我们已在 AMD Pensando 硬件上验证了其运行，还在进行其他厂商的实现。

生产合规套件：我们开发了一套合规套件，为硬件厂商提供工具，以证明其实现符合协议规范。

软件参考实现：我们的 libsoftmetaroce 库提供了完整的、可运行的传输栈，可在标准的 Linux UDP 套接字上运行，无需专用硬件。它作为硅片开发的权威行为模型，也是我们统一合规框架的基础。

借助 MetaRoCE，我们在规模外网络方面取得了显著进展——在数据中心内基于普通以太网实现高性能、弹性传输。但 AI 基础设施跨越多种距离和延迟环境，每种环境都带来我们正在积极解决的不同挑战：

规模向上扩展：在机架内，加速器交换小消息，每一纳秒都至关重要。MetaRoCE 消除了两大主要延迟来源：重排序缓冲区和 PFC。我们目前正在优化针对从一个处理单元直接发起到另一个处理单元的短内存操作的快速信号路径。

跨尺度：跨尺度功能使单个作业能够跨越数千公里的建筑。往返时间延伸到毫秒级，路径之间的小差异会累积起来。将路径视为一等公民正是让MetaRoCE能够自适应的原因，它偏好未拥塞的路径，并在每个层级追求公平。未来的工作是在公平共享受争用的远程长途链接方面。

存储/Kv缓存用例：分布式存储容易引发汇聚拥塞，即一次读取请求会分发到多个服务器，它们同时回复。接收方驱动的速率提示允许接收端（处理写入的存储服务器或处理读取的客户端）调节入站速率，无论请求是发送到十台服务器还是一千台服务器。新的维度是如何在速度不一的网络和各种大小请求中保持速率的准确性。

在十月，我们将在2026 OCP全球峰会上发布MetaRoCE规范、基于DPDK优化的软件参考实现以及我们的生产合规框架：https://www.opencompute.org/summit/global-summit?ref=engineeringatmeta。

我们在开放环境中进行构建，因为未来的挑战需要广泛的行业协作。如果你正在开发网卡、交换机或AI基础设施，我们欢迎你加入我们。

Meta相信通过开源技术建立社区。探索我们在人工智能、数据基础设施、开发工具、前端、编程语言、平台、安全、虚拟现实等领域的最新项目。

Meta的工程技术是为对我们如何解决大规模技术挑战感兴趣的工程师提供的技术新闻资源。

At Meta, we’ve been a strong driver behind the industry’s growing consensus that Ethernet should be the fabric of choice for AI infrastructure. We’ve already shown that RoCE can power distributed AI training at scale ：https://engineering.fb.com/2024/08/05/data-center-engineering/roce-network-distributed-ai-training-at-scale/ . Now, we’re building on that work with MetaRoCE, protocol designed from the ground up for Ethernet at million-GPU scale.

We’ve scaled up clusters of hundreds of thousands of GPUs ：https://engineering.fb.com/2025/09/29/data-infrastructure/metas-infrastructure-evolution-and-the-advent-of-ai/ , spread over multiple data centers and regions. Whether these clusters are training the next frontier model or serving inference at global scale, the network is in the critical path.

Collective operations like all-reduce and all-to-all synchronize thousands of accelerators during training, and the slowest transfer sets the pace for the entire job. In inference, low-latency communication between distributed model shards directly impacts response times for hundreds of millions of users. Even small amounts of network friction directly strand significant compute capacity.

Standard RoCE expects the network to deliver every frame in order, leveraging PFC and discouraging the packet spraying that provides performance in multiplane and large scale networks. MetaRoCE is built to provide high throughput, low tail latency, and operational simplicity as the network grows in the number of accelerators and the distances between them.

MetaRoCE’s core insight is simple: The fabric sees packets, but the NIC sees intent. Traditional architectures centralize intelligence in the fabric, relying on switches to enforce losslessness and maintain order.

By moving intelligence to the endpoint MetaRoCE decomposes the network into many fine-grained logical paths, each with its own real-time telemetry – per-path RTT, ECN state, and utilization. This visibility unlocks capabilities that are difficult to achieve with traditional RDMA.

MetaRoce sprays packets across many paths, so they arrive out of order by design. The transport treats out-of-order arrival as the normal case. Every packet carries its own destination, so data is written straight to its final memory location as it lands, with no reorder buffer and no head-of-line blocking.

Writes carry their destination in every packet. Sends carry the match to a posted receive buffer, so a Send lands correctly even when the messages ahead of it have not arrived, and without a round trip to learn where the data goes. Collective libraries can use two-sided messaging where it suits them rather than reducing everything to Write.

MetaRoCE gives each connection first class paths and sprays across them packet by packet. Each path carries a distinct UDP source port as its ECMP entropy, which the NIC can change at any time to move traffic off a bad route. On multiplane fabrics, plane selection falls entirely to the NIC, and the fabric is used only as well as the NIC sprays. Because each path keeps its own window and round trip estimate, the transport can tell congestion from failure and rebalance explicitly, so a hot or broken link slows one path instead of stalling the connection.

MetaRoCE treats the Ethernet fabric as lossy and does not ask it to be otherwise – no PFC, no pause frames. Because each path carries its own ordered sequence, a gap in its 256-bit selective acknowledgment bitvector is evidence of loss rather than of reordering. In other protocols a SACK mostly avoids resending data that already arrived; here it triggers retransmission of exactly the missing packet, on the path that lost it, the moment the gap appears.

MetaRoCE combines a conventional ECN-based, sender-driven AIMD congestion control with receiver-driven fair-share rate hints. Windows are kept per path as well as per connection, so a congestion mark trims the path that saw it and steers the next packets toward paths that are clear. In every acknowledgment, the receiver returns the share of its inbound bandwidth it has allocated to that sender, so senders approach the right speed directly rather than searching for it. Incast resolves in one or two round trips, with better fairness and lower tail latency.

MetaRoCE asks the fabric for two things every switch already has, ECN marking and ECMP. It does not require packet trimming, in-network telemetry, credit-based flow control, or switch-side spraying, and it does not break when a fabric offers them. The same transport runs over fat-tree, multiplane, deep-buffer, and shallow-buffer fabrics, and over vendor clouds whose configuration you don’t control. Nothing proprietary is involved, so the fabric stays free to optimize for cost and cabling.

A queue pair (QP) carries both an ordered stream of messages and bandwidth. Traditional RDMA gets more of either by opening more QPs (dozens per node pair), each with a congestion window blind to the rest and its own state on the NIC.

MetaRoCE separates the two. A single connection carries many independent ordered streams above, one per communicator or collective, and many paths below, under one congestion controller. The connection state stops growing with the parallelism of the workload.

The application layer remains mostly untouched – existing RDMA Verbs APIs and software stacks work without modification. Enhanced features like multiplane support are supported through extension APIs.

To accelerate hardware validation, we worked with AMD to implement MetaRoCE on their Pensando programmable NICs.

On a 64-node AMD GPU cluster running RCCL collectives, we directly compared MetaRoCE against RoCEv2 across all-reduce and all-to-all operations. The results were consistent with the design goals:

MetaRoCE consistently delivers higher throughput and lower flow completion times than RoCEv2.

Under packet loss conditions that would degrade RoCEv2, MetaRoCE maintains ~86% throughput at 1% packet loss and continues delivering useful bandwidth even at extreme 10% loss rates – converging gracefully rather than collapsing.

Multiplane validation across 4-plane and 8-plane topologies with up to 4,000 concurrent connections confirmed that throughput scales linearly with plane count.

During simulated plane failures, the protocol demonstrates graceful autonomous recovery – traffic redistributes without application involvement or operator intervention.

These results support MetaRoce’s core design choice. By designing for loss from day one and pushing intelligence to the edge, you get a transport that performs better in ideal conditions and degrades gracefully when things go wrong.

AI infrastructure benefits from shared standards that accelerate innovation across the ecosystem. MetaRoCE extends the same open, multi-vendor philosophy that the Open Compute Project (OCP’s) Ethernet Scalable Unified Network (ESUN) ：https://engineering.fb.com/2025/10/13/data-infrastructure/ocp-summit-2025-the-open-future-of-networking-hardware-for-ai/ initiative established for the fabric into the transport layer.

Open specification via OCP: The full protocol spec is being contributed to OCP, available for any vendor to implement and build interoperable hardware.

Multiple NIC implementations: MetaRoCE is designed to run across diverse NIC architectures – programmable and fixed-function alike. We’ve proven it on AMD Pensando hardware, with additional implementations underway from other vendors.

Production compliance suite: We have developed a compliance suite that gives hardware vendors the tools to prove their implementations match the protocol spec.

Software reference implementation: Our libsoftmetaroce library provides a complete, functional transport stack that runs on commodity Linux over standard UDP sockets without specialized hardware. It serves as the authoritative behavioral model for silicon development and the foundation of our unified compliance framework.

With MetaRoCE, we’ve made strong progress on scale-out networking – high-performance, resilient transport within the data center on commodity Ethernet. But AI infrastructure spans multiple distance and latency regimes, and each brings distinct challenges we’re actively working on:

Scale-up : Within a rack, accelerators trade small messages where every nanosecond matters. MetaRoCE removes two main sources of latency, the reorder buffer and PFC. We are now optimizing the fast signaling path for short memory operations issued directly from one processing element to another.

Scale-across : Scale-across enables a single job to span buildings thousands of kilometers apart. Round trips stretch into milliseconds, and small differences between paths add up. Treating paths as first class entities is what lets MetaRoCE adapt, preferring the uncongested ones and seeking fairness at every level. The work ahead is in fairly sharing contended long-haul links.

Storage/Kv-cache use cases : Distributed storage invites incast,where a single read fans out to many servers and they all reply at once. Receiver-driven rate hints let whichever side is receiving(a storage server taking writes or a client taking reads) moderate the inbound rate, whether the request went to ten servers or a thousand. The new dimension is keeping that rate accurate with networks of varying speed and requests of varying size.

In October, we’ll release the MetaRoCE specification, a DPDK-optimized software reference implementation, and our production compliance framework at the 2026 OCP Global Summit ：https://www.opencompute.org/summit/global-summit?ref=engineeringatmeta .

We’re building this in the open because the challenges ahead benefit from broad industry collaboration. If you’re building NICs, switches, or AI infrastructure, we invite you to join us.

Meta believes in building community through open source technology. Explore our latest projects in Artificial Intelligence, Data Infrastructure, Development Tools, Front End, Languages, Platforms, Security, Virtual Reality, and more.

Engineering at Meta is a technical news resource for engineers interested in how we solve large-scale technical challenges at Meta.