OpenAI MRC Multipath Reliable Connection Protocol

OpenAI has introduced Multipath Reliable Connection (MRC), a novel networking protocol designed to eliminate congestion and minimize the impact of hardware failures in supercomputer clusters used for frontier AI model training. Developed in partnership with AMD, Broadcom, Intel, Microsoft, and NVIDIA, MRC is now available as an Open Compute Project (OCP) specification to standardize high-performance AI networking.

Solving Network Bottlenecks in Synchronous AI Training

Synchronous pretraining requires thousands of GPUs to operate in lockstep; consequently, a single delayed data transfer caused by network congestion or hardware failure can stall the entire training job. As clusters scale toward the size of OpenAI's Stargate infrastructure, these "failure amplifiers" become more frequent and disruptive.

Traditional networking faces two primary challenges at this scale:

• Congestion: Unavoidable bottlenecks occur when multiple GPUs send data to the same destination simultaneously, but general network congestion often stems from inefficient routing.
• Failure Impact: In conventional networks, a single link or switch failure often crashes the training job, requiring a restart from a checkpoint or causing multi-second stalls while the network recomputes routes.

The MRC Architecture: Multi-Plane Networks and Packet Spraying

MRC extends RDMA over Converged Ethernet (RoCE) and incorporates techniques from the Ultra Ethernet Consortium (UEC) and SRv6-based source routing to create a more predictable networking fabric.

Multi-Plane Topology

Instead of using a single 800Gb/s link per network interface, MRC splits the interface into multiple smaller links (e.g., eight 100Gb/s links) connecting to separate parallel networks, or "planes."

This architectural shift allows OpenAI to connect over 100,000 GPUs using only two tiers of switches, whereas a conventional 800Gb/s network would require three or four tiers. This reduction in tiers lowers power consumption, reduces the number of components prone to failure, and decreases total network cost.

Adaptive Packet Spraying

To utilize the redundancy of multi-plane networks, MRC replaces single-path flows with packet spraying. Rather than assigning a transfer to one path, MRC sprays packets from a single transfer across hundreds of paths across all distinct planes.

• Out-of-Order Delivery: Packets include their final memory address, allowing the destination to deliver them to memory as they arrive, regardless of order.
• Load Balancing: MRC monitors paths for congestion and dynamically swaps congested paths for available ones to even out network load.
• Failure Recovery: If a packet is lost, MRC immediately stops using that path and retransmits the data. It then uses probe packets to determine if the path has recovered.
• Packet Trimming: To distinguish between path failure and destination congestion, switches can "trim" a packet—removing the payload but forwarding the header—to trigger an explicit retransmission request without marking the path as failed.

Simplifying Control Planes with SRv6 Source Routing

MRC eliminates the complexity of dynamic routing protocols like BGP (Border Gateway Protocol), which can be difficult to diagnose when they fail subtly.

By using IPv6 Segment Routing (SRv6), the sender encodes the full path each packet should take directly into the destination address as a sequence of switch identifiers. Switches simply follow a static routing table configured at setup, which never changes. If a path fails, the sender independently detects the loss and stops using that path, removing the need for switches to recompute routes.

Production Performance and Resilience

MRC is currently deployed in OpenAI's largest NVIDIA GB200 supercomputers, including sites with Oracle Cloud Infrastructure (OCI) in Abilene, Texas, and Microsoft's Fairwater supercomputers.

OpenAI reports the following production outcomes:

• Zero Impact from Link Flaps: Multiple link flaps per minute between tier-0 and tier-1 switches have had no measurable impact on synchronous pretraining jobs.
• Seamless Hardware Maintenance: Tier-1 switches can be rebooted or links repaired while in service without coordinating with training teams, as MRC automatically routes around the affected hardware.
• Graceful Degradation: If a GPU interface loses one of its eight ports, the training job survives. MRC recalculates paths to avoid the failed plane and notifies peers to stop using that plane for inbound traffic, resulting in a performance hit significantly lower than the physical capacity loss.

Summary of Key Technical Advantages

Feature	Impact on AI Training
Multi-Plane Design	Connects 100k+ GPUs with only 2 switch tiers, reducing power and cost.
Packet Spraying	Eliminates core congestion and reduces throughput variation between flows.
SRv6 Source Routing	Replaces complex dynamic routing with a static control plane, bypassing failures in microseconds.