Abstract
The shift toward large-scale AI training on GPU clusters has made RDMA over Converged Ethernet version 2 (RoCEv2) a critical infrastructure concern. Unlike InfiniBand, RoCEv2 runs over standard Ethernet switches but inherits RDMA’s zero-copy, kernel-bypass characteristics, pushing practical throughput for all-reduce operations close to line rate on 400GbE links. The tradeoff is that RoCEv2 has no built-in congestion control; it relies entirely on lossless Ethernet mechanisms and explicit congestion notification (ECN) to avoid queue buildup that can cause devastating retransmissions during large gradient synchronization steps.
Lossless Fabric Requirements
RoCEv2 requires a lossless Ethernet fabric. Priority Flow Control (PFC, IEEE 802.1Qbb) is the standard mechanism: a congested switch sends a PAUSE frame back to the upstream sender, halting traffic on specific priority lanes. In practice, PFC introduces head-of-line blocking and can propagate back-pressure across multiple hops, triggering PFC storms that stall entire fabric segments. The operational response from hyperscalers - Meta and Microsoft among them - has been to treat PFC as a last resort, relying primarily on Explicit Congestion Notification (ECN) with DCQCN (Data Center Quantized Congestion Notification) in the NICs to throttle senders before buffers fill. DCQCN, which combines RED-like marking at the switch with DCTCP-style rate reduction in the sender NIC, is implemented in Mellanox ConnectX NICs and is the dominant congestion control algorithm in production RoCEv2 clusters as of early 2026.
Key Tuning Parameters
Getting RoCEv2 right is a per-cluster exercise. The most impactful levers are: ECN marking thresholds (Kmin/Kmax) on switch egress queues, which must be calibrated to the traffic pattern - all-reduce has a very different burst profile than point-to-point parameter server traffic; the DCQCN alpha decay and beta increment rates in the NIC firmware; and the PFC watchdog timer, which kills PFC-paused ports that have been stalled beyond a threshold to break deadlocks. Getting the ECN thresholds wrong in either direction - too aggressive and you throttle unnecessarily, too conservative and you allow buffer overflow and retransmits - can cost 20-40% of effective all-reduce throughput on large collective operations. Tuning guides from Mellanox (now part of NVIDIA) and from the ROCEv2 working group within the Ultra Ethernet Consortium provide starting points, but they require adaptation to specific switch ASIC buffer configurations.
The Ultra Ethernet Consortium Direction
The Ultra Ethernet Consortium (UEC), launched in 2023 with backing from AMD, Intel, Microsoft, and others, is developing a transport-layer specification aimed at replacing both InfiniBand and RoCEv2 for AI workloads. UEC defines new congestion control mechanisms and multipath support at the transport layer, without relying on PFC. As of early 2026, the UEC specification is at draft stage, and deployment is still years out for most operators. In the interim, RoCEv2 with DCQCN and careful PFC configuration remains the standard path for GPU cluster networking, particularly as 400GbE and 800GbE switch deployments accelerate.