Abstract
Remote Direct Memory Access over Converged Ethernet version 2 (RoCEv2) allows a NIC to read and write application memory on a remote host without involving either host’s CPU in the data path, using standard UDP/IP as the network layer. The result is microsecond-latency, high-throughput data transfer at CPU costs far below what TCP achieves, which is why RoCEv2 has become the dominant fabric interconnect for GPU clusters running distributed training workloads and for HPC storage (NVMe-oF over RoCE) as NVMe-over-TCP’s latency ceiling becomes the bottleneck. Understanding why RoCEv2 requires explicit congestion management - and how to configure it - is the prerequisite for any team planning a 100G or 400G cluster fabric.
Why Lossless Ethernet Matters
RoCEv2 transports RDMA over UDP, and the RDMA transport itself has minimal congestion control. Unlike TCP, which backs off when it detects loss, an RDMA queue pair that encounters a dropped packet either retransmits at the original rate or, worse, stalls waiting for a NAK that may be delayed by a congested return path. Even a small amount of packet loss - sub-0.1% - can collapse throughput dramatically on a high-bandwidth-delay product path. This is why RoCEv2 deployments require lossless Ethernet, implemented through Priority Flow Control (PFC, IEEE 802.1Qbb). PFC pauses a specific traffic class on a port-to-port basis when the receiving buffer reaches a threshold, propagating backpressure rather than dropping packets. The fabric must be engineered so that PFC pause frames do not propagate indefinitely and create deadlocks - a real failure mode in poorly designed fat-tree topologies.
ECN, DCQCN, and Congestion Management
Priority Flow Control alone is insufficient for stable operation at scale. PFC pause storms can originate from a single slow receiver and propagate across multiple hops, impacting unrelated flows. The solution is Explicit Congestion Notification (ECN) combined with an end-to-end congestion control protocol. DCQCN (Data Center Quantized Congestion Notification), developed by Microsoft and Mellanox (now NVIDIA) and described in a 2015 SIGCOMM paper, combines ECN marking at the switch with a rate-limiting algorithm at the sender that reduces injection rate when ECN marks are received, and recovers gradually when they cease. Most Mellanox/NVIDIA ConnectX NICs implement DCQCN in firmware; Broadcom NICs use a similar scheme called HPCC in some configurations. Tuning ECN thresholds on the switch and the DCQCN parameters on the NIC for a specific cluster topology is non-trivial and significantly affects both throughput and fairness between competing flows.
Fabric Design and Operational Monitoring
A RoCEv2 fabric for a GPU cluster is typically a two-tier fat-tree - leaf and spine - with equal-cost multipath (ECMP) load balancing. Getting ECMP right matters because RoCEv2 generates large sustained flows that can hash to the same spine links, creating hotspots. Adaptive routing (available on Arista 7800R, Juniper QFX10000, and NVIDIA Spectrum switches) monitors per-port load and dynamically reroutes elephant flows, flattening utilization significantly compared to static ECMP. Operational visibility requires per-queue ECN marking rates, PFC pause frame counts, and RDMA NIC counters (rx/tx completion rates, NAK counts) exported from both the NICs and the switches. NVIDIA’s DOCA telemetry stack and Mellanox WinOF-2 / OFED packages expose these via standard interfaces; integrating them into a Prometheus/Grafana stack is the baseline for meaningful fabric health monitoring.