Abstract

As processor core counts scale past 64 on a single die, the network-on-chip (NoC) connecting them to caches, memory controllers, and I/O interfaces has become a dominant contributor to both die area and unloaded memory access latency. The mesh topology that served well at 16-32 core counts begins to show its limits - hop count grows with the square root of core count, and the wiring needed to implement low-diameter alternatives such as flattened butterflies or dragonfly topologies consumes routing layers that competing with power and signal integrity. This article examines the topology tradeoffs being made in current many-core designs and the microarchitectural innovations accompanying them.

Mesh Dominance and Its Limits

The 2D mesh remains the default NoC topology for most commercial high-core-count processors. Its appeal is regularity: every router is identical, wires connect only to immediate neighbors, and the layout tiles naturally with the compute and cache arrays it interconnects. Intel’s Xeon Sapphire Rapids uses a 2D mesh across its tile array, as does AMD’s EPYC with its inter-CCD Infinity Fabric. The penalty is hop count: a 10x10 mesh (100 tiles) has a worst-case diameter of 18 hops, each adding 2-4 nanoseconds of latency depending on router pipeline depth and wire delay. For remote LLC accesses and coherence traffic, the accumulated hop latency pushes the effective NUMA distance between tiles on opposite ends of the die above 50 nanoseconds - comparable to a first-generation HyperTransport socket-to-socket link. This is still acceptable for bandwidth-bound workloads but creates visible latency tails in pointer-chasing and lock-acquisition workloads.

Hierarchical and Concentrated Mesh Variants

One practical response to mesh diameter growth is hierarchy: cluster tiles into groups of 4-8, connect clusters with a higher-bandwidth global ring or mesh, and use local meshes within each cluster. This is essentially the approach Intel adopted with the ring-bus in Sandy Bridge through Skylake server parts, and the transition to a 2D mesh in Skylake-SP reflected the limits of a ring at 28 cores. A further refinement is the concentrated mesh (CMesh), where multiple compute tiles share a single router - reducing router count and die area at the cost of router port count. The TRIPS processor from the University of Texas, and more recently research implementations from MIT and Stanford, demonstrated that a 4:1 concentration ratio (four compute tiles per router) is close to the sweet spot for minimizing area while keeping injection bandwidth adequate for typical workload mixes.

Flattened Butterfly and Express Channels

For core counts above 64, low-diameter topologies become attractive. The flattened butterfly (FBfly) eliminates intermediate hops by providing direct connections between every pair of routers in a row and every pair in a column - a k-ary 2-fly. The diameter is always 2, regardless of network size, at the cost of router radix proportional to the square root of node count. For a 256-core mesh equivalent, each FBfly router needs 32 output ports, which is feasible but expensive in die area. A compromise popular in research is adding a small number of express channels to an otherwise standard mesh: long wires crossing multiple hops in a single cycle, switched in or out by routers at each end. Simulations from groups at Georgia Tech and the University of Wisconsin have shown that as few as 16 express channels on a 256-node mesh can reduce average hop count by 30-40 percent with area overhead under 5 percent, provided traffic patterns are not adversarial to the placement of those channels.

Traffic Pattern Awareness in Routing

Even within a fixed topology, routing algorithm choice significantly affects observed latency and throughput. Adaptive routing - where each router makes a per-flit forwarding decision based on current queue depths - can improve throughput under unbalanced load by 20-40 percent compared to deterministic dimension-order routing, but adds per-router complexity and can cause packet reordering that complicates cache coherence implementations. Apple’s M-series chips use a variant of adaptive routing with flow-level ordering guarantees, ensuring that messages within a coherence transaction are never reordered regardless of route variation. The implementation details are proprietary, but the approach is consistent with published techniques using virtual channel segregation per transaction type to maintain ordering without sacrificing adaptivity.

Looking Ahead

The industry trend toward chiplet disaggregation is partly a response to NoC scaling limits: rather than building a 256-core monolithic die with its associated NoC complexity, split the design into smaller tiles connected by a package-level fabric (UCIe, or vendor-specific equivalents). The package-level fabric operates at lower frequency but higher bandwidth per link than an on-die NoC wire, and latency is determined by serialization and framing overhead rather than hop count. This effectively exports the topology design problem from silicon to packaging, where per-connection cost is lower and the topology can be reconfigured by changing the interposer or package routing rather than respinning the die. Whether this path fully replaces on-die NoC scaling or complements it will depend on how package-level interconnect latency improves over the next two to three technology generations.