Abstract

CXL (Compute Express Link) 3.x introduced fabric switching and multi-host memory sharing, making rack-scale memory pooling architecturally feasible for the first time without specialized proprietary interconnects. The motivation for AI inference workloads is straightforward: large language model weights frequently exceed what a single host’s DRAM can hold, and pooled CXL-attached memory can present a larger flat address space to a GPU or AI accelerator via the host CPU. What the architectural papers underemphasize is the latency regime that engineers actually encounter, and how that latency interacts with the memory access patterns of transformer inference specifically.

CXL Latency vs. Local DRAM - the Numbers That Matter

Local DDR5-6400 DRAM latency from a processor’s perspective is roughly 70-90 nanoseconds end-to-end. A CXL 2.0 Type 3 memory expander attached via PCIe Gen 5 x16 adds approximately 100-150 nanoseconds of one-way latency attributable to the CXL.mem protocol overhead and the PCIe physical layer. CXL 3.0 with fabric switching introduces at least one hop through a CXL switch, adding another 50-100 nanoseconds per hop. For workloads that are bandwidth-bound - bulk matrix-vector multiply in the prefill phase of transformer inference, for example - this added latency is tolerable provided bandwidth is sufficient. For workloads that are latency-sensitive on a per-access basis, the overhead is more significant. The decode phase of autoregressive generation, which sequentially reads KV-cache entries for each generated token, falls into the latency-sensitive category and sees real degradation with pooled CXL memory compared to local DRAM.

NUMA Effects and Software Complexity

CXL pooled memory appears to the OS as a NUMA (Non-Uniform Memory Access) node with a penalty factor that is substantially larger than the CPU-to-CPU NUMA latency familiar from multi-socket servers. Linux’s NUMA balancing heuristics were designed for latency ratios of roughly 1.5-2x between local and remote; CXL pooled memory at two hops can present ratios of 3-5x. This means workloads that rely on OS NUMA placement - including most frameworks that call mmap and let the kernel handle placement - will not automatically route hot data to local memory. Engineers deploying pooled CXL for inference need to use explicit mbind calls or memory allocation libraries that are CXL-topology aware. Samsung’s CXL memory modules shipping in late 2025 include a companion userspace library for topology discovery, and the Linux kernel’s DAMON (Data Access Monitor) subsystem has been extended to generate placement hints for CXL tiers, but neither mechanism eliminates the need for application-level attention.

Where Pooling Delivers and Where It Does Not

The cases where rack-scale CXL pooling genuinely helps are: hosting multiple model replicas across hosts that share a pooled weight store (amortizing DRAM cost), disaggregated prefill-decode separation where the prefill host’s KV-cache is written to pooled memory for the decode host to read, and near-memory compute scenarios where an active CXL device with an embedded processor handles reduction operations locally. The cases where pooling adds cost without benefit are: single-host serving of models that fit in local DRAM, and latency-critical decode paths where per-token generation latency is a user-facing SLO. The rack-scale CXL story for inference in 2026 is real but narrow - it solves a cost and capacity problem for specific deployment shapes, not a general acceleration problem.