Abstract

Disaggregated memory systems - where DRAM capacity is accessed over a fabric rather than soldered to a host processor’s memory bus - break the assumptions baked into decades of cache coherence protocol design. Classical protocols such as MESI and MOESI assume that coherence traffic stays within a tightly coupled domain where round-trip latencies are measured in tens of nanoseconds. When memory is tens to hundreds of nanoseconds away over CXL or a proprietary fabric, the cost of an invalidation round-trip changes the protocol tradeoff space fundamentally. This article surveys how architects are adapting coherence for disaggregated topologies and where the open problems remain.

Why Classical Coherence Breaks Down at Fabric Latency

MESI coherence works well when the worst-case snoop response latency is a few dozen nanoseconds - roughly the round-trip across a dual-socket NUMA link. At fabric latencies of 150-500 nanoseconds, a write that must first invalidate remote sharers and wait for acknowledgement before proceeding will stall the issuing core for a substantial fraction of a microsecond per store, compounding badly under write-heavy workloads. Snoopy broadcast schemes are even worse: broadcasting an invalidation to every potential sharer on a large fabric saturates the interconnect at a fraction of the theoretical bandwidth. Directory protocols scale better but introduce their own latency overhead at the directory lookup step, and directory state for a disaggregated pool spanning terabytes requires either enormous directory storage or lossy compression schemes that cause unnecessary invalidations.

CXL.cache and the G-FAM Coherence Model

CXL 3.0 introduced Global Fabric Attached Memory (G-FAM), a coherence model that acknowledges fabric latency by relaxing the requirement that a processor’s caches remain fully coherent with remote memory at all times. Instead, G-FAM designates memory regions as either cache-coherent (with the full MESI overhead) or non-coherent (mapped as device memory, managed by software). The device allows applications to choose their coherence tier per allocation, pushing the protocol burden to the software stack for performance-critical paths. This approach is pragmatic but puts correctness responsibility on the programmer or runtime, which has historically been a source of subtle bugs in heterogeneous systems.

Lazy Invalidation and Speculative Forwarding

Several research prototypes and at least one shipping product (AMD’s MI300A APU, in its unified memory mode) have explored lazy invalidation, where a write is permitted to complete locally and the invalidation of remote copies is deferred until the next time a remote agent actually requests that line. Combined with speculative forwarding - where a directory predicts that a requesting node is likely to be the next writer and sends the line directly without waiting for all current sharers to acknowledge - the effective write latency under low contention can approach that of a coherent NUMA link. The catch is that under high contention the speculative forward can be wrong, requiring a recall and adding latency beyond what a conservative protocol would have incurred. Intel’s research on Scalable Memory Interconnect has published simulation results showing that hybrid lazy/speculative protocols outperform strict MESI by 18-30 percent on memory-bandwidth-bound HPC workloads at 256-node scale.

Software-Managed Coherence Regions

An alternative to hardware coherence is to expose disaggregated memory as explicitly non-coherent and provide software primitives - flush, invalidate, fence - that let runtimes enforce consistency at a coarser granularity. This is the approach taken by RDMA-based disaggregated memory systems such as Leap and FarMemory, and it is also how most GPU unified memory frameworks operate today. The cost is programming complexity: every region of memory that might be accessed from multiple nodes requires explicit lifecycle management. The benefit is that the common case - a memory region owned exclusively by one host at a time - incurs no coherence overhead at all. For cloud workloads where memory disaggregation is used primarily for capacity (swapping cold data to a remote pool) rather than sharing, this tradeoff is often favorable.

Open Problems

The field still lacks a widely agreed protocol for mid-latency fabrics in the 50-200 nanosecond range, where neither the tight-coupling assumptions of MESI nor the fully software-managed model of RDMA fits cleanly. Research groups at MIT CSAIL, Carnegie Mellon, and EPFL are each pursuing variants of hierarchical directory schemes that partition coherence domains to keep most traffic within a low-latency local cluster while still presenting a unified address space to applications. Whether any of these will be adopted by CXL 4.0 or a successor standard remains to be seen.