Abstract

CXL 2.0 and 3.0 introduce two distinct memory access protocols: CXL.cache, which provides host-managed cache coherence between the processor and a CXL device, and CXL.mem, which exposes device-attached memory as a non-coherent or coherent memory region depending on configuration. The interaction between these protocols and software-visible coherence models is subtle and consequential, particularly for systems that use CXL-attached DRAM as a persistent or semi-persistent memory tier alongside volatile local DRAM. This post examines the coherence semantics that system architects need to understand when mixing these memory regions in production deployments.

CXL.mem Coherence Modes and Their Implications

CXL.mem defines three coherence modes for attached memory: HDM-H (Host-managed Device Memory with host coherence), HDM-D (device-managed, essentially non-coherent from the CPU’s perspective), and HDM-DB (device-managed with Back-invalidation, where the device can request invalidation of CPU cache lines holding data from the device memory). The mode in use determines whether the processor’s cache hierarchy maintains coherence for accesses to CXL-attached memory regions or whether software must issue explicit flushes. HDM-H is the safest from a software correctness standpoint - accesses to CXL memory go through the processor’s coherence protocol and cached copies are automatically invalidated on conflict. The cost is bandwidth: every access requires the CXL link to participate in MESI (Modified-Exclusive-Shared-Invalid) state transitions, which adds latency overhead on top of the physical link penalty. HDM-D provides higher effective bandwidth for workloads that do large sequential reads or writes without fine-grained sharing, but requires software to manage coherence explicitly using clwb (cache line write-back) and sfence instructions before handing data to another agent.

Persistent Tier Use Cases and Correctness Pitfalls

The persistent memory use case - using CXL-attached DRAM with battery-backed or NVDIMM-style persistence - amplifies the coherence complexity because durability and coherence interact. A write to CXL.mem in HDM-H mode that is absorbed into the processor’s L3 cache is not persistent until it is flushed to the CXL device. The clwb instruction writes back a dirty cache line to memory without invalidating it, and the subsequent sfence provides ordering guarantees, but neither is sufficient alone to guarantee the write has crossed the CXL link and been committed to the device’s storage cells. The CXL 3.0 specification defines a persistence domain concept that addresses this: a CXL_FLUSH operation can be issued to the device to synchronize the device’s internal write buffers, but this operation is not yet uniformly supported across device implementations, and software using persistent CXL tiers needs to check device capability registers at initialization rather than assuming it is available.

Software Stack Support Through Early 2026

The Linux kernel’s memory tiering work in the 6.7 and 6.8 kernels added explicit CXL topology awareness to the NUMA subsystem, exposing CXL-attached memory regions with correct performance ratings so that the kernel’s NUMA balancing can treat them appropriately. The ndctl and cxl-cli tools provide userspace access to CXL device capability registers, including persistence domain queries. For database workloads using pmem-aware storage engines (like RocksDB with its PmemEnv plugin or Intel’s PMDK-based applications), the coherence model differences between local NVDIMM and CXL-attached persistent memory require configuration changes - specifically, setting the flush mode to account for the additional CXL link in the persistence boundary. Teams migrating pmem workloads to CXL-attached storage in 2026 should treat this as a correctness audit item, not just a performance tuning exercise.