Controller Memory Buffer (CMB): NVMe 2.0’s On-Controller Memory Feature

Non-volatile Memory Express (NVMe) has become the dominant interface protocol for high-performance storage devices. As workloads demand ever-lower latencies, the NVMe specification has evolved with features that reduce data-path overhead. One such feature is the Controller Memory Buffer (CMB), which exposes on-controller memory directly to the host system.

NVMe and the Host-Controller Interaction

In traditional NVMe architecture, all command queues, data buffers, and pointer structures (PRPs and SGLs) reside in host memory. When the host submits a command, the controller must perform PCIe read transactions to fetch the submission queue entry. Additional reads may be needed to retrieve PRP Lists or SGL segments. Each of these operations traverses the PCIe fabric—potentially crossing multiple switches—introducing latency and consuming bandwidth. For latency-sensitive workloads such as real-time analytics and AI inference pipelines, these round-trip overheads become a significant bottleneck.

Why the Controller Memory Buffer Was Introduced

Instead of the controller pulling commands and metadata from host memory, the host directly pushes them into memory on the controller itself. This eliminates controller-initiated PCIe read transactions for command fetches and PRP/SGL list retrievals. The result is reduced latency, improved PCIe fabric efficiency, and better performance in multi-switch topologies.

Understanding the CMB Feature in NVMe 2.0

The Controller Memory Buffer is a region of general-purpose read/write memory physically located on the NVMe controller. The controller advertises CMB support through the CAP.CMBS capability bit (bit 57 of the Controller Capabilities register at offset 0h). Once the host indicates its intent to use the CMB by setting CMBMSC.CRE to ‘1’, the controller exposes CMB properties through the following controller properties:

• CMBLOC (offset 38h) – Controller Memory Buffer Location: Defines where the CMB resides within the controller’s PCI BAR space via the Base Indicator Register (BIR) field and offset (OFST). This property also contains capability bits that govern operational flexibility: CQPDS indicates whether queues can be physically discontiguous in CMB, CQMMS controls whether a queue’s memory can span CMB and host memory, CDPMLS allows PRP Lists or SGLs to be split across CMB and host memory, CDPCILS permits PRP/SGL placement in CMB even when the command resides outside CMB, CDMMMS enables data and metadata to span both memory regions, and CQDA indicates whether Dword-aligned queue addressing is required.
• CMBSZ (offset 3Ch) – Controller Memory Buffer Size: Defines the CMB size via the SZ and SZU fields, with granularity ranging from 4 KiB to 64 GiB. This property also indicates what the CMB can be used for through support flags:
  • Submission Queue Support (SQS) – Admin and I/O submission queues can reside in CMB
  • Completion Queue Support (CQS) – Completion queues can be placed in CMB
  • PRP/SGL List Support (LISTS) – Pointer lists can be stored in CMB
  • Read Data Support (RDS) – Data for controller-to-host transfers can use CMB
  • Write Data Support (WDS) – Data for host-to-controller transfers can use CMB
• CMBMSC (offset 50h) – Controller Memory Buffer Memory Space Control: Configures the CMB’s controller address range. The CRE bit signals intent to use CMB, CMSE enables controller memory space, and CBA specifies the controller base address. This property is preserved across controller reset and function level reset.
• CMBSTS (offset 58h) – Controller Memory Buffer Status: Reports the current status of the CMB, including whether the controller base address is valid.
• CMBEBS (offset 5Ch) – Controller Memory Buffer Elasticity Buffer Size: Identifies the size of the optional write elasticity buffer, with granularity ranging from bytes to 1 GiB. This buffer allows PCIe write bursts to exceed the CMB’s sustained write throughput without back-pressuring the PCIe fabric.
• CMBSWTP (offset 60h) – Controller Memory Buffer Sustained Write Throughput: Reports the maximum sustained write throughput of the CMB, enabling host software to make informed decisions about data placement.

An important ordering requirement ensures correctness: the host must guarantee all CMB writes for a command have completed before updating the SQ tail doorbell, and the doorbell write must not use relaxed ordering.

Applications of Controller Memory Buffer

• Low-latency command submission: Submission queues in CMB allow the host to write commands directly into controller memory, eliminating controller-initiated PCIe reads.
• PRP/SGL fetch elimination: Storing PRP lists and SGLs in CMB gives the controller immediate local access without separate fabric reads.
• Small write optimization: The host can push small write payloads directly into CMB, avoiding DMA read overhead.
• Multi-switch fabric efficiency: CMB-based host pushes reduce cross-fabric traffic in complex PCIe topologies.
• Peer-to-peer and virtualization: CQs in CMB enable device-to-device communication; CMBMSC persistence across resets supports VM-assigned controllers.

Verification Challenges

The CMB feature introduces significant verification complexity. Because queues, pointer lists, and data can now reside in either host memory or controller memory—or span both depending on CMBLOC capability bits—the verification environment must exercise a combinatorial matrix of placement configurations. Key challenges include:

• Address range validation – Ensuring host-supplied addresses correctly map to the CMB’s controller address range and that out-of-range accesses are directed elsewhere.
• Mixed-memory compliance – Verifying the controller enforces or relaxes restrictions based on CMBLOC bits (CQMMS, CDPMLS, CDPCILS, CDMMMS) and aborts commands with Invalid Use of Controller Memory Buffer when constraints are violated.
• Ordering and coherency – Confirming CMB writes complete before doorbell updates and that relaxed ordering is not used on doorbell writes.
• Reset persistence – Validating CMBMSC preservation across controller reset and function level reset while CMB contents are correctly undefined after these events.
• Discontiguous queue handling – Testing queue operations when CQPDS is cleared versus set, ensuring contiguity enforcement is applied correctly.

Cadence’s NVMe Verification IP addresses these challenges with built-in protocol checkers that automatically validate CMB address mapping, mixed-memory placement rules, ordering requirements, and reset behavior. The VIP’s configurable host and subsystem models support all CMB usage modes—including queue placement in CMB, PRP/SGL list storage, data transfers via CMB, and physically discontiguous queue configurations—enabling verification teams to systematically cover the full CMB compliance matrix defined in the NVMe 2.0 base specification. More information on Cadence NVMe Verification IP is available at Simulation VIP for NVMe | Cadence

Reach out to Cadence Verification IP experts at talk_to_vip_expert@cadence.com