Cadence and Intel Demonstrate CXL 1.1/2.0 Interoperability

This weekend, timed to coincide with SC21, Cadence and Intel published a white paper on CXL interoperability (interop) testing titled Interop Shift Left: Using Pre-Silicon Simulation for Emerging Standards. The SC in SC21 stands for SuperComputing, although the conference calls itself 'The International Conference for High Performance Computing, Networking, Storage, and Analysis". I will provide a link to the white paper at the end (because I don't want you to go away yet and not come back!).

In 2016, I wrote about a demonstration of interoperability between Mellanox's PCIe 4.0 and Cadence's 16Gbps multi-link and multi-protocol PHY IP implemented in 16nm. Mellanox's Gilad Shainer said to me when I talked to him, "interoperability is the only way to prove standards compliance." I used his quote for the title of my blog post. Well, a lot has changed since then. For a start, NVIDIA acquired Mellanox. PCIe has advanced to version 5 (with 6 waiting in the wings). 16Gbps is now 112G. A 16nm process is now a 5nm process. But it remains the case that interoperability is the only way to prove standards compliance. In fact, more than that, it is the only way to prove that the standard itself is solid.

CXL

First, a little background on CXL, which stands for Compute eXpress Link. This was originally developed by Intel who then made it an open standard as CXL 1.0. That was revised to a 1.1 version. The new version, CXL 2.0, was developed by the CXL Consortium. Both CXL 1.0 and 2.0 are built on top of PCIe 5.0. There are still almost no components available with CXL 1.1, although it is expected that there will be memories, processors, FPGAs, and more. But version 2.0 is too new to have any of silicon available yet. Cadence has designed controller IP for CXL 2.0 and has a test chip fabricated (in 5nm) and just arrived back in our lab. This lack of product availability shows one of the challenges of developing products against new emerging standards.

Obviously, if there are issues with the standard, you want to discover them before you design silicon or IP. If there is a problem with the design, you want to find it before tapEout, or for IP, before release. To address both of these, the solution is to "shift left" and start the verification and interop testing earlier. As it says in the white paper:

To keep up with the torrid pace of design innovations, there is a fundamental change in strategy to massively shift-left the verification of new CXL devices with the host IP.  Intel and Cadence are working together on interoperability through co-simulation as the first proof point to successfully run complex cache-coherent flows by first training the link using alternate protocol negotiation. The CXL simulation interop demonstrates the ability to confidently build host and device IP, while also providing essential feedback to the CXL standards body.
...
For advanced silicon geometries, such as 7nm, 5nm, Intel 3, and Intel 20A, the development time from specification to architecture to implementation to verification to testchip tapeout to silicon characterization could exceed a year. Silicon design costs for an initial design in 5nm compared to 7nm were projected to be 31% higher in 2019; software costs for advanced SoCs through the Intel 20A node era were forecasted to grow at 51% CAGR.

As I said already, Cadence developed Controller IP for CXL 2.0 Controller. A major challenge in developing this IP while the specification was still evolving in 2020 was the lack of CXL 2.0 host platform availability for interop testing. The solution was to partner with Intel for simulation interop testing with the future processor designs that implement support for CXL. Through simulation, Intel could ensure backward compatibility against its existing CXL 1.1 solutions. In parallel, they could perform interop simulations on the evolving CXL 2.0 standard. This also benefited the CXL Consortium to have early adopters interoperating, with an opportunity to provide feedback on the specification while it was still in development. The standard was finally released on November 10th 2020, almost exactly a year ago.

Simulation

The Cadence CXL Endpoint RTL dropped straight into Intel’s RTL simulation environment, replacing their Endpoint bus functional model (BFM). See the diagram above.  A joint test plan was established to determine the area of testing, test strategy, and metrics to determine success. Test cases and traffic patterns included:

• CXL 1.1/2.0 link training
• CXL.io 1.1/2.0 discovery
• CXL.mem, CXL.cache, and mixed CXL.mem/CXL.cache traffic
• Traffic initiated by the host or Endpoint

The white paper obviously goes into a lot more detail on the challenges of executing this plan than is appropriate in a blog post like this.

The next step was to "kick it up a notch" as Emeril Lagasse says on his cooking programs.  Building on the initial 16-lane 32 GT/s CXL link-up in CXL mode, the next step was to enable application capability. We replaced the BFMs with a reference CXL EP application layer, supporting a small memory application. This RTL-based application layer models a CXL Type3 device memory application. Additionally, the application layer was the implementation of DVSEC, RCRB, and port registers that typically lie outside an IP implementation. In this way, the CXL 1.1 host could discover and enumerate the application using a standard enumeration with real software sequences, such as those used in host platform BIOS firmware.

The big advantages to this simulation-based approach were:

• Designers can count on the full debug capabilities of a simulator, unlike silicon or FPGA environments. Tools such as RTL waveforms, assertions, coverage, and logfiles are easily available and facilitate problem resolution.
• RTL simulation is not limited by FPGA clocks speeds. We can simulate at PCIe 5.0 x16 speeds in simulation, which is not possible in current FPGA implementations without making design tradeoffs that would make the implementation very different from an SoC version.

Benefits to the Wider Community

Obviously, the primary beneficiaries of this interop testing were Cadence and Intel themselves. But there are benefits to the wider CXL community. Examples include:

• Generation of errata to evolving CXL standard, particularly where the standard was open to interpretation. For example, when CXL.io-only mode was negotiated, it may have been interpreted as bypassing ARBMUX.
• Interop work performed by adopters allows new and evolving sections of CXL standard to be pipe cleaned and confirmed before being fully published, such as combinations of ARB/MUX being exposed to recovery during initial link training.
• Synchronization on CXL 2.0 errata between vendors, where small clarifications had been added to the standard and missed by one vendor.
• Opportunity to learn from alternative interpretations of a standard, and support as an alternate mode to increase interop robustness. For example, a requirement to retry after recovery, even during initial link training.

This interop verification was a pilot program and it exceeded our expectations.  We intend to pursue similar interop projects for future additions and revisions to standards as well as new standards that come up!

Watch a Demo

Download the White Paper

The white paper is formally Interop Shift Left: Using Pre-Silicon Simulation for Emerging Standards: A CXL Case Study by Martin James, Gary Dick, and Arif Khan (Cadence) and Suhas Pai and Brian Rea (Intel).

Sign up for Sunday Brunch, the weekly Breakfast Bytes email.

.