Boost your CXL Verification From IP to System-Level

24 Feb 2022 • 6 minute read

Knowingly or unknowingly, we are consuming huge volumes of data from getting up early with Google/Siri, doing day-to-day work, using maps while driving till the time we call it a day and retiring in bed while watching Netflix. We expect all these to be secure, seamless with the least latency.

These high computation workloads of applications communication systems and high-performance computing require tighter connectivity between processors, accelerators, advanced memory, and storage.

Although these devices are already connected through the ubiquitous PCI Express interconnect (PCIe), communication between processors demands orders of magnitude faster links than PCIe can address like CXL (Computer Express link).

In this blog, we will discuss CXL verification challenges from IP-level to system-level and demonstrate how these challenges can be significantly mitigated using Cadence Verification IP (VIP) solution for advanced verification methodologies.

What is CXL?

Compute Express Link (CXL) is an open interconnect standard built on top of PCIe that addresses these workloads' bandwidth and latency requirements by extending cache coherency beyond the processor for heterogeneous computing and enabling resource disaggregation. Pervasiveness, performance, Caching, memory disaggregation, and security are the main key metrics for next-gen interconnects.

CXL leverages the PCIe architecture and develops the extra fabric over and above.
It addresses the performance requirements with more bandwidth with low latency.
CXL enables PCIe infrastructure to work with caches in the system and interconnects.
It also enables the users/system architects to borrow the memory from the connected peripherals according to the workload of the required scenarios. Security is another major requirement as we look for safe data transfers while using these interconnects.

CXL is comprised of 3 sub-protocols

CXL.io: I/O semantics (mandatory) –The semantics is remarkably like PCIe, it is required for typical enumeration, discovery, register access, interrupts, initializations, I/O virtualization, DMA.

CXL.cache – caching semantics (Optional) – helps in device caching of host memory and host processor manages coherency

CXL.memory – memory semantics (Optional)- Memory access protocol allows the host to manage device attached memory like host memory and can be managed depending on workloads.

Verification challenges

CXL is a layered protocol is like PCIe. It offers performance benefits, but adds the verification load. The various verification checks required to be done are as mentioned below

IP Level Verification Challenges

Verification team need to ensure backward compatibility with the PCIe.
Mem/cache transaction layers (TL) and data link layers (DL) must be verified.
The cache of the host processor and CXL device is in sync.

CXL.io Verification Challenges

A robust PCIe verification strategy where we need to take care of discovery, link, and error management.
As each CXL device, creates a new PCIe enumeration hierarchy, hierarchy needs to be taken care of.
CXL uses a PCI link training state machine LTSSM (Link Training and Status State Machine), so the same link training process is followed except the Alternate Protocol Negotiation which negotiates the speed, performance, etc.
Apart from these, certain capabilities like Advanced Error Reporting Extended Capability (AER), Address Translation Services (ATS), Data Poisoning by Transmitter are mandatory in CXL and should not be broken when CXL is enabled.

The link layer communicates to the physical layer via fixed-sized data blocks called FLITs for latency optimization.

Dynamic Multiplexing of FLITs for Performance is a critical aspect for verifying a CXL compliance device. The verification strategy must take care of several types of flits (protocol, control, and data)
Link Layer verification challenges
- For some performance achievement and deadlock avoidance aspects, there are certain rules which are to be followed while verification related to the link layer.
  - Proven in the market being used by plenty of customers
  - A layered approach to provide flexibility of using callbacks for error correction, score boarding etc.
  - Full transaction control to inject traffic as per requirements
  - Trackers and verbose debug logs
  - Comprehensive protocol checks
  - Various Bypass modes to speed up the verification and provide flexibility protocol messages are divided into 6 messages, Verification team need to take care of all types of messages such as initialization, flow control, retry and error checking and ensure that the multiplexing is occurring correctly.

Caching and disaggregation challenge

Cache consistency between device and host and performance over-heads.
Asymmetrical nature of link - this is critical to verify when there are multiple CXL links at the system level
Complexity increases when there are type 2 flows (mix of .cache and .mem transactions) special handling for device accessing the device memory based on source/device bias.

Security Challenge

There are various ECNS across (DMTF, PCI_SIG and CXL). The verification teams must know about the DMTF specs and mapping to PCIe ECN and CXL IDE ECNS and at times verifying IDE is as complex as verifying a separate IP.

System-level Verification Challenges

Integration tests- As all the components are connected at the system level, determining the operational correctness is a challenge as it involves a lot of complexity and protocols in CXL.
Another big challenge is to optimize the performance – as performance and low latency are key parameters for CXL, so turnaround time for memory/cache access, etc. need to be optimized
Also, Software/ level Hardware tests must be mixed to optimize the cost of the debugging of long tests for the system

Cadence Verification Solutions

We, at Cadence, offer verification IP (VIP) for CXL and PCIe which is highly configurable to meet varied IP verification scenarios. It supports all three modes type 1,2 and 3 as per CXL specifications

Features

Proven in the market being used by plenty of customers
A layered approach to provide flexibility of using callbacks for error correction, score boarding etc.
Full transaction control to inject traffic as per requirements
Trackers and verbose debug logs
Comprehensive protocol checks
Various Bypass modes to speed up the verification and provide flexibility

TripleCheck for CXL

As it is challenging to complete the verification within the time limits, Cadence TripleCheck is comprised of test cases, vPlan, and coverage module and is a widely used product. It ensures that every aspect of the specification is covered and saves a lot of verification time. These test cases are directly extracted from the CXL protocol and cater to all the layers.

System Verification Solution

We also provide system-level solutions to our customers, as shown below System VIP components.

CXL System Traffic Library provides out-of-the-box test scenarios for validating complex CXL-based systems. System Performance Analyzer helps in analyzing the design bottlenecks. It enables the discovery of performance degradation within memory subsystems, interconnects, and peripherals. It is Integrated with Cadence VIP / AVIPs / MMP, supports custom/proprietary buses, and provides Unified analysis across simulation and emulation platforms. System verification scoreboard collects the data from all the IPs connected in the network and helps to find out where the data link is broken.

Conclusion

The VIP for CXL leverages Cadence's mature industry-leading VIP for PCIe. Built on top of an industry-known and -proven platform that was designed for easy integration in testbenches at the IP, system-on-chip (SoC), and system-level

Enables verification of both host and device designs for all device types (Type 1-3)
VIP for CXL runs on all simulators and supports SystemVerilog along with the widely adopted Universal Verification Methodology (UVM).
Helps verification teams to reduce time spent on environment development, redirect the saved time to cover a larger verification space, accelerate verification closure, and ensure end-product quality.
Layered architecture and callback mechanism help to verify CXL features at each functional layer.

Learn More