PCIe 6.0 Standard Ratified...and Cadence's Implementation

The big news about PCIe 6.0 is that the specification has been released by the PCI-SIG. I covered some of PCIe 6.0 in my post The History of PCIe: Getting to Version 6, although a lot of that post was about earlier versions of the standard. More recently I came back and fleshed out more details in my post TSMC OIP: N3, N4, and PCIe 6.0.

Roughly every three years, the bandwidth has been doubled. PCIe 6.0 does just that, doubling the performance to 64GT/s transfer speeds. See the graph below.

You don't need me to tell you how important data is. This is at the big picture level measured by how many videos are downloaded or how many emails are sent. But it is also at the level of individual servers and even on-chip bandwidth. The energy required to move the data around an SoC can be many times the energy required to process it. And the numbers are huge. I'm sure you've seen many other charts like the one below, showing how much data is created every day. Another amazing statistic is that 500 hours of video are uploaded to YouTube every minute (and one billion hours of YouTube are watched daily). Since an individual video is large, much larger obviously than, say, a WhatsApp message, this is a huge percentage of all the data on the internet, especially if you add in Netflix and all the other streaming services. All that data needs to get in and out of computer systems and onto SSDs and networks, and the primary way this is done is using PCIe.

The biggest difference between PCIe 5.0 and PCIe 6.0 is that the signaling has changed from NRZ to PAM4. The highest speed SerDes used in other applications made the same change a couple of years ago to achieve 112G speeds. However, PCIe 6.0 interfaces are compatible with the older levels of the standard, and drop back to lower speed NRS communication for devices that cannot handle the full rate. The diagram below shows the difference between NRZ and PAM4. At the top, NRZ transmits one bit on each clock cycle, either a low or high voltage. At the receiver, the eye diagram contains a single eye. At the bottom is PAM4, with two bits transmitted on each cycle, using four different voltage levels. At the receiver, the eye diagram has three eyes. The good news is that the amount of data is increased with no increase in the Nyquist frequency (a theoretical channel limit). The bad news is that the noise margin is reduced meaning that more aggressive equalization is required to make everything work.

The other highlights of the PCIe 6.0 specification are:

• 64GT/s raw data rate and up to 256GB/s via x16 configuration
• Pulse Amplitude Modulation with 4 levels (PAM4) signaling and leverages existing PAM4 already available in the industry
• Lightweight Forward Error Correct (FEC) and Cyclic Redundancy Check (CRC) mitigate the bit error rate increase associated with PAM4 signaling
• Flit (flow control unit) based encoding supports PAM4 modulation and enables more than double the bandwidth gain
• Updated Packet layout used in Flit Mode to provide additional functionality and simplify processing
• Maintains backwards compatibility with all previous generations of PCIe technology

There are two levels of error correction. Forward error correction (FEC) allows for some correction without requiring retransmission. For an explanation of FEC, see my post What the FEC is Forward Error Correction? However, there is also a CRC, which is used to check that the packet is good after FEC, and if not then there is a capability to retransmit it. As long as the FEC works fully most of the time, this has very limited overhead.

Cadence is not just a passive observer of the PCI standardization, we have been heavily involved in the whole process. Rishi Chugh, VP product management for our IP group, is quoted in the press material from PCI-SIG:

PCI-SIG’s announcement of the PCI Express 6.0 specification is a milestone in the evolution of the protocol that brings together PAM4 and NRZ technology to address the continually increasing demands of our hyperscale and intelligent applications in our data-centric world. As a contributor to the PCI Express 6.0 specification, Cadence supports the PCIe 6.0 specification release with complete high-quality PHY and controller IP and our verification IP. This comprehensive Cadence offering allows customers to get to market with a robust, high-performance solution while lowering risk and reducing development costs.

The Cadence PCIe 6.0 PHY and Controller

As Rishi said above, we already have a PHY and we actually have silicon being characterized in our lab. In some areas, the normal way to operate is get the spec standardized, and then.build product that meets the spec. But semiconductor IP doesn't really work like that. The rough details of the spec are known long in advance, and so the design can start. Going back a few years to PCIe 4.0, Gilad Shaner, then of Mellanox (since acquired by NVIDIA) said to me that:

Interoperability is the only way to prove standards compliance.

In a similar vein, we do the same thing with memory interfaces: DDR5 IP Test Chip Operates with Micron Prototype DRAM at 4400 MT/s.

Implementation of PCIe 6.0 makes use of a lot of digital signal processing (DSP) instead of analog interfaces. This matches much better to the capabilities and limitations of the most advanced process nodes. In fact, DSP techniques have been used for data rates above about 32Gbps. The advantages are:

• Technology scaling trend enabling low power DSP for performing advanced data equalization and recovery in small area
• Standard digital design flow achieving shorter design cycle, robust design margin, and higher DFT coverage
• Less sensitive to PVT, noise, and other environmental factors
• Smaller area than analog

It is worth emphasizing that PCIe 6.0 is not our first use of PAM4 signaling. It has been in use for 56G and 112G Serdes for a couple of years now. So this approach is mature and is considered a low-risk approach now. The chip is called Excelsior.  It is optimized for the highest performance 64G PCIe 6.0, and has fully autonomous startup and adaptation without ASIC intervention being required. PMA is the physical medium attachment; PCS is the physical coding sublayer.

Other features:

• PAM4 + NRZ dual mode signaling support
• DSP-powered adaptive equalization and data recovery
• Programmable Multi-rate CTLE
• Adaptive offset/gain/phase correction
• SRIS Clock Recovery support
• LC PLL for low jitter performance
• On-die AC termination and T-coil minimize return loss
• PIPE 6.0 supported
• Support PCIe L1 sub-states power management
• Native Bifurcation support

A PHY also requires a controller, and in practice other things such as verification IP, and drivers:

The complete solution consists of:

• Controller: Root Port, End Point, and Dual mode
• Software Core Driver and Linux Reference Driver
• Complete Solution: Controller, PHY, Drivers, and VIP

Video Demo

Watch a video demo of PCIe 5.0 and PCIe 6.0 (about 7 minutes):

Learn More

If you are a member, you can download the PCIe 6.0 specification from this page (or any other specification for that matter).

You can read the press release announcing the ratification and release of the standard.

You can read the Cadence datasheet on PCIe 6.0 and the press release about it being made available to early adopters.

Sign up for Sunday Brunch, the weekly Breakfast Bytes email.

.