New PCIe SI Challenges Conquered Using Clarity 3D Field Solver for PCB

Figure 1: High-performance PCIe-based graphics card

There is a trend in the data center industry moving toward heterogeneous computing in order to satisfy workloads that are computationally intensive. These trends are driving the development of software solutions that can distribute the workload among multiple computers of different core and memory configurations. Along with high-speed computing comes the need for high-speed data transfer. A key enabling technology for moving data is the PCIe bus, of which the latest generation’s standards (4.0/5.0) have dramatically increased bandwidth and lowered latency.

Though attractive from a data throughput standpoint, these performance increases are also generating substantial signal integrity (SI) design challenges. In order to meet PCIe 4.0/5.0 compliance testing, designers must now meet very stringent performance requirements, such as bit-error rate (BER) tests (32 GT/s function with NRZ signaling). This trend seems to be only accelerating this challenge as early versions of PCIe 6.0 continue to follow this trend. It is no surprise that iterative board spins and verification testing are not only time-consuming but costly. Engineers need to carefully design and test the interconnect design of the high-speed PCIe serial link. 

The following application note discusses PCIe 4.0/5.0 design challenges associated with signal integrity. Furthermore, insights into how designers are addressing these challenges with cutting-edge CAD tools is further detailed.

Detailed look at PCIe 4.0/5.0 requirements & compliance testing

The demand for greater throughput and performance from computer peripherals has led to each subsequent generation of PCIe doubling in bandwidth and transfer rate while halving the unit interval. Faster PCIe transfer rates are a necessary result of the need for data center applications to scale the bandwidth of memory, storage, and interconnects as processors are integrating a greater number of accelerators per CPU. For the transfer rate and bandwidth to increase, higher-frequency signals are generated. The loss of a conductive trace on a PCB is a function of the frequency of the signals traveling on that trace, meaning that the loss at higher frequencies (i.e., higher throughputs) is greater than at lower frequencies.

Hence, PCI-SIG has had to enhance the physical layer specification and increase the loss budget at the Nyquist frequency of the maximum transfer rate with each subsequent PCIe standard, along with reducing the acceptable jitter and eye height/width limits (eye characteristics). In the transition from PCIe 3.0 to 4.0, a move to lower-loss PCB substrates was required to meet the new specifications. With even tighter margins, ultra-low-loss PCB substrates are a critical tool in realizing PCIe 5.0 requirements.

Even though the dielectric loss and dispersion in higher-performance PCB substrates is substantially less than standard FR4, meeting PCIe 4.0/5.0 requirements is still a stretch for many designers. The channel loss, channel discontinuities, and crosstalk of a PCIe design result in greater system noise, eye closure, and poorer jitter performance. Maintaining a BER of less than one error in 1 trillion bits (10^–12 error rate) requires designers to carefully manage the budget in each area of the physical and electrical design. This includes minimizing loss in both the package and board design while accounting for other SI performance requirements.

A specified method of combating greater channel loss at high frequencies is the use of receiver equalization and transmitter de-emphasis. As PCIe 4.0/5.0 are considered closed-eye standards, meaning that the SI of the channel degrades inter-symbol interference and will force an eye closed even if the transmitter demonstrates zero jitter, link equalization is necessary to open the eye. Reach extensions tools, such as retimers and redrivers, have also been used since PCIe 3.0 to enable longer physical channels for server and storage systems. Reach extension tools are necessary when the physical length of the channel exceeds the PCIe specification, which often occurs with PCIe 4.0 systems that employ multi-connector topologies, cabled topologies, and single-connector add-in card topologies with >9.5-inch baseboard channels with mid-loss PCB substrates.

Challenges with designing PCIe 4.0/5.0 with SI in mind

The effectiveness and design of equalization, de-emphasis, and reach extension tools depend on how well the channel is understood and how tight the channel tolerances are. To gain this knowledge, precise modeling and physical design are essential. However, characterizing and modeling a high-speed digital channel is no easy feat, and even slight errors can have significant impacts on SI for high-speed signals.

The main reason for this is the intrinsic inductive and capacitive parasitics of conductive traces and how the insulating material around those traces impacts the parasitics. Moreover, PCB production tolerances are tight but not perfect, and the variation in dimensions of the traces and insulation material can lead to enough error to make parasitic models inaccurate. Another factor to consider is that a PCIe channel is not in isolation, and the channel’s traces electrically couple with all of the conductive structures in proximity. This means that mathematical models, 2D simulations, and non-inclusive approximations are typically too inaccurate to properly predict the parasitics to meet PCIe Gen 4.0/5.0 requirements.

Even if the parasitic extraction for a specific PCB physical layout is accurate, the parasitics may not be within an optimum range for the PCIe signal integrity enhancing circuitry and systems. Optimizing the traces and structures to minimize parasitics is most common and usually requires complex parametric analysis, the results of which must then be fed back into a simulator to determine if the physical structure meets the requirements of the PCIe circuits.

As the parasitic extraction and trace optimization techniques are typically rather error-prone, designers regularly resort to iterative testing of prototypes and physical designs until the layout yields channel characteristics within tolerance of the PCIe signal integrity enhancing circuitry. This type of rigorous testing either requires batch testing of a wide variation of physical layouts or a sequential cycle of design tweaks that adds significant delays and cost to the design process.

Using true 3D EM field solvers for PCB to reduce design cycle length & improve performance

A method of circumventing the delays from iterative testing associated with PCIe channel physical design is to use true 3D EM field solvers built to accommodate simulations of large PCB structures. If set up properly, a 3D EM solver can yield a much more accurate prediction of trace parasitics and export the results in a format that is readily integrated into a circuit simulator. Moreover, a 3D EM solver with high-enough accuracy and speed to capture small-dimension can even be used to provide EM simulation of both the IC package as well as the PCB traces. This approach provides a much more accurate and complete picture of the parasitic behavior and channel characteristics than cascading together the results of different IC and PCB parasitic extraction approaches. Additionally, performing a complete simulation of both large PCB structures and small IC packaging and circuit structures can yield models with less error than if the simulation were broken into parts, simulated separately, and then merged. The challenge in this approach is having a 3D EM solver without enough speed and capacity to make these simulations in a timely fashion.

Also, worth noting is that a 3D EM field solver with the ability to perform parametric optimization can be used to automatically converge on a PCB physical layout that meets design requirements and accounts for process variation and other production-tolerance variations. With a fast-enough 3D EM field solver with high enough capacity, the physical layout and prototype testing process can be dramatically reduced in time and iterations as compared to traditional methods of struggling with achieving desired SI with high-speed digital signals.

 Figure 2: Accurate 3D simulation tools with adequate capacity can produce models that are considerably better matched with experimental data of PCIe channel characteristics than approximations and reductive models.

Successful PCIe compliance certification with Cadence Design Flow with Clarity 3D Solver

Cadence Clarity 3D Solver can perform the functions mentioned above and includes other additional features that aid in saving design time and producing gold-standard simulation accuracy. With the ability to simulate using distributed multiprocessing technology, Clarity 3D Solver can be used for even extremely complex and intricate physical structures with a wide geometric variation much faster than solvers that rely on a single computing node. Greater processing resources enables a 3D solver to deliver more accurate models for signal integrity, power integrity (PI), and electromagnetic compliance (EMC) analysis, and does not require approximations or degrading of the structure geometries to be more “simulator friendly”. The result of using Clarity 3D Solver are models that are a much closer match to laboratory measurements and can be directly imported into industry-standard circuit simulators (i.e. Cadence Sigrity SystemSi) for incredible correlation between actual measurements and simulation.

These features are especially valuable for PCIe compliance and interoperability testing, such as the electrical testing done as part of the PCI-SIG Compliance Program during PCI-SIG Compliance Workshops [6]. At these workshops, both interoperability tests and compliance tests are performed, with the results reported as either “pass” or “fail”. A product must score a minimum of 80% pass ratings for interoperability tests and pass 100% of the compliance tests. Failure during a compliance workshop means that a product may not be labeled as PCIe compliant, which may eliminate a product's chances in a highly competitive marketplace and require a redesign cycle and retesting at the next PCI-SIG Workshop.

Clarity 3D Solver delivers a key advantage to designers for the electrical testing, which is comprised of platform and add-in card Transmitter and Receiver characteristic testing, as simulating with gold-standard accuracy interconnect models provided by Clarity brings confidence to the designers that they will be able to pass the first time. 

Moreover, Clarity is part of a best in class design and analysis flow only available from Cadence. This design flow includes a complete design environment with Allegro high-speed design rules, in-design signal integrity and power integrity analysis with Aurora, precision 3D model extraction with Clarity, and final-step PCIe pre-compliance simulation with SystemSI. The sophisticated and integrated solutions comprising the Cadence PCIe design flow, including Clarity 3D Solver, help to ensure a PCI product is brought to volume production on time and on budget.

References

1. https://pcisig.com/
2. PCI-SIG PCI 4.0 and PCI 5.0 Specifications
3. https://pcisig.com/faq?field_category_value%5B%5D=pci_express_4.0&keys=
4. https://pcisig.com/pci-express%C2%AE-retimers-vs-redrivers-eye-popping-difference
5. PCI Express Retimers vs. Redrivers: An Eye-Popping Difference
6. https://www.cadence.com/content/dam/cadence-www/global/en_US/documents/tools/system-analysis/Clarity-3d-solver-ds.pdf 

Tables