Felipe Goncalves

Community Member

Blog Activity
Options

Enhancing PCIe6.0 Performance: Flit Sequence Numbers and Selective NAK Explained

• 5 minute read

Introduction

The Flit Sequence Number is a mechanism introduced in the PCIe 6.0 specification, accompanying the transition to Flit Mode operation. This enhancement supersedes the legacy transaction layer packet (TLP) sequence numbering, along with its associated acknowledgment and replay protocols.

What Is a Flit Sequence Number?

Historically, each TLP carried an explicit sequence number, which, while contributing to link reliability, resulted in inefficient resource utilization. PCIe 6.0 addresses this inefficiency through the Flit Sequence Number protocol, which leverages implicit sequencing. In this model, the receiver infers sequence numbers without requiring explicit transmission. Furthermore, the sequence number is now applied at the flit level, enabling the encapsulation of multiple TLPs within a single flit. This significantly reduces overhead, allowing the reclaimed bandwidth to be repurposed for transmitting higher-value data.

Figure 1: DLP bit usage

In the image above, we can see the fields associated with this feature. The bits related to Flit Usage are used to determine the role of the current flit within the transmission process. An Idle Flit is used during the Recovery.Idle and Configuration.Idle to help the link transition to the L0 state, ensuring proper synchronization and readiness for data transfer. A NOP Flit is characterized by having all 236 bytes reserved, that it is the TLP field, it is important to highlight that the NOP Flit has DLLP information. A Payload Flit, on the other hand, contains actual TLP content and is responsible for delivering meaningful data across the link.

The Replay Command field defines the type of replay mechanism associated with the flit. It can indicate whether the flit is an Explicit Flit, a Selective Negative Acknowledgment (NAK), an ACK, or a Standard NAK. These commands are essential for managing retransmissions and ensuring data integrity. The Explicit Flit and Selective NAK types are particularly new and will be discussed in more detail throughout this blog.

The DLLP also includes the Prior Bit, which informs if the previous flit was payload. If the prior bit is 0, the previous flit can be a NOP or an Idle, in case it is the flit non-idle flit transmitted is a payload. This information helps the receiver determine whether a replay is necessary, thereby avoiding redundant retransmissions and improving the throughput. This distinction is crucial in scenarios where a device receives a corrupted flit that has been explicitly pointed out for replay. If the corrupted flit is identified as a NOP Flit, it does not carry any meaningful data and therefore does not require retransmission. As a result, the device can avoid sending a NAK and bypass the entire replay mechanism, which would otherwise consume bandwidth and processing resources.

Additionally, the DLLP Payload Type field defines the interpretation of bits 31 to 0, specifying the nature of the payload data and enabling correct parsing by the receiver. Finally, the Sequence Number field, located in bits 41 to 32, serves different purposes depending on the Replay Command type. It may be used to track the order of flits, identify missing packets, or coordinate selective retransmissions, depending on the context of the communication.

Selective NAK Replay for Efficient Flit Recovery

In case the prior bit of the flit that follows the corrupted flit is 1, the replay mechanism must be triggered, and the device can choose between the Selective or Standard NAK, if it is not a scenario in which the standard NAK is mandatory. The first one was introduced with this new feature, so that when this Selective NAK is triggered, the device that receives the Selective NAK must start replaying the NAK number plus 1—see the flow below:

Figure 2: Diagram of Selective NAK

In this diagram, Device A is transmitting a sequence of flits to Device B, specifically those with implicit sequence numbers 23, 24, 25, and 26. During this transmission, the flit with sequence number 24 is corrupted, which is visually indicated with the message in red. Upon detecting the corrupted flit, Device B initiates a selective replay mechanism by issuing a Selective NAK referencing sequence number 23. This request prompts Device A to retransmit only the necessary flits rather than the entire sequence. As part of the selective replay process, Device A sends an explicit flit containing sequence number 24, followed by another explicit flit with sequence number 26 if there isn't new payload flits to be transmitted, in case that there is a new one the Device A should transmit the new sequence number 27. This approach ensures that only the relevant data is retransmitted, optimizing link efficiency and minimizing unnecessary traffic. In the meantime, Device A removed the flit 23 from the Tx Retry buffer. And the Device B will remove the 25 and 26 from Rx Retry Buffer, because this will be processed with the 24. 

As Device B can request retransmission only for flits that were received with errors, the RX Retry Buffer was introduced. This buffer temporarily stores information about incoming flits until they are fully consumed by the receiver. Its indexing mechanism is essential for Device B to maintain accurate tracking of data from the transaction layer, ensuring that corrupted or missing flits can be identified and managed efficiently. By retaining this information, the device can selectively request retransmissions without affecting the integrity of successfully received data.

Verification Challenges and Solutions

This new feature introduces several challenges, one key example is maintaining accurate tracking of flits during selective NAK scenarios.

The monitor must continue tracking flits until it receives the replay of the corrupted flit. This behavior depends on the Rx Retry Buffer size, which must be properly aligned with the RTL implementation to prevent overflow and potential data loss. If the buffer overflows, it may trigger unexpected NAK transitions or force a switch to standard replay.

Since it's not possible to predict whether the RTL will immediately process or temporarily store incoming flits, the VIP must be designed to tolerate delays in both data handling and replay initiation. These delays are configurable within the VIP to match the expected behavior of the RTL.

In selective NAK scenarios, new flits may be inserted after the replayed corrupted flit, and the monitor must continue tracking these flits to ensure protocol consistency.

The replay type, selective or standard, can be configured in the VIP, enabling flexible testing of various RTL behaviors and recovery mechanisms. Additionally, the VIP supports error injection, allowing a wide range of test scenarios to validate robustness and compliance.

Conclusion

To conclude, the Flit Sequence Number feature plays a critical role in helping PCIe devices maintain proper tracking of transmitted data. It contributes to the organization of flits across the link and supports mechanisms like selective replay, ultimately improving throughput and reliability in high-speed communication environments.

Learn More

  • For more information on PCIe in general, and on the various PCI standards, see the PCI-SIG website