In our previous lecture, we explored how the Transaction Layer assembles our data into Transaction Layer Packets (TLPs). Once a TLP is built, it must safely survive the physical journey across the PCIe connection. This crucial responsibility falls to the Data Link Layer, which serves as the highly reliable middleman between the Transaction Layer above and the Physical Layer below.
Here is a deep dive into how the Data Link Layer utilizes Link management, Data Link Layer Packets (DLLPs), and Flow Control to guarantee error-free, highly efficient data transfers.
What are Data Link Layer Packets (DLLPs)?
While the Transaction Layer communicates using TLPs, the Data Link Layer generates and consumes its own specialized packets called Data Link Layer Packets (DLLPs).
Unlike TLPs, which are routed through the entire PCIe topology to reach a final destination, DLLPs are strictly local. They only travel between the Data Link Layers of two direct, neighboring devices on a single Link. The Transaction Layer is completely unaware of their existence, and they are never routed through Switches to other devices.
Because DLLPs represent overhead that consumes Link bandwidth, they are intentionally kept very small—always exactly 8 bytes. To ensure they arrive correctly, a 16-bit CRC is appended to the DLLP before it is handed down to the Physical Layer for transmission.
Link Management and The Ack/Nak Protocol
One of the primary jobs of the Data Link Layer is to protect TLPs from transient physical errors (like electrical noise) as they cross the wire. It accomplishes this using a hardware-based, automatic retry mechanism known as the Ack/Nak protocol.
Here is how the protocol protects your data:
- Prepping for Transmission: When a TLP is passed down from the Transaction Layer, the Data Link Layer appends a 12-bit Sequence Number and a 32-bit Link CRC (LCRC) to the packet.
- The Replay Buffer: Before sending the packet to the Physical Layer, the transmitting Data Link Layer stores a backup copy of it in a local Replay Buffer. It holds this copy until the neighboring device actively confirms it received the packet perfectly.
- Positive Acknowledgement (Ack): When the neighbor receives the TLP, its Data Link Layer checks the LCRC and Sequence Number. If everything is correct, it generates an Ack DLLP containing the sequence number of the good packet and sends it back to the transmitter. The transmitter then safely flushes that packet from its Replay Buffer.
- Negative Acknowledgement (Nak): If the receiver detects a corrupted TLP, it instantly drops the packet and returns a Nak DLLP. Upon receiving the Nak, the transmitter stops and automatically replays all unacknowledged TLPs from its Replay Buffer. Because most errors are transient, the replay usually succeeds the second time around, completely resolving the error in hardware without any software intervention.
Implementing Flow Control to Prevent Buffer Overflows
In legacy PCI architectures, if a device sent data to a receiver whose buffers were full, the receiver had to abruptly disconnect or force the sender to retry, wasting immense amounts of bus time and bandwidth.
PCIe elegantly completely prevents buffer overflows using a strict Flow Control mechanism managed entirely by the Data Link Layer.
- The Rule: A transmitter is strictly prohibited from sending a TLP unless it absolutely knows the receiver has sufficient buffer space to accept it.
- Advertising Buffer Space: Following a system power-up or reset, the receiving device automatically begins advertising the exact size of its available Virtual Channel (VC) receive buffers. It does this by periodically sending Flow Control Update DLLPs to the transmitter.
- Tracking Available Space: The transmitter keeps a running tally of this available space. As the receiver processes TLPs and frees up buffer room, it sends new Flow Control DLLPs to update the transmitter.
Why use DLLPs for Flow Control? You might wonder why the system uses dedicated DLLPs to report buffer space rather than just embedding the updates inside standard TLPs. The reason is to prevent devastating system deadlocks. If a device’s TLP receive buffers were completely full, it wouldn’t be able to accept a new TLP containing a buffer update. Because DLLPs are processed immediately by the Data Link Layer and do not use the TLP Virtual Channel buffers, Flow Control updates can always be sent and received regardless of how congested the TLP buffers are.
Through the clever use of lightweight DLLPs, the Data Link Layer flawlessly manages Flow Control and error correction entirely in the background, making it completely transparent to system software while guaranteeing maximum Link efficiency.