Post 7.1 – Data Link Layer in PCI Express: Ensuring Reliable Delivery

Ack/Nak protocol, sequence numbering, and replay logic explained

1 . Introduction

The Data Link Layer (DLL) sits between the Transaction Layer (TL) and the Physical Layer (PHY).
 Its core responsibility is reliability — ensuring that every TLP transmitted over the serial link is received exactly once, and without corruption.

Even though the Physical Layer may introduce occasional bit errors or alignment issues, the DLL guarantees that upper layers never see these problems.

💡 In simple terms:
 The DLL makes an unreliable physical channel behave like a perfect one.

2 . Key Functions of the Data Link Layer

Function	Description
Reliability (Ack/Nak System)	Detects errors via CRC (LCRC) and retransmits lost/corrupted packets
Replay Management	Stores recently sent TLPs for potential retransmission
Sequence Numbering	Tags each packet to maintain order and detect duplicates
Flow Control DLLPs	Exchanges credit information (as covered in Chapter 6)
DLLP Management	Transmits/receives Ack, Nak, Update, and other control packets

3 . The Reliability Challenge

High-speed serial links can occasionally experience:

• Bit flips (due to noise/jitter)
  
  
• Symbol misalignment
  
  
• Packet boundary errors
  
  

Unlike parallel buses (which use per-cycle parity/handshake), PCIe transmits long bursts of bits — entire packets.
 Hence, the receiver needs a way to:

1. Detect corrupted packets
   
   
2. Signal retransmission if needed
   
   
3. Keep everything in correct order
   
   

This is achieved via the Ack/Nak Protocol.

Each TLP transmitted across the link is protected by a Link CRC (LCRC).
 After receiving a packet, the receiver checks this CRC and responds with a DLLP:

Response Type	Meaning	Action Taken
Ack DLLP	TLP received correctly	Transmitter can discard it from Replay Buffer
Nak DLLP	TLP corrupted (LCRC fail or missing)	Transmitter must retransmit from Replay Buffer

This mechanism ensures that every packet is acknowledged, either positively (Ack) or negatively (Nak).

5 . Sequence Numbers

Every TLP sent on a PCIe link is tagged with a 12-bit Sequence Number (0–4095).
 It increments monotonically with each transmitted TLP.

Purpose	Benefit
To uniquely identify each packet	Detect duplicates or missing packets
To align Acks/Naks with the right packet	Ensures correct retransmission
To maintain order across the link	Guarantees in-order delivery per VC

⚙️ After 4095, the sequence number wraps around to 0.
 Each direction of the link maintains its own independent counter.

6 . Replay Buffer

Since retransmissions may be required, the transmitter keeps a Replay Buffer — a circular memory holding recently sent TLPs.

Event	Replay Buffer Action
New TLP transmitted	Added to buffer (pending Ack)
Ack received	Removed from buffer
Nak received	Replayed from buffer (retransmitted)
Timeout (no Ack)	Retransmit oldest pending packets

The size of this buffer determines the maximum outstanding unacknowledged packets that can exist on the link.

7 . Replay Timer

Each transmitter starts a Replay Timer when it sends an unacknowledged TLP.
 If no Ack arrives before the timer expires:

• It assumes packet loss.
  
  
• Retransmits from the Replay Buffer.
  
  
• Restarts the timer.
  
  

Replay timers are typically calibrated based on link round-trip delay and credit exchange latency.

Let’s walk through a normal transaction:

Step 1 – Transmission

• Transmitter sends 3 TLPs (Seq# 100, 101, 102).
  
  
• All three are stored in Replay Buffer.
  
  

Step 2 – Reception

• Receiver receives TLP#100, 101 correctly → sends Ack DLLP for 100–101.
  
  
• TLP#102 corrupted → sends Nak DLLP.
  
  

Step 3 – Retransmission

• Transmitter removes 100–101 from buffer.
  
  
• Retransmits TLP#102 (same sequence number).
  
  
• Receiver receives successfully → sends Ack 102.
  
  

Replay complete.
 All packets are now acknowledged.

9 . Ack/Nak DLLP Structure

Field	Description
DLLP Type	Indicates Ack/Nak
Sequence Number	Identifies last correctly received TLP
VC (Virtual Channel)	Acknowledgment is per-VC
CRC	Protects the DLLP itself

Ack/Nak DLLPs are short, link-local, and high-priority, ensuring rapid recovery.

10 . Replay Buffer Example

Sequence	Status	Action
100	Acked	Removed
101	Acked	Removed
102	Nak	Retransmit
103	Pending	Hold until replay complete

Replay happens in order, maintaining strict delivery order across the link.

11 . Design Implementation Notes

In controller RTL:

• Each VC has its own sequence counter and replay buffer.
  
  
• Ack/Nak DLLPs are handled by a dedicated DLLP engine.
  
  
• Replay logic uses a state machine:
  
  
  • IDLE → WAIT_ACK → REPLAY → RESUME
    
    

if (nak_received)

   replay_from(seq_number);

else if (ack_received)

   remove_acked_entries();

Replay Buffer must be large enough to hold all unacknowledged packets within a round-trip window.

12 . Error Recovery

If a packet is corrupted repeatedly or replay timeout exceeds threshold:

• The link transitions to Recovery state (LTSSM).
  
  
• Retraining occurs to re-establish synchronization.
  
  
• Credits and sequence numbers reset.
  
  

13 . Verification and Debug Tips

✅ Check Sequence Numbers – Ensure wrap-around works correctly (0→4095→0).
 ✅ Track Ack/Nak DLLPs – Ensure all TLPs are acknowledged eventually.
 ✅ Verify Replay Behavior – Replays should occur only after Nak or timeout.
 ✅ No Replay Storms – Excessive replay indicates LCRC or Ack handling bug.
 ✅ Buffer Management – Ensure no replay buffer overflows under max load.

NOTE:

• Data Link Layer = Reliability layer of PCIe.
  
  
• Uses Ack/Nak DLLPs, Sequence Numbers, and Replay Buffers to ensure error-free transmission.
  
  
• Each direction of the link operates independently.
  
  
• Replay occurs automatically and transparently to upper layers.
  
  
• Together with flow control, DLL guarantees lossless and in-order communication.