PCIe Series — PCIe-12: Data Link Layer — DLLPs and Reliability — VLSI Trainers
PCIe Series · PCIe-12

Data Link Layer — DLLPs and Reliability

How the Data Link Layer guarantees reliable TLP delivery — DLLP types and formats, sequence numbers, LCRC, the Replay Buffer, the ACK/NAK protocol, REPLAY_TIMER, and how Gen 6 flit-based reliability changes the picture.

📋 What the Data Link Layer Does

The Data Link Layer (DLL) sits between the Transaction Layer above it and the Physical Layer below it. Its single most important job is guaranteed delivery of TLPs across one link hop. If a TLP is corrupted or lost in the Physical Layer, the DLL detects this and automatically retransmits the packet — without any involvement from software, drivers, or the Transaction Layer.

Data Link Layer — Position and Responsibilities Transaction Layer Data Link Layer Physical Layer Data Link Layer Responsibilities ① Assign 12-bit Sequence Number to every outbound TLP ② Append 32-bit LCRC (Link CRC) to every outbound TLP ③ Keep a copy of every sent TLP in the Replay Buffer until ACK arrives ④ Detect errors via LCRC and Sequence Number · send ACK or NAK DLLP back ⑤ Automatically retransmit (replay) TLPs when NAK or timeout occurs
Figure 1 — The Data Link Layer wraps every TLP with reliability metadata before sending it to the Physical Layer. The Transaction Layer sees only clean, error-free TLPs — it has no visibility into the ACK/NAK protocol happening below it.

The DLL also manages flow control credit updates (carried in UpdateFC DLLPs) and link power management signalling (PM DLLPs). These topics are covered in later posts. This post focuses on the reliability mechanism — the ACK/NAK protocol.

📋 How DLL Wraps a TLP

Every TLP that the Transaction Layer hands down to the Data Link Layer gets two additions before it is passed to the Physical Layer for transmission:

DLL Wrapping — Sequence Number Prepended, LCRC Appended SeqNo 12-bit 2 bytes TLP Header 3 or 4 DW — all types unchanged from TL Payload if present 1–1024 DW LCRC 32-bit 4 bytes DLL adds prepended from Transaction Layer from Transaction Layer DLL calculates appended LCRC covers: SeqNo + Header + Payload + ECRC
Figure 2 — DLL wrapping. The Sequence Number is prepended (2 bytes, top 4 bits reserved, lower 12 bits the counter value). The LCRC is appended (4 bytes, CRC-32 covering all fields including SeqNo). The Transaction Layer’s TLP content is unchanged — DLL adds reliability metadata around it.

📋 DLLPs — What They Are

DLLPs (Data Link Layer Packets) are the private language of the Data Link Layer. They carry ACK and NAK signals, flow control credit updates, and power management transitions. DLLPs are fundamentally different from TLPs in several ways:

📋 DLLP Format

DLLP — 8 Bytes Total (Fixed Size) DLLP Type Byte 0 [7:0] 8 bits — identifies DLLP DLLP-Specific Fields Bytes 1–3 (24 bits) Content varies by DLLP type 16-bit CRC Bytes 4–5 covers bytes 0–3 Physical Layer adds framing around the 6-byte core: Gen 1/2 (8b/10b): SDP K-character before + END K-character after Gen 3–5 (128b/130b): 2-byte SDP token before · no END token
Figure 3 — DLLP structure. Byte 0 is the type code. Bytes 1–3 carry type-specific content (sequence number for ACK/NAK, credit counts for flow control). Bytes 4–5 are the 16-bit CRC protecting bytes 0–3. Physical Layer adds framing — the total transmitted size including framing is 8 bytes in all generations.

📋 DLLP Types

DLLP TypeType byte encodingPurpose
Ack0000 0000bPositive acknowledgement — the receiver tells the transmitter which TLPs arrived correctly. Carries a 12-bit Sequence Number representing the last successfully received TLP.
Nak0001 0000bNegative acknowledgement — a TLP failed LCRC check or arrived out of sequence. Transmitter must replay from the indicated Sequence Number.
PM_Enter_L10010 0000bRequest to enter L1 link power state.
PM_Enter_L230010 0001bRequest to enter L2/L3 link power state.
PM_Active_State_Request_L10010 0011bRequest for ASPM L1 entry.
PM_Request_Ack0010 0100bAcknowledge a power management request.
Vendor Specific0011 0000bVendor-defined content. Bytes 1–3 are vendor-defined.
InitFC1-P/NP/Cpl0100–0110 0xxxbFlow control initialisation phase 1 — advertise receive buffer sizes.
InitFC2-P/NP/Cpl1100–1110 0xxxbFlow control initialisation phase 2 — confirm values.
UpdateFC-P/NP/Cpl1000–1010 0xxxbUpdate flow control credits as receive buffers drain.

xxx = VC number (0–7). Flow control DLLPs are covered in PCIe-13. PM DLLPs are covered in PCIe-16.

📋 Sequence Numbers

Every TLP that the Data Link Layer transmits gets a 12-bit Sequence Number stamped on it before it is sent. The Sequence Number counter starts at 0 after reset and increments by 1 for each TLP. When it reaches 4095 it wraps back to 0. This wrap is expected and handled correctly — “later” in sequence terms means the next sequential value after wrap-around, not necessarily numerically larger.

Sequence Number Counter and Wrap-Around Reset → 0 0 TLP 1 1 TLP 2 2 TLP 3 → ··· 4094 4095 max value wrap → 0 0 next TLP 1 2 Safety rule: NEXT_TRANSMIT_SEQ and last ACK’d Sequence Number must never differ by more than 2048 If they reach 2048 apart, the transmitter stops accepting new TLPs from the Transaction Layer until the gap closes
Figure 4 — 12-bit Sequence Number counter. Counts 0 through 4095 then wraps back to 0. “Later” is defined as the next value in counting order, accounting for wrap-around. Transmitter and receiver must never be more than 2048 apart or the numbering scheme becomes ambiguous.

Three outcomes when a TLP arrives at the receiver

Sequence Number comparisonMeaningAction
SeqNo == NEXT_RCV_SEQ (expected) Good TLP — this is exactly the packet we were waiting for Accept, forward to Transaction Layer, increment NEXT_RCV_SEQ, schedule ACK
SeqNo < NEXT_RCV_SEQ (earlier) Duplicate — we have already received and accepted this TLP Silently drop. Send ACK with NRS−1 so transmitter knows current progress
SeqNo > NEXT_RCV_SEQ (later) Gap — a TLP was lost or dropped between this one and the last good one Discard this TLP. Send NAK with NRS−1. Set NAK_SCHEDULED flag

📋 LCRC — Link CRC

The LCRC (Link CRC) is a 32-bit CRC appended by the transmitter DLL to every TLP before it is sent. The receiver DLL recalculates the same CRC over the received bits and compares the result with the received LCRC. Any single-bit or burst error in the TLP body will produce a mismatched CRC and trigger a NAK.

The LCRC covers: Sequence Number + TLP Header + Data Payload (if any) + ECRC (if present). It does not cover the Physical Layer framing symbols.

LCRC is per-hop only. When a TLP passes through a switch, the switch’s DLL checks the incoming LCRC, strips it, and the switch’s DLL appends a freshly calculated LCRC before forwarding the TLP out the egress port. This means LCRC only detects errors on the individual link segment. Errors that corrupt a TLP inside a switch’s internal fabric — after the LCRC check but before the new LCRC is calculated — are invisible to LCRC. That is exactly why ECRC (the Transaction Layer’s optional end-to-end CRC, enabled by the TD bit in DW0) exists.

📋 Replay Buffer

Before sending any TLP, the transmitting DLL stores a complete copy of it — including the prepended Sequence Number and appended LCRC — in the Replay Buffer. This copy stays in the buffer until an ACK DLLP arrives confirming successful reception. Only then is it purged.

Replay Buffer — Stores Unacknowledged TLPs Transmitter DLL Replay Buffer TLP SeqNo=7 (sent, waiting ACK) TLP SeqNo=8 (sent, waiting ACK) TLP SeqNo=9 (just sent) TLP SeqNo=9 Receiver DLL LCRC check ✓ SeqNo check ✓ Forward to TL Schedule ACK(9) ACK DLLP(9) On ACK(9): purge SeqNo ≤ 9 from buffer
Figure 5 — Replay Buffer holds all sent-but-unacknowledged TLPs. When ACK(9) arrives, everything with SeqNo ≤ 9 is purged. The buffer must be large enough to hold all in-flight TLPs for a round-trip ACK latency without stalling the Transaction Layer.

Sizing the Replay Buffer

The Replay Buffer is not a fixed hardware size mandated by the spec — it is a design choice. It needs to be large enough to hold all TLPs sent during one ACK round-trip without stalling. The round-trip latency includes: physical link propagation delay, ACK generation time at the receiver, and ACK transmission delay back to the sender. For a Gen 6 x16 link, the ACK round-trip allows many large TLPs to be in flight simultaneously.

📋 ACK/NAK Protocol — Normal Operation

Under normal conditions (no errors), the protocol is simple: transmitter sends TLPs with incrementing Sequence Numbers, receiver accepts them, and periodically sends ACK DLLPs to confirm delivery.

Normal ACK Operation — One ACK Can Acknowledge Multiple TLPs Transmitter Device A Sends TLPs SeqNo=5,6,7 TLP SeqNo=5 TLP SeqNo=6 TLP SeqNo=7 Receiver Device B All three pass LCRC ✓ SeqNo in order ✓ All forwarded to TL ACK latency timer running Timer expires: Send single ACK(7) ACK DLLP — SeqNo=7 One ACK(7) purges TLPs 5, 6, and 7 from the Replay Buffer — cumulative acknowledgement reduces ACK traffic overhead
Figure 6 — Normal ACK operation. Three TLPs are sent with SeqNos 5, 6, 7. All pass LCRC and SeqNo checks at the receiver. The ACK latency timer coalesces them into a single ACK(7). Transmitter purges SeqNos ≤ 7 from its Replay Buffer. No separate ACK is needed for each TLP.

The key efficiency feature is cumulative acknowledgement. A single ACK DLLP with SeqNo=N tells the transmitter that TLPs with SeqNo 0 through N all arrived successfully. This means heavy traffic with many TLPs per round trip generates only one ACK per batch, keeping DLLP overhead low.

📋 NAK and Replay — Error Recovery

NAK — TLP 6 Fails LCRC · Replay from 6 Onward Transmitter Sends 5,6,7,8 Receives NAK(5) Purges ≤5 Replays 6,7,8 TLP 5 → received OK TLP 6 → LCRC FAIL discarded · NAK pending TLP 7 → discarded (out of SeqNo) TLP 8 → discarded (out of SeqNo) Receiver TLP 5: OK → forward to TL TLP 6: LCRC fail → discard Set NAK_SCHEDULED flag TLP 7: SeqNo gap → discard TLP 8: SeqNo gap → discard Send NAK(5) = last good NAK DLLP — SeqNo=5 Transmitter purges ≤5, replays 6, 7, 8 — Receiver accepts them in order, sends ACK(8) Replayed TLPs 6, 7, 8 → all pass ✓ ACK(8) → buffer purged
Figure 7 — NAK recovery. TLP 6 fails LCRC at the receiver. TLPs 7 and 8 are discarded because they arrived out of sequence. Receiver sends NAK(5) — meaning SeqNo 5 was the last good TLP. Transmitter purges everything ≤ 5 and replays TLPs 6, 7, 8. All three pass on replay. Receiver sends ACK(8).

Important NAK rules

📋 REPLAY_TIMER and REPLAY_NUM

The REPLAY_TIMER is a watchdog that runs whenever there are unacknowledged TLPs in the Replay Buffer. If it expires without receiving any ACK or NAK that makes forward progress, the transmitter automatically replays everything in the buffer — even without receiving a NAK. This handles scenarios where the NAK itself was corrupted and discarded.

REPLAY_TIMER and REPLAY_NUM — Watchdog and Escalation REPLAY_TIMER Starts: when a TLP is transmitted and not yet acknowledged Resets: when ACK/NAK makes forward progress Stops: when Replay Buffer becomes empty (all TLPs acked) On expiry: replay entire buffer · increment REPLAY_NUM reset and restart REPLAY_TIMER Covers: lost NAK · receiver fails to send ACK · no response REPLAY_NUM (2-bit counter) Counts how many replay attempts have occurred Values: 00 → 01 → 10 → 11 → rolls to 00 Resets to 00: when an ACK makes forward progress On rollover (11→00): 4 failed replay attempts → force Physical Layer to retrain the link → LTSSM enters Recovery state
Figure 8 — REPLAY_TIMER fires when no ACK/NAK arrives in time. REPLAY_NUM counts replay attempts. After 4 failed replays (REPLAY_NUM rolls over from 11 to 00), the DLL forces link retraining — something is seriously wrong with the physical link and it needs to re-establish timing and equalization.

📋 ACK Latency Timer

The receiver’s ACK Latency Timer controls how long the receiver may hold off sending an ACK in order to batch multiple TLPs into one acknowledgement. When the timer expires, the receiver must send an ACK covering all TLPs it has received since its last ACK.

The timer value is calculated from the link speed, link width, and maximum payload size — it is set long enough to guarantee that a fast transmitter can keep its link busy without stalling waiting for ACKs, but short enough that the transmitter’s REPLAY_TIMER never expires before an ACK arrives during normal operation.

Link configurationApproximate ACK latency (typical)
Gen 1 x1, 128B max payload~237 symbol times ≈ 950 ns
Gen 1 x16, 128B max payload~48 symbol times ≈ 190 ns
Gen 3 x16, 256B max payloadMuch shorter due to 128b/130b and higher speed
Gen 6 x16Flit-level ACKs — see Gen 6 section below

📋 All Error Cases and How They Resolve

Error scenarioDetectionResolution
TLP has LCRC error Receiver DLL: recalculated CRC ≠ received LCRC Discard TLP. Send NAK(NRS−1). Transmitter replays from the failed TLP onward.
TLP lost en route (Physical Layer drops it) Next arriving TLP has a SeqNo gap (greater than expected) Discard the out-of-sequence TLP. Send NAK(NRS−1). Transmitter replays from the missing TLP.
ACK DLLP has CRC error Transmitter DLL: 16-bit DLLP CRC mismatch — discard silently No immediate action. Wait for next ACK or NAK. If REPLAY_TIMER expires, replay all. Often resolves harmlessly — next successful ACK covers all prior TLPs.
NAK DLLP has CRC error Transmitter DLL: 16-bit DLLP CRC mismatch — discard silently Receiver will not send another NAK (NAK_SCHEDULED prevents it). REPLAY_TIMER eventually expires, triggering replay. Receiver accepts replayed TLPs, clears NAK_SCHEDULED, sends ACK.
No ACK/NAK arrives at all REPLAY_TIMER expires at transmitter Replay entire buffer. Increment REPLAY_NUM. After 4 consecutive failures, force link retraining.
4 replay attempts all fail REPLAY_NUM rolls from 11b to 00b DLL forces LTSSM to Recovery state. Link retrains. If link cannot recover, link goes down — hardware error escalation to AER.
Duplicate TLP arrives (replay after already-acked TLP) SeqNo < NEXT_RCV_SEQ at receiver Silently discard duplicate. Send ACK(NRS−1) so transmitter knows current progress. No error raised.

📋 Transmit Priority Order

When the DLL has multiple things to transmit simultaneously, it must prioritise them correctly. The recommended order (highest priority first):

  1. Complete any TLP or DLLP already in progress on the link
  2. Ordered Sets (from the Physical Layer)
  3. NAK DLLPs
  4. ACK DLLPs
  5. Flow Control Update DLLPs
  6. Replayed TLPs from the Replay Buffer
  7. New TLPs from the Transaction Layer
  8. All other DLLPs (power management, vendor)

NAKs and ACKs are prioritised above new TLPs to ensure the transmitter on the remote end receives error feedback quickly. A delayed NAK means the remote transmitter must keep more TLPs in its Replay Buffer for longer — potentially stalling its Transaction Layer.

⚡ Data Link Layer in Gen 6

Gen 6 introduces flit-based framing at the Physical Layer, and this changes how the Data Link Layer’s reliability mechanism operates. The core concepts — Sequence Numbers, LCRC, Replay Buffer, ACK/NAK — are all preserved but operate at a different granularity.

Flit-level vs TLP-level reliability

Gen 1–5 vs Gen 6 — Reliability Granularity Gen 1–5 — Per-TLP Sequence Numbers TLP SeqNo=5 TLP SeqNo=6 TLP SeqNo=7 Each TLP has its own 12-bit SeqNo + LCRC ACK/NAK acknowledges individual TLPs Error on TLP 6 → replay TLP 6, 7 only Physical Layer: STP/END or sync headers per TLP Gen 6 — Per-Flit Sequence Numbers Flit SeqNo=N · TLP5 + TLP6 + TLP7 packed Flit SeqNo=N+1 Each 256-byte flit has its own SeqNo FEC corrects errors within a flit first If FEC cannot correct → replay entire flit All TLPs in a replayed flit replay together
Figure 9 — Gen 1–5 assign Sequence Numbers per TLP. Gen 6 assigns Sequence Numbers per flit. Multiple TLPs are packed into each 256-byte flit. FEC corrects most errors without any replay. Only uncorrectable flit errors trigger ACK/NAK replay at the flit level. A replayed flit brings all its TLPs with it.

How Gen 6 DLL reliability works

Gen 6 in practice: fewer replays, simpler ACK/NAK. Because FEC handles the vast majority of bit errors silently, Gen 6 links experience far fewer NAK-triggered replays than Gen 5 links operating on the same physical channel. The ACK/NAK protocol is still there — it handles the rare uncorrectable flit errors — but the typical Gen 6 link operates almost entirely in the FEC-correction path, with the DLL replay mechanism as a fallback for catastrophic errors. This means the Replay Buffer in a Gen 6 device can be smaller than an equivalent Gen 5 device operating at the same throughput.

📋 Quick Reference

ConceptKey Point
DLL purposeGuaranteed per-hop TLP delivery. Detects and automatically corrects transmission errors without software involvement.
DLL wrappingPrepends 12-bit Sequence Number · Appends 32-bit LCRC (covers SeqNo + header + payload + ECRC)
DLLP8-byte packets local to one link. Not routed. Never reach Transaction Layer. 4 bytes content + 2-byte CRC + 2-byte Physical Layer framing.
ACK DLLPType byte=0x00. Carries 12-bit SeqNo of last successfully received TLP. Cumulative — one ACK covers all TLPs up to that SeqNo.
NAK DLLPType byte=0x10. Carries 12-bit SeqNo of last good TLP (NRS−1). Transmitter replays from (SeqNo+1) onward.
NEXT_RCV_SEQ (NRS)12-bit counter at receiver. Tracks expected next Sequence Number. Increments on every good TLP forwarded to Transaction Layer.
Sequence Number outcomesSeqNo=NRS: good, accept. SeqNo < NRS: duplicate, discard silently. SeqNo > NRS: gap, NAK.
LCRC32-bit CRC per TLP. Per-hop only — stripped and recalculated at each switch. Does not protect against in-switch fabric corruption (that is ECRC’s job).
Replay BufferStores copy of every sent-but-unacked TLP. Purged when ACK covers the TLP. Replayed on NAK or REPLAY_TIMER expiry.
NAK_SCHEDULED flagSet when receiver sends a NAK. Cleared only when the expected TLP arrives correctly. Only one NAK outstanding at a time.
REPLAY_TIMERWatchdog. Fires if no forward progress from ACK/NAK. Triggers full buffer replay. Resets when ACK makes progress or buffer empties.
REPLAY_NUM2-bit counter. Counts consecutive replay attempts. Rolls 11→00 after 4 failures → forces Physical Layer retraining (LTSSM Recovery).
ACK Latency TimerReceiver coalesces ACKs until timer fires. Ensures ACK arrives before transmitter’s REPLAY_TIMER. Tuned per link speed/width/MPS.
Transmit priorityIn-progress packet → Ordered Sets → NAK → ACK → FC Updates → Replay TLPs → New TLPs → Other DLLPs
Gen 6 — FEC firstRS FEC corrects most errors silently. ACK/NAK operates at flit level (not TLP level). Fewer replays in practice. Same REPLAY_TIMER / REPLAY_NUM logic.
Gen 6 — per-flit SeqNoSequence Numbers and Replay Buffer operate on 256-byte flits. A NAK replays the entire flit including all TLPs inside it.
Scroll to Top