Flow Control

📋 Why Flow Control Exists

PCIe is a packet-switched, full-duplex link. At Gen 6 speeds a x16 link can deliver ~122 GB/s in each direction simultaneously. A device at one end of the link has finite receive buffers — if the transmitter sends TLPs faster than the receiver can process and consume them, the buffers overflow and TLPs are dropped.

PCI solved this with a very different mechanism: bus-wide retry signals (STOP#, RETRY#). If a target was busy it asserted RETRY# and the initiator tried again. On a shared bus this is cheap — the bus is stalled anyway. On a PCIe point-to-point link there is no retry signal. Dropping a TLP and retransmitting it costs a full round-trip latency and wastes link bandwidth.

PCIe’s answer is credit-based flow control: the receiver tells the transmitter exactly how much buffer space it has available. The transmitter never sends more than what the receiver has space to hold. No overflow, no drops, no retries — just a clean credit accounting system operating at the Data Link Layer.

📋 The Credit Concept

A credit is a unit of receive buffer space. The receiver counts its available buffer space in credits and advertises that count to the transmitter. The transmitter maintains a running credit balance. Before sending any TLP, it checks whether it has enough credits for that TLP. If yes, it deducts credits and sends the TLP. If no, it waits.

Credits are not consumed permanently. When the receiver’s Transaction Layer processes a TLP and frees the buffer space that held it, the receiver returns those credits to the transmitter via an UpdateFC DLLP. The transmitter adds them back to its balance. This circulation continues indefinitely during normal operation.

📋 Six Credit Types

Flow control credits are tracked independently for six buffer types. The three TLP categories (Posted, Non-Posted, Completion) each have separate header and data credit pools, giving six credit types in total. This separation is what allows the mandatory ordering rules (completions must pass non-posted) to be implementable — they use physically separate credit pools.

📋 Credit Unit Sizes

A credit is not a byte — it is a fixed-size unit chosen to balance accounting precision against counter width. Different credit types have different unit sizes:

Worked example — credits consumed by one TLP

A Memory Write (MWr) with a 256-byte payload sent to a receiver that has advertised PH and PD credits:

• PH credits consumed: 1 (one header, regardless of payload size)
• PD credits consumed: 256 bytes ÷ 16 bytes per credit = 16
• NPH, NPD, CPLH, CPLD: unchanged — MWr does not use these pools

📋 Transmitter Credit Check

Before the transmitter DLL passes any TLP to the Physical Layer, it performs a credit check. The check is a simple comparison of two counters per credit type:

• CREDITS_LIMIT — the total credits available from the receiver (updated by UpdateFC DLLPs)
• CREDITS_CONSUMED — how many credits have been used by TLPs sent but not yet returned

A TLP may be sent if and only if: CREDITS_LIMIT − CREDITS_CONSUMED ≥ credits_needed_for_this_TLP

This check is performed independently for each credit type. A TLP can only be sent when all relevant credit pools (header + data, for the appropriate TLP category) have sufficient space. If either pool is exhausted, the transmitter must wait — the TLP sits in the Transaction Layer’s egress queue until UpdateFC returns enough credits.

📋 FC DLLP Format

All Flow Control DLLPs — InitFC1, InitFC2, and UpdateFC — use the same 8-byte format. Only the type byte changes.

📋 FC Initialisation — Two Phases

No TLPs may be sent on a link until flow control initialisation is complete. This process happens automatically in hardware immediately after Physical Layer link training succeeds. It involves a two-phase handshake (FC_INIT1 and FC_INIT2) carried out simultaneously in both directions.

Why two phases?

The two-phase design handles the case where the two devices finish phase 1 at different times. If Device A finishes phase 1 before Device B, A transitions to phase 2 and starts sending InitFC2 DLLPs. Device B ignores InitFC2 while it is still in phase 1 waiting to receive enough InitFC1 repetitions to be confident the values were received reliably. Since A is still sending InitFC1 DLLPs during phase 2 (they carry the same credit values), B eventually receives them, completes phase 1, and transitions to phase 2. Both sides then converge to DL_Active.

📋 Infinite Credits and Why They Exist

A credit value of 0x000 during FC initialisation has a special meaning: infinite credits. A receiver that advertises infinite credits is guaranteeing that its receive buffer for that credit type will never overflow — the transmitter may send without checking that credit pool.

Which devices advertise infinite credits

📋 UpdateFC — Returning Credits

After FC initialisation, the receiver continuously sends UpdateFC DLLPs as its Transaction Layer processes TLPs and frees buffer space. UpdateFC DLLPs use the same format as InitFC DLLPs (same byte layout) but with type bytes in the 0x80/0x90/0xA0 range instead of the 0x40/0x50/0x60 range.

UpdateFC carries a cumulative count — the total number of credits that have ever been available to the transmitter, not just the credits being returned in this DLLP. This means a single delayed UpdateFC DLLP does not create a problem — the next one that arrives will contain the accumulated total, including all the credits the previous DLLP would have returned.

📋 Counters Inside the Transmitter

The transmitter maintains two counters per credit type, per Virtual Channel. Together they implement the credit check:

Available credits at any moment = CREDITS_LIMIT − CREDITS_CONSUMED (modulo 2¹²). The check is performed as a signed comparison accounting for the wrap-around nature of the counter — the two values must never be more than half the counter range (2048) apart, or the counter semantics become ambiguous.

📋 Minimum and Maximum Advertisements

📋 Per-Virtual-Channel Independence

All six credit types are tracked independently per Virtual Channel. A 6-credit-type × 8-VC system has 48 independent credit pools. Each VC initialises its own flow control when that VC is enabled by software — VC0 initialises automatically (it is always enabled), VC1–VC7 initialise when software enables them.

This per-VC independence is what enables Quality-of-Service guarantees. A high-priority video stream in VC7 can be guaranteed a minimum bandwidth allocation because its credit pool is completely separate from the VC0 best-effort DMA traffic pool. Even if VC0 PD credits are exhausted by a flood of best-effort writes, VC7 PD credits are unaffected and video TLPs continue to flow.

⚡ Flow Control in Gen 6

The flow control credit model — six types, credit unit sizes, FC_INIT handshake, infinite completion credits, UpdateFC return — is completely unchanged in Gen 6. Flow control is a Transaction Layer and Data Link Layer mechanism. Gen 6 changes only the Physical Layer.

FC DLLPs inside Gen 6 flits

In Gen 1–5, FC DLLPs travel between TLPs in inter-packet gaps or dedicated framing windows. In Gen 6, there are no inter-packet gaps — TLPs are packed continuously into 256-byte flits. FC DLLPs are embedded inside flits alongside TLPs. The flit header contains a field that identifies which bytes within the flit carry DLLPs. The receiver’s Data Link Layer unpacks DLLPs from the flit before passing TLPs to the Transaction Layer.

The DLLP content — type byte, HdrFC, DataFC, CRC — is identical. Only the physical packaging changes. An UpdateFC-P DLLP carrying HdrFC=42, DataFC=100 has exactly the same bit pattern in Gen 6 as in Gen 1.

Credit check is unchanged per TLP

In Gen 6, multiple TLPs are packed into one flit. The credit check still happens per-TLP, not per-flit. Before each TLP is added to a flit, the transmitter checks whether sufficient credits exist for that TLP. The flit packing is a Physical Layer concern — by the time the TL hands TLPs to the DLL for flit packing, the credit check has already been performed and cleared.

Gen 6 and high-bandwidth credit scaling

At ~122 GB/s on a Gen 6 x16 link, credit pools drain far faster than at Gen 1 speeds. A receiver’s PD credit pool of 128 credits (2048 bytes) at Gen 1 would take roughly 800 µs to drain. At Gen 6, the same pool drains in nanoseconds. This means UpdateFC DLLPs must be sent far more frequently in Gen 6 systems to avoid credit starvation. Hardware implementations of Gen 6 endpoints must service their UpdateFC generation at much tighter intervals than Gen 1 designs. The protocol is unchanged — the timing expectations are far more aggressive.

📋 Quick Reference