Post 6.7 – Credit Tracking Example and Debugging in PCI Express

Understanding live credit flow, link stalls, and verification strategies

---

1 . Introduction

Flow control in PCIe is invisible during normal operation — it works silently behind every packet.
However, debugging or verifying PCIe requires you to see how credits move and confirm that every transmission follows the credit rules.

In this post, you’ll walk through real credit flow examples, learn to identify credit starvation, and get hands-on debugging techniques used by protocol engineers.

---

2 . Credit Lifecycle Recap

The credit system manages buffer space between transmitter and receiver using six counters per Virtual Channel:

Credit Type	Purpose
PH	Posted Header
PD	Posted Data
NPH	Non-Posted Header
NPD	Non-Posted Data
CPLH	Completion Header
CPLD	Completion Data

The transmitter’s local credit counters are initialized from the receiver’s advertised capacities, then continuously updated via Flow Control DLLPs.

---

3 . Visualizing Credit Flow

Let’s look at the credit movement sequence between a Root Complex (RC) and Endpoint (EP):

[1] Link enters L0

     ↓

[2] EP advertises credits (PH=8, PD=32)

     ↓

[3] RC transmits a Memory Write TLP (1 PH + 8 PD)

     ↓

[4] RC decrements PH/PD counters (PH=7, PD=24)

     ↓

[5] EP processes the TLP and frees buffer

     ↓

[6] EP sends Flow Control Update DLLP

     ↓

[7] RC increments PH/PD counters

This loop continues as long as traffic is active.
If any credit counter hits zero, transmission of that TLP type pauses until replenished.

---

4 . Example 1 – Posted Write Flow

Initial State:

• Receiver (EP) advertises PH=8, PD=32
  
• Transmitter (RC) starts with same counters
  

Operation Sequence:

Step	Event	PH	PD
1	Send 1st Write (4 DW payload)	7	28
2	Send 2nd Write (8 DW payload)	6	20
3	Send 3rd Write (16 DW payload)	5	4
4	Next Write (8 DW) blocked – PD < 8	—	—
5	FC Update DLLP arrives (+8 PD)	5	12
6	Write resumes	4	4

💡 Flow Control Update DLLPs restore data credits as the receiver empties its posted data buffers.

---

5 . Example 2 – Read/Completion Flow

1. Root Complex → Memory Read (Non-Posted):
   
   • Consumes 1 NPH credit.
     
2. Endpoint → Completion with Data:
   
   • Consumes 1 CPLH + N CPLD (payload size in DW).
     
3. Root Complex processes Completion:
   
   • Sends Flow Control Update DLLP for CPLH/CPLD.
     
4. Endpoint credits increment → ready for next completion.
   

This ensures bidirectional synchronization — reads drive completions, completions return credits.

---

6 . Example 3 – Mixed Traffic with VCs

In a system with VC0 (bulk data) and VC1 (control):

VC	Credit Type	Active Packets	Status
VC0	PH/PD	Memory Writes	PD almost exhausted
VC1	NPH	Config Reads	Still active
VC1	CPLH/CPLD	Completions	Unaffected

Even if VC0 stalls due to PD depletion, VC1 continues transmitting because its flow control is independent.
This demonstrates credit isolation across Virtual Channels.

---

7 . Common Flow Control Debug Issues

Issue	Symptom	Possible Root Cause
Credit starvation	Link stalls unexpectedly	Transmitter ran out of credits; receiver not sending FC updates
Credit mismatch	Continuous stalls or deadlock	Incorrect advertised credit count
Ack/Nak storm	Excessive DLLPs seen	Replay buffer issues or LCRC errors
Uneven credit replenishment	Only one credit type refilled	Partial DLLPs or VC misconfiguration
Flow Control DLLPs missing	No updates observed	Receiver logic not generating FC updates

---

8 . Debugging Techniques

A. Using a Protocol Analyzer

• Look for Flow Control DLLPs between bursts of TLPs.
  
• Confirm that the frequency matches traffic intensity.
  
• Check that credit values in the DLLP fields are changing dynamically.
  

Expected Pattern:

TLP Burst → Pause → Flow Control DLLP → Resume Transmission

If the pause never ends → credit starvation.

---

B. In Simulation

Use scoreboards and assertions to track credits.

// Check before transmission

assert(credits_ok(type)) else

  $fatal(“TLP sent without enough credits: %s”, type);

// Track replenishment

cover(property (@(posedge clk)

   credits_increased_after_fc_update));

Visualizing credit counters in waveform viewers (e.g., SimVision, DVE) helps correlate stalls with Flow Control Update DLLPs.

---

C. Hardware Debug

In real hardware or FPGA prototyping:

• Check for credit starvation counters in PCIe IP debug registers.
  
• Use logic analyzers or protocol checkers to verify FC Update intervals.
  
• Many IPs (like Synopsys, Cadence, Xilinx) expose “Credit Available” status bits.
  

---

9 . Tips for Flow Control Verification

✅ Initialization must complete before any TLP is sent.
✅ All credits must be replenished periodically under continuous load.
✅ Replay events must not affect credit accounting.
✅ VC arbitration must not block higher-priority completions.
✅ Credit counters should never go negative.

---

NOTE:

• Credits continuously cycle through consumption (Tx) and replenishment (Rx).
  
• Flow Control DLLPs are essential for keeping the link alive and balanced.
  
• Each VC’s credits are tracked independently to prevent starvation.
  
• Most PCIe link stalls trace back to credit starvation or update synchronization errors.
  
• Credit observability and verification are crucial for a robust PCIe design.
  

---