Architecture — Topology and Components

📋 The Tree Rule and Why It Exists

PCIe topologies must be strict trees — no loops, no rings, no meshes. Every node connects to exactly one upstream parent and zero or more downstream children. This is a deliberate constraint inherited from PCI, and it exists for one reason: software backward compatibility.

PCI’s configuration software uses a simple recursive depth-first algorithm to enumerate buses. The algorithm assigns bus numbers in the order it discovers bridges. It works perfectly on trees. It breaks completely on graphs with loops. Rather than require a new OS to be written for PCIe, the PCISIG preserved the tree constraint — and because the constraint is preserved, every PCI configuration driver ever written works unmodified on a PCIe system.

📋 Bus, Device, Function — BDF Addressing

Every PCIe Function is uniquely identified by a 16-bit BDF address: 8 bits of Bus Number, 5 bits of Device Number, and 3 bits of Function Number.

• Bus Number (8 bits) — up to 256 buses in one domain (0–255). Bus 0 is always the Root Complex’s internal virtual bus. Configuration software assigns all other bus numbers during enumeration.
• Device Number (5 bits) — up to 32 device slots per bus. Because PCIe links are point-to-point, a PCIe endpoint always ends up as Device 0 on its bus. The Root Complex and Switches have virtual buses that can have multiple device slots for their embedded ports.
• Function Number (3 bits) — up to 8 functions per device. Most devices have only Function 0. Multi-function devices (a network card with a storage controller on the same chip, for example) can expose multiple functions, each with its own configuration space.

The BDF is called the Requester ID when embedded in a request TLP (so the completer knows where to send the completion) and the Completer ID when embedded in a completion TLP (so the requester can match it to its outstanding request). Tags — a separate 8-bit or 10-bit field — distinguish multiple in-flight requests from the same function.

📋 Root Complex Internals

From the outside, the Root Complex looks like a single block at the top of the PCIe tree. From the inside — and from software’s perspective — it is a collection of bridges sharing a virtual internal bus.

What the spec says about the RC

The PCIe spec deliberately does not fully define the Root Complex internals. It gives a list of required and optional behaviours but leaves the implementation to the vendor. In broad terms the RC must:

• Present a virtual PCI Bus 0 to configuration software, with Root Ports appearing as PCI-to-PCI bridges
• Generate and accept PCIe transactions on behalf of the CPU and system memory
• Support the full TLP routing and flow-control mechanisms
• Implement ACS (Access Control Services) in Gen 3+ for peer-to-peer isolation

In modern systems the RC is split across the CPU package (memory controller, high-bandwidth root ports for GPU) and the Platform Controller Hub or I/O Hub (USB, SATA, lower-bandwidth PCIe slots). The CPU ↔ PCH link (Intel’s DMI, AMD’s equivalent) is itself based on PCIe electricals. Both sides together are the RC from software’s point of view.

📋 Switch Internal Structure

A Switch looks deceptively simple from the outside. From the inside it is a collection of PCIe ports connected by an internal virtual bus — and that internal bus is what software enumerates as PCI bridges.

How a Switch routes a packet

When a TLP arrives at the Switch’s upstream port, the switch does the following:

• Memory/IO TLP: checks the target address against each downstream port’s Base/Limit registers (programmed during enumeration). If the address falls within a port’s range, the packet is forwarded out that port.
• Config TLP (Type 1): the TLP header contains a Bus Number. The switch checks if that bus is downstream of one of its ports and forwards it on. When the target bus is the secondary bus of a downstream bridge, the Type 1 config TLP becomes a Type 0 config TLP on exit.
• Completion TLP: routed by Requester ID — the bus number in the Requester ID field tells the switch which port is upstream toward the original requester.
• Message TLP: routed based on the routing field in the message header — some go up (to Root), some go down (broadcast), some go to a specific address.

📋 Endpoint Types

The endpoint type is declared in a field in the PCIe Capability structure in configuration space. Software reads this field to understand what the device is and what it can do. All modern devices designed specifically for PCIe (Gen 3 and later) are Native PCIe Endpoints and use only memory-mapped I/O.

📋 Every Port Implements All Three Layers

This applies without exception — Root Ports, Switch upstream ports, Switch downstream ports, and Endpoint ports all implement the Transaction Layer, Data Link Layer, and Physical Layer. The logic is the same regardless of the device type; what varies is what the Transaction Layer does with the decoded TLP (route it, deliver it to the device core, or generate a completion).

📋 Transaction Types — Posted vs Non-Posted

Every PCIe transaction is either posted or non-posted. The distinction drives fundamental protocol differences.

📋 All TLP Types

▶ Memory Read — End-to-End Example

Let’s walk through an NVMe SSD reading data from system memory. The SSD is behind a Switch. This example shows every hop and every packet.

▶ Memory Write — Posted Example

A GPU DMA engine writing rendered frame data to system memory. Posted — no completion comes back.

📋 ACK/NAK Protocol — Per-Hop Reliability

PCIe’s Data Link Layer guarantees delivery of every TLP between adjacent devices using ACK and NAK DLLPs. This is important to understand: it is per-hop, not end-to-end.

• The Sequence Number is 12 bits (0–4095), wrapping around. Transmitters use it to identify which TLPs the receiver is acknowledging or rejecting.
• An ACK for sequence number N means “I have received everything up to and including N — you can discard all of those from your replay buffer.”
• A NAK for sequence number N means “I found a problem at N — please resend from N onwards.”
• If no ACK or NAK arrives within a timeout period, the transmitter replays all unacknowledged TLPs.
• In Gen 6, the replay mechanism is modified to work within the flit-based framing model — but the fundamental ACK/NAK logic is preserved.

📋 Bus Enumeration — Step-by-Step Walk-Through

Enumeration is the process by which BIOS/OS firmware discovers the PCIe topology and assigns bus numbers to every bus segment. It uses a depth-first search — it always goes as deep as possible before backtracking.

⚡ Gen 6 Topology Considerations

The topology model — tree, BDF addressing, root/switch/endpoint roles, enumeration algorithm — is unchanged in Gen 6. What Gen 6 changes is the physical link speed and framing, not the topology or software model.

However, Gen 6’s higher bandwidth and flit-based framing introduce some practical topology considerations:

• Retimer requirements. At 64 GT/s PAM4, channel loss limits trace length to approximately 4–6 inches between a CPU root port and a device. Longer channels require PCIe Retimers — active clock-forwarding devices that regenerate the signal. Retimers are transparent to software (they have no BDF) but add latency. In data centre and AI systems, 1–2 retimers per link are common.
• CXL and Gen 6. CXL (Compute Express Link) 3.0 runs on the PCIe 6.0 PHY. The topology management for CXL 3.0 includes multi-headed and fabric topologies that go beyond PCIe’s strict tree — but the PCIe Gen 6 electrical layer is the same.
• No topology changes for software. A Gen 6 GPU still appears in the same BDF at Bus N/Dev 0/Function 0 as it always has. The enumeration algorithm is identical. Existing drivers work without change.
• Flit-mode impact on Switches. A Gen 6 Switch internally operates in flit mode — packing and unpacking flits as TLPs are routed between ports. The TLP routing decisions are identical, but the data is physically carried in 256-byte flit containers with FEC protection on each link.

📋 Quick Reference