In previous lectures, we learned that PCI Express (PCIe) shifted away from a shared parallel bus to a high-speed, point-to-point serial connection. Because a single point-to-point Link can only connect two interfaces together, building a complete system requires a way to fan out connections to multiple devices.
To maintain crucial backward compatibility with legacy PCI software, PCIe organizes these connections into a simple “inverted tree” topology, meaning no loops or complex routing structures are allowed.
Here is a breakdown of the core elements that make up the PCIe tree structure.
1. The Root Complex (RC)
At the very top of the PCIe hierarchy is the Root Complex.
- The Function: The Root Complex serves as the primary interface between the system’s processor (CPU), the memory, and the rest of the PCIe topology. It acts on behalf of the CPU to communicate with all other devices in the system.
- The Hardware: The Root Complex is not necessarily a single chip. It may contain multiple components, such as processor interfaces and DRAM interfaces. In many modern architectures, the memory controller and routing logic of the Root Complex are actually integrated directly into the CPU package itself (often referred to as “Uncore” logic).
2. Switches
Because the Root Complex only has a limited number of “Root Ports,” the system needs a way to expand the number of devices it can support. This is the job of Switches.
- The Function: Switches provide fanout or aggregation capabilities, allowing multiple devices to attach to a single upstream PCIe port. They act as highly efficient packet routers, recognizing which path a specific packet needs to take based on its address or routing information.
- The Structure: Every Switch has exactly one Upstream Port (the port facing toward the Root Complex) and can have several Downstream Ports (facing away from the Root Complex). To the system’s configuration software, a Switch simply appears as a collection of virtual PCI-to-PCI bridges sharing an internal bus.
3. Bridges
While Switches route traffic within the PCIe fabric, Bridges exist to provide an interface to completely different bus protocols.
- Forward Bridges: These allow an older legacy bus card (like PCI or PCI-X) to be plugged into a modern PCIe system.
- Reverse Bridges: These do the opposite, allowing a modern PCIe card to interface with an older PCI system.
4. Endpoints (Native vs. Legacy)
At the very bottom of the branches in the PCIe tree topology are the Endpoints. These are the actual peripheral devices doing the work, such as an Ethernet controller or a graphics card. Endpoints only implement a single Upstream Port and act as the ultimate initiators and Completers of transactions on the bus.
PCIe defines two distinct classes of Endpoints:
- Native PCIe Endpoints: These are devices designed from scratch specifically for the PCIe architecture. They operate purely as Memory-Mapped IO (MMIO) devices.
- Legacy PCIe Endpoints: These are devices that were originally designed for older bus operations (like PCI-X) but have been adapted with a new PCIe interface. Because of their legacy roots, these Endpoints designate themselves as “Legacy” so they are allowed to make use of older features that are strictly prohibited in newer Native PCIe designs, such as utilizing IO address space, handling IO transactions, and supporting Locked requests.