In previous lectures, we discussed how PCI Express (PCIe) completely transformed data transfer by moving from a legacy parallel bus to a packet-based serial connection. However, moving to a serial connection presented a unique physical challenge: what to do with all the dedicated physical wires (side-band signals) that older architectures used for system interrupts, error reporting, and power management?
Here is a look at how PCIe elegantly eliminated the need for these physical pins using Message TLPs and a clever technique called Implicit Routing.
Replacing Wires with Message TLPs
To achieve its design goal of drastically reducing the physical pin count on the hardware, PCIe introduced a brand-new type of transaction: the Message TLP.
Instead of relying on dedicated side-band signals, PCIe communicates critical system events using in-band packets that travel directly over the standard data Links. By acting as “virtual wires,” Message TLPs are used to convey a wide variety of system events, including:
- Power Management events.
- Legacy INTx interrupt signaling.
- Error signaling.
- Hot Plug signaling and Slot Power Limit settings.
- Locked Transaction support and Vendor-specific signaling.
The Genius of Implicit Routing
While Message TLPs can technically be routed using a target memory Address or a specific device ID (BDF), the vast majority of messages utilize a highly efficient method known as implicit routing.
In implicit routing, the packet does not need to contain the exact memory address or ID of its final destination. Instead, the packet header contains a simple 3-bit routing code that specifies a general, well-known destination within the PCIe topology.
Navigating Upstream and Downstream
Implicit routing works seamlessly because every Switch and routing element within the PCIe tree inherently understands the directional concepts of “upstream” and “downstream”. The hardware naturally knows that the Root Complex always sits at the very top of the topology, while the Endpoints sit at the absolute bottom.
Because of this built-in directional awareness, switches can easily route messages without needing a specific address. Here is how devices handle these implicit routing codes:
- Route to the Root Complex (Upstream): If an Endpoint encounters an error or fires an interrupt, it generates a message that is implicitly routed to the Root Complex. When a Switch receives this message on any Downstream Port, it simply forwards the packet directly out of its Upstream Port, passing it up the chain until it successfully reaches the Root Complex.
- Broadcasts (Downstream): If the Root Complex needs to announce an event to the entire system, it generates a Broadcast message. When a Switch receives this Broadcast message on its Upstream Port, it automatically duplicates the packet and forwards it out of every single Downstream Port, ensuring all branches of the tree receive the message.
- Directional Error Checking: Because this directional flow is strictly defined by the protocol, switches actively police the traffic. For example, if a Switch receives an implicitly routed Broadcast message on a Downstream Port (meaning the broadcast is traveling backward, or upstream), it recognizes the violation and automatically handles the packet as a Malformed TLP.
- Endpoint Handling: Because Endpoints sit at the very bottom of the branches, their job is simple. They check the routing sub-field and accept messages appropriate for them (such as Broadcasts or messages that implicitly terminate at the receiver), while ignoring messages that implicitly target the Root Complex.
By replacing dozens of physical side-band pins with implicitly routed Message TLPs, PCIe drastically reduced manufacturing complexity while maintaining full support for crucial legacy system events.