In Lecture 17, we introduced the fundamentals of transaction ordering. Now, we are looking at the exact reason why those strict ordering rules exist in the first place: The Producer/Consumer Model.
The Producer/Consumer model is the most common method for data delivery in both legacy PCI and modern PCIe systems. In this lecture, we will trace the sequence of events that make this model work, and look at the disastrous race conditions that happen if ordering rules fail!
Part 1: The Five Elements of the Model
To pass data safely between two devices, the Producer/Consumer model relies on a strict handshake using five specific elements:
- The Producer: The device generating and sending the data.
- The Data Buffer: The target memory space where the data is temporarily stored.
- The Flag Semaphore: A marker used by the Producer to tell the Consumer, “I have finished sending the data.”
- The Consumer: The device receiving and processing the data.
- The Status Semaphore: A marker used by the Consumer to tell the Producer, “I have finished reading the data, you can send more.”
(Note: The Flag and Status semaphores can reside in the same physical device or different devices.)
Part 2: The Perfect Sequence (No Errors)
When the system is working perfectly, the delivery handshake looks like this:
The Producer Delivers Data:
- Step 1: The Producer sends one or more Memory Writes (Posted Requests) to move data into the Memory Buffer.
- Step 2: The Consumer periodically checks the Flag semaphore using a Memory Read (Non-Posted Request) to see if data is ready. Initially, the Flag is
0(not ready). - Step 3: Once the Producer finishes sending all the data, it sends a Memory Write to set the Flag semaphore to
1. - Step 4: The Consumer checks the Flag again, sees that it is
1, and knows the data is officially ready for consumption. - Step 5: The Consumer sends a Memory Write to clear the Flag back to
0.
The Consumer Reads Data:
- Step 6: The Consumer performs Memory Reads to fetch the newly delivered data from the Memory Buffer.
- Step 7: Once the Consumer has finished reading all the data, it sends a Memory Write to set the Status semaphore to
1. - Step 8: The Producer, which has been periodically checking the Status semaphore, sees it is now
1. This confirms the buffer is empty and it is safe to send the next batch of data.
Part 3: The Disaster of Race Conditions
The sequence above assumes that every packet arrives instantly and sequentially. But a PCIe topology is complex. What happens if there are no ordering rules enforced by the switches in the fabric? A race condition occurs.
Let’s look at how the sequence fatally fails without strict ordering rules:
- The Producer sends the Memory Write (data) to the buffer. However, the data gets temporarily stuck in a Switch’s flow control buffer due to a lack of credits.
- The Producer immediately follows up by sending a Memory Write to set the Flag =
1. - If there are no ordering rules, the Switch might allow the small Flag update packet to bypass the stalled data packets.
- The Consumer reads the Flag, sees a
1, and thinks the data is ready. - The Consumer eagerly reads the Memory Buffer, but because the real data is still stuck inside the Switch, the Consumer fetches old, invalid data.
Part 4: How PCIe Ordering Rules Save the Day
To completely avoid this race condition, PCIe enforces Strong Ordering for transactions that share the same Traffic Class (TC).
When the Consumer performs the Memory Read to check the Flag, the virtual PCI bridge inside the switch must not allow the contents of the flag to be forwarded ahead of the previously posted data. By rule, a subsequent transaction is absolutely not allowed to pass a previously Posted Request.
Because of this strict rule, the Flag update is mathematically guaranteed to stay “in line” directly behind the data payload. The Consumer will never see the Flag flip to 1 until every single byte of data has successfully cleared the switches and landed safely in the memory buffer!