PCIe Series — PCIe-06: Memory Read and Write TLPs — VLSI Trainers
PCIe Series · PCIe-06

Memory Read and Write TLPs

Full bit-level headers for MRd, MWr, and MRdLk — 3DW and 4DW variants side by side, Requester ID, Tag, Byte Enables explained with worked examples, address rules, the 4 KB boundary constraint, and how Gen 6 flit packing affects none of this.

📋 Memory TLPs — the Three Types

Memory transactions are the workhorse of PCIe. Almost every DMA transfer, GPU frame render, and NVMe command queue operation is a memory read or write. There are three memory TLP types:

Three Memory TLP Types MRd Memory Read Request Non-posted · no payload Type = 0_0000 · Fmt = 000 or 001 Completion with data (CplD) must return Covered in PCIe-07 MWr Memory Write Request Posted · carries data payload Type = 0_0000 · Fmt = 010 or 011 No completion — fire and forget DLL per-hop ACK still applies MRdLk Memory Read Locked Non-posted · legacy atomic support Type = 0_0001 · Fmt = 000 or 001 CplDLk returns · locks VC0 until Unlock Legacy devices only · avoid in new design
Figure 1 — Three memory TLP types. MRd and MWr differ only in Fmt (no-data vs with-data) and in being non-posted vs posted. MRdLk shares the MRd header layout but uses Type=0_0001 and triggers legacy lock behaviour in switches.

The key distinction between MRd and MWr is the Fmt field: MRd has Fmt bit[1] = 0 (no payload), MWr has Fmt bit[1] = 1 (payload follows header). The Type field is the same — 0_0000 — for both. The receiver tells them apart purely by the Fmt field before reading any further.

📋 MRd — 3DW Header (32-bit address)

Use the 3DW MRd when the target address is below 4 GB (fits in 32 bits). The header is 12 bytes — three Doublewords.

MRd 3DW Header — Memory Read, 32-bit Address Fmt = 000 · Type = 0_0000 · No data payload · 12 bytes total Byte 0 Byte 1 Byte 2 Byte 3 Fmt 000 [30:28] Type 0_0000 [27:24] · MRd TC · LN · TH · TD · EP · Attr · AT Byte 1 + Byte 2 — same as DW0 common fields Covered in PCIe-05 (common header) Length [9:0] DWs requested (not DWs sent — no payload) Byte 2 bits [1:0] + Byte 3 bits [7:0] DW0 Byte 4 Byte 5 Byte 6 Byte 7 Requester ID [15:0] Bus[15:8] · Device[7:3] · Function[2:0] Byte 4 = Bus · Byte 5 bits[7:3]=Dev · [2:0]=Fn Tag [7:0] Byte 6 — identifies this request Matched in CplD that comes back Last DW BE [7:4] — 4 bits 0000 if Length=1 First DW BE [3:0] — 4 bits Byte 7 bits [3:0] DW1 Byte 8 Byte 9 Byte 10 Byte 11 Address [31:2] Upper 30 bits of the 32-bit target address Bytes 8–11 bits [7:2] — 30 bits total R [1:0] = 00 DW align DW2 Header ends at Byte 11 · Total = 12 bytes · No data payload follows · DLL adds SeqNo before Byte 0 · LCRC after Byte 11
Figure 2 — MRd 3DW header (Fmt=000). DW0 carries Fmt+Type+TC+flags+Length. DW1 carries Requester ID (16-bit BDF) + Tag (8-bit) + Last DW BE (4-bit) + First DW BE (4-bit). DW2 carries the 30-bit address (bits [31:2]), with bits [1:0] always 00 — the address is always DW-aligned. No payload follows.

📋 MRd — 4DW Header (64-bit address)

Use the 4DW MRd when the target address is ≥ 4 GB (requires more than 32 bits). The header grows to 16 bytes — four Doublewords. DW0 and DW1 are identical to the 3DW variant. DW2 and DW3 together carry the full 64-bit address.

MRd 4DW Header — Memory Read, 64-bit Address Fmt = 001 · Type = 0_0000 · No data payload · 16 bytes total Byte 0–3 DW0: Fmt=001 · Type=0_0000 · TC · LN · TH · TD · EP · Attr · AT · Length[9:0] Identical to 3DW DW0 except Fmt=001 (4DW header, no data) DW0 Byte 4–7 Requester ID [15:0] Bus · Device · Function Bytes 4–5 Tag [7:0] Byte 6 Last DW BE [7:4] Byte 7 bits [7:4] First DW BE [3:0] Byte 7 bits [3:0]DW1 Byte 8–11 Address [63:32] — Upper 32 bits of 64-bit address Bytes 8, 9, 10, 11 — all 32 bits used · no reserved bits in upper DW For addresses < 4GB this should be 0x00000000 (but 3DW header is preferred) DW2 Byte 12–15 Address [31:2] — Lower 30 bits of address Bytes 12–15 bits [7:2] — 30 bits · bits [1:0] always 00 DW-aligned address — lower 2 bits forced to 0 R [1:0] always 00 DW3
Figure 3 — MRd 4DW header (Fmt=001). DW0 and DW1 are identical to the 3DW variant. DW2 carries Address[63:32] — the upper 32 bits. DW3 carries Address[31:2] plus two reserved zero bits. The total header is 16 bytes. No data payload follows in either the 3DW or 4DW MRd.
The spec says: “An address below 4 GB must use a 3DW header. The spec states that receiver behaviour is undefined if a 4DW header is used for an address below 4 GB with the upper 32 bits of the 64-bit address set to zero.” In other words — if your address fits in 32 bits, always use the 3DW variant. Using 4DW for a sub-4GB address wastes 4 bytes of header and may confuse some receivers.

📋 Requester ID and Tag Field

DW1 of every memory request header carries two fields that together form the Transaction ID — the unique identity of this request within the PCIe fabric:

Transaction ID = Requester ID + Tag — How They Work Together Requester ID (16 bits) = BDF of sender Bus [15:8] 8 bits · 0–255 Device [7:3] 5 bits · 0–31 Function [2:0] 3 bits · 0–7Completer echoes this in the CplD — Switch uses it to route CplD back Example: NVMe SSD at BDF 03:00.0 → Requester ID = 0x0300 Tag (8 bits) — identifies this specific request Standard Tag 8 bits = 256 tags Extended Tag 10 bits = 1024 tagsSender assigns a unique Tag per in-flight request CplD echoes it → sender matches CplD to outstanding MRd Transaction ID = Requester ID (16-bit BDF) + Tag (8–10 bits) This combination is globally unique in the fabric — no two outstanding requests from the same function may share a Tag value
Figure 4 — Transaction ID composition. Requester ID is the BDF of whoever sent the request. Tag is a counter the sender increments per request. Together they uniquely identify every outstanding non-posted transaction. The completer echoes both in the CplD so routing and matching work correctly.

Tag field modes

ModeBits usedMax outstanding requests per functionHow to enable
Default8 bits (Tag[7:0])256On by default in all PCIe devices
Extended Tag10 bits (Tag[9:0])1024Set Extended Tag Enable bit in Device Control register · both endpoints must support it
Why does Gen 6 benefit from extended tags? At 64 GT/s per lane, a x16 Gen 6 link can transfer ~122 GB/s. An NVMe SSD reading 4 KB blocks would need roughly 30 million requests per second to saturate that link. With only 256 tags at 8 bits, the sender could run out of unique tags for in-flight requests, forcing it to wait for completions before sending new requests. Extended 10-bit tags (1024 values) are therefore strongly recommended for high-bandwidth Gen 6 devices.

📋 Byte Enables — First DW and Last DW

Byte 7 of every memory request header carries two 4-bit Byte Enable fields. They select which individual bytes within the first and last Doublewords of the transfer are active. This is how PCIe supports sub-DW-granular transfers without needing a separate length-in-bytes counter.

Byte Enable Fields — Bit-to-Byte Mapping Byte 7 of DW1 header BE[7] Last Byte 3 BE[6] Last Byte 2 BE[5] Last Byte 1 BE[4] Last Byte 0 BE[3] First Byte 3 BE[2] First Byte 2 BE[1] First Byte 1 BE[0] First Byte 0 ← Last DW BE [7:4] → ← First DW BE [3:0] → Bit = 1 → that byte IS valid / active · Bit = 0 → that byte is NOT active (ignore / don’t update) 0xF / 0xF — all bytes valid First BE = 1111 · Last BE = 1111 All 4 bytes in first DW valid All 4 bytes in last DW valid Most common — full-DW transfers First BE=0xF · Last BE=0xF 0x1 / 0x0 — byte 0 only First BE = 0001 · Last BE = 0000 Only Byte 0 of the first DW is valid Last BE = 0000 means Length = 1 DW Single-byte access within a DW First BE=0x1 · Last BE=0x0 0xE / 0x3 — unaligned span First BE = 1110 · Last BE = 0011 Skip Byte 0 of first DW (start offset 1) Only Bytes 0–1 of last DW valid Unaligned transfer — spec allows this First BE=0xE · Last BE=0x3
Figure 5 — Byte Enable fields in Byte 7 of DW1. The lower nibble [3:0] applies to the first DW of the transfer; the upper nibble [7:4] applies to the last DW. Bit=1 means valid. When Length=1 (single DW transfer), Last BE must be 0x0 — only First BE is meaningful.

📋 Byte Enable Rules from the Spec

These rules come directly from Table 5-5 and the byte enable section of Chapter 5 in the MindShare PCIe specification:

#RuleWhy it matters
1Byte enable bits are high-true. A bit = 1 means that byte is valid; bit = 0 means ignore that byte.Receivers must check this before writing to registers or memory. A 0 bit means “leave that byte unchanged.”
2If Length = 1 DW, the Last DW BE must be 0x0 (all bits zero). Only First DW BE is used.There is only one DW — it is both the first and the last. Last BE would be redundant and must be cleared.
3If Length ≥ 2 DW, the First DW BE must have at least one bit set.A transfer of 2+ DWs with no valid bytes in the first DW would be nonsensical. At least one byte must be active in the starting DW.
4If Length ≥ 3 DW, both First and Last DW BE bits must be contiguous — no holes in the enable pattern.The middle DWs are always fully valid. Only the first and last DWs can have partial bytes. If Length ≥ 3, the BEs only define the start and end byte offsets — they cannot select non-contiguous bytes.
5Discontinuous BE patterns are allowed only if Length = 1.For a single-DW transfer you can have bits like 0101 (Bytes 0 and 2 but not 1 and 3). This is valid for narrow register accesses.
6A write with Length = 1 and no BEs set (all zero) is legal but has no effect on the completer.This is used as a memory-flush mechanism. The completer accepts the request and returns a completion but writes nothing. This forces all previously posted writes from the requester to drain through the switch before the completion returns.
7A read with Length = 1 and no BEs set causes the completer to return 1 DW of undefined data.Same flush purpose as rule 6 — the ordering rule ensures the completion cannot come back until all earlier posted writes have propagated. The data is meaningless and should be discarded.

▶ Byte Enable Worked Examples

These examples show how Byte Enables select the valid bytes for transfers of different sizes and alignments.

Byte Enable Examples — Memory Layout View DW Address Byte 3 Byte 2 Byte 1 Byte 0 BE value Length / Notes Example A — Read 4 bytes from 0x100 (full DW, aligned)0x100 valid valid valid valid F / 0 Length=1 · First BE=0xF · Last BE=0x0 Example B — Read 1 byte from 0x102 (byte offset 2 within DW)0x100 skip valid skip skip 4 / 0 Length=1 · First BE=0x4 (bit[2]=1) · Last=0x0 Example C — Read 6 bytes from 0x102 to 0x107 (spans 2 DWs) 0x100 valid valid skip skip E First BE 0x104 skip skip valid valid 3 Last BE Length=2 · First BE=0xE · Last BE=0x3 Addr=0x100 (DW aligned) · skip bytes 0–1 at start Example D — Read 12 bytes from 0x200 (3 full DWs)0x200 all valid (First DW) F First BE0x204 middle DW — always fully valid (no BE)0x208 all valid (Last DW) F Last BE Length=3 · First BE=0xF · Last BE=0xF
Figure 6 — Byte Enable examples showing which bytes are active (coloured) vs skipped (grey). Example A: full 4-byte DW read. B: single byte at offset 2. C: 6-byte unaligned read spanning two DWs — First BE=0xE skips byte 0, Last BE=0x3 takes only bytes 0–1. D: 3-DW aligned read — all bytes valid, middle DW has no BEs (always fully valid).

📋 MWr — Memory Write with Payload

MWr is the posted counterpart to MRd. The header layout is identical to MRd (same DW1 with Requester ID, Tag, BEs; same DW2/DW3 address structure). The two differences are in DW0: Fmt has the data bit set (Fmt bit[1]=1), and the data payload immediately follows the last header DW.

MWr — Header + Payload Layout (4DW shown) Fmt = 011 · Type = 0_0000 · Payload follows header · Posted — no completion DW0: Fmt=011 (4DW+data) · Type=0_0000 · TC · Flags · Length Fmt=010 for 3DW MWr (32-bit address) · Fmt=011 for 4DW MWr (64-bit address) DW0 DW1: Requester ID [31:16] · Tag [15:8] · Last DW BE [7:4] · First DW BE [3:0] Identical to MRd DW1 — same layout, same fields DW1 DW2: Address [63:32] — upper 32 bits (4DW only; not present in 3DW) DW2 DW3: Address [31:2] — lower 30 bits R DW3 Data Payload — 1 to 1024 DW (4 to 4096 bytes) Immediately follows last header DW · Length field in DW0 specifies how many DWs
Figure 7 — MWr 4DW header + payload. The header is identical to MRd except Fmt=011 (data present). The payload follows DW3 immediately — no gap. Since MWr is posted, there is no completion TLP returned. The Data Link Layer’s per-hop ACK DLLP still confirms delivery to the adjacent device, but that ACK is invisible to the Transaction Layer.

MWr vs MRd — the two differences

PropertyMRdMWr
Fmt bit[1] (data present)0 — no payload1 — payload follows
Transaction typeNon-postedPosted
Completion returned?Yes — CplD (data) from completerNo — fire and forget
Tag purposeMatches incoming CplD to outstanding readPresent in header but not used by completer for completion matching
Ordering ruleCannot pass earlier posted writesCannot pass earlier posted writes (strict ordering unless RO=1)
Flow control bufferNon-Posted (NPH + NPD credits)Posted (PH + PD credits)
Type[4:0] encoding0_00000_0000 (same — Fmt distinguishes)

📋 MRdLk — Memory Read Locked

MRdLk is a legacy transaction inherited from PCI’s locked transaction protocol. The header is identical to MRd except for the Type field: Type = 0_0001 instead of 0_0000. The Fmt values are the same (000 for 3DW, 001 for 4DW).

MRdLk is a legacy feature — avoid in new designs. When a switch sees an MRdLk it locks VC0 for all requesters except the one that issued the lock, preventing any other Posted writes through VC0 until an Unlock Message TLP arrives. This is a significant performance hazard in modern multi-device systems. The PCIe spec strongly discourages it — native PCIe Endpoints must not use MRdLk. It exists only for bridges to PCI/PCI-X legacy devices.
FieldMRdMRdLk
Fmt[2:0]000 (3DW) or 001 (4DW)000 (3DW) or 001 (4DW) — identical
Type[4:0]0_00000_0001
Header layoutRequester ID, Tag, BEs, AddressIdentical — same byte positions
Completion typeCplD — completion with dataCplDLk — locked completion with data
Side effect at switchesNoneLocks VC0 until Unlock Message received
Allowed onAll endpoint typesLegacy Endpoints and PCI/PCI-X bridges only

📋 Payload Rules

These rules apply to all TLPs that carry a data payload (MWr, CplD, IOWr, CfgWr). They come directly from Chapter 5 of the spec:

Payload Rules — Four Key Constraints ① Length refers to payload only The Length field in DW0 counts payload DWs — not header DWs. Header is always 3 or 4 DWs depending on Fmt. Length tells how many additional DWs of data follow the header. ② First payload byte = lowest address The byte immediately after the last header DW maps to the Address field value (start address). Bytes are in ascending address order with no gaps or padding. ③ Max_Payload_Size limit Payload must not exceed Max_Payload_Size in Device Control register. Values: 128, 256, 512, 1024, 2048, or 4096 bytes. Violation → Malformed TLP error (AER reported). ④ Always integral DWs (4-byte multiples) Length is always in whole DWs. Partial-DW granularity is handled by First and Last DW Byte Enables — the payload itself is always padded to a full DW count.
Figure 8 — Four payload rules from the spec. These apply to MWr and any other TLP with a data payload. The Length-in-DWs requirement and the Max_Payload_Size limit are the two most commonly encountered in practice.

📋 Address Rules — Alignment and the 4 KB Boundary

The address carried in any memory TLP header must follow two hard rules:

Two Hard Address Rules ① Address is always DW-aligned Address bits [1:0] are hardwired to 00 in every header. The PCIe address always points to a 4-byte (DW) boundary. To access a byte at offset N within a DW: → Address = DW-aligned base of that DW → Set the appropriate First DW BE bit Example: access byte at 0x103 → Address = 0x100, First BE = 0x8 (bit[3]=1 = Byte 3) (byte 3 of DW at 0x100 is address 0x103) ② No 4 KB boundary crossing A single TLP must not span a 4096-byte page boundary. Rule: (start_address + length_in_bytes) must not cross a multiple of 0x1000 (4096 bytes). 0x0000–0x0FFF 0x1000–0x1FFF ← 4 KB page boundary Checking this is optional for receivers but if seen it may be treated as a Malformed TLP by the completer.
Figure 9 — The two hard address rules. DW alignment is enforced by the reserved bits [1:0] in the header — the sender physically cannot express a non-DW-aligned address; byte granularity is achieved via Byte Enables. The 4 KB no-crossing rule means software DMA engines must split large transfers that cross page boundaries into two separate TLPs.

📋 When to Use 3DW vs 4DW

SituationUseReason
Target address ≤ 0xFFFF_FFFF (below 4 GB)3DW (Fmt=000/010)Shorter header saves 4 bytes per TLP. Spec says to use 3DW when address fits in 32 bits.
Target address > 0xFFFF_FFFF (≥ 4 GB)4DW (Fmt=001/011)The upper 32 bits of the address must be carried in DW2.
Target address is 0, but system may have > 4 GB RAM3DWIf address fits in 32 bits, use 3DW regardless of system RAM size.
Using 4DW for a <4GB address (upper DW = 0)Undefined per specSpec says receiver behaviour is undefined. Some devices accept it; avoid this in practice.
MRd targeting PCIe device registers (BAR)Depends on BAR address32-bit BARs are always below 4 GB → 3DW. 64-bit BARs may be above 4 GB → 4DW required.

⚡ Memory TLPs in Gen 6

The MRd, MWr, and MRdLk header formats are completely unchanged in Gen 6. Every bit position, field width, encoding, and rule described in this post applies identically to a Gen 6 link. Gen 6 changes only the Physical Layer — the TLP header the Transaction Layer produces is the same byte pattern whether the link is Gen 1 at 2.5 GT/s or Gen 6 at 64 GT/s.

What flit packing means for memory TLPs

Gen 6 performance tip for memory writes. At Gen 6 x16 speeds (~122 GB/s per direction), a single large MWr TLP with 4096-byte payload approaches the theoretical flit utilisation maximum. But many smaller MWr TLPs (e.g. 64-byte cache-line writes) pack multiple TLPs per flit, and the Gen 6 flit header amortises cleanly. The PCIe 6.0 spec’s flit-based framing was specifically designed to be efficient for both large streaming writes and small cacheline-sized transactions.

📋 Quick Reference

ItemValue / Rule
MRd Type field0_0000 · Fmt = 000 (3DW no-data) or 001 (4DW no-data)
MWr Type field0_0000 · Fmt = 010 (3DW with-data) or 011 (4DW with-data)
MRdLk Type field0_0001 · same Fmt options as MRd · legacy only
3DW header size12 bytes (DW0 + DW1 + DW2) · address in DW2 bits [31:2]
4DW header size16 bytes (DW0 + DW1 + DW2 + DW3) · address in DW2[63:32] + DW3[31:2]
Use 3DW whenTarget address fits in 32 bits (below 4 GB). Spec requires 3DW for sub-4GB addresses.
DW1 layoutRequester ID [31:16] · Tag [15:8] · Last DW BE [7:4] · First DW BE [3:0]
Requester ID16-bit BDF of sender (Bus[15:8] · Device[7:3] · Function[2:0])
Tag8-bit (256) standard · 10-bit (1024) extended · matched in CplD · unique per in-flight request
First DW BEByte 7 bits [3:0] · bit=1 means byte valid · selects which bytes in the first DW are active
Last DW BEByte 7 bits [7:4] · must be 0x0 when Length=1 · bit=1 means byte valid in last DW
BE discontinuousAllowed only when Length=1. For Length≥3, First and Last BEs must be contiguous bit patterns.
Address alignmentAlways DW-aligned — bits [1:0] of address are reserved (always 00). Byte offset via Byte Enables.
4 KB ruleNo single TLP may cross a 4096-byte (0x1000) boundary. Split at the page boundary if needed.
Max payload1 to 1024 DW (4 to 4096 bytes) · limited by Max_Payload_Size in Device Control register
MWr posted?Yes — no completion returns. DLL ACK DLLP confirms per-hop delivery only.
MRd posted?No — CplD returns with requested data and the echoed Tag value.
Gen 6 impactNone on header format — same bits, same positions, same rules. Flit packing is Physical Layer only.
Coming next — PCIe-07: Completion TLPs — the full Cpl and CplD header, Completer ID, Completion Status codes (SC/UR/CRS/CA), Byte Count, Lower Address, split-completion reassembly, and Completion Timeout.
Scroll to Top