Address Translation — AT Field, ATS, and PRI

📋 The Virtualisation Problem

In a standard PCIe system, a device uses physical host memory addresses directly. When it performs a DMA write, the address in the TLP header is a physical address that the Root Complex maps straight to system RAM. This works perfectly for a single operating system with full trust in its devices.

Virtualisation breaks this model. A virtual machine believes it owns contiguous physical memory starting at address 0. In reality that memory is scattered across the real physical address space by the hypervisor. A PCIe device assigned to a virtual machine must be able to use the VM’s view of addresses — called Guest Physical Addresses (GPA) — while the hardware translates them transparently to real Host Physical Addresses (HPA) before they reach memory.

Without an IOMMU, either the hypervisor must reprogramme every device BAR and DMA address every time a VM migrates (expensive and slow), or devices can DMA to arbitrary physical addresses — a severe security hole that lets a malicious VM corrupt any other VM’s memory or the hypervisor itself.

📋 What an IOMMU Does

An IOMMU (Input/Output Memory Management Unit) sits in the Root Complex and intercepts every inbound DMA request before it reaches the memory controller. It maintains page tables — similar to the CPU’s MMU page tables — mapping each device’s view of addresses to real physical addresses.

The IOMMU provides three security and isolation guarantees:

1. Isolation — a device assigned to VM-A cannot access memory owned by VM-B or the hypervisor, even if compromised or malicious
2. Remapping — a VM can be migrated to different physical memory without reprogramming the device. The IOMMU page tables are updated; the device sees the same guest addresses
3. Translation — the device uses logical addresses; the IOMMU silently translates to physical. The device driver never needs to know physical addresses

The challenge: IOMMU translation adds latency to every DMA access — the IOMMU must walk its page tables before forwarding each TLP. For high-throughput devices like Gen 5/6 GPUs and NVMe arrays, this latency is unacceptable. Address Translation Services (ATS) solves this.

📋 The AT Field in TLP Header DW0

The AT (Address Type) field is a 2-bit field at DW0 bits 11:10 of every memory and AtomicOp TLP header. It tells the Root Complex and any intermediate IOMMU-aware hardware whether the address in this TLP has already been translated by the IOMMU — or whether it still needs translation.

📋 AT Field Encodings

📋 Address Translation Services — Overview

ATS is the mechanism that allows a device to pre-fetch and cache translated addresses from the IOMMU, so that subsequent DMA accesses use AT=10 (already translated) and bypass the per-access page-table walk. The device builds its own Translation Lookaside Buffer (TLB) of GPA→HPA mappings, keyed by GPA range.

📋 ATS Translation Request TLP

An ATS Translation Request is a standard Memory Read Request (MRd) with AT=01. The device sends it to ask the IOMMU (via the RC) to translate a GPA into an HPA. The header format is identical to an MRd — the only distinguishing feature is AT[1:0] = 01 in DW0.

What the Length field means in a Translation Request

In a normal MRd, Length specifies how many DWs of data to return. In an ATS Translation Request, Length specifies the size of the GPA window the device wants translated — how large a contiguous block of GPA space it wants a mapping for. The IOMMU returns the HPA base of that window along with attributes indicating the access permissions and cache hints.

📋 ATS Translation Completion

The Root Complex/IOMMU responds to an ATS Translation Request with a standard Completion with Data (CplD). The payload carries the translated address plus permission and attribute bits. The format of the payload is defined by the ATS capability:

The device stores the returned HPA in its TLB, tagged with the original GPA, its size, and the permission bits. All subsequent DMA accesses to that GPA range use AT=10 with the cached HPA, completely bypassing the IOMMU table walk. The latency saving is significant — a table walk on a modern system takes 50–200 ns; AT=10 bypasses it entirely.

📋 ATS Invalidation — Keeping Caches Fresh

When the IOMMU needs to change a mapping — for example when a VM is migrated, a memory page is reclaimed, or an IOMMU page table entry is updated — it must ensure the device’s TLB no longer holds a stale translation for that address range. The ATS Invalidation mechanism handles this.

The Invalidate Request is carried as an ATS Invalidation Request TLP (a specialised NP TLP type). Invalidation completion is mandatory — the IOMMU must not remap a page until the device confirms its TLB is clean for that range. This prevents a device from issuing an AT=10 DMA to an HPA that the IOMMU has already remapped to a different GPA.

📋 ATS Configuration Registers

ATS support is declared and negotiated via the ATS Extended Capability structure, which lives in the PCIe Extended Configuration Space (above offset 0x100) of the endpoint. The key registers are:

ATS is only active when the platform enables it. Software discovers the ATS capability by scanning the extended capability chain in config space for Capability ID = 0x000F. Both the endpoint and the platform (RC) must support ATS before the device may issue AT=01 or AT=10 TLPs. Issuing AT≠00 without ATS negotiation is an Access Control Services (ACS) violation.

📋 Page Request Interface — Overview

ATS Translation Requests work well when the target GPA is mapped and the page is already resident in physical memory. But in a virtualised system pages can be paged out to disk, swapped, or simply not yet allocated. When a device sends an AT=01 Translation Request for a GPA that has no current valid mapping, the IOMMU cannot simply return an error — the device needs to wait while the page is brought back into memory.

The Page Request Interface (PRI) is the mechanism that handles this gracefully. Instead of failing the DMA and crashing the device driver, PRI allows the device to signal the OS page manager that a page fault has occurred and to wait for the OS to service it before resuming DMA.

📋 PRI Messages

PRI operates through three TLP types, all implemented as standard Message TLPs (Msg) or Non-Posted Requests:

Page Request Group (PRG)

A PRG is a group of related page requests that all belong to the same DMA transaction. The device gives each group a PRG index. When all pages in a PRG are resident, the OS sends a single Page Response referencing the PRG index. This allows one OS response to unblock multiple related DMA pages at once rather than responding one page at a time.

📋 PRI Full Flow

The complete PRI sequence for a device performing DMA to a paged-out memory region:

1. Device issues ATS Translation Request (AT=01) for GPA range. IOMMU checks page tables — page not resident.
2. IOMMU responds with UR or a specific “needs paging” completion status. Device’s ATS logic sees the miss.
3. Device sends Page Request Message to the RC, specifying the GPA, access permissions, and PRG index. Device pauses this DMA context — it does not abort, just waits.
4. RC forwards the Page Request to the OS page manager (via an interrupt or kernel notification path).
5. OS pages in the required memory from disk or swap, updates the IOMMU page tables, marks the page as resident.
6. OS sends a Page Response to the device (via the RC) indicating success and the PRG index that was satisfied.
7. Device receives the Page Response. It retries the ATS Translation Request (AT=01) for the same GPA.
8. This time IOMMU finds the page resident, returns the HPA via CplD.
9. Device adds the translation to its TLB. Subsequent DMA uses AT=10. DMA proceeds normally.

📋 ACS — Access Control Services

Access Control Services (ACS) is the security layer that prevents devices from abusing the AT field. Without ACS, a malicious device could set AT=10 on a TLP carrying an arbitrary HPA, bypassing the IOMMU entirely and reading or writing any physical memory location.

ACS is configured via the ACS Extended Capability structure (Capability ID = 0x000D) in the config space of switches and Root Ports. A switch port with ACS enabled will reject AT=10 TLPs from devices that have not been validated, returning a UR completion to the offending device.

⚡ AT Field, ATS, and PRI in Gen 6

The AT field encoding, ATS protocol, and PRI message format are all unchanged in Gen 6. They are Transaction Layer features — Gen 6 changes only the Physical Layer. ATS Translation Requests and PRI Page Request messages are packed into Gen 6 flits the same way any other TLP is, with no modifications to their header or payload formats.

ATS relevance in Gen 6 scale-out systems

• Gen 6 bandwidth demands make ATS more valuable. At ~8 GB/s per lane (x16 = ~122 GB/s), an IOMMU page walk on every DMA access would consume a significant fraction of total memory bandwidth. ATS eliminates this cost — its value scales directly with link bandwidth, making it more important in Gen 6 than in any prior generation.
• CXL 3.0 extends ATS. CXL 3.0 (which runs on the PCIe 6.0 PHY) adds coherent memory sharing across devices. CXL uses the same AT field with the same encodings but introduces additional coherency-aware translation semantics. The AT=10 bypass concept carries forward unchanged.
• PASID (Process Address Space Identifier). Gen 6 deployments increasingly use PASID — a TLP prefix that identifies which process’s address space a translation belongs to. PASID allows a single PCIe device to hold separate address translations for multiple processes simultaneously, enabling fine-grained multi-tenant isolation. PASID is carried as a TLP Prefix and works alongside the AT field, not instead of it.
• PRI in SR-IOV/MR-IOV. Gen 6 NVMe, GPU, and network devices often expose hundreds of Virtual Functions via SR-IOV. Each VF can independently issue ATS requests and PRI page faults. The OS must handle PRI responses per-VF. The AT field and PRI message formats are the same for VFs as for physical functions.

📋 Quick Reference