Peer To Peer DMA

Peer To Peer DMA means sending data directly from one device to another without going through the systems memory or CPU. Sending data directly between devices on the PCIe bus has clear advantages if you can make it work: There is no CPU overhead or need for synchronization. It is very hard to do P2P DMA on Linux.

Kernel Support[edit]

Linux has had P2P DMA support since at least Linux 5.6. There is no support on Linux 4.x kernels. P2P DMA requires CONFIG_PCI_P2PDMA under Device Drivers ▸ PCI support. Most, if not all, Linux distributions enable it in their kernels. This kernel driver will only work if two devices that to do direct peer to peer DMA are behind the same PCIe bridge. Forwarding between hierarchy domains is not supported^[1] by the PCI_P2PDMA driver and there is only support for RDMA and NVMe drivers. It was primarily designed for P2P DMA between NVMe devices and it behaves accordingly, it is therefore not ideal as a general-purpose P2P DMA driver.

Hardware Support[edit]

There is no standard official way of telling if a chipset supports P2P DMA over the PCI Express bus or not. The Linux kernel solves this by using a whitelist with known "good" chipsets.

All AMD Zen chipsets support (for both reads and writes)
Previous AMD chipsets had some support for writes only.

Why P2P DMA is so hard on Linux[edit]

AMD-employed kernel and Mesa developer Alex Deucher held a very interesting presentation on the state of P2P DMA on Linux at the X.Org Developers Conference 2020 in September 2020:

Copyright X.Org Developers Conference 2020. Licensed under Creative Commons CC-BY.