Peer To Peer DMA
Peer To Peer DMA means sending data directly from one device to another without going through the systems memory or CPU. Sending data directly between devices on the PCIe bus has clear advantages if you can make it work: There is no CPU overhead or need for synchronization. It is very hard to do P2P DMA on Linux.
Linux has had P2P DMA support since at least Linux 5.6. There is no support on Linux 4.x kernels. P2P DMA requires
CONFIG_PCI_P2PDMA under ▸ . Most, if not all, Linux distributions enable it in their kernels. This kernel driver will only work if two devices that to do direct peer to peer DMA are behind the same PCIe bridge. Forwarding between hierarchy domains is not supported by the PCI_P2PDMA driver and there is only support for RDMA and NVMe drivers. It was primarily designed for P2P DMA between NVMe devices and it behaves accordingly, it is therefore not ideal as a general-purpose P2P DMA driver.
There is no standard official way of telling if a chipset supports P2P DMA over the PCI Express bus or not. The Linux kernel solves this by using a whitelist with known "good" chipsets.
- All AMD Zen chipsets support (for both reads and writes)
- Previous AMD chipsets had some support for writes only.
Why P2P DMA is so hard on Linux
- kernel.org: PCI Peer-to-Peer DMA Support