Xilinx Answer 65444 Xilinx PCI Express DMA Drivers and Sftware Guide Imprtant Nte: This dwnladable PDF f an Answer Recrd is prvided t enhance its usability and readability. It is imprtant t nte that Answer Recrds are Web-based cntent that are frequently updated as new infrmatin becmes available. Yu are reminded t visit the Xilinx Technical Supprt Website fr the latest versin f this Answer. Intrductin The Xilinx PCI Express DMA IP prvides high-perfrmance direct memry access (DMA) via PCI Express. The PCIe DMA can be implemented in Xilinx 7 Series XT and UltraScale devices. This answer recrd prvide drivers and sftware that can be run n a PCI Express rt prt hst PC t interact with the DMA endpint IP via PCI Express. The drivers and sftware prvided with this answer recrd are designed fr Linux perating systems and can be used fr lab testing r as a reference fr driver and sftware develpment. Thrugh the use f the PCIe DMA IP and the assciated drivers and sftware yu will be able t generate high-thrughput PCIe memry transactins between a hst PC and a Xilinx FPGA. PCIe DMA Driver fr Linux Operating Systems Dependencies The current driver implementatin uses the fllwing Kernel functins and must be included in yur OS kernel versin. The fllwing Linux Kernels have been tested. - RedHat 6.2 - Fedra 20 - CentOS 7.0 - Ubuntu 14.04 Time Functins PCIe Functins Kernel Memry functins Lading the Driver After uncmpressing the drivers and sftware file, the prvided directries cntain the surce cde fr a kernel mde driver and a few example applicatins that demnstrate the basic capabilities f the PCIe DMA driver and IP. The steps belw describe hw t install the driver. The steps belw describe hw t install the driver and must be perfrmed with rt permissins. Prgram the Xilinx FPGA with a bitfile that makes use f the PCIe DMA IP. Bt the PCIe hst PC int Linux. Cpy the driver and sftware files t the Linux PC. Uncmpress the driver and sftware package. G t the driver directry and type make t cmpile the driver and assciated files. Cpyright 2017 Xilinx Xilinx Answer 65444 Xilinx PCI Express DMA Drivers and Sftware Guide 1
Cpy the 60-xdma.rules file t yur lcal system directry. $Linux> cp etc/udev/rules.d/60-xdma.rules /etc/udev/rules.d/60-xdma.rules $Linux> cp etc/udev/rules.d/xdma-udev-cmmand.sh /etc/udev/rules.d/xdma-udev-cmmand.sh G t the tests directry and run the lad_drivers.sh script. $Linux>./lad_driver.sh This script will perfrm the fllwing functin: Remve the current xdma mdule driver Insert the xdma mdule driver int the kernel Check the installed devices t certify that an apprpriate PCIe DMA device was fund Reprt a passing (return 0) r failing (return 1) result t the screen Nte: By default the driver uses interrupts t determine when DMA transfers are cmpleted. If supprted by the driver, pll mde can be enabled when inserting the kernel mdule. Pll mde uses plling rather than interrupts t determine when DMA transfers are cmpleted. Refer t the delivered readme.txt and lad_driver.sh files t enable pll mde. Additinal infrmatin abut the IP can als be fund in the pll mde sectin f PG195. At this pint the PCIe DMA driver is ready fr use t perfrm DMA transfers between the hst PC and the Xilinx FPGA device. Running the Applicatin Sme basic applicatins that use the PCIe DMA kernel mdule driver have been included fr reference. The steps belw describe hw t cmpile and run the example applicatins using the tests/run_test.sh script. G t the tests directry and type make t cmpile the sftware and assciated files. Run the prvided test using the tests/run_test.sh script. $Linux>./run_tests.sh This script is designed t run with the PCIe example design which implements a 4KByte BRAM buffer in the user prtin f the design. This is because such DMA transfers shuld be limited t 4 KByte transfers. Fr a 4 channel design this script transfers 1024 bytes n each channel. The fllwing functins are perfrmed by the run_test.sh script.: Determines hw many h2c and c2h channels are enabled in the PCIe DMA IP Determines if the PCIe DMA cre is cnfigured fr memry mapped r streaming mdes Perfrms data transfers n all available h2c and c2h channels Verifies that the data written t the device matches the data that was read frm the device Reprts pass (return 0) r fail (return 1) cmpletin status t the user A few f the key cmmands used in the tests/run_test.sh script are identified belw. Read a 32-bit register frm the PCIe DMA cntrl registers at a specified ffset (0x0000). (The register map detailed in the PCIe DMA prduct guide fr reference.) $Linux>./reg_rw /dev/xdma0_cntrl 0x0000 w Perfrm PCIe DMA memry mapped h2c data transfer. (Channel=0, size=1024, address ffset=0x0000, number f transfer=1) $Linux>./dma_t_device -d /dev/xdma0_h2c_0 -f data/datafile0_4k.bin -s 1024 -a 0000 -c 1 Cpyright 2017 Xilinx Xilinx Answer 65444 Xilinx PCI Express DMA Drivers and Sftware Guide 2
Perfrm PCIe DMA memry mapped c2h data transfer. (Channel=0, size=1024, address ffset=0x0000, number f transfer=1) $Linux>./dma_frm_device -d /dev/xdma0_c2h_0 -f data/utput_file.bin -s 1024 -a 0000 -c 1 Perfrm PCIe DMA streaming c2h data transfer. (Channel=0, size=1024, number f transfer=1) $Linux>./dma_frm_device -d /dev/xdma0_c2h_0 -f data/utput_file.bin -s 1024 -c 1 Perfrm PCIe DMA streaming h2c data transfer (Channel=0, size=1024, number f transfer=1) $Linux>./dma_t_device -d /dev/xdma0_h2c_0 -f data/datafile0_4k.bin -s 1024 -c 1 Perfrmance Measurement Anther sample applicatin has als been prvided t reprt XDMA thrughput perfrmance in bth the Hst t Card (H2C) and Card t Hst (C2H) directins. This applicatin shuld nly be used with XDMA example designs fr the XDMA in AXI Memry Mapped mde. This applicatin cannt be used t measure thrughput perfrmance using the example design fr the XDMA in AXI Stream mde. When using the AXI Stream mde f the XDMA, the example design implements a lpback between the H2C and C2H channels. This is nt handled by sftware applicatin and will nt reprt the crrect perfrmance fr the design. The prvided perfrmance applicatin measures XDMA thrughput fr transfer sizes that are a pwer-f-tw between 64 bytes t 4 Mbytes. Larger transactin sizes reduces the impact f verhead and results in higher thrughput fr XDMA transactins. The fllwing instructins can be used t run the prvided perfrmance applicatin. G t the tests directry and type make t cmpile the sftware and assciated files. Run the perfrmance test using the tests/perfrm_hwcunt.sh scipt $Linux>./perfrm_hwcunt.sh The thrughput measurement results will be reprted in the hw_lg_h2c.txt file fr the Hst-t-Card directin and the hw_lg_c2h.txt file fr the Card-t-Hst directin. The data rate thrughput is reprted as a percentage f maximum thrughput fr the link. Example 1: The maximum thrughput fr a Gen3 x8 PCIe Link is 8 Gbytes/s; s fr a Gen3 x8 PCIe design a reprted data rate f 0.81 crrespnds t (8 Gbytes/s*.81) = 6.4Gbytes/s Example 2: The maximum thrughput fr a Gen3 x16 PCIe link is 16 Gbytes/s; s fr a Gen3 x16 PCIe design a reprted data rate f.78 crrespnds t (16 Gbytes/s *.78) = 12.4 Gbytes/s. Mdifying the driver fr yur wn PCIe Device ID During the PCIe DMA IP custmizatin in Vivad yu can specify a PCIe Device ID. This Device ID must be recgnized by the driver in rder t prperly recgnize the PCIe DMA device. The current driver is designed t recgnize the PCIe Device IDs that get generated with the PCIe example design when this value has nt been mdified. If yu have mdified the PCIe Device ID during IP custmizatin yu will need t mdify the PCIe driver t recgnize this new ID. Yu may als want t mdify the driver t remve PCIe Device IDs that will nt be used by yur slutin. T mdify the PCIe Device ID in the driver yu shuld pen the driver/xdma-cre.c file and search fr the pcie_device_id struct. This struct identifies the PCIe Device IDs that are recgnized by the driver in the fllwing frmat: { PCI_DEVICE(0x10ee, 0x8038), }, Add, remve, r mdify the PCIe Device IDs in this struct as desired fr yur applicatin. The PCIe DMA driver will nly recgnize device IDs identified in this struct as PCIe DMA devices. Once mdified the driver must be uninstalled, recmpiled, and reinstalled fllwing the directin in the Lading the Driver sectin. Cpyright 2017 Xilinx Xilinx Answer 65444 Xilinx PCI Express DMA Drivers and Sftware Guide 3
Enabling the PCIe t AXI-Lite Master interface in the PCIe DMA Driver During IP custmizatin in Vivad the PCIe DMA IP can be custmized t enable a PCIe t AXI-Lite Master interface. This selectin is available n the PCIe:BARs tab f the PCIe custmizatin GUI. This interface expses an AXI Lite Memry Mapped interface t the user which can be used fr cntrl r ther functins and als can be cnnected t AXI Intercnnect IP. In the example design this interface is cnnected t a 4Kbytes Bram. Data width fr this interface is 32bits nly. This memry regin can be accessed thrugh the fllwing cmmand using the prvided sftware applicatin. Here is an example f hw t read frm the AXI-Lite at a specified ffset (0x0000). $Linux>./reg_rw /dev/xdma0_user 0x0000 w Here is an example f hw t write t the AXI-Lite Interface at a specified ffset (0x0000) with specific data (0x1234567). $Linux>./reg_rw /dev/xdma0_user 0x0000 w 0x1234567 Enabling the PCIe t DMA Bypass interface in the PCIe DMA Driver During IP custmizatin in Vivad the PCIe DMA IP can be custmized t enable a DMA bypass interface. This selectin is available n the PCIe:BARs tab f the PCIe custmizatin GUI. This interface expses an AXI Memry Mapped interface that bypasses the DMA and can be cnnected t an AXI system thrugh the AXI Intercnnect IP. This memry regin can be accessed thrugh the fllwing cmmand using the prvided sftware applicatin. Here is an example f hw t read frm the bypass channel at a specified ffset (0x0000). $Linux>./reg_rw /dev/xdma0_bypass 0x0000 w Here is an example f hw t write t the bypass channel at a specified ffset (0x0000) with specific data (0x1234567). $Linux>./reg_rw /dev/xdma0_bypass 0x0000 w 0x1234567 Applicatin prgram reg_wr has 32Kbytes allcated space as default. If PCIe t AXI Lite Master r PCIe t DMA Bypass interface selected size is less than 32Kbytes and try t use reg_rw applicatin fr read/write will prduce an errr. If selected size is less than 32Kbytes mdify this define in reg_rw.c t crrespnding value and cmpile (make) the file. #define MAP_SIZE (32*1024UL) Cpyright 2017 Xilinx Xilinx Answer 65444 Xilinx PCI Express DMA Drivers and Sftware Guide 4
Enabling Debug Messaging in the Driver T aid develpment and debug f the PCIe DMA driver, yu can enable debug messaging by setting the XDMA_DEBUG define t 1. T make this mdificatin, pen the include/xdma-cre.h file and search fr the #define that sets the XDMA_DEBUG variable t 0. Change the 0 t a 1 as shwn belw. #define XDMA_DEBUG 1 Once mdified, the driver must be uninstalled, recmpiled, and reinstalled by fllwing the directins in the Lading the Driver sectin. Yu can view the messages frm the kernel driver by using the Linux dmesg cmmand. $Linux> dmesg This can be used t debug failures r t view the DMA peratinal messages. Uninstalling the PCIe DMA Driver Standard Linux cmmands shuld be used t uninstall the driver and delete the rules that were added during the installatin prcess. # Uninstall the kernel mdule. $Linux> rmmd -s xdma # Delete the ma rules that were added. $Linux> rm -f /etc/udev/rules.d/60-xdma.rules $Linux> rm -f /etc/udev/rules.d/xdma-udev-cmmand.sh Updates and Backward Cmpatibility The fllwing features were added t the PCIe DMA IP and driver in Vivad 2016.4. Sample Perfrmance Measurement applicatin and script. The fllwing features were added t the PCIe DMA IP and driver in Vivad 2016.1. These features cannt be used with PCIe DMA IP if the IP was generated using a Vivad build earlier than 2016.1. Pll Mde: Earlier versins f Vivad nly supprt interrupt mde which is the default behavir f the driver. Surce/Destinatin Address: Earlier versins f Vivad PCIe DMA IP required the lw-rder bits f the Surce and Destinatin address t be the same. As f 2016.1 this restrictin has been remved and the Surce and Destinatin addresses can be any arbitrary address that is valid fr yur system. Revisin Histry 10/06/2015 Initial Release 05/14/2016 Updated fr 2016.1 01/25/2017 Updated fr 2016.4 04/21/2017 Added cmmands fr axi-lite access 06/08/2017 Few changes in the driver; see change lg in the driver zip file. 09/22/2017 Added explanatin n memry limit fr reg_rw.c Cpyright 2017 Xilinx Xilinx Answer 65444 Xilinx PCI Express DMA Drivers and Sftware Guide 5