PCI Express Reference Design

By the end of this tutorial you will have successfully used the BRK8310 with an XEM8310 to initiate DMA transfers between the BRK8310+XEM8310 gateware and the host PC. The steps will guide you through the process of generating the gateware, configuring your host system, bringing up your hardware, and successfully running the test procedure to validate your system.

This tutorial utilizes Xilinx’s DMA/Bridge Subsystem for PCI Express IP’s example design along with Xilinx’s provided example drivers. This tutorial uses the Ubuntu operating system, but Windows 10 drivers are also available.

Requirements:

  • BRK8310
  • XEM8310
  • One of the following supported Ubuntu distributions: 
    • 16.04 LTS
    • 18.04 LTS
    • 20.04 LTS

Resources

Top-Level Architecture

The DMA/Bridge Subsystem for PCI Express IP’s example design is generated by Vivado. The example design implements a 4KByte BRAM buffer that is read or written to via DMA. Xilinx’s provided drivers/applications initiate the DMA transfers to and from the BRAM on the XEM8310. 

Tutorial

We provide two tutorials that result in different power-on procedures:

  • Standard – The BRK8310 and host PC are powered separately. The system is powered and programmed in a manual sequence that ensures a proper enumeration. This method is less complicated and allows more flexibility for the implemented design.
  • Tandem – The BRK8310 and host PC are powered using the host PC’s PSU and the PCIe device enumerates when the PC is powered on. The bitfile is split into two stages. The first stage includes just the PCIe components that enumerate the PCIe device, the second stage includes the rest of the implemented design. This segmented bit file is loaded into the FPGA flash onboard the XEM8310 and configures the FPGA at power-on. The smaller first stage ensures that the PCIe device is available to the host PC in under 100ms (PCIe specification), allowing a successful enumeration. This method is more complicated and comes with some caveats for the implemented design.

Standard Configuration

Create a New Vivado Project

  1. Open a New Vivado project.
  2. Select the xcau25p-ffvb676-2-e FPGA part.
  3. Configure the DMA/Bridge Subsystem for PCI Express IP from the IP wizard. 
  4. Select a lane width of x8.
  5. Select the maximum link speed available through the GUI.
  6. All other settings can be kept at their default. Press “OK” to create the IP.
  7. Right click on the IP in the hierarchy view and select “Open IP Example Design”.

Make Board-Specific HDL Modifications

1. Open up the example design’s xilinx_xdma_pcie_x0y0.xdc constraint file.

2. Remove the LOC constraints for ports sys_rst_n, sys_clk_p, and sys_clk_n. Substitute them with the following constraints. Package pin information can be obtained from the XEM8310’s Pins page.

set_property PACKAGE_PIN E22 [get_ports sys_rst_n] set_property PACKAGE_PIN T7 [get_ports sys_clk_p] set_property PACKAGE_PIN T6 [get_ports sys_clk_n]
Code language: CSS (css)

3. Generate the bitstream.

Install Drivers

  1. Acquire the sources at the following Git repository: XMA
  2. Follow the README’s instructions for installation.

Power-on Procedure

The PCIe device must be powered and available at the time the host PC enumerates the PCIe bus at power-on. Follow the following procedure to configure your hardware correctly:

  1. Power-off your host PC.
  2. Plug in the BRK8310 into a compatible PCIe port.
  3. Power-on your XEM8310 using the barrel jack power connector. See Power Connectors.
  4. Program the PCIe DMA bitfile into your XEM8310. 
  5. Power-on your host PC. 

Run the Tests

  1. Run sudo lspci -vvv to see verbose information about the devices enumerated onto the PCIe bus of the host PC. The device will be enumerated under the name “Xilinx Corporation Device”.
  2. Run the following Xilinx provided tests from the XMA Git repository:
    1. run_tests.sh
    2. perform_hwcount.sh

Tandem Configuration

Create a New Vivado Project

  1. Open a New Vivado project.
  2. Select the xcau25p-ffvb676-2-e FPGA part.
  3. Configure the DMA/Bridge Subsystem for PCI Express IP from the IP wizard. 
  4. Select a lane width of x8.
  5. Select the maximum link speed available through the GUI.
  6. Select Mode “Advanced”.
  7. Select “Tandem PROM” for “Tandem Configuration or Dynamic Function eXchange”
  8. All other settings can be kept at their default. Press “OK” to create the IP.
  9. Right click on the IP in the hierarchy view and select “Open IP Example Design”.

Make Board-Specific HDL Modifications

1. Open up the example design’s xilinx_xdma_pcie_x0y0.xdc constraint file.

2. Remove the LOC constraints for ports sys_rst_n, sys_clk_p, and sys_clk_n. Substitute them with the following constraints. Package pin information can be obtained from the XEM8310’s Pins page.

set_property PACKAGE_PIN E22 [get_ports sys_rst_n] set_property PACKAGE_PIN T7 [get_ports sys_clk_p] set_property PACKAGE_PIN T6 [get_ports sys_clk_n]
Code language: CSS (css)

3. Add the following constraints to allow for x4 SPI configuration:

set_property BITSTREAM.CONFIG.EXTMASTERCCLK_EN disable [current_design] set_property CONFIG_MODE SPIx4 [current_design] set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 4 [current_design] set_property BITSTREAM.CONFIG.SPI_FALL_EDGE YES [current_design] set_property BITSTREAM.CONFIG.CONFIGRATE 80 [current_design] set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design] set_property BITSTREAM.CONFIG.UNUSEDPIN Pulldown [current_design]
Code language: CSS (css)

4. Generate the bitstream.

Loading the Flash

Follow the instructions at Flash Memory to load the tandem bit file onto the FPGA flash of the XEM8310.

Install Drivers

  1. Acquire the sources at the following Git repository: XMA.
  2. Follow the README’s instructions for installation.

Power-on Procedure

 Follow the following procedure to configure your hardware correctly:

  1. Power-off your host PC.
  2. Plug in the BRK8310 into a compatible PCIe port.
  3. Connect a 6-pin PCIe-style power connector from your host PC’s PSU to the 6-pin power connector on the BRK8310. See Power Connectors.
  4. Power-on your host PC.

Run the Tests

  1. Run sudo lspci -vvv to see verbose information about the devices enumerated onto the PCIe bus of the host PC. The device will be enumerated under the name “Xilinx Corporation Device”.
  2. Run the following Xilinx provided tests from the XMA Git repository:
    1. run_tests.sh
    2. perform_hwcount.sh

Tandem Caveats

The use of FrontPanel becomes more complicated when using a tandem bitfile. The FrontPanel Host Interface is on bank 65. Bank 65 is also the configuration bank and the configuration bank must come up in the first stage when using the “Tandem PROM” setting.

After your design has been placed, floorplanning Pblocks are used to specify which parts of the FPGA fabric are to come up in stage one. If you require use of FrontPanel with a tandem bitfile, our recommendation is to place the FrontPanel okHost within the stage one Pblock wrapping bank 65. Endpoints can be placed outside of this Pblock and can come up in the second stage.

Alternatively, the “Tandem PCIe” setting can be used as it doesn’t require bank 65 to come up in stage one. In “Tandem PCIe” the first stage is loaded onto the FPGA flash and the second stage of the bitfile lives on the host PC. At power-on the first stage is loaded and the device is enumerated onto the PCIe bus. Drivers on the host PC then load the second stage bitfile through the PCIe link. Xilinx provides examples of “Tandem PCIe” that you can use as a reference.