PCI Express Packet Analysis with Down Stream Port Model and PIO Example Design. Deepesh Man Shakya. Page 1 of 66

Transcription

1 PCI Express Packet Analysis with Down Stream Port Model and PIO Example Design Deepesh Man Shakya Page 1 of 66

2 Table of Contents Introduction...3 PIO Example Design...3 Downstream Port Model...4 Files Hierarchy...4 PIO Example Design Schematics...5 Downstream Port Model Architecture...15 Xilinx_pci_exp_dsport.vhd...16 RX_APP (pci_exp_usrapp_rx.vhd)...16 PROC_READ_DATA...16 PROC_DECIPHER_FRAME...16 PROC_3DW/PROC_4dW...17 PROC_PARSE_FRAME...17 TX_APP (pci_exp_usrapp_tx.vhd)...19 pio_writereadback_test0 (tests.vhd)...21 test_interface.vhd...24 PROC_SYSTEM_INITIALIZATION...24 PROC_BAR_INIT...24 PROC_BAR_SCAN...25 PROC_BUILD_PCIE_MAP...26 PROC_BAR_PROGRAM...27 PROC_TX_SYNCHRONIZE...28 PROC_TX_TYPE0_CONFIGURATION_WRITE...28 PROC_TX_TYPE1_CONFIGURATION_READ...30 PROC_READ_DATA / PROC_PARSE_FRAME / PROC_DECIPHER_FRAME / PROC_4DW...30 PROC_TX_IO_WRITE...30 PROC_TX_MEMORY_WRITE_ PROC_WAIT_FOR_READ_DATA...33 PIO Example Design Packet Analysis...34 Fmt and Type...36 Configuration Write...37 Configuration Write Completion...41 Memory Write 64 / Memory Read 64 / Completion...42 Memory Write 32 / Memory Read 32 / Completion...51 Limitations and Features of Downstream port model...53 Limitations of PIO Example Design...54 Playing with Packets...54 Length Parameter Modification...54 Poisoning Memory Read TLP...56 Poisoning Configuration Write Request...58 Testing with a new TLP...60 Appendix...62 Generic TLP Header Fields...62 IO Request Header Format...64 Memory Request Header Format...64 Configuration Request Header Format...65 Message Request Header Format...65 Completion Header Format...65 Few points to note...65 References...66 Page 2 of 66

3 Introduction This document discusses the PIO example design and the downstream port model that comes with the generation of the PCI Express Block Plus core. The main goal of this document is to provide detail information on the architecture of the PIO example design and the simulation setup that includes downstream port model. The PIO example design simulation emulates the packet transaction between the real root complex and the endpoint. We will look into how the initialization process takes place, how the configuration transaction is initiated by the downstream port model and how the normal Memory Read, Memory Write and IO Read Write transaction is initiated by the host. We will also look into the generation of the completion packets by the endpoint. Latter part of this document goes through the packet analysis of the TLPs generated by downstream port model and the corresponding completions generated by the endpoint example design. PIO Example Design The PIO example design is a simple target-only application that interfaces with the Endpoint for PCIe core's Transaction (TRN) interface. Following are the main features of the PIO example design: Four transaction-specific 2 kb target region using the internal Xilinx FPGA Block RAMs, providing a total target space of 8192 bytes. Supports single DWORD payload Read and Write PCI Express transactions to 32/64 bit address memory spaces and IO space with support for completion TLPs. Utilizes the core's trn_rbar_hit_n[6:0] signals to differentiate between TLP destination Base Address Registers. Provides separate implementations optimized for 32-bit and 64-bit TRN interfaces. The following block diagram shows the PIO example design components: Page 3 of 66

4 Downstream Port Model Downstream port model acts as a root complex but it is not really a "root complex". The model represents only the downstream port interface which allows training with the endpoint. A complete root complex functionality represents a lot of other things. The downstream port and the provided testbench only provide just enough tools to do writes and read to the user design. The downstream port model is not a full blown simulation model as a true BFM available from third party vendors. However, it enables enough functionality to do basic testing of the user design. Downstream port model provides a mechanism for generating downstream PCI Express TLP traffic to access the user application. It also provides a mechanism to receive upstream PCI Express TLP traffic from the customer design in a simulation environment. Downstream port model initializes the core's configuration space, creates TLP transactions, generates TLP logs, and provides an interface for creating and verifying tests. The following diagram shows the high level architecture of downstream port model. Files Hierarchy Following screen shots gives the files hierarchy for the PIO example design simulation setup. All the files are generated during the generation of the PCI express block plus core. This hierarchy has been captured by creating an ISE project of the VHDL files provided in the example design and the downstream port model. Page 4 of 66

5 PIO Example Design Architecture The following diagram shows the top level schematics of the PIO Example design. The following schematic shows the block instantiations inside the above block. Page 5 of 66

6 The following schematic gives the top level view of the PIO example design user application. Page 6 of 66

7 The following schematic gives the top level view of the PCIe endpoint block plus core. Page 7 of 66

8 The following schematic shows the PIO Module inside the user application block. Page 8 of 66

9 The rest of the IOs in the user application block are tied off as shown below: Page 9 of 66

10 Following schematic shows a block (pio_ep) inside the pio interface. Inside PIO_EP, a memory block and the RX and TX engines are defined as shown below: Page 10 of 66

11 The following schematic shows the close-up view of the EP_MEM_ACCESS module. Page 11 of 66

12 Inside this module, there is a module called EP_MEM module where entire memory blocks used in the PIO example design have been instantiated. The PIO_EP_MEM_ACCESS module processes data written to the memory from incoming Memory and IO Write TLPs and provides data read from the memory in response to Memory and IO Read TLPs. The EP_MEM module processes 1 DWORD 32- and 64 bit addressable Memory and IO Write requests based on the information received from the RX Engine. The following schematic shows the internals of the EP_MEM_ACCESS module. The following schematic shows the internals of the EP_MEM module. Page 12 of 66

13 The following schematic shows the close-up view of the EP_RX_64 (Receive Engine) block. Page 13 of 66

14 The following schematic shows the close-up view of the TX engine block. PIO_64_TX_ENGINE (PIO_32_TX_ENGINE) module generates completions for received memory and IO read TLPS. The PIO design does not generate outbound read or write requests. However, user can add this functionality to further customize the design. Page 14 of 66

15 The following table shows the inputs required by the TX engine to generate completion packets. After the completion is sent, the TX engine asserts the compl_done_i output indicating to the RX engine that it can assert trn_rdst_rdy_n and continue receiving TLPs. Downstream Port Model Architecture Downstream port model consists of following components: Page 15 of 66

16 Xilinx_pci_exp_dsport.vhd This block essentially acts as a root complex. However, the model shouldn't be strictly treated as a root complex as it doesn't provide many features that a real root complex would normally provide. The endpoint PCIe block plus at the user side transmits TLPs across the PCI express link to the Downstream Port (dsport) model. The dsport and the PCIe block plus core are responsible for the data link layer and physical layer processing when communicating across the PCI Express fabric. dsport_cfg configures the downstream port model. RX_APP (pci_exp_usrapp_rx.vhd) Following are different procedures defined in RX_APP: 1. PROC_READ_DATA 2. PROC_DECIPHER_FRAME 3. PROC_3DW 4. PROC_4DW 5. PROC_PARSE_FRAME PROC_READ_DATA This procedure reads receive transaction data line (trn_rd) and stores it in frame_store_rx as shown below: PROC_DECIPHER_FRAME This procedure extracts the information from the data collected by PROC_READ_DATA as shown below: Page 16 of 66

17 PROC_3DW/PROC_4dW These procedures print the frame information to the output log as shown below: PROC_PARSE_FRAME PROC_PARSE_FRAME calls PROC_DECIPHER_FRAME, PROC_4DW and PROC_3DW (last two writes to the tx.dat and rx.dat file) Page 17 of 66

18 The following code gives the RX_APP state machine: Page 18 of 66

19 TX_APP (pci_exp_usrapp_tx.vhd) The usrapp_tx block sends TLPs to the dsport block for transmission across the PCI Express Link to the Endpoint DUT. Transaction sequences or test programs are initiated by the usrapp_tx block to stimulate the endpoint device's fabric interface. All test programs are defined inside the test_interface.vhd. All transaction sequences are defined in tests.vhd file. There are different tests that you can perform based on the VHDL or Verilog version of the core you generate. Following table gives the details of entire test suite that you can perform with the downstream port model. Page 19 of 66

20 Page 20 of 66

21 We will discuss here a test flow with reference to pio_write_readback_test0 test which is available both in verilog and vhdl version of the core generation. All downstream port model tests follow the same six steps as listed below: 1. Perform conditional comparison of a unique test name 2. Set up master timeout in case of simulation hangs 3. Wait for Reset and link-up 4. Initialize the configuration space of the endpoint 5. Transmit and receive TLPs between the Downstream Port Model and the Endpoint DUT 6. Verify that the test succeeded An entire source code relating to the TX_APP is presented here along with the description from the userguide (in the form of screen shot) and relevant description of the procedures where needed. The main objective behind publishing source code for all procedures defined in the TX_APP is to allow readers to get understanding of the working mechanism of the PIO example design without needing to generate the core and browse through the source code and the user guide. pio_writereadback_test0 (tests.vhd) This section will present the entire source code for pio_writereadback_test0 test suite provided in the downstream port model. The code involves number of procedure calls. Source code will be presented for each procedure used in the test. PROC_SYSTEM_INITIALIZATION will cause the test program to wait for the system reset to deassert as well as the endpoint's trn_lnk_up_n signal to assert. This is an indication that the endpoint is ready to be configured by the test program via the Downstream Port Model. PROC_BAR_INIT will perform a series of Type 0 Configuration Writes and Reads to the Endpoint core's PCI Configuration Space, determine the memory and IO requirements of the endpoint, and then program the endpoint's Base Address Registers so that it is ready to receive TLPs from the Downstream Port Model. In the following part of the source code, the sample program work together to cycle through all the endpoint's BARs and determine whether they are enabled, and if so determine their type, for example, Mem32, Mem64, or IO). Whether the BAR is enabled or not is checked by probing BAR_ENABLED[] global array. A non-zero value indicates that the corresponding BAR is enabled. If the BAR is not enabled then test program flow will move on to Page 21 of 66

22 check the next BAR. The previous call to PROC_BAR_INIT performed the necessary configuration TLP communication to the endpoint device and filled in the appropriate values into the BAR_ENABLED[ ] array. If the array element is enabled (that is, non-zero), the element's value indicates the BAR type. A value of 1, 2, and 3 indicates IO, Memory 32, and Memory 64 spaces, respectively. Page 22 of 66

23 Page 23 of 66

24 test_interface.vhd All the procedures called in the tests.vhd are defined in test_interface.vhd. Following are the procedures defined in this file: PROC_SYSTEM_INITIALIZATION PROC_BAR_INIT PROC_BAR_SCAN PROC_BUILD_PCIE_MAP PROC_DISPLAY_PCIE_MAP PROC_BAR_PROGRAM PROC_TX_SYNCHRONIZE PROC_TX_TYPE0_CONFIGURATION_WRITE PROC_TX_TYPE1_CONFIGURATION_READ PROC_READ_DATA PROC_PARSE_FRAME PROC_DECIPHER_FRAME PROC_4DW PROC_TX_IO_WRITE PROC_TX_IO_READ PROC_TX_MEMORY_WRITE_32 PROC_TX_MEMORY_READ_32 PROC_WAIT_FOR_READ_DATA PROC_TX_MEMORY_WRITE_64 PROC_TX_MEMORY_READ_64 PROC_SYSTEM_INITIALIZATION This procedure waits for transaction interface reset and linkup between the Downstream Port Model and the Endpoint DUT. This task must be invoked prior to the Endpoint core initialization. PROC_BAR_INIT PROC_BAR_INIT will perform a series of Type 0 Configuration Writes and Reads to the Endpoint core's PCI Configuration Space, determine the memory and IO requirements of the endpoint, and then program the endpoint's Base Address Registers so that it is ready to receive TLPs from the Downstream Port Model. Page 24 of 66

25 PROC_BAR_SCAN This procedure performs a sequence of PCI Type 0 Configuration Writes and Configuration Reads using the PCI Express fabric in order to determine the memory and IO requirements for the Endpoint. The task stores this information in the global array BAR_RANGE[]. This task should only be called after PROC_SYSTEM_INITIALIZATION. Page 25 of 66

26 PROC_BUILD_PCIE_MAP [From User Guide] Performs memory/io mapping algorithm and allocates Memory 32, Memory 64, and IO space based on the Endpoint requirements. This task has been customized to work in conjunction with the limitations of the PIO design and should only be called after completion of PROC_BAR_SCAN. [From Source Code] This checks whether the BAR_RANGE has been defined or not. If it is then that BAR is enabled. If not the BAR is disabled. Page 26 of 66

27 PROC_BAR_PROGRAM Page 27 of 66

28 PROC_TX_SYNCHRONIZE The main function of this procedure is to synchronize the trn_clk and trn_tdst_rdy_n signals. Before a TLP is transferred it waits for trn_clk to detect its positive edge and trn_tdst_rdy_n to be asserted. Within PROC_TX_SYNCHRONIZE calls PROC_READ_DATA and PROC_PARSE_FRAME PROC_PARSE_FRAME calls PROC_DECIPHER_FRAME PROC_4DW and PROC_3DW. These chain of procedures are called to log the outgoing TLPs into output log (i.e. tx.dat). This is shown in the following source code snippet. PROC_TX_TYPE0_CONFIGURATION_WRITE Page 28 of 66

29 This procedure sends a Type 0 PCI Express Config Write TLP from Downstream Port Model to reg_addr_ of Endpoint DUT with tag_ and first_dw_be_ inputs. Completion returned from Endpoint DUT will use contents of global COMPLETE_ID_CFG as completion ID. Inputs to this procedure are as follows: First PROC_TX_SYNCHRONIZE is called. The first call only synchronizes the trn_clk and trn_tdst_rdyn_n signals. After that the TLP information is put into trn_td_c. The second PROC_TX_SYNCHRONIZE call synchronizes the signals as well as logs the TLP information into local buffer to be parsed and sent to output log. The whole outgoing TLP information is sent to the output log after the transmission of the last TLP data. Page 29 of 66

30 PROC_TX_TYPE1_CONFIGURATION_READ This procedure sends a Type 1 PCI Express Config Read TLP from Downstream Port Model to reg_addr_ of Endpoint DUT with tag_ and first_dw_be_ inputs. CplD returned from Endpoint DUT will use contents of global COMPLETE_ID_CFG as completion ID. The definition of this procedure is same as that for PROC_TX_TYPE0_CONFIGURATION_WRITE. PROC_READ_DATA / PROC_PARSE_FRAME / PROC_DECIPHER_FRAME / PROC_4DW These procedures are the general procedures used to log the TLP information in the output log i.e. tx.dat and rx.dat. The description for these procedures has been provided in the RX_APP section. PROC_TX_IO_WRITE This procedure sends a PCI Express IO Write TLP from Downstream Port Model to IO address addr_[31:2] of Endpoint DUT. CplD returned from Endpoint DUT will use contents of global COMPLETE_ID_CFG as completion ID. The code snippet for PROC_TX_IO_WRITE is as follows: Page 30 of 66

31 PROC_TX_MEMORY_WRITE_32 The inputs for this procedure are as follows: The header format for 32 bit address memory write TLP is as follows: The code snippet for the PROC_TX_MEMORY_WRTE_32 is as shown below: Page 31 of 66

32 Page 32 of 66

33 PROC_WAIT_FOR_READ_DATA This procedure waits for the next completion with data TLP that was sent by the Endpoint DUT. On successful completion, the first DWORD of data from the CplD will be stored in the global P_READ_DATA. This task should be called immediately following any of the read tasks in the TPI that request Completion with Data TLPs to avoid any race conditions. By default this task will locally time out and terminate the simulation after 1000 transaction interface clocks. The global cpld_to_finish can be set to zero so that local time out returns execution to the calling test and does not result in simulation timeout. For this case test programs should check the global cpld_to, which when set to one indicates that this task has timed out and that the contents of P_READ_DATA are invalid. Page 33 of 66

34 This procedure is called from the main test (pio_writereadback_test0) code as follows: PIO Example Design Packet Analysis In the previous section a complete description of the pio_writereadback_test0 test was given. In this section a detail description of TLP packet analysis will be presented by simulating the example design based on pio_weritereadback_test0 downstream port model test suite. To recap the whole working mechanism of this test a flow chart is presented below: Page 34 of 66

35 Page 35 of 66

36 Fmt and Type In packet analysis, it is important to understand what Fmt and Type value in the packet indicate to. The following table shows the Fmt and Type encodings: Page 36 of 66

37 Configuration Write The following waveform shows the zoomed out view of the example design simulation output with the default settings. The first TLP that is initiated by the dsport model is the configuration write transaction. If you open tx.dat you will see following information of the first configuration write transaction: This configuration write transaction is called from the PROC_BAR_PROGRAM. The first call to PROC_TX_TYPE0_CONFIGURATION_WRITE is to program BAR0. The code snippet with the parameters for the first call is as shown below. Our main goal here is to trace the parameters in the tx.dat and the simulation waveform. Page 37 of 66

38 Let s bring in the PROC_TX_TYPE0_CONFIGURATION_WRITE definition: Page 38 of 66

39 Let s look at the header format for the configuration write transaction: Based on the source code provided for the configuration write call and the definition of the procedure, let s track down the parameter values for the above header format. R = 0 Fmt = 10 Type = R = 0 TC = 000 Reserved = 0000 TD = 0 EP = 0 Attr = 00 R = 00 Length = Requester ID = COMPLETER_ID_CFG ( This value is the global constant definition) Tag = 0f (This is passed from the PROC_TX_TYPE0_CONFIGURATION_WRITE call) Last DW BE = st DW BE = f (This is passed from the PROC_TX_TYPE0_CONFIGURATION_WRITE call) Bus Number/Device Number/ Function Number = COMPLETER_ID_CFG Reserved = 0000 Ext Reg Number / Register Number = x010 (provided in the procedure call) The reg_data is the content to be programmed into BAR (0) as shown below: Page 39 of 66

40 reg_data(7 downto 0) & reg_data(15 downto 8) & reg_data(23 downto 16) & reg_data(31 downto 24); Here BAR(0) = reg_data Therefore, Reg_data (7 downto 0) = 0x00 Reg_data (15 downto 8) = 0x00 Reg_data (23 downto 16) =0x00 Reg_data (31 downto 24) = 0x10 Now let s check the waveform and see if the above parameter values are reflected in the waveform or not. The first 64 bits of trn_td in binary representation of the waveform is as follows: The second 64 bits of trn_td in binary representation of the waveform is as follows: If we break this down and put it in the header format above, we will get following: Page 40 of 66

41 There are altogether 9 configuration writes which is also seen in the waveform: Configuration Write Completion The yellow box in the waveform below shows the configuration write completiond. Since there were 9 configuration writes, there are corresponding 9 configuration write completions. Let s check rx.dat for the output log of the first configuration write completion. Let s look at the completion header format: If you zoom in the waveform to trace the first configuration write completion TLP, we get following: Page 41 of 66

42 In the waveform, both completer ID and Requester ID are same i.e. 01A0. The important bit is the Tag. In the first configuration write the Tag was 0F. We see the same tag for the completion as well indicating the completion for the specific configuration write. Now let s look at the second configuration write log and the corresponding configuration write completion log and verify that both have the same Tag or not. From tx.dat From rx.dat Memory Write 64 / Memory Read 64 / Completion Let s explore the Memory_Write_64 TLP as shown in the yellow box below: Page 42 of 66

43 A close-up view is as shown below. It is broken into two parts for clarity. The first two hexadecimal bits (60) indicates that this TLP is Memory Write TLP with 4DW header i.e. the write TLP is addressing the memory location with 64 bits address. Before going further let s check how the BARs were configured. You can see it in the modelsim console. As you can see here, in our current example design configuration the first BAR has been enabled for 64 bits address. Hence, the second bar is automatically disabled. The third BAR i.e. BAR 2 has been enabled to map to 32 bits memory address location. If you refer to the flow chart presented above, the first TLP that will be sent will be PROC_TX_MEMORY_WRITE_64. Let s explore how this procedure has been defined. Page 43 of 66

44 Page 44 of 66

45 PROC_TX_MEMORY_WRITE_64 procedure has been called in pio_writereadback_test0 as follows: Now let s take a look at 4DW Memory Request header: Let s check the corresponding parameters for the TLP as assigned in PROC_TX_MEMORY_WRITE_64 and the input parameters passed during this procedure call. R = 0 Fmt = 11 Type = R = 0 TC = 000 (Passed during procedure call) Reserved = 0000 Page 45 of 66

46 TD = 0 EP = 0 Attr = 00 R = 00 Length = (passed during the procedure call) Requester ID = COMPLETER_ID_CFG (This value is a global constant definition) Tag = 04 (passed during the procedure call) Last DW BE = 0000 (passed during procedure call) 1 st DW BE = 0xf (passed during procedure call) Bus Number/Device Number/ Function Number = COMPLETER_ID_CFG Address [63:32] = BAR (1) = 0x Address [31:2] = BAR (0) = 0x R =00 Let s put first 32 bits from the above parameters together and see if this matches with what we see in the waveform: 0110_0000_0000_0000_0000_0000_0000_0001 From the waveform we get: Let s look into second 32 bits from the above parameter values. We will look into hexadecimal values. 01A0_04_0_f = 01A0040f Now, let s look in the waveform: Next 64 bits is the memory address, from the above parameter definition, should be: _ In the waveform we see the same value: In this TLP, the length of payload size is specified to be 1. Therefore, the payload data of 1DW is attached with this TLP. In pio_writereadback_test0, DATA_STORE is assigned values as follows: Page 46 of 66

47 Now let s go back to PROC_TX_MEMORY_WRITE_64 definition i := i+8; The first condition in the if statement is true therefore DATA_STORE (4,5,6,7) are filled with 0. The while loop will run only once. The above source code will transfer 64 bits data with first 32 bits as and rest 0. Let s see in the waveform if the data is correctly transferred or not: The output log in tx.dat for this TLP is as shown below: Page 47 of 66

48 The yellow box in the following waveform shows the TLP sent from the dsport reaching the trn receive interface at the endpoint. The zoomed in view is as shown below: The data in the yellow box is the garbage data. Since the length of the payload is 1 (i.e. 1DW) this value doesn t count. In pio_writereadback_test0, Write TLP is followed by a Read TLP just to make sure that the value is written correctly. In the waveform below, the blue box is the Write TLP discussed above and the yellow box is the Read TLP (this can be verified by checking the first two Hexadecimal bits which is 20 here). The output log for this read TLP in the tx.dat is as follows: c Page 48 of 66

49 Memory read is the non-posted transaction. Therefore, a completion packet should be sent upstream by the endpoint example design. This completion packet should be visible at the trn transmit interface of the endpoint example design and also at the trn receive interface of the dsport model. Let s trace the packet at these two interfaces in the simulation waveform. In the waveform below, the yellow box shows the completion packet at the user s side. By zooming into the yellow box we see: Before analyzing this packet let s check the format of the completion packet. The first 8 bits in the completion packet i.e. 4A indicates it is a Completion with Data packet. Let s look at the output log for this completion packet received: Page 49 of 66

50 As you can see in the waveform, the correct value is returned in the completion packet. The main interest here is the Tag value. In the Memory read packet the Tag value was 0x05 (shown below for reference) If we check the Tag value in the completion packet it is the same i.e. 0x05 indicating that this completion belongs to the memory read packet that was sent earlier. Also the completion status field is 0x0 indicating that completion is successful: As seen in the yellow box in the completion packet, this is the byte count field. Byte count is defined as follows: Page 50 of 66

51 Since there are four bytes remaining to be received after this 64 bits, the value of 4 here is correct. The following waveform shows the packet being received at the receive trn interface of the dsport model. Following requirement as specified in the specification should be satisfied by the completion packet: Memory Write 32 / Memory Read 32 / Completion The yellow box in the following waveform shows the Memory Write TLP that addresses 32 bit memory address. The memory write TLP is immediately followed by Memory Read TLP to the same memory location. This TLP is included in the same box. Page 51 of 66

52 The close up view of the Memory Write TLP (followed by the Memory Read TLP) is shown below: 40 indicates Memory Write TLP addressing 32 bit memory. 00 indicates Memory Read TLP addressing 32 bit memory. The output log for the memory write packet in tx.dat is as follows: The output log for the memory read packet in tx.data is as follows: The yellow box in the following waveform shows the completion for the above Memory Read packet. Page 52 of 66

53 The close up view of the completion packet is shown below: The output log for this packet in rx.dat is: The yellow box in the following waveform shows the Memory Write and Memory Read from the Expansion Rom which is mapped to BAR6. Limitations and Features of Downstream port model Following are the limitations of downstream port model: 1. The PIO design was created to support at most one IO BAR, one Mem64 BAR, and two Mem32 BARs (one of which must be the EROM space), the Downstream Port Model by default makes a check during device configuration that verifies that the core has been configured to meet this requirement. A violation of this check will cause a warning message to be displayed as well as for the offending BAR to be gracefully disabled in the test bench. This check can be disabled by setting the pio_check_design variable to zero in the pci_exp_usrapp_tx.v file. Page 53 of 66

54 2. Dsport model consists of a parallel test. This test involves more than one process thread. The test sample_smoke_test1 is an example of a parallel test with two process threads. Parallel tests are very useful when verifying that a specific set of events have occurred, however the order of these events are not known. 3. Currently the VHDL version of the Downstream Port Model Test Bench does not support Parallel tests. 4. The Downstream Port Model has a 128-byte MPS capability in the receive direction and a 512-byte MPS capability in the transmit direction. 5. The downstream port and the provided testbench only provide just enough tools to do writes and read to the user design. The downstream port model is not a full blows simulation model as a true BFM available from third party vendors. However, it enables enough functionality to do basic testing of the user design. Limitations of PIO Example Design 1. The PIO design is a simple target-only application that interfaces with the Endpoint for PCIe core s Transaction (TRN) interface. 2. The PIO design only supports single DWORD payload Read and Write PCI Express transactions to 32/64 bit address memory spaces and IO space with support for completion TLPs. 3. The example design supports one IO space BAR, one 32-bit Memory space (that cannot be the Expansion ROM space), and one 64-bit Memory space. If these limits are exceeded, only the first space of a given type will be active accesses to the other spaces will not result in completions. 4. Each space is implemented with a 2 kb memory. If the corresponding BAR is configured to a wider aperture, accesses beyond the 2 kb limit wrap around and overlap the 2 kb memory space. 5. The PIO design successfully processes single DWORD payload Memory Read and Write TLPs and IO Read and Write TLPs. Memory Read or Memory Write TLPs of lengths larger than one DWORD are not processed correctly by the PIO design; however, the core does accept these TLPs and passes them along to the PIO design. If the PIO design receives a TLP with a length of greater than 1 DWORD, the TLP is received completely from the core and discarded. No corresponding completion is generated. 6. PIO design handles Memory writes and IO TLP writes in different ways: the PIO design responds to IO writes by generating a Completion Without Data (cpl), a requirement of the PCI Express specification. 7. The PIO_32_TX_ENGINE and PIO_64_TX_ENGINE modules generate completions for received memory and IO read TLPs. The PIO design does not generate outbound read or write requests. However, users can add this functionality to further customize the design. Playing with Packets Length Parameter Modification As stated in the earlier section, the PIO example design will not generate completion for TLP whose payload is greater that 1. Let s see the behaviour of the simulation if the memory read TLP has Length parameter of 2. Let s change the parameter value in the pio_writereadback_test0 as shown below: If we simulate the resulting design, the simulation times out. The TLP is passed from the dsport to the endpoint. It appears at the receive trn interface of the endpoint of the example design as well but the completion is never generated and hence the simulation times out. In the yellow box in the following waveform, we can see that the TLP with the modified length parameter leaves the trn transmit interface at the dsport model. Page 54 of 66

55 In the waveform below, in the yellow box, the TLP does arrive at the user side but completion is never generated. Let s take a closer look at the modified TLP and check whether the new length value shows up in the outgoing packet or not. Page 55 of 66

56 Poisoning Memory Read TLP Now let s see what happens if the Memory_Read_32 TLP is poisoned. The TLP is poisoned by assigning the value of ep as 1 as shown below. The following code snippet is from PROC_TX_MEMORY_READ_32 procedure in test_interface.vhd.. The packet is generated and passed to the endpoint backend application. The user application detects that the TLP with poisoned data has been received. The message is printed on the console window as shown below: The user guide says following: Page 56 of 66

57 Let s see if we see similar behavior in our simulation or not: Looking at the waveform, it is not clear what the behavior of the completion should be for the poisoned incoming non-posted packet. The completion packet seems to be generated correctly with completions status set to successful. This needs to be clarified. Page 57 of 66

58 Poisoning Configuration Write Request Let s see what happens when configuration write request is poisoned. This is done by modifying the ep bit in PROC_TX_TYPE0_CONFIGURATION_WRITE procedure in test_interface.vhd as shown below: PCI Express specification says: We see this behavior in our simulation as well. Following is the message printed on the console when poisoned configuration write is transmitted from the dsport model. In the following waveform, the red box is the poisoned configuration write request, the yellow box is the corresponding completion for this request. We will first look into whether the output configuration write request is poisoned or not. Then we will look into the incoming completion to see whether the Completion Status field is set to 001 indicating Unsupported Request. Page 58 of 66

59 The following waveform shows the close-up view of the first outgoing configuration write request with poisoned bit ep set to 1. The header format for the configuration request TLP is presented below for the quick reference for the readers. Following waveform shows the corresponding completion with completion status set to Unsupported Request. The header format from the completion TLP is presented below for the quick reference for the readers. Page 59 of 66

60 Testing with a new TLP Let s insert a new TLP that will write to BAR mapped to 32-bit memory address and then read it back. This is done by modifying the tests.vhd file as follows: We are not creating any fancy TLP here. We are just replicating the PROC_TX_MEMORY_WRITE_32 TLP to read and write different data set to the same memory location as would be done by PROC_TX_MEMORY_WRITE_32. For this, a new procedure is defined in test_interface.vhd called PROC_TX_MEMORY_WRITE_321. The content of this procedure will basically remain the same except for DATA_STORE. A new array has been defined called DATA_STORE1. A different set of values is stored in this array as shown in the code snippet above. Let s look at the resulting waveform. In our previous simulation we had 3 TLPs going down from dsport to the endpoint. In this case we will have 5 TLPs. Page 60 of 66

61 Let s take a closer look at the new outgoing TLP. We should see the following data payload with this TLP. DATA_STORE1(0) := X"DE"; DATA_STORE1(1) := X"AD"; DATA_STORE1(2) := X"BE"; DATA_STORE1(3) := X"EF"; The output log in tx.dat for this TLP is as shown below: A Memory Read TLP is issued, let s if we see the same data pattern in the completion sent back from the endpoint for this Memory Read TLP. The output log in rx.dat for this completion packet is as shown below: Page 61 of 66

62 Appendix Some reference contents have been added in this section to make it easier for the readers to do packet analysis and hence debug their design. The contents in this section have been taken from chapter-4 of the PCI Express System Architecture book from Mindshare. Generic TLP Header Fields Page 62 of 66

63 Page 63 of 66

64 IO Request Header Format Memory Request Header Format Page 64 of 66

65 Configuration Request Header Format Message Request Header Format Completion Header Format Few points to note 1. Byte enables bits are high true. A value of 0 indicates the corresponding byte in the data payload should not be written by the completer. A value of 1, indicates it should. 2. If the header Length field indicates a transfer is more than 1DW, the first DW Byte Enabled must have at least one bit enabled. 3. A write request with a transfer length of 1DW and no byte enables set is legal, but has no effect on the completer. 4. If a read request of 1 DW is done with no byte enable bits set, the completer returns a 1DW data payload of undefined data. This may be used as a Flush mechanism. Because of ordering rules, a flush may be used to force all previously posted writes to memory before the completion is returned. 5. The first byte of the data in the payload (immediately after the header) is always associated with the lowest (start) address. 6. Receivers also must check for discrepancies between the value in the Length field and the actual amount of data transferred in a TLP with data. Violations are also handled as Malformed TLPs. Page 65 of 66

66 7. Requests must not mix combinations of start address and transfer length which will cause a memory space access to crass a 4KB boundary. While checking is optional in this case, receivers checking for violations of this rule will report it as a Malformed TLPs. References 1. PCI Express Base Specification Revision 1.1,March 28, LogiCORE IP Endpoint Block Plus v1.9 for PCI Express,UG341 September 19, PCI Express System Architecture by Ravi Budruk, Don Anderson, Tom Shanley, MindShare, Inc. Page 66 of 66