Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide

Transcription

1 Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide Important Note: This downloadable PDF of an answer record is provided to enhance its usability and readability. It is important to note that answer records are Web-based content that are frequently updated as new information becomes available. You are reminded to visit the Xilinx Technical Support Website and review Xilinx Answer for the latest version of this solution. Introduction This document describes techniques for debugging issues related to Integrated PCIe Block Wrapper in Virtex -6 FPGA by using ChipScope analyzer. You should have a clear understanding on the flow of packets through different interfaces of your design when debugging PCIe issues. You should be able to identify what types of packets are seen on those interfaces by analyzing the packet header. This document provides a detailed description on tracking packets through different interfaces in the core. Some helpful techniques on how to select signals for triggering in ChipScope analyzer and how to set the correct trigger to capture packets on different interfaces are discussed. The main objective of this document is to help you debug your design by going into the low level details of the core. This document will help you to debug issues related to both link training and PCIe packet traffic. ChipScope analyzer screenshots are provided to illustrate how to analyze packets and signals. Signal and Packet Analysis Interfaces Figure 1 shows a block diagram of different components in Integrated PCIe Block in Virtex-6 FPGA. It shows different interfaces between those components. The interfaces are numbered to make it easier to reference in rest of the document. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 1

2 Figure 1 Top-level Functional Blocks and Interfaces The main interfaces are as follows: 1. PCIe Link Interface 2. Transceiver Interface 3. MIM Interface 4. TRN TX Interface 5. TRN RX Interface 6. PIO TX Interface 7. PIO RX Interface 8. CFG Interface 9. Physical Layer Interface In this document, it is illustrated on how to use ChipScope analyzer to capture and track a particular packet along each of these interfaces. It does not offer much flexibility in triggering for the correct capture with only one trigger port. If you need to trigger only when a completion packet is presented to the core for a particular incoming read, you would need to trigger only after two conditions are met; memory read packet on the receive interface and the corresponding completion packet on the transmit interface. In the following sections, techniques to capture packets in similar scenarios are discussed. Inserting ChipScope Core This section describes the process of using ChipScope Pro Core Inserter to capture PCIe signals. For more flexibility in triggering and hence being able to trigger in a specific condition, 8 trigger ports have been created. These groups can be categorized as follows: LTSSM signal Global signals TX/RX PIPE interface control/data signals TX/RX MIM interface control/data signals TX/RX TRN interface control/data signals In the example below, all of the data signals have also been included as trigger signals. The main reason for doing this is to be able to trigger ChipScope analyzer based on the data being received or transmitted (e.g. if it is required to Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 2

3 trigger on the incoming TLP at the GT receive interface, the trigger could be set by assigning FB to the GT receive interface data signals). The ChipScope Pro Core Inserter flow is used to capture signals in the design. The input file is the design NGC file. In the case that the signals are not visible in the ChipScope Pro Core Inserter GUI, the reason could be because these signals are optimized away during synthesis. You can avoid this by using KEEP attribute in the source file to stop XST from optimizing a particular signal. This is illustrated in the example code below: In VHDL, declare the KEEP attribute in the file architecture before the begin keyword: attribute keep: string After KEEP and the signal have been declared, specify the VHDL constraint as follows: attribute keep of signal_name: signals is true ; In Verilog, add the following: (* KEEP = "{TRUE}" *) wire signal_name; Following are the steps for using ChipScope Pro Core Inserter. Figure 2 Select NGC File Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 3

4 Figure 3 Select 8 Trigger Ports Figure 3 shows the selection of 8 trigger ports. Each port can be separately configured depending upon the number of signals to be included in each port. Figure 4 shows the Data Width and Data Depth selections, and the width selected for each port. Figure 4 - Set Capture Parameters for Each Port Figure 5 shows an overview of all three ports: Clock Port, Trigger Ports and Data Port. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 4

5 Figure 5 Final Port Structure after Selecting Signals for Each Port Figure 6 shows the module of hierarchy of the example design. The list of signals in each module can be viewed by selecting the modules in the Structure/Nets window. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 5

6 Figure 6 - Example Design Project Structure in ChipScope Analyzer Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 6

7 Trigger Port 0 Figure 7shows a list of signals selected for Trigger Port 0. All of these signals are related to GTs. Also, trn_lnk_up_n signal is listed in this trigger port. When trn_lnk_up_n signal is asserted, it indicates the core has finished link training and is now in L0 state. While you are debugging issues related to link training, it would be useful to check the status of these signals in ChipScope analyzer. These signals should be checked as the first step in debugging link training issues. All of these signals should toggle correctly. GTXRXRESET: This port is asserted High, and then de-asserted to start the full GTX transceiver RX reset sequence. The sequence takes about 120 µs to complete, and systematically resets all subcomponents of the GTX transceiver RX. RXRESETDONE: This port goes High when GTX transceiver RX has finished reset and is ready for use. TXDETECTRX: This input activates the receive detection sequence. The sequence ends when PHYSTATUS is asserted to indicate that the results of the test are ready on RXSTATUS. TXRESETDONE: This port goes High when the GTX transceiver TX has finished reset and is ready for use. PLLLKDET: This active-high PLL frequency lock signal indicates that the PLL has achieved lock. The GTX transceiver and its clock outputs are not reliable until this condition is met. Figure 7 Signals in Trigger Port 0 Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 7

8 Trigger Port 1 This port includes the ltssm signal which is the main signal when debugging link training issues. Whenever there is a problem with the link, it is always advised to check this signal in ChipScope analyzer to see what state the core is in. Under normal working conditions, this signal will show the value indicating L0 state. If the link is bad due to signal integrity issues on the board, the ltssm frequently goes into recovery state which is an indication of a bad link. Trigger Port 2 Figure 8 - Add Signals in Trigger Port 1 This port includes signals coming out of the GT interface. Probing the signals on this interface helps determine whether there are any issues at the physical level. If the host is sending packets and there is no toggling on the rxdata or txdata, it would indicate that the GTs are not properly constrained. During link training, the physical ordered sets TS1s and TS2s are exchanged between the link partners. If the link is not coming up, you should always check this interface to make sure TS1s and TS2s are correctly exchanged. The data on txdata and rxdata are scrambled, however, the control characters such as FB (Start of TLP), FD (End of TLP) and so on, are not scrambled. If you think that a packet is not coming into the core, or is not going out, probe rxdata and txdata respectively, and look for the FB character followed by the FD character. The FB character is qualified by pipe_rxn_char_is_k<n> signal indicating that the corresponding character on rxdata is a control character. The presence of FB on rxdata or txdata indicates the presence of a TLP. In the case when you are generating a MSI interrupt and the MSI packet is not making it to the root complex, the best way to check is to make sure the packet has gone out of the core or not. There is no signal in the core that indicates a packet has successfully transmitted to the PCIe link but if you can see FB on the txdata, that would be a very strong indication that the packet has gone out of the core. MSI packet has a definite size. Check for FB on the GT interface on txdata, and then FD after that. Count the number of bytes between these two characters, if it is equal to the number of a valid MSI message, this would indicate that the MSI packet generated by the core has gone out. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 8

9 Figure 9 - Add Signals in Trigger Port 2 (Part 1) Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 9

10 Figure 10 - Add Signals in Trigger Port 2 (Part 2) Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 10

11 Figure 11 - Add Signals in Trigger Port 2 (Part 3) Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 11

12 Trigger Port 3 Figure 12 - Add Signals in Trigger Port 3 (Part 1) Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 12

13 Figure 13 - Add Signals in Trigger Port 3 (Part 2) Trigger Port 4 This port includes all signals at the receive side of the RX buffer interface (MIM RX Interface). All incoming packets are first stored in this buffer before sending out to the user interface. If a packet is not making it to the user interface, first check this interface by triggering on mim_rx_wen and verify if you can see the packet on this interface or not. If you do see the packet written into the buffer, check whether the packet is read back from the buffer or not. If you see the packet on this interface and not on the user interface, this would be an indication that the packet has been Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 13

14 dropped inside the core. This is very unlikely to happen but if you run into any such scenarios, please create a WebCase with Xilinx Technical Support. Trigger Port 5 Figure 14 - Add Signals in Trigger Port 4 Similar to the incoming packets, the outgoing packets are also buffered in the TX Buffer before they are transmitted. As discussed for the receive buffer, perform a similar test for the receive buffer in the case where you could see a packet being transmitted by the user application on the TRN TX interface (Figure 1, Interface 4), but it is not showing up at the host. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 14

15 Figure 15 TX Memory Interface Signals in Trigger Port 5 Trigger Port 6 This port includes all receive side signals between the core and the user application. If the host is sending packets downstream and you think your backend application is not receiving those packets, then the first thing to do is to check this interface. One probable cause for the packet not making it to the user interface might be that the incoming packet is not hitting the BAR range. If this is the case, the corresponding signals: trn_rbar_hit_n<n> will not be asserted. In that case, you should make sure the application at the host is targeting the correct memory address. If you want to check whether or not any packets are making it to the user application, set your trigger on trn_rsof_n. In the case where the host is performing memory read from the endpoint user application and you are not getting the corresponding completion back from the endpoint, you should first verify whether or not you can see Memory Read TLP on the trn receive interface. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 15

16 Figure 16 TRN Receive Interface Signals in Trigger Port 6 Trigger Port 7 This port includes the TX side of the signals between the user application and the core. If the host is not receiving completions for the memory reads, probe this interface and verify whether or not the completion packet has been generated by your user application and presented to the core. In the case when there are no sufficient credits at the host, the core de-asserts trn_tdst_rdy_n. This signal is also deasserted if the core is not ready to accept packets from the user application. If you do not see any packets coming in at the host from the endpoint, trigger ChipScope analyzer on this signal and make sure it is not de-asserted indefinitely. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 16

17 Figure 17 TRN Transmit Interface Signals in Trigger Port 7 Clock Signal Figure 18 - Clock Signal Select Data the same as Trigger in Figure 4. This will add all trigger signals in the data waveform as well. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 17

18 Figure 19 - Add Data Signals After inserting the ChipScope IP core, run the implementation again (without running the synthesis) to generate the bitstream. After programming the device on ML605, open ChipScope analyzer, the ChipScope IP core should be detected by the analyzer. LTSSM Signal Analysis After the FPGA configuration, the two connected devices go through the link training process. The main states to consider while debugging link training issues are DETECT, POLLING, CONFIGURATION, and L0. Please refer to Chapter 4 of the PCI Express Base Specification for detailed descriptions of these LTSSM states. The section provides a list of signals to capture for debugging a link training issue in the ChipScope tool along with a detailed analysis of the signals. A number of ChipScope tool waveform screenshots are provided for each LTSSM state to illustrate the toggling of signals. If you are capturing signals in the ChipScope tool, compare your captures with the screenshots provided below to make sure the signals in your design are toggling correctly. The captures are taken from the PIO Example Design of Virtex-6 FPGA Integrated Block for PCI Express v1.7 generated from the CORE Generator graphical user interface for ML605 board. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 18

19 LTSSM States The signal below indicates which LTSSM state the core is currently at. The core goes to recovery state to achieve bit lock and symbol lock. If the ChipScope tool capture shows frequent transition to the recovery state, it normally indicates a noisy link. pl_ltssm_state<5:0> Different states of the link training state machine (indicated by pl_ltssm_state) are shown in Table 1: b:Det Quiet b:Det Quiet Gen b:Det Active b:Det Active Second b:Pol Active b:Pol Config b:Pol Comp Pre Send Eios b:Pol Comp Pre Tim-out b:Pol Comp Send Pattern b:Pol Comp Post Send Eios b:Pol Comp Post Time-out b:Cfg Lwidth St b:Cfg Lwidth St b:Cfg Lwidth Ac b:Cfg Lwidth Ac b:Cfg Lnum Wait b:Cfg Lnum Acpt b:Cfg Complete b:Cfg Complete b:Cfg Complete b:Cfg Complete b:Cfg Idle b:L b:L1 Entry b:L1 Entry b:L1 Entry b:L1 Idle b:L1 Exit b:Rec Rcvrlock b:Rec Rcvrcfg b:Rec Speed b:Rec Speed b:Rec Idle b:Hot Rst b:Disabled Entry b:Disabled Entry b:Disabled Entry b:Disabled Idle b:Dp Cfg Lwidth St b:Dp Cfg Lwidth St b:Dp Cfg Lwidth St b:Dp Cfg Lwidth Ac b:Dp Cfg Lwidth Ac b:Dp Cfg Lnum Wait b:Dp Cfg Lnum Acpt b:To 2 Detect b:Lpbk Entry b:Lpbk Entry b:Lpbk Active b:Lpbk Exit b:Lpbk Exit b:Lpbkm Entry b b: Reserved Table 1 : LTSSM States pipe_rx_status indicates the receiver status and error codes as shown in Table : Data Received OK 001: One Skip Symbol (SKP) added 010: One SKP removed 011: Receiver detected 100: 8B/10B decode error 101: Elastic Buffer overflow 110: Elastic Buffer underflow 111: Receive disparity error Table 2 : pipe_rx_status Signal Figure 20 shows the LTSSM state at Pol Config on Trigger Port 1 Figure 20 - Set Different LTSSM States to Trigger Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 19

20 DETECT State Figure 22 shows a capture of the GT interface signals when the ChipScope tool is triggered on Detect Active state (pl_ltssm_state<5:0> = ). Figure 21 - Trigger LTSSM State - Detect Active (Part 1) Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 20

21 Figure 22 - Trigger LTSSM State - Detect Active (Part 2) During the DETECT state, the receiver detection takes place on each lane. If the detection process is done correctly, the following sequence should be observed in the ChipScope tool capture; this is also described in (Xilinx Answer 39525). PCIe Hard Block asserts TxDetectRx. GTP performs receiver DETECT. After the receiver is detected, GTP asserts pipe_rx_phy_status and puts 011 on pipe_rx_status to indicate the receiver is present. PCIe Hard Block then de-asserts TxDetectRx and pipe_tx_elec_idle. Receiver detect is used by the transmitter to detect if a receiver is present at the other end of the link. The Virtex-6 FPGA Integrated Block for PCI Express and GTX Transceivers support this functionality. However, due to various reasons such as board signal integrity or issues with the link partner receiver, sometimes it is helpful to bypass the receiver detect function and allow the core to go straight to the LTSSM polling state. For more details on how to bypass receiver detect, refer to (Xilinx Answer 45859). For more details on debugging receiver detect issue, see (Xilinx Answer 39380). Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 21

22 The ChipScope Pro tool capture of the above sequence is shown in Figure 23. Figure 23 - Detect State Details Another signal that you should pay attention to is the plllkdet signal. This port indicates that the VCO rate is within acceptable tolerances of the desired rate; in other words, the assertion of this signal indicates that the internal PLL is successfully locking on to the incoming reference clock. The clock_locked signal, as shown in Figure 24, indicates whether or not the fabric PLL has successfully locked. For successful link training, both the plllkdet and clock_locked signals should be asserted. Therefore, before capturing other signals to debug link training issues, it is advised to check these signals first to make sure both of them are asserted. Figure 24 - plllkdet and clock_locked Signals During DETECT When each link partner enters into POLLING state, it begins transmitting TS1 ordered sets. However, each link partner might not enter POLLING at the same time, so it is possible that the Xilinx endpoint might be transmitting TS1s on pipe_tx_data while still receiving 00h on the pipe_rx_data pins. Hence, in the ChipScope Pro tool, when TS1 appears at pipe_tx_data, pipe_rx_data might still be 00. POLLING State To check whether or not TS1 transmission has started, trigger the ChipScope tool when ltssm_state enters POLLING. The screenshots below show the ChipScope tool capture of the signals when the endpoint device enters POLLING.ACTIVE state. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 22

23 Figure 25 - Trigger LTSSM State Polling.Config (Part 1) Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 23

24 Figure 26 - Trigger LTSSM State Polling.Config (Part 2) Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 24

25 CONFIGURATION State Figure 27 - Trigger LTSSM State Configuration Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 25

26 Figure 28 - Trigger LTSSM State - Configuration Idle (Part 1) Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 26

27 Figure 29 - Trigger LTSSM State - Configuration Idle (Part 2) After CONFIGURATION, the next state is L0 which is the normal working state. As shown in Figure 28, the initial phase of link training completes after trn_lnk_up_n is asserted. For more information on this link trained signal, see (Xilinx Answer 39522). At times, the link might not come up if configured for higher lane width due to signal integrity issues on the board. In such cases, the best way to check is to force the link to train down by taping off the upper lanes as described in (Xilinx Answer 38988). For more information on debugging link training issues, see (Xilinx Answer 34873). Common errors that can occur are 8b/10b errors which can lead to the link being lost, the link not coming up, and the link training down to a lower lane width. For more information on 8b/10b errors, refer to (Xilinx Answer 39590). When ASPM is enabled at the host, it might cause an issue during link training. It is recommended to disable ASPM. For more details on how to do disable ASPM, see (Xilinx Answer 36325). Although trn_link_up_n is asserted, sometimes the device might not be detected by the system. This can be caused by the FPGA configuration timing. For more information and on how to debug this issue, see (Xilinx Answer 34800). During normal running of the system, it could freeze. Although this might be due to various reasons, one probable cause could be completion timeouts which is described in (Xilinx Answer 35034). Other causes could be the generation of malformed TLPs due to incorrect use of trn_trem_n as described in (Xilinx Answer 35748). Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 27

28 Tracking Received TLPs This section describes how to track an incoming TLP packet all the way from the GT RX interface to the user application RX interface. As shown in Figure 30, an incoming TLP packet (highlighted with red arrows) will arrive on the RX GTX interface (2), and goes into and comes out of the MIM interface (3), and the core then presents it on the TRN RX interface (4). Figure 30 Incoming TLP Packets on Different Interfaces The packet that was tracked in the example below was performed by writing to the backend memory of the example design user application and reading back from the same memory location in the PciTree. The incoming TLP packet on the GT RX interface is identified by checking the FB (start of TLP packet) character on rxdata. The example design was generated for four lanes, the FB (1byte) will start on pipe_rx0_data, and FD (which indicates the end of TLP packet) will be on pipe_rx3_data. The ChipScope analyzer setup, described earlier in this document, has 7 trigger ports in total. In M2 and M3, there are PIPE RX and TX interface signals respectively. To trigger on an incoming FB, trigger on either the lower 8-bit or the upper 8-bit of pipe_rx0_data by setting it to FB as shown below: pipe_rx0_data = 16 b FBXX & pipe_rx0_char_is_k = 2 b10 OR pipe_rx0_data = 16 b XXFB & pipe_rx0_char_is_k = 2 b01 The input pipe_rx0_char_is_k signal determines the control bit(s) for the received data; the lower bit corresponds to the lower byte of pipe_rx0_data, while the upper bit corresponds to the upper byte. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 28

29 Tracking Vendor Defined Messages Figure 31 and Figure 32 show how to set the trigger in Trigger Port M2 to track the start of the TLP packet (FB) which could be on either the higher or lower byte of pipe_rx0_data[15:0]. Figure 31 - Trigger FB on pipe_rx_data with Upper Data Byte Set If the above setting does not trigger, try setting the FB trigger on the lower byte of the data as shown below: Figure 32 - Trigger FB on pipe_rx_data with Lower Data Byte Set Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 29

30 Meanwhile, you must enable M2 to be in the Trigger Condition Equation as shown in Figure 33. Figure 33 - Enable M2 as Trigger Press the trigger button in the ChipScope tool. The next step is to perform Memory Write in PciTree. The expectation is that the ChipScope tool will get triggered with the incoming Memory Write packet. However, in this particular test, even without performing Memory Write in the PciTree, ChipScope gets triggered due to Vendor Defined messages that are sent by the host to the Endpoint. Figure 34 shows a TLP coming from the host to the Endpoint. FB is on pipe_rx0_data and the corresponding FD is on pipe_rx3_data on the GTX interface. Figure 34 - Vendor Defined TLP Packet on GTX Interface Figure 35 shows a close-up view of the packet on the MIM interface. The xx73. on the mim_rx_wdata when mim_rx_wen is asserted indicates a Vendor Defined Message. Figure 35 - Vendor Defined TLP Packet on MIM Interface The same packet travels further to the TRN interface as shown in Figure 36. The signal trn_rsof_n is asserted with 0x7300_0001_0000_007F on trn_rd. The data starting with 0x73 means a Vendor Defined Message TLP. Figure 36 Vendor Defined TLP Packet on TRN Interface Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 30

31 Tracking Memory Write TLP Since the ChipScope tool is getting triggered anyway due to the host sending these Vendor Defined Messages continuously to the Endpoint core, the main question here is how to trigger only when Memory Write TLP is received in the core. 0x40 at the start of a TLP indicates a Memory Write TLP packet. Set the trigger condition when trn_rsof_n is asserted and trn_rd = 0x40xx_xxxx_xxxx_xxxx as shown in Figure 37. Figure 37 - Trigger on trn_rsof_n and Memory Write TLP Then, set the trigger condition to M6 as shown in Figure 38. Figure 38 - Enable M6 Port to Trigger Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 31

32 After pressing the trigger button in the ChipScope tool, perform a Memory Write 0xdeadbeef to 0x in the PciTree as shown in Figure 40. After the ChipScope tool triggers, the next step is to track back the packet in these interfaces: TRN -> MIM -> PIPE, and verify whether or not the 0xdeadbeef packet appears on all these interfaces. Figure 39 PciTree Xilinx PCI Express Block Plus Configuration Space Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 32

33 Figure 41 shows deadbeef on the TRN interface. Figure 40 - Memory Write using PciTree Figure 41 - Memory Write TLP packet on TRN Interface Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 33

34 If the incoming Memory Write TLP is correct and is hitting the correct BAR address, trn_rbar_hit_n<0> should be asserted as shown in Figure 42. Figure 43 shows the same packet on the MIM interface. Figure 42 - trn_rbar_hit_n<0> is Asserted Figure 43 - Memory Write TLP Packet on MIM Interface Figure 44 shows the same packet on the PIPE interface. As shown in the figure, FB on pipe_rx0_data with pipe_rx0_char_is_k asserted indicates the start of the Memory Write TLP. The corresponding FD on pipe_rx3_data with assertion of pipe_rx3_char_is_k indicates the end of the Memory Write TLP. The packet in Figure 44 is definitely a Memory Write TLP since you are tracking the packet back from TRN and MIM interface. It is not possible to tell whether the packet is Memory Write TLP, Memory Read TLP, or some other packet types by just checking the packet data on the PIPE interface because the packets are scrambled. Figure 44 - Memory Write TLP Packet on PIPE Interface Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 34

35 Tracking Memory Read and Completion TLP When performing a Memory Read request, as it is a Non-Posted request, there will be a corresponding completion TLP coming back from the Endpoint. As shown in Figure 45, the memory read TLP (highlighted in red) will go through interfaces (2) -> (3) -> (4) -> (6); and the completion TLP (highlighted in green) will come back through interfaces (7) - > (5) -> (3) -> (2). Figure 45 Tracking Memory Read and Completion Packet To track both Memory Read TLP and the corresponding completion packet, use triggers on both the TRN RX and TRN TX side. When you perform a memory read, trn_rsof_n is asserted with a TLP header trn_rd 0x00xx_xxxx_xxxx_xxxx on the TRN RX interface. The 0x00 means a 32-bit Memory Read request. On the TX side of the TRN interface, there will be the corresponding Completion. To trigger on this packet, set trigger with trn_tsof_n asserted and trn_td = 0x4Axx_xxxx_xxxx_xxxx. Figure 46 and Figure 47 show the correct trigger settings as described. Figure 46 - Triggers for Memory Read Packet Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 35

36 Figure 47 - Triggers for Completion Packet from Endpoint As the Memory Read comes first, then is followed by the completion packet, perform the trigger sequence as shown in Figure 48. Figure 48 - Enable Sequence Trigger Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 36

37 After setting the triggers as described above, press the trigger button in the PciTree as shown in Figure 49. button, and then click on Figure 49 - Auto Read Memory in PciTree Figure 50 shows a capture of TRN interface with Memory Reads and the corresponding completion packets. Figure 50 - Memory Read and Completion Packet Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 37

38 Figure 51 shows a close-up view of Figure 50. On the TRN RX interface, trn_rsof_n is asserted first with 0x00xx_xxxx_xxxx_xxxx, which means a 32-bit Memory Read request. On the TRN TX interface, there is the corresponding completion packet with the assertion of trn_tsof_n as indicated by the cursor. Figure 51 - Completion Packet In Figure 51, there are two completion packets on the TRN TX interface. The main question here is which completion packet belongs to the Memory Read on the TRN RX interface. This can be known by checking the TAG field of the packet header. Figure 52 shows the Memory Read request header format. Figure 52 - Request Header Format for 32-bit Addressing of the Memory In Figure 51, trn_rd = 0x0000_0001_0000_070F. 0x0000_0001_0000_070F= 0000_0000_0000_0000_0000_0000_0000_0001_0000_0000_0000_0000_0000_0111_0000_1111 Figure 53 shows the analysis of the Memory Read packet header which gives the TAG as 0x07. Figure 53 - Memory Read Packet Header Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 38

39 The two completion packets in Figure 51 are as follows. trn_td = 0x4a00_0001_0200_0004_0000_0704_0000_0000 trn_td = 0x4a00_0001_0200_0004_0000_0808_EFBEADDE The Completion header format is shown in Figure 54. Figure 54 - Completion Header Format The breakdown of the first completion packet header will be as follows: 0x4a00_0001_0200_0004_0000_0704_0000_0000= 0100_1010_0000_0000_0000_0000_0000_0001_0000_0010_0000_0000_0000_0000_0000_0100_ 0000_0000_0000_0000_0000_0111_0000_0100_0000_0000_0000_0000_0000_0000_0000_0000 Figure 55 - Completion Packet Header Figure 55 shows the TAG value in this completion packet as 0x07 which is the same as the TAG value in the Memory Read TLP. The second completion packet has TAG field of 0x08. Since multiple trigger was done in the ChipScope tool, auto memory read in PciTree reads all memory locations in the address range, they are read back as all 0x0000_0000 which is shown in Figure 56. Figure 56 - Memory Read Request with Completion Packets with data = 0x0000_0000 Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 39

40 Figure 57 and Figure 58 show the Memory Read packet in Figure 53 on MIM interface. Figure 57 - Memory Read on MIM Interface Figure 58 - Memory Read (0xdeadbeef) on MIM Interface Figure 59 shows the Memory Read data 0x0000_0000 on MIM interface shown in Figure 56. Figure 59 - Memory Read (0x ) on MIM Interface If you do not see completion packets coming to the host, make sure the completion packet is sent by the user application to the Endpoint core by probing the TRN interface, then check the MIM interface, and then the PIPE interface. If you see the completion packet on the TRN interface, you can check whether or not that packet made it to the PIPE interface by tracking the corresponding FB character on the PIPE TX interface. Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 40

41 Set M3 pipe_tx0_data [0:7] = FB or pipe_tx0_data [8:15] = FB as shown in Figure 60 and Figure 61. Figure 60 - Trigger on FB on pipe_tx0_data Higher Byte Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 41

42 Figure 61 - Trigger on FB on pipe_tx0_data Lowe Byte Set the trigger as follows: M6 (Memory Read on TRN RX Interface) -> M7 (completion packet on TRN TX Interface) -> M3 ( FB on PIPE Interface). Figure 62 Sequence Trigger to Track Outgoing FB on pipe_tx_data for the Corresponding Completion Packet on TRN TX Interface Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 42

43 Figure 63 and Figure 64 shows FB and FD for completion packets on TX PIPE interface. Figure 63 - Memory Read on PIPE Interface Conclusion Figure 64 - FB and FD Shown on PIPE Interface This document presented different aspects of debugging issues in the Virtex-6 FPGA Endpoint Block Wrapper by going through the packet and signal analysis. If the user of the core is experiencing issues with link training or the packets are not transmitted or received correctly, it is recommended to go through this document and check the provided suggestions. It is expected that with this document, the user will be able to capture the relevant signals specific to the issue in the ChipScope tool, and perform analysis of the captured waveform to figure out where the problem might be. If this document does not help to resolve the problem, please create a WebCase with Xilinx Technical Support. Attach all of the captured ChipScope Pro tool waveforms, and the details of your investigation and analysis. References 1. Virtex-6 FPGA Integrated Block for PCI Express User Guide 2. Xilinx PCI Express Documents 3. Virtex-5 Integrated PCI Express Block Plus - Debugging Guide for Link Training Issues 4. Virtex-5 Endpoint Block Plus for PCI Express - Debugging and Packet Analysis Guide with Downstream Port Model and PIO Example Design 5. IP Release Notes 6. Xilinx Solution Center for PCI Express 7. ChipScope Pro Tutorials 8. ChipScope Pro User Guide Revision History 08/16/ Initial Release Xilinx Answer Virtex-6 Integrated PCIe Block Wrapper Debugging and Packet Analysis Guide 43