Freescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, ColdFire+, C-Ware, the Energy Efficient Solutions logo, Kinetis,

Transcription

1 June 2012 Freescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, ColdFire+, C-Ware, the Energy Efficient Solutions logo, Kinetis, mobilegt, PowerQUICC, Processor Expert, QorIQ, Qorivva, StarCore, Symphony and VortiQa are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. Airfast, BeeKit, BeeStack, CoreNet, Flexis, MagniV, MXC, Platform in a Package, QorIQ Qonverge, QUICC Engine, Ready Play, SafeAssure, the SafeAssure logo, SMAROS, TurboLink, Vybrid and Xtrinsic are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners Freescale Semiconductor, Inc..

2 PCI Express features a rich set of error detection, logging and signaling mechanism. PCI Express Data Link Layer offers hardware protocol to automatically guard against possible data corruption for transaction in flight over a link. For complicated errors uncorrectable by hardware automatically, PCI Express offers further classification as fatal and non-fatal errors. Advanced Error Reporting (AER) Capability provides flexible error handling by allowing some fatal uncorrectable error to be signaled and handled as non-fatal error such that further recovery action could be implemented by system hardware and software together. Unsupported Request and Completion Timeout Errors are some of the common errors that might potentially impact system stability. Properly handling these errors draws increasingly attention in most PCI Express applications. 2

3 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 3

4 4

5 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 5

6 CPU RCRB Memory Bus# 0 Virtual PCI Bridge Virtual PCI Bridge Virtual PCI Bridge Bus# 1 PCIe Root Complex (RC) Bus# 6 Bus# 7 PCIe Endpoint Bus# 3 Bus# 2 Switch PCIe Endpoint PCIe-PCI/PCI-X Bridge Bus# 8 Bus# 4 Bus# 5 PCIe Endpoint PCIe Legacy Endpoint 6

7 Root Complex Device Downstream Port Upstream Port Links Switch Downstream Port Endpoint Endpoint Upstream Port Endpoint 7

8 Link Collection of two ports and their interconnecting lanes Lane A set of differential signal pairs: one pair for Tx and another for Rx. TX0 P N RX0 P N Upstream Device TXn P N RXn P N Downstream Port Port Lane 0 Lane n Physically, a group of transmitters and receivers located on the same chip that define a link Logically, an interface between a component and a PCI Express Link x1, x2, x4, x8, x16,.. xn (Link) A by-n link is composed of N lanes Channel (P-N Pair) P N RX0 P N TX0 Link Downstream Device P N P N RXn TXn Upstream Port 8

9 Packet-based split-transaction protocol Provides R/W logical transactions to software 4 basic transaction types: memory, I/O, configuration and message 32-bit and 64-bit memory addressing Three routing methods Address routing (memory and I/O) ID routing (configuration) Implicit routing (messages) Transactions are carried by Transaction Layer Packets (TLPs) Root Complex or Switch Ack Req D DLLP FC Update DLLP Req 4 TLP Req C TLP Ack Req C DLLP Ack Req 3 DLLP Req 3 TLP Req D TLP FC Update DLLP Ack Req 4 DLLP End point 9

10 10

11 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 11

12 The Goal To guard against potential data corruption The Scope within the PCI Express System Fabric A transaction in flight over a link or within the PCI Express fabric Errors occur on the PCI Express interface itself Errors occur on behalf of transaction initiated on PCI Express Does NOT cover errors that occur within the component that are unrelated to a particular PCI Express transaction 12

13 Error Reporting & Signaling Completion & Message Error Logging Mandatory Baseline Capability Optional Advanced Error Reporting (AER) Capability Error Detection Where & How to Detect Error 13

14 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 14

15 15

16 Types of the Correctable Errors Bad TLP (bad LCRC or incorrect Sequencer Number) Bad DLLP Replay Timer Timeout Replay Number Rollover Receiver Error (for example, Framing error) Characteristics of the Correctable Errors Automatic HW recovery action, no software intervention Performance will degrade No information loss Normally software does not need to monitor 16

17 Example 1: 1) Sender sends a TLP (Transaction Layer Packet) over the link 2) Recipient receives a Bad TLP with bad LCRC due to link signal integrity 3) Recipient s DLL detects the Bad TLP and discards it 4) Recipient sends a NAK DLLP to sender 5) Sender HW automatically resends (Retry or Replay) the same TLP Example 2: 1) Sender sends a TLP (Transaction Layer Packet) over the link 2) Recipient never acknowledges the packet (no ACK is returned to sender) 3) Sender s Replay Timer expires 4) Sender resends the same TLP and logs the Replay Timer Timeout error status 17

18 Uncorrectable Non-Fatal Errors Unsupported Request (UR) Completion Timeout (CTO) Completer Abort (CA) Unexpected Completion Poisoned TLP Received ECRC Check Failed Uncorrectable Fatal Errors Link Training Error DLL Protocol Error Receiver Overflow Flow Control Protocol Error Malformed TLP Error 18

19 Characteristics of Uncorrectable Errors Impact the functionality of the interface No mechanism defined in the specification to correct these errors Reporting an uncorrectable error is similar to asserting SERR# in PCI/PCI-X Further Classification of Uncorrectable Errors Fatal Errors Errors cause the particular link and related hardware unreliable Reset of the component on the link may be required How to handle is platform implementation specific Non-Fatal Errors Errors cause a particular transaction to be unreliable But the link is otherwise fully functional This classification provides related HW or SW an opportunity to recover the error without resetting the components on the link and disturbing other transactions in progress 19

20 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 20

21 Poisoned TLP Rcvd, ECRC Error, Unsupported Request, Completion Timeout, Completer Abort, Unexpected Completion, Receiver Overflow, FC Protocol Error, Malformed TLP DLL Protocol Error Training Error Rx Side Uncorrectable Error Checking Seq# Header Header TLP TLP TLP Transaction Layer Data Data Data Link Layer LCRC Physical Layer Correctable Error Checking Bad TLP & DLLP, Replay Timer TO, Replay# Rollover Errors Downstream Port Framing Start Seq# Header Data LCRC Framing End Receiver Error (Framing) Link Downstream Device Upstream Port 21

22 RC Mode Applicable for RC Only 22

23 EP Mode 23

24 Applicable for RC Only 24

25 Finer granularity in defining the actual type of error occurred within each classification For each type of uncorrectable error, the AER provides the ability: To specify its severity, and, To determine if it should be reported as fatal or non-fatal error Support for logging errors Enable/disable RC to report errors to the system Identify source of error Ability to mask reporting individual types of errors 25

26 Uncorrectable Uncorrectable Error Status Register Provides additional information for uncorrectable error Uncorrectable error status bit is set independent of the Mask Register setting Uncorrectable Error Mask Register Provides 11 uncorrectable error mask bits to let device control whether to mask or unmask a particular uncorrectable error condition to be reported to RC. A masked error: Is not logged in the Header Log Register Does not update the First Error Pointer Is not reported to the RC by device Only affects the error reporting, not the status bits Uncorrectable Error Severity Register Provides 11 uncorrectable error severity control bits to let device control whether to treat a particular uncorrectable error condition detected as fatal or non-fatal. 26

27 Correctable Correctable Error Status Register Provides additional information for 6 correctable errors Correctable error status bit is set independent of the Mask Register setting Correctable Error Mask Register Provides 6 correctable error mask bits to let device control whether to mask or unmask a particular correctable error condition to be reported to RC A masked error: Is not logged in the Header Log Register Is not reported to the RC by device Only affects the error reporting, not the status bits 27

28 RC-Only Root Error Status Register ERR_COR Received status bit is set whenever the Correctable Error is detected by root port or upon receipt of an ERR_COR Message. Multiple ERR_COR Received status bit is set when a new correctable error condition is true and the ERR_COR Received bit is already set. ERR_FATAL/NONFATAL Received status bit is set whenever the Uncorrectable Error is detected by root port or upon receipt of an ERR_FATAL or ERR_NONFATAL Message. Multiple ERR_FATAL/NONFATAL Received status bit is set when a new uncorrectable error condition is true and the ERR_FATAL/NONFATAL Received bit is already set. Error status bit is set independent of the Root Error Command Register setting Root Error Command Register Provides independent control to enable interrupt generation for all three categories of errors: Correctable, Uncorrectable Fatal and Uncorrectable Non-Fatal Source ID Register Captures the Error Message ID associated with the First Fatal and First Non-Fatal Error Messages received by the RC. Captures the Error Message ID associated with the ERR_CORR message received by the RC. 28

29 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 29

30 Completion TLP Header Byte 0 R Fmt =x0 Type = R TC R T D E P Attr R Length Byte 4 Completer ID Compl Status B C M Byte Count Byte 8 Requester ID Tag R Lower Address Status Code Completion Status Definition 000b Successful Completion (SC) 001b Unsupported Request (UR) can also be reported via Error Message 010b Configuration Request Retry Status (CRS) 011b Completer Abort (CA) can also be reported via Error Message 100b 111b Reserved 30

31 Error 4DW Error Message TLP Header Byte 0 R Fmt =01 Type = R TC = 000 R T D E P Attr =00 R Length = Byte 4 Byte 8 Byte 12 Requester ID Tag Message Code (Reserved) (Reserved) Fmt = 01b: Message Request without Data Type [4:3] = 10b: Message Request Type [2:0] = 000b: Message Routed to RC Message Code 0x30h 0x31h 0x33h Name ERR_COR ERR_NONFTATAL ERR_FATAL 31

32 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 32

33 e500mc Core Completion Queue 7 Internal Completion w/bad data UR Cpl CoreNet PCIe Root Complex 1 Load Instruction Load Queue 4 PCIe Endpoint MRd P P4080 host software issues a load instruction via core targeting a PCI Express downstream EP 2. The load instruction is decoded and dispatched to issue queue. It will then be tracked at completion queue. 3. The guarded load instruction reaches the bottom of the completion queue. That s its term to be executed. 4. The load instruction is converted to an outbound MRd transaction and passed down through CoreNet, OCN, PCI Express RC, which will convert it into a PCI Express MRd transaction and send through link to reach the downstream EP. 5. The downstream EP for some reason cannot handle the MRd. It then returns a Completion TLP w/status as Unsupported Request. 6. PCI Express RC terminates this MRd transaction locally and returns the status marked as bad data. 7. Core noticed the result of this transaction marked with bad data. Bit it expects the correct return data before it can complete/finish the instruction. The core can t complete this load instruction normally. 33

34 Options available when receiving return data marked as bad 1) Pass the bad data to the core Not a good choice. Only for PQ2s. It will cause further damage to the system with the propagated bad data. 2) Stall further execution of the code Little bit better choice. This is the PQ3 products implementation choice. Side effect is that core will hang unless an asynchronous machine check handler in place to rescue. 3) Terminate the memory transaction while prevent the load instruction from completion. Pause further code execution. Best Choice. e500mc core s implementation choice. This is the Synchronous Error Report Machine Check feature in e500mc core. Core s memory transaction is terminated with a data error such that the core is not hung awaiting the transaction to return good data Stops further propagation of bad data by taking a synchronous error report machine check. Code execution is paused and wait for further information from the other asynchronous machine check or interrupt handler to resolve this exception. 34

35 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 35

36 What does UR stand for? UR = Unsupported Request Watch out! This UR (Request) is NOT that UR (Completion). Always clarify the reference first. One reference of UR Unsupported Request The TLP Type is a Posted or Non-Posted Request It means the completer receives a Posted or Non-Posted Request (for example, MWr/Rd, IOWr/Rd, etc.); however, can t handle it for some reason. Therefore, it treats this request as Unsupported Request. The recipient of this Request TLP is the Completer. This TLP is NOT a Bad TLP. It may be a totally valid TLP without any error. The other reference of UR Completion with UR completion status The TLP Type is Completion It means once the completer treats a Non-Posted Request as Unsupported Request, it will return a Completion TLP with its status code marked as UR (001b). The recipient of this Completion TLP is the Original Requester. Some might simply call it UR Response, UR Completion or Completion w/ur status. 36

37 Why is a Request TLP treated as UR at Completer? Request type not supported Request does not reference address space mapped within device Request contains address that cannot be routed to any egress port of a bridge or a switch (address not specified in any port s Base and Limit Registers) Another very common cause is that the PCIe switch s downstream port experienced link down due to EP s hardware or software failure. A transaction targeting a device on a PCI bus, but is Master Aborted after the request was accepted by the bridge. A configuration access that targets an un-implemented Bus, Device or Function results in termination of the transaction, and a completion transaction is returned with UR status. This is very common during PCI/PCIe device tree enumeration. It can be safely ignored if the configure cycle is initiated by Config_Addr/ Config_Data mechanism This is the only kind of UR can be ignored! Message request received with unsupported or undefined message code. 37

38 The UR is a PCIe Uncorrectable Non-Fatal Error. Who is mainly responsible for detecting and reporting UR Error? The base specification places the burden on the Completer. The recipient of the original request is responsible. Or, the one decides to treat a request as UR is responsible for reporting error. The original Requester has minimum information for an UR Error Requester Completer MRd Request Root Complex or Switch UR Completion TLP End point TLP 38

39 Completer s The scenario is as a Completer, It receives an Unsupported Request TLP from the original transaction Requester (or Initiator) It can send an ERR_FATAL or ERR_NONFATAL Error Message (depending on severity setting) toward RC, if error reporting is enabled with either Command Register [SERR_EN] or Device Control Register [URR and FER or NFER], regardless the TLP type It may return a Completion with UR status if the UR TLP type is Non-Posted I am here! I am going to treat you as UR. Requester Root Complex or Switch UR Completion MRd Request ERR_x Message TLP Completer End point TLP PCIe Error Message 39

40 Completer s The UR Error (receiving an Unsupported Request) is logged in the Completer s configure space: In the traditional PCI-compatible area: None In the PCIe Baseline Error Reporting (PCIe Capability) area: Freescale PCIe Configure Offset 0x56, Device Status Register [URD, NFED] In the optional Advanced Error Reporting (PCIe AER) area: Freescale PCIe Configure Offset 0x104, Uncorrectable Error Status Register [URE] Freescale PCIe Configure Offset 0x11C-128, Header Log Registers 40

41 Requester s The scenario is: as a Requester, It sends a request TLP out to a completer, which could be RC or EP The completer for some reason treats this request as an Unsupported Request: If the original request TLP is Posted, the completer discards this UR request, however, will return NO completion. It may send PCIe error message toward the RC. If the original request TLP is Non-Posted (MRd for example), the completer will return a Completion with UR status. I am here! What s the matter with you? Why do you send me a UR Completion? Requester Root Complex or Switch UR Completion MRd Request TLP ERR_x Message Completer End point TLP PCIe Error Message 41

42 Requester s The UR Error (receiving a UR Completion) is logged in the Requester s configure space: In the traditional PCI-compatible area: Freescale PCIe Config. Offset 0x06, Status Register [Received Master Abort] EP Mode Only Freescale PCIe Config. Offset 0x1E, Secondary Status Register [RMA] RC Mode Only. In Type 1 Header. In the optional Advanced Error Reporting (PCIe AER) area: Freescale PCIe Configure Offset 0x130, Root Error Status Register [NFEMR, EFNFR] RC Mode Only. Freescale PCIe Configure Offset 0x136, Error Source ID Register [Error_Source_ID] RC Mode Only. Contains the Source (Requester) ID information of the first ERR_NONFATAL messaged received/logged in the Root Error Status Register 42

43 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 43

44 What is Completion Timeout? The Requester sends a Non-Posted transaction to some Completer in the PCIe system fabric The Completer never returns the required completion The Requester keeps waiting and eventually its local Completion Timer expires, which results in a Completion Timeout event Typical Completion Timer recommended by base spec is 50 msec Completion Timeout error is logged PCI Express Error interrupt can be triggered if enabled No data is returned core can t finish the related load instruction. Very Bad! Condition for Completion Timeout to Occur The request TLP must be transmitted out to the link to kick off the Requester s Completion Timer 44

45 Completion Timeout is a PCIe Uncorrectable Non-Fatal Error Who is responsible for detecting and reporting CTO Error? Of course, it s the original Requester. I am responsible to report! You never send me the completion TLP! Requester Root Complex or Switch Completion MRd Request TLP End point Expected Cpl TLP 45

46 The Completion Timeout Error is logged in the following registers at Requester side: In Freescale proprietary memory-mapped register area: Freescale PCIe controller MM offset 0xE00, PEX Error Detect Register [PCT] In the optional Advanced Error Reporting (PCIe AER) area: Freescale PCIe Config. Offset 0x104, Uncorrectable Error Status Register [CTO] 46

47 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 47

48 MPIC Error Interrupt Summary Register 0 (EISR0) Offset 0x4_3900 FMAN2 FMAN1 BMAN QMAN PME SRIO PEX3 PEX2 PEX1 elbc DDR MC2 DDR MC1 CPC2 CPC1 Int RAM ECC PAMU CoreNet May only select 1 of 3 destinations mcp int cint IVPR + IVOR1 IVPR + IVOR4 IVPR + IVOR0 Machine Check ISR External Input ISR Critical Input ISR 48

49 Internal Interrupts MPIC_EISR0 : Error Interrupt Summary Register PEX3 Error PEX2 Error Set when active PEX1 Error Interrupts from all internal logic and IP blocks within SoC ORed Error Interrupt Internal Interrupt # 0 MPIC_IIVPR0 MPIC_IILR0 Select one interrupt output target mcp int cint Core 7 Send to the destination core MPIC_IIDR0 49 Core 0

50 Brief Overview of PCI Express Architecture and Protocol Introduction to PCI Express Errors Error Classification, Detection, Logging, Signaling PCI Express Uncorrectable Errors impacting system stability PCI Express Uncorrectable Error Case Study - UR UR Error Definition and Clarification UR Cause, Detection and Logging PCI Express Uncorrectable Error Case Study Completion Timeout CTO Error Definition and Clarification CTO Cause, Detection and Logging P4080 SoC Platform and PCIe Error Interrupt Generation Path PCI Express Outbound MRd/UR Completion Error Handling Summary Reference 50

51 5 4 e500mc Core 7 e500mc Core 0 1 Load Instruction Queue Internal Completion w/bad data 2 PEX3 RC MRd CoreNet P4080 Case Study Assumption: P4080 Core 0 issues a Load to downstream EP, which is connected underneath P4080 PEX3 RC. The downstream EP for some reason can t handle the MRd. It then returns a Completion TLP w/status as Unsupported Request. PCI Express RC terminates this MRd transaction locally and return the status marked as bad data. Meanwhile, it asserts the PEX3 error interrupt. P4080 Core 0 is configured to take the Synchronous Error Report Machine Check interrupt. P4080 Core 7 is configured as a dedicated core to handle the PEX3 interrupt. The cint is selected as the interrupt output target of the Internal Interrupt # 0, which is the ORed Error Interrupt 3 UR Cpl PCIe Endpoint 51

52 Core 0 issues a load targeting the downstream EP connected underneath PEX3 RC The Guarded Load instruction reaches the bottom of the completion queue. It gets converted to an Outbound MRd going through CoreNet OCN PEX3 RC PEX3 RC sends out this MRd on the link and reaches the downstream EP The downstream EP can t handle this MRd. It treats this received MRd as Unsupported Request - It returns a Completion TLP w/ur Status to PEX3 RC - It may also return an ERR_NONFATAL message, if enabled in EP s Device Control Reg [URR] As the completer of this MRd, the EP logs this error in its: - Device Status Reg. [URD, NFED] - Uncorrectable Error Status Reg [URE] - Header Log Registers PEX3 RC terminates this MRd transaction and returns an internal completion to the requester, core 0 - The result of this transaction is marked as with error and Bad Data PEX3 RC logs this error in its: - Secondary Status Reg [RMA], when receiving the UR completion - Root Error Status Reg [NFEMR, EFNER], if receiving an ERR_NONFATAL message 52

53 Core 0 receives the result of this load from PEX3 RC and CoreNet, notices the result has error with Bad Data associated - It marks the result of this load with an Synch. Error Report Exception Because the guarded load already reached the bottom of the Core 0 s completion queue, it s this load s turn to complete this instruction. But Core 0 can t complete the instruction normally with the result of error status and Bad Data Core 0 now takes a Synch. Error Report Machine Check Interrupt - The Save/Restore Register contains pointer to this load instruction, in other words, the MCSRR0 contains the address of this load instruction - The MCSR [LD] bit is set to reflect this situation Core 0 s instruction execution is paused and waits for further information to resolve this exception once brought into handler 53

54 Core 0 receives the software semaphore notice from Core 7 - It is able to associate the pending CoreNet Bad Data Synchronous Machine Check interrupt with Core 7 s interrupt source & cause information Software semaphore from Core 7 Core 0 can further confirm this with: - Its MCSRR0 contains the target address of this load instruction; - It can find out the LAW (Local Access Window) configuration; - With above information, Core 0 can confirm this load was sent to PEX3 RC Core 0 can now decide how to proceed further with recovery option Which is vendor & application dependent! 54

55 Since PEX3 RC has the following errors logged in its: - Secondary Status Reg [RMA], when receiving the UR completion - Root Error Status Reg [NFEMR, EFNER], if receiving an ERR_NONFATAL message PEX3 RC fires the interrupt through MPIC with cint, which is routed to Core 7 Core 7 is interrupted with the cint, and saves machine states Core 7 execution resumes at interrupt handler address of: - IVPR [32:47] IVOR0 [48:59] 0b

56 Core 7 cint handler checks MPIC CISR1 [0] s assertion to confirm the cause is from ORed Error Interrupt source mapped to Internal Interrupt 0 Core 7 cint handler checks EISR0 to further figure out the exact source of the ORed error interrupt - It finds EISR0 [13] is set the interrupt is triggered by PEX3 Core 7 cint handler polls all the PEX3 RC error registers to find out the detailed error information Core 7 cint handler now knows the interrupt from PEX3 RC is caused by receiving a completion with UR status Core7 cint handler uses software semaphore to notify each core and find out who is related to this PEX3 RC interrupt event 56

57 Freescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, ColdFire+, C-Ware, the Energy Efficient Solutions logo, Kinetis, mobilegt, PowerQUICC, Processor Expert, QorIQ, Qorivva, StarCore, Symphony and VortiQa are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. Airfast, BeeKit, BeeStack, CoreNet, Flexis, MagniV, MXC, Platform in a Package, QorIQ Qonverge, QUICC Engine, Ready Play, SafeAssure, the SafeAssure logo, SMAROS, TurboLink, Vybrid and Xtrinsic are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners Freescale Semiconductor, Inc..

58 PCI Express error management is a challenging task for system developers A good understanding of the PCI Express specification s definition on PCI Express error detection, logging, signaling as well as scope enables us to get a solid grasp of the subject. The session elaborates from the detail definition to detection, logging, signaling and handling of some common uncorrectable non-fatal errors like Unsupported Request and Completion Timeout that impact system stability. With the session material, system developers will get a good understanding on handling the common Unsupported Request and Completion Timeout uncorrectable non-fatal errors in a Freescale QorIQ product based PCI Express system. UR error during PCI Express bus enumeration phase can be safely ignored, if the configuration cycle is initiated by using the Config_Addr/Config_Data mechanism! Facebook.com/Freescale Tag yourself in photos and upload your own! Tweeting? Please use hashtag #FTF2012 Session materials will be Look for announcements in the FTF Group on LinkedIn or follow Freescale on Twitter 58

59 Freescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, ColdFire+, C-Ware, the Energy Efficient Solutions logo, Kinetis, mobilegt, PowerQUICC, Processor Expert, QorIQ, Qorivva, StarCore, Symphony and VortiQa are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. Airfast, BeeKit, BeeStack, CoreNet, Flexis, MagniV, MXC, Platform in a Package, QorIQ Qonverge, QUICC Engine, Ready Play, SafeAssure, the SafeAssure logo, SMAROS, TurboLink, Vybrid and Xtrinsic are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners Freescale Semiconductor, Inc..

60 Books: PCI System Architecture, Fourth Edition, Tom Shanley, Don Anderson, MindShare, Inc., 2002 PCI Express System Architecture, Ravi Budruk, Don Anderson, Tom Shanley, MindShare, Inc., 2006 Freescale App Notes: AN4311, SerDes Reference Clock Interfacing and HSSI Measurements Recommendations PCI-SIG Specifications: PCI Local Bus Specification, Revision 2.3, March 29, 2002 PCI Bus Power Management Interface Specification, Revision 1.2, March 3, 2004 PCI Express Base Specification, Revision 1.0a, April 15, 2003 PCI Express Base Specification, Revision 1.1, March 28, 2005 PCI Express Base Specification, Revision 2.0, December 20, 2006 PCI Express Card Electromechanical Specification, Revision 1.1, March 28, 2005 PCI Express Card Electromechanical Specification, Revision 2.0, April 11,

61 White papers: Agilent Soft Touch 2.0 Midbus Probe User Guide RIAL&ckey= &lc=eng&cc=US&nfr= &pselect=SR.GENERAL 61

62