SEU Mitigation Techniques for SRAM based FPGAs

Transcription

1 SEU Mitigation Techniques for SRAM based FPGAs Ken Chapman 30 th September 2015 TWEPP2015 Copyright Xilinx.

2 Some Aviation History DH.89 Dragon Rapide Introduced hp piston engine Wright Flyer Model A 1 35hp piston engine Page 2 Copyright Xilinx.

3 Over 17 Years of Rosetta and Beam Testing Copyright Xilinx.

4 Channel Xilinx SEU; Past, Present and Future UG116 Soft Event Rates at sea level New York (Now) 2015 (Now) Events per million bits per 10 9 hours. Configuration memory (CRAM) and Block Memory (BRAM) in user design. Things were definitely getting worse so this had to be addressed by design. Xilinx FPGAs now have a lower susceptibility than they have ever had. Source Gate Coming soon Tests indicate <5 FIT/Mb Advanced 16nm FinFET 5 reduction in sensitive area Design Rules 40+ Patents & Trade Secrets STI Drain Substrate STI FinFET Transistor Page 5 Copyright Xilinx.

5 Risk Assessment Whole Device Let s take a look at the XC7K325T which is a mid-range Kintex-7 device 326,000 logic cells (i.e. not small!) 1Mb 0.5Mb of available user flip-flops 75.1Mb of static configuration 16.5Mb of available user RAM Page 8 Copyright Xilinx.

6 Risk Assessment Resources Actually Used Every design is different so obviously better to work with actual values. But let s accept some typical figures for now 0.15Mb of used flip-flops 30Mb of used or Essential Bits 12Mb of used RAM 1-7Mb of Critical Bits Page 9 Copyright Xilinx.

7 Detecting and Correcting Errors Quickly 7-Series and UltraScale devices have dedicated error detection and correction circuits. - Primarily for SEU but acts as an ongoing comprehensive BIST during device lifetime. Automatic scan of all static configuration cells (continuously happening in the background) Device-level CRC Detection (verifies corrections) ECCERROR=1 4.6ms to 70.7ms depending size of device INIT_B=1 CRCERROR=0 Frame-level ECC detection and correction 3,232 bits in each 7-Series Frame Including 13-bit ECC (0.4% of bits!) Page 23 Copyright Xilinx.

8 Observe What Happens & What Actions Are Needed Aircraft have become extremely reliable so are the pilots now the weakest link? Must not overload and distract pilots but also need to keep them engaged. UltraScale+ will be ~50 less susceptible to SEU than Virtex-5 That s a big change. Do not implement mitigation schemes that make things less reliable! Soft Error Mitigation (SEM) exploits partial reconfiguration to facilitate controlled simulation of SEUs (not everyone has a particle accelerator you know ). 500 simulated SEU in XC7325T. Red dots = Disturbance observed ( Critical Bits ). Experiments = Knowledge (not guessing) Do you need mitigation? Where do you need mitigation? What type of mitigation is suitable? Will mitigation be more disruptive than doing nothing? Page 28 Copyright Xilinx.

9 Upsets in Data Flip-Flops 0.15Mb of used flip-flops 7-Series Flip-flops in Xilinx FPGAs are the lowest risk but if their contents are really that critical then you must include mitigation in your design. But can your design truly improve the reliability without making things worse? Mitigation schemes typically employ one or combinations of the following - Parity bits and calculations for detection. - ECC bits and calculations for detection and correction. - Dual Modular Redundancy (DMR) detects errors but not location. - Triple Modular Redundancy (TMR) detects and can isolate failing module. Key Observations - All schemes increase design size and therefore increase underlying upset rates. - Watch out for single and common points of failure: where is the weakest link? - Extreme care needed to avoid introducing complexity related failures (emulate). - In-circuit error detection and/or correction should reveal anything you really care about so these schemes should, by default, cover all critical bits. Page 42 Copyright Xilinx.

10 Upsets in Data RAM 7-Series Mainly driven by sheer quantity, the risk of data corruption in memory is significantly greater that that held in flip-flops. 12Mb of used RAM BRAM has multiple applications Data and sample storage (RAM) Data buffers (e.g. FIFO mode) Program memory (ROM) Coefficients and Tables (ROM) State Machines (ROM) 16nm UltraScale+ If you care about BRAM contents then you must provide mitigation in your design. In this case Xilinx have provided you with ECC protection - Every BRAM has this option; you decide where and when to enable it bit data with 8-bit ECC (Simple Dual Port and FIFO). - 4-way physical interleaving of memory cells. Key Observations - Zero un-correctable errors observed during any real SEU testing. - Very unlikely that any redundancy schemes would improve reliability. - ECC corrects a value as it is read so consider writing back ROM contents. Page 43 Copyright Xilinx.

11 Upsets in Configuration 16nm UltraScale+ 7-Series The Engine is becoming ever more reliable.. Only the critical bits really matter. 30Mb of used or Essential Bits 1-7Mb of Critical Bits Built-in error detection and correction means that the vast majority of upsets only result in a temporary soft error (e.g. 4.6ms to 70.7ms in 7-Series). Page 44 Copyright Xilinx.

12 Categorisation of Configuration Upsets All Configuration Cells 100% Detection Observed results for a variety of real applications ( normalised for device utilisation ) 60-80% Completely miss the design so have no effect. - All detections are reported. - Be careful NOT to overreact and lower reliability. - Essential Bit Mask can classify upset locations % Touch the design. - Typically less than 1 in 3 have any effect. - Be careful NOT to overreact and lower reliability. - Consider Prioritised Essential Bits Mask <10% will be observed to have any effect. - 1% to 5% is a more typical observation rate. <1% have an undesirable effect on operation. - Critical bits are those that really matter. Exploit error injection (SEU simulation/emulation) to learn and appreciate which circuits really matter. Page 45 Copyright Xilinx.

13 The PCIe-based readout system for the LHCb experiment K.Arnaud, J.P. Duval, J.P. Cachemiche, Cachemiche,P.-Y. F. Réthoré F. Hachon, M. Jevaud, R. Le Gac, Rethore Centre de Physique des Particules def.marseille Centre de Physique des Particules de Marseille TWEPP2015 1st October 2015 The PCIe-based readout system for the LHCb experiment 1

14 Initial architecture choice for LHCb Triggerless: All events fragments are routed toward a single CPU blade Front-ends Front End Cluster Intermediate crates GBT protocol Commercial protocol Network Computer blades Commercial protocol Front End Cluster Front End Cluster Front End Cluster Front End Cluster Front End Cluster ATCA based TWEPP2015 1st October 2015 Long life cycle The PCIe-based readout system for the LHCb experiment Short life cycle 5

15 New readout/trigger scheme Move back-end card into already existing CPU blades and software trigger TWEPP2015 1st October 2015 The PCIe-based readout system for the LHCb experiment 10

16 PCIE40 Synoptic view Up to 48 optical inputs/outputs Serial signal up to 10 Gbps Bidirectional serial links for the TFC output bandwidth ~100 Gbps though PCIe Gen3 x16 Generic board: Data Acquisition from the FE boards TFC supervisor TFC & ECS distribution to/from the FE boards TWEPP2015 1st October 2015 The PCIe-based readout system for the LHCb experiment 16

17 PCIE40 Challenging board Design of: 114 high speed serial signals up to 10 Gbps including PCI Express GEN3x16. Reduce form factor to be compatible with most CPU blades Developed with the largest ALTERA FPGA currently available (Arria 10) Use of engineering sample (ES1 & ES2) Production chip still not released Power dissipation up to 160W 10 different voltages. Precise power sequencing required. Transport high current (50A/0.9V) to the FPGA core. Risk of PCB delamination by thermal effect TWEPP2015 1st October 2015 The PCIe-based readout system for the LHCb experiment 17

18 Simulation of current propagation on the ground shape of the power supply Mezzanine board CADENCE SIGRITY suite Previous routing Final routing Very high density Of current (150A/mm²) Solution: two separated ground shape 50A/mm² TWEPP2015 1st October A/mm² The PCIe-based readout system for the LHCb experiment 18

19 PCIE40 Prototypes 2 prototypes: equipped with Arria10 ES1 & ES2 TWEPP2015 1st October 2015 The PCIe-based readout system for the LHCb experiment 19

20 Common Readout Unit (CRU) A New Readout Architecture for ALICE Experiment Jubin Mitra VECC, Kolkata, India For the ALICE Collaboration

21 ALICE LS1 ( ) After LS2 * ( ) Future Present (RUN3/RUN4 (RUN2) UPGRADE YEAR 2021) 1 nb -1 (PbPb) Collisions and Collection >10 nb -1 (PbPb) 6 pb -1 (pp) 50 nb -1 (ppb) cm -2 s -1 At Peak Luminosities 6 x cm -2 s -1 8 khz (PbPb) Corresponding to Collision Rate Of 500 Hz (PbPb) Maximum Readout Rate Hardware triggers Event multiplicity Calorimeter energy Track p T To summarize: Why Do We Need CRU? Trigger Mechanism 50 khz (PbPb) 200 khz (pp and ppb) >200 khz (PbPb) 1 MHz (pp) A minimum bias event (Non- Upgraded detector) A self-triggered Continuous fashion (upgraded detector) 6

22 Example CRU connection with TPC TPC Front-end Card The CRU re-orders the data samples according to their position in the pad row allowing a more efficient cluster search. Control and Monitoring DCS Data Storage Storage Network Physics & Monitoring Data GBT CRU Physics Data PCIe x16 FLP Farm Network EPN GBT Trigger, Control and Configuration FPGA 1 st stage of data reduction using cluster or tracklet finder 2 nd level of data reduction. Reconstruct the tracks and associate them to their primary vertex Pic courtesy ALICE TPC TDR 11

23 CRU Form Factor Evaluation Features Prototype version 1 Prototype version 2 DDL3 10 Gigabit Ethernet PCIe Gen 3 Trigger and Busy line Distribution ATCA Processor Advantage Modularity Directly connected to the O 2 Disadvantage Not Enough memory for data clustering not possible Compatibility of the board depends on future PCs PCIe form factor 15

24 CRU CANDIDATE BOARDS EVALUATED AMC 40 Developed By: CPPM Marseille 16

25 CRU internal block connections CTP FLP Server LTU FEE FEE GBT TTS User (detector specific) Logic CRU Ctrl. PCIe CRU Detector Data Frames Detector Control Frames RAM Common CRU FW/SW Components provided by the CRU team Extensible by the Detector teams Developed by the Detector and DCS teams Developed by the Detector and O2 teams CPU DCS FLP DCS Software FLP DAQ Software NIC/IB NIC to EPN Servers to DCS Servers CTP Central Trigger Processor DCS Detector Control System EPN Event Processing Node FLP First Level Processor GBT Giga-Bit Transceiver LTU Local Trigger Unit 19

26 DDL3: PCIe Link Testing and DMA Performance Measurement Avalon- Memory Master Slave Interface PIPE Interface Serial Data Transmission Application Layer (User Logic) Bridge and DMA Engine PCIe Hard IP Block PHY IP Core for PCIe (PCS/PMA) PCIe Gen2 x8 Performance Measurement: Signalling Rate = 5 Gbps per lane x 8 = 40 Gbps Useful Data Throughput = 32 Gbps As, Gen2 use 8B/10B encoding which introduces a 20% overhead # In Arria 10 Engineering Sample1 we have PCIe Gen2, but Engineering Sample2 and production chips run in GEN3 Using PLX8747 PCIe Switch for multiplexing 2 banks of x8 lanes 23