Reliability, Availability, and Serviceability (RAS) on AArch64. Fu Wei (Linaro LEG) Supreeth Venkatesh (ARM)

Transcription

1 Reliability, Availability, and Serviceability (RAS) on AArch64 Fu Wei (Linaro LEG) Supreeth Venkatesh (ARM)

2 AGENDA 1. Brief introduction of RAS 2. RAS on AArch64 3. Definition, Importance, History Overview Hardware support RAS Extension Software Architecture ARM-Trusted-Firmware, UEFI, APEI tables SDEI Prototype Solution for Firmware First Error Handling MM Secure Partition, Secure Partition Manager Uncorrected error -- HEST & MM Demo time Status and Future Plans

3 Brief introduction of RAS What is RAS? Why do we need RAS? History of RAS

4 What is RAS? -- Definition Reliability Continuity, Computation needs be correct and reliable. Availability Readiness, System needs to remain available as long as possible. Serviceability Ability to undergo modifications and repairs,system should provide information to administrator to aid in system servicing. The RAS architecture primarily cares about ERRORs produced from HARDWARE.

5 Why do we need RAS? -- Importance Impacts Continuity, Computation needs be correct and reliable. So we have to maintain system very well, and Operating Expense (OPEX) for maintenance is inevitable. Inevitability Although faults are rare, enterprise systems can be very large. So failures are inevitable. OPEX for maintenance is reduced by 1. replacing only failed parts 2. scheduled maintenance (is cheaper than unscheduled service outages)

6 Why do we need RAS? -- Importance Inevitability <DRAM Errors in the Wild: A Large-Scale Field Study> by Bianca Schroeder, Eduardo Pinheiro, Wolf-Dietrich Weber Benefit from ECC in DIMM Important Conclusion: The incidence of memory errors and the range of error rates across different DIMMs to be much higher than previously reported. Memory errors are strongly correlated. The incidence of CEs increases with age, while the incidence of UEs decreases with age (due to re-placements). Error rates are unlikely to be dominated by soft errors. Single-bit error --> CE Avoid (multi-bit errors)ues from beginning (Single-bit error, CEs) The statistical data of CEs/UEs could be a reference for maintenance to reduce the cost of unscheduled service outage.

7 Server without RAS How to avoid "Inevitability"? To Be Successful in Business, You Need a Little Luck. /* _ooooo_ o o 88". "88 ( -_- ) O\ = /O /`---'\.' \\ // `. / \\ : // \ / _ -:- - \ \\\ - /// \_ ''\---/'' \.-\ `-` /-. / `..' /--.--\ `..."" '< `. \_< >_/.' >'"". : `- \`.;`\ _ /`;.`/ - ` : \ \ `-. \ \ / /.-` / / ======`-. `-. \ /.-`.-'===== = `=---=' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^ Buddha blessed No Error Forever */ --Richard Branson Emperor Yongzheng Defeat Ba A Ge (Bug)

8 History ECC in memory controllers and I/O RAMs Machine Check Architecture (MCA) A mechanism in which the CPU reports hardware errors to the OS model-specific registers (MSRs) set up machine checking record detected errors the info they contain is CPU specific Machine Check Exception (MCE) signals the detection of an uncorrected machine-check error handler collect information about error from MSRs Utility: mcelog Linux kernel EDAC (Error Detection and Correction) designed to report and possibly act on hardware errors inspect the hardware directly (system-specific handling and decoding.) only support memory controller and PCI/AGP errors Firmware (first) FF APEI UEFI PCI-E: Advanced Error Reporting (AER)

9 RAS on AArch64 Overview of Hardware & Software Prototype Solution for Firmware First Error Handling

10 Hardware support for RAS CPU ARMv8-A architecture (a mandatory extension to ARMv8.2) EL2, EL3, or both Virtualization extension or Security extensions or both GICv3 Interrupt routing modes Private and shared interrupts (PPI/SPI) Ability to set an interrupt pending event signaling and delegation Interrupt groups/priority RAS Extension ESB (Error Synchronization Barrier) instructions RAS Extension registers Corrupted data poisoning

11 RAS Extension ESB instruction ESB (Error Synchronization Barrier) can be used to isolate Unrecoverable errors. Software can determine that: The error was reported as Unrecoverable. The preferred return address of the SEI is an ESB instruction. The software between that ESB and the previous ESB can be isolated. ESB might update DISR_EL1 / DISR (Deferred Interrupt Status Register) and VDISR_EL2 / VDISR (Virtual Deferred Interrupt Status Register) RAS Extension registers: Feature Register/Component ID Register Error Record Register Feature Control Record Primary Syndrome Record Address Register Record Miscellaneous Registers Hypervisor Configuration Register Virtual SError Exception Syndrome Register Secure Configuration Register Or Interrupt Register for Fault-Handling and Recovery Device Affinity and Architecture Register

12 RAS Extension -- gather HW error info for FW ESB instruction Help to locate Error RAS Extension registers Provide the error info to FW Control the Interrupt by FW ARMv8-A RAS extensions standardize the interface between HW and FW

13 Software Architecture Firmware First error handling requires standard interfaces between multiple SW components.

14 Software Architecture

15 Firmware ARM Trusted Firmware Reference EL3 Runtime (BL31) Trusted boot firmware Standard power control (PSCI) Optional Trusted OS integration Optional Compatible with other firmware (like EDK2) Applicable to all segments Open Source at GitHub with BSD-3-clause license UEFI Unified Extensible Firmware Interface. Firmware interface between the platform and the operating system. Predominate interfaces are in the boot services (BS) or pre-os. Few runtime (RT) services. On AArch64, it (tianocore EDK2) works with ARM TF as BL33 in EL2

16 APEI (ACPI Platform Error Interfaces) For last crash BERT HEST For runtime APEI For Storage ERST Provides a standard way to convey error info from Firmware to OS EINJ For Testing

17 APEI tables HEST (Hardware Error Source Table) Key info: HOW to get trigger WHERE are the error records HOW to release records mem For ARM64 : GHES v2 HOW to get trigger: Notification Structure WHERE are the error records: Error Status Address For IA-32 : MCE/CMC/NMI For PCI: AER Root Port/Endpoint/Br idge For generic hardware: GHES (Generic Hardware Error Source) V1/V2 (GAS : Generic Address Structure) HOW to release records mem: Read Ack Register

18 APEI tables BERT: Boot Error Record Table Record fatal errors, then report it in the second boot CPER (in the Appendix of UEFI spec) Common Platform Error Record, with this help, OS can get all kinds of error we could think of.

19 APEI tables -- ERST & EINJ ERST: Error Record Serialization Table Operation abstract, provides details necessary to communicate with on-board persistent storage for error recording EINJ: Error Injection Table Operation abstract, provides a generic interface which OSPM can inject hardware errors to the platform without requiring platform specific software.

20 SDEI usage in RAS Software Delegated Exception Interface An interface between FW & OS, for registering, notifying and servicing system events using SMC/HVC. SDEI Specification (ARM DEN0054A)

21 Prototype Solution for Error Handling MM Secure Partition, Secure Partition Manager Uncorrected error -- HEST & MM

22 What Are We Doing Define standard interfaces to enable FF handling of AP RAS errors Demonstrate use of RAS extensions Demonstrate interfaces with reference software and platforms uncorrected DIMM & CPU errors

23 MM Secure Partition MM Secure Partition implements management functions, runs in S-EL0 to achieve isolation from S-EL1 & EL3 Leverages existing firmware code based on EDK2: Standalone MM Partition communicates with ARM TF through a standard interface: MM_COMMUNICATE SMC Partition is managed by ARM TF ARM TF BL31 stage owns EL3 and S-EL1 Secure partition resources are described in BL31 platform port Minimise code in EL3 and delegate RAS error handling

24 Secure Partition Manager (SPM) Secure Partition Manager in BL31 exports standard ABI to Initialize the partition Delegate SMC requests to the partition

25 Uncorrected Error -- HEST & MM

26 Uncorrected Error -- HEST & MM System boot: BootROM-->BL2-->BL3x a. BL31 initializes SPM (includes MM dispatcher) and SDEI dispatcher. b. UEFI (BL33), DXE, UEFI Platform Driver: i. query SPI (Secure Partition Image, BL32) for error source info ii. SPI return error source info back to UEFI iii. UEFI map in and mark error record region as Runtime Services Data Region iv. Update/add error source info in HEST OS starts running: HEST driver scan HEST table and register error handlers by SDEI UE occurred, the event will be routed to EL3 (SPM) SPM routes the event to RAS error handler in S-EL0 (MM Foundation) MM Foundation creates the CPER blobs by the info from RAS Extension SPM notifies SDEI to call the corresponding OS registered handler OS gets the CPER blobs by Error Status Address block, process the error, try to recovery. report the error event by RAS event rasdaemon log error info from RAS event to recorder

27 Demo Time Prototype RAS solution on FVP arm-trusted-firmware (bl1, bl2, bl31) tianocore edk2 (bl32, bl33) Linux kernel, Shell command

28 Status and Future Plans Current development status Ongoing development TODO list for Reference Solution

29 Current development status Hardware ARM engineers are working FVP, LEG team is developing on QEMU RAS spec has released (ARM DDI 0587A) Firmware SDEI SDEI Specification released (ARM DEN0054A) SDEI added as hardware error notification type in ACPI 6.2 Linux SDEI client implementation v3 patchset has been posted on kvmarm and devicetree mailing list. ACPICA support for SDEI up-streamed SDEI DT bindings acked ARM TF support posted to github and includes SDEI Dispatcher Framework for managing interrupts handled in EL3 OS (Linux): APEI on ARM64 can be enabled in kernel. Memory failure support merged

30 Ongoing development ARM TF Simplify error interrupt handling for platform ports Framework for handling External aborts (EA) in design RAS Extensions support in design ESB RAS Error Record driver EDK2 Driver for creating APEI HEST under development Library for creating APEI CPERs under development Prototyping use of Standalone MM partition to create error records on QEMU OS(Linux) KVM changes for virtualizing SDEI under development

31 TODO list Hardware Test on a real hardware (ARMv8.2, including RAS extension) Firmware ARM-TF Support for Double fault handling Support for v8.4 RAS Extensions EDK2: Support for BERT ERST and EINJ implementation

32 Acknowledgments ENGINEERS AND DEVICES WORKING TOGETHER Achin Gupta (ARM) John Feeney (Red Hat) Leif Lindholm (ARM) Supreeth Venkatesh (ARM)

33 Thank You #SFO17 BUD17 keynotes and videos on: connect.linaro.org For further information: