ARM SERVER STANDARDIZATION (and a general update on some happenings at Red Hat) Jon Masters, Chief ARM Architect, Red Hat
6+ YEARS OF ARM AT RED HAT Red Hat ARM Team formed in March 2011 Bootstrapped ARMv8 architecture for RHEL (and Fedora, and CentOS, etc.) Co-created ARM server industry standards along with Linaro Enterprise Group (LEG) Collaborated with every ARM server chip semiconductor during design and verification Shipped Red Hat Enterprise Linux Server for ARM (Development Preview) Watch this space
Source: By Dr-text - Datacenter, CC BY-SA 3.0 https://commons.wikimedia.org/wiki/file:datacenter-mivitec-munich.jpg
Source: By Alan Chia - Lego Color Bricks, CC BY-SA 2.0 https://commons.wikimedia.org/w/index.php?curid=6068229
Source: By Arp - https://commons.wikimedia.org/wiki/file:maxscan_oe509_collage.jpg
Source: http://www.thecompaqstory.com ( Open, by Rod Canion)
SERVER BASE SYSTEM ARCHITECTURE
SYSTEM ON CHIP CPU Core Cluster Interconnect
SYSTEM ON CHIP Memory Controller
SYSTEM ON CHIP Serdes (PCIe, Fabric, SATA )
SYSTEM ON CHIP Workload Customized IP Compression Engine
SILICON COMPLIANCE Adherence to the ARM Server Base System Architecture (SBSA) Core compliance is (well) handled by ARM s architecture validation Uncore components sometimes need work (PCIe IP example) PCIe quirks are common across arches but need upstreaming Always design with backward compatibility in mind
SERVER BASE BOOT REQUIREMENTS
FIRMWARE COMPLIANCE Real servers are built using standard platforms. In the ARM community, we defined the ARM Server Base Boot Requirements (SBBR). We use UEFI and ACPI because they offer the necessary pieces for platform abstraction required in general purpose OSes. There is one driver for the Real Time Clock (RTC) in RHEL We have defined interfaces for provisioning ARMv8 servers (not your ARMv8 server), and for upgrading firmware easily over time
VALIDATION AND COMPLIANCE
OUT OF BOX EXPERIENCE Having a high performance server core isn t the first step to world domination Things need to just work in the way that users already expect them to Unbox, plug into rack, provision via the network using standard tooling Practice good hygiene by giving users a boring experience Use the Linaro Enterprise Reference Platform, other examples as a base
WHAT NOT TO DO Don t ship some hack with your custom hacked up distro and kernel of the day Don t ship a special OS for ARMv8 server X that has no upgrade path By all means pre-install an example, but make sure it isn t a hack (if it says it is RHEL it needs to actually be RHEL) If you are unable to avoid a hack, have a good plan and work it through upstream
Source: By Rob Young, Bluff Sign Post, CC BY-SA 2.0 https://commons.wikimedia.org/wiki/file:bluff_sign_post,_new_zealand_(870336830).jpg
WORKLOAD OFFLOAD ACCELERATORS Traditional compute tasks replaced with CPU managed data movement to dedicated or reconfigurable accelerators ARMv8 servers integrate many workload accelerators including network packet processors, compression engines, and security/crypto processors Use otherwise dark silicon more effectively (e.g. phone all day battery) Source: Cavium Networks (ThunderX)
RECONFIGURABLE COMPUTING FPGAs allow for custom logic to be tailored to a specific VM, attached onchip/package, interconnect (e.g. CCIX) Logically coherent or non-coherent with regard to host system memory Coherent is easier to program (e.g. CUDA) but requires cache snoop hw Non-coherent device memory access using OS DMA primitives Unified Memory (CUDA) (transparent migration), memory hierarchy (OpenCL)
RECONFIGURABLE COMPUTING Red Hat Trilby is a simple Red Hat PCIe FPGA platform developed for research Implements a trivial search algorithm (not for high performance) Uses hand-written Verilog, a custom DMA engine, and a custom drivers Interfacing to Apache Spark is possible through a native (JNI) library etc. Red Hat Trilby PCIe-attached FPGA
ARM SERVER STANDARDIZATION (and a general update on some happenings at Red Hat) Jon Masters, Chief ARM Architect, Red Hat
6+ YEARS OF ARM AT RED HAT Red Hat ARM Team formed in March 2011 Bootstrapped ARMv8 architecture for RHEL (and Fedora, and CentOS, etc.) Co-created ARM server industry standards along with Linaro Enterprise Group (LEG) Collaborated with every ARM server chip semiconductor during design and verification Shipped Red Hat Enterprise Linux Server for ARM (Development Preview) Watch this space
Source: By Dr-text - Datacenter, CC BY-SA 3.0 https://commons.wikimedia.org/wiki/file:datacenter-mivitec-munich.jpg
Source: By Alan Chia - Lego Color Bricks, CC BY-SA 2.0 https://commons.wikimedia.org/w/index.php?curid=6068229
Source: By Arp - https://commons.wikimedia.org/wiki/file:maxscan_oe509_collage.jpg
Source: http://www.thecompaqstory.com ( Open, by Rod Canion)
SERVER BASE SYSTEM ARCHITECTURE
SYSTEM ON CHIP CPU Core Cluster Interconnect
SYSTEM ON CHIP Memory Controller
SYSTEM ON CHIP Serdes (PCIe, Fabric, SATA )
SYSTEM ON CHIP Workload Customized IP Compression Engine
SILICON COMPLIANCE Adherence to the ARM Server Base System Architecture (SBSA) Core compliance is (well) handled by ARM s architecture validation Uncore components sometimes need work (PCIe IP example) PCIe quirks are common across arches but need upstreaming Always design with backward compatibility in mind
SERVER BASE BOOT REQUIREMENTS
FIRMWARE COMPLIANCE Real servers are built using standard platforms. In the ARM community, we defined the ARM Server Base Boot Requirements (SBBR). We use UEFI and ACPI because they offer the necessary pieces for platform abstraction required in general purpose OSes. There is one driver for the Real Time Clock (RTC) in RHEL We have defined interfaces for provisioning ARMv8 servers (not your ARMv8 server), and for upgrading firmware easily over time
VALIDATION AND COMPLIANCE
OUT OF BOX EXPERIENCE Having a high performance server core isn t the first step to world domination Things need to just work in the way that users already expect them to Unbox, plug into rack, provision via the network using standard tooling Practice good hygiene by giving users a boring experience Use the Linaro Enterprise Reference Platform, other examples as a base
WHAT NOT TO DO Don t ship some hack with your custom hacked up distro and kernel of the day Don t ship a special OS for ARMv8 server X that has no upgrade path By all means pre-install an example, but make sure it isn t a hack (if it says it is RHEL it needs to actually be RHEL) If you are unable to avoid a hack, have a good plan and work it through upstream
Source: By Rob Young, Bluff Sign Post, CC BY-SA 2.0 https://commons.wikimedia.org/wiki/file:bluff_sign_post,_new_zealand_(870336830).jpg
WORKLOAD OFFLOAD ACCELERATORS Traditional compute tasks replaced with CPU managed data movement to dedicated or reconfigurable accelerators ARMv8 servers integrate many workload accelerators including network packet processors, compression engines, and security/crypto processors Use otherwise dark silicon more effectively (e.g. phone all day battery) Source: Cavium Networks (ThunderX)
RECONFIGURABLE COMPUTING FPGAs allow for custom logic to be tailored to a specific VM, attached onchip/package, interconnect (e.g. CCIX) Logically coherent or non-coherent with regard to host system memory Coherent is easier to program (e.g. CUDA) but requires cache snoop hw Non-coherent device memory access using OS DMA primitives Unified Memory (CUDA) (transparent migration), memory hierarchy (OpenCL)
RECONFIGURABLE COMPUTING Red Hat Trilby is a simple Red Hat PCIe FPGA platform developed for research Implements a trivial search algorithm (not for high performance) Uses hand-written Verilog, a custom DMA engine, and a custom drivers Interfacing to Apache Spark is possible through a native (JNI) library etc. Red Hat Trilby PCIe-attached FPGA