The Evolution of the ARM Architecture Towards Big Data and the Data-Centre

Transcription

1 The Evolution of the ARM Architecture Towards Big Data and the Data-Centre 8th Workshop on Virtualization in High-Performance Cloud Computing (VHPC'13) held in conjunction with SC 13, Denver, Colorado John Goodacre Director, Technology and Systems ARM Processor Division ARM Cambridge 1

2 Abstract In recent years the rapid evolution of the ARM Architecture has lead to a potential inflection point where the architectural and compute capabilities the ARM partners deliver into the low cost, low power, consumer focused products has direct applicability to the requirement of compute in the data centre and the processing of big data. This talk will review these capabilities and the discuss the potential impact to the data centre. Finally, we'll take a glance into a project that aims to use a holistic approach to compute, ranging from 3D fabrication of an application specific, scalable compute element with virtualized access to system level resources so to balance to cost and requirement between compute, IO and memory. 2

3 ARM: Technology Supplier Advanced digital products are incorporating more and more ARM technology from processor and multimedia IP to software Processor IP Design of the brain of the chip Physical IP Software development tools Design of the building blocks of the chip 3

4 ARM IP Business Model Our customers are primarily the Semiconductor partners We sell mostly to their OEM We work with our customer to support the ecosystem 4

5 Various IP Licensing Models Arch. Subscription Implementation Term Single Use Design Foundry Program Higher license value Higher IP value Greater commercial / technical interaction Increasing internal investment by partner Increasing ARM / Partner business knowledge Greater flexibility Greater commercial / technical interaction Platform licenses End User license Free Library Program DesignStart Academic / Research Processors Physical IP 5

6 ARM Architecture Evolution 6

7 Performance, Functionality ARM Applications Processors Cortex-A Series Low-Power Leadership Cortex-A15 >2GHz+ in 28HPM Virtualization 1TB physical addressing big.little with Cortex-A7 Cortex-A57 ARMv8 64-bit Same power, 3x the performance of todays superphones 40LP Cortex-A8 Cortex-A5 Cortex-A9 Shipping in mobile since nd generation 1-4X SMP 4x1750DMIPS@700MHz+ in Cortex-A53 ARMv8 64-bit Same performance, ¼ the power of todays superphones Cortex-A7 1/5 the power of Cortex-A15 Architectural alignment with Cortex-A Future 7

8 ARM VIRTUALIZATION EXTENSIONS 8

9 Capabilities introduced in Cortex-A15 Full backwards compatibility with the Cortex-A9 Supporting the ARMv7 Architecture Addition of Virtualization Extension (VE) Run multiple OS binary instances simultaneously Isolates multiple work environments and data Supporting Large Physical Addressing Extensions (LPAE) Ability to use up to 1TB of physical memory With AMBA 4 System Coherency (AMBA-ACE) Other cached devices can be coherent with processor Many core multiprocessor scalability Basis of concurrent big.little Processing 9

10 Large Physical Addressing 40-bit physical addressing Virtual memory (apps and OS) still has 32bit address space However support more memory resident applications Offering up to 1 TB of physical address space Supporting multiple 32bit OS images through virtualization What does this mean for ARM based systems? More applications resident at the same time Multiple resident virtualized operating systems Common global physical address in many-core 10

11 Virtualization Extensions: The Basics New Non-secure level of privilege to hold Hypervisor Hyp mode New mechanisms avoid the need Hypervisor intervention for: Guest OS Interrupt masking bits Guest OS page table management Guest OS Device Drivers due to Hypervisor memory relocation Guest OS communication with the interrupt controller (GIC) New traps into Hyp mode for: ID register accesses and idling (WFI/WFE) Miscellaneous difficult System Control Register cases New mechanisms to improve: Guest OS Load/Store emulation by the Hypervisor Emulation of trapped instructions through syndromes 11

12 How does ARM do Virtualization Extensions to the v7-a Architecture, available on the Cortex -A15, Cortex-A7 CPUs and future processors Second stage of address translation (separate page tables) Functionality for virtualizing interrupts inside the Interrupt Controller Functionality for virtualizing all CPU features, including CP15 Option of a MMU within the system to help virtualize IO Hypervisor runs in new Hyp exception mode / privilege HVC (Hypervisor Call) instruction to enter Hyp mode Uses previously unused entry (0X14 offset) in vector table for hypervisor traps Hyp mode exception link register, SPSR, stack pointer Hypervisor Control Register (HCR) marks virtualized resources Hypervisor Syndrome Register (HSR) for Hyp mode entry reason 12

13 Exceptions Exception Returns Virtualization: Third Privilege Guest OS same kernel/user privilege structure HYP mode higher privilege than OS kernel level VMM controls wide range of OS accesses Hardware maintains TZ security (4 th privilege) Non-secure State User Mode (Non-privileged) 1 Secure State App1 App2 App1 App2 Secure Apps Guest Operating System1 Guest Operating System2 Supervisor Mode (Privileged) 2 Secure Operating System Virtual Machine Monitor / Hypervisor Hyp Mode (More Privileged) 3 TrustZone Secure Monitor (Highest Privilege) 13

14 Virtual address map of each application Physical Address Map Memory the Classic Resource Before virtualisation the OS owns the memory Allocates areas of memory to the different applications Virtual Memory commonly used in rich operating systems Translation s from translation table (owned by the OS) 14

15 Virtual Memory in Two Stages Stage 1 translation owned by each Guest OS Stage 2 translation owned by the VMM Hardware has 2-stage memory translation Tables from Guest OS translate VA to IPA Second set of tables from VMM translate IPA to PA Allows aborts to be routed to appropriate software layer Virtual address (VA) map of each App on each Guest OS 15 Intermediate Physical address map of each Guest OS (IPA) Physical Address (PA) Map

16 Classic Issue: Interrupts An Interrupt might need to be routed to one of: Current or different GuestOS Hypervisor OS/RTOS running in the secure TrustZone environment Basic model of the ARM virtualisation extensions: Physical interrupts are taken initially in the Hypervisor If the Interrupt should go to a GuestOS : Hypervisor maps a virtual interrupt for that GuestOS App1 App2 Virtual Interrupt App1 Guest OS 1 App2 App1 Guest OS 2 App2 Operating System Physical Interrupt Physical Interrupt VMM System without virtualisation System with virtualisation 16

17 Interrupt Virtualization Virtualisation Extensions provides : Registers to hold the Virtual Interrupt CPSR.{I,A,F} bits in the GuestOS only appling to that OS Physical Interrupts are not masked by the CPSR.{I.A.F} bits GuestOS changes to I,A,F no longer need to be trapped Mechanism to route all physical interrupts to Monitor Mode Already utilized in TrustZone technology based devices Virtual Interrupts are routed to the Non-secure IRQ/FIQ/Abort Guest OS manipulates a virtualized interrupt controller Actually available in the Cortex-A9 to aid paravirtualization support for interrupts 17

18 Virtual Interrupt Controller New Virtual GIC Interface has been Architected ISR of GuestOS interacts with the virtual controller Pending and Active interrupt lists for each GuestOS Interacts with the physical GIC in hardware Creates Virtual Interrupts only when priority indicates it is necessary GuestOS ISRs therefore do not need calls for: Determining interrupt to take [Read of the Interrupt Acknowledge] Marking the end of an interrupt [Sending EOI] Changing CPU Interrupt Priority Mask [Current Priority] 18

19 Virtual GIC (Global Interrupt Controller) GIC now has separate sets of internal registers: Physical registers and virtual registers Non-virtualized system and hypervisor access the physical registers Virtual machines access the virtual registers Guest OS functionality does not change when accessing the vgic Virtual registers are remapped by hypervisor so that the Guest OS thinks it is accessing the physical registers GIC registers and functionality are identical Hypervisor can set IRQs as virtual in the HCR Interrupts are configured to generate a Hypervisor trap Hypervisor can deliver an interrupt to a CPU running a virtual process using register lists of interrupts 19

20 Resource Ownership Software-only approaches Access to resources by GuestOS intercepted by the VMM VMM interprets the GuestOS s intent Provides its own mechanism to meet that intent Mechanism of interception varies Paravirtualisation adds a hypercall to the source code Binary translation adds a hypercall to the binary Exceptions in Hardware provide an trapping of operations Hypercalls can be more efficient More of the intent to be expressed in a single VMM entry Hardware assisted approaches: Provide further indirection to resources Accelerating trapped operations by syndrome information 20

21 Helping with Virtual Devices ARM I/O handling uses memory mapped devices Reads and Writes to the Device registers have specific side-effects Creating Virtual Devices requires emulation: Typically reads/writes to devices have to trap to the VMM VMM interprets the operation and performs emulation Perfect virtualization means all possible devices loads/stores emulated Fetching and interpreting emulated load/store is performance intensive Syndrome information on aborts available for some loads/stores Syndrome unpacks key information about the instruction Source/Destination register, Size of data transfer, Size of the instruction, SignExtension etc If syndrome not available, then fetching of the instruction for emulation still required 21

22 Devices and Memory Providing address translation for devices is important Allows unmodified device drivers in the GuestOS If the device can access memory, GuestOS will program it in IPA ARM virtualisation adds option for a System MMU Enables second stage memory translations in the system A System MMU could also provide stage 1 translations Allows devices to be programmed into guest s VA space System MMU natural fit for the processor ACP port ARM defining a common programming model Intent is for the system MMU to be hardware managed using Distributed Virtual Messages found in AMBA 4 ACE 22

23 System MMU 23

24 Virtualization and TrustZone TrustZone coexists alongside a VMM Normal World App1 (EPG) App2 App1 (Flash) App2 Guest Operating System1 Guest Operating System2 Virtual Machine Monitor (VMM) or Hypervisor Secure Monitor Secure Apps Secure RTOS Secure World HARDWARE (Memory, ARM CPU, I/O Devices) TrustZone offers a specialised type of Virtualisation Only 2 Worlds not extendable (except through paravirtualization) Although VMM can also span both worlds Fourth privilege level is provided by CPU s secure monitor mode Non-symmetrical - The two Worlds are not equal Secure world can access both worlds (33bit addressing) 24

25 Spanning Hypervisor Framework ARM Non-Secure Execution Environment User Exception Level (Application Layer) A P P API A P P A P P A P P API ARM Secure Execution Environment Secured Services Domain SoC SoC Management Domains (ST/TEI) Resource API Platform Resources NoC smmu Coherence Kernel/Supervisor Exception Level Open OS & Drivers Driver (OpenMP RT) Open OS (Linux) Resource Driver RTOS/uKernel Para-virtualization with platform service API s Virtualized TZ Call Secured hw timer (tick) HYP(ervisor) Exception Level Accelerators (eg DSP / GPU) Virtualized TZ Call Open Platform Hypervisor Full Virtualization (KVM) Heterogeneous TZ Monitor Exception Level TrustZone Monitor Enabling secure and mix-criticality systems through virtualization This project in ARM is in part funded by ICT-Virtical, a European project supported under the Seventh Framework Programme (7FP) for research and technological development Project website 25

26 Flexibility Through Design Same core technology thousands of application specific solutions Built by hundreds of ARM technology licensee partners 26

27 Bridge IO Options Targeted IO Advantage of a Targeted Solution Discrete Processor System On Chip Specific Memory Types ARM business model enables efficiency of system design as In addition to processor design Targeted Memory Types One-size can not fit all applications Compromise across all important metric, compute vs memory vs IO Energy overhead from abstractions (eg PCIe) between IO and CPU ARM model supports hundreds of companies building hundreds of application optimized System on Chip solutions The challenge is ROI cost of development vs. market opportunity 27

28 COTS-on-Silicon 40+ billion cores to date 100+ billion by Approx 10B additional core in

29 Scalability Concept: Compute Unit Coherent view into memory hierarchy of this compute unit Compute Unit Local Coherent Interconnect Path to local memory addressable by this unit Path to address locations not accessible by this unit The Compute Unit unifies a number of localized compute resources Potentially both general purpose and other local heterogeneous accelerators Provides coherent and symmetric access across local resources Enabling a SMP capable operating system and resource sharing Each Compute Unit is registered at a partition within a system s global address space (GAS), including units with heterogeneous capability Any unit can access any remote location in the GAS (including cached) DMA can transfer between (virtually address cached) memory partitions 29

30 Example: Heterogeneous Compute Unit Access to remote partitions AMBA Extensions Interface (Master) Cortex-A15 Cluster CPU 0 L2 Cache CPU 2 CPU 1 CPU 3 Cortex-A7 Cluster CPU 0 L2 Cache CPU 2 CPU 1 CPU 3 Mali T600 series GPU Shader Core 0 Shader Core 1 L2 Cache SMMU Shader Core 2 Shader Core 3 Debug & Trace NIC 400 CoreSight Resources Mgt System Power JTAG & Trace PMIC/ APB Bus Cache Coherent Interconnect (CCI-400) AMBA Extensions Interface (Master) Unified coherent virtual memory DMC-400 LPDDR2/DDR3 Controller OpenCL TBA - H.S.A. NIC 400 TrustZone enabled 2012 Compute Subsystem Shared pointers Shared data Shared dispatch On-Chip Memories (RAM, ROM) Base Peripheral DDR PHY or DDR Memory Local memory partition AMBA Extensions Interface (Slave) Requests from remote partitions 30

31 EUROSERVER Vision: Started Sept 13 Local I/O and Remote Routing Cortex-A15 Cluster Cortex-A7 Cluster CPU 0 CPU 2 CPU 0 CPU 2 L2 Cache L2 Cache CPU 1 CPU 3 CPU 1 CPU 3 Cache Coherent Interconnect (CCI-400) AMBA Extensions Interface (Master) Mali T600 series GPU Shader Core 0 Shader Core 1 L2 Cache Shader Core 2 Shader Core 3 SMMU AMBA Extensions Interface (Master) Debug & Trace CoreSight Resources Mgt System Power NIC 400 JTAG & Trace PMIC/ APB Bus 1) Start with a standard scalable compute unit die Compute Node Multi-Chip-Module Compute Chiplet Active Interposer (2D SoC + TSV) Bumps TrustZone enabled Compute Chiplet Package Substrate DMC Compute Subsystem LPDDR2/DDR3 Controller DDR PHY or DDR Memory AMBA Extensions Interface (Slave) On-Chip Memories (RAM, ROM) NIC 400 Base Peripheral 2) Connect a few together along with its local memory partition Memories 3) Put a few on a board With shared virtualized IO s Consumer supply 256xCPU 4) In future, integrate into a single MCM Balls Non-Volatile Memory Compute Node (64xCPU) Compute Node Compute Node Configurable Multi- Protocol I/O Interfaces Delivering a result 100x better than today Compute Node 31

32 EUROSERVER: System Hierarchy Component Hierarchy Per hierarchy Software view Design Choice Rack Shelves 8-16 System shmem MPI Base board Chassis shelf 2-4 Clustering (supporting virtually mapped shmem) Power / cooling density Thermal density and local storage requirement Mezzanine board IO Specialized node 2-8 processor boards Virtually mapped regions, hostos Balance of IO vs communication Processor Board Daughter board 2 nodes < 100ns latency Virtual mapped regions, guestos Physical sizing Packaged 2D TSV flipchip Compute Node 2-8 Units Virtual shared memory (ccnuma) Balance compute vs communication Micro-bumped Silicon chiplet Compute Unit 4-16 CPU + GPGPU Virtual memory Coherent SMP Balance compute again memory MCM in Package Local Memory 8-32GB per unit Unified memory Primary Memory per processor 32

33 Prototype Example Configuration Processor Board (local router + remote bridge) Remote routing Up to 4x 100Gb/s optical or 8x 40Gb/s Ethernet Processor Board (local router + remote bridge) Technology Dependent 4 16 Nodes / board 40Gb/s between nodes 2Tb/s switching Node Node Node Node CPU / node Unit Unit Unit Unit Unit Unit Unit Unit 4-16GB per unit 2 48 P0 P1 Flat Global Address Space 8x CPU per unit, 4x unit per node, 8xG3 PCIe/40Gb/s per node, 9x nodes per local router Latency from unit to unit via processor board ~ 0.5 us, ability to read/write L2 cache of remote unit 33

34 Use of Virtualization IO Multiple nodes share the physical cost of utility interfaces Multiple Access Mac used to share high-speed interfaces NIC masters directly into each guests local memory partition Processing IO Node is the VMM host operating as a ccnuma machine over multiple physical compute nodes Capable of 1:1 allocation between hardware and guest Memory Any guest can be allocated memory up to the entire system image Numa cost rules help with shared memory access latencies Communication Direct processor read/write semantic between system memory Shared memory MPI with DMA for long haul transfers 34

35 Summary ARM Architecture Has evolved into a full 64bit, secure, high-performance solution capable of hardware accelerated virtualization ARM Licensee Design both the processor core and system to target the requirements of specific applications and markets The flexibility opportunity of ARM based design for HPC hasn t really started We see integration of the high-speed IO around the ARM core We see scalable connectivity being integrated on the device The full opportunities of integrated system design has yet to be realized for enterprise and HPC requirement 35