Architetture di calcolo e di gestione dati a alte prestazioni in HEP IFAE 2006, Pavia

Transcription

1 Architetture di calcolo e di gestione dati a alte prestazioni in HEP IFAE 2006, Pavia Marco Briscolini Deep Computing Sales Marco_briscolini@it.ibm.com

2 IBM Pathways to Deep Computing Single Integrated Systems Single systems image SMP's Clusters of dense systems Grids Access to remote resources available on demand Computational and data-intensive Autonomic Computing Self-managing and self-healing systems React to conditions and redistribute workload On Demand Computing Delivery of standardized processes, applications and infrastructure over the network Utility-like model -- quick access to incremental capacity

3 % CGR 1000 IBM Blue Gene Human brain ops TeraFlops IBM Deep Blue 1 ASCI Lizard brain ops Mouse brain ops

4 Pharmacogenomics Proteins Metabolic Pathways Human Genome SNPs Petabytes Combinatorial Chemistry HTS Computational Biology MIPs ESTs Moore's Law

5 Intel Xeon (including EM64T) is the processor in 51% of the systems but IBM POWER architectures account for 15.4% of TOP500 systems. Itanium2; 79; 16% Source: TOP500 Processor Technology - June 05 PA RISC; 36; 7% PPC 440; 16; 3% POWER5; 8; 2% PowerPC; 9; 2% PowerPC includes 3 Apple Xserve systems Opteron; 25; 5% POWER3; 7; 1% Alpha; 5; 1% POWER4; 37; 7% POWER4 includes 2 Hitachi SR11000 systems MIPS, 1, 0% EM64T; 76; 15% SPARC, 5, 1% Cray ; 9; 2% NEC; 7; 1% Other; 4; 1% Xeon; 175; 36% POWER Architecture Combined is is 15.4% 15.4%

6 IBM has 6 of TOP10 Cray Others SGI Fujitsu NEC Dell Apple HP IBM IBM has doubled the results of the #1 spot with the DOE/LLNL BlueGene/L machine and together with other BlueGene/L Clusters and JS20 Cluster at Barcelona, IBM has 6 TOP10 entries, the only vendor with more than one. C ount TOP10 Supercomputers # Ven-dor IBM IBM SGI NEC IBM IBM Calif Digital IBM IBM Rmax TFlops Installation DOE/NSSA/LLNL (32 racks BlueGene/L) BlueGene at Watson (20 racks BlueGene/L) NASA/Columbia (Itanium2) Japan Earth Simulator Barcelona Super-computer (JS20) ASTRON Netherlands (6 racks BlueGene/L) LLNL Thunder (Itanium2) EPFL Switzerland (4 racks BlueGene/L) AIST - Japan (4 racks BlueGene/L) Source: 0 Jun04 Nov04 Jun05 10 Cray Sandia Natl Lab (Red Storm, Cray XT3)

7 IBM Systems Industry Leadership & Choice Clusters / Virtualization Large SMP Scale Up / SMP Computing p595 High BW SSI LPAR RAS High Performance Switch Linux Cluster (1350) AIX Clusters (1600) p575 High density POWER5 based e326 IntelliStation 1U Opteron-based POWER-based Intel-based Opteron-based High Density Rack Mount BladeCenterTM x336 1U/2p density Intel-based Scale Out / Distributed Computing Denser form factors Rapid deployment Flexible architectures Switch integration LINUX

8 Dual Core As frequency increase is limited due to power limitation Dual core is a way to : 2 x Peak Performance per chip (and per cycle) But at the expense of frequency (around 20% down) Other features also increase Flop/cycle FMA for POWER (and IA64) = 4 Flops/cycle/core VMX for PowerPC = 8 Flops/cycle/core (32bit only) IBM implements all these technologies: POWER4 introduced dual core in 2001 JS20 introduced dual heterogeneous core in 2003 JS21 will have quadruple heterogeneous core in 1Q06

9 Multi-Core and Multi-Chip There are two ways to put more CPUs on a chip : Multi-core has several cores on the same die Multi-chip glues several chip on the same module One and the other can be used depending on time: Multi-core is most costly to develop since it s a new chip design Multi-core and multi-chip can be used simultaneously Single Chip/Dual core Dual Chip Single Module

10 Multi-Core and Multi-Chip Multi Core: AMD and Intel are calling dual processor What IBM is calling dual core chip (with two processors) Therefore comparison is uneasy What they call 4 dual core processors is indeed a 8 way!!! Multi Chip: Intel is also calling multi-chip processor what we call multi-chip module For example the p550q is a multi-chip processor with two quadruple chip module with dual-core chips : 8 ways with two modules only

11 POWER Processor Roadmap POWER4 TM POWER4+ TM POWER5 TM POWER5+ TM POWER6 TM 180 nm 1+ GHz Core 1+ GHz Core 130 nm 1.2+ GHz Core GHz GHz Core Shared L2 130 nm > GHz Core > GHz Core Shared L2 90 nm >> GHz Core >> GHz Core Shared L2 Distributed Switch 65 nm Ultra high frequency cores L2 caches Advanced System Features Shared L2 Distributed Distributed Switch Switch Distributed Switch Distributed Switch Chip Multi Processing - Distributed Switch - Shared L2 Dynamic LPARs (16) Reduced size Lower power Larger L2 More LPARs (32) Simultaneous multi-threading Sub-processor partitioning Dynamic firmware updates Enhanced scalability, parallelism High throughput performance Enhanced cache/memory subsystem Autonomic Computing Enhancements

12 PowerPC : The Most Scalable Architecture POWER3 POWER2 PPC 603e PPC 401 PPC 750 PPC 405GP PPC 750CXe POWER4 PPC 750FX PPC 440GP PPC 750GX POWER5 PPC 970FX PPC 440GX POWER6 Desktop Games Embedded Servers Binary Compatibility PPC 970FX

13 BlueGene/L

14 BlueGene/L Hardware Principles A large number of nodes (65,536) Low-power nodes for density High floating-point performance System-on-a-chip technology Nodes interconnected as 64x32x32 three-dimensional torus Easy to build large systems, as each node connects only to six nearest neighbors full routing in hardware Cross-section bandwidth per node is proportional to n 2 /n 3 Auxiliary networks for I/O and global operations Applications consist of multiple processes with message passing Strictly one process/node

15 BlueGene/L Compute System-ona-Chip ASIC 5.5GB/s P LB (4:1) 5.6GF peak node 2.7GB/s 32k/32k L1 440 CP U Double FPU 32k/32k L1 440 CPU I/O proc Double FPU L2 snoop L Multiported Shared SRAM Buffer GB/s Shar ed L3 dir ectory for EDRAM Includes ECC ECC 22GB/s 4M B EDRAM L3 Cache or Memory l 128 Ethernet Gbit JTAG Access Torus Tree Global Interrupt DDR Control with ECC 5.5 GB/s Gbit Ethernet JTAG 6 out and 6 in, each at 1.4 Gbit/s link 3 out and 3 in, each at 2.8 Gbit/s link 4 global barriers or interrupts 144 bit wide DDR 256MB

16 BlueGene/L Interconnection Networks 3 Dimensional Torus Interconnects all compute nodes (65,536) Virtual cut-through hardware routing 1.4Gb/s on all 12 node links (2.1 GB/s per node) Communications backbone for computations 350/700 GB/s bisection bandwidth Global Tree One-to-all broadcast functionality Reduction operations functionality 2.8 Gb/s of bandwidth per link Latency of tree traversal in the order of 5 µs Interconnects all compute and I/O nodes (1024) Ethernet Incorporated into every node ASIC Active in the I/O nodes (1:64) All external comm. (file I/O, control, user interaction, etc.) Low Latency Global Barrier and Interrupt Control Network

17 BlueGene/L System Software Architecture User applications execute exclusively on the compute nodes and see only the application volume as exposed by the user-level APIs The outside world interacts only with the I/O nodes and processing sets (I/O node + compute nodes) they represent, through the operational surface functionally, the machine behaves as a cluster of I/O nodes Internally, the machine is controlled through the service nodes in the control surface goal is to hide this surface as much as possible

18 Software Stack in BlueGene/L Compute Node CNK controls all access to hardware, and enables bypass for application use User-space libraries and applications can directly access torus and tree through bypass As a policy, user-space code should not directly touch hardware, but there is no enforcement of that policy Application code can use both processors in a compute node Application code User-space libraries CNK Bypass BlueGene/L ASIC

19 High Performance and Scalability on LINPACK Linpack Scalability on BlueGene/L 100 Percentage of peak Performance (Rmax): 1435 Gflop/s Problem size (N): TOP500 rank: Number of nodes

20 MPI Latency and Bandwidth on BlueGene/L ½ roundtrip latency: 3000 cycles MHz Measured with Dave Turner s Measured in heater mode; bound to increase a bit in co-processor mode Measured on nearest neighbors: HW latency is only about 1200 cycles Composition of roundtrip latency: HW 32% High level (MPI) 26% Bandwidth: Measured with a custom made program that sends nearest neighbor messages Heater mode Only eager protocol suboptimally implemented (224 byte packet payload instead of 240) Max bandwidth = * B TP (864 MBytes/s send, receive) packet overheads 29% msg layer 13%

21 Artist s Rendition of Full BlueGene/L System Source:

22 PowerPC 970 and POWER4 Comparaison Memory Bandwidth 12.8 GB/s POWER4+ SMP Optimized Balanced system/bus design One or two processor cores Memory Bandwidth 6.4 GB/s PowerPC 970 CPU Core L2 Cache ( 512KB ) SIMD Units BIU Single processor core 2 Load/store units 2 Fixed point units 2 IEE Floating point units 2 SIMD sub-units (VMX 32 bit) Branch unit Condition register unit CPU Core CPU Core L2 Cache Chip- Chip Interconnect L2 Cache CPU Core Chip- Chip Interconnect 8 Execution pipeline 2 load / store units 2 fixed point units 2 DP multiply-add execution units 1 branch resolution unit / 1 CR execution unit 8 prefetching streams L3 Cache L3 Cache Main Memory Main Memory

23 BladeCenter Server Overview Enterprise-class shared infrastructure Shared power, cooling, cabling, switches means reduced cost and improved availability Enables consolidation of many servers for improved utilization curve High performance and density: 168 processors in a 42U rack

24 The portfolio continues to build... HS20 2-way Xeon HS40 4-way Xeon JS20 PowerPC Featu res Intel Xeon DP EM64T Mainstream rack dense blade High availability apps Optional HS HDD Intel Xeon MP processors 4-way SMP capability Supports Windows, Linux, and NetWare Two PowerPC 970 processors 32-bit/64-bit solution for Linux & AIX 5L Performance for deep computing clusters LS20 AMD Blade Two socket AMD Dual core ready Similar feature set to HS20 Edge and mid-tier workloads Collaboration Web serving Target Apps Back-end workloads Large mid-tier apps 32-bit/64-bit HPC UNIX server consolidation 32- or 64-bit HPC stellar performer Common Chassis and Infrastructure

25

26

27 Cell Processor Based Workstation (CPBW) (Sony Group and IBM) First Prototype Powered On 16 Tera-flops in a rack (est.) ( equals 1 Peta-flop in 64 racks ) Optimized for Digital Content Creation, including Computer entertainment Movies Real-time rendering Physics simulation CELL Processor Board High BW Sys Networks Mgmt I/O Bridge CELL Processor (2-Way SMP) Memory 16 TFlop rack Storage

28 Very Basic Cluster Design Management A Cluster that is designed like this is ready to perform a variety of compute functions. Node Network Switch Storage Nodes Storage Nodes Compute Nodes Compute Nodes Compute Nodes Compute Nodes Compute Nodes Compute Nodes Compute Nodes Compute Nodes

29 Could I have an example as picture? Sure... Terminal Servers Ethernet switches Mgmt. Lan from Myrinet 2000 Management VLAN Storage Controllers Cluster Nodes RS RS-485 Management Node Myrinet2000 Cluster VLAN 100 Mbit Enet (integrated) Gigabit Enet Myrinet 2000 Falcon 100 Mbit Enet Gigabit Enet Myrinet 2000

30 Interconnect Performance Interconnect Type Bi-directional Bandwidth* Unidirectional Bandwidth* Small Message Latency I/O Bus Equivalent Gigabit Ethernet 250 MB/s (184 MB/s) 125 MB/s (111 MB/s) microsecond 66 MHz PCI Single-port Myrinet 500 MB/s (463 MB/s) 250 MB/s (229 MB/s) ~4.5 microsecond 100 MHz PCI-X Dual-port Myrinet 1000 MB/s (>850 MB/s) 500 MB/s (448 MB/s) ~4.2 microsecond 133 MHz PCI-X Quadrics 1067 MB/s (950 MB/s) 1067 MB/s (950 MB/s) ~1.5 microsecond 133 MHz PCI-X Single-port InfiniBand 2000 MB/s (1750 MB/s) 1000 MB/s (914 MB/s) ~3.9 microsecond 4x PCI-E * B/W quoted as Peak and (Payload)

31 Infiniband - TopSpin Bandwidth (MBps/s) MPI over InfiniBand MPI over Myrinet MPI over Quadrics Latency (us) MPI over InfiniBand MPI over Myrinet MPI over Quadrics Message Size (Bytes) Message Size (Bytes) Throughput InfiniBand 840 MBps Quadrics 300 MBps Myrinet 220 MBps GigE 120 MBps Latency (small msg) 5.5 us 5 us 7 us 70 us CPU Utilization 1-3% Not available Not available 50% Source: Ohio State and Topspin

32 Systems Management for HPC CSM Clusters Diagnostics Diagnostics Monitoring Monitoring Sensors Sensors HPC Stack Parallel Environment GPFS Loadleveler Deployment Deployment Integration Integration AIX SUSE RedHat Interoperability Interoperability Adapter Adapter Setup Setup Monitoring Monitoring Myrinet GigE Federation InfiniBand Hardware Hardware Control Control Deployment Deployment pseries Cluster 1600 BlueGene/L BladeCenter x&p xseries (Intel, AMD) Cluster 1350

33 How does GPFS work? Network Shared Disk (NSD) Direct Attach (DA)

34 How does GPFS work? Each file spans multiple servers and disks Imagine a big virtual RAID across servers and disks Network Shared Disk (NSD) concept Disks appear to be local Transparent I/O over cluster interconnect Application GPFS Daemon NSD NSD GPFS Daemon Application Linux Linux Disk Device Driver Interconnect Device Driver Interconnect Device Driver Disk Device Driver Disk Interconnect Disk

35 GPFS: Storage Nodes - Data Access from any GPFS client via NSD - Disk processing off-loaded from application through NSD servers - High Avalaibility and Recoverability - High Speed/Bandwidth Network suggest for Scalability Myrinet/Infiniband ready

36 I/O Network

37 GPFS in GRID environment