Resources Current and Future Systems. Timothy H. Kaiser, Ph.D.

Transcription

1 Resources Current and Future Systems Timothy H. Kaiser, Ph.D. 1

2 Most likely talk to be out of date History of Top 500 Issues with building bigger machines Current and near future academic machines 2

3 Top 500 list Ranks computers based on performance on a linear solve 3

4 Top500 Benchmarks Spring 12 4

5 Trends 5

6 Colorado State Cray model XT6m Opera1onal January 2011 Peak performance 12 teraflops Dimensions: 7.5 E. (h) x 2.0 E. (w) x 4.5 E. (d) Computer par11on 52 compute nodes 2 processors / node 104 AMD Magny Cours 64- bit 1.9 GHz total processors 12 cores / processor; 1,248 total cores 32 GB DDR3 ECC SDRAM / node; TB total RAM 6

7 NCAR's Computational and Information Systems Laboratory (CISL) invites NSFsupported university researchers in the atmospheric, oceanic, and closely related sciences to submit large allocation requests by September 17, University researchers supported by an NSF award can request up to 30,000 GAUs as a Small Allocation request. Up to 10,000 GAUs are available to graduate students and post-docs; no NSF award is required. 7

8 NCAR & CISL systems Yellowstone A 1.5-petaflops high-performance computing system with 72,288 processor cores and 144 terabytes of memory. Production computing operations will begin in the summer of Bluefire NCAR's 77-teraflops IBM Power6 system used by the Climate Simulation Lab (CSL) and Community Computing Facilities. Janus The Janus system is a Dell Linux cluster that is housed on the CU- Boulder campus and has a high-speed networking connection to NCAR's computing and data storage systems. Lynx A Cray XT5m system deployed as a testing platform and available to NCAR users. Mirage and Storm CISL operates two data analysis and visualization clusters, with software packages including NCL, Vapor, Matlab and IDL, for its user community. GLADE The central GLADE file system significantly expands the disk space available to CISL users and allows users to access their data from both HPC and DAV systems. HPSS CISL has migrated its archival storage resource to the High-Performance Storage System (HPSS) environment, which currently stores more than 12 PB of data in support of CISL computing facilities and NCAR research activities. 8

9 9

10 NCAR%Resources%!!at!the!NCAR!Wyoming!Supercompu6ng!Center!(NWSC)! Centralized!Filesystems!and!Data!Storage!(GLADE)! >90!GB/sec!aggregate!I/O!bandwidth,!GPFS!filesystems! 10.9!PetaBytes!iniJally!K>!16.4!PetaBytes!in!1Q2014! High!Performance!CompuJng!(Yellowstone)! IBM!iDataPlex!Cluster!with!Intel!Xeon!E5K2670!processors!with!Advanced!Vector! Extensions!(AVX)! 1.50!PetaFLOPs!!28.9!bluefireKequivalents!!4,518!nodes!!72,288!cores! 145!TeraBytes!total!memory! Mellanox!FDR!InfiniBand!full!fatKtree!interconnect! Data!Analysis!and!VisualizaJon!(Geyser!&!Caldera)! Large!Memory!System!with!Intel!Westmere!EX!processors! 16!nodes,!640!WestmereKEX!cores,!16!TeraBytes!memory,!16!nVIDIA!Quadro!6000!GPUs! GPUKComputaJon/Vis!System!with!Intel!Sandy!Bridge!EP!processors!with!AVX! 16!nodes,!256!E5K2670!cores,!1!TeraByte!memory,!32!nVIDIA!M2070Q!GPUs! Knights!Corner!System!with!Intel!Sandy!Bridge!EP!processors!with!AVX! 16!Knights!Corner!nodes,!256!E5K2670!cores,!>1600!KC!cores,!1!TB!memory!!Early!2013!deliver! NCAR!HPSS!Data!Archive! >100!PetaByte!capacity!(with!5!TeraByte!cartridges,!uncompressed)! 10! Codenamed! Sandy!Bridge!EP! 2

11 Yellowstone Compute 72,288 processor cores 2.6-GHz Intel Sandy Bridge EP with Advanced Vector Extensions (AVX) 8-flops clock 4,518 nodes IBM dx360 M4, dual socket, 8 cores per socket TB total system memory 2 GB/core, 32 GB/node, DDR FDR Mellanox InfiniBand interconnect Full fat tree, single plane Bandwidth 13.6 GBps bidirectional per node; latency 2.5 µs Peak bidirectional bisection bandwidth: 31.7 TBps petaflops peak 1.20 petaflops estimated HPL 11

12 XSEDE Extreme Science and Engineering Discovery Environment Mostly the same people as TeraGrid Mostly the same machines 12

13 XSEDE Machines: Yellowstone will be added to this list 13

14 Future Directions in HPC Four important concepts that will effect math software - Jack Dongarra Effective use of many-core Exploiting mixed precision in our numerical computations Self adapting / auto tuning of software Fault tolerant algorithms Barriers to progress are increasingly on the software side. Hardware has a half-life measured in years, while software has a half-life measured in decades. High performance ecosystem out of balance: HW, SW, OS, Compilers, Algorithms, Applications. 14

15 Trends Hardware Large number of cores Less memory per core More Flops/Watt Better Interconnect Software Hybrid programming Directives based 15

16 Top500 Benchmarks Spring 12 16

17 GPU GPU computing is the use of a GPU (graphics processing unit) together with a CPU to accelerate general-purpose scientific and engineering applications. GPUs do real computation Vendors have taken GPU systems and repackaged them to do computation Vendors IBM AMD Nvidia Intel NVidia Tesla M2090 GPU 17

18 Not a completely new concept Think coprocessor Main processor passes off some work to coprocessor Remember the 8087? Same issues Programs must be written to take advantage Getting data to/from coprocessor 18

19 Programming (Bottom Level) Program is written in two parts CPU GPU Computation starts on CPU Data is prepared on CPU Data is sent back to CPU Data and Program (subroutine) are sent to GPU Subroutine run on GPU as a thread 19

20 Issues Complexity Separate code for GPU Easy to write tough to get to run well Bottleneck between CPU and GPU Mixed precision Efficiency on the GPU Small amount of fast memory Massive number of threads must be managed 20

21 GPUs Many more cores Does not support normal process Expected to run multiple threads per core Very small fast memory MUCH less memory per core 21

22 Issues Complexity Directives based programming similar to OpenMP Libraries Bottleneck between CPU and GPU Getting Better Mixed precision (Some) newer GPUs have better ratio of performance Efficiency on the GPU More memory and flatter hierarchy Better thread management 22

23 CSM s old GPU node (2009) # of Tesla GPUs 4 # of Streaming Processor Cores 960 (240 per processor) Frequency of processor cores to 1.44 GHz Single Precision floating point performance (peak) 3.73 to 4.14 TFlops Double Precision floating point performance (peak) 311 to 345 GFlops Floating Point Precision IEEE 754 single & double Total Dedicated Memory 16 GB Memory Interface 512-bit Memory Bandwidth 408 GB/sec Max Power Consumption 800 W System Interface PCIe x16 or x8 Software Development Tools C-based CUDA Toolkit 23

24 Today s NVIDA offerings 24

25 Intel Many Integrated Core (MIC) What? Many (>50) cores on a chip Each core is x86 type processor Why? Massive parallelizm Same (MoL) instruction set as other X86 When? Knights Corner Prerelease product PCI card Available very soon as Xeon Phi, also PCI card 25

26 Intel MIC differences X86 instruction set Can in theory, run full os on the card Should most likely run threads (OpenMP) Uses the same compilers as normal Intel processors Codes optimized for current generation processor will run well on MIC Threading Vectorization 26

27 Next few slides taken from Dr. Jay Boisseau Director of TACC 27

28 MIC Architecture Many cores on the die L1 and L2 cache Bidirectional ring network Memory and PCIe connection MIC (KNF) architecture block diagram Knights Ferry SDP Up to 32 cores 1-2 GB of GDDR5 RAM 512-bit wide SIMD registers L1/L2 caches Multiple threads (up to 4) per core Slow operation in double precision Knights Corner (first product) 50+ cores Increased amount of RAM Details are under NDA Double precision half the speed of single precision (canonical ratio) 22 nm technology 28

29 What we at TACC like about MIC (and we think that you will like this, too) Intel s MIC is based on x86 technology x86 cores w/ caches and cache coherency SIMD instruction set Programming for MIC is similar to programming for CPUs Familiar languages: C/C++ and Fortran Familiar parallel programming models: OpenMP & MPI MPI on host and on the coprocessor Any code can run on MIC, not just kernels Optimizing for MIC is similar to optimizing for CPUs Make use of existing knowledge! Key elements of this talk highlighted! 29

30 Differences Coprocessor vs. Accelerator Architecture: x86 vs. streaming processors HPC Programming model: Threading/MPI: Programming details coherent caches vs. shared memory and caches extension to C++/C/Fortran vs. CUDA/OpenCL OpenCL support OpenMP and Multithreading vs. threads in hardware MPI on host and/or MIC vs. MPI on host only offloaded regions vs. kernels Support for any code: serial, scripting, etc. Yes No Native mode: Any code may be offloaded as a whole to the coprocessor 30

31 Programming Models Ready to use on day one! TBB s will be available to C++ programmers MKL will be available Automatic offloading by compiler for some MKL features Cilk Plus Useful for task-parallel programing (add-on to OpenMP) May become available for Fortran users as well OpenMP TACC expects that OpenMP will be the most interesting programming model for our HPC users 31

32 IBM Blue Gene Q New machine from IBM Evolution from BGL and BGP Many cores / node with less memory / core but more than L or P Very energy efficient 4 of the top 8 on top 500 list 32

33 BGQ Rack 208 Tflop 62.5 kw 1 rack 1024 nodes cores 1 node = 16+1 cores 16 Gbytes or 1Gbyte/core Footprint < 31 ft2 33

34 BGQ Proprietary Parts Processors Designed for HPC 4 threads/core Advanced speculative operation Transactional memory Networks 5D torus Collective and barrier Floating point addition in network Special IO Nodes 34

35 5D Torus What the? 35

36 Summary Core count is going up Memory / core is going down Threading will become more important Hybrid will be critical 36