Supercomputing: scientific instruments and precursors for new technologies what does this mean for Switzerland? Thomas C.

Transcription

1 Supercomputing: scientific instruments and precursors for new technologies what does this mean for Switzerland? Thomas C. Schulthess

2 Today s state of the art climate simulation (resolution T85 ~ 148 km)

3 Experimental climate running at higher resolution (resolution T341 ~ 37 km)

4 Why resolution is such an issue for Switzerland 70 km 35 km 1X 8.8 km 100X 2.2 km 10,000X 0.55 km 1,000,000X Source: Oliver Fuhrer, MeteoSwiss

5 Premise: 3 pillars of 21. century scientific method Theory (since antiquity) combined with experiment (since Galilei & Newton) and simulation (since Metropolis, Teller, von Neumann, Fermi, s) Excellence in Science requires excellence in all three areas: theory, experiment, and simulations

6 Computer performance and application performance increase ~10 3 every decade ~100 Kilowatts ~5 Megawatts MW ~1 Exaflop/s 1.35 Petaflop/s Cray XT processors 100 million or billion processing cores (!) 1.02 Teraflop/s Cray T3E processors 1 Gigaflop/s Cray YMP 8 processors First sustained GFlop/s Gordon Bell Prize 1988 First sustained TFlop/s Gordon Bell Prize 1998 First sustained PFlop/s Gordon Bell Prize 2008 Another 1,000x increase in sustained performance

7 Moore s Law is still alive and well illustration: A. Tovey, source: D. Patterson, UC Berkeley

8 Limits of CMOS scaling Oxide layer thickness ~1nm Source: Ronald Luijten, IBM-ZRL t ox /α Voltage, V/α GATE n+ n+ source drain L/α p substrate, doping WIRING W/α SCALING Voltage: Oxide: Wire width: Gate Width: Diffusion: Substrate: V/α t ox /α W/α L/α x d /α α N A CONSEQUENCE: Higher density: α 2 x d /α α Higher speed: α N A Power/ckt: 1/α 2 Power density: The power challenge today is a precursor of more physical limitations in scaling atomic limit! constant

9 1000 fold increase in performance in 10 years: > previously: double transistor density every 18 months = 100X in 10 years frequency increased > now: only 1.75X transistor density every 2 years = 16X in 10 years frequency almost the same Need to make up a factor 60 somewhere else Source: Rajeeb Hazra s (HPC@Intel) talk at SOS14, March 2010 SARA 26th Annual Superdag

10 Source: Rajeeb Hazra s (HPC@Intel) talk at SOS14, March 2010 SARA 26th Annual Superdag

11 Petaflop/s = bit floating point operations / sec. which takes more energy? 64-bit floating-point fused multiply add or moving three 64-bit operands 20 mm across the die 934, x = 49,370, = 49,370, mm this takes over 3x the energy! loading the data from off chip takes > 10x more yet source: Steve Scott, Cray Inc. moving data is expensive exploiting data locality is critical to energy efficiency If we care about energy consumption, we have to worry about these and other physical considerations of the computation but where is the separation of concerns?

12 Von Neumann Architecture: Memory Memory CPU Control Unit Arithmetic Logic Unit accumulator I/O unit(s) Input Output stored-program concept = general purpose computing machine

13 Memory hierarchy to work around latency and bandwidth problems Functional units CPU Expensive, fast, small Registers Internal cash ~100 GB/s ~ 6-10 ns External cash ~50 GB/s Cheap, slow, large Main memory (RAM) ~10 GB/s ~ 75 ns

14 Distributed vs. shared memory architecture Distributed memory Interconnect CPU Memory Shared memory

15 Interconnect types on massively parallel processing (MPP) systems distributed memory Switch(es) / router(s) RAM RAM RAM RAM CPU CPU CPU... CPU... NIC & Router NIC & Router NIC & Router... NIC & Router NIC NIC NIC NIC & Router NIC & Router NIC & Router... NIC & Router CPU CPU... CPU CPU CPU CPU... CPU RAM RAM RAM RAM RAM RAM RAM

16 Larger parallel computers only solve part of the problem 2x 2x Run on 4x the number of processors Sequential >2x Calculations have to be more efficient: better implementation, better algorithms, more suitable systems Time

17 Applications running at scale on ORNL Fall 2009 Domain area Code name Institution # of cores Performance Notes Materials DCA++ ORNL 213, PF Materials WL-LSMS ORNL/ETH 223, PF Chemistry NWChem PNNL/ORNL 224, PF 2008 Gordon Bell Prize Winner 2009 Gordon Bell Prize Winner 2008 Gordon Bell Prize Finalist Materials OMEN Duke 222, TF Chemistry MADNESS UT/ORNL 140, TF Materials LS3DF LBL 147, TF Seismology SPECFEM3D USA (multiple) 149, TF 2008 Gordon Bell Prize Winner 2008 Gordon Bell Prize Finalist Combustion S3D SNL 147, TF Weather WRF USA (multiple) 150, TF

18 Algorithmic motifs and their arithmetic intensity Arithmetic intensity: number of operations per word of memory transferred Finite difference / stencil in S3D and WRF Rank-1 update in HF-QMC Sparse linear algebra Matrix-Vector Vector-Vector BLAS1&2 Fast Fourier Transforms FFTW Rank-N update in DCA++ QMR in WL-LSMS Linpack (Top500) Dense Matrix-Matrix BLAS3 O(1) O(log N) O(N) Supercomputers are designed for certain algorithmic motifs which ones?

19 Leaving the comfort zone of separated concerns Decompose application into algorithmic motifs Understand the hierarchy of motifs the dwarfs are not all dwarfs Design systems or components therefore against these dwarfs Proper (theoretical) description of heterogenous systems... Algorithmic motifs and their arithmetic intensity Arithmetic intensity: number of operations per word of memory transferred Rank-1 update in HF-QMC Finite difference / stencil Sparse linear algebra BLAS1 BLAS2 O(1) O(log N) O(N) Eckert and Mauchly s invention is named after von Neumann! FFT Rank-N update in DCA++ QMR in WL-LSMS Linpack Dense matrix-matrix BLAS3

20 Relationship between simulations and supercomputer system Science? Simulations Model & method of solution Port codes developed on workstations > vectorize codes > parallelize codes > petascaling and soon exascaling Basic numerical libraries Programming environment Supercomputer Runtime system Operating systems Computer Hardware

21 Relationship between simulations and supercomputer system Simulations + Theory + Experiment Science Model & method of solution Mapping problem to supercomputer system > Algorithm re-engineering > Software refactoring > Domain specific libraries/languages, etc. > Focus on scientific / engineering problem > Requires interdisciplinary effort / team Basic numerical libraries Programming environment Runtime system Supercomputer Operating systems Co-Design Computer Hardware

22 Swiss Platform for High-Performance and High- Productivity Computing (, see Scientific problem Simulations + Theory + Experiment Swiss Universities / Federal Institutes of Technology (presently 11 domain science projects in HP2C Platform) Interdisciplinary teams consisting of: > model & method development > application software design / engineering > system software (everything between apps & hardware) > numerical libraries / programming environments > mapping methods onto computer hardware/ systems > hardware design / engineering Swiss National Supercomputing Center (CSCS) & U. of Lugano (USI) (collaboration with industry: Cray, IBM, Mellanox, SCS) Supercomputer IT manufacturers system integrators

23 Projects of the platform (see Gyrokinetic Simulations of Turbulence in Fusion Plasmas (ORB5) Laurent Villard, EPF Lausanne Ab initio Molecular Dynamics (CP2K) Jürg Hutter, Univ. of Zurich Computational Cosmology on the Petascale Geoge Lake, Univ. of Zurich Selectome, looking for Darwinian evolution in the tree of life Marc Robinson-Rechavi, Univ. of Lausanne Cardiovascular Systems Simulations (LifeV) Alfio Quarteroni, EPF Lausanne Modern Algorithms for Quantum Interacting Systems (MAQUIS) Thierry Giamarchi, Univ. of Geneva Large-Scale Parallel Nonlinear Optimization for High Resolution 3D- Seismic Imaging (Petaquacke) Olaf Schenk, Univ. of Basel 3D Models of Stellar Explosions Matthias Liebendörfer, Univ. of Basel Large Scale Electronic Structure Calculations (BigDFT) Stefan Gödecker, Univ. of Basel Regional Climate & Weather Model (COSMO) Isabelle Bey, ETH Zurich/C2SM Lattice-Boltzmann Modeling of the Ear Bastien Chopard, U. of Geneva

24 New building under construction in Lugano Computer room area (1500 m 2 ) Power & cooling ~ 15 MW (upgradable) (PUE ~ 1.2) Proximity to academic institution (USI) Extensible Facilitate seamless computer hardware upgrades/changes Current CSCS building in Manno: PUE ~1.7 i.e. 1 MW delivered to computer requires 1.7 MW electrical power

25 Supercomputing Ecosystem Resource allocation based on recommendation from a panel: PRACE Tier 0 Leadership > scientific review (external peers, SNF, ERC,...) > technical review (external Leadership peers) > readiness review (CSCS internal) Tier 1 Regional / National Resource usage Regional fully under / National the control of its owner (institution or individual project) Tier 2 Leadership Robust produciton systems Advanced development Regional / National systems Institutional production systems Computational Science and Engineering Prototypes Local/institutional supercomputer Local/institutional supercomputer Local/institutional supercomputer Time (a few years)

26 High-risk & high-impact projects of the ( New procurement Cray XT processors Upgrade Cray XT proc. Dual core upgrade Cray XT cores 2008 Upgrade Cray XT cores 2009 Hex-core upgrade cores Final upgrade Cray XT Procurement next generation supercomputer HPCN initiative Begin construction of new building New building complete

27 Roles we could play in Europe Why is Switzerland not a hosting partner in PRACE? Contribute to problem driven co-design most scientific problems are universal an not limited to Switzerland! HP2C can be extended to a European platform For a start: ETH/CSCS is leading WP8 in PRACE 2IP Our intention: give access to national supercomputing resources open to international audience starting 2012 when peta-scale systems are installed Access based only on scientific excellence, technical soundness and readiness Consistent with Swiss tradition in science: passport or country of origin doesn t matter No preset quotas funding agency may start asking questions if foreign usage goes well over 1/3 Allocation decisions are made locally (i.e. in Switzerland) and based on review process that meets Swiss standards for scientific user facilities

28 Summary and conclusions Excellence in science requires excellence in all three areas: theory, experiment, and simulations Moore s Law will not carry the day opportunities for innovations! Take into account physical considerations of the computation Build the right system for the problem at hand Top500.org is suboptimal for many problems! Rewrite application codes even if they are large Domain problems and supercomputing technology (center and industry) need to be integrated into interdisciplinary efforts e.g. HP2C.ch Highly energy efficient new CSCS building in Lugano energy cost savings alone will pay for new construction Keep up with rapid developments by focusing on innovation and full ecosystem rather than building just the fastest machine Starting in 2012: access to national system will be open to all, irrespective on nationality and based on scientific/technical quality & readiness only

29 QUESTIONS / COMMENTS?