SuperMUC. PetaScale HPC at the Leibniz Supercomputing Centre (LRZ) Top 500 Supercomputer (Juni 2012)

Transcription

1 SuperMUC PetaScale HPC at the Leibniz Supercomputing Centre (LRZ) Dieter Kranzlmüller Munich Network Management Team Ludwig Maximilians Universität München (LMU) & Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities Top 500 Supercomputer (Juni 2012) D. Kranzlmüller SuperMUC PetaScale LRZ 2 1

2 Ernst A. Graf D. Kranzlmüller SuperMUC PetaScale LRZ 3 Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities With 156 employees + 38 extra staff for more than students and for more than employees including scientists Institute Building Visualisation Centre Cuboid containing computing systems 72 x 36 x 36 meters Institute Building Lecture Halls Ernst A. Graf D. Kranzlmüller SuperMUC PetaScale LRZ 4 2

3 Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities Computer Centre for all Munich Universities IT Service Provider: Munich Scientific Network (MWN) Web servers e Learning E Mail Groupware Special equipment: Virtual Reality Laboratory Video Conference Scanners for slides and large documents Large scale plotters IT Competence Centre: Hotline and support Consulting (security, networking, scientifc computing, ) Courses (text editing, image processing, UNIX, Linux, HPC, ) D. Kranzlmüller SuperMUC PetaScale LRZ 5 Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities Regional Computer Centre for all Bavarian Universities Computer Centre for all Munich Universities D. Kranzlmüller SuperMUC PetaScale LRZ 6 3

4 Virtual Reality and Visualisation Centre (V2C) 5 sided projection room + large scale high resolution powerwall D. Kranzlmüller SuperMUC PetaScale LRZ 7 Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities European Supercomputing Centre National Supercomputing Centre Regional Computer Centre for all Bavarian Universities Computer Centre for all Munich Universities SuperMUC SGI UV SGI Altix Linux Clusters Linux Hosting and Housing D. Kranzlmüller SuperMUC PetaScale LRZ 8 4

5 PRACE Timeline PRACE Partnership for Advanced Computing in Europe HPCEUR HET D. Kranzlmüller SuperMUC PetaScale LRZ 9 The European Scientific Case Weather, Climatology, Earth Science degree of warming, scenarios for our future climate. understand and predict ocean properties and variations weather and flood events Astrophysics, Elementary particle physics, Plasma physics systems, structures which span a large range of different length and time scales quantum field theories like QCD, ITER Material Science, Chemistry, Nanoscience understanding complex materials, complex chemistry, nanoscience the determination of electronic and transport properties Life Science system biology, chromatin dynamics, large scale protein dynamics, protein association and aggregation, supramolecular systems, medicine Engineering complex helicopter simulation, biomedical flows, gas turbines and internal combustion engines, forest fires, green aircraft, virtual power plant D. Kranzlmüller SuperMUC PetaScale LRZ 10 5

6 PRACE Timeline PRACE Partnership for Advanced Computing in Europe HPCEUR HET PRACE MoU D. Kranzlmüller SuperMUC PetaScale LRZ 11 PRACE Timeline PRACE Partnership for Advanced Computing in Europe HPCEUR HET PRACE MoU PRACE Preparatory EU-Grant: INFSO-RI , 10 Mio. Phase PRACE Operation PRACE Implementation Phase (1IP, 2IP, 3IP) PRACE (AISBL), a legal entity with (current) seat location in Brussels D. Kranzlmüller SuperMUC PetaScale LRZ 12 6

7 PRACE Research Infrastructure Created Establishment of the legal framework PRACE AISBL created with seat in Brussels in April (Association Internationale Sans But Lucratif) 20 members representing 20 European countries Inauguration in Barcelona on June 9 D. Kranzlmüller SuperMUC PetaScale LRZ 13 Gauss Centre for Supercomputing (GCS) Joint activity of the 3 German National HPC Centres John von Neumann Institut für Computing (NIC), Jülich Leibniz Supercomputing Centre (LRZ), Garching near Munich Höchstleistungsrechenzentrum Stuttgart (HLRS), Stuttgart Largest and most powerful supercomputer infrastructure in Europe Foundation of GCS (e.v.) April, 13th, Principal Partner in PRACE (Partnership for Advanced Computing in Europe) D. Kranzlmüller SuperMUC PetaScale LRZ 14 7

8 PRACE Research Infrastructure Created Establishment of the legal framework PRACE AISBL created with seat in Brussels in April (Association Internationale Sans But Lucratif) 20 members representing 20 European countries Inauguration in Barcelona on June 9 Funding secured for Million from France, Germany, Italy, Spain Provided as Tier 0 services on TCO basis Funding decision for 100 Million in The Netherlands expected soon 70+ Million from EC FP7 for preparatory and implementation Grants INFSO RI and Complemented by ~ 60 Million from PRACE members D. Kranzlmüller SuperMUC PetaScale LRZ 15 PRACE Tier 0 Systems 1 st Tier 0 System Jugene: BlueGene/P in GCS@Juelich 72 Racks, 1 PFlop/s peak 35% of capacity provided to PRACE 2 nd Tier 0 System Curie: Bull Cluster with Intel CPUs operated by CEA 1.6 PFlop/s peak in Oct (1 st step in 10/2010) Largest fraction of capacity provided to PRACE Next Procurements (in alphabetical order) BSC, CINECA, GCS@HLRS, GCS@LRZ Procurement plan based on analysis of user requirements and market D. Kranzlmüller SuperMUC PetaScale LRZ 16 8

9 PRACE Tier 0 Access Single pan European Peer Review Early Access Call in May proposals asked for 1870 Million Core hours 10 projects granted with 328 Million Core hours Principal Investigators from D (5), UK (2) NL (1), I (1), PT (1) Involves researchers from 31 institutions in 12 countries 1st Regular Call closed on August proposals received asked for 2900 million core hours 33 proposals have fulfilled the technical assessment 360 million core hours available for a 12 months allocation period starting November 2010 Further calls being scheduled (every 6 months) 2nd regular call will include both Jugene and Curie D. Kranzlmüller SuperMUC PetaScale LRZ 17 PRACE Tier 0 at LRZ: SuperMUC Video: SuperMUC rendered on SuperMUC by LRZ D. Kranzlmüller SuperMUC PetaScale LRZ 18 9

10 PRACE Tier LRZ: SuperMUC Internet Achive and Backup ~ 30 PB Snapshots/Replika 1.5 PB (separate fire section) $HOME 1.5 PB / 10 GB/s NAS 80 Gbit/s Desaster Recovery Site Peak: 3 PF expected: <3 MW >150,000 cores pruned tree (4:1) non blocking SB-EP 16 cores/node 2 GB/core non blocking WM-EX 40cores/node 6.4 GB/core GPFS for $WORK $SCRATCH 10 PB 200 GB/s Compute nodes 18 Thin node islands (each >8000 cores) Compute nodes 1 Fat node island (8200 cores) SuperMIG I/O nodes D. Kranzlmüller SuperMUC PetaScale LRZ 19 LRZ Building Extension Picture: Horst-Dieter Steinhöfer Figure: Herzog+Partner für StBAM2 (staatl. Hochbauamt München 2) Picture: Ernst A. Graf D. Kranzlmüller SuperMUC PetaScale LRZ 20 10

11 Energy Efficient Supercomputing at LRZ 10,000,000 SuperMUC IBM 1,000,000 = 1 Peta fold every 4 years Double every 14.1 Montths HLRB2: SGI Altix Linux-Cluster 100,000 GFlop/s, GByte 10,000 1, = 1Tera... KSR IBM SP2 Cray T90 Fujitsu HLRB1: Hitachi SR Cray Y-MP2 Cray Y-MP8 GFlop/s GByte = 1Giga 1 D. Kranzlmüller SuperMUC PetaScale LRZ 21 Energy Consumption at LRZ Powerconsumption in MWh C HPC Workshop, Shanghai, September 29, 2011 D. Kranzlmüller SuperMUC PetaScale LRZ 22 11

12 SuperMUC Cooling D. Kranzlmüller SuperMUC PetaScale LRZ 23 IBM System x idataplex Direct Water Cooled Rack idataplex DWC Rack w/ water cooled nodes (front view) idataplex DWC Rack w/ water cooled nodes (rear view of water manifolds) Torsten Bloth, IBM Lab Services - IBM Corporation D. Kranzlmüller SuperMUC PetaScale LRZ 24 12

13 IBM idataplex dx360 M4 Torsten Bloth, IBM Lab Services - IBM Corporation D. Kranzlmüller SuperMUC PetaScale LRZ 25 D. Kranzlmüller SuperMUC PetaScale LRZ 26 13

14 Cooling Concept Dedicated Free Cooling D. Kranzlmüller SuperMUC PetaScale LRZ 27 Cooling Infrastructure 3rd Floor Pictures: StBAM2 (staatl. Hochbauamt München 2) D. Kranzlmüller SuperMUC PetaScale LRZ 28 14

15 Cooling Infrastructure Rooftop Picture: StBAM2 (staatl. Hochbauamt München 2) D. Kranzlmüller SuperMUC PetaScale LRZ 29 SuperMUC HPL Energy Consumption Power (Machine Room, kw) Power (PDU, kw) Power (kw) :3018:0019:3021:0022:300:00 1:30 3:00 4:30 6:00 7:30 9:0010:3012:0013:30 Time (Clock) Energy Consumption (kwh) Power (infrastructure, kw) Energy (Machine Room, kwh) Integrated Power (PDUs, kwh) Energy efficiency (single number in GFlops/Watt) 9,380E 01 (PDU, 10 minutes resolution, whole run) 9,359E 01 (PDU, 1 minutes resolution, whole run, without cooling) 9,359E 01 (PDU, 1 minutes resolution, whole run, cooling included) 8,871E 01 (machine room measurement, whole run) 7,296E 01 (infrastructure measurement, whole run) D. Kranzlmüller SuperMUC PetaScale LRZ 30 15

16 SuperMUC & Green IT D. Kranzlmüller SuperMUC PetaScale LRZ 31 SuperMUC Energy Efficient Supercomputing Probably most powerful x86 system in Europe (3PetaFlops peak) Use for science in Europe (PRACE), Germany (GCS) and Bavaria (KONWIHR) System with > cores, 324 TeraByte Main Memory Most energy efficient General Purpose Supercomputer in Europe in 2012 Hot liquid cooling Reuse of waste heat Hardware and software tools for clock scaling and optimization ( dynamic frequency scaling, CPU throttling WIKIPEDIA) D. Kranzlmüller SuperMUC PetaScale LRZ 32 16

17 Energy Efficiency at LRZ Measures around SuperMUC New Contract (Spot Market / Evaluation of alternative Technologies) Optimization of Building and Cooling Infrastructure (additional cooling loop) Hot liquid cooling PUE < 1,1 Cooperation LRZ / TUM / LMU / IBM on Tools and Provider / User Strategies Cooperation with Building Management YIT PRACE 1 IP Evaluation Prototype SGI Ultraviolet 2 IP Evaluation Prototype T Platforms Exascale EU Project DEEP: System based on Accelerator Architecture (Intel MIC) Cooling and Prototype Evaluation Exascale EU Project Mont Blanc System based on low power commercially available embedded CPUs Next generation HPC machine with a range of embedded technology Software applications to run on this new generation of HPC systems D. Kranzlmüller SuperMUC PetaScale LRZ 33 SuperMUC PetaScale LRZ Dieter Kranzlmüller kranzlmueller@lrz.de D. Kranzlmüller SuperMUC PetaScale LRZ 34 17