The Energy Challenge in HPC

Transcription

1 ARNDT BODE Professor Arndt Bode is the Chair for Computer Architecture at the Leibniz-Supercomputing Center. He is Full Professor for Informatics at TU Mü nchen. His main research includes computer architecture, distributed and parallel computing, parallel tools and energy efficient data centers. Prof. Bode was the former Vice President and CIO of TU Mü nchen. He is now a member of the Bavarian Academy of Sciences and Humanities. The Energy Challenge in HPC

2 The Energy Challenge in HPC 13th HPC Connection Workshop: Towards Exascale Computing Frankfurt, June 21 st, 2016 Arndt Bode Herbert Huber Chairman of the Board, Leibniz-Rechenzentrum of the Bavarian Academy of Sciences and Humanities and Technische Universität München Head of HPC Division

3 June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 2

4 June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 3

5 LRZ: Organisational Embedding Bavarian State Ministry of Education, Science and the Arts Ludwig-Maximilians- Universität München Bavarian Academy of Sciences and Humanities Technische Universität München Other Bavarian Universities LRZ Advisory Board Board of Directors of the LRZ Prof. Bode (Chairman), Prof. Bungartz, Prof. Hegering, Prof. Kranzlmüller LRZ: Member of PRACE Gauss Centre for Supercomputing GCS Leibniz Supercomputing Centre June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 4

6 LRZ: Our Mission IT Service Provider for Munich Universities , Web, Multimedia, IT Security, HelpDesk, Virtual Reality, Trainings, etc. Regional Computing Centre for Bavarian Universities and Research institutions ~ 50 PByte Storage/Archive Digital Archive of the Bavarian State Library Munich Scientific Network Linux Cluster ~16000 cores German National Supercomputing Centre European Supercomputing Centre Participating in large European e-infrastructures High Performance Computing High Speed Networks Grid/Cloud Computing June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 5

7 Why does LRZ care about Energy? Normalized TCO: Estimated Worldwide Data Center Power Consumpti : 6% Europe Source: DataCenterDynamics Focus, Volume 3, Issue 33, Jan/Feb 2014 June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 6

8 The four pillars of Energy Efficient Data Centers External Influences/Constraints Data Center (Goal: Reduce Total Cost of Ownership) Neighboring Buildings SIMOPEK Advanced Adsorption Cooling CooLMUC SuperMUC SIMOPEK Power Consumption Modeling, Simulation & Optimization using MYNTS SIMOPEK Data Collection using PowerDam V.2.0 PowerDam V.1.0 Global Optimization Strategy Utility Providers Improve PUE (Power Usage Effectivness) Reduce Hardware Power Consumption Optimize Resource Usage Tune System Optimize Performance Pillar 1 Building Infrastructure Pillar 2 HPC System Hardware Pillar 3 HPC System Software Pillar 4 HPC Applications Open Access 4 Pillar Framework Paper: June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 7

9 Leibniz Supercomputing Centre: Dark Center June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 8

10 Sections of the Data Centre Cooling Towers Supercomputer Linux-Cluster, Hosting/Housing, Servers Archive/Backup, Disks, Servers Cooling, Water & Air Processing Power, Transformers, UPS June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 9

11 Air versus Water Cooling June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 10

12 Overview Cooling Infrastructure 11 June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 11

13 LRZ: Warm Water Distribution Infrastructure Pipe to wet cooling tower Pipe from hydraulic sperator Pipe from wet cooling tower Pipe to hydraulic sperator Heat Exchanger KLT14 Pipe to hydraulic sperator Heat Exchanger KLT13 June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 12

14 LRZ Warm Water Cooling Infrastructure 3rd Floor LRZ False Floor with Cooling Pipes June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 13

15 Warm Water Cooling Infrastructure on the Roof: Four 2 MW Cooling Towers June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 14

16 June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 15

17 SuperMUC System

18 SuperMUC System Phase 1 (IBM System x idataplex): 3.2 PFlops peak performance 9216 IBM idataplex dx360m4 nodes in 18 compute node islands 2 Intel Xeon E processors and 32 GB of memory per compute node 147,456 compute cores Network Infiniband FDR10 (fat tree) Phase 2 (Lenovo NeXtScale WCT): 3.6 PFlops peak performance 3072 Lenovo NeXtScale nx360m5 WCT nodes in 6 compute node islands 2 Intel Xeon E5-2697v3 processors and 64 GB of memory per compute node 86,016 compute cores Network Infiniband FDR14 (fat tree) Common GPFS file systems with 10 PB and 5 PB usable storage size respectively Common programming environment Direct warm-water cooled system technology June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 17

19 SuperMUC Phase 1 System Power Consumption: 800 kw (idle) 3700 kw (Linpack) June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 18

20 SuperMUC Phase 2 System Power Consumption: 300 kw (idle) 1500 kw (Linpack) Photo: Torsten Bloth, Lenovo Lenovo NeXtScale Water Cool (WCT) system technology Water inlet temperatures 30 C 45 C All season chiller-less cooling 72 Racks 48 Compute 9 Infiniband 6 In-Row Cooler 9 Management + Storage The Energy Challenge in HPC - 13th HPC Connection Workshop June 21st,

21 SuperMUC Phase 2 Photos: Torsten Bloth, Lenovo High Energy Efficiency Usage of Intel Xeon E5 2697v3 processors Direct liquid cooling 10% power advantage over air cooled system 25% power advantage due to chiller-less cooling Energy-aware scheduling 6% power advantage ~40% power advantage Total annual savings of ~1.8 Mio. for SuperMUC Phase 1 and 2 The Energy Challenge in HPC - 13th HPC Connection Workshop June 21st,

22 Reuse of Waste Heat at LRZ: CooLMUC-1 June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 21

23 Adsorption Chiller June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 22

24 CooLMUC-2 Lenovo NeXtScale Water Cool (WCT) system technology Water inlet temperatures 30 C 50 C All season chiller-less cooling 384 compute nodes 466 TFlop/s peak performance 8 Racks 6 compute 1 Infiniband 1 management June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 23

25 CooLMUC-2: Waste Heat Re-Use for Chilled Water Production Cold capacity of adsorption chillers June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 24

26 CooLMUC-2 Energy Efficiency Ratio ERE and Coefficient of Performance COP (ERE = 0.3) CooLMUC-2 power consumption CooLMUC-2 heat output into warm water cooling loop Cold capacity of adsorption chillers (COP ~ 0,5 0,6) June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 25

27 Total Cooling Performance of CoolMUC-2 June 21st, 2016 The Energy Challenge in HPC - 13th HPC Connection Workshop 26

28 Thank you!