Energy Efficiency for Exascale. High Performance Computing, Grids and Clouds International Workshop Cetraro Italy June 27, 2012 Natalie Bates

Transcription

1 Energy Efficiency for Exascale High Performance Computing, Grids and Clouds International Workshop Cetraro Italy June 27, 2012 Natalie Bates

2 The Exascale Power Challenge Where do we get a 1000x improvement in performance with only a 10x increase in power? How do you achieve this in 10 years with a finite development budget? 2

3 How to address this challenge? You can only improve what you can measure. but, this is easier said than done.

4 High Performance Computing, Energy Efficiency and Sustainability Compute System Energy Efficiency Sustainability Data Center Infrastructure

5 Energy Efficiency Roadmap to Exascale Data Center Thermal Pods Free Best Liquid Cooling Practices Cooling Location Compute System Processors: low power designs Interconnect aware: on + off chip Data locality mgmt on chip Memory: photonics and 3D Interconnects Si photonics Energy Efficiency Tools Si Power Features Data Center Middleware Instrumentation Sys admin tools System App Tuning Instrumentation Metrics PUE Flops/ MWatt ERE CUE Power Methodology TUE GUPS/ MWatt Sub-system/ MWatt 2006 Exascale Demonstration

6 Energy Efficiency Roadmap to Exascale Data Center Thermal Pods Free Best Liquid Cooling Practices Cooling Location Compute System Processors: low power designs Interconnect aware: on + off chip Data locality mgmt on chip Memory: photonics and 3D Interconnects Si photonics Energy Efficiency Tools Metrics Si Power Features PUE Flops/ MWatt Data Center Middleware Instrumentation Sys admin tools System App Tuning Instrumentation ERE CUE Power Methodology TUE GUPS/ MWatt Sus-system/ MWatt 2006 Exascale Demonstration

7 A quick word Energy Efficient HPC Working Group Driving energy conservation measures and energy efficient design in HPC Forum for sharing of information (peerto-peer exchange) and collective action Open to all interested parties EE HPC WG Website Energy Efficient HPC Linked-in Group With a lot of support from Lawrence Berkeley National Laboratory

8 Agenda We know how to build efficient data centers Power Usage Effectiveness (PUE): its all about the 1 Refining the metric - TUE But we can go beyond energy efficiency Energy re-use and carbon emissions ERE and CUE Instrumentating for energy/power Data center and system power management Energy/power measurement methodology Current best practices Location, location, location Cheap, renewable energy

9 Power Usage Effectiveness (PUE) simple and effective

10 PUEs: Reported and Calculated PUE EPA Energy Star Average reported in Intel Jones Farm, Hillsboro 1.41 ORNL CSB 1.25 T Systems & Intel DC2020 Test Lab, Munich 1.24 Google 1.16 Leibniz Supercomputing Centre (LRZ) 1.15 National It s all Center about for the Atmospheric 1. Research (NCAR) 1.10 Yahoo, Lockport 1.06? Should we Facebook, focus on Prineville driving the 0.06 down? 1.07 Or work on the 1.0? National Renewable Energy Laboratory (NREL) 1.06

11 Refining PUE for better comparison - TotalPUE PUE does not account for cooling and power distribution losses inside the compute system ITPUE captures support inefficiencies in fans, liquid cooling, power supplies, etc. TUE provides true ratio of total energy, (including internal and external support energy uses) TUE preferred metric for inter-site comparison EE HPC WG Sub-team proposal

12 Combine PUE and ITUE for TUE

13 Agenda We know how to build efficient data centers Power Usage Effectiveness (PUE): its all about the 1 Refining the metric - TUE But we can go beyond energy efficiency Energy re-use and carbon emissions ERE and CUE Instrumentating for energy/power Data center and system power management Energy/power measurement methodology Current best practices Location, location, location Cheap, renewable energy

14 I am re-using waste heat from my data center on another part of my site and my PUE is 0.8!

15 I am reusing waste heat from my data center on another part of my site and my PUE is 0.8! While re-using excess energy from the data center can be a good thing to do, it should not be rolled into PUE. The definition of PUE does not allow this. There is a new metric to do this; ERE

16 ERE adds energy reuse to the PUE concept Rejected Energy (a) Cooling (f) Reused Energy Utility (e) IT (g) (b) UPS (c) PDU (d)

17 PUE & ERE resorted. PUE EPA Energy Star Average 1.91 Intel Jones Farm, Hillsboro 1.41 T Systems & Intel DC2020 Test Lab, Munich 1.24 Google 1.16 NCAR 1.10 Yahoo, Lockport 1.08 Facebook, Prineville 1.07 Energy Reuse Leibniz Supercomputing Centre (LRZ) 1.15 ERE <1.0 National Renewable Energy Laboratory (NREL) 1.06 ERE <1.0

18 Carbon Usage Effectiveness (CUE) Ideal value is 0.0 Example, the Nordic HPC Data Center in Iceland is powered by renewable energy CUE ~ 0.0

19 Agenda We know how to build efficient data centers Power Usage Effectiveness (PUE): its all about the 1 Refining the metric - TUE But we can go beyond energy efficiency Energy re-use and carbon emissions ERE and CUE Instrumentating for energy/power Data center and system power management Energy/power measurement methodology Current best practices Location, location, location Cheap, renewable energy

20 Lawrence Livermore National Laboratory Case Study Create an operational, event, and real-time facility power management infrastructure of all external and internal data sources ALC Goal = Lower power utilization and coordinate facility power management to HPC system power management initiatives SGI Ice Cube Environmental Monitoring Building Management System Nexus Computational Power Submetering Evaluating LLNL site load flow is crucial for exascale Coordinate future data intensive runs with entire site operations Skummee HPC SNMP Information Siemens SCADA Electric Utility Data Data Sources SLURM HPC Resource Manager LLNL Weather & Area Disaster Information MV90/ EEM Sitewide Metering Forseer Environmental Monitoring

21 Agenda We know how to build efficient data centers Power Usage Effectiveness (PUE): its all about the 1 Refining the metric - TUE But we can go beyond energy efficiency Energy re-use and carbon emissions ERE and CUE Instrumentating for energy/power Data center and system power management Energy/power measurement methodology Current best practices Location, location, location Cheap, renewable energy

22 Methodology complexities and issues Fuzzy lines between the computer system and the data center, e.g., fans, cooling systems Shared resources, e.g., storage and networking Data center not instrumented for computer system level measurement Measurement tool limitations, e.g., frequency, power verses energy dc system level measurements don t include power supply losses

23 Proposed Methodology Improvements Current power measurement methodology is very flexible, but compromises consistency between submissions Proposal is to keep flexibility, but keep track of rules used and quality of power measurement 3 Levels of power measurement quality Sampling rate; more measurements means higher quality Completeness of what is being measured; more of the system translates to higher quality Common rules for start/stop times Vision is to continuously raise the bar with higher levels

24 dc Measurement Methodology (Current Proposed) requires reporting ac to dc conversion losses L1 = data supplied by the power supply manufacturer L2 = loss model based on 2-point measurements done within prior 2 years L3 = loss model based on 4-point measurements done within prior 1 year L3 total energy measurement dc voltage and current measured at least once per ac period (100 or 120 times per second, depending on the country)

25 Power/Energy Measurement Methodology (Current Proposed) Level Aspect 1: Time Fraction & Granularity 1 20% of run Power measurement > 1 per second Report > 1 avg. power measurement 2 100% of run Power measurement > 1 per second Report > 10 avg. power measurements 3 100% of run Total integrated energy measurement Report > 10 running total energy measurements Aspect 2: Machine Fraction (larger of) 1/64 of machine or 1kW (Larger of) 1/8 of machine or 10kW Whole machine Aspect 3: Subsystems Measured [Y] Compute nodes [ ] Interconnect net [ ]Storage [ ]Storage Network [ ]Login/Head nodes

26 Testing proposed improvements: early adopter phase 5 early adopters to use methodology for June 12 submissions to Top500 and Green500 Lawrence Livermore National Laboratory Leibniz Supercomputing Center Oak Ridge National Laboratory Argonne National Laboratory Université Laval, Calcul Québec, Compute Canada Seeking feedback Alpha June 12 List Beta November 12 List Next revision expected by September 2012

27 Agenda We know how to build efficient data centers Power Usage Effectiveness (PUE): its all about the 1 Refining the metric - TUE But we can go beyond energy efficiency Energy re-use and carbon emissions ERE and CUE Instrumentating for energy/power Data center and system power management Energy/power measurement methodology Current best practices Location, location, location Cheap, renewable energy

28 Location, location, location 75% of Top500 Supercomputers are located in three countries

29 Shift from energy to cost and carbon as an upper bound First and foremost, drive system design for energy efficiency low power processor designs, on and off chip interconnect awareness, data locality management, memory photonics and 3D stacking, photonic interconnects Build a data center with PUE ~ 1 and ERE << 1 Use renewable energy sources only Site where electricity cost is ~$0.03 kwatt/hour Which is better? A 20MW, $20M B 60MW, $20M, zero carbon footprint

30 Thank you!