Energy Efficiency and Water-Cool-Technology Innovations

Transcription

1 Energy Efficiency and Water-Cool-Technology Innovations Karsten Kutzer April 10th 2018 Swiss Conference 2018 Acknowledgments: Luigi Brochard, Vinod Kamath, Martin Hiegl (Lenovo) Julita Corbalan (BSC)

2 Why care about Power and Cooling? Data Center limitations Increasing Electricy Cost Performance- Power relation Application Diversity Waste Heat Reuse Leading the Industry in Energy Aware HPC 2

3 Performance-Power relation NVIDIA / AMD GPU ARM SOC XEON AMD PHI NERVANA/CREST NVIDIA SXM Maintaining Moore s Law with increased competition is resulting in higher component power Increased memory count, NVMe adoption, and I/O requirements are driving packaging and feature tradeoffs (superset of features doesn t fit in 1U) Shared cooling fan power savings no longer exist for dense 2S nodes architectures due to nonspreadcore CPU layout high airflow requirements Intel Xeon Processor & Spec_fp Rate Sandy Bridge / IvyBridge Haswell For highest performance systems will have to reduce density or move to optimized cooling TDP CFP2006 Rate 3

4 Application Diversity CPU bound BQCD case Memory bound BQCD case Turbo ON: 157 GFlops DC node[w] CPU pkg 0 [W] RAM pkg 0 [W] Turbo ON: 65 Gflops DC node[w] CPU pkg 0 [W] RAM pkg 0 [W] CPU pkg 1 [W] RAM pkg 1 [W] CPU pkg 1 [W] RAM pkg 1 [W] Node runs on full Power CPU provides full performance while running at full power Node still runs on full Power CPU provides less performance while still running at full power How much energy do we waste on non-cpu bound application? SD650 with 2 sockets 8168 and 6 x 16GB DIMMs; room temp = 21 C, inlet water = 45 C, 1.5 lpm/tray 4

5 Waste Energy reuse - ERE Energy Waste Direct Reuse Indirect Reuse Pictures: Leibniz Supercomputing Centre How much energy do we waste by not using the system heat? 5

6 Energy Aware HPC Best CPU choice with max TDP supported Best performance fully utilizing the system Best TCO / Performance for maximized ROI Best use of limited DataCenter capacities Best Carbon Footprint for eco responsible HPC 6

7 The three Pillars Hardware Software Infrastructure Leading the Industry in Energy Aware HPC 7

8 Hardware

9 Water Cooling Technologies Direct Water Cooling 9

10 Lenovo Cooling Technologies Air Cooled Air Cooled w/ Rear Door Heat Exch. Direct Water Cooled Standard Air flow with internal fans cooled with the room climatization Broadest choice of configurable options supported Relatively inefficient cooling PUE ~ ERE ~ Choose for broadest choice of customizable options Air cooled but heat removed with RDHX through chilled water Retains high flexibility Enables extremely tight rack placement Potentially room neutral PUE ~ ERE ~ Choose for increased energy efficiency with broad choice Most heat removed by onboard-waterloop with up to 50 C temperature Supports highest TDP CPU at densest footprint Higher performance Free cooling PUE <=1.1 ERE <=1.1 Choose for max performance and high energy efficiency 10

11 Return on Investment for DWC vs RDHx New data centers: Water cooling has immediate payback. DWC Existing air-cooled data center payback period strongly depends on electricity rate $0.06/kWh $0.12/kWh $0.20/kWh RDHx 11

12 Rear Door Heat Exchanger Up to 27 C Cold Water Cooling Up to 100% Heat Removal Efficiency on 30kW No moving parts or power required Tenthousands of nodes install base Long ago

13 Lenovo Rear Door Heat Exchanger Feature Air Movement Heat removal Water temperature Water Volume Water Flow Rate Door Dimensions Door Assembly Weight Connection RDHx times more efficient than cold air Provided by the systems in the rack At 18 o C Water temp, 27 o C inlet air temp: 100% for 30kW; 90% for 40kW Min 18 C / 64.4 F for ASHRAE Class 1 Min 22 C / 71.6 F for ASHRAE Class 2 Max 27 C 9 Liters / 2.4 Gallons Min 22.7 liters / 6 gallons per minute Max 56.8 liters / 15 gallons per minute Depth: 129mm/5in. Height: 1950mm/76.8in. Width: 600mm/23.6in. Empty: 39kg/85lbs Filled 48kg/105lbs ¾ inch quick connect (Supply: Parker SH6-63W; Return: Parker SH6-62-W; or equivalent) Torsten Bloth 13

14 Lenovo RDHx2 Typical Environment 14

15 Direct Hot Watercooling Up to 50 C Hot Water Cooling Up to 90% Heat Removal Efficiency World Record Energy Reuse Efficiency 30+ patents on market leading design > Lenovo nodes - All rights globally reserved. 15

16 Lenovo ThinkSystem SD650 Feature Processors Form factor Memory Slots Max Memory Storage NIC PCIe Power SD650 2 Intel Purley Generation processors per node Socket-F for Intel Omnipath supported >120W all Skylake Shelves supported 1U Full wide tray double-node / 6U12N Chassis 12x DDR4 (R/LR) 2667MHz DIMM 4x Intel Apache Pass DIMM ready 2x SATA slim SSD / 1x NVMe, 2x M.2 SATA SSD 1x 1 GBaseT, 1x 1 GbE XCC dedicated 1x x16 PCIe for EDR Infiniband / OPA100 1x x16 ML2 for 10Gbit Ethernet (in place of Storage) 1300W/1500W/2000W Platinum and 1300W Titanium USB ports Up to 1x front via dongle cable + 1x internal (2.0) Cooling System MGMT / TPM Dimensions No fans on chassis, PSU fans only Up to 50 C warm water circulated through cooling tubes for component level cooling XCC, dedicated port or shared TPM, Pluggable TCM 915mm depth, front access w/ front I/O Torsten Bloth 16

17 ThinkSystem SD650 Top-Down View Water Outlet 60 C Power Board CPUs 6 DIMMs per CPU 2 AEP per CPU x16 PCIe Slot Disk Drive M.2 Slot 50 C Water Inlet *) two nodes sharing a tray and a waterloop *) inlet water temperature 50 C with configuration limitations (45 C without configuration limitations) 17

18 SD650 Improved Node Water Cooling Architecture Focus on maximizing efficiency for high (up to 50 C) inlet water temperatures Device cooling optimization by minimizing water to device temperature differences dt CPU < ~0.1 K / W dt Memory < ~1 K / W dt Network < ~1 K / W Direct water cooling of processors, memory, voltage regulation devices and IO devices (Network and Disk) Memory Water chanels Water circuit traverses all critical components to optimize cooling. DISK Conductive plate 18

19 HPL Temperature & Frequency on SD650 with 8168 Non AVX instructions AVX instructions Non AVX instructions PL2 (short term RAPL limit) is 1.2 x TDP PL1 (long term RAPL limit) is TDP SD650 with 2 sockets 8168 and 12 x 16GB DIMMs; room temp = 21 C, inlet water = 40 C, 1.5 lpm/tray 19

20 Performance Optimization ThinkSystem SD530 Standard Performance ~ 2.15 TeraFlop/s sustained HPL w/ SKL C 2.4Ghz 150W HPC [GF] AC node DC node CPU Temp Turbo OFF Turbo ON /s sustained HPL w/ SKL C 2.4Ghz 150W Turbo OFF Turbo ON ThinkSystem SD650 High Performance Mode ~ 2.34 TeraFlop/s sustained HPL w/ SKL C 2.4Ghz 150W +9% +18% Turbo OFF Turbo ON SD530 and SD650 with 2 sockets 6148 and 12 x 16GB DIMMs; room temp = 21 C, inlet water = 18 C, 1.5 lpm/tray 20

21 Software

22 Becoming Energy Aware REPORTING MANAGING 22

23 SD650 DC Power Sampling/Reporting Frequency AC power at chassis level (through FPC) With xcat With ipmi DC power and energy at node level through XCC With hw_usage library With ipmi With RAPL With Allinea With LSF or LEAR New for Lenovo ThinkSystem SD650 XCC/BMC FPGA HSC node power Sensor 100Hz 100Hz 100Hz 10KHz XCC/BMC NM/ME HSC Sensor High Level Software 1Hz 1Hz 10Hz 1KHz 200Hz RAPL CPU/memory (energy MSRs) Meter 500Hz 23

24 SD650 advanced Accuracy for Power and Energy Node DC Power readings Better than or equal to +/-3% power reading accuracy down to the node s minimum active power (~40-50W DC). Power granularity <=100mW At least 100Hz update rate for node power readings Node DC Energy meter Accumulator for Energy in Joules (~10 weeks until meter overflow) High accuracy, fast sampling ipmi raw oem cmd Maintains compatibility with Node Manager INA226 (used for metering) FPGA (FIFO) XCC ME (Node Manager) SN (used for capping) Bulk 12V Rsense Node 12V 24

25 25

26 Energy Aware Run time: Motivation Power and Energy has become a critical constraint for HPC systems Performance and Power consumption of parallel applications depends on: Architectural parameters Runtime node configuration Application characteristics Input data Configure application for Architecture X Manual best frequency Difficult to select manually and it is a time consuming process (resources and then power) and not reusable It may change along time It may change between nodes Select optimal frequency Execute with N frequencies: calculate time and energy 26

27 EAR Automatic and Dynamic CPU Frequency Architecture characterization Application characterization Outer loop detection (DPD) Application signature computation (CPI,GBS,POWER,TIME) Performance and power projection Users/System policy definition for frequency selection (with thresholds) MINIMIZE_ENERGY_TO_SOLUTION - Goal: To save energy by reducing frequency (with potential performance degradation) - We limit the performance degradation with a MAX_PERFORMANCE_DEGRADATION threshold MINIMIZE_TIME_TO_SOLUTION - Goal: To reduce time by increasing frequency (with potential energy increase) - We use a MIN_PERFORMANCE_EFFICIENCY_GAIN threshold to avoid that application that do not scale with frequency to consume more energy for nothing 27

28 read EAR Functional Overview Learning Phase (at EAR installation*) Kernel Execution Coefficients Computation * or every time cluster configuration is modified (more memory per node, new processors...) Coeffcients Database Execution Phase (loaded with application) CPU Frequency Dynamic Patter Detection detects outer loop Compute power and performance metrics for outer loop Optimal frequency calculation Energy Policy 28

29 BQCD_CPU with EAR MIN_ENERGY_TO_SOLUTION BQCD_CPU: Outer loop size detected(mpi rank 0) BQCD_CPU: Outer loop size detected(mpi rank 8) Big loop detected Acuumulated me Acuumulated me M P I Frequency BQCD_CPU:Frequency(mpi rank 0) Accumulated me Policy is applied F: 2.6Ghz 2.4Ghz Frequency BQCD_CPU:Frequency(mpi rank 8) Accumulated me M P I R A N K 0 Avg.Power (W) Itera on me (usecs) BQCD_CPU:Measured POWER(mpi rank 0) Accumulated me BQCD_CPU:Measured Itera on me(mpi rank 0) Accumulated me Power is reduced Iteration time is similar Avg.Power (W) Itera on me (usecs) BQCD_CPU:Measured POWER(mpi rank 8) Accumulated me BQCD_CPU:Measured Itera on me(mpi rank 8) Accumulated me R A N K 8 29

30 Infrastructure

31 PUE, ITUE, TUE and ERE Power Usage Effectiveness (PUE) says how much power a datacenter uses is not used for computing. It is the ratio of total power to the power delivered to computing equipment. It does not take into account how effective a server uses the Power it gets. Ideal value is 1.0 IT Usage Effectiveness (ITUE) measures how much power a system uses is not used for computing. It is the ratio of the power of IT equipment to the power of the computing components. Multiplied with the PUE it gives the Total-Power Usage Effectiveness (TUE) Ideal value is 1.0 PUE = ITUE = Total Facility Power Total IT Power Total IT Power Total Compute Power Energy Reuse Effectiveness (ERE) integrates the reuse of the power dissipated by the computer. It is the ratio of total power considering also reuse to the power delivered to computing equipment. An ideal ERE is 0.0. If no reuse, ERE = PUE ERE = (Total Facility Power Total Reuse Power) Total IT Power 31

32 Lenovo Cooling Technologies Air Cooled Air Cooled w/ Rear Door Heat Exch. Direct Water Cooled Direct Water Cooled w/ Adsorption Chilling Standard Air flow with internal fans cooled with the room climatization Air cooled but heat removed with RDHX through chilled water Most heat removed by onboard-waterloop with up to 50 C temperature Waste heat reused to generate coldness to cool non-dwc components Broadest choice of configurable options supported Relatively inefficient cooling PUE ~ ERE ~ Retains high flexibility Enables extremely tight rack placement Potentially room neutral PUE ~ ERE ~ Supports highest TDP CPU at densest footprint Higher performance Free cooling PUE <=1.1 ERE <=1.1 Retains highest TDP, footprint and performance Potentially all system heat covered through DWC PUE <=1.1 ERE <1 Choose for broadest choice of customizable options Choose for increased energy efficiency with broad choice Choose for max performance and high energy efficiency Choose for max performance and max energy efficiency 32

33 Value of Direct Water Cooling with Free Cooling Energy Savings Reduced noise level in the DataCenter Reduced server power consumption Lower processor power consumption (~ 5%) No fan per node (~ 4%) Reduce cooling power consumption At 45 C free cooling all year long ( ~ 25%) Energy Aware Scheduling Only CPU bound jobs get max frequency (~ 5%) 35-40% Total Saving CAPEX Savings Less conventional chillers for the Computing System 33

34 Adsorption Chilling The method of using solid materials for cooling via evaporation. Adsorption Cooling Water to Hybrid Cooling Tower 26 /32 C Evaporation Chilled Water to Storage etc. Racks 23 /20 C Desorption Hot Water from Compute Racks 52 /46 C Condensation Cooling Water to Hybrid Cooling Tower 26 /32 C Adsorption chiller consists of two identical vacuum containers, each containing two heat exchangers and water. Adsorber (Desorber) Coated with the adsorbent (e.g. zeolite) Evaporator (Condenser) Evaporation and condensation of water Adsorption process has 2 phases in the adsorption phase the water on the evaporator is taken in by the coated material in the adsorber. Through that evaporation the evaporater and the water flowing through it does cool down while the adsorber fills with water vapor and heats up the water flowing through it. When the adsorber is saturated the process is reversed. Adsorber Module 1 Evaporater Desorber Condenser Module 2 in the desorption phase hot water is passed through the adsorber acting as a desorber rather as its desorbing the water vapor and dispensing it to the evaporator which is acting as condenser at that point condensing the vapor back to water. Again the process is reversed when the adsorber is emptied. 34

35 Value of Direct Water Cooling with Adsorption Chiller Energy Savings Reduced noise level in the DataCenter Maximum TDP CPU Choice Reduced server power consumption Lower processor power consumption (~ 5%) No fan per node (~ 4%) Reduce cooling power consumption At 50 C free cooling all year long (~ 25%) Heat Reuse generate 600kW cooling capacity (> 5%) 40-50% Total Saving Energy Aware Runtime Frequency optimization during runtime (~ 5%) CAPEX Savings Less conventional chillers for the Computing System 35

36