Multi-scale Thermal Management Challenges in Data Abundant Systems

Size: px

Start display at page:

Download "Multi-scale Thermal Management Challenges in Data Abundant Systems"

Richard Norman
6 years ago
Views:

Bakir, A. Fedorov, S. Yalmanchili, K. Schwan, B. Sammakia, A. Ortega, D.

1 Multi-scale Thermal Management Challenges in Data Abundant Systems Yogendra Joshi G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA Collaborators: M. Bakir, A. Fedorov, S. Yalmanchili, K. Schwan, B. Sammakia, A. Ortega, D. Agonafer Sponsors: DARPA, NSF, Sandia National Laboratories 1st International Workshop on Abundant-Data System Technology, Stanford University, April 22-23, /38

2 Outline Multi-scale cooling of IT, and the energy requirements Top 500 cooling strategies Advanced air cooling (Titan) Indirect liquid cooling (K-Computer) Growing energy consumption and Green 500 Hybrid liquid/air cooling (Piz-Daint) Liquid (Tsubame KFC) Growing sustainability focus 3D stacked die as potential enabler for high performance computing Integrated electrical and fluidic I/Os technology Electrical/thermal co-optimization of fluidic cooling 2/38

Multi-scale Nature of Thermal Management

server to ambient heat rejection Similar

3 Multi-scale Nature of Thermal Management in Information Technology Systems ~2 m ~35 mm ~0.6 m ~2 m aisle rack server chip dm m cm mm ~10+ m Room (internal) ~10 m Chiller (external) O(10 5 ) length scales from server to ambient heat rejection Similar variation from onchip interconnects to chip scale Multi-scale time dependence also important

4 Data Center Energy Consumption and Cooling The Koomey Report (N.Y. Times, 2011) In 2000, Data center energy consumption in the U.S. was ~1% of total consumption For the next decade, it doubled every five years The growth rate in the U.S. has slowed somewhat more recently, ascribed to energy efficiency initiatives In the U.S. growing at ~5 billion kwh/yr (1 new 500 MW plant), and worldwide at 3x Energy Consumption for Cooling Power and cooling costs become equal to commodity server costs in 1-3 years LBNL benchmarking (~2003) in the bay area showed 23%-54% of incoming power was used for cooling Subsequent $50M DoE program focused on energy efficiency program focused on energy efficiency improvements Current average PUE thought to be in the range, with some new facilities showcasing significantly better performance 4/38

5 Data Center Energy/Sustainability Metrics PUE = Total energy consumption/it energy consumption Energy Reuse Effectiveness (ERE )= (Total Energy Reuse Energy)/IT Energy With significant energy reuse, a value of ERE below 1 is possible and should be the focus of new data center facilities. Carbon Usage Effectiveness (CUE) expressed in kg CO 2 equivalent/kwh: CUE = Total CO 2 emissions caused by the total data center energy/it Energy Water Usage Effectiveness (WUE) in litres/kwh: WUE = Annual site water usage/ IT energy Water Usage Effectiveness also including the source contribution : WUE source = (Annual energy source water usage + Annual site water usage) / IT energy Presentation by M. K. Patterson at Georgia Tech Workshop on Sustainable High Performance Computing Infrastructure, Feb 2012

6 Top 10 in Top 500 List (November 2013) Rank * Site Computer Manufac. Country 1, 1 National Supercomputer Center Guangzhou 2, 2 Oak Ridge National Lab 3, 3 Lawrence Livermore National Lab 4, 4 RIKEN Advanced Institute for Computational Science 5, 5 Argonne National Lab 6, 42 7, 6 Swiss National Supercomputing Centre Texas Advanced Comp. Center/U. of Texas 8, 7 Forschungszentrum Juelich (FZJ) 9, 8 Lawrence Livermore National Lab 10, 9 Leibniz Rechenzentrum Tianhe-2, Xeon 12C 2.200GHz, Intel Xeon Phi 31S1P Titan, Cray XK7, Opteron C 2.200GHz, Cray Gemini interconnect, Sequoia, BlueGene/Q, Power BQC 16C 1.60 K computer, SPARC64 2.0GHz, Tofu interconnect Mira, BlueGene/Q, Power BQC 16C 1.60GHz, Custom Piz Daint, Cray XC30, Xeon 8C 2.600GHz, Aries interconnect, NVIDIA K20x Stampede, PowerEdge C8220, Xeon 8C 2.7GHz, Intel Xeon Phi JUQUEEN, BlueGene/Q, Power BQC 16C 1.600GHz, Custom Vulcan, BlueGene/Q, Power BQC 16C 1.600GHz, Custom SuperMUC, idataplex DX360M4, Xeon E C 2.70GHz, Infiniband Power (MW) Rmax (PFLOPS) GFLOPS/kW NUDT China Cray Inc. United States IBM United States Fujitsu Japan IBM United States Cray Inc. Switzerland Dell United States IBM Germany IBM United States IBM Germany * Current and Previous Rank 6/38

HOT AISLE PDU Raised-Floor, Air-Cooled Data Center SERVER FAN CPU Temp. Rack Air Flow Rate RETURN PLENUM RACK SERVER COLD AISLE Server Inlet Air Temp. Temp. Field Flow Field RACK HOT AISLE CRAC COOLING COIL HUMIDIFIER ELECTRIC HEATER Outside Air Temp.

Example: VCR chiller with air cooled condenser, Hot-Aisle- Cold-Aisle (HACA) arrangement.

7 HOT AISLE PDU Raised-Floor, Air-Cooled Data Center SERVER FAN CPU Temp. Rack Air Flow Rate RETURN PLENUM RACK SERVER COLD AISLE Server Inlet Air Temp. Temp. Field Flow Field RACK HOT AISLE CRAC COOLING COIL HUMIDIFIER ELECTRIC HEATER Outside Air Temp. ECONOMIZER Return Air Temp. Water Supply Temp. PUMP Water Return Temp. CONDENSER FAN CHILLER COMPRESSOR SUPPLY PLENUM Tile Air Flow Rate CRAC FAN Supply Air Temp. Example: VCR chiller with air cooled condenser, Hot-Aisle- Cold-Aisle (HACA) arrangement. Cooling power consumed for: Chiller pump, compressor and condenser fan Air movement CRAC and server fans, CRAC = computer room air conditioning unit Experiments, Only CRAC 1 7/38

25 25 MW, 6,600 ton cooling facility Air

2 MW cooling capacity Chilled water at 5.

8 Titan (Oak Ridge) 40,000 sq. ft raised floor PUE ~ MW, 6,600 ton cooling facility Air cooled components using chilled air 23.2 MW cooling capacity Chilled water at 5.5 o C chills air itan computer board; 4 AMD Opteron (16-Core CPU) + 4 NVIDIA Tesla K20X GPUs

9 Titan Power Delivery 200 custom 19 Cray XK7 cabinets 480 V power to cabinets 299k-amd-x86-cores-and-186k-nvidia-gpu-cores

10 Cabinet level Liquid Cooling Evolution in High Performance Computing Chilled Water Cooling Free Cooling With Warm Water- No Chiller! The Cray XC Supercomputer Series: Energy-Efficient Computing Greg Pautsch, Duncan Roweth, and Scott Schroeder Cray Inc., November /38

11 The Fujitsu K-Computer

12 But the Power Keeps Increasing

13 Power Efficiency is Inching Up Too Top Gflops/kW 526 MW for Top 1 system to reach 1 Exaflop

14 Rank* Top 10 in Green 500 List (November 2013) Simulate climate change, don t create it Rmax (PFLOPS) Power (kw) Gflops/kW Name Manufacturer Site Processor Family Accelerat or/co- Processor 1, TSUBAME-KFC NEC Tokyo Institute of Technology, Japan Intel IvyBridge NVIDIA K20x 2, Wilkes Dell Cambridge University, U.K. Intel IvyBridge NVIDIA K20 3, HA-PACS TCA Cray Inc. University of Tsukuba, Japan Intel IvyBridge NVIDIA K20x Swiss National 4, Piz Daint Cray Inc. Supercomputing Centre, Switzerland Intel SandyBridge NVIDIA K20x 5, Romeo Bull SA ROMEO HPC Center, France Intel IvyBridge NVIDIA K20x 6, TSUBAME 2.5 NEC/HP Tokyo Institute of Technology, Japan Intel Nehalem NVIDIA K20x 7, IBM University of Arizona, U.S.A. Intel IvyBridge NVIDIA K20x 8, IBM Max-Planck-Gesellschaft MPI/IPP, Germany Intel IvyBridge NVIDIA K20x 9, IBM Financial Institution, U.S.A. Intel IvyBridge NVIDIA K20x 10, CSIRO GPU Cluster Xenon Systems CSIRO, Australia Intel SandyBridge Nvidia K20m * Green 500, Top 500 Heterogeneous systems (using computing brains consisting of CPUs and GPUs) dominate the top 10 spots The No. 1 machine today will require 222 MW to get to ExaFLOP 14/38

15 GFLOPS/kW Green 500 Power Efficiency Evolution U. of Tennessee, 45 kw Combined Commodity Intel CPUs with GPUs IBM Blue Gene/Q, 39 kw Accelerator Based TSUBAME KFC, 28 kw Intel CPUs, NVIDIA GPUs Liquid Immersion Cooled 0 Oct-06 Feb-08 Jul-09 Nov-10 Apr-12 Aug-13 Dec-14 Do the high efficiencies scale for computing architectures? 15/38

16 Piz-Daint (CRAY-XC) The Cray XC Supercomputer Series: Energy-Efficient Computing Greg Pautsch, Duncan Roweth, and Scott Schroeder Cray Inc., November /38

17 Piz-Daint Cooling Transverse Cabinet Cooling Processor Module (Pair of Dual Sockets) The Cray XC Supercomputer Series: Energy- Efficient Computing, Greg Pautsch, Duncan Roweth, and Scott Schroeder Cray Inc., November /38

18 Tsubame KFC: Liquid Immersion Cray-2, 1985 Fluorinert From New York Times, February 11, 2014 The Tsubame KFC in Japan is submerged in a tank of mineral oil

19 S. Matsuoka, Being Very Green with Tsubame 2.5 Towards 3.0 and Beyond to Exascale, 19/38

Smart Air/Water Cooling: Cooling Energy Optimization with Time-varying CPU Workload Hypothesis A data-driven algorithm can improve parametric granularity of measured CPU temperature data with cooling

75m Power distribution unit Downflow CRAC APC In row cooler HP rack HP rack IBM rack IBM Blade Center rack Mixed HP/ IBM rack CEETHERM Data Center Layout 3 5 7 9 11 13 1 2 4 6 10 12 14 8 Upflow CRAC

20 Smart Air/Water Cooling: Cooling Energy Optimization with Time-varying CPU Workload Hypothesis A data-driven algorithm can improve parametric granularity of measured CPU temperature data with cooling resource setpoints as parameters. Thereby, it facilitates controller design to optimize dynamic allocation of cooling resources, using critical CPU temperature as optimization constraint. 8.75m Power distribution unit Downflow CRAC APC In row cooler HP rack HP rack IBM rack IBM Blade Center rack Mixed HP/ IBM rack CEETHERM Data Center Layout Upflow CRAC APC In row cooler Mixed HP/ IBM rack HP rack IBM rack IBM Blade Center rack Server Simulator Upflow CRAC Dell Power edge 64 Node Rack 640 node rack Storage IBM Blade Center IBM Blade Center IBM Blade Center IBM Blade Center Downflow CRAC Spare IBM Blade Center IBM Blade Center IBM Blade Center IBM Blade Center IBM Blade Center IBM Blade Center Power distribution unit Power distribution unit Storage Network D O O R Downflow CRAC Downflow CRAC D O O R 6.4m 6.4m Experimental zone Computational a zone GT Data Center Lab T f x, y, z; Workload, t, T, P CRAC,Supply RDHx i i i i 20 i

21 Dynamic Workload Profiles Amplitude: 25%, Period: 3600 s Amplitude: 75%, Period: 1200 s Amplitude: 25%, Period: 1200 s Amplitude: 25%, Period: 3600 s Amplitude: 95% * J. Liu and A. Terzis, "Sensing data centres for energy efficiency," Phil. Trans. R. Soc , /38

22 Controller to Optimize Cooling Cost 22 Rajat Ghosh, Ph.D. Dissertation, Georgia Tech, 2013

23 Optimal Cooling for Type-1 Load Peak Half Period Flash Crowd 23/38

24 Big Data for Big Energy Savings 51.4% average (rms) saving air E C T out 18.5% average (rms) saving E P /38

25 Current Technology for Waste Heat Recovery Smarter Data Centres-Enabling Business Change and Innovation, Dr. Bruno Michel, Manager Advanced Thermal Packaging, IBM Zurich Research Laboratory 04 May 2012

26 New Georgia Tech HPC Facility Slide courtesy Ron Hutchins, CTO Georgia Tech Multi-disciplinary/cross-disciplinary living laboratory combining high performance computing research data center and office tower spanning a city block; targeted completion ~ /38

27 Conclusions Air cooling in data centers is currently ubiquitous, and is likely to stay for lower density facilities, and portions of high density facilities Higher density facilities have shown significant improvements in energy efficiency through incorporation of indirect liquid cooling, and significantly more by direct liquid immersion Advanced cooling technologies will have to be used to get to exa-flop and beyond Some of these technologies may subsequently migrate to lower power density facilities Co-optimization of IT resources and cooling are critical to get the best energy efficiency. 27/38

28 Can 3D Stacked Die Electronics Help Future High Performance Computing? 1. System Throughput 3D eliminates low-bandwidth off-chip links that stall benefits of processor throughput Enables high-throughput architectures 2. System Power Reduces system capacitance, losses, and power in signaling: on-chip wires=50-70% total chip capacitance Today s off-chip links: mw/gbps ~20% less 3D: <1 mw/gbps chip pwr 3. Heterogeneous Integration Provide monolithic like performance for photonics, MEMS, sensors, non-volatile memory, etc with CMOS 4. System Form factor, Cost, Yield, and Density Reduce chip size, which improves yield and cost Provides a new way to increase device density Slide courtesy Prof. M. Bakir

Impact of Fluidic Cooling on Power Dissipation (1) Memory(DRAM) Memory(SRAM) Processor(Logic) Sekar et al, IITC 2008 1 P ( T T ambient ) R thermal (2) total dd gates dd leak0 2 P ac V T f N V T I e V

29 Impact of Fluidic Cooling on Power Dissipation (1) Memory(DRAM) Memory(SRAM) Processor(Logic) Sekar et al, IITC P ( T T ambient ) R thermal (2) total dd gates dd leak0 2 P ac V T f N V T I e V t ( T ) V nk T/q Freq. Power Temp. Air cooling: ~0.6 C/W 3 GHz 102 W 88 o C Single Phase Liquid Cooling: ~0.25 C/W L. Zheng, et al., Novel Electrical and Fluidic Microbumps for Silicon Interposer and 3-D ICs, TCPMT, GHz 83 W 47 o C Thermal ancillary technologies are critical to minimizing power dissipation and increasing reliability and performance t Slide courtesy Prof. M. Bakir

Several teams, including Stanford, IBM S. Isaacs, Y. Zhang, Y. J. Kim, Y. Joshi, M.

30 Two-Phase Fluidic Cooling for Higher Heat Fluxes Flow Direction 4 heaters Triangular wakes display 2-D spreading feature Advantage in flow stability compared to microchannels Rapid and periodic nucleation and departure Current DARPA program; Several teams, including Stanford, IBM S. Isaacs, Y. Zhang, Y. J. Kim, Y. Joshi, M. Bakir, Two-Phase Flow and Heat Transfer in Pin-Fin Enhanced Micro-Gaps with Non-Uniform Heating, Micro-Nano Heat Mass Transfer 2013, The University of Hong Kong, December 2013.

31 Co-Optimization of 3D Stacked ICs with Fluidic Cooling Initial Structure Natural Convection,, h amb =10 W/m 2 K Simplified Structure Natural Convection, h amb =10 W/m 2 K Inlet Tier 2 Outlet Inlet Tier 2 Active layer Microbump Tier 1 SiO 2 & Metal layer Tier 1 SiO 2 & Metal layer Outlet Interposer BT substrate Effective Convection, h eff PCB Natural Convection, h amb =10 W/m 2 K For simplification, an effective heat transfer coefficient is applied at the bottom of SiO 2 & Metal layer. Z. Wan, H. Xiao, Y. Joshi, S. Yalamanchili, Electrical/Thermal Co-Design of Multi-Core Architectures and Microfluidic Cooling for 3D Stacked ICs, Proc. of 19th Int. Workshop on Thermal Investigations of ICs and Systems, 2013, pp H. Xiao, Z. Wan, S. Yalamanchili, Y. Joshi, Leakage Power Characterization and Minimization in 3D Stacked Multi-core Chips with Microfluidic Cooling, Proc. of 30th SEMI-THERM Symposium, /38

32 Application Binaries Coupled Power-Thermal Simulation Cycle-based Simulator with Timing Model Counter Structure Floorplan for processor Power Traces Power Library Analysis Model Floor Plan Floorplan for L2 cache Thermal Map Thermal Library Analysis Model 1. Power model simulations are run for 500M clock cycles to warm up the processor state and reach a region of interest. Then the power for each block is obtained as the dynamic power. 2. The dynamic power is input to the compact thermal model. Leakage power is updated after every iteration. 32/38

33 Realistic Power Map Chip power Dynamic power Leakage power P d 1 2 f 2 scv dd P s I leak T, V dd V dd where C is the switching capacitance, V dd is the supply voltage, f s is the switching factor per clock cycle, I leak is the leakage current, T is the average temperature of the block. Dynamic power for every block FPU (W) INT (W) DL1 (W) SC (W) FE (W) L2 (W) Core Core Other cores FPU (float-point unit), INT (integer unit), DL1 (L1 data cache), SC(Out-of-Order scheduler), FE (pipeline frontend and L1 instruction cache) V dd = 0.8V, f s =3GHz; Maximum heat flux: 301 W/cm 2 Leakage power VS Temperature Leakage power increases exponentially with temperature. 33/38

34 Workload Driven Cooling Solution Optimization Constraint Fixed pumping power Objectives Minimize the junction temperature and thermal resistance Determine Cooling System Design, e.g., pin fin geometries, flow rate, etc. Optimization algorithm: Genetic algorithm is used to do the optimization. The compact model is embedded into a genetic algorithm as a function. After optimization, the optimized pin fin dimensions are obtained. Objectives Compact thermal model Genetic algorithm Optimized pin fin dimensions 34/38

35 Optimization Results D p : Pin fin diameter S L : Longitudinal spacing S T : Transversal spacing H p : Pin fin height m : mass flow rate P: Pressure drop Pumping power:0.03 W Optimization for non uniform power map. D p (mm) S L (mm) S T (mm) H p (mm) m (g/s) P (kpa) T max,logic ( ) (optimized) Configuration of two microgaps with Processor at tier 2 is used. Pump power is 0.03 W. Dimensions of 1, 2, 3 are selected from existing literature. The maximum temperature is reduced to Larger pin fin dimensions produce larger mass flow rate and lower pressure drop. The larger mass flow rate could reduce the bulk fluid temperature rise at the outlet. 35/38

03 W D p (mm) S L (mm) S T (mm) H p (mm) m (g/s) P (kpa) T max,logic ( ) 1 100 200 200 200 1.6 18.2 149.

36 Hotspot Optimization Results Processor L2 cache Optimization for non uniform power map with hotspot; Core 7 memory module power increased to 10x Pumping power:0.03 W D p (mm) S L (mm) S T (mm) H p (mm) m (g/s) P (kpa) T max,logic ( ) (optimized) /38

37 Architecture Performance Given that optimized pin fin structure provides best 4 pin fin structures thermal dissipation, its 1.2 Energy Per Instruction 0.9 (EPI) has the lowest value, 0.6 indicating that the energy 0.3 saved throughout execution 0 is 40% over the worst case Predicted leakage power of the 16 core microarchitecture under different pin fin organizations: Normalized EPI comparison among all Processor(W) L2 cache (W) Total leakage (W) (optimized) /38

38 Conclusions 3D Inter-tier microfluidic cooling shows significant performance enhancement compared to air cooling. Increasing pumping power could reduce both the maximum temperature and leakage power. Two tiers and two microgaps with the high power dissipation tier under double side cooling shows best thermal performance. Optimum pin fin configuration depends on the power map. For non-uniform power dissipation without hotspot, large pin fin dimensions which produce large mass flow rate are better than small pin fin dimensions. For non-uniform power dissipation with hotspots, small pin fin dimensions which produce larger convection surface area are better than large pin fin dimensions. 38/38

It s a Multicore World. John Urbanic Pittsburgh Supercomputing Center

It s a Multicore World. John Urbanic Pittsburgh Supercomputing Center It s a Multicore World John Urbanic Pittsburgh Supercomputing Center Waiting for Moore s Law to save your serial code start getting bleak in 2004 Source: published SPECInt data Moore s Law is not at all