3D MPSoCs with Active Cooling

Size: px

Start display at page:

Download "3D MPSoCs with Active Cooling"

Abraham Bishop
5 years ago
Views:

1 System-Level Thermal Management of 3D MPSoCs with Active Cooling Prof. David Atienza, Embedded Systems Laboratory (ESL), Ecole Polytechnique Fédérale de Lausanne (EPFL) MPSoC 11, July 4 th 8 th 2011 (Beaune, France) Acknowledgements: IBM Zürich, LSM and LTCM-EPFL, ECE-Boston University

expensive, we try to get denser infrastructures

Dyer, ITHERM 2006 45% of energy overhead in

densities (with lower cooling Reed, 2009 costs)?

2 The Big Picture for Energy Efficiency: Large-Scale Computing in Datacenters Area is expensive, we try to get denser infrastructures New containers: 2500 servers each, >10x density Dyer, ITHERM % of energy overhead in cooling, how to get higher computational densities (with lower cooling Reed, 2009 costs)? Air-cooled datacenters are very inefficient Cooling needs as much energy as IT and thrown-away For a 10MW datacenter ~US$ 4M wasted per year 2 Datacenter energy overhead, ASHRAE

3 Promises Advantages of 3D vs. 2D Multi-Processor System-on-Chip (MPSoCs) ICs Reduce average length of on-chip global wires Increase number of devices reachable in given time budget Greatly facilitate massive storage integration (i.e., logic-ram stacks) A4 X-section CPU RAM RAM Three-Dimensional Multi-Processor System-on-Chip (MPSoC) IC Concept 3

Run-Time Heat Spreading in 3D MPSoCs: More Complex Cooling Needs! 5-tier 3D stack: 10 heat sources and sensors 10000 9000 8000 7000 Layer 2 420 415 Inject between 4W 1.

2010] width (um) 5000 4000 3000 400 398 2000 1000 396 h (um) width 10000 9000 8000 7000 6000 5000 4000 3000 2000 1000 5 th Tier Layer 5 0 0 2000 4000 6000 8000 10000 length (um) 397 396 395 394 393 4

4 Run-Time Heat Spreading in 3D MPSoCs: More Complex Cooling Needs! 5-tier 3D stack: 10 heat sources and sensors Layer Inject between 4W 1.5W width (um) length (um) Layer nd Tier [Garcia and Atienza, Microelectornics Journal 2010] width (um) h (um) width th Tier Layer length (um) wid dth (um) Layer length (um) length (um) 3 rd Tier Large and non-uniform th Tier heat propagation! (up to 130º C on top tier) 4 394

5 Energy-Centric Design for Datacenters: System-Level View Multi-scale energy management solution needed Architecture SW: middleware, power management, network design, etc. Energy-proportional datacenters: t Integrated t layers of software-hardware to maximize Mflops/Watt Technology HW: cooling, processing and storage systems, etc. Barroso & Hölzle,

6 Zero-Emission Datacenter: Liquid Cooling Technology and Predictive Energy Management Datacenters are intelligent heaters 30-40% of carbon footprint in Europe using district heating networks Direct re-use of heat output 3D MPSoC architectures Aquasar Hot -liquid datacenter outlet temperature server: >70 C, 80% payback large CO 2 of -emission electricity reduction costs ºC Water conditioning Water 60ºC Courtesy: IBM Zürich 6

7 NanoTera CMOSAIC Project: Design of 3D MPSoCs with Advanced Cooling 3D systems require novel electro-thermal co-design Academic partners: EPFL and ETHZ Industrial: IBM Zürich and T.J. Watson 3D MPSoC datacenter chip: microchannels etched on back side to circulate (controlled) liquid coolant adjustment of coolant flux task scheduling and power management System Level Active Cooling Manager (fast 3D heat flow prediction) 7

8 Creating a Fast Thermal Model: Compact RC-Based Stack Model with TSVs Chip-Level thermal model q bj 6 b f 1 fj T f b 0 j q RC Network of q b_top q b_back Si/metal layer cells q b_left q b_front q q b_right q b_bottom Convective boundary conditions between layers for each tier separately q b_top = h top A(T a -T top ) TSVs change resistivity of interlayer material q b_bottom = h bottom A(T a -T bottom ) 8

9 Modeling Liquid Cooling as RC-Network in 3D MPSoC stacks Local junction temperature modeled as 4-resistor based compact ttransient tthermal model l(4rmb (4RM-based CTTM) Rtot = Rcond + Rconv + Rheat [Sridhar, et al., ICCAD 2010] 3D-ICE: 3D IC Emulator with liquid cooling 4 convective 2 voltage-controlled otageco t o ed resistances current sources z y R conv x 9

10 Manufacturing of 5-Tier 3D Chips with Liquid Channels in Multiple Tiers Adding multi-tier liquid id cooling in-/out-lets t 10

11 Manufacturing of 5-Tier 3D Chips with Liquid Channels in Multiple Tiers Inlet/Outlet Opening 11

12 Manufacturing of 5-Tier 3D Chips with Liquid Channels in Multiple Tiers Maximum error of 3.5% between measurements and RC-based 3D thermal model with liquid q cooling g T Temperat ure (oc) ml/min 3D-ICE CVX 5-tier ml/min 3D-ICE CVX 5 5-tier tier ml/min 3D-ICE CVX 5 5-tier tier Time (s)

13 Active-Adapt3D: Active cooling management for 3D MPSoCs 3D MPSoC temperature control at system-level: Electrical l based: task scheduling, and DVFS (µsec or few ms) Mechanical based: run-time varying flow rate (hundreds of ms) Fuzzy logic-based controller and thermal-aware aware scheduler 1. Design-time analysis: extraction of set of thermal management rules [Coskun, Atienza, et al., MICRO 2. Run-time thermal management: utilization of rules in scheduler and subsequently fuzzy logic controller using both mechanical and electrical methods to achieve: Inputs: Thermal balance Workload information DVFS actuator Energy efficiency Floorplan, package Scheduler (TALB) 3D-ICE: Fast Transient Temperature Response for Each Unit 2011] Flow rate actuator (pump, valve) 13

14 Active-Adapt3D: Active cooling management for 3D MPSoCs 3D MPSoC Coolant Inlet temperature Coolant Flowcontrol at system-level: 700 Electrical l based: task scheduling, and DVFS (µsec or few ms) Mechanical 600 based: run-time varying flow rate (hundreds of ms) Fuzzy 500 logic-based controller and thermal-aware aware scheduler Height (um) Location of fthe hotspot t 1. Design-time 400 analysis: extraction of set of thermal management rules Coolant Outlet 2. Run-time 300 thermal management: utilization of rules in scheduler and subsequently fuzzy logic controller using both mechanical and electrical elements to 200 achieve: Inputs: 100 Thermal balance Workload information DVFS actuator Floorplan, package 0Energy efficiency Scheduler Distance along (TALB) Channel length (um) 3D-ICE: Fast Transient Temperature Response for Each Unit New insights about suitable thermal-aware aware scheduling and task assignment for 3D MPSoCs! Flow rate actuator (pump, valve) 14

15 Temperature-Aware Load Balancing (TALB) Scheduler for 3D MPSoCs Dispatching Queues Threads Load balancing (Default): Balances queue-length, but does not consider core locations TALB: [Coskun, et al., DATE 2010] Thermal index ( ) per core i i i l weighted l queue thermal(t(k)) ( Core-1 Core-2 Core-3... Chip For cores at locations 1, 2 and 3: Chip 1 15

16 3D-ICE: predicted max. core temperat. TALB: Workload Pump power Integrated Flow Rate and DVFS Fuzzy Controller for 3D MPSoCs Takagi-Sugeno fuzzy controller Fuzzy Controller IF x 1 =a 1 AND x 2 =a 2 THEN y 1 =f 1 (x 1,x 2,..) Extract cooling rules at design time Stable (circle criteria, BIBO stability, ) Rules for multi-outputs at run-time Adjusted Flow Rate and DVFS Look-up table (depends on 3D chip and pump properties) [Sabry, et al., ICCAD 2010] Electrical: DVFS (VF) per core Mechanical: Flow rate (FL) Cores next to the inlet: no thermal reduction needed IF D is L THEN VF is H AND FL is L 4-tier chip: ~150 rules 16

17 Experiments Active Thermal Management 3D MPSoCs with Microchannels Target 3D systems based on 3D ICs with Sparc-Power cores Power values and workloads from real traces measured in Sun platforms (database queries, web services, etc.) Cores and caches in separate layers 3D crossbar as interconnect Channels: Width 100µm and height 50µm Three flow rate settings, default at 32ml/min 17

18 ergy n Run-time thermal Management for 3D Chips: thermal evaluation For hot spot threshold 85 o C, thermal violations: 0% Energy reduction: malized ene onsumptio Norm co 3 2,5 2 1,5 1 0,5 0 70% average coolant energy (max. savings: 77%) 52% average total system energy (max. savings: 85%) [Coskun, Atienza, et al., MICRO 2011] AC_LB AC_TDVFS_LB LC_LBLB LC_FUZZY_LB LB 4 tier AC_LB 4 tier LC_LB LB 4 tier LC_FUZZY_LB Pumping network energy System Energy Promising figures for thermal control in 3D MPSoCs, thermal gradients of less than five degrees/tier 18

Conclusions: Aquasar 2010 First Chip-Level

19 Conclusions: Aquasar 2010 First Chip-Level Liquid Cooled Server 3D MPSoCs: Interdisciplinary work Fast RC thermal models for 3D ICs with inter-tier variable liquid fluxes (error of less than 4%) Layout combining electrical and mechanical constraints Next generation of thermal-aware proactive controllers (task control, flow rate and DVFS) Holistic control reduces significantly the energy cost for the whole system (80% power savings) Green datacenters: energy efficient Roadrunner: 445 Mflops/Watt Water-Cooled IBM BladeCenter HS22 Aquasar: 2250 MFlops/Watt Back side Water conditioning Courtesy: IBM Zürich 19

20 Key References and Bibliography 3D Thermal modeling and FPGA-based emulation 3D-ICE: Compact transient thermal model for 3D ICs with liquid cooling via enhanced heat transfer cavity geometries, A. Sridhar,etal.Proc. of ICCAD 2010, USA, November 2010 ( Emulation-based transient thermal modeling of 2D/3D systems-on-chip with active cooling, PabloG.DelValle,DavidAtienza,Elsevier Microelectronics Journal, December Thermal management for 3D MPSoCs 3D Stacked Systems with Active Cooling: The Road to Single-Chip High- Performance Computing, Ayse K. Coskun, David Atienza, Mohamed M. Sabry, J. Meng, IEEE MICRO, June/July Fuzzy Control for Enforcing Energy Efficiency in High-Performance 3D Systems, M. Sabry, Ayse K. Coskun, David Atienza, Proc. of ICCAD 2010, USA, November Energy-Efficient Variable-Flow Liquid Cooling in 3D Stacked Architectures, Ayse K. Coskun, David Atienza, et al., Proc. of DATE 2010, Germany,March2010. Modeling and Dynamic Management of 3D Multicore Systems with Liquid Cooling, Ayse K. Coskun, et al., Proc. of VLSI-SoC 2009, Brazil,October2009. (Best Paper Award) 20

21 Nano-Tera.ch Swiss Engineering Programme European Commission QUESTIONS? Swiss National Science Foundation

Thermal Modeling and Active Cooling

Thermal Modeling and Active Cooling for 3D MPSoCs Prof. David Atienza, Embedded Systems Laboratory (ESL), EE Institute, Faculty of Engineering MPSoC 09, 2-7 August 2009 (Savannah, Georgia, USA) Thermal-Reliability