-1-

Transcription

1 -1- ARTIST Summer School in Europe 2010 Autrans (near Grenoble), France September 5-10, 2010 Thermal-Aware Design of 2D/3D Multi-Processor System-on on-chip Architectures Invited ds Speaker: David idai Atienza, Professor and Director of Embedded Systems Laboratory (ESL), EPFL

2 Evolution of Electronics to Multi-Processor System-on on-chip (MPSoC MPSoC) Roadmap continues: nm Multi-Processor System-on on-chip (MPSoC) architectures CMOS 90nmCMOS 65nm CMOS 45nm PE Local Memory hierarchy CPU i/o I/0 PE PE PE I/0 SRAM SRAM PE SRAM I/O I/O P E R I P H E R A L S SDR RAM main memor ry 2

3 Evolution of Electronics to Multi-Processor System-on on-chip (MPSoC MPSoC) Roadmap continues: nm Multi-Processor System-on on-chip (MPSoC) architectures CMOS 90nmCMOS 65nm CMOS 45nm [Cell Multi-Processor PS3] I/0 PE PE 80-tile PE 1.28TFLOPS I/0 SRAM MPSoC INTEL SRAM [ISSCC 07] PE SRAM I/O I/O P E R I P H E R A L S SDR RAM main memor ry 3

4 MPSoCs are Spreading Fast # of cor res 512 [Amarasinghe06] 256 Picochip PC102 Cisco CSR-1 Intel 128 Tflops Pentium P2 P3 Raw Raza XLR Niagara Boardcom 1480 Xbox360 PA-8800 Opteron Power4 Athlon PExtreme Power6 Yonah Itanium P4 Itanium 2 Ambric AM2045 Cavium Octeon Cell Opteron 4P Xeon MP Tanglewood ?? 4

5 Design Issues in MPSoCs MPSoCs have very complex architectures Advanced components and CAD tools very expensive Time-closure issues, system speed decreased Aggravated thermal issues Hot-spots, non-uniform thermal gradients [Sun, 1.8 GHz Sparc v9 Microproc] [Santarini, EDN, March 05] [Sun, Niagara Broadband Processor] [Coskun et al 07, Sun] High chances of thermal wear-outs and very short lifetimes! 5

6 Thermal Issues Become More Critical for 3D-MPSoCs I/O Pherip. I/0 SDRAM PEs layer SRAM PEs layer SRAM 3D Integ. / PE PE PE I/0 PE I/O I/O P E R I P H E R A L S SRAM SRAM I PE SRAM SDRAM main memory More power and more non-uniform heat spreading! 6

7 Reliability Degradation Factors in MPSoCs 7

8 Reliability Degradation Factors in MPSoCs Thermal Hot Spots 8

9 Reliability Degradation Factors in MPSoCs 9

10 Reliability Degradation Factors in MPSoCs 90 Fatigue failures increase with: Magnitude of variation Frequency of cycles 80 T (C) T Time (sec) Caused by: Power on/ off Power management (turning off cores) 10

11 Reliability Degradation Factors in MPSoCs 11

12 Reliability Degradation Factors in MPSoCs Spatial Gradients 12

13 Advocating Thermal-Aware 2D/3D MPSoC Design Integration of HW/SW modeling and management Heat Flow Models Fast Thermal Exploration HW Thermal monitoring HW Tuning knobs HW-Based Thermal Management Policies SW-Based Thermal Management Policies 13

14 Outline Part 1: Thermal Modeling and Management for 2D MPSoCs Part 2: Thermal Modeling and Management for 3D MPSoCs with Active Cooling Acknowledgements: Prof. Ayse K. Coskun (Boston University and Sun Microsyst.).), Dr. Srinivasan Murali (inocs and EPFL), Prof. Jose L. Ayala (UCM), Thomas Brunschwiler and Dr. Bruno Michel (IBM Zürich), Prof. Stephen Boyd (Stanford University) 14

15 Thermal Modeling, Analysis and Management of 2D Multi-Processor System-on-Chip Prof. David Atienza Alonso Embedded Systems Laboratory y( (ESL) Institute of EE, Faculty of Engineering ARTIST Summer School 2010, Autrans (France)

16 Outline MPSoC thermal modeling and analysis HW-based thermal management for MPSoCs SW-based thermal management for MPSoCs Conclusions 2

17 Outline MPSoC thermal modeling and analysis HW-based thermal management for MPSoCs SW-based thermal management for MPSoCs Conclusions 3

18 MPSoC Thermal Modeling Problem Continuous heat flow analysis Capture geometrical characteristics of MPSoCs Explore different packaging features and heat sink characteristics Time-variant heat sources Transistor switching depends on MPSoC run-time activity (software) Dynamic interaction with heat flow analysis Very complex computational problem! 4

19 MPSoC Thermal Modeling State-of of-the the-art MPSoC Modeling and Exploration 1. SW simulation: Transactions, cycle ycle-accurate (~100 KHz) [Synopsys Realview, Mentor Primecell, Madsen et al., Angiolini et al.] At the desired cycle-accurate level, they are too slow for thermal analysis of real-life life applications! 2. HW prototyping: Core dependent (~ MHz) [Cadence Palladium II, ARM Integrator IP, Heron Engineering] Very expensive and late in design flow, no thermal modeling, only used for functional validation of MPSoC architectures! Heat Flow Modeling: 1. Software thermal/power models models [Skadron et al., Kang et al.] Too computationally intensive and not able to interact at run-time with inputs from MPSoC components! 5

20 MPSoC Thermal Modeling State-of of-the the-art MPSoC Modeling and Exploration 1. SW simulation: Transactions, cycle ycle-accurate (~100 KHz) [Synopsys Realview, Mentor Primecell, Madsen et al., Angiolini et al.] At the desired cycle-accurate level, they are too slow for thermal analysis of real-life life applications! Combination of cycle-accurate MPSoC behavior 2. HW prototyping: Core dependent (~ MHz) and IC heat flow modeling at run-time is unheard of [Cadence Palladium II, ARM Integrator IP, Heron Engineering] Very expensive and late in design flow, no thermal modeling, only used for functional validation of MPSoC architectures! Heat Flow Modeling: 1. Software thermal/power models models [Skadron et al., Kang et al.] Too computationally intensive and not able to interact at run-time with inputs from MPSoC components! 6

21 Orthogonalizing MPSoC Thermal Modeling and Analysis I/O CPU CPU I/O sniffer SRAM sniffer sniffer CPU SRAM sniffer Energy sniffer sniffer CPU sniffer SRAM sniffer sniffer FPGA MPSoC Behavior Emulation on FPGA Temperature (T) SW thermal estimation tool (Host PC) Framework: MPSoC behavioral model on reconfigurable HW interacting with efficient thermal estimation 7

22 Chip and Package Heat Flow Modeling Model interface Input: power model of MPSoC components, geometrical properties Output: temperature of MPSoC components at run-time Thermal circuit: 1 st order RC circuit si Heat flow ~ Electrical current ; Temperature Si thermal ~ conductivity Voltage Heat spreader and IC composed of depends 160 elementary on temperature blocks Th hermal cond ductivity(w/ /mºk) Cu cu cu cu cu 130 si si si si si si si Thermal capacitance conductance matrix matrix si C si,1 G 1,2 -G 1,2 C si,2g 2,1 G 2,1. -G 21 G 21 Temperature change C cu,n (IMEC & Freescale, 90nm) Actual value Temperature (in Celsius) Ct k =-G(t k )t k +p k ;k= 1..m Temperature vector at instant k power consumption vector 8

23 SW Thermal Estimation Tool for MPSoCs. C t k = - G (t k )t k + p k ; k = 1..m Creating linear approximation while retaining variable Si thermal conductivity: Si thermal conductivity linearly approx. : G i,j (t k ) = I + q t k Numerically integrating. in discrete time domain the t k : t k+1 = A(t k )t k + Bp k ; k = 1..m A(t k ) = (I - d t C G(t k )) ; B = d t C -1 Complexity scales linearly with the number of modeled cells (simulated on P4@ 3GHz) tion w estimat me (S.) Time step chosen small enough for convergence thermal library validated against 3D finite element 200 model (IMEC & Freescale) 0 Heat flow Tim Si thermal conductivity dependent on temperature 60 sec of MPSoC heat flow analysis ) vity(w/mºk Non-linear 120 Therm mal conducti thermal estim Actual value Linear fit Proposed linear thermal estimation Temperature (in Celsius) Number of Cells 9

24 Case Study: HW 4-Core MPSoC MPSoC Philips board design: 4 processors, DVFS: 100/500 MHz Plastic packaging Software: Image watermarking, video rendering Power values for 90nm: Element Max Power Max Power (mw) (mw) 100 MHz 500 MHz Processor 2,92 x ,02 x 10 3 D-Cache 1,42 x ,10 x 10 2 I-Cache 1,42 x ,10 x 10 2 Priv Mem 0,61 x ,75 x 10 2 AMBA 0,31 x ,68 x

25 Results: Thermal Validation 4-core Philips MPSoC MPARM: Cycle-accurate SW architectural simulator Complete power/thermal models tuned to Philips/IMEC figures Simulations too slow: 2 days for 0.18 real sec (12 cells) HW thermal emulation able to validate policies at run-time Dynamic Voltage and Frequency Scaling 400 Many weeks of Scaling (DVFS) simulation?! based thresholds Emulation time 45 sec (128 cells)! Average temperature of emulated 4-core 4 Very fast validation of MPSoC 420 run-time thermal behavior and management DVFS ON: 500/100 MHz. in) Temp perature (Kelvi Package limit (~85ºC) ,0 10 1,0 20 2,0 30 3,0 40 4,0 50 5,0 60 6,0 70 7,0 80 8,0 Time (seconds) 500 MHz. 100 MHz. Simulation Simulation in MPARMin MPARMEmulation Emulation with DFS 11

26 Outline MPSoC thermal modeling and analysis HW-based thermal management for MPSoCs SW-based thermal management for MPSoCs Conclusions 12

27 Temperature Management is Power Control under Thermal Constraints Power consumption of cores determines thermal behavior Power consumption depends on frequency and voltage Setting frequencies/voltages can control power and temperature Optimization problem: frequency/voltage assignment in MPSoCs under thermal constraints Meet processing requirements Respect thermal constraint at all times Minimize power consumption 13

28 HW-Based Thermal Management State-of of-the the-art Static approach: thermal-aware aware placement to try to even out worst-case thermal profile [Sapatnekar, Wong et al.] Computationally difficult problem (NP-complete) Not able to predict all working conditions, and leakage changing dynamically, it is not useful in real systems No formalization of the thermal optimization problem Dynamic approach: HW-based dynamic thermal management Clock gating g based on time-out [Xie et al., Brooks et al.] DVFS based on thresholds [Chaparro et al, Mukherjee et al,] Heuristics for component shut down, limited history [Donald et al] Techniques to minimize power, they only achieve thermal management as a by-product 14

29 Formalization of Thermal Management Problem in MPSoCs Control theory problem Optimal frequency assignment module, 2-phase approach: Observable: bl Geometrical properties and behavior Heat 1) flow Design-time model and phase: thermal Find profile optimal estimation sets of frequencies Performance the cores counters for different working conditions Controlable: 2) Run-time Max. phase: throughput Apply one under of the thermal predefined constraints sets Tuning found knobs: in phase frequencies/voltages 1 for the required of system the system performance (DVFS) Observed system: MPSoC Run-time HW DVFS support Processor cores Control output: Cores freqs Performance counters (average frequency) Thermal sensors Observer and control system Optimal frequency assignment module Thermal state Thermal profile estimation Requirements: Max. Throughput Constraints: Max. temperature 15

30 Pro-Active HW-Based Thermal Control: Phase 1 Design-Time Predictive model of thermal behavior given a set of frequency assignments Allowed core power values and frequencies Optimization problem: Constraints: Chip floorplan Non-linear offline problem Optimal Performance constraint: on average, freq. is f avg frequency assignment Thermal equation: Si conductivity depends on temp Thermal equation Meet temp. constraints module at all time points Power Power equation: equation quadratic based dependence on frequency on freq. Packaging, heat spreader information Phase inputs Minimize esu sum of power consumption of cores Method outputs Table of cores frequencies assignments Frequency in predefined range 16

31 Making Power and Thermal Constraints Convex Power constraint adaptation Change non-affine (quadratic equality): Solve convex problem and get table of optimal frequencies p max for (f different i,k ) 2 / (f max ) working 2 = p i,k ; i conditions = 1,..,n, k in polynomial time (number of processors) To convex inequality: Thermal constraint adaptation p 2 /(f 2 max (f i,k ) max ) p i,k ;i=1 1,..,n, k Use worst case thermal conductivity in the range of allowed temperatures, and iterate (if needed) to optimum 17

32 Pro-Active HW-Based Thermal Control: Phase 2 - Run-Time Time, Putting It All Together Use table of frequencies assignments and index by actual conditions at regular run-time intervals Targeted operating frequency of cores Current temperature of cores Method inputs Run-time optimal DVFS assignment HW module 1) Index table output of phase 1 with current working conditions 2) Compare to current assignment to cores and generate required signaling to modify DVFS values Phase output Run-time DVFS changes for processors 18

33 Case Study: 8-Core Sun MPSoC MPSoC Sun Niagara architecture 8 processing cores SPARC T1 Max. frequency each core: 1 GHz 10 DVFS values, applied every 100ms Max. power per core: 4 W Execution characteristics of workloads [Sun Microsystems]: Mixes of 10 different benchmarks, from web-accessing to multimedia 60,000 iterations of basic benchmarks, tens of seconds of actual system execution Sun s Niagara MPSoC 19

34 Results: Thermal Constraints Respected DVFS: 2-phase Convex method: Total run-time of benchmarks 180 sec Proposed method achieves better throughput than standard DVFS while satisfying thermal constraints 106 sec (45% less exec. time) 20

35 Outline MPSoC thermal modeling and analysis HW-based thermal management for MPSoCs SW-based thermal management for MPSoCs Conclusions 21

36 MPSoC System-Level Architecture: HW and SW Layers MPOS To Core #1 (30% load) Task Manager LOAD FREQUENCY 100 % TASK MIGRATION To Core #2 (60% load) To Core #1 Can we control the MPSoC To Core #2thermal profile by (70% load) TASK B (35% FSE LOAD load) controlling 50 % software 50% execution? 0 % To Core #2 (60% load) TASK A TASK C To FSE Core LOAD #1 FSE LOAD (30% 40% load) 40% PROC 1 PROC 2 To Core #1 (70% load) To Core #2 (35% load) SW layers introduced to better exploit the HW of MPSoCs Applications divided in tasks: blocks of operations to be executed Multi-processor Operating System (MPOS) distributes the tasks Load balancing: equal ldi distribution tib ti of work kb between processors 22

37 Task Migration for Load vs. Thermal Balancing Plain load balancing No improvement in workload distribution possible: no migration 100 % LOAD FREQUENCY TEMPERATURE TEMPERATURE PROCESSOR 1 TEMP. MEAN TEMP. PROCESSOR 2 TEMP. TASK B FSE LOAD Hot-spot! 50 % 50% 40% TIME TIME 0 % TASK A FSE LOAD 40% TASK C FSE LOAD 40% PROC 1 PROC 2 23

38 Task Migration for Load vs. Thermal Balancing Heat&Run: Load balancing with local knowledge of temperature in MPSoC components 100 % LOADTASK MIGRATION FREQUENCY LOAD FREQUENCY TEMPERATURE TIME 100 % 50 % 50 % 0 % 0 % TASK B FSE TASK LOAD B 40% 50% 40% 50% TASK TASK A FSE LOAD 40% FSE LOAD FSE LOAD 40% TASK TASK C FSE LOAD 40% FSE LOAD 40% PROC 1 PROC 2 PROC source 1 PROC target 2 24

39 Task Migration for Load vs. Thermal Balancing Heat&Run: Load balancing with local knowledge of temperature in MPSoC components Helping with hot-spots, but no thermal balancing TEMPERATURE LOAD FREQUENCY TASK MIGRATION Existing approaches do not consider 100 % global thermal dynamics for task migration TIME 50 % 0 % TASK A FSE LOAD 40% TASK B FSE LOAD 50% 40% TASK C FSE LOAD 40% PROC source 1 PROC target 2 25

40 Task Migration for Load vs Thermal Balancing Contribution: Migration strategy for thermal balancing Global knowledge of temperature at MPOS level Adjusted to particular thermal dynamics of each platform Formalization Dynamic number of tasks, no control theory formalization possible Knapsack problem, move N largest tasks between cores: estimated increase in temperature and minimizing performance penalty TEMPERATURE TEMPERATURE UPPER TRESHOLD LOWER TRESHOLD TIME TIME Reduces hot spots and reaches thermal balancing Reduces hot-spots and reaches thermal balancing 26

41 Case Study: Freescale MPSoC Board Hardware 3 RISC processor cores 16KB caches, 32KB shared mem. AMBA bus, 2GB ext. mem Software uclinux-based MPOS Multimedia applications: audio and video Two packaging options Mobile embedded SoCs (slow temperature variations) High performance SoCs (fast temperature variations) 27

42 Results and Comparisons Good thermal balancing Average: 40.5ºC, variations of < 3ºC Small performance overhead ( 2 migrat/s) +/-3º Comparisons with other policies Load balancing inefficient (>7ºC diffs) 400MHz (1% overhead) Good performance and uniform temperature adjusting globally to thermal dynamics with MPOS Heat&Run inefficient or causes many deadline misses (40% below performance requirements) Contribution: performance requirements met for both types of packaging 28

43 Adapt2D-MIGRA: Combination of HW and SW-Based Pro-Active Thermal Management Initial: Large gradients New: Thermal balancing HW-based management: Convex-based dynamic voltage and frequency scaling (DVFS) exploration SW-based management: Proactive task scheduling and migration Support of multi-processor operating system: Solaris Multi-Core Good thermal control o in commercial ca MPSoCs in 90nm, what about 3D integration? 29

44 Outline MPSoC thermal modeling and analysis HW-based thermal management for MPSoCs SW-based thermal management for MPSoCs Conclusions 30

45 Conclusions Progress in semiconductor technologies enables new MPSoCs Thermal/reliability issues must be addressed for safe human interaction Thermal monitoring and control are key Clear benefits of thermal-aware aware design methods for MPSoCs Novel, fast and low-cost thermal modeling approach at system-level Formalization of HW-based thermal management problem as convex, and solved in polynomial time New SW-based thermal balancing method with very limited overhead Validation on commercial 2D- MPSoCs (Sun, (Sun, Freescale, Philips) Fast exploration of thermal behavior of complex MPSoCs Effective HW- and SW-based pro-active thermal management 31

46 Key References and Bibliography Thermal modeling and FPGA-based emulation HW HW-SW Emulation Framework for Temperature-Aware Design in MPSoCs, D. Atienza, et al. ACM TODAES,, Vol. 12, Nr. 3, pp. 1 26, August Thermal management for 2D MPSoCs Thermal Balancing Policy for Multiprocessor Stream Computing Platforms, F. Mulas, et al., IEEE T-CAD,, Vol. 28, Nr. 12, pp , December Processor Speed Control with Thermal Constraints, A. Mutapcic, S. Boyd, et al. IEEE TCAS-I,, Vol. 56, Nr. 9, pp , Sept Inducing Thermal-Awareness in Multi-Processor Systems-on-Chip Using Networks-on-Chip, E. Martinez, et al., Proc. ISVLSI Temperature Control of High-Performance Multi-core Platforms Using Convex Optimization, S.Murali, et al., Proc. DATE,

47 Swiss National Science Foundation QUESTIONS? Acknowledgements: European Commission UCSD / Sun Microsystems IMEC / Philips IBM Zürich Bologna / Freescale semiconductors33

48 Thermal Modeling and Management for 3D MPSoCs with Active Cooling Prof. David Atienza Alonso Embedded Systems Laboratory y( (ESL) Institute of EE, Faculty of Engineering ESL/EPFL 2010 ARTIST Summer School 2010, Autrans (France)

49 Promises Advantages of 3D vs. 2D Chips Reduce average length of on-chip global wires Increase number of devices reachable in given time budget Greatly facilitate heterogeneous integration (e.g. logic-dram stacks) ESL/EPFL 2010 [Figures: Ray Yarema, Fermilab] Samsung Wafer Stack Package (WSP) memory 2

50 Thermal-Reliability Issues in 3D Chips Latest chips increase power density Non-uniform hot-spots in 2D chips In 3D chips, heat affects several layers! (even more cool components) [Sun, 1.8 GHz Sparc v9 Microproc] Courtesy: [Sun, [IBM and Irvine Sens.] Niagara Broadband Processor] ESL/EPFL

51 Thermal-Reliability Issues in 3D Chips Latest chips increase power density Non-uniform hot-spots in 2D chips In 3D chips, heat affects several layers! (even more cool components) [Sun, 1.8 GHz Sparc v9 Microproc] Courtesy: [IBM and Irvine Sens.] Higher chances of thermal [Sun, Niagara wear-outs Broadband and Processor] very short lifetimes! ESL/EPFL

52 Run-Time Heat Spreading in 3D Chips 5-tier 3D stack: 10 heat sources and sensors Layer Inject between 4W 1.5W width (um) length (um) Layer nd Tier width (um) h (um) width ESL/EPFL th Tier Layer length (um) wid dth (um) Layer length (um) length (um) 3 rd Tier Large and non-uniform th Tier heat propagation! (up to 130º C on top tier) 5 394

53 NanoTera CMOSAIC Project: Design of 3D MPSoCs with Advanced Cooling 3D systems require novel electro-thermal co-design Academic partners: EPFL and ETHZ Industrial: IBM Zürich ESL/EPFL

54 NanoTera CMOSAIC Project: Design of 3D MPSoCs with Advanced Cooling 3D systems require novel electro-thermal co-design Academic partners: EPFL and ETHZ Industrial: IBM Zürich 3D stacked MPSoC chips: microchannels etched on back side to circulate liquid coolant ESL/EPFL 2010 adjustment of coolant flux task scheduling and execution control System Level Active Cooling Manager (3D heat flow prediction) 7

55 Outline Introduction 3D chip thermal modeling framework Validation of 3D thermal model Liquid cooling modeling Liquid id cooling model validation Close-loop 3D MPSoCs thermal management with active cooling Experiments and conclusions ESL/EPFL

56 Compact RC-Based Tier Thermal Model Gate-level thermal model q bj 6 b f 1 fj T f b 0 j q RC Network of q b_top q b_back Si/metal layer cells q b_left q b_front q q b_right q b_bottom 2D tier modeled as heat flux moving between adjacent cells I-1 I I+1 (q bi ) I (q bi ) I+1 face i Convective boundary conditions between layers in tier q b_top = h top A(T a -T top ) q b_bottom = h bottom A(T a -T bottom ) ESL/EPFL 2010 [Atienza et al., TODAES 2007] 9

57 Complete 3D Chip Thermal Modeling Multi-level execution for thermal convergence in 3D Local (2D-tier), liquid channels and global (3D) propagation N iteration ns Evaluate local temperature for each cell Feedback temperature Update with neighbour temperature difusion Tier-lev vel conve ergence Go to next tier or microchannel ESL/EPFL 2010 [Ayala et al., NanoNet 2009] 10

58 3D Chip Thermal Library Validation Extensible set of layers in 3D stack up to 9 tiers and heat spreader Pre-defined layers: Silicon, copper (10 layers), glue, overmold, interposer, bump Configurable nr. of cells and iterations per tier Initially 10ms thermal interval (1000 iterat./tier) Multi-tier test chip manufactured at EPFL: ESL/EPFL

59 3D Chip Thermal Library Validation Extensible set of layers in 3D stack up to 9 tiers and heat spreader Pre-defined layers: Silicon, copper (10 layers), glue, overmold, interposer, bump Configurable nr. of cells and iterations per tier Initially 10ms thermal interval (1000 iterat./tier) Multi-tier test chip manufactured at EPFL: Three types of tiers ESL/EPFL

60 3D Thermal Library Validation: Creating Various 3D Thermal Maps Flexibility for thermal characterization ESL/EPFL

61 3D Thermal Library Validation: Creating Various 3D Thermal Maps Flexibility for thermal characterization ESL/EPFL

62 3D Thermal Library Validation: Creating Various 3D Thermal Maps Flexibility for thermal characterization 10 heat sources and sensors per layer, accesible to be simultaneously l activated t ESL/EPFL

63 3D Thermal Library Validation: Correlation with 5-Tier 3D Stack e (mv) Sensor Voltag D Chip, EPFL, Layer 3 characterization ti Blue Curve: 3D current -heat model for D8 Pink curve: Heater current measured in D8 Dev8 D7HD8S Heater Current (ma), applied to Dev 7 ESL/EPFL 2010 Sensor Voltage ( mv) [Ayala et al., Nano-Nets 09] 3D Chip, EPFL, multi-tier tier characterization Bue/Pink Curve: D7 (tier 1) and D8 (tier 4) Red Curve: 3D current-heat model for D8 Dev6 Dev7 Div6_Iheat Heater Current (ma), applied to Dev 7 16

64 3D Thermal Library Validation: Correlation with 5-Tier 3D Stack ESL/EPFL 2010 e (mv) Sensor Voltag D Chip, EPFL, Layer 3 characterization ti Blue Curve: 3D current -heat model for D8 Pink curve: Heater current measured in D8 Dev8 D7HD8S Heater Current (ma), applied to Dev 7 Variations of less than 1.5% between 3D stack measurements and new 3D thermal model mv) Sensor Voltage ( [Ayala et al., Nano-Nets 09] 24 3D Chip, EPFL, multi-tier tier characterization Bue/Pink Curve: D7 (tier 1) and D8 (tier 4) Red Curve: 3D current-heat model for D8 Dev6 Dev7 Div6_Iheat Heater Current (ma), applied to Dev 7 17

65 Modeling Through Silicon Vias (TSVs) in 3D Stacks TSVs: Size: 5-10um x um TSVs change resistivity of interlayer material (IM) Figure: LSM-EPFL Modeling Granularities: 1. Homogeneous distribution, one R value for the IM 2. Different R value per unit (core, cache, etc.) 3. Exact locations of TSVs Higher accuracy Higher complexity Source: IBM Zürich and Y.Heights ESL/EPFL

66 TSV Modeling Accuracy in 3D Stacks ESL/EPFL 2010 Chosen to model TSV groups in localized positions of 3D MPSoCs 19

67 Liquid Flux Model for Laminar Flow Local junction temperature modeled as RC network: R tot = R cond + R conv + R heat Heat source Thermal resist. of Si Chip back-side temperature Si base Heating Flow rate and thickness area Total area density R tot = 1/(G si /t + 1/R b ) + A/(bA t ) + A/(VPc p ) Dependence of thermal resistance in liquid flux modeled as a quadratic form Variable value of coolant flux (Φ) R heat aφ + bφ 2 ; b << a T 1 P 1 Thermal resistance of q 2 wiring P 2 ESL/EPFL 2010 [Atienza et al., THERMINIC 09 and DATE 10] 20 q 1

68 3D Thermal Model with Liquid Cooling New set of layers in 3D stack 3D stack (up to 9 tiers) 1 microchannel and coolant flow per tier 5-tier stack with microchannels and manifold cooling seal manufactured at IBM/EPFL Enables different multi-tier liquid flux injection Liquid Micro-Heater PCB Micro-Channels Source: IBM & ESL, EPFL ESL/EPFL

69 Manufacturing of 5-Tier 3D Test Chip with Liquid Channels in Multiple Tiers Front-side Back-side Figure: IBM & ESL, EPFL Adding multi-tier liquid cooling in-/out-lets ESL/EPFL 2010 Multi-tier active cooling technology feasible for 3D-stacked chips 22

70 Correlation Results: Liquid Cooling and 3D Heat Transfer Temperature evolution at the junction (T j ) Tested range: to 0.15 L/min q 2 P 2 q 1 Similar accuracy results at different channels T 1 P 1 Avg Max temp Error= 0.6% ESL/EPFL 2010 [Atienza et al., THERMINIC 09] 23

71 Correlation Results: Liquid Cooling and 3D Heat Transfer Temperature evolution at the junction (T j ) Tested range: to 0.15 L/min Similar accuracy results at different channels q 2 2 P q 1 Variations of less than 1% between measurements and RC-based 3D thermal model with liquid cooling T 1 P 1 Avg Max temp Error= 0.6% ESL/EPFL 2010 [Atienza et al., THERMINIC 09] 24

72 Complete 3D Chip Thermal Modeling Flow with Liquid Cooling Inputs: Workload information Floorplan, TSV areas, package Temperature (for dynamic policies) Scheduler (Reactive, Proactive) Inputs: Workload information Activity of cores Power Manager (DPM) Inputs: Power trace for each unit Floorplan, package and die properties (Niagara-1), TSV area percentage/distribution Flow rate 3D Thermal Simulator w. Liquid Cooling based on EPFL-IBM 3D chips (Integrated within internal HotSpot tool version) Transient Temperature Response for Each Unit ESL/EPFL

73 Run-Time HW/SW Thermal Modeling Framework for 3D Chips Exploitation of both hardware and software benefits Zero-delay MPSoC architecture simulation Multi-Proc. OS + DVFS + Task Migration I/O CPU CPU Sw app 1... Sw app N SRAM SRAM SRAM I/O sniffer sniffer sniffer CPU sniffer sniffer sniffer CPU sniffer sniffer sniffer Energy of 2D components MPSoC Behavior Emulation on FPGA Detailed thermal analysis of 2D MPSoC layout ESL/EPFL 2010 [D. Atienza et al., TODAES 2007] standard Ethernet connection & dedicated HW monitor Software Thermal Model cu cu cucu cu si si si si si si si si si Temp. (T) of 2D components Host PC 26

74 Run-Time HW/SW Thermal Modeling Framework for 3D Chips Exploitation of both hardware and software benefits Zero-delay MPSoC architecture simulation Multi-Proc. OS + DVFS + Task Migration I/O CPU CPU Sw app 1... Sw app N SRAM SRAM SRAM I/O sniffer sniffer sniffer CPU sniffer sniffer sniffer CPU sniffer sniffer sniffer Energy of 3D components MPSoC Behavior Emulation on FPGA standard Ethernet connection & dedicated HW monitor Temp. of 3D components N th Tier 3D Stack Thermal Model 1 st Tier Host PC ESL/EPFL 2010 [D. Atienza, THERMINIC 2009] 27

75 Thermal Management for 3D-MPSoCs with Liquid Cooling Active-Adapt3D: Combined policy manager (Best-Paper Award at IEEE/IFIP VLSI-SoC S 2009) Predictive, floorplan-based task assignment and DVFS Close-loop variable liquid cooling control T 80 C Increment flow rate ; T < 80 C Decrement Policy can be applied reactively or proactively System Temperature Dynamics Flow Rate Tuner Thermal Sensors REACTIVE Temperature Measurements ESL/EPFL 2010 Temperature Forecast PROACTIVE ARMA Based Predictor 28

76 Adaptive Thermal-Aware Task Assignment Policy for 3D MPSoCs Cores on layers closer to the heat sink can be cooled faster in comparison to cores further away Adapt-3D assigns a thermal index ( ) to each core in order to distinguish the location of the cores Higher Core more prone to hot spots i For cores at locations 1, 2 and 3: Chip Chip 1 ESL/EPFL 2010 [Coskun and Atienza, DATE 09] 29

77 Adaptive Thermal-Aware Task Assignment Policy for 3D MPSoCs Probability of receiving workload at time t: P t P t 1 W For each core Weight: W inc dec 1 W init if T i W if T init i avg avg T T preferred preferred Cool core Hot core Empirical constants W init T preferred T avg E.g., 80 o C Measuredby sensors ESL/EPFL

78 Experiments 3D Thermal Management: 3D MPSoCs with Microchannels Target 3D systems based on 3D version Sun UltraSPARC T1 Power values and workloads from real traces measured in Sun platforms (multimedia players, web servers, databases, etc.) Cores and caches in separate layers Channels: Width 400um, Depth 250um. Four flow rate settings, default at 15ml/min. ESL/EPFL 2010 (EXP1-2) (EXP3) (EXP4) 31

79 Thermal Management for 3D Chips: Active-Adapt3D Adapt3D Comparisons Predictive task scheduling, active cooling and floorplan- aware DVFS achieves less than 5% hotspots Promising figures for thermal control in 3D-MPSoCs ESL/EPFL 2010 [Coskun and Atienza, DATE 10] 32

80 Thermal Management in 3D Chips: Active-Adapt3D Adapt3D Comparisons Variable multi-tier tier flow control useful for 3D systems with 3+ layers. Proactive thermal management achieves: 75% reduction in spatial gradients on average -- for fixed flow rate 97% reduction in spatial gradients on average -- for variable flow rate ESL/EPFL 2010 *LC: Multi-tier variable liquid cooling Cooling power savings up to 67% to worst-case flux [Coskun and Atienza, DATE 10] 33

81 Conclusions Complexity of coming 3D MPSoC chips requires novel thermal modeling approaches Application of simple RC-based methods demonstrated, validated with 3D test chip Initial model of liquid cooling channels in 3D chips Simple RC laminar flow model, works well with variable liquid fluxes (errors of less than 2%) Integrated the compact model into custom HotSpot tool New thermal management: feedback controller adjusts flow rate to allowed temperature with job assignment and DVFS Proactive control improves the hot spot reduction to 95% for systems with variable flow rates, and reduces thermal variations Dynamic flow rate adjustment is helpful in reducing the energy cost of fthe pump and overall system (67% power savings) ) ESL/EPFL

82 Key References and Bibliography 3D Thermal modeling and FPGA-based emulation 3D-ICE: Compact transient thermal model for 3D ICs with liquid cooling via enhanced heat transfer cavity geometries, A. Sridhar,etal.Proc. of ICCAD 2010, USA, November Transient Thermal Modeling of 2D/3D Systems-on-Chip with Active Cooling, David Atienza, Proc. of THERMINIC 2009, Belgium, October, Thermal management for 3D MPSoCs Fuzzy Control for Enforcing Energy Efficiency in High-Performance 3D Systems, M. Sabry, Ayse K. Coskun, David Atienza, Proc. of ICCAD 2010, USA, November Energy-Efficient Variable-Flow Liquid Cooling in 3D Stacked Architectures, Ayse K. Coskun, David Atienza, et al., Proc. of DATE 2010, Germany, March Modeling and Dynamic Management of 3D Multicore Systems with Liquid Cooling, Ayse K. Coskun, et al., Proc. of VLSI-SoC S 2009, Brazil, October (Best Paper Award) Dynamic Thermal Management in 3D Multicore Architectures, Ayse K. Coskun, et al., Proc. of DATE 2009, France, April ESL/EPFL 2010

83 Nano-Tera.ch Swiss Engineering Programme European Commission QUESTIONS? Swiss National Science Foundation ESL/EPFL 2010