*an NSF Industry/University Cooperative Research Center (IUCRC)

Size: px

Start display at page:

Download "*an NSF Industry/University Cooperative Research Center (IUCRC)"

Alyson Brown
5 years ago
Views:

1 The Center for Energy-Smart Electronic Systems* - Some Research Projects in Energy-Efficiency Improvements for Data Centers by Prof. Kanad Ghose *an NSF Industry/University Cooperative Research Center (IUCRC) Enabling Energy Efficient Data Centers

The Center for Energy-Smart Electronic Systems Vision: To create electronic systems that are self sensing and self regulating, and are optimized for energy efficiency at any desired performance level.

2 The Center for Energy-Smart Electronic Systems Vision: To create electronic systems that are self sensing and self regulating, and are optimized for energy efficiency at any desired performance level. Focus: Reducing Energy Use by Data Centers ES2 works in partnership with industry and academia to develop systematic methodologies for operating information technology, telecommunications, and electronic systems and cooling equipment.

3 3 Why Focus on Data Centers? Data centers are an intrinsic part of the modern society o Finance, Government, Healthcare, e-commerce o Entertainment o Clouds (IaaS) o Web-based information services o Large variety of web-based services for smart phones/tablets Number of data centers growing annually Requires 130+billion KWH of electricity annually to operate US Data Centers o 2.7% of the US electricity consumption o Often 90% of the power drawn from the grid is wasted o Large carbon footprint, many operators use their own diesel generators 4/9/2014 ES2 BRIEFING

The ES2 Team Binghamton University Bahgat Sammakia - Center Director

University of Texas at Arlington Dereje Agonafer Site Director and

Site Director and PI Amy Fleisher PI Gerald Jones PI Aaron Wemhoff

4 The ES2 Team Binghamton University Bahgat Sammakia - Center Director and PI Kanad Ghose Site Director and co-pi Bruce Murray co-pi The University of Texas at Arlington Dereje Agonafer Site Director and PI Alan Bowling Co-investigator Villanova University Alfonso Ortega Site Director and PI Amy Fleisher PI Gerald Jones PI Aaron Wemhoff PI The Georgia Institute of Technology Yogendra Joshi Site Director and PI

5 5 The NSF IUCRC Research Model Industry Federal Govt Academia Advanced Concepts 4/9/2014 The NSF shares cost of research with Member Companies Member Companies vote on projects that they wish to support Member Companies are true collaborators

6 Industry/University Cooperative Research Centers ( About 60 Active Centers) 6 4/9/2014

7 ES2 is an NSF Industry / University Cooperative Research Center 7 The National Science Foundation provides base funding to operate the center at each site Member Company membership fees fund the research projects Member Companies vote on projects that they wish to support Member Companies are true collaborators o Providing expertise to the projects o Providing data, used equipment, and facilities o Providing Internships to the Research Students 4/9/2014 E3S BRIEFING

8 8 Member Benefits Opportunity to share in Center strategic planning through representation on the Industrial Advisory Board (IAB) Access to a shared pool of research Opportunity to conduct research exclusive and confidential to a specific member Early access to publications, short courses, electronic systems, databases Ph.D.-level consultative services Opportunity for development of business-to-business relationships Access to PhD-level graduate students, mentoring and recruitment opportunities, workforce training programs, and internships Access to research equipment, infrastructure and expertise at partner campuses and at the New York State Center of Excellence in Small Scale Systems Integration and Packaging (S3IP), and Opportunities to develop research partnerships across the I/UCRC network. 4/9/2014 ES2 BRIEFING

9 4/9/2014 9

10 The Nature of the Problem: Raw Numbers for Data Centers Lifecycle: 90% of energy expended during the operations Breakdown of operating costs: o Equipment energy consumption: 45% o HVAC: 50% (about 25% to 30% in water chilling facilities) o Remainder: 5% Breakdown inside a modern server: o CPU energy: 20% to 25% o Memory energy: 30% to 45% o Interconnection energy: 10% to 15% Total energy consumptions: o 2.7% of total national electricity consumption in US, 130+ Billion KWH annually o 1.6% for Western Europe Others 5% Interconn 12% Others 28% HVAC 50% Equip 45% CPU 22% Memory 38%

11 The Nature of the Problem (contd.) Server overprovisioning is a common practice 100% Power Energy Efficiency Performance is not proportional to power: significant power dissipated when servers are idling and ready to take on requests 0% Typical Operating Region Individual Server Utilization 100% Poor energy efficiency (performance per Watt) on average Situation becoming worse with smaller form factors GOAL: IT Equipment, Entire Data Center should be energyproportional

because of long lag times Overprovisioning is commonly used to handle sudden load surges o Using sensed temperature to adjust cooling

12 The Nature of the Problem (contd.) HVAC equipment typically use water chillers to supply cold air: this approach to cooling is difficult to control in real time because of long lag times Overprovisioning is commonly used to handle sudden load surges o Using sensed temperature to adjust cooling system performance is a viable first step, but not enough New cooling systems: o Warm-water cooling, airside cooling o Seeing increased deployment, but reliability issues uncertain in some cases Cooling systems are not energy-proportional either

13 13 Energy Conservation: Hardware Level Cores: Utilization-based DVFS/power state setting, improved datapath design significant accomplishments here (microarchitecture, circuits, processes) DRAM (DDR-3): It s all in the memory controller but who makes these at this time? DC Power distribution Disks: Disk with power states, head parking off surfaces when idle etc. New and emerging Flash/SSD and NVRAM technologies Battery/Solar powered servers Alternative power sources, co-generation 4/9/2014 ES2 BRIEFING

14 14 OS/System Level Energy Conservation Utilization based DVFS/Power state management o Changes the CPU gear only Virtualization: forces equipment to operate near high load/high energy-efficiency regions through consolidation Energy-aware programming Workload allocation driven by utilization o Practical solutions are NOT automatic but rather in the form of configuration tools Others 4/9/2014 ES2 BRIEFING

containment o Floor tiles with fans ( active tile ), fan trays, chimneys.

15 Dynamic and Spot Cooling Systems The beginnings o Headmount, Rackmount, in-line o Airside economization Airflow control o Partitions for aisle containment o Floor tiles with fans ( active tile ), fan trays, chimneys. o CRACs with adjustable louvers Water cooling: cabinet and chip-level Missing: synergistic control

16 Modular Everything Modular water chilling systems Modular power generator and cooling Modular data centers o Fast deployment o Everything localized, customized and integrated better energy-efficiency

17 The Missing Synergy Isolation of solutions at the equipment level and cooling system level: these need to be coordinated o Must control of IT equipment, workload allocation and cooling systems synergistically Lack of coordination implies loss of overall efficiency The solution HAS to be interdisciplinary: focus on the integration and coordination at the system level Need to also focus on the OS and other system software components, control system and their interfaces

18 18 Holistic Control: The Challenges IT: how to operate IT equipment in an energy proportional fashion? o Need to match active server deployment with load changes automatically o Need to predict system load well in advance to permit deployment of additional servers are load surges occur Cooling System: how to provision cooling dynamically to match actual needs? o Need to worry about finite response times, thermal lags o Need to worry about complex air flow dynamics inside and outside the servers What kind of control system is right? o Must be scalable o Must be stable and responsive 4/9/2014 ES2 BRIEFING

19 19 Addressing the Challenges Data centers are complex systems Approach has to be interdisciplinary o Data analytics for prediction: load, temperature etc. o OS level infrastructures o Physics-based models for thermodynamic analysis and trend prediction o Control system to address complex system dynamics Solutions developed have applications to other systems, such as smart building, transportation/traffic management in large cities etc. 4/9/2014 ES2 BRIEFING

trends Techniques for DYNAMIC workload scheduling and control of cooling

20 20 Five Year Roadmap Identify energy consumption inefficiencies CHIP to COOLING TOWER Synergistic methods for computing workload and thermal trends Techniques for DYNAMIC workload scheduling and control of cooling Optimized, validated, dynamically controllable air and liquid cooling methods 4/9/2014

21 Unique Aspects of ES2 Research Determining key intrinsic energy consumption inefficiencies at every level, from CHIP to COOLING TOWER Develop techniques for managing data centers synergistically using predictive models for computing workload and thermal trends o Addressing energy-efficiency improvements at multiple spatial and temporal scales Linux kernel level techniques for implementing energy-aware virtualization, scheduling and synergistic management of the IT equipment and cooling systems Evaluation and development of optimized airflow and liquid cooled system management techniques o Using sophisticated modeling and simulation methods o Validated with data Mechanisms to utilize waste heat

22 22 EXAMPLES OF ES2 RESEARCH PROJECTS 4/9/2014 ES2 BRIEFING

Energy-Efficient Scheduling in Data Centers Load Balancer Team: Ghose and Sammakia (BU) Goals: Develop verified OS infrastructures to realize predictable energy dissipation and operate servers at

data centers dynamically matches server capacity and cooling resources with actual load with negligible performance impact Outcomes: Prototype system for Linux servers, control methodology and

23 Energy-Efficient Scheduling in Data Centers Load Balancer Team: Ghose and Sammakia (BU) Goals: Develop verified OS infrastructures to realize predictable energy dissipation and operate servers at high energy-efficiency levels Refine existing technology and prototype to enable deployment, demonstrate energy savings realized with negligible performance implications Addresses overprovisioning in data centers dynamically matches server capacity and cooling resources with actual load with negligible performance impact Outcomes: Prototype system for Linux servers, control methodology and demonstration of energy savings realized in an operational data center evaluated with actual power measurements and representative workload Incoming http Requests Front-End Controller Back-end servers (BES) Load info from back end servers Server management directives Simplified depiction multi-level load balancers, aggregation and access layer networks are NOT shown Energy per transaction as a function of system load: lower curve current prototype, upper curve base system

24 Energy-Efficient Scheduling in Data Centers: Energy Savings Actual loads used to manage number of active servers 24 Small performance loss for sudden increases in load 4/9/2014 ES2 BRIEFING

25 Number of Online M achines System Load (%) Energy-Efficient Scheduling in Data Centers: Coping with Performance Losses Actual loads used to predict short-term load trends based on dynamics of actual load history Performance loss limited to less than 1% on the average Active Machine Count System Load Time Into Test (sec.) Time Into Test (sec.) Without Prediction With Prediction Without Prediction With Prediction 4/9/2014 ES2 BRIEFING

26 Energy-Efficient Scheduling in Data Centers: Core Temperature Variation Temperature-awareness limits cores from heating up 26 Also incorporates wear-leveling 4/9/2014 ES2 BRIEFING

Sample Project: Role of Integrated and Silicon Photonics Transceivers in Data Centers Team: Ghose (BU) Goals: Develop cycle-accurate simulation models for

links Benefits of direct PCI3e links from NIC to multicore CPU (Intel Xeon 5 family) Explore alternative topologies for interconnection, extensions within

27 Sample Project: Role of Integrated and Silicon Photonics Transceivers in Data Centers Team: Ghose (BU) Goals: Develop cycle-accurate simulation models for transceivers, multi-lane Fibre links, couplers, switches, losses and errors Use DC benchmarking apps to study performance and energy savings over copper links Benefits of direct PCI3e links from NIC to multicore CPU (Intel Xeon 5 family) Explore alternative topologies for interconnection, extensions within Open Compute servers Outcomes: Simulation framework, models Study of copper vs. multi-lane optical fibre based DCINs Determine best combination of copper and fibre

28 DCIN Configurations Explored Front-end Firewall and Load Balancing Switches Core Switch Core Switch Aggregation Layer Switches Aggregation Layer Switches Connections from ToR switches to servers in rack not shown Top-of-Rack Switch Multi-lane fibers with on-board close-proximity integrated optical transceivers: Generic DC: inter-rack HPC: rack backplane May have another layer of access switches here SAN Fabric Storage Array

Sample Project: Dynamic Thermal Management and Control in a Data Center Team: Sammakia (BU) and

Center operating conditions to enable operation as dynamic selfsensing and self-regulating systems

consumption for a broad range of specified performance metrics Outcomes: Containment Systems Dynamic,

Modeling methodology for data centers physical parameter such as fan curves and thermal capacity.

29 Sample Project: Dynamic Thermal Management and Control in a Data Center Team: Sammakia (BU) and Agonafer (UTA) Goals: Develop verified thermal analysis tools to provide real time assessment of Data Center operating conditions to enable operation as dynamic selfsensing and self-regulating systems Enable Data Centers to run at optimal or near optimal conditions in order to minimize energy consumption for a broad range of specified performance metrics Outcomes: Containment Systems Dynamic, predictive, verified, 3D data center for air cooling and hybrid ( air plus liquid) cooling systems. Modeling methodology for data centers physical parameter such as fan curves and thermal capacity. Design guidelines for thermal management of data center accounting for steady state and dynamic conditions CRAC Calibration

Experimental Characterization of Water Cooled Server Cabinets Water-cooled server racks

Instead, the cooling is provided by a closed system employing an air-water (fin-tube)

Fans 46 44 42 Intel S1 40 38 CPU1 S1 36 34 32 CPU2 S1 30 28 26 Outlet S1 24 22 Outlet2

30 Experimental Characterization of Water Cooled Server Cabinets Water-cooled server racks are typically isolated units, so no heat load is removed by the room cooling system. Instead, the cooling is provided by a closed system employing an air-water (fin-tube) heat exchanger 30 Front 16 Instrumented Servers 4/9/2014 ES2 BRIEFING Rear Circulation Fans Intel S CPU1 S CPU2 S Outlet S Outlet2 S Intel S Characterization of the transient behavior in the cabinet

31 31 Sample Project: Models and Metrics for Dynamic Air and Hybrid Liquid Cooled Data Centers Based on Computational and Experimental Approaches Team: Sammakia, Ortega, Bowling Develop a full 3D, physics based, experimentally verified, dynamic CFD model of data center rooms cooled using air and hybrid air/liquid cooling systems The model will be used to assess energy efficiency and develop performance metrics for hybrid air/ liquid cooling systems immersed in an air-cooled data center 4/9/2014

32 Temp deg C Static pressure (in. H2O) CRAC internal resistance Flow bench apparatus 32 2U server for testing Calibrated operating point Prediction for 20kW at 100%FS 34 Physics based models Real time NN Answers Uncalibrated operating point Flow rate (CFM) x adaptive NN CFD Temp Data Point 4/9/2014

33 33 Other Ongoing Projects Energy-aware virtualization: o VM allocation, provisioning and migration in data centers and clouds o Per-VM energy budget based management Role of stacked DRAM and Hybrid Memory Cubes in Open Compute platforms 4/9/2014 ES2 BRIEFING

in collaboration with government and industry

will house I/UCRC in Energy- Smart Electronic

research laboratory is under construction:

34 Research infrastructure-binghamton SUNY Binghamton is a leader in microelectronics R&D in collaboration with government and industry New $30 million Center of Excellence building will house I/UCRC in Energy- Smart Electronic Systems A fully instrumented new data center research laboratory is under construction: comes online in Spring 2014 Data Center Research Lab at BU

35 35 The ES2 Data Center Research Lab 4700 square feet facility, 40 racks, 4 aisles, partitionable Fully instrumented to level of individual servers (power, load, temperatures, air flow, performance) Air, water and airside cooling, room-level CRACs and local cooling Waste heat reuse Additional facility for housing container- based data center at lower level Solar-power supplemented lighting Usage: research, benchmarking and best-practice development and education 4/9/2014 ES2 BRIEFING

36 36 Contact Information Current Members Dr. Bahgat Sammakia Binghamton University, Center Director Dr. Kanad Ghose Binghamton University Site Director Logistics & Additional Information: Andrea Palmeri Project Administrator 4/9/2014 E3S BRIEFING

The Center for Energy Smart Electronic Systems

The Center for Energy Smart Electronic Systems January 2014 an NSF Industry/University Cooperative Research Center Enabling Energy Efficient Data Centers for New York and the Nation The Center for Energy