Data Center Trends and Power Management Roger Schmidt IBM Chief Thermal Architect Sept 12, 2006
Contents Power Trends/Impacts on Data Center IBM s Product Innovation Industry Actions Final thoughts
What Customers are Asking I am out of power in my data center, so what can be done? How do I handle the hot spots in my data center resulting from high server powers? How do I get real time information from servers on temperature and power so I can run my data center more energy efficiently?
Facility/Network Concerns Excessive heat Insufficient power Insufficient raised floor Excessive facility cost Poor location None of the above 0% 5% 10% 15% 20% 25% 30% 35% Power and cooling will be a top 3 issue with all CIO s in the next 6-12 months Michael Bell Gartner Group
Everyone Has A Computer Room Branch Office Client/Web Hosting Site
IT Equipment is HOT And HEAVY P Series 595 31 W x 66 D x 80 H 23 kw 3014 Lbs Toyota Camry 3276 Lbs P-Series 690 62 W x 50 D x 80 H 32 kw 4233 Lbs
New Datacom Equipment Power Density Chart
Rack Level Heat Load Trend 35 Rack Level Heat Loads Rising at an Exponential Rate Rack Level Heat Load (kw) 30 25 20 15 10 5 1994 1996 1998 2000 2002 2004 2006 Year of Announcement
System Power Comparison System Power Varies as a Function of Configuration and Workload It is NOT Just the Processor 100% 90% 80% 70% 60% 50% 40% 30% Processor / Cache Memory IO Subsystem Cooling Power 20% 10% 0% Blade p520 p575 p595 z990 z900
Basic Data Center Air Flow HOT OUTPUT RETURN AIR - Perforated tiles do not support high flowrates required by high density servers - Underfloor obstructions from chilled water pipes and cables RAISED FLOOR AREA 3 Server Rack 2 4 COLD INPUT AIR PEDESTAL A/C unit 1 CONCRETE SUBFLOOR
Air Blockages beneath Raised Floor Chilled Water Pipes Cables
The Problem in Data Centers Chilled air does not reach top of racks Cold Aisle Hot Aisle Cold Aisle
CFD Model of a Raised Floor Data Center Computational Fluid Dynamics Highly complex flow patterns and localized rack inlet air heating focus of many studies by R. Schmidt and Co-workers (1997-2004)
The New Economics of IT; A Paradigm Shift Power and cooling spend will exceed new server spending (Gartner 2006) 2000 Raw processing horsepower is the primary goal, while the infrastructure to support it is assumed ready 2006 Raw processing horsepower is a given, but the infrastructure to support deployment is a limiting factor Three Cooling Challenges 1. The System 2. The Rack 3. The Datacenter Spending (US$B) $90 $80 $70 $60 $50 $40 $30 $20 $10 $0 New server spending Power and cooling U.S. Market 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Installed base (M units) 18 16 14 12 10 8 6 4 2 0
Contents Power Trends/Impacts on Data Center IBM s Product Innovation Industry Actions Final thoughts
IBM delivers business value with innovation at all levels Information on Demand Capacity on Demand SOA Grid Computing Infrastructure Infrastructure Solutions Supercomputing Capacity on Demand x86 Product Families Mainframe Technology & Packaging Storage IBM Power z/architecture Architecture Blades Cell Broadband X-Architecture Engine
IBM POWER Architecture Track Record of Consistent Predictable Delivery 2001 2003 POWER4+ 2004 POWER5 2006 POWER5+ 2007 POWER6 POWER4
Power 6 will provide leadership for years to come Innovations in all areas of system design System Energy Management RAS Performance Increased performance with increased energy efficiency Flexibility Features and function Technology Current development Status
The Datacenter IBMs focus on power efficiency allows our systems to run with less power than our competitors For some customers this level of efficiency is still inadequate Older data center floors were not designed to handle the heat load that today's servers place on it Many customers run out of power and cooling before running out of rack space For these customers, IBM has developed a suite of additional tools to employ; Hot Spot Area IBM Cool Blue rack based heat exchanger Smart Tools Power Calculator, Power Executive Virtualization Cool blue Solution Rear Door Heat Exchanger
Understanding Where Costs are Critical Normal Operation of Server Air flow IBM s s Cool Blue Rear Door Heat exchanger can remove over 50% of a rack s s heat output Front Cold Back Hot No new fans or electricity needed. Attaches to back of rack (adds 5 ) No rearrangement of datacenter Cost effective; 1KW cooling = $286 Perf tile Cable Opening Tile floor Underfloor Chilled Air Subfloor Improved Operation of Server Rear Door Heat Xchanger Air flow The Cool Blue Heat exchanger adds cooling capacity at ~1/4 the cost of traditional methods Perf tile Front Cold Back Hot IBM Enterprise Rack Cable Opening Underfloor Chilled Air water lines Tile floor Subfloor Rear Door Heat Exchange
Comparison with and without Cool Blue
Typical Data Center Power Distribution and Energy Savings with Water Server Load 46.2% Example: Air Cooled Server Racks Electricity Consumption(kW) with Air Cooled Servers Other 12.4% Lighting 2.2% Office Space Conditioning 1.3% Electrical Room Cooling 3.6% Cooling Tower Plant 3.5% Data Center Cooling 30.8% Electricity Consumption(kW) with Water Cooled Servers Description Other 402 402 Lighting 73 73 Office Space Conditio 41 41 Electrical Room Coo 118 118 Cooling Tower Plant 114 114 Data Center Cooling 1000 418 Server Load 1500 1500 Total Load 3248 2666 Server Load 56.3% Water Cooled Server Racks Other 15.1% Lighting 2.7% Office Space Conditionin 1.5% Electrical Room Cooling 4.4% Cooling Tower Plant 4.3% Data Center Cooling 15.7% Savings of ½ million dollars/year at $0.10 / kw-hr
Livermore Data Center 3 rd Fastest Supercomputer High Power Rack Clusters within 10,000 sq ft can require 10 MW including supporting power and cooling 77.8 77.9 80.1 83.1 80.7 77.9 86.5 86.1 87.2 85.2 86 85.1 85.8 85 76.7 Annual Energy Cost for 10 MW at $0.10/kW-hr = $8.8 M 75 New Building Cost for 10,000 sq ft = $15 M
Energy Cost Around the World (US cents/kw-hr) GERMANY 7.7 AK 10.99 UK 6.7 CA 11.53 TX 7.90 MN 6.31 NY 12.98 NC 6.70 FRANCE 5.0 ITALY 16.2 JAPAN 12.7 KOREA 5.9 TAIWAN 5.7 HI 16.19 BRAZIL 4.7 CHINA 15.0 AUSTRALIA 6.1 ARGENTINA 3.3 NEW ZEALAND 6.0
IBM Site and Facilities Services - Providing Data Center Solutions Deep Global Expertise Dedicated team since 1986 Mechanical & Electrical Engineers, Space planners and PM s on staff Most with over 20 years experience A team of 450 people globally Designed and built over 30 million SF of raise floor from Boulder and Brussels to Bangalore We have learned what works because we use it Built and run over 400 of our own datacenters, with 6M sq ft currently in operation Built over 120 of our own disaster recovery facilities that support our clients Built and reside in 4 of top 10 green buildings in Japan Built state of the art clean rooms for our own leading processor technology manufacturing Clear understanding of current technologies and a vision of future with tight linkage to product development and research teams Well established partnerships with key providers ensuring best choices to fill your requirements including: APC, Liebert, etc.
IBM and the Energy Efficient Data Center IBM servers provide leadership energy efficiency today Power5 efficiency continuing to Power6 Server power management features - PowerExecutive IBM is delivering leadership energy management technology Enable customers to maximize datacenter computational efficiency Enable customers to minimize datacenter operating costs IBM has the skill and resources to assist our customers in planning and deploying data center solutions IBM s long term commitment to providing solutions for our customers Source:Brad McCredie, Analyst meeting, July 20, 2006
Contents Power Trends/Impacts on Data Center IBM s Product Innovation Industry Actions Final thoughts
Worldwide Focus on Climate/Energy Energy Usage becoming a critical global issue Climate Change Energy Security and Availability Energy Cost Attracting the attention of regulators, legislators, and customers Carbon caps Energy Efficiency Energy Saving Law Japan Energy Star EPA researching server farm energy usage Green grid Best practices in a data center Benchmarks on power/performance
IBM Providing Industry Leadership - ASHRAE TC9.9 Committee Chair of committee : Roger Schmidt IBM Chief Thermal Architect Other Books being developed Contamination in datacom equipment center High density data centers - case studies and other considerations Seismic considerations in datacom equipment centers Current practices in Datacom Facility TCO analysis and Energy Efficiency Source: Roger Schmidt
Thermal Guidelines Book Chapter 1 Introduction Overview Chapter 2 Equipment Environment Specifications Chapter 3 Environmental Measurement Chapter 4 Equipment Placement & Airflow Patterns Chapter 5 Equipment Manufacturer s Heat & Airflow Reporting People from these firms participated: ANCIS, APC, Ceyba, Cisco, CRS, Cray, DataAire, Dell, DLB, Echelon, EMC, Engineered Refrigeration Sys, Flomercs Inc, Fujitsu, Hellmer Medved Engineers, Hitachi, HP, IBM, Innovative Research, Intel, LBNL Labs, Liebert, Lucent, Mallory & Evans, Motorola, NCR, Sun, Telecordia, Unisys, Uptime Institute
Power Trend Book - Overview Chapter 1 Introduction Chapter 2 Background Chapter 3 Load Trends & their Applications Chapter 4 Air Cooling of Computer Equipment Chapter 5 Liquid Cooling of Computer Equipment Appendix A Collection of Terms Appendix B Additional Trend Chart Information / Data Appendix C Electronics, Semiconductors, Microprocessors, ITRS Appendix D Micro Macro Overview of Datacom Equipment Packaging 2005 ASHRAE TC9.9 People from these firms participated: Alcatel, ANCIS, ATI, Cisco, Cray, DataAire, Dell, DOD, DLB, EMC, EYP, Fanniemae, Freescale, Fujitsu, Hellmer Medved Engineers, HP, IBM, Intel, LBNL Labs, Liebert, Mallory & Evans, Motorola, Nortel, Sun, Syska & Hennesey, Taylor Engineering, Uptime Institute
Data Center Considerations - Overview Part 1 Datacom Facility Basics Chapter 1 Introduction; Chapter 2 Design Criteria; Chapter 3 HVAC Load Considerations; Chapter 4 Computer Room Cooling Overview; Chapter 5 Air Distribution; Chapter 6 Liquid Cooling Part 2 Other Considerations Chapter 7 Ancillary Spaces; Chapter 8 Contamination; Chapter 9 Acoustical Noise Emissions; Chapter 10 Structural & Seismic; Chapter 11 Fire Detection & Suppression; Chapter 12 Commissioning; Chapter 13 Availability & Redundancy; Chapter 14 Energy Efficiency People from these firms participated ANCIS, APC, Bell South, Citigroup, Data Aire, Dell, DOD, DLB, EDS, EYP, Fannie Mae, Fluent, Fujitsu, Heapy Engineering, HP, IBM, Intel, LBNL, Liebert, Mallory & Evans, Nelson Acoustical, Nortel, Rice University, Stulz, Sun, Syska & Hennessy, Tier 4 Consulting, Wright Line
Liquid Cooling Book Overview Chapter 1 Introduction Chapter 2 Cooling Services for Equipment Cooling Systems Chapter 3 Facility Piping Architecture Chapter 4 Liquid Cooling Implementation for Datacom Equipment Chapter 5 Liquid Cooling Infrastructure Requirements for Chilled Water Systems Chapter 6 Liquid Cooling Infrastructure Requirements for Chilled Water Systems and Datacom Equipment Cooling Systems People from these firms participated: APC, Aavid, Cray, DataAire, Dell, DLB, EYP, Hellmer Medved Engineers, HP, IBM, Intel, Liebert, Lytron, Mallory and Evans, NCR, NSA, Panduit, Rittal, Sanmina, SGI, Spraycool, Sun, Trane
Contents Power Trends/Impacts on Data Center IBM s Product Innovation Industry Actions Final thoughts
Final Thoughts Thermal issues will become ever more severe over time, and fundamental innovations are required to provide a route forward Power efficient products as revealed today resolve customer challenges already limiting their ability to leverage existing assets This is not going to get any easier