Data Center Temperature Design Tutorial. Ian Seaton Chatsworth Products

Transcription

1 Data Center Temperature Design Tutorial Ian Seaton Chatsworth Products

2 What are we going to do today? Explore a process for determining what data center operating temperature will deliver the best total cost of ownership, while meeting uptime and reliability objectives, for a wide range of boundary variables

3 What are we going to do today? Background of ASHRAE environmental guidelines for data centers Difference between comfort cooling and data center cooling Minimum best practices Considerations for a design without cooling economization Considerations for a design with cooling economization Considerations for a design without mechanical cooling How to weigh impact of design elements on computer equipment reliability and availability BREAK

4 What are we going to do today? Quiz Design element assessment practice exercise Special considerations for a building retrofit project Special considerations for an efficiency upgrade project Summary/review

5 Learning Objectives You will understand how to integrate ASHRAE TCO guidelines into a building design project. You will be able to articulate the differences between comfort cooling and data center cooling. You will be able to apply the different data center class temperature thresholds to the process of evaluating relative merits of competing design concepts. You will be able to de-couple inherent power usage effectiveness (PUE) errors from total building energy use estimates. You will be able to calculate the impact on computer equipment reliability of specific building design strategies.

6 Appendix C Detailed Flowchart for the Use and Application of the ASHRAE Data Center Classes ANSI/BICSI Data Center Design and Implementation Best Practices Appendix G Design for Energy Efficiency

7

8 2015 Update Thermal Guidelines for Data Processing Equipment, ASHRAE, 2012

9 Dry Bulb Temperatures Server Classes Recommended Allowable A to to 89.6 A to to 95 A to to 104 A to to 113

10

11 Let s Get Started Desire to Reduce TCO and Explore Higher Temperatures My data center project is? New Construction Major Retrofit Existing, looking for Efficiency Gains

12 NOTES FOR APPENDIX C FLOW CHARTS 1. To use the highest inlet temperatures, ensure excellent airflow segregation is in place to avoid recirculation with warmer IT outlet flows. 2. Higher temperature IT loses its primary benefit if mixed with standard IT equipment, a common cooling system must meet the most demanding IT equipment requirements. 3. Ensure no chiller option choice meets availability requirements of the IT workload needs; investigate site extreme temperature and humidity conditions and economizer uptime risk. 4. Ensure the operational and safety aspects of high hot-aisle temperatures are understood; temperature of 140⁰F to 160⁰F may be expected. 5. Ensure higher airflow for IT equipment above the recommended range is understood. Data center airflow may have to increase dramatically.

13 New Construction Include best practices: airflow management, containment, VFD, control cooling at IT inlet Will you have an economizer? No Yes

14 Economizer? No Note 1 Operate data center within recommended environmental range at IT inlet. Investigate warmer chilled water temperature. Procure Class A1 or A2 ITE going forward

15 NOTES FOR APPENDIX C FLOW CHARTS 1. To use the highest inlet temperatures, ensure excellent airflow segregation is in place to avoid recirculation with warmer IT outlet flows. 2. Higher temperature IT loses its primary benefit if mixed with standard IT equipment, a common cooling system must meet the most demanding IT equipment requirements. 3. Ensure no chiller option choice meets availability requirements of the IT workload needs; investigate site extreme temperature and humidity conditions and economizer uptime risk. 4. Ensure the operational and safety aspects of high hot-aisle temperatures are understood; temperature of 140⁰F to 160⁰F may be expected. 5. Ensure higher airflow for IT equipment above the recommended range is understood. Data center airflow may have to increase dramatically.

16 Maximize Airflow Segregation Maximize Airflow Segregation When: T i max = TSLA Then: T i min - 10⁰F = LWT

17 Maximize Airflow Segregation

18 Economizer? No Note 1 Operate data center within recommended environmental range at IT inlet. Investigate warmer chilled water temperature. Procure Class A1 or A2 ITE going forward?

19 Where to procure Class A1 Servers? Where to procure Class A1 servers?

20 Economizer? No Note 1 Operate data center within recommended environmental range at IT inlet. Investigate warmer chilled water temperature. Assess accelerating technology refresh

21 Chiller Efficiency at Higher Set Points Why T i min is so important Chart from ASHRAE Data Center Design and Operation Book#6: Best Practices for Datacom Facility Energy Efficiency

22 With economizer Will you have an economizer? No Yes 1. Do detailed TCO analysis considering hours of economization at various Ti (including added cost of A3/A4 servers and increased energy use at higher Ti 2. Evaluate acceptable reliability of higher Ti 3. Assess impact of Class A3/A4 servers 4. Define optimum Tmax

23 Yes, with Economizer

24 TCO Worksheet, with economizer Sample Scenario Label Total 500kW IT load with 100 average 5kW each, in 5 rows of 20 cabinets each. Seven 30 ton N+2 redundancy, on slab floor with chimney cabinets

25 TCO Analysis Steps 1.Obtain relevant outdoor temperature bin data 2.Determine (estimate) total sensible cooling load 3.Obtain economizer operating cost estimate 4.Determine cost of expediting technology refresh from legacy Class A2 servers and cost between A3 and A4 servers

26 Outdoor Ambient Operating Conditions limatedata.html Content/Tools/NAmericanFreeCooling Tool

27 Water-Side Economization Parallel Economizer On or Off (may limit free cooling hours) Simple pump control Dead band increases approach temperature Should be controlled by engineer Easier retrofit Better ROI for larger data centers Long, cold winter Series Economizer Partial economization adds free cooling Complex pump control Approach figured on return temperature Can be controlled by automation More complex retrofit Better ROI for smaller data centers Temperate climate

28 Affinity Law Fan Energy Economies 10 CRAH units with 12 Hp fan motors 797, % rpm 581,130 90% rpm 408,146 80% rpm 273,426 70% rpm

29 Affinity Law Fan Energy Economies 2 MW data center case study example, N+2 CRAH redundancy $290,765 to operate 44 94% fan rpm $42,075 to operate 44 49% fan rpm $141,417 to operate 24 90% rpm $500,000 capital expense avoidance Year One $1,390, CRAHs and no containment $1,142, CRAHs and containment $741, CRAHs and containment

30 Affinity Law Fan Energy Economies 2 MW data center case study example, N+2 CRAH redundancy $290,765 to operate 44 94% fan rpm $42,075 to operate 44 49% fan rpm $141,417 to operate 24 90% rpm $500,000 capital expense avoidance Year Five $2,553, CRAHs and no containment $1,307, CRAHs and containment $1,307, CRAHs and containment

31 Affinity Law Fan Energy Economies 2 MW data center case study example, N+2 CRAH redundancy $290,765 to operate 44 94% fan rpm $42,075 to operate 44 49% fan rpm $141,417 to operate 24 90% rpm $500,000 capital expense avoidance Year Ten $4,087, CRAHs and no containment $1,518, CRAHs and containment $2,014, CRAHs and containment

32 Energy Use in Data Centers IT Energy (UPS Load) Cooling equipment Power Conversion Lighting

33 Cooling Requirements in Data Centers IT Load Electrical room Battery room Power substation CRAC/CRAH fan motors Power Conversion Losses Lighting Make-up air Pump motors Bypass air Building and room skin losses

34 Energy Use in Data Centers

35 Determine Sensible Cooling Load

36 Forecast the Life of the Data Center Source: Datacom Equipment Power Trends and Cooling Applications, 2 nd edition, ASHRAE, 2012, Table 4.1 Volume Server Power Trends

37 Apply Power Trends to Plan How many technology refreshes is the data center being planned to see? How much expected application growth? Plot refresh growth from point on the trend curves where you are today. Determine how much of the forecasted load to include in initial capacity.

38 Using CFD to fine-tune TCO Analysis 64% airflow volume < 1.5% over-supply Higher supply air temperature

39 Using CFD to fine-tune TCO Analysis All server inlet temperatures are within recommended environmental envelope

40 Server Airflow versus Temperature 1.4 = 40% increase in airflow 40% increase in airflow = 274% increase in fan energy (1.4 3 = 2.74) Example 700 watt server with 70 watts for fan 68 Fan energy increased to Thermal Guidelines for Data Processing Environments, Fourth Edition, ASHRAE Total server energy increased from 700 to

41 Server Airflow versus Temperature Example Implications 500 servers = 840,000 kw hours hours = 1.5 million kw hours avoidance 4 ½ year = 750,000 kw hours avoidance Realistic scenario 10% of year running CRACs when ambient exceeds 86 50% of year running no CRACs % of year running no CRACs below 68

42 Server Airflow versus Temperature Convection heat transfer from a surface to an airflow is governed by the equation q = ha(t w T f ) Where q = the heat transfer rate h = the convection heat transfer coefficient A = the surface area T w = the temperature of the surface T f = the temperature of the fluid Sensible cooling CFM = 3.1W ΔT Where CFM = cubic feet per minute airflow w = watts (sensible heat load) ΔT = temperature rise in ⁰F

43 Server Airflow versus Temperature Hewlett-Packard research project reported at 2011 Uptime Symposium Server Inlet Total Energy Temperature Cost _ PUE 68⁰F $ ⁰F $ ⁰F $ ⁰F $

44 Thermal Guidelines for Data Processing Environments, Fourth Edition, ASHRAE Server Airflow versus Temperature

45 Server Airflow versus Temperature Hypothetical Illustrative Example One Megawatt data center watt servers A3 95 F = 8% fan energy penalty (or watts) Build new data center in Seattle

46 Server Airflow versus Temperature Seattle Illustrative Example with A3 Servers 3 hours a year at or above 96 3 hours a year right at 95 3 hours a year at hours a year at hours a year at hours a year at hours a year at hours at 68 F or lower

47 Server Airflow versus Temperature 3 hours on chiller = 0 penalty 3 8% $.10 per kw/h = $0.24 penalty 3 5% $.10 per kw/h = $0.15 penalty % $.10 per kw/h = $0.85 penalty % $.10 per kw/h = $2.57 penalty 220 1% $.10 per kw/h = $2.22 penalty 442 $.10 per kw/h = $2.27 penalty $.10 per kw/h = $0 penalty Total fan energy penalty for Class A3 servers in 1MW Seattle data center = $8.25

48 Server Airflow versus Temperature WITH ALLOWABLE MAXIMUM SERVER INLET OF 95 Total energy savings for 8757 hours without a chiller = $195, : 200% up to 700% 5 year ROI 33 to 83 week payback 2015 and following: 2.4 million % 1 year ROI 22 minute payback

49 Server Airflow versus Temperature San Antonio Illustrative Example with A3 Servers 9 hours a year at or above 96 4 hours a year right at hours a year at hours a year at hours a year at hours a year at hours a year at hours at 68 F or lower

50 Server Airflow versus Temperature 9 hours on chiller = 0 penalty 4 8% $.10 per kw/h = $3.20 penalty 161 5% $.10 per kw/h = $80.50 penalty % $.10 per kw/h = $ penalty % $.10 per kw/h = $ penalty % $.10 per kw/h = $ penalty $.10 per kw/h = $84.60 penalty $.10 per kw/h = $0 penalty Total fan energy penalty for Class A3 servers in 1MW San Antonio data center = $805.30

51 Server Airflow versus Temperature WITH ALLOWABLE MAXIMUM SERVER INLET OF 95 Total energy savings for 8751 hours without a chiller = $195, : 200% up to 650% 5 year ROI 34 to 84 week payback 2015 and following: 24,134% 1 year ROI 1.5 day payback

52 Cooling Cost Roll-up, example 12kW to operate 5 cooling 64% capacity 41.8kW to operate 65 F H 2 O leaving temperature 0.06 kw for fresh air (N+1) 0.56kW for electrical room (N+1) 0.06kW for battery room (N+1) 15kW for substation (N+1, but not simultaneously running redundancy 608,119 kw hours per year $58,800 cooling budget at $0.10 per kw hour (.23 ANNUALIZED MECHANICAL LOAD COMPONENT)

53 TCO Calculation

54 TCO Calculation

55 TCO Calculation

56 The Big Worry: How to know the effect of temperature on ICT equipment reliability/life? How to know how long it is OK to operate in allowable temperature range? The X Factor 1. Baseline = 24/7 68 F server inlet temperature 2. OEM historical data on user failure reports 3. Performance variations above, at and below baseline Premise is that the data center temperature follows mother nature

57 Evaluate acceptable reliability of higher Inlet Temperatures T i Failure Rate x-factor (⁰F) Lower Bound Average Bound Upper Bound Thermal Guidelines for Data Processing Environments, Fourth Edition, ASHRAE TC9.9

58 Implications of X - Factor If 1000 servers produced 30 failures during warranty period (3% failure rate) running all year at 68⁰F, then those servers would produce 44 failures running all 90⁰F inlet temperature BUT 438 hours below 59⁰F 876 hours at 59⁰F F ⁰F ⁰ ⁰F ⁰F Results in 29 failures, or one fewer than running all 68⁰F

59 Implications of X - Factor Thermal Guidelines for Data Processing Environments, Fourth Edition, ASHRAE TC9.9

60 Evaluate acceptable reliability of higher Inlet Temperatures Time at temperature weighted failure rate calculations for IT equipment in Chicago Inlet Temperature The X factor % of Hours 59⁰F T 68⁰F ⁰F T 77⁰F ⁰F T 86⁰F ⁰F T 95⁰F % 17.2% 10.6% 4.6% 2011Thermal Guidelines for Data Processing Environments Expanded Data Center Classes and Usage Guidance, White Paper, ASHRAE TC9.9 Net x-factor = 0.99

61 TCO Preliminary Results Analysis shows positive returns? No Yes No Will you have a chiller? Yes Operate data center within recommended environmental range at IT inlet. Investigate warmer chilled water temperature. Procure Class A1 or A2 ITE Think Technology Refresh

62 Economizer with Chiller Notes 4,5 Yes Set economizer to provide free cooling from 59⁰F to lowest TCO Tmax, select ITE for Ti as: A1: 90⁰F A2: 95⁰F A3: 104⁰F A4: 113⁰F Ensure all data center components capable of meeting ASHRAE Class

63 NOTES FOR APPENDIX C FLOW CHARTS 1. To use the highest inlet temperatures, ensure excellent airflow segregation is in place to avoid recirculation with warmer IT outlet flows. 2. Higher temperature IT loses its primary benefit if mixed with standard IT equipment, a common cooling system must meet the most demanding IT equipment requirements. 3. Ensure no chiller option choice meets availability requirements of the IT workload needs; investigate site extreme temperature and humidity conditions and economizer uptime risk. 4. Ensure the operational and safety aspects of high hot-aisle temperatures are understood; temperature of 140⁰F to 160⁰F may be expected. 5. Ensure higher airflow for IT equipment above the recommended range is understood. Data center airflow may have to increase dramatically.

64 CRAC/CRAH Selection Cooling capacity requirement (growth + redundancy) Efficiency (EER/SEER or COP) CRAH-1w/EC EER = 58 CRAH-2wEC EER = 133 CRAH-3 centrifugal EER = 43 Close-Coupled EER = 38 CRAC-1 EER = 14 CRAC-2 EER = 11 kw per ton Chiller New Chiller Existing Reciprocating.78 to to 1.2 or higher Screw.62 to to.85 or higher Centrifugal High.50 to.62 NA Centrifugal Moderate.63 to to.80 or higher CRAC (DX) 1.0 to to 2.5 or higher Free-cooling opportunities

65 Economizer with Chiller Notes 4,5 Yes Set economizer to provide free cooling from 59⁰F to lowest TCO Tmax, select ITE for Ti as: A1: 90⁰F A2: 95⁰F A3: 104⁰F A4: 113⁰F Ensure all data center components capable of meeting ASHRAE Class

66 Annual Hours of Operation with Mechanical Cooling and Class A2 Servers Recommended inlet temperatures Class A2 allowable inlet temperatures Economizer with Data Center Containment HA/CA Water-Side Wheel Air-Side IEC Chicago 70% 14.6% 14.6% 12.7% 10.2% 70% 0% 0.2% 0% 0% Seattle 92.5% 4% 2.9% 2.4% 0% 92.5% 0% 0.1% 0% 0% San Antonio 98% 58.4% 36.5% 33.4% 42.2% 98% 0.9% 1% 0.1% 18.6% Denver 71.9% 6.8% 13% 12% 22.2% 71.9% 0% 2% 0.7% 0%

67 Annual Hours of Operation with Mechanical Cooling and A3 Servers Recommended inlet temperatures Class A3 allowable inlet temperatures Economizer with Data Center Containment HA/CA Water-Side Wheel Air-Side IEC Chicago 70% 14.6% 14.6% 12.7% 10.2% 70% 0% 0% 0% 0% Seattle 92.5% 4% 2.9% 2.4% 0% 92.5% 0% 0% 0% 0% San Antonio 98% 58.4% 36.5% 33.4% 42.2% 98% 0.6% 0% 0% 0.6% Denver 71.9% 6.8% 13% 12% 22.2% 71.9% 0% 0% 0% 0%

68 Economizer without Chiller Notes 3,4,5 No Set economizer to provide free cooling from 59⁰F to maximum annual outdoor temperature, select ITE for Ti as: A1: 90⁰F A2: 95⁰F A3: 104⁰F A4: 113⁰F Ensure all data center components capable of meeting ASHRAE Class

69 NOTES FOR APPENDIX C FLOW CHARTS 1. To use the highest inlet temperatures, ensure excellent airflow segregation is in place to avoid recirculation with warmer IT outlet flows. 2. Higher temperature IT loses its primary benefit if mixed with standard IT equipment, a common cooling system must meet the most demanding IT equipment requirements. 3. Ensure no chiller option choice meets availability requirements of the IT workload needs; investigate site extreme temperature and humidity conditions and economizer uptime risk. 4. Ensure the operational and safety aspects of high hot-aisle temperatures are understood; temperature of 140⁰F to 160⁰F may be expected. 5. Ensure higher airflow for IT equipment above the recommended range is understood. Data center airflow may have to increase dramatically.

70 Economizer without Chiller Notes 3,4,5 No Set economizer to provide free cooling from 59⁰F to maximum annual outdoor temperature, select ITE for Ti as: A1: 90⁰F A2: 95⁰F A3: 104⁰F A4: 113⁰F Ensure all data center components capable of meeting ASHRAE Class

71 Final Note No single design concept is best for all situations Well executed airflow management is the foundation for everything Newer ICT equipment opens more design options Free cooling does not necessarily mean exposure to outside elements ICT procurement/specification decisions affect mechanical design and performance Mechanical design decisions affect ICT deployment Consider EVERYTHING while you are still in design phase

72 10 MINUTE BREAK

73

74 QUICK REVIEW

75 What do you remember from before the break? What are the differences between comfort cooling and data center cooling?

76 What do you remember from before the break? What are the ASHRAE recommended temperature thresholds?

77 What do you remember from before the break? What are the ASHRAE allowable temperature thresholds for class A3 servers?

78 What do you remember from before the break? Where can you buy Class A1 servers today?

79 What do you remember from before the break? How can your PUE go down and your operating costs actually increase for doing the same amount of work?

80 What do you remember from before the break? How do you determine if your PUE anomaly is actually a serious issue?

81 What do you remember from before the break? What might make a Q factor of 1.3 acceptable?

82 Let s apply that 1. Open file: BICSI Practice Exercise w-omaha data.xls from the provided USB device 2. Go to the Input tab 3. Enter data only in yellow shaded cells 4. N.B.: Only ONE approach temperature (not both wet and dry) 5. Use AFM factor (airflow management) on card provided 6. You can pick maximum server inlet temperature, class of servers, and type of economization

83 Why We re not Examining Class A1 Servers Today Class A1 Sever Chart shows fan airflow Each point is an airflow that needs to be cubed to get energy 95 F is or 564% 40 watts X 564% = watts 400 watts 40 watts watts = watts 2500 servers = 1464KW total server energy = 464kW server fan energy penalty

84 Why We re Examining Class A2 and A3 Servers Today Class A2 and A3 charts show actual server energy Each point is a total server energy variation 95 F = or 112.5% 400 watts X 112.5% = 450 watts 2500 servers X 450 watts = 1125kW total server energy = 125kW server fan penalty

85 Let s apply that 1. Open file: Omaha BICSI Practice Exercise.xls from the provided USB device 2. Go to the Input tab 3. Enter data only in yellow shaded cells 4. N.B.: Only ONE approach temperature (not both wet and dry) 5. Use AFM factor (airflow management) on card provided 6. You can pick maximum server inlet temperature, class of servers, and type of economization 7. Enter server failure experience 8. Enter actual server inlet temperature experience 9. Short report on findings

86 Let s apply that Points of reference: 2.0 PUE = 8,760,000 kw hours for cooling energy 1.5 PUE = 3,504,000 kw hours for cooling energy 1.25 PUE = 1,314,000 kw hours for cooling energy WRITE THAT DOWN!!!

87 Let s apply that GO TO WORK WHAT DID YOU DISCOVER?

88 The Rest of the Story Desire to Reduce TCO and Explore Higher Temperatures My data center project is? New Construction Major Retrofit Existing, looking for Efficiency Gains

89 Major Retrofit Include best practices: airflow management, containment, VFD, control cooling at IT inlet Yes Do you have or are you adding an economizer? Is the new space thermally and airflow segregated from existing IT? Yes No No

90 With Economizer and Segregated New Space Yes 1. Do detailed TCO analysis considering hours of economization at various Ti (including added cost of A3/A4 servers and increased energy use at higher Ti 2. Evaluate acceptable reliability of higher Ti 3. Assess impact of Class A3/A4 servers 4. Define optimum Tmax No Analysis shows positive returns? Yes Will you have a chiller? Yes No

91 With Economizer and Segregated New Space Segregated high density space with cold aisle containment and partitioned return air space

92 Desire to Reduce TCO and Explore Higher Temperatures My data center project is? New Construction Major Retrofit Existing, looking for Efficiency Gains

93 Existing Data Center, Looking for Efficiency Gains Include best practices: airflow management, containment, VFD, control cooling at IT inlet Implement best practices for airflow management, hot/cold aisle, containment, blanking panels, correct airflows, VFDs (see BICSI , ASHRAE Datacom book series and EU CoC best practices No Do you have good airflow mgt? Rack Cooling Index & Return Temp. Index? Yes

94 Rack Index Over Temperature Rack Cooling Effectiveness in Data Centers and Telecom Central Offices: The Rack Cooling Index (RCI), Magnus Herrlin, ASHRAE Transactions, Volume 111, Part 2 (DE )

95 Rack Index Under Temperature Rack Cooling Effectiveness in Data Centers and Telecom Central Offices: The Rack Cooling Index (RCI), Magnus Herrlin, ASHRAE Transactions, Volume 111, Part 2 (DE )

96 Rack Index Under Temperature Rack Cooling Effectiveness in Data Centers and Telecom Central Offices: The Rack Cooling Index (RCI), Magnus Herrlin, ASHRAE Transactions, Volume 111, Part 2 (DE ) Ian s suggestion for an improvement target: Set your minimum recommended temperature within 3⁰F of your maximum recommended temperature and then set your minimum allowable temperature 1-2⁰F below your minimum recommended temperature and set 100% as your project target

97 Existing Data Center, Looking for Efficiency Gains With Effective Airflow Management Yes Correct control strategy No Do you measure/ control IT inlet temperature? Yes Do you have an economizer? No Yes

98 NOTES FOR APPENDIX C FLOW CHARTS 1. To use the highest inlet temperatures, ensure excellent airflow segregation is in place to avoid recirculation with warmer IT outlet flows. 2. Higher temperature IT loses its primary benefit if mixed with standard IT equipment, a common cooling system must meet the most demanding IT equipment requirements. 3. Ensure no chiller option choice meets availability requirements of the IT workload needs; investigate site extreme temperature and humidity conditions and economizer uptime risk. 4. Ensure the operational and safety aspects of high hot-aisle temperatures are understood; temperature of 140⁰F to 160⁰F may be expected. 5. Ensure higher airflow for IT equipment above the recommended range is understood. Data center airflow may have to increase dramatically.

99 Final Note No single design concept is best for all situations Well executed airflow management is the foundation for everything Newer ICT equipment opens more design options Free cooling does not necessarily mean exposure to outside elements ICT procurement/specification decisions affect mechanical design and performance Mechanical design decisions affect ICT deployment Consider EVERYTHING while you are still in design phase

100 QUESTIONS? Ian Seaton Chatsworth Products, Inc. (512)