Power consumption and efficiency of cooling in a Data Center

Transcription

1 Power consumption and efficiency of cooling in a Data Center Satoshi Itoh, Yuetsu Kodama, Toshiyuki Shimizu and Satoshi Sekiguchi (AIST), Hiroshi Nakamura (The University of Tokyo), Naohiko Mori (NTT Communications Corporation) 1 This research was partially supported by the New Energy and Industrial Technology Development Organization (NEDO) research project entitled Research and Development Project for Green Network/System Technology (Green IT Project)

2 Motivation and Purpose 2 There are many different ways to save energy in a data center. Lower power server More efficient power supply and cooling facility More efficient server operation How much does the improvement contribute to energy efficiency? Necessity of green metric Existing green metrics: PUE and server green metric PUE is too macroscopic Pitfalls Development of model and metrics for data center and server system In this paper, measure various temperature and power consumption using our testing laboratory and discuss mainly on cooling facility and fan Examples Data center A Data center B Server Performance 50GFlops 50GFlops Performance/Watt ( GFlops / W) Power facility Cooling Facility Large Fan PSU with UPS Alternating current, no UPS Low airflow No fan DC/DC trans. no UPS Direct current with UPS Heat remove by air pressure Total power / year 30GWh 30GWh PUE* *PUE=Total power consumption / power consumption by IT loss loss

3 Model of Power Consumption for Data Center Power Consumption of Datacenter Pdc Power consumption of IT equipment Pit Power loss in Power Unit Ppu Ep:Power efficiency Power consumption for Cooling Facility Pcf Ec:Cooling efficiency COP : Coefficient of Performance NW Pns Storage Pst Server Psv 3 Pdc : Total Power Consumption of Data Center Pit : IT equipment Pns : Network switch Pst : Storage Psv : Server Ppu : Power Unit Pcd : Cooling Device Pdc = Pit + Ppu + Pcf Pit = Pns + Pst + Psv Pit / (Pit + Ppu) = Ep Pcd = Pit Ec Ep : Power efficiency of Power Unit Ec : Cooling efficiency

4 Model of Power Consumption for Server Components FAN Pf Motherboard Pb Disk Pd Memory Pm CPU Pc PSU Pp In operation Po Psv FAN rpm Mother board Disk access Mem access CPU load PSU Pt : Total Server Power Consumption = Psv Pb : mother board (almost const.) Pd : Disk Pm : Memory Pc : CPU Pf : FAN Pp : power loss in PSU Po / (Po + Pp) = Epsu Pf = Pf_0 (rpm)^3 rpm = rpm_min + f(server Temp.) Pc = Pc_idle + load * (Pc_max - Pc_idle) Pm = Pm_no-access + Pm_access Pd = Pd_no-access + Pd_access 4

5 5 Overview of the testing laboratory

6 Floor layout of testing laboratory Photo of CRAC CRAC #13 Temperature measurement points CRAC #14 CRAC #15 Booth and vinyl curtain on the floor CRAC #16 Wall #4 #5 #6 #7 #1 #2 #3 Flow panel under the floor Rack Louver (Grill) 600mm 600mm

7 Flow control panel under the floor Separate the space under the floor using cardboards 7

8 Side view of testing laboratory Temperature measurement points Booth and vinyl curtain Ceiling Wall CRAC Rack Floor Flow panel under the floor

9 IT equipment 9 1U servers IBM x3250 Xeon G, 90 nodes NEC Express 5800/i120Rg-1 Xeon 2.33G, 20 nodes Storage IBM System Storage DS3400 Dual FC 42X IBM System Storage DS3200 Dual SAS 22X HP StorageWorks Modular Smart Array 2012FC Blade servers HP BladeSystem c7000 X2 / BL460c Xeon5350 Quad 2.66GHz, 3 nodes Xeon5160 Dual 3.00GHz, 13 nodes Xeon5160 Dual 3.00GHz 2CPU, 16 nodes Web server Rack #1-4, 6 Blade server Rack #5 NW switch (Mgt) NW switch (Data) Web server Web server Web server Web server 32 Blade enclosure 2 32 Filler panel Power meters KVM switch Filler panel 21 Filler panel 21 Web server Web server Web server Web server Web server Web server Web server Web server DB server DB server Storage Web server 8 8 Web server 7 7 Web server 6 6 Web server 5 Blade enclosure 1 5 Web server 4 4 Web server 3 3 Web server 2 2 Web server 1 1

10 Power meter at Power unit and CRAC Yokokawa Cramp Power Meter CW120 Power consumptions per rack/balde chassis/ CRAC are measured every 2 seconds Power unit CRAC 10

11 Power meter for individual equipment Ohsaki Electric Watt Checker MWC-01 Power consumptions of 128 equipment are measured every second Measured data is recorded through USB network Voltage(RMS) (unit) Range Accuracy (V) AC100V±10% AC200V±10% ±1% Current(RMS) (A) 0.00~20.00 ±1% Power (W) 0~2200(AC100V) 0~4400(AC200V) ±2% Frequency (Hz) 47.0~63.0 ±2% power factor 0.00~1.00 ±0.03 Electric Energy (kwh) 0.00~9999 ±2% Output interval 1 second 11

12 Temperature sensors GRAPHTECmidi LOGGER GL800 with type K thermocouple Temperature at 87 points is measured every 1 second 12

13 Power (W) LINPACK Whole of IT equipment consumes maximally 27kW Rack Server CPU (CPU=2core) Freq. (GHz ) # of nodes Stand by (W) Idle power (W) LINPACK G Flops Power (W) per node GFlops /W Rack1,2,3,4 IBM x3250 Xeon Rack 6 NEC Express Xeon 5148 X Rack5bdcb1 HP BL460 (a) Xeon 5355 x 1 (CPU=4core) HP BL460 (b) Xeon 5160 x Rack5bdcb2 HP BL460 (c) Xeon 5160 X per rack/chassis kW 3.1kW 3.1kW 3.2kW 3.2kW 4.9kW 3.7kW linpack rack6(w) rack4(w) rack3(w) rack2(w) rack1(w) bdcb2(w) bdcb1(w) idle IBM x3250 NEC Express HP BL460 (a) HP BL460 (b) HP BL460 (c) 13

14 spped of virtual fan (%) Power consumption of Blade Fan Power (W) HP BladeSystem c7000 has many sensors We measured power consumption and speed of fans with different kinds of load: idle, LINPACK and SPECpower Speed of fan increases linearly with CPU temperature after 57 Power consumption of fan can be represented by constant and cube of fan speed. Power = 22.1 (rpm/10000)^ Fans consume roughly 0.8~1kW which is 16~20% of system y = x power = 22.1(rpm/10000) processor temp. (degree) ,000 10,000 15,000 fan speed (rpm) 14

15 Cooling capability of CRAC CRAC : GV-15 (2003/08 products) Maker: SINKO INDUSTRIES LTD. Maximum volume: 14,000 CMH Rated power: 7.5kW Typical (default) setting Temperature of air : 15 Volume of airflow : half (roughly) Methods to change capability of CRAC Volume of airflow Temperature of air Number of CRACs 15

16 Estimation of heat removing The heat what the CRAC can remove is estimated by Quantity of heat (kw) = specific gravity airflow specific heat temperature = 1.3Kg/m CMH 1.0KJ/Kg 5 = 84000kJ/h = 23.3kW The maximum air flow produced by the CRAC we used is CMH (Cubic meter per hour). temperature means difference in temperature between entrance and exit of airflow. When we run LINPACK on all nodes, temperature of return air is about 20 degrees and temperature is 5 degrees. Thus one CRAC has a capability to remove heat of 23kW. Because the power consumption of IT equipment with LINPACK is roughly 27kW, one CRAC is not enough to remove all heat. 16

17 Power consumption (W) 12:32:30 12:51:38 13:10:46 13:29:54 13:49:02 14:08:10 14:27:18 14:46:26 15:05:34 15:24:42 15:43:50 16:02:58 16:22:06 16:41:14 17:00:22 17:19:30 17:38:38 17:57:46 18:16:54 18:36:02 18:55:10 Volume of airflow 19:14:18 19:33:26 19:52:34 Power Power consumption 20:11:42 (W) Modify openness of dumper to change the volume of airflow. Because openness of dumper is not accurate, we measured actual volume of airflow at the air duct using anemometer. Power consumption of CRAC increases linearly to volume of airflow Openness (Graduations) of dumper 50% 100% 25% 10% 50% Rated power: 7.5kW Time Graduations Speed of of dumper air (m/s) Power (W) 100% % % % % Speed of air (m/s) Speed of air (m/s)

18 Power consumption of CRAC (W) 11:30:00 11:45:32 12:01:04 12:16:36 12:32:08 12:47:40 13:03:12 13:18:44 13:34:16 13:49:48 14:05:20 14:20:52 14:36:24 14:51:56 15:07:28 15:23:00 15:38:32 15:54:04 16:09:36 16:25:08 16:40:40 16:56:12 17:11:44 17:27:16 17:42:48 17:58:20 18:13:52 18:29:24 18:44:56 Temperature of air Power Consumption (W) Modify mixture ratio of cold water and hot water to change the temperature of air. The volume of airflow is fixed to almost half of full speed. Power consumption of CRAC decreases 14W per 1 degree Temperature of air (degree Celsius)

19 Power consumption of CRACS in operation 13:00:01 13:16:55 13:33:49 13:50:43 14:07:37 14:24:31 14:41:25 14:58:19 15:15:13 15:32:07 15:49:01 16:05:55 16:22:49 16:39:43 16:56:37 17:13:31 17:30:25 17:47:19 18:04:13 18:21:07 18:38:01 18:54:55 19:11:49 19:28:43 19:45:37 20:02:31 20:19:25 20:36:19 20:53:13 21:10:07 21:27:01 21:43:55 The temperature of air is set to 15. The volume of airflow is fixed to almost half of full speed. Change #CRACs from 1 to 4 and 2. LINPACK run on all of equipment, but started gradually. Because parameters are set manually, power consumptions are not the same. They spread from 4500W to 5000W, deviation is about 10%. The same CRAC (for example #14) shows different values, when number of active CRACs is different. IT power (W) AC13(W) AC14(W) AC15(W) AC16(W) 27kW Cooling power (W)

20 Number of CRACs and room temperature Temperature( ) Change number of CRACs in operation ; 1 to 3 We measure room temperature (front, inside, rear of rack) Difficult to judge sufficiency of cooling capability LINPACK LINPACK Front Inside Rear LINPACK 15 Front Inside Rear top middle bottom Front Inside Rear

21 Temperature( ) Number of CRACs and CPU temperature Horizontal axis represents height of IBM s servers (in unit of U) Lower position is cooler in the case of 1 CRAC Lower, higher, when number of CRACs increases 3CRACs is necessary to keep them under 60, Tc critical temp Tc Tc 55 rack1 55 rack rack2 rack3 rack rack2 rack3 rack Tc rack1 rack2 rack3 rack Flow with 1 CRAC is too weak to be reached to top area and cold air is consumed in the low area of the rack. Flow with 3 CRACs is strong, cold air cannot be caught in the low area. We guess total capability of fans in IBM x3250 is not so large.

22 Summary of changing number of CRACs Temperature ( ) 22 Difficult to judge sufficiency by monitoring room temperature. Monitoring CPU temperature is a possible way to judge it. In order to keep the CPU temperature lower than 60, 3 CRACs are necessary. Power consumption by 3 CRACs (14kW), is necessary to remove heat of IT equipment (27kW). The maximum volume of airflow by 1 CRAC is CMH and can remove 23kW. 1 CRAC with half of airflow can remove roughly 12kW. The necessary of 3 CRACs seems to be reasonable. The efficiency of cooling is Ec = 27kW / 14kW = inside of rack rear of rack front of rack Number of CRACs CPU max Tc CPU average

23 Actual airflow We measured speed of air at air duct of CRAC, floor louver, and front/rear of rack by anemometer and estimated airflow. It was found that quantity of air absorbed to rack is less than 50% of total flow of CRAC in the case of 2 CRACs. The remaining air seems to reach directly to ceiling. It is clear that there is a lot of energy loss. Rack Ceiling Wall CRAC <Notice> The figure does not represent accurately actual environment. Rack is rotated 90 degree from the actual position. Floor Flow panel under the floor Louver (Grill)

24 Improvement of cooling efficiency We constructed front cover so that all of air was led into racks. CPU temperature in the case of 1 CRAC with front cover, temperature of air : 15, the volume of air : 50% CPU temperatures are at least 5 lower than those of 3 CRACs. Power consumption of 1 CRAC is about 4.5kW. The efficiency of cooling is Ec = 27kW / 4.5kW = Tc rack1 rack2 rack3 rack

25 More improvement of cooling efficiency CPU temperature (degree Celsius) CPU temperature (degree Celsius) Reduce power consumption by decreasing volume of airflow. Only 20% of full volume of air is enough for these IBM servers, if front cover is used. Then cooling power is 2.1kW The efficiency of cooling is Ec = 27kW / 2.1kW = % Tc Bottom Rack Position (U) Top 20% Tc rack1 rack2 rack3 rack4 rack1 rack2 rack3 rack Bottom Rack Position (U) Top 25

26 Summary Constructed testing laboratory : monitor temperature and power consumption of IT equipment and cooling facility Power consumption of fan behave as cube of fan speed. Cooling power depends on not only power consumption of IT equipment, but also configuration of facility, rack and environment. It is possible to reduce cooling power by leading cool air to rack directly. Monitoring temperature of CPU and controlling number of CRACs and volume of air are useful methods to reduce cooling power while maintaining CPU temperature lower than threshold. 26