How Intel Technologies & High Temperature Data Centers saves Energy and Money.

Transcription

1 Intel Intelligent Power Management Intel How Intel Technologies & High Temperature Data Centers saves Energy and Money. Power Thermal savings through the use of Intel s Intelligent Power Management in conjunction with Virtual Machine Device Queues (Intel VMDq) deployed in planned future High Temperature Cloud Data Center Environment. Intel Corporation October 2011

2 Contents Executive Summary 2 The POC proved the following: 2 Methodology 2 Software and Hardware Tools 3 Business Challenge 3 Technical Overview 3 Node Manager and Data Center Manger 10Gbps Networking with VMDq 3 3 High Temperature Ambient (HTA) 4 Test Environment for the POC Summary 4 Test Environment Physical Architecture 4 Test Environment Logical Architecture VMDq off 4 Test Environment Logical Architecture VMDq on 4 Test Cases for the POC Summary 4 High Temperature Ambient Data Center Operations 5 DaNang Climate Data Summary 5 Overall Results 6 Glossary 7 Executive Summary Intel s Power Management Technologies known as Node Manager (NM) and Data Center Manager (DCM) in combination with Intel s VMDq and an assessment of potential High Temperature Ambient (HTA) Data Center operations were jointly tested over a 4 month Proof of Concept (POC) with Department of Information and Communications of DaNang City at (DICD) the existing DaNang Building2 Software Park Data Center in DaNang, Vietnam. The objective was to provide a Technology Proof Point on the use of these Intel technologies. Intel Intelligent Power Node Manager Intel Intelligent Power Management is an Intel platform-software-based tool that provides policy-based power monitoring and management for individual servers, racks, and/or entire data center. Intel VMDq Virtual Machine Device Queues (VMDq) help offload network I/O data processing from the hypervisor software to the network silicon. High Temperature Ambient (HTA) Raising the operating temperature within the computer room in a data center decreases chiller energy costs and increases power utilization efficiency. Table 1 Intel Intelligent Power Node Manager, Intel VMDq and HTA explained. The POC proved the following: Using Intel 10Gbps Ethernet adapter with VMDq will increase the network throughput 25% while increasing per server power consumption by only 2 Watts. Applying a Node Manager/Date Center Manager Minimum Power Policy saves 30 Watts of power per server while still meeting maintaining the business Service Level Agreements. High Temperature Ambient Data Center Operations could cut Data Center energy costs by 50%. This paper describes the procedures and results from testing Intel Intelligent Power Management Technologies within a High Ambient Temperature operation at the DaNang Data Center. Methodology All three phases of the engagement utilized Intel s Technical Project Engagement Methodology (TPEM). This is not meant to be a detailed look at each step of the methodology but a guideline to the approach. # Descriptions HTA NM/DC M VMDq 1 Customer Goals and Requirements 2 Data Gathering 3 Design - DC room constructed - Server platform X X X chosen - Virtualization Software 4 Instrumentation - HTA- Covered under X data collection 5 Design Test Cases 6 Run Test Case 7 Data Collection 8 Analysis 9 Data Modeling 10 Reporting and Recommendations Table 2 - uses Intel s Technical Project Engagement Methodology (TPEM).

3 Software and Hardware Tools Three tools were used to measure and model the DaNang Government data center environment: 1. Intel Data Center Manager/Node Manager: (NM/DCM) was used to collect data on the workload, the inlet temperature and the actual power usage (both idle and under workload). 2. HTA Discovery and Analysis tool (HTA DA) was used to model the effect of high temperature operation using the data collected in step NTttcp 10Gbps (AKA NLoad) is a Microsoft tool that is a multi-threaded, asynchronous application that sends and receives data between two 10Gbps endpoints and reports the network performance for the duration of the transfer. NLoad allowed the team to focus on the correct installation and function of the installed 10Gbps Network Interface Card (NIC) subsystems. Business Challenge The DaNang egovernment is planned to interconnect employees, citizens and business communities by combining high speed network connectivity with seamless egovernment cloud services. To better align with these innovative egovernment cloud service offerings; there will be a need to increase the compute capabilities in the Data Centers. This has resulted in corresponding increases in rack and room power densities. The DaNang Government wants to utilize the latest technologies to gain the best performance and best energy efficiency across their planned digital city. To achieve this, DaNang Governement is working with the Intel CloudBuilders program, which is a crossindustry initiative aimed at making it easier to build, enhance, and operate cost efficient cloud infrastructure, to enhance the egovernment offering. Technical Overview The DaNang Government s Building2 Software Park DC has already been constructed so there is no possibility to alter any of the dimensions. Therefore the power & thermal optimization problem becomes a problem solving exercise in thermodynamics, i.e.: dissipating heat generated by the servers. This is done by transferring the heat to the air and then ducting hot exhaust air from the servers. The hot air is then passed through the CRAC units where it is cooled by the chilled water (CW) and blown back into the computer room. The CW is supplied in a separated water loop and cooled by one or more chillers located outside the computer room. The DaNang data center has a 120kW maximum facility power capacity. This power budget must cover both server power and Data Center cooling. This means that the more efficient the cooling infrastructure, the more power is available for the compute infrastructure. This ratio of total power divided by server power is called Power Utilization Efficiency (PUE). Older data centers often have PUEs of 3 or 4, while the latest designs are closer to 1.5. At the DaNang Government data center there was 100m 2 available space for IT equipment. Node Manager and Data Center Manger Intel Intelligent Power Node Manager provides users with a powerful tool for monitoring and optimizing DC energy usage, enhancing cooling efficiency, and identifying thermal hot spots in the DC. It can provide historical power consumption trend data at the server, rack and Data Center level. For the DaNang Government, inlet thermal monitoring of node temperatures was a key capability to identify potential hot spots in the DC in real time. 10Gbps Networking with VMDq Intel Virtualization Technology (Intel VT) refers to the hardware assists for virtualization that Intel offers across its server platforms (CPU, Chipset, I/O) to provide improved system performance, security, efficiency, and a more powerful virtualization solution. Intel VT for Connectivity is the portion of Intel VT designed to improve network I/O in virtualized servers and includes VMDq. VMDq is a network silicon-level technology that off loads the network I/O management burden from the hypervisor to the Ethernet Controller. Multiple queues and sorting intelligence in the silicon support enhanced network traffic flow in the virtual environment, freeing processor cycles for application work. This improves efficiency in data transactions toward the destined virtual machines (VM) and increases overall system performance. VMQ is Microsoft s Hyper-V queuing technology that makes use of the VMDq capabilities of the Intel Ethernet controller to enable data packets to be delivered to the VMs with minimal handling in software. The Shared Memory feature allows the data packets to DMA directly into the VM s memory, thereby avoiding a copy between the memory of the Management OS and the VM s memory i. 3

4 High Temperature Ambient (HTA) Intel defines HTA as an integrated solution that combines the server platform thermal design with data center infrastructure to raise the operating temperature in the computer room. The result is that this combined approach to integrated design will decrease cooling costs and increase power efficiency. Test Environment for the POC Summary Test Environment Physical Architecture The test environment was set up in a distinct area within the production data center, away from the core network or production servers. Devices Server Platform Server OS Tools Descriptions Physical Layout: 2 Canisters within 1 enclosure. Xeon: 2* Gb memory Intel 520-DA10Gb NIC: IXGBE Oplin PCIe Dual Oplin 82598; direct connect SPF+ Intel NM v GB2.5"SATA HDD * 2 Lab environment: Windows Server 2008R2 Power management tool: Intel Data Center Manager v1.5 Intel s Power Thermal Utility (PTU); Maximizes the CPU usage to 100% Temperature meter Microsoft Performance Monitoring (perfmon) NTttcp 10Gb Network Stress Tool (NLoad) Table 3 - NM/DCM and VMDq Test Environment Test Environment Logical Architecture VMDq off Figure 1 - VMDq "off" In the figure above, the hypervisor s virtual switch manages network I/O. All data passes through the virtual switch in the Management OS. All network traffic is directed to a single thread routing process on a single processor core. Test Environment Logical Architecture VMDq on Figure 2 - VMDq "on" In the figure above, the VMQ and VMDq technologies, data packets are delivered directly to the VM without data copy and bypassing the virtual switch. Network traffic is delivered directly to multiple threads running on multiple cores. Test Cases for the POC Summary There were 6 Test Cases used: 1. Baseline; NM/DCM Power Monitoring captured the power usage. 2. Baseline with HyperV ; NM/DCM Power Monitoring captured the power usage Gbps Networking with VMDq off and 4 VMs running NLoad; NM/DCM Power Monitoring captured the power usage Gbps Networking with VMDq off and 4 VMs running NLoad; NM/DCM Power Monitoring captured the power usage % 10Gbps Network with VMDq on with 100% CPU utilization; NM/DCM Power Monitoring captured the power usage. 4

5 6. 100% 10Gb Network with VMDq on with 100% CPU utilization; NM/DCM Minimal Power Usage Policy on and Power Monitoring captured the power usage. High Temperature Ambient Data Center Operations Before a cloud service provider considers the use of HTA in the data center, the external ambient temperature conditions play a large role in determining how the data center can utilize a variety of cooling strategies such as: Airside Economizer: Outside air is brought into building and distributed via a series of dampers and fans. Waterside Economizer: The chilled water from the tower is then used in the air conditioners inside the data center. DaNang Climate Data Summary ii The 2008 climate data provided by the US Dept of Commerce National Oceanic and Atmospheric Administration (NOAA) was analyzed and there is an opportunity to investigate running the current DaNang Data Center in HTA mode for increased power efficiency.. is set to 27 C, there will be an opportunity to run Free Cooling up to 6 months of the year based on the average maximum daily temperature. With the maximum daily temperature being an average of the whole day s temperature data there will be times within a 24 hours period where the temperature would be below the 27 C data center set point. Also, when the average low is included in the analysis it is clear that there is a possibility to run Free Cooling for longer than 5 months per year. Further detail data on DaNang climate would be needed, such as hourly temperature data samples to further clarify the total number of hours of Free Cooling that could be possible. Without the use of Free Cooling : 365days x 24hrs x PUE:3 x 120kW x $0.1/kWh = $315,360USD With the use of Free Cooling : (365days 180days) x 24hrs x PUE:3 x 120kW x $0.1/kWh = $159,840USD Possible saving with Free Cooling : $315,360 $159,840 = $155,520/annum Month Temperature (Celsius) Average Record Relative Humidity % Min Max Min Max Avg Jan Feb March April May June July Figure 5 DaNang Climate Data 2008 With the current relative humidity for data centers set at between 30-60% iii ; the DaNang Data Center will have a challenge to maintain the top end. Aug Sept Oct Nov Dec Table 4 DaNang Climate Data 2008 The chart below graphs the recorded minimum and maximum temperatures at DaNang. The graph shows that, if the current data center computer room 5

6 Table 6 DaNang Current Relative Humidity Data 2008 When designing the future DaNang Data Center; Node Manager power and thermal monitoring could be used as the actual power/thermal. Overall Results 1. Using Intel 10Gbps Ethernet adapter with VMDq will increase the network throughput 25% while increasing per server power consumption by only 2 Watts. 2. Applying a Node Manager/Date Center Manager Minimum Power Policy saves 30 Watts of power per server while still meeting maintaining the business Service Level Agreements. 3. High Temperature Ambient Data Center Operations could cut Data Center energy costs by 50%. 6

7 Glossary Power Usage Effectiveness (PUE) PUE is a measure of how efficient a computer data center uses its power; specifically, how much the power is actually used by the IT equipment (in contrast to cooling and other overhead lower PUE is better). The relationship between the cooling delivered by the chiller and the input work to the compressor is referred to as the Refrigeration, and is defined as the heat absorbed by the evaporator is the quantity of cooling that can be supplied. Supplying this cooling requires an input of mechanical work to the compressor, normally in the form of electrical energy to drive a motor. BMC CDC DC DCM NM POC VM AHU ASHRAE CAPEX CFD CRAC DC DCIE Delta T HTA LV MIPS MV ODM TX Free Cooling or Airside-Economizer Board Management Controller Cloud Data Center Data Center Intel Data Center Manager Intel Intelligent Power Node Manager Proof of Concept Virtual Machine Air Handling Unit, used inter-changeably with CRAC The American Society of Heating, Refrigerating and Air-Conditioning Engineers Capital Expenditure Computational Fluid Dynamics Computer Room Air Conditioner Unit, used inter-changeably with AHU Data Center Data Center Infrastructure Efficiency Delta Temperature, typically refers to supply and return temperature of cooling systems or servers. High Ambient Temperature Low Voltage Million Instructions Per Second Medium Voltage Original Design Manufacturer Transfer The ability to supply cooling when it is required, without having to pay for the actual generation of the low temperature iv.

8 INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked reserved or undefined. Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling , or by visiting Intel s Web site at Copyright 2011 Intel Corporation. All rights reserved. Intel, the Intel logo Intel s Power Management Technologies inside are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. i Advanced Virtualization I/O Queuing Technologies - An Intel-Microsoft Perspective; VMDq-VMQ white paper.pdf ii iii ASHREA Data Center Specifications 2011 iv komfortkyla.pdf 8