Intel Cloud Builders Guide to Cloud Design and Deployment on Intel Platforms

Transcription

1 Intel Cloud Builders Guide Intel Xeon Processor-based Servers Lenovo ThinkServer SmartGrid Technology Intel Cloud Builders Guide to Cloud Design and Deployment on Intel Platforms Lenovo ThinkServer SmartGrid Technology Intel Xeon Processor 5500 Series Intel Xeon Processor 5600 Series Intel Xeon Processor E Series Audience and Purpose This reference architecture outlines the usage of energy management technologies as part of monitoring, planning, provisioning, and optimizing strategies in cloud data centers to reduce energy costs and to address carbon emissions for green IT goals. It is intended for data center administrators and enterprise IT professionals who seek energy management solutions to achieve better energy efficiency and power capacity utilization within new or existing data centers. The techniques and results described can be used as a reference to understand energy management solutions implemented with the use of hardware and software components. The reader should be able to develop appropriate energy management solutions based on the design options presented using Lenovo ThinkServer Smart Grid Technology and ThinkServer Series Servers implementing Intel Node Manager. October 2012

2 Table of Contents Executive Summary...3 Introduction...3 Smart Grid Overview...3 Intel Power Management Technologies Overview...4 Intel Node Manager...4 Intel Data Center Manager...5 Test Bed Blueprint...5 Smart Grid Solution Architecture...5 Hardware and Software Architecture...6 Physical Architecture...6 Installation and Configuration...7 Usage Models...8 Power Monitoring, Energy Consumption Report and Ranking, Power Trend Analysis...8 Power Control and Optimizing Rack Density...11 Group Policy Prioritized Power Control...13 Business Continuity Policy...18 Things to Consider...22 Power Management...22 Security...22 Hardware

3 Executive Summary A green data center is an inevitable requirement that data centers must meet at a certain stage. It encompasses overall system problems of data centers such as the whole building and room infrastructure, air conditioning, UPS, servers, and other IT equipment. Therefore green data centers need to meet the demands below: Power monitoring and capping capabilities at all levels of the data center (system, rack identification, and data center). What can be done at an individual server level becomes much more compelling once physical or virtual servers are scaled up significantly. Aggregation of the power consumed at the rack level and management of power within a rack group to ensure that the total power does not exceed the power allocated to a rack. Higher level aggregation and control at the row or data center level to manage power budget within the average power and cooling resources available. Optimization of productivity per watt through management of power at the server, rack, row, and data center levels to optimize total cost of ownership (TCO). Application of standards-based power instrumentation solutions available in all servers to allow management for optimal data center efficiency. Extension of instrumentation to enable load balancing or load migration based on power consumption, and close coupled cooling for the management of pooled power and cooling resources. Intel works with end users to create an open data center roadmap of usage models that address key IT pain points for more secure, efficient, and simple cloud architectures built on a foundation of transparency. This paper describes an energy management reference architecture based on Lenovo, Intel, and ThinkServer Smart Grid Technology solutions with usage models aimed at data center power efficiency and optimal utilization of provisioned power and cooling capacity. The goal of power management is to optimize productivity per watt in order to reduce TCO. Requirements include the capability to monitor and cap power in real-time at server, rack, zone, and data center levels. This means the ability to monitor and manage aggregated power consumption within a rack, zone, or data center based on available power and cooling resources. In this reference architecture we used ThinkServer Series Servers with Intel Node Manager and Smart Grid Technology Software based on Intel Data Center Manager (Intel DCM) to provide data center energy efficiency through real time power monitoring of the servers, power capping and policy based on power management. Introduction Data center designers have often had to rely on inadequate power usage data in deploying servers and racks. Lacking actual power usage information for each server based on its intended load, they stay on the conservative side and populate server racks based on avail able server power information using the Figure 1: Lenovo ThinkServer Smart Grid Technology declared power rating from the server nameplate. This nearly always leads to an overstatement of a server s actual power need, resulting in underutilization of the budgeted rack power. Likewise, relying on overstated power demands often leads to providing more cooling than is necessary for the actual loads. In fact, data center designers often use overstated power demand and heat generation figures instead of actual temperatures. The price they pay for overestimation is additional costs for unnecessary cooling. To enable data center owners to bet-ter utilize rack power and better control server performance and power consump tion to optimize server density, Intel has developed an innovative solution known as Intel Node Manager. Smart Grid Overview Lenovo ThinkServer Smart Grid Technology software is a Lenovo data center power management solution related to the ThinkServer server series. Smart Grid can provide the intensive and remote energy consumption management ability for the servers supporting Intel Node Manager. Smart Grid achieves power and performance management with no agent, using browser/server architecture and an IT administrator to centrally manage power of servers/groups in the local area 3

4 network. It also supports centralized management of servers/groups spanning the Internet. Smart Grid consists of two components: the Smart Grid server and the client browser. The energy monitoring and control of the Smart Grid is based on the Intel DCM platform through which Smart Grid can grasp the power trending with analysis and step-by-step statistics. According to the power consumption change, Smart Grid can make the reasonable and effective energy saving policy, and then the policy will be issued to Intel Node Manager, which is embedded to the board through the power control interface. Intel Node Manager regulates the CPU basic frequency according to the received policy and controls the power consumption. Figure 2: Energy monitoring and control using Smart Grid and Intel Node Manager The major benefits of Smart Grid include: Smart Grid helps IT administrators know their business and power consumption trends and do better rack planning, therefore increasing rack density. Smart Grid supplies a soft power meter to each server/group, which is convenient for grasping the real-time power consumption. Smart Grid offer a friendly interface to set power policy for servers/groups to achieve power saving. The interface also remembers historical power trending and provides recommended default policies. Smart Grid monitors performance (CPU and memory utilization) and provides dynamic power policy for Web servers or database application s which combines performance (in-band) with power saving (out-of -band). Smart Grid provides multi-dimensional reports, letting customers know the history power trend in detail. Smart Grid uses emergency solutions to ensure business continuity during a data center outage. The process by which Smart Grid manages the power consumption includes: discovering and importing the server, monitoring the power consumption, learning the power trending, carrying out the policy, analyzing the power savings, and suggesting how to regulate the power policy. The whole process achieves a closed-loop management, which helps to find the best power saving solution for each server and reduces the power consumption. Intel Power Management Technologies Overview Intel Node Manager Intel Node Manager was implemented on Intel server chipsets starting with the Intel Xeon processor 5500 series platforms. Intel Node Manager 2.0 was introduced in 2012 on server platforms supporting the Intel Xeon processor E series. Intel Node Manager provides power and thermal monitoring and policy based on power management for an individual server and is exposed through a standards-based IPMI interface on supported Baseboard Management Controllers (BMCs). Intel Node Manager requires an instrumented power supply that conforms to the PMBus standards. Intel Node Manager allows data center managers to set a power bud get for a server, enabling up to 20 percent denser deployments. It s a key ingredient to dynamic power management. In fact, when paired with Intel Data Center Manager (Intel DCM); a software technology that provides power and thermal monitoring and management for servers, racks, aisles, and other groups of servers), the combined solution can help optimize entire data centers. Figure 3: Intel Node Manager 2.0 can monitor and apply power limiting on the whole server, including CPU and memory subsystems Some of the key features of Intel Node Manager include: Real-time power monitoring Platform (server) power limiting Power threshold alerts Smart Ride-Through or SmaRT that increases the server s resiliency to transient faults where the voltage dips for a few cycles 4

5 Closed Loop System Throttling or CLST that enables optimized platform designs that can continue operating under impaired power supply situations due to power supply unit failures Intel Node Manager 2.0 enabled platforms introduced in 2012 bring improved power limiting range and faster response to reach the target power limit through the use of the Running Average Power Limiting (RAPL) interfaces supported in the Intel Xeon processor E series. It also allows a finer degree of power control useful for certain applications, making it possible to monitor and apply power limiting not only on the whole server, but also on the CPU and memory subsystems or domains individually as shown in Figure 3. Intel Data Center Manager Intel DCM provides power and thermal monitoring and management for servers, racks, and groups of servers in data centers. Management Console Vendors (ISVs) and System Integrators (SIs) can integrate Intel DCM into their console or command-line applications to provide high value power management features. These technologies enable new power management paradigms and minimize workload performance impact. Test Bed Blueprint Smart Grid Solution Architecture Smart Grid is compatible with two ways of collecting data - in-band and out of band. The collector gathers the performance, configuration, and event data of the equipment through in-band and Intel DCM collects the power consumption of equipment through out-of-band. After data acquisition, the data process layer deals with the original data through analysis and statistics in order to provide the related services for the upper applications such as performance threshold, event forward, strategy management, and resource discovery. The Smart Grid divides the system into several modules including power and performance monitoring, event management, statistics, equipment management, and so on which can be used by end users for monitoring and management. Figure 4: Smart Grid Architecture 5

6 Hardware and Software Architecture According to the different number of servers, Smart Grid needs different configurations of hardware. The required hardware and software resources are shown in Table 1. Hardware Requirements Operating Systems Client To manage 500 servers: Intel Xeon 1CPU, 8GB RAM, at least 300GB disk space, 1Gb NIC To manage 1000 servers: Intel Xeon 2CPU, 16GB RAM, at least 600GB disk space, 1Gb NIC Windows Server* 2008 R2 Standard x64 Windows Server* 2008 R2 Enterprise x64 Windows Server* 2008 R2 Datacenter x64 Windows* Small Business Server 2011 Standard Windows* Small Business Server 2011 Essentials SUSE LINUX Enterprise Server 11 (x86 & x86_64) Red Hat Enterprise Linux 5 (x86 & x86_64) Red Hat Enterprise Linux 6 (x86 & x86_64) Note: It is strongly recommend to run Smart Grid on Windows Server 2008 R2 Web browser IE7 or greater, Firefox4.0 or greater Table 1: Smart Grid hardware and software requirements Physical Architecture Smart Grid used B/S architecture to manage the power consumption of servers in a local area network instead of installing proxy software on managed servers. It just needs to install the program on the server-side and then the end users use it through the Web browser. Figure 5: Smart Grid Management Node 6

7 Installation and Configuration Installation Steps Smart Grid uses the B/S architecture, so the program needs to be installed on the server-side. In the Smart Grid installation CD, there are three versions of Smart Grid packages: 1. smartgrid_32.exe is the version for Windows 32bit 2. smartgrid_64.exe is the version for Windows 64bit 3. smartgrid.zip is compatible Linux 32bit and 64bit Installing WINDOWS Version Put the Smart Grid Installation CD into the drive; the program will run automatically. Complete the installation on the basis of setup wizard step by step. Note: If the program doesn t run automatically in the drive, please enter the catalogue of the CD and click the setup.exe in order to start the wizard. Installing LINUX Version Put the Smart Grid Installation CD into the drive and enter the catalogue of the CD (such as cd/mnt/cdrom). Perform the below command: chmod +x install.sh./install.sh The default catalogue is /opt/smartgrid. If you want to specify the installation directory (such as /opt/sgt), perform the command below:./install.sh /opt/sgt According to the version of Linux, choose the appropriate RPM package to install ipmiutil. For example, perform the command below: rpm -i ipmiutil _rhel6. x86_64.rpm If the system version is RedHat Linux 6.1 or greater, the required package wasn t included in the default installation. The below packages need to be installed manually: ld-linux.so.2 libxml2.so.2 libpq.so.5 libtermcap.so.2 libreadline.so.5 Configuration When operating the Smart Grid for the first time, use the automatic discovering equipment in the wizard. If the network has changed later, you can trigger the manual discovering to find the new equipment with the Smart Grid. There are two ways to discover devices in Smart Grid. You can click the Discover Servers button in the home page or click the Discover or Add buttons on the List View page. 7

8 To collect the devices by IPMI, there are several necessary parameters including BMC address, user name, password, and subnet mask. Usage Models Power Monitoring, Energy Consumption Report and Ranking, Power Trend Analysis Power Monitoring Power monitoring is a critical capability that enables us to characterize workloads and identify opportunities to increase data center energy efficiency. Smart Grid can monitor the real-time power trending and provide the soft instrument, which is the equivalent of installing a meter for free. It is convenient for grasping the real-time power consumption. The real time power utilization of server is 125W in the screen shot below. 8

9 Power Consumption Ranking Smart Grid provides multi-dimensional power consumption ranking and reporting. It can help the users to analyze the power consumption for all servers in order to find the server which generates the highest power. In this way, the user can screen out the server which needs to be adjusted in order to achieve the highest cost savings. All rankings support TOP10, TOP20, TOP50 switching and the user can customize the ranking search by clicking the More button. Power Consumption Report According to the different sizes of reports, all reports show support for today, the last seven days, and the last 30 days switching, and so on. Every report can be exported, including the trending diagram and detailed data. The real-time power trending of server group: 9

10 The view of each server s power consumption in the group: The power consumption of the server group: Power Trend Analysis Smart Grid can grasp the variation of the power consumption through the data collection of the servers and then deal with the data collected through analysis and statistics in order to make a reasonable policy for the server. The use case about power control will illustrate the power analysis in detail. Conclusion Server power monitoring, racking, report, and analysis all help us find and manage data center hotspots and thereby reduce the chances of hardware failure. The analysis of the historical usage of power consumption data enables us to develop forecast models for data center planning. These forecast models are very accurate as they are based on the real-time data. 10

11 Power Control and Optimizing Rack Density In the past, there has been no dynamic power management technology which allows the data center to optimize power utilization. Data center managers had to face many power management challenges with the top issue being over-allocation of power. Power allocation to servers does not match actual server power consumption. Power is typically allocated for the worst case scenario based on server nameplate. Static allocation of power budget based on the worst case leads to inefficiencies and does not maximize use of available power capacity and rack space. The over-allocation resulted in another problem, in that there is a lot of space on racks. When users need more compute capacity, they have to pay more for additional racks. The traditional data center lacks effective ways to optimize the power and performance at rack level dynamically. To improve power utilization, data centers need to track actual power and cooling consumption, and dynamically adjust workload and power distribution for optimal performance at rack and data center levels. The use case below describes how to control the power consumption with Smart Grid. Power Monitoring for Server Node Smart Grid can monitor the real-time power trending and provide the soft instrument, which is the equivalent of installing a meter for free. It is convenient for grasping the real-time power consumption. Smart Grid can grasp the effective power of the server in the learning phase, which helps set the power capping value for user. The real time power utilization of server is 125W in the screen shot below. 11

12 To simulate a workload, ensure that on the node and monitor the power consumption, the power of the server is 250W. Set the policy and make the power budget 200W. 12

13 Enable the policy and check the power trending. The power remains at 200W which indicates the power capping is working. The use case is helpful in older data centers that over-provision for power and in host settings with power quotas in effect. In old data centers in which the servers are allocated power quotas, the main goal is to optimize the rack utilization so as to place as many servers in a rack as the power limit allows in order to maximize the rack space utilization. The pre-requisite of the use case is to measure the power utilization of the servers in the rack over a period of time and capture the maximum power utilized at the peak workload. To create a new policy for demonstrating power capping, ensure that the power usage of the server will not go beyond the threshold value specified. Make sure that the threshold value is little lower than the maximum power utilization of the server captured earlier. Setting the threshold value lower than the maximum will allow us to demonstrate the power capping policy. Note: In reality, the threshold value would be higher than the maximum power utilization to ensure that the workload performance is not impacted. Enable the policy and initiate the workload on the server for which the policy is enabled so as to increase the power utilization. The power utilization of the server stays within the threshold limit set from the server. Note: Intel Node Manager always tries to honor the power capping value. But in certain cases, where the workload demand is very high, it will try to come down to the lowest p-state possible, if not to the level requested. Conclusion The result of the use case shows that Intel Node Manager can reduce power consumption at node level without much impact to performance. When the same concept applied at rack level with a combination of Intel Node Manager and Smart Grid working together, the rack-level capacity could also increase within the power capping value without performance impacts. The difference between the total power cap assigned to the rack and the power quota allocated for the rack provides guidance as to how many additional servers with similar power cap settings can be added to the rack without overshooting the power quota allocated. Since we will continue to add additional servers into the rack, the overall performance of the system increases further but stays within the power envelope allocated by the host provider. Group Policy Prioritized Power Control Smart Grid supports the server group managed by policy. Users can make different servers in the same group and specify the priority for each server. When we set power capping value for the group, the Smart Grid will allocate the power budget according to the priority of servers and make sure the server with higher priority obtain the power supply preferentially. 13

14 Two Servers in a Group Put two servers in a group and add the same workload for each server. Let the servers run stably for a period. The screen shot shows that the power of the two servers is 400W. The power of after increasing the workload is 400W. The power of after increasing the workload is 400W. 14

15 The power consumption of the server group is 800W. Set the Priority for Each Server Make sure one of the servers is critical, the other is low. 15

16 Set the Policy to the Server Group The power capping value is set to 650W. Check the power trending respectively. Power Consumption Trending about Each Server and Server Group The power of is reduced to 380W or so. 16

17 The power of is reduced to 270W or so. The power of the server group is reduced to 650W or so. Conclusion We could use capping to distribute the power that remains on group; the Smart Grid will allocate the power budget according to the priority of servers and make sure the server with higher priority obtain the power supply preferentially. 17

18 Business Continuity Policy Once we have employed server power monitoring to characterize the application workloads, we can extend the power capping to help maintain business continuity in the event of a power outage. Smart Grid provides a solution to ensure the equipment or group run and consume at a low level in the case of emergency. In the case of UPS limited, protect the key application and maintain the business by extending power supply time. Under Normal Circumstances The power consumption of server group is 650W. The power of is 380W. 18

19 The power of is 270W. System Enters the Emergency Solution Click the Start Emergency Solution button, which requests that the system maintain the lowest power consumption. 19

20 Choose the server group which will start emergency solution. In this test case, we choose the Lenovo server group. The power consumption of server group is 500W or so. 20

21 The power of is 247W or so. The power of is 248W or so. Conclusion In emergency circumstances, we can use the emergency policy to make the server maintain the lower power consumption in order to extend the power supply time until the power supply recovers. 21

22 Things to Consider Power Management Power management should be considered after careful analysis of the workload performance under various power capping. As mentioned earlier, there are many usage models for which a power management solution would be very beneficial. At the same time, there are scenarios wherein power management may not be the right option. For example, if a highly-sensitive production workload is very processor intensive and the host is already highly utilized, the addition of a power cap would inadvertently affect the performance of the system. Security Security is one of the key considerations in server deployments, either virtualized or bare-metal. In a cloud deployment scenario, from both the perspective of the service provider and consumer, it is highly recommended to use platforms that support Intel Trusted Execution Technology (Intel TXT), such as Intel Xeon processor 5600 series, along with supporting software platforms to create a trusted cloud environment that enjoys strong protection against compromise Hardware If Smart Grid is used to manage other servers that are not ThinkServer, it has to support Intel Node Manager which is embedded in other servers such as the Dell idrac7 server. In addition, Smart Grid needs to provide API if it will have been integrated with the management of the room infrastructure, application facilities, and data center. 22

23 Disclaimers Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. See products/processor_number for details. INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked reserved or undefined. Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling , or by visiting Intel s Web site at Copyright 2010 Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Xeon inside, and Intel Intelligent Power Node Manager are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others.