High-Performance Computing Clusters with IPMI

Managing and Monitoring High-Performance Computing Clusters with IPMI This article introduces the Intelligent Platform Interface (IPMI) specification in the context of high-performance computing clusters discussing example implementations and important IPMI features from a cluster-level management and monitoring perspective. BY YUNG-CHIN FANG, GARIMA KOCHHAR, AND RANDY DEROECK High-performance computing (HPC) clusters are widely used for compute-intensive, transaction-intensive, and I/O-intensive applications. The benefits that enterprises can derive from standards-based HPC clusters compared to large symmetric multiprocessing (SMP) based supercomputers are well known, including scalability, ease of technology refresh, reusability of components, and disaster recovery capabilities. 1 However, cluster mean time between failures (MTBF) is inversely proportional to the scale of the cluster, given the low MTBF of aggregated standards-based components. Thus, cluster management can become critical for maintaining the cost-effectiveness of large-scale clusters as compared to small clusters. Timecritical cluster applications can suffer greatly from unexpected downtime resulting from component failure or working-environment failure. In an ideal situation, component failure would be prevented without using host resources by remotely monitoring the hardware health of all cluster components, including CPU, bus cycle, and memory. Under such circumstances, failing components could be detected and replaced before the performance of the cluster was affected. The foundation for strong overall systems management lies in the hardware-level instrumentation of data center platforms that is, monitoring physical server characteristics such as motherboard and chassis device temperature, CPU and power supply module voltage, fan rpm, power supply status, and access to important information about hardware inventory. Although most original equipment manufacturers (OEMs) incorporate instrumentation for their platforms into the circuit level, many hardware management solutions are either proprietary or nonstandardized. Consequently, administrators are often required to implement management solutions based on multiple standards that are not interoperable in a heterogeneous environment. Also, in the past, administration of largescale HPC clusters and Web or server farms suffered from a lack of standardized hardware-based, out-of-band remote 1 For more information about HPC clusters compared to large SMP-based supercomputers, visit http://www1.us.dell.com/content/topics/global.aspx/solutions/en/clustering_hpcc?c= us&cs=555&l=en&s=biz&~tab=2. 58 POWER SOLUTIONS Reprinted from Dell Power Solutions, October 2004. Copyright 2004 Dell Inc. All rights reserved. October 2004

Server software interface Agent instrument bridge management capability. Advanced Power (APM) is an example of an existing standard that is defined as a subset of server management specifications. However, APM does not have well-defined interfaces that allow it to provide higher-level management capability. The Intelligent Platform Interface (IPMI) specification can help address the need for a consolidated remote hardware management standard. IPMI defines common platform instrumentation interfaces that can help enable interoperability between motherboard/baseboard and chassis, between baseboard and server management software, and even between servers. It utilizes intelligent hardware that enables administrators to monitor and access platform instrumentation even when a server is powered down or locked up. This monitoring capability provides information that enables systems management, recovery, and asset tracking, thereby helping organizations to drive down total cost of ownership (TCO). The IPMI initiative The IPMI initiative comprises three separate specifications: IPMI, 2 the Intelligent Platform Bus (), 3 and the Intelligent Bus (). 4 The IPMI specification, which is the main specification, defines the messages and system interface to platform management hardware. The specification defines an internal management bus for extending platform management within a chassis (intraplatform management). Finally, the specification defines the external management bus between IPMI-enabled systems (interplatform management). Figure 1 illustrates the role of and connections. Peripheral chassis Agent instrument bridge Figure 1. IPMI intraplatform () and interplatform () connections Peripheral chassis Agent instrument bridge Layered management One of the key principles of IPMI is to provide a layered management framework based on a modular design that provides management value at each level of integration. The net value is designed to increase as more software, hardware, and firmware levels from processor, chip set, and BIOS down to baseboard, chassis, and so forth are included. The intelligence for management should reside at the appropriate level such that each level can retain its integrity even if the levels are separated by design. This implies extensibility, reuse, and scalability across server product lines. Figure 2 illustrates how IPMI fits into the management stack. IPMI management components Intelligent platform management refers to the autonomous monitoring and recovery features implemented directly in platform management hardware and firmware so that monitoring, logging, recovery, and inventory information is available independent of the host s main processors, BIOS, and operating system (OS). The following sections describe representative components of the IPMI specification that are most relevant to HPC clusters. Figure 3 illustrates the overall IPMI architecture. Baseboard management Many vendors design and manufacture baseboard management s (BMCs), which define a chip on the baseboard that serves as the centralized processor for hardware-level platform management. Dell has included an on-board BMC in its eighth-generation servers, including the Dell PowerEdge 1850 and PowerEdge 2850 servers. Motherboard Hardware Software software standards IPMI applications Service provider Figure 2. IPMI and layered management stack In-band remote access Service provider interface Instrumentation code IPMI interface code IPMI hardware interface BMC Standard remote interface (such as Remote Procedure Call [RPC] and SNMP) Standard software interface (such as Distributed Interface- Interface [DMI-MI] and Common Information Model [CIM]) Standard software interface (such as DMI-Component Interface [DMI-CI] and Windows Instrumentation [WMI]) IPMI interface 2For information about the IPMI 2.0 specification, visit ftp://download.intel.com/design/servers/ipmi/ipmiv2_0rev1_0.pdf. 3For information about the 1.0 specification, visit ftp://download.intel.com/design/servers/ipmi/ipmb1010ltd.pdf. 4For information about the 1.0 specification, visit ftp://download.intel.com/design/servers/ipmi/_1013.pdf. www.dell.com/powersolutions POWER SOLUTIONS 59

Remote management card RS-485 transceivers LAN LAN Modem Serial connector Auxiliary connector Auxiliary connector Serial bridge (optional) PCI management bus Network (LAN) Sideband interface to NIC, such as SMBus Serial port sharing Serial BMC SEL SDR Nonvolatile storage Sensors and control circuitry Voltages, temperatures, fans, power, reset control, and so on Private management buses management (satellite ) SEEPROM sensors Fans, temperatures, power supplies, and so on board Motherboard Motherboard serial System bus System interface IPMI messages SEEPROM Temperature T sensor Memory board SEEPROM Processor board serial EEPROM (SEEPROM) Redundant power board Figure 3. IPMI architecture designers can choose the proper BMC for the platform under development. Additional management s including s for chassis management, redundant power supply management, and local area network (LAN) management can be distributed on other boards within the system and communicate with the BMC via a standard interconnect: the bus. Such s are also referred to as satellite s. Many servers are equipped with more than 100 on-board sensor devices and chips. These sensors are connected to the BMC via the bus. The bus facilitates the inter-ic (I 2 C) bus and systems management bus (SMBus). 5 These buses are dedicated low-speed, low-cost management buses. The IPMI management framework has multiple management buses that run to multiple devices. The satellite s communicate with the BMC through IPMI messages on the bus. The BMC controls the system event log (SEL) sensor, sensor data record (SDR), and field replaceable unit () information as described in the System event logs and Sensor data records sections later in this article. The BMC provides a connection to the LAN and can send messages over the LAN as described in the LAN interface section. The bridge provides a mechanism for transferring internal messages on the bus to devices on the external bus. Figure 4 shows the various devices that connect to the BMC via the bus and the bus. The BMC,, and implementation is designed to make the hardware management and monitoring architecture a stand-alone computer subsystem. The hardware management and monitoring architecture has its own processor, bus, memory, and ROM. This approach enables administrators to remotely manage and monitor hardware health conditions without consuming host CPU cycles or host bus bandwidth. Thus, HPC cluster applications can run without interruption by the out-of-band management traffic or activities. device Status and control bridge device SDR repository device SDR repository SEL device SEL Figure 4. BMC sensor information and connection to the external network device Sensors 5 For information about I 2 C and SMBus, visit http://www.semiconductors.philips.com/acrobat/literature/9398/39340011.pdf and http://www.smbus.org/specs, respectively. 60 POWER SOLUTIONS October 2004

System event logs The BMC provides a central, nonvolatile SEL. Satellite s detect system events and log them to the SEL. These events are added to the SEL via commands sent out by the satellite s to the BMC on the bus. The SEL entry includes the sensor name and type, helping to ensure that the data can be interpreted without additional knowledge of the sensor or access to the SDRs. Figure 5 displays out-of-band access to SEL records via the Dell Remote Access Controller 4/I (DRAC 4/I). Example SEL entries include chassis intrusion messages, CPU thermal trip events, CPU configuration errors, fan speed messages, chassis and baseboard temperature logs, and so on. SELs are reported by Platform Event Traps (PETs), described in the Watchdog timers section. The SEL can also be queried from the management station by using IPMI shell commands. Dell servers usually include centralized management console software such as Dell OpenManage IT Assistant. When a system event occurs, the BMC can automatically send the event trap using Alert Standard Forum (ASF) format. A system administrator can install and configure Dell OpenManage IT Assistant software to receive and monitor the SEL across an HPC cluster and computing facilities. IT Assistant can receive notification of events from compute nodes and either notify the system administrator by e-mail or page, or launch an application to help remedy the situation. Based on this input, the system administrator can take proactive action to help prevent failures. Sensor data records Sensor data records contain information about the type and number of sensors on the platform, sensor threshold support, event-generation capabilities, and type of reading each sensor provides. For example, a vendor could describe a device s threshold-based sensor in the SDR such that an event is assigned by default different voltage values on different systems (such as +5 volts on one, and 12 volts on another) just by changing the SDRs. Sample SDR entries include CPU voltage and temperature, chassis ambient temperature, fan rpm, and so on. The primary purpose of the SDR is to describe the platform s sensor configuration to the systems management software. SDRs also include information about the number and type of devices connected to the system s bus. The SDR repository is a single, nonvolatile storage area that contains all the SDRs and is managed by the BMC. The repository provides a mechanism to retrieve SDRs via out-of-band interfaces such as remote management cards and other devices connected to the bus. The SDR repository is independent of the host processor, BIOS, OS, and systems management software. Centralized management console software can be used to manage and monitor SDRs. System administrators can set the Figure 5. DRAC 4/I system event log threshold conditions to trigger an event before the event actually occurs. For example, a system administrator can lower the CPU temperature overheat threshold, causing the BMC to trigger the SEL to send a notification when the lowered CPU temperature threshold approaches giving the administrator a chance to resolve the event proactively. Web, command-line, and Simple Network Protocol (SNMP) interfaces can be used to access the SDR. The system administrator can decide which interface to use based on the requirements of the situation. For example, the system administrator of a large-scale computing facility will usually set the sensor thresholds to a narrower range, thereby allowing more time to respond to unexpected situations. LAN interface IPMI messages can be transferred between the BMC and a remote management console over an Ethernet LAN using User Datagram Protocol (UDP). The IPMI messages are encapsulated in Remote and Control Protocol (RMCP) packets at the LAN and sent out on the network. RMCP is a simple request response protocol defined by the Distributed Task Force (DMTF) 6 that can be delivered using UDP datagrams. The RMCP packet format is also used for the DMTF ASF specification. Using the RMCP format allows management applications to operate with both IPMI-based and ASF-based systems. The IPMI LAN interface can be implemented using a LAN that is dedicated to the BMC, but typically the network interface card (NIC) is shared by the system and the BMC. The BMC talks to the LAN over an bus (either I 2 C or SMBus). The LAN has the capability to detect packets sent to the management port (port 623), and these packets are delivered to 6 For more information about DMTF, visit http://www.dmtf.org. www.dell.com/powersolutions POWER SOLUTIONS 61

the BMC. The packets can also be sent to the system at the same time they are delivered to the BMC. If the incoming packets are not encapsulated in RMCP packets, they are sent only to the host CPU and not to the BMC. The BMC uses the shared LAN interface to inject packets into the network, and these packets are interleaved with network packets generated by the system. If a shared LAN is used, it can be designed such that the interface for the management port is powered by standby power. This approach enables the BMC to be active and available even when the system is locked up or powered down. LAN alerts received by the management application running on the remote management system enable cluster administrators to remotely respond to an alert several ways: by powering up the system, powering down the system, or forcing a reboot either to the default boot device or to an alternate boot device such as a Preboot Execution Environment (PXE) server (see Figure 6). This functionality can save valuable time by providing the capability to fix problems remotely from a management console. If thresholds are set low enough, failing equipment can be detected and replaced before cluster performance degrades. Remote console via a LAN IPMI 1.5 defines the serial over LAN (SOL) feature, which allows serial console traffic to be redirected over an IPMI session via the BMC. SOL is designed to provide an administrator at a remote console with text-based access to the redirected BIOS and OS console of the managed server. Once console redirection is set up in the managed server s OS, a remote administrator can invoke the SOL capability to provide remote text-based access, which includes the managed server s BMC, BIOS, and OS. From the remote console, an administrator needs a local proxy to establish a Telnet connection to the managed server s BMC via a specified logical network port. The Telnet proxy wraps the commands into the RMCP packets. These packets Remote management console UDP datagrams to management port LAN LAN BMC Sideband connection Outgoing packets (such as SMBus or I 2 C) from system software PCI System bus Figure 6. LAN in an IPMI implementation Managed system All incoming packets Datagrams generated by BMC SEL, SDR, Satellite are received by the managed server s BMC (via the LAN ) and contain information indicating that the remote administrator requires an SOL connection. The BMC bridges communication over the managed server s serial port circuit and sends it to the OS. The OS response is redirected to the COM port. The BMC wraps this response and sends the output to the remote management station s Telnet proxy. In this way, a remote administrator can access the managed server provided that the managed server s OS is functioning. If the OS is not functional, the remote administrator can access the host BMC to power manage the server, set up the BIOS, or both. A remote management console is a commonly required management feature for HPC clusters and data centers. The SOL implementation described in this article can help eliminate one management fabric from the cluster, thus helping to reduce rack cable density and cost because there is one less cable to manage. In addition, because SOL can be implemented via either a dedicated NIC or a shared NIC, administrators can use a shared NIC to help reduce the cost for other types of out-of-band remote management hardware such as the serial port concentrator; keyboard, video, mouse (KVM) switch and cable products; proprietary out-of-band card; and fabric. Watchdog timers IPMI provides a standardized interface for a system watchdog timer that can be used by the managed server s BIOS, OS, and OEM applications. For example, if so configured, a watchdog timer can periodically read the BMC SEL (or receive an interrupt for a new SEL entry) and send out PETs on the defined LAN interface. These PETs can be picked up by the Platform Event Filters (PEFs) of a properly configured remote management station and used to trigger pager or e-mail alerts. Advanced Configuration and Power Interface Advanced Configuration and Power Interface (ACPI) 7 is an open standard for remote power management to configure and manage power devices on the motherboard. It allows OS-directed power management in addition to BIOS-directed power management unlike APM, which puts all power management control in the BIOS. The idea behind ACPI is that the OS has maximum knowledge of the state of the system and does not have the size limitations of the BIOS. At the hardware level, ACPI allows power-up, power-down, and power cycling. At the OS level, extended kernel modules are needed for Linux platforms to allow operations such as hibernate and graceful shutdown. ACPI is widely used in the cluster deployment phase for remote power-up of a new node with no OS, and in the cluster operational phase to remote power cycle a locked-up node. 7 For more information about the ACPI standard, visit http://www.acpi.info and http://www.intel.com/technology/iapc/acpi. 62 POWER SOLUTIONS October 2004

Terminal mode Terminal mode defines how IPMI messages can be transferred using printable characters. It also includes a limited number of English ASCII text commands including the SYS command set for such actions as getting a high-level system status and causing a system reset or power state change. Examples of SYS commands include the following: HPC cluster operation Monitor sensor readings remotely Monitor SEL readings remotely Provide a remote text console (eliminates one out-of-band management fabric) Provide remote power management (power up, power down, power cycle) SYS PWD U USERNAME password Authenticates and activates a terminal mode session SYS POWER OFF Directs the BMC to perform an immediate system power-down SYS HEALTH QUERY Causes the BMC to return a high-level system health report Many of the commands in the SYS command set are in hexadecimal form, and the SYS command set is usually implemented via a serial port. Most vendors wrap this IPMI ASCII command set to make it human-readable by introducing a command-line interface (CLI). Hence there are different CLI encapsulation implementations of the SYS command set. A set of commands that is provided by Dell on the Dell OpenManage CDs, ipmish, is one example of this. The example used in the SEL section of the CDs contains ipmish commands issued to read the SEL of the node. These commands can also be used to query the power state of the machine and to power manage it (power up/power down/power cycle). The ipmish commands equivalent to the previous SYS commands include the following: The IPMI specification helps enable an interoperable, extensible, scalable, and highly available architecture, which many OEMs have adopted for developing server management architecture. As discussed in this article, IPMI includes many existing systems management specifications and protocols and helps make them interoperable by defining common interfaces between these systems management specifications and the protocols themselves as well as between these systems management specifications and OS-level agents and other management protocols. As a result, IPMI can help to enhance platform manageability, availability, and productivity while helping to maintain scalability and security. IPMI also can potentially reduce the maintenance and management costs of computing resources by enhancing node uptime through facilitating the proactive replacement of failing components and eliminating one management fabric from the cluster. Yung-Chin Fang is a senior consultant in the Scalable Systems Group at Dell. He specializes in cyberinfrastructure management and high-performance computing. He participates in open source groups and standards organizations as a Dell representative and has published dozens of technical and conference papers. ipmish ip 10.10.10.1 u user p password sel power off The BMC with IP address 10.10.10.1 authenticates the user and performs an immediate power-down ipmish ip 10.10.10.1 u user p power sel get -last 45 The BMC with IP address 10.10.10.1 authenticates the user and returns the last 45 events from the SEL Interoperable cluster management using IPMI For HPC cluster remote hardware health condition management and monitoring, IPMI allows a separate management fabric that can be used by OS-level management applications. IPMI can be used independently of the managed server s OS, even when the system is powered down. During the two main phases of HPC cluster management, IPMI enables the following functions: HPC cluster deployment Power up a node remotely for deployment Check a node s health condition remotely Power cycle a node remotely to bring up the cluster configuration after the OS has been deployed Garima Kochhar is a systems engineer in the Scalable Systems Group at Dell. She has a B.S. in Computer Science and Physics from Birla Institute of Technology and Science (BITS) in Pilani, India. She has an M.S. from The Ohio State University, where she worked in the area of job scheduling. Randy DeRoeck is a systems engineer in the Scalable Systems Group at Dell. Randy has a B.S. in Computer Science from Arkansas State University. FOR MORE INFORMATION IPMI adopters list: http://developer.intel.com/design/servers/ipmi/adopterlist.htm IPMI home page: http://developer.intel.com/design/servers/ipmi www.dell.com/powersolutions POWER SOLUTIONS 63