HP Cluster Platform Overview

Transcription

1 HP Cluster Platform Overview Abstract This document describes the benefits, hardware support, software support, and installation considerations of HP Cluster Platform systems. HP Part Number: Published: June 2012 Edition: 1

2 Copyright 2012 Hewlett Packard The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Acknowledgements Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Warranty WARRANTY STATEMENT: To obtain a copy of the warranty for this product, see the warranty information website:

3 Contents 1 Concepts...4 Architecture...4 Nodes...4 Console and Administrative Networks...5 Console Network...5 Administrative Network...6 Connecting Console and Administrative Networks...7 Regions...7 System Interconnect...8 InfiniBand...8 Ethernet...9 Physical design Configuration guidelines...12 Hardware...12 Control Nodes...12 Application Nodes...12 Additional information...12 Operating environments...12 Cluster management...12 Applications and tools...13 Cluster interconnects Supported HP Cluster Platform components...14 System Interconnects...14 Servers...14 Administrative and Console Networks...15 Racks Installation considerations...16 Planning for the installation...16 Customer responsibilities...16 HP service provider responsibilities...16 Installation time...16 Personnel...16 Disposing packing materials...17 Site requirements...17 Unpacking space...17 Data center requirements...17 Rack information...17 Rack stability Support and other resources...19 Contacting HP...19 New and changed information in this edition...19 Typographic conventions Documentation feedback...20 Glossary...21 Index...23 Contents 3

4 1 Concepts This chapter describes HP Cluster Platform architecture and technology. An HP Cluster Platform is a factory-integrated package of HP ProLiant nodes (servers), management networks, high-speed interconnects, and optional storage. HP Cluster Platform configurations deliver high performance computing. Configurations are based on a common base architecture, which allows for flexible customized systems, including choices for server and interconnect types. NOTE: See your HP services representative to discuss your specific requirements. Architecture Nodes HP Cluster Platform configurations combine a set of independent computers into a unified system through high-speed networking technologies. Configurations can scale from a small number of nodes to thousands of nodes, where one node equates to an individual server. However, the practical number of nodes is determined by hardware and software thresholds. An HP Cluster Platform provides: Preconfigured building blocks based on a range of technologies that can be easily and reliably built using standard manufacturing processes Flexibility to build clusters of varying size and configuration using those building blocks Future growth and scalability HP assistance with planning the configuration that fits your needs and ordering Cluster Platform products Hardware that is shipped installed in racks and pretested at the factory Optional preinstalled and tested software The major components of an HP Cluster Platform are: Nodes (page 4) Console and Administrative Networks (page 5) System Interconnect (page 8) Optionally, an HP Cluster Platform can include storage devices. A node is a ProLiant server. The two types of nodes are Control and Application: Control Node Provides basic cluster management services, which are defined by the software running on the HP Cluster Platform, not by HP Cluster Platform itself. There must be one Control Node. A second Control Node can be added to provide a high-availability environment. The Control Node can have external, direct attached storage. Application Node Used primarily or exclusively to run computational tasks and applications. The majority of nodes in the cluster are Application Nodes, which are organized in node groups. A Cluster Platform must have at least one Application Node group. There may be additional node groups. All nodes in a node group are configured identically. Figure 1 (page 5) illustrates a typical HP Cluster Platform set of nodes. 4 Concepts

5 Figure 1 Typical HP Cluster Platform set of nodes Control Node 2. Application Nodes 3. Additional node group 4. Additional node group The Control Nodes are also considered a separate node group. A Cluster Platform comprises at least two node groups (Control Node and one other group) and can contain many node groups. NOTE: There is no limit to the number of node groups. However, there is a configuration-dependent maximum number of supported nodes. Node groups can be used to configure nodes that run compute intensive applications, provide services and access to external capabilities, such as Login services, SAN connections, and gateways to external networks. Console and Administrative Networks This section describes the networks that are primarily used for management of the cluster. Console Network The Console Network is an Ethernet network used for typical console functions, such as power ON/OFF, boot, and hardware status verification. Command protocol is determined by the node type, but is typically ilo or IPMI. All nodes are connected to the Console Network. The Console Network is implemented with one or more 10/100 Ethernet switches distributed in the racks with the nodes. If there are two or more switches, one switch is designated as the Console Root switch and the remaining switches are designated as Console Leaf switches. The Console Root switch aggregates the Console Leaf switches in a two-level tree topology. The Control Node is connected to the Console Root switch. Application Nodes are typically connected to the Console Leaf switches. In some configurations, the Application Nodes may be connected directly to the Console Root switch. Figure 2 (page 6) illustrates the Console Network connections. Architecture 5

6 Figure 2 Console Network connections Console Root switch 2. Console Leaf switches Administrative Network HP Cluster Platforms use a 1 gigabit/second Ethernet (GbE) network to manage and administrate the cluster at the operating system level, for example, image distribution or job management. All nodes are connected to the Administrative Network using a GbE port. The Administrative Network is implemented with one or more GbE switches distributed in the racks with the nodes. If there are two or more switches, one switch is designated the Administrative Root switch and the remaining switches are designated Administrative Leaf switches. The Administrative Root switch aggregates the Administrative Leaf switches in a two-level tree topology. The Control Node is connected to the Administrative Root switch. The Console Root switch can also be connected to the Administrative Root switch. Application Nodes are typically connected to Administrative Leaf switches. However, if the console port of an Application Node is connected to the Console Root switch, that Application Node is also connected to the Administrative Root switch. Figure 3 (page 6) illustrates the Administrative Network connections. Figure 3 Administrative Network Administrative Root switch 2. Administrative Leaf switches 6 Concepts

7 Connecting Console and Administrative Networks Regions Depending on your specific needs, the Console and Administrative Networks can be connected in one of the following ways: The Console Root switch is connected to the Administrative Root switch. The Console Root switch is connected to a second Ethernet port on the Control Node, keeping the Console and Administrative Network separate. For very large systems, the cluster is divided into regions. Each region is a collection of nodes under an Administrative Root switch. Regions are connected through a Super Root switch. Figure 4 (page 7) illustrates a cluster with two regions. Figure 4 Cluster with regions Region 1 2. Region 2 3. Super Root switch NOTE: As in smaller HP Cluster Platforms, HP Cluster Platform with regions require one Control Node and can have a second optional Control Node. If you use a second Control Node in a large HP Cluster Platform, HP recommends that you install it in a different region than that which houses the first Control Node. Architecture 7

8 System Interconnect The System Interconnect is an optional separate network that provides communication between nodes. The two technology options for System Interconnect are: InfiniBand Ethernet If System Interconnect is present, all nodes except the Control Node must be connected to it. Although connecting the System Interconnect to the Control Node is optional in an HP Cluster Platform, your operating system software can require that it be connected. Figure 5 (page 8) illustrates the System Interconnect connections. Figure 5 System Interconnect 1 InfiniBand 1. System Interconnect InfiniBand provides ultra high bandwidth and low latency, and is ideal for cluster computing, low latency messaging systems, scalable clustered database, and data warehouse applications. For more information, see the InfiniBand Quickspecs at quickspecs/13078_div/13078_div.html and the InfiniBand matrix at h20311.www2.hp.com/hpc/us/en/infiniband-matrix.html. Interconnect switches that provide connections for nodes are described as Leaf switches. An interconnect switch that provides connections to other switches is called a Spine or Core switch. The InfiniBand fabric is implemented using a topology often referred to as a fat tree. Cluster Platform supports two variations on this topology. The first is referred to as one-to-one fat tree. This topology is characterized by symmetrical hierarchical connections providing a constant cross-sectional bandwidth. Figure 6 (page 9) illustrates the connection scheme. If a Leaf switch has a total of n ports, then n/2 are used to connect to nodes and the remaining n/2 create connections distributed over the Spine switches. 8 Concepts

9 Figure 6 One-to-one fat tree topology Nodes 1. Root switch 2. Leaf switches 3. N number of nodes The second variation on the topology is referred to as two-to-one fat tree. This topology also has symmetrical connections between the Leaf and Spine switches, but has half has many as in the one-to-one topology. For example, if a Leaf switch has n ports, then typically 2n/3 ports connect to nodes and n/3 connect to Spine switches. Figure 7 Two-to-one fat tree topology Nodes Ethernet 1. Root switch 2. Leaf switches 3. N number of nodes Ethernet networks are implemented as tree topologies as illustrated in Figure 8 (page 9). With this topology, there are fewer connection paths between nodes on different Leaf switches than in a fat tree. Consequently there is a greater potential for congestion, though the performance impact is dependent on the network use by the application. Figure 8 Tree topology Nodes 1. Root switch 2. Leaf switches 3. N number of nodes Architecture 9

10 Physical design The physical design of an HP Cluster Platform is based on modular building blocks that are created when the cluster is built at the factory. These building blocks are defined as follows: Compute Building Block (CBB) The majority of the cluster consists of Application Nodes that are organized in single-rack modules called CBBs. A CBB typically contains Application Nodes, Administrative Network Leaf switches, Console Network Leaf switches, PDUs, and optionally System Interconnect switches. Utility Building Block (UBB) The UBB contains supporting infrastructure components including: Control Node Administrative Root switch Console Root switch Keyboard, video, and mouse (KVM) connected to the Control Node Additional space in the UBB can be used for other components, such as disk storage systems. The rack space usage is defined for each specific HP Cluster Platform implementation or is reserved space. To ensure adequate airflow and cooling, HP recommends that you not use any unoccupied rack space for components that are not part of the cluster. Interconnect Building Block (IBB) IBB racks contain System Interconnect switches. Spine/Core switches are typically placed in the IBB; Leaf switches can be placed in the IBB or distributed into CBB racks with nodes. In a small configuration, System Interconnect switches can be placed in the UBB to conserve floor space. An IBB may contain multiple System Interconnect switches and a cluster may contain more than one IBB. Figure 9 (page 11) illustrates a typical HP Cluster Platform system. 10 Concepts

11 Figure 9 Example of a typical HP Cluster Platform system Additional Node Group 2. Administrative Network switches 3. Console Network switches 4. Application Nodes 5. Additional Node Group in UBB 6. Control Node 7. Keyboard, video, mouse (KVM) 8. Administrative Root switch 9. Console Root switch 10. System Interconnect 11. Power distribution units (PDUs) 12. Compute Building Blocks (CBBs) 13. Interconnect Building Block (IBB) 14. Utility Building Block (UBB) Physical design 11

12 2 Configuration guidelines Hardware This chapter describes HP Cluster Platform configuration guidelines. Control Nodes Typically, Control Nodes have: Two multicore processors At least 1 GB memory per core processor One or more internal disks A drive capable of reading DVDs Redundant power supplies if available Optional connection to external storage The Control Node is typically connected to the System Interconnect if one is included in the Cluster Platform. However, this connection is optional. Application Nodes Typically, Application Nodes have: Two multicore processors At least 1 GB or memory per core processor Optional external disks Typically, Application Nodes do not have: Redundant power connections CD or DVD drives Additional information This section describes sources for additional information about HP Cluster Platform. For the most up-to-date information about HP Cluster Platform, see the HP Cluster Platform website h20311.www2.hp.com/hpc/cache/ html. Operating environments HP Cluster Platform supports HP ProLiant servers running open source Linux (Red Hat and SUSE) and Windows HPC Server 2008 (HPCS). Linux can be installed and tested at the factory before your HP Cluster Platform ships. Windows operating system must be installed on-site. For the latest information on operating system support, see cache/ html. Cluster management HP Cluster Platform supports cluster management options from HP as well as partner developed solutions. 12 Configuration guidelines

13 For the latest information on cluster management, see html and the Unified Cluster Portfolio website at HPC/cache/ html. HP Cluster Platform supports the following cluster management tools: HP Insight Cluster Management Utility (Insight CMU) For more information, see: Windows HPC Server (HPCS) 2008 For the latest information, see: Additional software for HP Cluster Platforms running Linux include: HPC Linux Value Pack Altair PBSPro Platform LSF Adaptive Moab Applications and tools HP Cluster Platform provides, with its partners, a complete software development environment for all supported operating systems and supports leading applications in high-performance computing. For the latest information about the available applications and tools for HP Cluster Platform, see Cluster interconnects HP provides a choice of interconnect products, including InfiniBand and Ethernet, from industry-leading vendors. For the latest information on supported cluster interconnects, see HPC/cache/ html. Additional information 13

14 nl nl nl nl 3 Supported HP Cluster Platform components This chapter describes the currently supported components used in an HP Cluster Platform. System Interconnects Table 1 (page 14) lists the supported System Interconnects used in an HP Cluster Platform. Table 1 Supported System Interconnects Type InfiniBand FDR InfiniBand QDR Vendor Mellanox Voltaire Mellanox QLogic Fabric management UFM or OpenSM UFM or OpenSM UFM or OpenSM IFS or OpenSM Servers See your HP representative for the most up-to-date information on supported System Interconnects available with HP Cluster Platform configurations. For more information on HP supported interconnects, see _div.html. Table 2 (page 14) lists the supported families of servers. Work with your HP representative to ensure that you choose the best servers for your requirements. Table 2 Supported servers Gen8 BL460c Gen8 DL160 Gen8 DL380p Gen8 SL230s Gen8 SL250s Gen8 DL385p Gen8 BL465c Gen8 G7 BL2x220c G7 BL460c G7 BL465c G7 DL165 G7 DL385 G7 DL380 G7 SL390s G7 G6 BL280c G6 DL160 G6 DL170e G6 SL160s G6 SL170s G6 For more information on servers, see the following websites: HP ProLiant servers most often used for HPC: HP DL Rack Mount Servers: HP BL BladeSystem: HP SL Scalable System and HP Integrity Systems: 14 Supported HP Cluster Platform components

15 Administrative and Console Networks Table 3 (page 15) lists the switches supported for use in the Administrative and Console Networks. Table 3 Supported switches for Administrative and Console Networks Network Administrative Network Console Network Vendor HP Networking Switch model 2910al 2620 Racks For more information about HP Networking switches, see products/switches/index.aspx. Supported racks for HP Cluster Platform are: HP G2 Series HP Intelligent Series Rack For more information about racks, see Rack information (page 17) and h18004.www1.hp.com/products/servers/proliantstorage/racks/index.html. Administrative and Console Networks 15

16 4 Installation considerations Planning for the installation The customer and HP support share the responsibilities for installation planning. The required installation planning tasks must be scheduled and completed to ensure successful installation of an HP Cluster Platform. Customer responsibilities Prior to an HP Cluster Platform installation, you must: Understand safety requirements associated with installing an HP Cluster Platform. Work with your HP services representative to ensure that you understand the installation requirements for an HP Cluster Platform. Verify that the installation site meets all installation requirements by consulting with your HP services representative. Provide electrical hardware, including cables, connectors, and receptacles that are required to connect the HP Cluster Platform to site power. As needed, work with your HP services representative to create an installation plan. Allow enough time to complete any changes to the plan to ensure that your site is ready when the equipment arrives. HP service provider responsibilities The HP service provider is responsible for: Providing assistance as needed during the installation planning process for your specific site and operational configuration. Coordinating HP resources to ensure a successful installation and configuration. Installation time Personnel When planning an installation schedule, consider the following tasks: Unpacking the Cluster Platform Rack preparation Inter-rack cabling Depending on the number of networking cables, the installation can take up to four hours per rack. In addition to the actual installation time, you might need to plan and schedule downtime for adjacent systems, particularly if they are moved or powered down to facilitate the cluster installation. Plan for the availability of key personnel to provide specific services or to assist in moving and unpacking the cluster. Availability of the following key personnel ensures that the HP service representative can install the cluster with maximum efficiency. System or site administrators Site electrical engineers Security personnel for access to restricted areas. Ensure that security badges or pass keys are available to HP service personnel, where appropriate. 16 Installation considerations

17 nl nl Suitably equipped and trained personnel to assist in heavy lifting. While component removal is not anticipated, post-installation troubleshooting might require the removal of components. WARNING! The weight of some rack-mounted components requires the use of a mechanical lift or at least two people to remove and replace the component in the rack. Before replacing a component, see the component's QuickSpecs at for weight information. Shipping room personnel if there are problems locating items in a shipment, if there are damaged items, or for relocating the pallets, if needed. Key personnel who are involved in the installation hand-off. When the installation and hardware verification process is complete, trained administrators might need to be present to boot the cluster and then begin verifying its operation. Disposing packing materials Shipping an integrated cluster generates far less packaging than the individual components that it contains. However, large clusters use a substantial amount of packaging material that is not reusable. The bulk of the packaging material is recyclable, and is labeled as such. Plan on providing a number of dumpsters into which this packaging can be sorted and recycled. Site requirements The site must accommodate the delivery and movement of the equipment from the receiving dock to the installation location in the data center. Unpacking space When planning for the installation, consider the additional space required to unpack the rack and remove it from its shipping pallet. If there is insufficient space in the computer room, you might need to clear some space in an anteroom or corridor and then roll the system to its installation destination. The following considerations might be relevant: You need 122 cm (48 inches) of clearance all the way around the pallet and above the rack to allow the removal of the packing materials. For operating access, maintain at least 86 cm (34 inches) of clearance in front of the rack to allow the door to open all the way. Data center requirements NOTE: Some single-rack dense cluster models contain a zero-u PDU that is mounted on the side of the rack and accessible by removing the side panel from the rack. Do not block access to the side panel by placing the rack close to other equipment. You must be able to remove the panel to reach the switches and connectors on the zero-u PDUs. Rack information For information about the HP Series, HP G2 Series, and HP Intelligent Series racks, see: Best practices for HP Series and HP G2 Series Racks, available at contenttype=supportmanual&lang=en&cc=us&docindexid=64179&taskid=101& prodtypeid=329290&prodseriesid= HP Rack G2 Series QuickSpecs, available at Site requirements 17

18 nl HP Intelligent Series Rack QuickSpecs, available at Power Rack stability Cluster Platform configurations use a variety of power cords and plugs and you must ensure that your computer room can supply adequate power to your Cluster Platform. The HP Power Calculator utility ( Power%20Calculator%20Catalog.xls) enables IT planners to estimate the power needs of HP ProLiant systems. The power calculator estimates are based on typical component usage. However, power consumption increases with the level of system load utilization. High Performance Computing applications fully utilize the system and therefore have higher power consumption. Consult your HP representative for help with the specific power requirements for your Cluster Platform configuration. Rack stability protects personnel and equipment. WARNING! To reduce the risk of personal injury or damage to equipment: Extend leveling jacks to the floor. Ensure that the full weight of the rack rests on the leveling jacks. Install stabilizing feet on the rack. In multiple-rack installations, fasten racks together securely. Extend only one rack component at a time. Racks can become unstable if more than one component is extended. 18 Installation considerations

19 5 Support and other resources Contacting HP For worldwide technical support information, see the HP support website: Before contacting HP, collect the following information: Product model names and numbers Technical support registration number (if applicable) Product serial numbers Error messages Operating system type and revision level Detailed questions New and changed information in this edition This is a new version of the document. Typographic conventions Table 4 Document conventions Convention Blue text: Table 4 (page 19) Blue, underlined text: Bold, blue, underlined text Element Cross-reference links Website addresses addresses Bold text Keys that are pressed Text typed into a GUI element, such as a box GUI elements that are clicked or selected, such as menu and list items, buttons, tabs, and check boxes Italic text Text emphasis Monospace text File and directory names System output Code Commands, their arguments, and argument values Monospace, italic text Code variables Command variables Monospace, bold text Emphasized monospace text WARNING! Indicates that failure to follow directions could result in bodily harm or death. NOTE: Provides additional information. TIP: Provides helpful hints and shortcuts. Contacting HP 19

20 6 Documentation feedback HP is committed to providing documentation that meets your needs. To help us improve the documentation, send any errors, suggestions, or comments to Documentation Feedback Include the document title and part number, version number, or the URL when submitting your feedback. 20 Documentation feedback

21 Glossary A Administrative Network Application Node 1GbE network used to manage administrative functions. Nodes (servers) used for running computational tasks. B building blocks Units of the Cluster Platform (a single rack) used to house like nodes and associated hardware by function. See CBB, UBB, or IBB. C CBB Console Network Control Node Compute Building Block Unit of a Cluster Platform comprised of Application Nodes. An Ethernet network used for console functions. The node (server) used for cluster management services. There must be one Control Node in a Cluster Platform, though additional Control Nodes can be added for high availability. F FDR Fourteen Data Rate speed of some InfiniBand products. I IBB IFS ilo InfiniBand Insight CMU IPMI Interconnect Building Block unit of a Cluster Platform comprised of System Interconnect switches. InfiniBand Fabric Suite (QLogic). Integrated Lights-Out. Ultra high bandwidth and low latency network. HP Insight Cluster Management Utility. Intelligent Platform Management Interface. K KVM Keyboard, video, and mouse. N node node group A node equates to a single server within the Cluster Platform. Organization of Application Nodes, which are configured identically. A Cluster Platform must have at least one node group. O OpenSM InfiniBand subnet manager for Linux. P PDU Power Distribution Unit. Q QDR Quad Data Rate speed of some InfiniBand products. R regions Divisions of large Cluster Platform systems each having an Administrative Root switch. 21

22 S SAN System Interconnect Storage Area Network. An optional Ethernet or InfiniBand network that provides communication between nodes. U UBB UFM Utility Building Block unit of a Cluster Platform comprised of supporting infrastructure, such as Control Node, Administrative and Console Network root swtiches, and KVM. Unified Fabric Manager (Voltaire). 22 Glossary

23 Index A Administrative Network, 6 Administrative Root switch, 10 Application Node, 4, 10, 12 B building blocks, 10 C CBB, 10 clusters base modules, 10 management, 12 operating systems, 12 Compute Building Block see CBB configurations for HP Cluster Platform, 4 Console Network, 5 Console Root switch, 10 contacting HP, 19 Control Node, 4, 5, 10, 12 conventions document, 19 text symbols, 19 Core switch, 8, 10 CPU type, 4 D document conventions, 19 documentation providing feedback on, 20 E Ethernet, 8 H hardware configuration, 12 help obtaining, 19 HP technical support, 19 HP Cluster Platform configurations, 4 I IBB, 10 InfiniBand, 8 Insight CMU, 13 installation, 16 Interconnect Building Block see IBB interconnects, 13 L Leaf switch, 8, 10 Administrative, 6 Console, 5 N network Administrative, 6 Console, 5 Ethernet, 9 InfiniBand, 8 regions, 7 supported Administrative and Console, 15 System Interconnect, 8 node groups, 4, 5, 10 nodes Application, 4, 12 Control, 4, 12 group, 4 typical set, 4 O one-to-one fat tree topology, 8 operating systems, 12 P physical design, 10 power, 18 ProLiant servers, 4 R racks QuickSpecs, 17 stability warning, 18 supported, 15 regions, 7 Root switch Administrative, 6 Console, 5 S servers supported, 14 site requirements, 17 software, 12 Spine switch, 8, 10 Supper Root switch, 7 switch Core, 8, 10 Leaf, 5, 6, 8, 10 root, 5, 6 Spine, 8, 10 Super Root, 7 symbols in text, 19 System Interconnect, 8, 10 System Interconnects supported, 14 T technical support HP, 19 text symbols, 19 23

24 topology one-to-one fat tree, 8 tree, 9 two-to-one fat tree, 9 tree topology, 9 two-to-one fat tree topology, 9 typographic conventions, 19 U UBB, 10 Utility Building Block see UBB W warning rack stability, Index