CMU : Cluster Management Utility

Transcription

1 CMU : Cluster Management Utility Release Notes Version 4.0, January 2009 Version 4.0: January 2009 Preface 1

2 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Intel, the Intel logo, Itanium, Xeon, and Pentium are trademarks or registered trademarks of Intel Corporation in the United States and other countries. Linux is a U.S. registered trademark of Linus Torvalds. Red Hat and RPM are registered trademarks of Red Hat, Inc. UNIX is a registered trademark of The Open Group. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR and , Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor s standard commercial license. Preface 2

3 Contents 1 Preface Intended audience Related documents Terminology Conventions in this document New and changed features New Operating System Releases New Hardware support Enhancements Diskless compute nodes pre_reconf.sh implementation of a CMU collectl client cmu integration with serviceguard (optional) performance gauge in GUI Implementation of automatic/transparent SSH tunnel from CMU GUI Support of SLES devices id s Other improvements Known bugs DL785 with more than 128 GB of RAM will fail netbooting Cloning will fail on nodes with more than 16 disks Colplot html files lost on cmu rpm unistall SFSg3 issue locale not supported by Xlib, locale set to C Cloning on large size disks may fail GUI xterm failure after serviceguard failover Things to know Blades C class BL460 image deployment time NIC1/NIC2 labels and eth port names mismatch CMU expects the boot partition on partition No LVM nor software RAID support bits servers are no longer supported SELinux and CMU On the management node SELinux on the compute nodes SuSE specific SLES10 SP1 device uuid are now supported Limitations SATA drive support with SLES Itanium specific New servers supported Limitations Fibre channel adapter delay Network boot delay Only partition 0 is supported on cell based systems Power off command on cell based systems...16 Preface 3

4 1 Preface 1-1 Intended audience This guide is intended primarily for system managers and operators who want to configure or manage a large collection of systems in a HPC Cluster Architecture (called cluster in this document). Users should be familiar with the installation and administration of RedHat Linux or SuSE Linux. Distribution-specific commands will not be detailed. 1-2 Related documents The following documents should be used as reference: "Cluster Management Utility Installation Guide". "Cluster Management Utility User's Guide. "Cluster Management Utility Diskless User's Guide. "Cluster Management Utility on Serviceguard Installation Guide". "Cluster Management Utility Hardware preparation Guide". Appropriate Operating System Installation and User Documentation. 1-3 Terminology Table 1 Terminology Term Cluster Definition Collection of commodity servers or workstations linked by a network, in order to build a distributed memory supercomputer. This architecture is commonly called a HPC cluster, running the Linux operating system. Node A node, formally a compute node, is a single machine (server or workstation) which runs a Linux operating system. Each node is a member of the whole cluster. Management Node CMU is installed and runs from the Administration Disk, which is typically mounted on the Management Node. Typically, the management node is also the image server. GUI Client Workstation The CMU graphical user interface can be accessed from any client workstation that can reach the CMU Management Node via the network. Several user on different workstation can use the CMU GUI at the same time. The GUI Client workstation can be running Linux or Windows Preface 4

5 Table 1 Terminology Term Image server or Primary Server Definition This is the Primary Server where clone images are stored. This image server will propagate only to the nodes dedicated as Secondary Server in each Network Entity. CMU runs on that machine that is also named Management Node. Secondary server This is a dedicated node in a Network Entity where the clone image is temporary stored. It will propagate the clone image only to the other nodes that are defined inside the entity. Target Node A compute node that is to receive a clone image from a secondary server. Administration disk CMU is installed on the administration disk, which is located in the image server. A dedicated space may be allocated to the clone images. Master disk or Golden Image A master disk is a reference disk where the user manually installed the operating system and layered products. Its content will be duplicated on the other nodes of a group. Target Disk The hard drive on a target node where the clone image is installed. Management Network CMU communicates with the compute nodes and terminal servers over the management network, which is typically a private, Ethernetbased local area network (LAN) that interconnects all nodes in a cluster. Network Entity This is a group of machines that are connected under an Ethernet switch. In each Network Entity, one of the nodes is temporarily dedicated as the Secondary Server, which means that it has the same role as the Image Server machine. A node belongs to a unique Network Entity. Logical Group This is a group of similar machines (same hardware) which are running the same image. A node can belong to several logical groups Clone image This is the compressed image of the installation done on a master disk. One clone image per logical group is needed. Preface 5

6 Table 1 Terminology Term Management Card Definition This stands for the hardware device included into a server to enhance its remote manageability. The management card supported by CMU are ILO/ILO2, Integrity ILO and LO100i. ILO/ILO2 Select this card in the CMU GUI when your server is a recent ProLiant DL3xx/DL5xx or P-Class C-Class Blade with an ILO or ILO2 management card Integrity ILO Select this card in the CMU GUI when your server is an Integrity server Lights-Out 100i Select this card in the CMU GUI when your server is a DL14x server with a LO100i management card. EFI EFI is the command line BIOS of the HP Integrity servers 1-4 Conventions in this document NOTE: Unless otherwise noted, press the Return or Enter key after entering commands or responses to command prompts. The following conventions are used in this guide: Table 2 Conventions Conventions Description # A pound sign (#) is the default superuser prompt. % A percent sign (%) is the default user prompt. <Return> Ctrl/C A boxed symbol indicates that the user must press the named key on the keyboard. This symbol indicates that the user must press the Ctrl key while he simultaneously presses another key (in this case, C). % cat In interactive examples, typed user input appears in a bold typeface. Bold typeface is also used to emphasize important words or phrases. Monospaced or italics UPPERCASE, lowercase [y] In text, this typeface indicates the exact name of a command, routine, partition, pathname, directory, or file. This typeface is also used in interactive examples and other screen displays. Linux differentiates between lowercase and uppercase characters. Literal strings that appear in text, examples, syntax descriptions, and function definitions must be typed exactly as shown. In a prompt, square brackets indicate that the enclosed item is the default response. For example, [y] means the default response is Yes. Preface 6

7 2 New and changed features 2-1 New Operating System Releases Support for: RH3U9, RH5U2, SLES10SP2 (also unofficial support for upcoming Lenny). 2-2 New Hardware support DL785 (patch for above 128GB of RAM provided on CMU CDROM), BL2x220c, DL185, BL260c, DL Enhancements Diskless compute nodes CMU is now supporting a diskless mode. Parts or all of the compute nodes into a CMU cluster can now be running diskless assuming the following limitations: diskless is only supported for x86/x86_64 family; diskless is only supported for a maximum of 192 compute nodes; diskless is only supported for Redhat 4.4 and Redhat 5.0 and 5.2; please refer to the diskless documentation for details on how to implement a diskless cluster pre_reconf.sh Allows to run arbitrary commands after netboot before cloning. Example: flashing firmwares (platform dependent). See CMU Users Guide for details implementation of a CMU collectl client CMU is now able to defer the node performance data collection to collectl. This is critical for diskless clusters. Metrics can be mixed between collectl and cmu legacy. Refer to CMU Users Guide for details cmu integration with serviceguard (optional) CMU has been re-audited and re-engineered for a better serviceguard support. GUI/monitoring only is secured. Refer to documentation performance gauge in GUI Left click in center of any metric flower to see the perf gauge. It allows easy visualisation of for example global throughput. New and changed features 7

8 2-3-6 Implementation of automatic/transparent SSH tunnel from CMU GUI For Linux workstation displaying the GUI, no more configuration of the X server is necessary (exit removal of no-listen flag for X servers). This only works on x86/x86_64 and IA64 Linux. Contact us if this is a problem Support of SLES devices id s /dev/by-id syntax in /etc/fstab is now supported. necessary. No more manual editing of /etc/fstab is Other improvements CMU migration to dhcp instead of bootp CMU migration to debian during netboot/backup CMU does not depend anymore on 'hostname -i'. It is still the default mode (backward compatibility) but it can now be set to any interface in the system. Refer to CMU Installation Guide for details. faster cloning engine: CMU is now parallelizing disk formatting processes (good with non SAS drives). better rpm implementation (better dependency handling...) code cleanup (customer bugs, valgrind bugs...) more states in the cloning engine (status window) avoiding locking ssh session when starting cmuservice (cmuserver start would lock ssh sessions) more complete cmu_* command set (cmu_add_node bugfix since last release) better handling of /etc/hosts entries console broadcast code rewrite for better performance/reliability autodetection of the name of the ifcfg-* file to reconfigure for a RH like OS (fix BL460 nic inversion problem) remove unused ifcfg-eth-id* files from backup node for SLES intelligent redisplay of flowers on GUI resize remote file browser ( kickstart + pdcp ) cleanup of temporary files hdparm is part of the netboot image for pre_reconf.sh scripts (ex: bl2x220c cloning) rewrite of the default set of metrics in both collectl & cmu legacy syntax (warning SLES10SP1 people and smartarray) better feedback on cmu_cn rpm installation problems (dependency feedback) upgrade to pdsh fanout = pdcp wrapper into GUI/cli implementation of cmu_mod_node to modify a compute node (or part of it) from the command line or in interactive mode thttpd webserver can now serve colplot New and changed features 8

9 3 Known bugs 3-1 DL785 with more than 128 GB of RAM will fail netbooting Apply the patch patch-cmu_4.0.0-x86_ proposed in the patches section of the CDROM. 3-2 Cloning will fail on nodes with more than 16 disks Cloning will fail on compute nodes with more than 16 disks. To fix the problem, apply the patch patch-cmu_4.0.0-x86_ proposed in the CDROM. 3-3 Colplot html files lost on cmu rpm unistall Uninstalling CMU will erase /opt/cmu/www therefore erasing /opt/cmu/www/colplot containing html files. History data are not affected, provided that the /opt/cmu/www directory is not used as common collectl directory for compute nodes. Workaround: after the new cmu installation, copy again files from the original colplot www directory to /opt/cmu/www as follows: Redhat: # cp -a /var/www/html/colplot /opt/cmu/www/colplot SuSE: # cp -a /srv/www/htdocs/colplot /opt/cmu/www/colplot 3-4 SFSg3 issue SFSg3 provided kickstart file install the systems with a firewall setting forbidding cmu monitoring to properly function. Opening UDP ports > (or stopping the firewall ) will make SFSg3 CMU friendly. 3-5 locale not supported by Xlib, locale set to C Sometimes when you try to open an xterm from the shell the command is successful but you get the warning: Warning: locale not supported by Xlib, locale set to C On /opt/cmu/log/cmuserver-0.log you get the following messages: Workaround: process 0 success???, but error stream not empty. The display is not :0</message>process 1 failed. The display is not :1</message>Display info for client is empty. Xserver is not working properly export LANG=en_US.UTF Cloning on large size disks may fail Known bugs 9

10 When cloning on large size disks (e.g. 750GB), the cloning process fails and types the following line on the pop up window: Cloning finished, 0 nodes cloned, 0 nodes finished Workaround: add this line in cmucerbere-clone.orig: export CMU_NO_STATUS_CHANGE_TIMEOUT=7200 This example sets the timer to 2 hours. 3-7 GUI xterm failure after serviceguard failover On serviceguard configuration, if you have a GUI session opened on a Windows workstation and a failover occurs, once CMU resumes on the alternate CMU server xterms creation from the GUI will fail with the message cannot access your local X server ; this affects commands like ssh, pdsh, backup, cloning from the GUI. Workaround: exit and re-enter privileged mode on the GUI. This will clear the problem. Known bugs 10

11 4 Things to know 4-1 Blades C class BL460 image deployment time On Blades C Class BL 460 with SATA controllers, if Cache Write Back is disabled the deployment time of the cmu image can take a long time. Writing on the disk can be as slow as 8MB/sec, which means for example that writing a 1GB image can take more than 2 hours. When this occurs, the cloning process will fail due to timeout. To avoid this, enable write back cache. Note: enabling cache write back option is available only on 128 MB cache modules. WORKAROUND: if you cannot avoid this problem, add this line in cmucerbere-clone.orig: export CMU_NO_STATUS_CHANGE_TIMEOUT=7200 This example sets the timer to 2 hours. 4-2 NIC1/NIC2 labels and eth port names mismatch Some servers (e.g. BL460, DL145) initialize embedded Ethernet ports in descending PCI bus order raher than ascending order. As generally Linux assigns eth port names in the same sequence as they are discovered, this may result in NIC1 NIC2 labels on the back front of the servers not being in the same order as eth0 eth1 Linux names. This can be easily detected when you install the first node. WORKAROUND: Either: swap the ethernet cables (when possible) or edit /opt/cmu/scripts/cmucerbere-clone.orig and modify the following line: to export CMU_NBOOT_MEDIA=eth0 export CMU_NBOOT_MEDIA=eth1 Note: this beheaviour may be Linux distribution dependent. 4-3 CMU expects the boot partition on partition 1 By default, the /boot partition is expected to be on partition 1. If you want to use another partition, you have to specify it to cmu by editing /opt/cmu/scripts/cmucerbere-clone.orig and adding the following line (the example is for partition 3): export CMU_BOOTMGR_PARTITION=3 Failure to do that will result in the cloned node to be unable to boot. 4-4 No LVM nor software RAID support Things to know 11

12 Do not install LVM on the CMU compute nodes has CMU does node support LVM or Software RAID during backup/cloning bits servers are no longer supported. CMU is no longer supported on 32 bit Servers. Nevertheless, 32 bit software on 64 bits servers is still supported. Things to know 12

13 4-6 SELinux and CMU On the management node SELinux has been tested on x86_64 and IA64 and is working fine on the management node with Redhat4. Starting with Redhat 5, the default SELinux policy shipped will stop the tftp server to function. You can change the default setting by executing the command: setsebool -P tftpd_disable_trans=0 Restart xinetd after. Note: This setting is persistent accross reboots SELinux on the compute nodes If you want to use SELinux on compute nodes, be aware of the following points: Relabeling cloned nodes Cloned nodes has to be SELinux-relabelled. In order to do this, starting with version 3.1 Update2, the default reconf.sh file in /opt/cmu/etc/reconf.sh includes the following commented line: #touch ${CMU_RCFG_PATH}/.autorelabel Uncommenting this line will allow cloning SELinux compute nodes. This will force a full filesystem relabeling and a reboot after cloning. Note: if you already have a backup and want to clone a SELinux golden node you will have to manually edit all /opt/cmu/image/*/reconf.sh). Installing cmu_cn rpm on compute nodes fails if SELinux is set to "enforcing". WORKAROUND: Set SELinux to "disabled" on compute nodes before installing monitoring rpms, then set it back to "enforcing". Things to know 13

14 5 SuSE specific SLES10 SP1 device uuid are now supported /dev/by-id syntax in /etc/fstab is now supported. necessary. No more manual editing of /etc/fstab is 5-2 Limitations SATA drive support with SLES 9.0 By default SLES 9.0 is using a old SATA driver which is not supported by CMU. Please use the parameter "amd74xx.skip_sata=1" at boot time and change hd into sd in your fstab SuSE specific 14

15 6 Itanium specific 6-1 New servers supported Since Release 3.1 update 2, the following itanium server have been qualified under SLES 10 SP1: RX8640 RX7640 SX1000 (Superdome) Qualification test have been run with the following MP and EFI firware: MP version EFI Version RX8640 B [14.61] RX7640 B [14.61] Superdome [14.61] 6-2 Limitations Fibre channel adapter delay Some Fibre channel device may delay the scan of boot device in the efi shell. This delay may cause malfunctioning of some cmu command (scan Mac address, Network boot). If this happen, proceed as follows: edit the file /opt/cmu/tools/lib_cmu_common.tcl change the value of CMU_FC_TIMEOUT (default = 0) ; a value of 30 should fix the problem Network boot delay Depending on the size of the partition, some cell systems may take a relatively long time to perform the hardware initialization during boot. This may cause some CMU functionality to fail (scan Mac address, Network boot, backup ). If this happens, proceed as follows: edit the file /opt/cmu/tools/lib_cmu_common.tcl change the value of ITANIUM_NETBOOT_TIMER (default = 480). A specific timer allows to handle a similar problem during cloning operations. Proceed as follows: edit the file /opt/cmu/scripts/cmucerbere-clone.orig export CMU_CERBST_NET_BOOTING_TIMEOUT=600 (default = 480) Itanium specific 15

16 6-2-3 Only partition 0 is supported on cell based systems On cell based systems, CMU commands as boot, power-on, power-off, backup, cloning operate only on partition Power off command on cell based systems As CMU has visibility only on hardware partition 0, the choice has been taken not to power off the entire cabinet when executing the power off command. If you need to change this behavior, you have to install the patch patch-cmu_3.2.0-ia available on the CMU CDROM. Note: this patch is not mandatory. Itanium specific 16

17 17