CMU : Cluster Management Utility CMU diskless user s guide Version 4.0, January 2009 Version 4.0 January 2009
2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Intel, the Intel logo, Itanium, Xeon, and Pentium are trademarks or registered trademarks of Intel Corporation in the United States and other countries. Linux is a U.S. registered trademark of Linus Torvalds. Red Hat and RPM are registered trademarks of Red Hat, Inc. UNIX is a registered trademark of The Open Group. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor s standard commercial license.
CMU diskless user s guide... 1 Version 4.0, January 2009... 1 1 Preface... 4 1-1 About this document... 4 1-2 Intended audience... 4 1-3 Related documents... 4 2 CMU Diskless installation... 5 2-1-1 Introduction... 5 2-2 Limitations... 5 2-3 Prerequisites... 5 2-3-1 Administation node prerequisites... 5 2-3-2 Golden node prerequisites... 5 2-4 CMU Management node configuration... 6 2-4-1 Initial CMU installation... 6 2-4-2 modify the tftp Server configuration (compared to cmu classic)... 6 2-4-3 Activating the diskless feature... 6 2-5 Using CMU Diskless feature via GUI... 7 2-5-1 Populate the CMU Database... 7 2-5-2 Create a diskless image... 7 2-6 Using the CMU Diskless feature via the CLI...10 2-6-1 Populates the CMU Database...10 2-6-2 Register a diskless image...10 2-6-3 Add the nodes into the logical group...10 2-6-4 Boot the compute nodes...10 2-7 Understanding the structure of a diskless image...11 2-7-1 Generic considerations...11 2-7-2 CMU implementation...11 2-7-3 Customizing your diskless image...11 2-7-4 Using reconf-diskless-image.sh...12 2-7-5 Using reconf-diskless-snapshot.sh...13 2-7-6 Templates and image file...14 2-7-7 Best practices for diskless clusters...14
1 Preface 1-1 About this document This guide describes how to configure a diskless Cluster using HP s Cluster Management Utility (CMU) V4.0 on HP systems. CMU is a software dedicated to the administration of HPC clusters. This software runs with RedHat Entreprise Linux 3.0/4.0/5.0 Suse SLES 10.1 (x86/x86_64/integrity), SuSE SLES 9.0/10.1 (x86/x86_64) CMU runs on X86-64 system to manage a farm of X86-64 of HP ProLiant servers, or on IA64 system (server or workstation) to manage a farm of HP Integrity servers. 1-2 Intended audience This guide is intended primarily for system managers and operators who want to configure or manage a large collection of systems in a HPC Cluster Architecture (called cluster in this document). Users should be familiar with the installation and administration of RedHat Linux or SuSE Linux. Distribution-specific commands will not be detailed. 1-3 Related documents The following documents should be used as reference: "Cluster Management Utility User's Guide" describes the use of HP s cluster management software. Cluster Management Utility Hardware Preparation Guide describes required hardware settings. Appropriate Operating System Installation and User Documentation.
2 CMU Diskless installation 2-1-1 Introduction This chapter describes how to activate and use the diskless functionality of CMU. For building diskless clusters, CMU uses functionalities and management functions available with the RedHat diskless environment. In the CMU implementation of this environment, most of the operating system is shared from the CMU Administration node between the compute nodes. Additionaly, each compute nodes has its own read/write directories hosted on the administration server for the rest of the operating system. Each time a compute node boots, it mounts most of the OS via NFS as readonly and its own directory, still via NFS, as read-write. Each client has its own read-write directory so that one client can not affect the others. The exported operating system is copied from a third node, referred to as Golden Node, which must be installed with the same RedHat release as the Administration node. Once the installation is complete, the Golden Node can also be network booted as compute node. 2-2 Limitations CMU version 4.0 supports the diskless functionality only with RedHat 4 Update 4 and RedHat 5 Update 2. CMU version 4.0 supports diskless clusters up to a size of 192 nodes. Future versions of CMU will overtake this limit. 2-3 Prerequisites 2-3-1 Administation node prerequisites Install RedHat on your head node. Check that system-config-netboot RPM is installed, if not install it. Install CMU on your Administration server following the procedure documented in the CMU Installation Guide. It is important to follow strictly the current documentation. 2-3-2 Golden node prerequisites Install the same Red Hat distribution as the Administration server on another server that you choose as a diskbased golden node. Any software to be used on the clients must be installed on this system. Please note that 3 packages are mandatory to install on the golden node: The following packages are mandatory for the golden node: busybox-anaconda, dhclient, bind-utils
2-4 CMU Management node configuration 2-4-1 Initial CMU installation Install and configure the head node following every step as for a standard disk based CMU installation. When the classical installation has been done properly, you can follow the next steps to transform it into a diskless CMU installation. 2-4-2 modify the tftp Server configuration (compared to cmu classic) Once the classic CMU configuration is completed, tftp is installed and started on your head node. At this point modify /etc/xinetd.d/tftp as follows : # default: off # description: The tftp server serves files using the trivial file transfer \ # protocol. The tftp protocol is often used to boot diskless \ # workstations, download configuration files to network-aware printers, \ # and to start the installation process for some operating systems. service tftp { disable = no socket_type = dgram protocol = udp wait = yes user = root } server = /usr/sbin/in.tftpd server_args = /tftpboot /opt/cmu/ntbt/tftp -v per_source = 11 cps = 100 2 flags = IPv4 Restart xinetd to reload the tftp configuration. # /etc/init.d/xinetd restart 2-4-3 Activating the diskless feature Edit /opt/cmu/etc/cmuserver.conf and activate CMU_Diskless: #This file is used by the cmuserver script #to run the cmu services. #either change the default value or export CMU_RMI_PORT #if you don't want to use 1099 CMU_RMI_PORT=${CMU_RMI_PORT:-1099} #if set to true will allow cmu to use CMU_PAM_AUTH=${CMU_PAM_AUTH:-false} #local mini web server port CMU_THTTPD_PORT=${CMU_THTTPD_PORT:-80}
#cmu kickstart feature true/false CMU_KS=${CMU_KS:-false} #cmu diskless feature true/false CMU_DISKLESS=${CMU_DISKLESS:-true} #cmu monitoring can be diverted to XC supermon #CMU_MONITORING=${CMU_MONITORING:-supermon} CMU_MONITORING=${CMU_MONITORING:-CMU} CMU_SUDO=${CMU_SUDO:-} Then restart the cmu server: # /opt/cmu/cmuserver restart This completes the Administration node configuration. Next paragraphs describe how to configure a CMU diskless cluster. 2-5 Using CMU Diskless feature via GUI 2-5-1 Populate the CMU Database Register nodes into the CMU database using the scan-node process, as described in CMU User s Guide. There is no difference when adding diskless or diskfull nodes into CMU. 2-5-2 Create a diskless image Pre-requisites Your golden node with disk must be properly installed with RedHat 4 Update 4 and all the applications you need. The 3 mandatory RPMS need to be installed on the golden node busybox-anaconda dhclient bind-utils ssh passwordless must be configured between your head node and your compute node for the root user. To do this, please add the content of /root/.ssh/id_rsa.pub of your cmu administration server into the file /root/.ssh/authorizedkeys on the golden node. Alternatively, you execute the following command: # /opt/cmu/tools/copy_ssh_keys.exp ip golden node Please enter the root password (not echoed): The golden node must be up and running. There is enough space on your head node in /opt/cmu/image to contain a copy of the content of all local file systems currently mounted on your golden node. If you want to use monitoring, the compute node monitoring rpm must be installed on the golden node before creating the diskless logical group. Refer to the CMU User Guide for details. CAUTION: If you plan to use the collectl option for monitoring, be sure that collectl is configured in such way as to avoid any collectl disk I/O operation. Refer to the CMU User Guide, paragraph 7-5 Using Collectl for gathering monitoring data for details.
Create a new diskless logical group As for classical CMU, diskless images are associated to a logical group. To do that click on the ADD button, in the Logical Group Management Frame. As you can see, there is a new diskless checkbox, to the right of the group name entry. Check this checkbox to get the diskless options. NOTE: If you can not see this checkbox, this means that diskless is not activated properly. Go back to section 2-4-3 Activating the diskless feature and correct the error. Enter your logical group name. Then enter the IP of your golden node, and click on get kernel info button. This button will launch a program that ssh to your golden node to get the list of kernels available on this node. Select one of these kernel and click on OK. This launches the diskless image building process. WARNING! This operation may last several minutes, as files are copied from all filesystems of your golden node to build the diskless image. Register some nodes in the golden image. When your diskless group has been created, select some nodes and add them into the diskless image logical group. This operation creates all the personalised read-write directories for each node corresponding to the diskless image. The more nodes you add, the longer this operation takes.
Diskless boot your nodes Select the compute nodes you just added into the diskless logical group and right click to launch a boot command on theses nodes. Then click on the network checkbox. The list of all the diskless image registered into CMU appear. NOTE: the cmu network image is also listed here. This is the CMU classic network boot image, used for cloning and backup.
In the list box, select your diskless image name and click on ok. The compute nodes will then boot on your diskless image... you got a diskless cluster now. 2-6 Using the CMU Diskless feature via the CLI Please read section 2-5-2 and verify your cluster is matching all the pre-requisites Then follow the procedure to create your diskless cluster within cmu command line 2-6-1 Populates the CMU Database Populate the CMU database with your compute nodes (Scannode in Gui) or add_node in CLI 2-6-2 Register a diskless image Start CMU cli: #/opt/cmu/cmucli To create the diskless group you need to know, the IP address of the golden node and the name of the kernel of this golden node that will be used by the diskless nodes. If you don't know which kernel name you should provide, use the probe_kernel command as follows: cmu> probe_kernel 16.16.185.192 2.6.9-42.EL 2.6.9-42.ELsmp kabi-4.0-0 kabi-4.0-0smp cmu> add_logical_group mytestimage 16.16.185.192 2.6.9-42.ELsmp 2-6-3 Add the nodes into the logical group Then simply add your node into your logical group : cmu> add_to_logical_group node1 noden to mytestimage 2-6-4 Boot the compute nodes Finally boot the nodes : cmu> boot net mytestimage node1 - noden
2-7 Understanding the structure of a diskless image 2-7-1 Generic considerations The diskless functionality of CMU is based on the diskless image build process provided by the system-config-netboot RPM. A diskless image is composed of 2 directories : root which contains the rsync of the golden node / and which is mounted in read/only mode by the diskless compute nodes and used as / snapshot which contains one subdirectory per node. Each subdirectory contains the files mounted in read/writes modes per each compute nodes. snapshot contains also 2 files : files which is the list of files that must be present in snapshot. This list is provided by the system-config-netboot RPM. Do not be modify it! files.custom is the list of files you need to add in the snapshot directory of each node. 2-7-2 CMU implementation Like every CMU image, all the directories and files related to a CMU diskless image are stored in /opt/cmu/image/<imagename>. When you create the diskless logical group, CMU invokes the rsync and system-config-netboot commands and populates the /opt/cmu/image/<imagename>/root directory. When you register a set of nodes into a diskless logical group, CMU invokes the systemconfig-netboot command and populates the /opt/cmu/image/<imagename>/snapshot/<nodename> for each of the nodes following the files and directories listed in: /opt/cmu/image/<imagename>/snapshot/files and /opt/cmu/image/<imagename>/snapshot/files.custom When booting a node with a diskless image, CMU modifies the dhcptab to order the nodes to network boot on the given diskless image. 2-7-3 Customizing your diskless image If you need to customize your diskless image, CMU provides 2 defaults scripts reconf-diskless-image.sh and reconf-diskless-snapshot.sh in /opt/cmu/etc. When your diskless image is created, theses 2 scripts are copied in the image directory and executed. The next section will describe how to modify these scripts to customize your image.
2-7-4 Using reconf-diskless-image.sh The script reconf-diskless-image.sh is executed at the end of the image building process. You should put in this script any customization that must be applied in the root directory of the image, i.e in the read-only part of the image mounted by the nodes. You should also put in this script all the changes that must be done before the creation of the snapshot directories(i.e the personalized read/write directory for each compute nodes). For example this is where you can customize the list of files to be copied in the snapshot directory. The reconf-diskless-image.sh content provided by CMU is: #!/bin/sh CMU_RCFG_PATH=$1 CMU_RCFG_OSTYPE=$2 CMU_RCFG_IMAGENAME=$3 CMU_PATH=$4 echo "Reconfiguration script" echo "rootpath:$cmu_rcfg_path" echo "ostype:$cmu_rcfg_ostype" echo "imagename:$cmu_rcfg_imagename" echo "cmupath:$cmu_path" echo "disabling SELINUX" echo " # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - SELinux is fully disabled. SELINUX=disabled # SELINUXTYPE= type of policy in use. Possible values are: # targeted - Only targeted network daemons are protected. # strict - Full SELinux protection. SELINUXTYPE=targeted" > $CMU_RCFG_PATH/etc/selinux/config ### copy /etc/hosts cp -a /etc/hosts ${CMU_RCFG_PATH}/etc/ echo "adding $CMU_PATH into the read/write dirs list" echo " ${CMU_PATH}/ " >> $CMU_PATH/image/$CMU_RCFG_IMAGENAME/snapshot/files.custom echo "creating ssh equivalence between the head node and the image" mkdir ${CMU_RCFG_PATH}/root/.ssh/ cp -a /root/.ssh/id_rsa.pub ${CMU_RCFG_PATH}/root/.ssh/ cp -a /root/.ssh/id_rsa ${CMU_RCFG_PATH}/root/.ssh/ cp -a /root/.ssh/identity.pub ${CMU_RCFG_PATH}/root/.ssh/ cp -a /root/.ssh/identity ${CMU_RCFG_PATH}/root/.ssh/ cp -a /root/.ssh/authorized_keys2 ${CMU_RCFG_PATH}/root/.ssh/ cp -a /root/.ssh/authorized_keys ${CMU_RCFG_PATH}/root/.ssh/ cp -a /root/.ssh/config ${CMU_RCFG_PATH}/root/.ssh/config rm -f ${CMU_RCFG_PATH}/root/.ssh/known_hosts ### Insert your custom reconfiguration scripts here ### End insert
exit 0 CMU calls this script at the end of the image creation process, so that /opt/cmu/image/<imagename>/root is already populated. The script is invoked with 4 input parameters : CMU_RCFG_PATH: the path to the root directory of the image i.e /opt/cmu/image/<imagename>/root CMU_RCFG_OSTYPE : this is the type of OS detected by CMU currently rh4u4 or untested. This parameter is currently unused but will allow later to do specific OS reconfiguration CMU_RCFG_IMAGENAME: This is the name of the image into CMU CMU_PATH: This is the path to the CMU directory (by default /opt/cmu) 2-7-5 Using reconf-diskless-snapshot.sh The script reconf-diskless-snapshot.sh is invoked at the end of each node registration i.e when the snapshot directory of a compute node has just been created. Please add in this script all the customizations which are node specific. For example if you have a network card that needs a static IP address you add the creation of the ifcfg- file in this directory. WARNING! in order to be in the snapshot directory the file needs to be listed in file.custom, so do not forget to add it in the reconf-diskless-image.sh script. Here is the content of the default reconf-diskless-snapshot.sh script provided by CMU: #!/bin/sh CMU_RCFG_PATH=$1 CMU_RCFG_HOSTNAME=$2 CMU_RCFG_IP=$3 CMU_RCFG_IMAGENAME=$4 echo "Reconfiguration script" echo "rootpath:$cmu_rcfg_path" echo "hostname:$cmu_rcfg_hostname" echo "IP:$CMU_RCFG_IP" echo "imagename:$cmu_rcfg_imagename" echo "Cleaning cmu log and temporary files" rm -rf $CMU_RCFG_PATH/opt/cmu/log/* rm -rf $CMU_RCFG_PATH/opt/cmu/tmp/* ### Insert your custom reconfiguration scripts here ### End insert exit 0
This script is invoked once per node, at the end of the image registration process, when the snapshot directory of the node has been populated. The script is invoked with 4 input parameters : CMU_RCFG_PATH: The path to the nodes snapshot direcotry i.e /opt/cmu/image/<imagename>/snapshot/<nodename> CMU_RCFG_HOSTNAME: The name of the node CMU_RCFG_IP : The ip of the node CMU_RCFG_IMAGENAME: The name of the diskless image 2-7-6 Templates and image file If the customizations defined in the reconf files are valid for all your diskless image then you may update the reconf file templates in /opt/cmu/etc. When a need image will be created the updated reconfiguration files will be used. WARNING! the reconfiguration files of the previously created images will not be updated If your customizations are valid for one image only then you just need to modify the reconfiguration files of the /opt/cmu/image/<imagename> directory 2-7-7 Best practices for diskless clusters Do not update the diskless image directly in /opt/cmu/image/<imagename>/root! It is very tempting to modify directly the /opt/cmu/image/<imagename>/root directory (using chroot /opt/cmu/image/<imagename>/root or not) to update the image, but this is a very bad, dangerous and unsupported practice : /opt/cmu/image/<imagename>/root does not contains the exact rsync of the / directory of the golden node. Some of the files of this directory are modified during the diskless image building process to clean the image and transform it into a diskless compatible image. When modifying directly the /root directory you may change one of these modified files and breaks the diskless image The snapshot directories are not synchronized. The registration process copies the files listed into files and files.custom in each node's snapshot directory. When modifying directly the /root directory you may change one of files, but as the snapshot is not updated this will have no effect on your compute nodes. The golden node is not updated. So if you rebuild the diskless image properly following the complete image creation process you will loose every modifications done directly in the /root directory Do not hesitate to make several diskless images. If you need to modify your diskless image, do not hesitate to create a new diskless image into CMU. The image building process and the nodes registration process do not require to stop the diskless cluster, so your users can still work with the previous image. When your new diskless image is ready, just reboot your nodes to put it into production. If you are unhappy with your changes, just reboot again your nodes on the previous image. This methodology is far safer than a online modification directly in the /root directory, that may breaks the production diskless image.
Do not hesitate to use you golden node as a diskless compute node If the boot order of your golden node is properly setup (i.e pxeboot before local hard drive boot). You can choose to boot your golden node on the diskless image and use it as a diskless compute node. When you need to refresh your diskless image, just choose the normal option in the CMU boot menu and your golden node will be removed from the DHCPTAB and will reboot on its local hard drives