Remote Initialization and Configuration of Cluster Nodes

Transcription

1 Remote Initialization and Configuration of Cluster Nodes February 11 th, 2003 Thomas Jefferson National Accelerator Facility Walt Akers, Jie Chen, Chip Watson, Ying Chen, Kelvin Edwards 1. OVERVIEW AND OBJECTIVES Motivation When building clusters of computers, it is highly desirable to have each of the compute nodes configured as close to identical as possible. When the cluster contains only a few systems, this can easily be accomplished by hand. However, as the number of machines begins to grow, there is an increasing likelihood that variations will creep into the configuration due to human error. Consequently, the best approach to ensure uniformity in the configuration of each machine is to automate the configuration process. In addition to ensuring that machines are identical, a fully automated configuration process also allows many systems to be built simultaneously and drastically reduces the time and human intervention required to bring a new cluster online or to perform significant updates to an existing cluster. Our goal in investigating this approach at Jefferson Lab was to develop a remote build system that would limit human intervention to an absolute minimum, and still produce a final system configuration that was ready for immediate use. Computing Environment Our cluster consists of 128 single-processor, Intel compute nodes with myrinet interconnects. The operating system for each node is RedHat Linux version 7.3 employing kernel version While it is not consequential to the build process, the batch system that we are using to deploy jobs is OpenPBS with the locally developed UnderLord scheduling system. 2. HARDWARE REQUIREMENTS PXE Intel s Pre-Boot Execution Environment In order to employ the remote build procedure described in this document, each of the compute nodes must support the Intel Pre-Boot Execution Environment (PXE). When set as the primary boot device in the BIOS, PXE will broadcast the MAC address of the ethenet card to locate a DHCP/boot server. It will then boot from the remote server and begin configuration or regular operation. 3. SOFTWARE REQUIREMENTS All of the software being used for remote configuration is available freely from the internet. Whenever possible the location where this software may be obtained is provided. As we refine our remote build process, we will provide as much of this software as possible from our web site: Note: For all of the application code that we downloaded from the internet, we created a build repository in /usr/local/remoteboot. Following the successful installation of these applications we backed-up the original sources/binaries for safe-keeping. I highly recommend using this approach since free software comes and goes.

2 Dynamic Host Configuration Protocol Server ftp://ftp.isc.org/isc/dhcp/dhcp-latest.tar.gz The Dynamic Host Configuration Protocol (DHCP) is a mechanism by which a node may solicit an IP address using its Ethernet card s hardware (MAC) address. The DHCP server maintains a table containing the relationship between IP and MAC addresses, as well as, other information (gateways, routing, netmask) that a node will require during configuration. Trivial File Transfer Protocol The Trivial File Transfer Protocol (TFTP) is a lightweight mechanism that allows files to be transferred between a server and a client. During initialization, several configuration files must be downloaded from the server to the compute node using TFTP. BpBatch BpBatch is a versatile remote-boot processor, which can be downloaded for free from the Web. It can perform a large variety of actions on a computer at boot-time, before any operating system operation has started. BpBatch is a PXE secondary bootstrap. It takes control at an early stage of the boot process of the computer. Most importantly, it takes control before any operating system. Red Hat KickStart KickStart is a mechanism within RedHat that allows a system to be built using a configuration file and a mounted NFS image of the RedHat distribution. KickStart uses the RedHat Package Manager (RPM) system to automatically install and configure the operating system and supported applications. Additionally, KickStart provides a scripting language that allows additional configuration steps to be performed both before and after the installation of packages. 4. NODE PREPARATION While every effort was taken to eliminate any human interaction with the compute node prior to fulloperation, there are several pre-configuration steps must be performed. The following section provides step-by-step instructions for preparing nodes for remote build/configuration. Step 1) Create a list of machine names and assign an IP Address to each name. a) Assign each name to a physical, unconfigured node. Step 2) BIOS Modification and Address Acquisition a) Power up each machine b) Change the BIOS to set the primary boot device to PXE/UNDI. c) Restart the node and record the MAC address that is displayed as PXE/UNDI attempts to connect to a DHCP server. d) Associate each MAC address with the appropriate IP Address. Step 3) Record this address assignment information for future use. Step 4) Power down the compute node. 5. TFTP INSTALLATION AND CONFIGURATION The Trivial File Transfer Protocol is used exhaustively by BpBatch and by the Intel s PXE/UNDI boot protocol. Use the following steps to install the TFTP server. Step 1) Obtain TFTP from Step 2) Extract the distribution as a subdirectory of the /usr/local/remoteboot directory. Step 3) Install TFTP distribution. Since the incom tftpd daemon is distributed in a precompiled form you must select the version that most closely matches your operating system. Since we were building against RedHat Linux 7.3 kernel , we selected tftpd.libc6.

3 a) Copy /usr/local/remoteboot/tftp.dist/tftpd.libc6 /usr/sbin/tftpd b) Create the /tftpboot directory if it does not already exist. Step 4) Export the /tftpboot directory from the /etc/exports table. Add the following line to the /etc/exports table to globally export the /tftpboot directory: /tftpboot *(ro) Step 5) Start /usr/sbin/tftpd with the following options: -c 64 : Set the number of concurrent connections to 64 -d /tftpboot : Set the current directory for the daemon -h : Enable read ahead buffers -i 0: Exit after num seconds of inactivity (0 = never) -k 5: Specify transmission keep alive time -l /var/log/tftp.log : specify log file -r : restrict access to the current directory -s : Set segment size to 1408 and port to 59 -v 2: Set message verbosity to 2 (high) Step 6) Add the following command to /etc/rc.d/rc.local : /usr/sbin/tftpd -c 64 -d /tftpboot -h - i 0 -k 5 -l /var/log/tftp.log -r -s v 2 >& \ /var/log/tftp.out & 6. BPBATCH INSTALLATION BpBatch, the remote boot environment, will be used to execute the configuration scripts that start the RedHat KickStart. This product is provided as a precompiled executable. Use the following steps to install BpBatch on your boot server. Step 1) Obtain BpBatch from Step 2) Extract the distribution into the /usr/local/remoteboot/bpbatch directory Step 3) Copy bpbatch.p, bpbatch.ovl and bpbatch.hlp from /usr/local/remoteboot/bpbatch to /tftpboot 7. BPBATCH CONFIGURATION Step 1) Create the /tftpboot/boot directory Step 2) Change the permission of /tftpboot/boot to 755 Step 3) Create the boot scripts. Boot scripts are loaded and executed by the BpBatch environment after it has loaded on the remote system. Like any other shell script, they define the course of action that BpBatch will follow. All BpBatch boot scripts MUST have the file extension.bpb. At the simplest level, there are two boot scripts that will be required. a) Create /tftpboot/boot/kickstart-7.3.bpb This script causes the RedHat installer to install the complete operating system from an NFS mounted mirror of the RedHat installation disk. This file should have the following contents:

4 Turn off local caching to avoid disk space problems set CacheNever="On" Call linuxboot to start the build: The first argument is the kernel file name: linux-7.3/vmlinuz The second argument is the command line parameters, in this case it tells linux where to find the kickstart config file. The third argument is the name of the ram disk image. The kernel and ram disk files will be download from the server using TFTP. linuxboot "linux-7.3/vmlinuz" "ks=nfs:qcdadm01:/tftpboot/linux- 7.3/ks.cfg" "linux-7.3/initrd-everything.img" b) Create /tftpboot/boot/boot.bpb This script will cause the system to boot from the hard drive. In case you re wondering, this design dictates that each compute node will always boot from the server first. The script from the server will dictate what actions are taken next: either build or boot. The boot.bpb file should have the following contents: hidebootprom hdboot Step 4) Create links for each compute node in the /tftpboot/boot directory. In the /tftpboot/boot directory, create a link for each of your compute nodes. The name should consist of the IP address with the.bpb extension. For example: if hpc2m001 has IP address , then a link named bpb should be created in this directory and should initially point to the /tftpboot/boot/kickstart-7.3.bpb script. After the node has been built, the link should point to /tftpboot/boot/boot.bpb. 8. REDHAT KICKSTART CONFIGURATION RedHat s KickStart is a powerful system configuration tool that allows the system administrator to have far-reaching power and flexibility in the installation and configuration of the operating system and applications. This power and flexibility, however, requires that the administrator have an indepth understanding of how the system should be configured to the finest detail. This document is not intended to provide a complete description of how KickStart works or to demonstrate all of its capabilities. On the contrary, this section will outline the most fundamental configuration possible, leaving the site developer to determine which options they require and how best to implement them. The following steps should be performed to use KickStart as part of this remote build procedure. Step 1) Identify or install a RedHat mirror site. A RedHat mirror site is an NFS mountable directory containing the RedHat distribution. During the system configuration, this directory will be mounted on the compute node and the packages will be installed from there. For our purposes, we have installed a RedHat mirror on the server qcdadm01 in the /mirror directory. Step 2) Export the RedHat mirror directory from the /etc/exports table. Add the following line to the /etc/exports table to globally export the /mirror directory: /mirror *(ro)

5 Step 3) Create the /tftpboot/linux-7.3 directory This will be the repository for the KickStart configuration file, the RedHat boot kernel and the initial ramdisk image. Step 4) Change the permission of /tftpboot/linux-7.3 to 755 Step 5) Copy the boot kernel and the initial ramdisk image to the /tftpboot/linux-7.3 directory. a) cp /mirror/redhat/linux/7.3/en/os/i386/images/pxeboot/vmlinuz /tftpboot/linux-7.3 b) cp /mirror/redhat/linux/7.3/en/os/i386/images/pxeboot/initrd-everything.img /tftpboot/linux-7.3 Step 6) Create the /tftpboot/linux-7.3/ks.cfg file The ks.cfg file is the master configuration script for KickStart. For the purposes of our design, we installed this file in the /tftpboot/linux-7.3 directory, from which it is downloaded during configuration by the compute node. The contents of the ks.cfg file is left completely to the discretion of the system administrator, however, a brief sample file is provided for the purpose of illustration. The nfs server entry is used to identify the server from which the mirror will be mounted to perform the installation. Note: The program ksconfig is quite useful in generating these files from a pre-configured system System Language lang en_us Language modules to install langsupport en_us US Keyboard keyboard us No mouse mouse none Eastern timezone timezone --utc America/New_York Encrypted root password. This value may be obtained using the grub-md5-crypt utility. rootpw --iscrypted PUT YOUR ENCRYPTED PASSWORD HERE Reboot this system after installation reboot

6 Perform installation using text mode text System bootloader configuration bootloader --location=mbr --uselilo Begin system installation install For the initialization use qcdadm01 ( ) /mirror/redhat/linux/7.3/en/os/i386 nfs --server dir /mirror/redhat/linux/7.3/en/os/i386 Clear all partitions clearpart --all Disk partitioning information part /boot --fstype ext3 --size asprimary part / --fstype ext3 --size asprimary part /opt --fstype ext3 --size 2000 part swap --size 1500 part /scratch --fstype ext3 --size grow Use dynamic networking network --bootproto dhcp System authorization information auth --useshadow --enablemd5 --enablenis --nisdomain CCCHP Firewall configuration firewall --disabled Do not configure the X Window System skipx

7 Here kickstart will install all required packages. Step 7) Add a post processing directive to the ks.cfg file. As the last step in the configuration of the system, we will direct kickstart to download a file named ks.done from the tftp server. A separate application on the server will be monitoring the tftp logs, and will detect when this file is downloaded. Once this file is downloaded by a compute node, the application will automatically change the nodes link in /tftpboot/boot from kickstart-7.3.bpb to boot.bpb. Consequently, the next time the compute node is booted, it will be directed to boot from it s local hard drive. Note: the server name qcdadm01 should be changed to the name of your remoteboot server. %post echo "get linux-7.3/ks.done" /usr/bin/tftp qcdadm01 Step 8) Create the /tftpboot/linux-7.3/ks.done file This is a trigger file that will be downloaded at the successful completion of a nodes configuration. The contents of this file are unimportant. To create the file type the following: a) echo kickstart finished >> /tftpboot/linux-7.3/ks.done Step 9) Compile and install the relinker daemon The relinker daemon will keep a continuous watch on the tftp log file. Whenever the file ks.done is downloaded using tftp, the relinker will change the link in the /tftpboot/boot directory for the specific compute node from the kickstart-7.3.bpb script to the boot.bpb script. Consequently, on the next reboot the compute node will boot from the newly installed operating system. To build the relinker application perform the following steps. a) Create a C++ source file named /usr/local/remoteboot/relinker.c with the following contents: include <stdio.h> include <string.h> include <unistd.h> include <ctype.h> int main () { char buf[1024]; while(fgets(buf, 1024, stdin)!=null) { char * ipptr = NULL; if(strstr(buf, "RRQ")!=NULL && strstr(buf, "/ks.done")!=null && (ipptr = strstr(buf, "from= "))!=NULL) { ipptr+=6; char * ptr = ipptr; while(*ptr && (isdigit(*ptr) *ptr=='.')) ptr++;

8 *ptr = 0; char linkname [128]; char targetname[128]; sprintf(linkname, "/tftpboot/boot/%s.bpb", ipptr); sprintf(targetname, "/tftpboot/boot/boot.bpb"); if(!access(linkname, 00)) unlink(linkname); symlink(targetname, linkname); return 0; b) Compile the source code using the following command: g++ -o relinker relinker.cc c) Copy the relinker daemon to the /usr/sbin directory. d) Add the following lines to the /etc/rc.d/rc.local script. This command should immediately follow the command that starts the tftp daemon. Pipe the output of the tftp Daemon through the relinker program. When the newly built node downloads this file, the relinker will detect the condition and will cause the nodes bpbatch file to be updated. tail -f /var/log/tftp.out /usr/sbin/relinker & 9. DHCP INSTALLATION The Dynamic Host Configuration Protocol server is the backbone of the remote build system. While understanding how to properly configure this application is a time-consuming process, it is a worthwhile investment. Step 1) Obtain DHCP from ftp://ftp.isc.org/isc/dhcp/dhcp-latest.tar.gz Step 2) Extract the distribution as a subdirectory of the /usr/local/remoteboot directory Step 3) Build and Install DHCP distribution (as root) a) Change directories to /usr/local/remoteboot/dhcp-3.0 b) Execute./configure c) Execute make d) Execute make install 10. DHCP CONFIGURATION Step 1) Collect required information a) Domain name : your domain name. This is the search domain that will be listed in /etc/resolv.conf. In our case: "jlab.org". b) Domain name server addresses : The ip addresses of your DNS name servers. These are the addresses listed as nameservers in /etc/resolv.conf.

9 In our case: and c) Subnet mask: The subnet mask for ethernet cards being configured via dhcp. This value will be listed in the NETMASK entry of /etc/sysconfig/network-scripts/ifcfg-eth0. In our case: d) Gateways : The address of the primary gateway for nodes that are being configured via dhcp: This value is defined as GATEWAY in the /etc/sysconfig/network file. In our case: e) Broadcast Address: The broadcast address used by the server. This value will be listed in the BROADCAST entry of /etc/sysconfig/network-scripts/ifcfg-eth0. In our case: f) Refer to your list of node names, IP addresses and MAC addresses that you obtained during the node preparation. Step 2) Create /etc/dhcpd.conf The following dhcpd.conf file is constructed from the information collected from each node during node preparation, and the general network information collected at the beginning of dhcp configuration. ======================= DHCP configuration file. ISC DHCP server v3.0 Suffering the ongoing modifications of Walt Akers Mar 11, 2002 ======================= ======================= Assert that we know what IP addresses are valid for each machine requesting an address. This assures that if a machine asks for an address that is not valid on a particular subnet, then dhcpd will respond with a DHCPNAK, forcing the client to request a new address. ======================= authoritative; ======================= Deny unregistered MAC Addresses ======================= deny unknown-clients; ======================= Deny clients that are using the bootp protocol. ======================= deny bootp; ======================= Turn off Dynamic DNS updates ======================= ddns-update-style ad-hoc; ddns-updates off; ======================= Define the special options required by BpBatch

10 Depending on your PXE Version, either option 135 or option 155 is used to supply arguments to BpBatch. The supplied option is always the location of the boot script. ======================= option option-135 code 135 = string; option option-155 code 155 = string; ======================= Since we are using static IP addresses, we will set the default lease time to inifinity ======================= default-lease-time -1; ======================= Option definitions common to all supported networks... Note: use the domain-name and domain-name-servers you collected during the data collection stage of dhcp configuration. ======================= option domain-name "jlab.org"; option domain-name-servers , ; ======================= This block defines the subnet of the HPC Cluster. HPC Cluster. Note: Each supported subnet must have a subnet definition, and you should use the values that you collected during the data collection stage of dhcp configuration. The routers option is the address of the GATEWAY. ======================= subnet netmask { option broadcast-address ; option subnet-mask ; option routers ; ======================= This group defines the actual nodes in the HPC Cluster that will be dynamically configured. Note: This section will declare the node specific information that was collected during node preparation. ======================= group { =============== This is the name of the boot file that will be retrieved from the TFTP server for all members of this group. Note: This boot file's location is relative to the /tftpboot directory on the TFTP server. =============== filename "bpbatch.p"; =============== This is the name of the tftp server that will be contacted to download the bootfile. Note: In our implementation the dhcp server and the tftp server are the same machine. If you wish to use a different system as the tftp server, this is where you

11 would define it. =============== next-server qcdadm01.jlab.org; =============== Host Specifications: Each host will contain a hardware ethernet address, a corresponding internet address, and option-135 which specifies the bootfile that BPBatch should use. The path to the boot file is relative from the /tftpboot directory on the TFTP server. =============== host qcd2m001 { hardware ethernet 0:30:48:23:45:3d; fixed-address ; option option-135 "boot/ "; host qcd2m002 { hardware ethernet 0:30:48:23:45:07; fixed-address ; option option-135 "boot/ "; host qcd2m003 { hardware ethernet 0:30:48:23:45:f; fixed-address ; option option-135 "boot/ "; host qcd2m004 { hardware ethernet 0:30:48:23:45:3; fixed-address ; option option-135 "boot/ "; Step 3) Create an empty dhcpd lease file a) Execute the following command: touch /var/state/dhcp/dhcpd.leases Step 4) Configure the dhcp daemon to automatically start when the system is booted by adding the following command to /etc/rc.d/rc.local immediately before the tftp daemon is started. Start the dhcpd server /usr/sbin/dhcpd 11. START THE REMOTE BUILD ENVIRONMENT At this point, everything necessary to remotely configure and start a compute node should be in place. Notably, all of the daemons required for this application are started from the /etc/rc.d/rc.local script, so it will be necessary to start them by hand the first time. Since most of these applications will complain if there are errors in the configuration files, this will give you an opportunity to discover any typos or inaccuracies that may have occurred during configuration. To start the remote build environment on the server, perform the following steps: Step 1) Start the dhcp server by typing the following command: /usr/sbin/dhcpd Step 2) Start the tftp daemon by typing the following command (on one line): /usr/sbin/tftpd -c 64 -d /tftpboot -h -i 0 -k 5 -l /var/log/tftp.log -r -s v 2 >& /var/log/tftp.out &

12 Step 3) Start the relinker by typing the following command: tail -f /var/log/tftp.out /usr/sbin/relinker & Note: In some cases you may need to restart the NFS daemon to ensure that the required file systems are exported. If all of these daemons started correctly, then the system should now ready to remote boot and configure compute nodes. 12. USING RPM TO AUTOMATICALLY INSTALL A REMOTE BUILD SERVER The preceding sections provide an exhaustive overview of how Jefferson Lab s Remote Build environment is configured. As you might imagine, much of this configuration information can be gleaned from the running system. This has allowed us to develop a general purpose RPM that automatically configures a Linux host to provide Remote Build services. Using the Jefferson Lab RPM eliminates many common configuration errors by automatically installing all of the required applications and generating system specific scripts. To install the Remote Build environment from the Jefferson Lab RPM, the following steps must be performed. Step 1) Download the RemoteBuild RPM via ftp from Jefferson Lab at the following address: ftp.jlab.org:/pub/hpc/remoteboot i386.rpm Step 2) Collect ethernet addresses and perform the node preparation steps as defined in Section 4 of this document. Step 3) Create an /etc/ethers file that associates the IP address for each node with the hardware ethernet address. The RemoteBuild RPM will use the list of nodes in the /etc/ethers file to generate boot scripts and DHCP entries for each of the compute nodes. Refer to the ethers man page for a description of the /etc/ethers file s syntax. Step 4) Install a complete copy of the RedHat distribution in a mirror directory. A copy of each version of RedHat that you wish to support will need to be installed. For our purposes, we ll assume that you ll be installing RedHat version 7.3 and that the distribution is installed in /mirror/redhat/linux/7.3/en/os/i386. Step 5) Export the /mirror directory globally: a) Add an entry in the /etc/exports file that exports the /mirror directory. It should look like this. /mirror *(ro) b) Restart the NFS daemons to ensure that the file system is exported and that NFS is operating properly. Use the following command. /etc/rc.d/init.d/nfs restart Step 6) Set the environment variables that will be used by the RemoteBoot RPM.

13 Unlike many RPM s that built exclusively from default parameters, the RemoteBoot RPM has a variety of configuration options. Before proceeding with the build, set the following environment variables. a) REDHAT_VERSION : The version of RedHat that the RPM will configured to provide. The default REDHAT_VERSION is 7.3. b) REDHAT_SERVER : The server that will provide the RedHat distribution. If the distribution is stored on a different server, then provide that server s hostname or IP address. By default, the hostname of the current host will be used. c) REDHAT_PATH : The complete path to the RedHat distribution on the specified REDHAT_SERVER. In our example, the REDHAT_PATH would be /mirror/redhat/linux/7.3/en/os/i386. Step 7) Install the RemoteBoot RPM. From the directory where the RemoteBoot RPM is installed, enter the following command: rpm i RemoteBoot i386.rpm This will automatically install all of the daemons that are required to operate the remote build/boot server and will generate a default KickStart configuration file for building each new node. Step 8) Update the /tftpboot/linux-7.3/ks.cfg file. Following the completion of the install process, you ll want to update the KickStart configuration file to install and configure compute nodes in accordance with your local policies. Once the KickStart configuration file has been updated, you ll be ready to begin building nodes from this server.