PBS Professional Integration with HP CMU

Transcription

1 Adam Diaz July Copyright 2010 Altair Engineering, Inc. All rights reserved. PBS, PBS Works, PBS Professional, PBS Analytics, PBS Catalyst, e-biochem, e-compute, and e-render are trademarks of Altair Engineering, Inc. and are protected under U.S. and international law and treaties. All other marks are the property of their respective owners. This paper is for informational purposes only, and may contain errors; the content is provided as is, without express or implied warranties of any kind.

2 Table of Contents I. Abstract... 2 II. Introduction to PBS Professional... 2 III. Introduction to HP CMU... 4 IV. Upgrading PBS Professional... 4 V. Green Provisioning... 7 VI. OS Provisioning... 9 VII. Conclusion VIII. References

3 I. Abstract As a workload management solution, PBS Professional integrates well with a variety of cluster management software. Due to recent customer requests, we have identified a number of areas where scheduling features in PBS Professional can be enabled through the use of HP s cluster management software package called Cluster Management Utility (CMU). This paper will detail the technical information on recent integration efforts between PBS Professional and HP CMU. These efforts take the form of a series of HP CMU enabled PBS Professional configuration options and accompanying scripts. These scripts are intended to be a starting place for administrators to examine and understand these functions as a whole and then edit the various configuration files and scripts to suit their site s needs. Many of the integration points discussed are the result of recent advancements by the PBS Professional team to create easy to modify scripted integration access points that do not require changes to PBS Professional source code. What follows are the details on three such features: 1. Upgrading PBS Professional 2. Green Provisioning 3. OS Provisioning II. Introduction to PBS Professional PBS Professional is the professional version of the Portable Batch System (PBS), a flexible workload management solution, originally developed to manage aerospace computing resources at NASA. PBS Professional has since become the leader in supercomputer workload management and the de facto standard on Linux clusters. Today, growing enterprises often support hundreds of users running thousands of jobs across many different types of machines in diverse geographical locations. In this distributed heterogeneous environment, it can be extremely difficult for administrators to collect detailed, accurate usage data, or to set system-wide resource priorities. As a result, many computing resources are under utilized, while others are over utilized. 2

4 At the same time users are confronted with ever expanding array of Operating Systems and platforms. Each year, scientists, engineers, designers and analysts must waste countless hours learning the nuances of different computing environments, rather than being able to focus on their core priorities. PBS Professional addresses these problems for computing-intensive industries such as science, engineering, finance and entertainment. Now you can use the power of PBS Professional to better control your computing resources. This allows you to unlock the potential in the valuable assets you already have, while at the same time, reducing dependency on system administrators and operators, freeing them to focus on other activities. PBS Professional can help you effectively manage growth by tracking real usage levels across your systems and enhancing utilization of future purchases. PBS Professional consists of two major component types: user-level commands and system daemons/services. A brief description of each is given here to help you understand how the pieces fit together. Commands PBS Professional supplies both command line programs that are POSIX d conforming and a graphical interface. These are used to submit, monitor, modify and delete jobs. These client commands can be installed on any system type supported by PBS Professional and do not require local presence of any of the other components of PBS Professional. There are three command classifications user commands which any authorized user can use, operator commands, and manager (or administrator) commands. Operator and manager commands require special privilege (like root access) to use. Server The Server daemon/service is the central focus of PBS Professional. All commands and other daemons/services communicate with the Server via an Internet Protocol (IP) network. The Server s main function is to provide the basic batch services such as receiving/creating a batch job, modifying the job, and passing the job to the execution node. Normally, there is one Server managing given set of resources. 3

5 Execution Node (MOM) The execution node or MOM is the daemon/service which actually places the job into execution. The process pbs_mom, is informally called MOM as it is a mother of all processes for that job. (MOM is reverse-engineered acronym that stands for Machine Oriented Mini Server). MOM places a job into execution when it receives a copy of the job from a Server. MOM creates a new session for each job and gathers information about the resource usage of that job. MOM also has the responsibility for communicating with all Moms assigned to the job and returning the job s output to the user when directed to do so by the Server. Scheduler The job Scheduler daemon/service, pbs_sched, implements the site s scheduling policy, controlling when each job is run and on which resource. The Scheduler may communicate with the various MOMs to query the state of system resources and with the Server for availability of jobs to execute. III. Introduction to HP CMU HP CMU is an efficient and robust tool for the management of Linux-based nodes in HPC clusters and compute farms. A simple graphic interface enables centralized console management, monitoring of cluster-wide and node-specific metrics, and software installation. HP CMU enables rapid provisioning and management of multiple images, making it ideal for a site with frequent changes to the software configuration across the cluster or on a subset of the cluster. HP CMU is highly flexible, and is being used to deploy a range of software environments, from simple compute farms to highly customized, application-specific configurations. HP CMU is available for HP ProLiant and Integrity servers and HP BladeSystems, with Red Hat Enterprise Linux and Novell SUSE Linux operating systems. IV. Upgrading PBS Professional PBS Professional already has a number of upgrade paths as detailed in the installation manual for the product. These methods however do not make use of any automated technology to install, uninstall or upgrade PBS Professional across a cluster. There are well functioning but basic shell scripts provided for the installation of individual systems 4

6 but the details of how to do this installation across many thousands of nodes are left to the administrator. While the scope of techniques for accomplishing basic installation and removal of PBS Professional is a well worn path, many times a simple upgrade to a working system that does not require reprovisioning all compute nodes is desired. This is the gap that the pbs_upgrade scripts attempt to fill. At some point this type of functionality may become part of the core PBS Professional product. This script is a prototype of what that technology may resemble once completed. Installing PBS Professional in what is called a rolling fashion 1 is currently not recommended or tested. For instance you might upgrade the head node and offline the existing execution nodes that are running jobs. In this case the job running on each execution node would finish but new jobs would not be accepted to these node. Once the currently running job is completed on a particular execution node, the upgrade is then performed for that execution node and then that node is again ready to accept jobs from the system. The technical challenges of this process are due to many factors including the potential scope of changes to all three daemons that may take place between even minor versions as part of our normal development process. Altair is considering how to address this issue in a future version of PBS Professional. This set of scripts does not attempt to address this issue. When upgrading a PBS Professional installation, administrators need to understand the exact process being undertaken in order to avoid job loss. Altair strongly recommends that procedures like the use of this upgrade script should be tested on a non production system prior to attempting its use on a production system. While the original intent was to simply provide an upgrade script, there are technically three scripts provided. The process of testing the upgrade script gave rise to the need to automate not only the full raw installation but also the removal of PBS Professional. The additional two scripts are provided for your reference and use. 1 An overlay upgrade of the head node daemon pbs_server and pbs_sched with mixed versions of the pbs_mom daemon on execution node. 5

7 The upgrade script itself simply needs to be made executable in some directory on the head node. A PBS Professional installation package needs to be placed in the directory with it. Overview of upgrade script actions: Backup current PBS configuration Automatically finds list of existing execution nodes from qmgr. Shuts down PBS Professional Upgrades the head node Installation of execution nodes Setup PBS Professional integration with Platform MPI (the preferred MPI of HP formerly HP-MPI) - commented out by default. Cleanup- commented out by default Caveats for upgrade script use: Script runs as root as required by the installer. Licensing- this script does not install or upgrade Altair License manager. This must be done by hand. Assumes license server is at 7788@headnodename. Assumes MPI is already installed. Valid only for PBS Professional 10 or higher. Valid only for supported versions of PBS Professional and HP CMU (mainly RHEL 5+/SLES 11+ x86_64). Must be run from head node Passwordless ssh/scp (including root) must be working Pdsh/pdcp using passwordless ssh needs to be installed and configured Assumes scratch area /tmp. Fairshare entries not updated. Queue configuration and custom resources should be closely checked by hand. As with most scripts, the usefulness of this script will vary but again the point is that this is the starting point for those who want to automate the installation of PBS Professional across a running system. The customization of things like an alternate MPI setup (or use 6

8 of pbsrun_wrap) could easily be added to this script to make it much more robust. We recommend that administrators do a complete backup of your existing setup prior to trying any form of upgrade. V. Green Provisioning Green Provisioning in PBS Professional is a feature that is really a collection of configuration options and scripts configured in such a way that execution nodes can be powered on and off depending upon the workload of the system. Green Provisioning, at its core, is currently powered by a series of Perl scripts containing a variety of configuration options surrounding the exact behavior of when nodes are powered on and off. Green Provisioning already has documentation 2 describing the various options and scripts used to configure this feature and those details will not be repeated here. Rather what will be shown is how this feature was modified to enable its action under HP CMU. Starting at line 52 of power_nodes.pm the following function was changed from the original to simply pass the node name to the script that performs the power up called cmuwakeonlan: if ($mac_address =~ /[``;&-]/ ) { warn "Invalid characters found in mac address($mac_address), skipping\n"; } else { } #$cmd = WOL_CMD." -i $ip_bcast_address $mac_address"; $cmd = WOL_CMD." $node"; log_debug "\t\tgoing to power up $node: $cmd"; system($cmd); The script cmuwakeonlan actually only performs the boot action through HP CMU and the name of this script was simply borrowed from the original script used to wake up nodes. The original script wakeonlan (included in the package of scripts) functions independently of HP CMU using the broadcast of a magic packet to the specified NIC. 2 Documentation for the Green Provisioning feature is not included in the standard PBS documentation at this time. A separate word doc is provided with the tar ball of Green Provisioning scripts. Green Provisioning is technically a way to configure existing PBS Professional features versus a built in solution to PBS Professional source code. 7

9 This script describes: The 'magic packet' consists of 6 times 0xFF followed by 16 times the hardware address of the NIC. This sequence can be encapsulated in any kind of packet, in this case an UDP packet targeted at the discard port. The HP CMU enabled version is much simpler in that we simply pass the boot command to the HP CMU command line interface via a script placed in /tmp in order to boot the node. The powering down of nodes is simply built into the do_power_down_node subroutine in power_nodes.pm where the Linux command poweroff is passed to the node via ssh. You may find that this also needs to be adjusted depending upon the desired behavior at your site. #!/bin/sh CMU_PATH=/opt/cmu if [[ $# < 1 ]]; then echo "usage: $0 <CMU node name>." exit 1 fi nodename=$1 # temp file to hold CMU clone commands tmpfile=`mktemp /tmp/cmu_boot_xxxxxxxx` echo "boot $1" > $tmpfile echo "exit" >> $tmpfile # cmucli boot method $CMU_PATH/cmucli $tmpfile #clean up rm -f /tmp/$tmpfile This script is sufficient to boot the required nodes provided the number of powered on nodes was insufficient to fulfill the job to be placed. It should be noted that you will need to carefully read the documentation on this feature. It may require some interactive configuration to provide appropriate time for nodes to boot. Please see the following: ttw setting in power management config file pwr_mgt_power_down_delay delay in secs to wait before powering down a quiet node pwr_mgt_power_up_delay delay in secs to wait for node to power up 8

10 Administrators may also find that in testing these values to find appropriate ranges for their system, you may need to clear offline flags in nodes repeatedly. This process can become tedious. There is a Perl script included to assist in this as well as other commands that will assist you in handling groups of nodes by passing a node file including: build-pbs-nodes provides resource installation assistance. clean-pbs-nodes provides resource removal assistance clear-flags-in-nodes removes offline comments and flags after nodes set offline. Again, it will probably be necessary to interactively work with this feature to find appropriate values for the settings mentioned above. We recommend that you only enable this feature on a smaller subset of nodes until you are comfortable with its behavior. VI. OS Provisioning Operating System (OS) Provisioning is a feature new to PBS Professional This is a feature that is enabled by the use of Python Hooks which are scripts run by PBS Professional under certain conditions along the path of job submission. In short, it is possible to now enable a hook that will launch an external provisioning script. Internally, PBS Professional will take care of the logic necessary to find each node that needs to be reprovisioned. The provision launching hook script will be called once for each of those nodes. Depending upon how provisioning is configured (aggressive versus avoid settings) in the scheduler and the requested operating systems passed via the qsub aoe, requested nodes will be repositioned accordingly. Details on the basic use of OS provisioning and its configuration are detailed in the PBS Professional 10.2 Administrators guide 3 and will not be reproduced here. There are two scripts necessary to enable OS Provisioning within PBS Professional enabled by HP CMU commands. The first is a simply python hook script which calls the actual HP CMU command script: 3 See Chapter 9 called Provisioning. This chapter number may change between versions but will likely remain as an entire chapter on provisioning. Please consult the table of contents of the PBS Professional Administrator s Guide. 9

11 import pbs import os e = pbs.event() vnode = e.vnode aoe = e.aoe ret = os.system("/root/prov/cmu_prov.sh " + aoe + " " + vnode); if ret!= 0: e.reject("reboot provisioning failed",ret) else: e.accept(0) Secondly the script cmu_prov.sh which actually runs the provisioning operation: #!/bin/sh CMU_PATH=/opt/cmu # check args # first one should be a valid CMU logical group # the rest should be valid CMU client nodes if [[ $# < 2 ]]; then echo "usage: $0 <CMU logical group> <CMU node to clone> [CMU node to clone]." exit 1 fi group=$1 node=$2 lg=`$cmu_path/bin/cmu_show_logical_groups` found=0 for l in $lg; do done if [ "$l" == "$group" ]; then found=1 break fi if [ "$found"!= "1" ]; then echo "Error, $group is not a valid CMU logical group" exit 1 fi # temp file to hold CMU clone commands tmpfile=`mktemp /tmp/cmu_clone_xxxxxxxx` echo "clone \"$group\" $2" > $tmpfile echo "exit" >> $tmpfile # run cloning $CMU_PATH/cmucli $tmpfile # Make sure node moved active logical groups 10

12 tmpfile2=`mktemp /tmp/cmu_group XXXXXXXX` echo "group $group" > $tmpfile2 echo "exit" >> $tmpfile2 nodeval=`$cmu_path/cmucli $tmpfile2 grep -c $node` if [ "$nodeval"!= "1" ];then echo "Error, node $node was not provisioned successfully." exit 1 else echo "Provisioning of node $node successful." fi #cleanup rm -f $tmpfile $tmpfile2 Prior to using these scripts, the administrator is required to create logical groups and backups via standard HP CMU commands. Please see the HP CMU users guide for more detail. The full details and an additional full example of how to install and setup OS provisioning is also included in the PBS Professional 10.2 Administrators guide in the chapter on OS Provisioning. These two scripts can simply be substituted as necessary following those simple instructions. It should be noted that similar to the Green Provisioning script the OS provisioning also has some timeout values that will need to be tuned on a per system basis to provide optimal functioning. These values include the hook alarm value and the server level setting node_fail_requeue. In most cases administrators will need to set these values to around 5 to10 minutes. The required timing might further be affected by other settings not discussed in this paper such as scheduler_iteration. It should also be noted that these are set in qmgr as a value in seconds. 11

13 VII. Conclusion In conclusion, these three features are wonderful examples of the integration possible between workload management and cluster management systems. The examples shown in this document are but a few of the ways PBS Professional and HP CMU can be integrated. Many additional levels of integration can be accomplished through the use of external scripts, programs, or python hooks within PBS Professional to create custom actions to fulfill your operating requirements. Scheduling based upon a variety of external conditions like temperature monitoring systems can just as easily be integrated to provide a scheduling policy that fits your sites needs. Additionally PBS Professional can pass information back to HP CMU for display in its graphical interface. The ideas shown above are simply quick examples of how simple integration can provide synergistic functionality by using lightly modified yet off the shelf products from both Altair and HP. Please contact Altair at pbssales@altair.com and we will be happy to help you customize your scheduling policy. 12

14 VIII. References HP CMU PBS Professional Altair Engineering SSH PDSH RHEL SLES POSIX1003.2d _desc.html t f E. Big Beaver Rd. Troy, MI USA Copyright 2010 Altair Engineering, Inc. All rights reserved. PBS, PBS Works, PBS Professional, PBS Analytics, PBS Catalyst, e-biochem, e-compute, and e-render are trademarks of Altair Engineering, Inc. and are protected under U.S. and international law and treaties. All other marks are the property of their respective owners. This paper is for informational purposes only, and may contain errors; the content is provided as is, without express or implied warranties of any kind. 13