Using and Administering

Transcription

1 IBM LoadLeeler Version 5 Release 1 Using and Administering SC

2

3 IBM LoadLeeler Version 5 Release 1 Using and Administering SC

4 Note Before using this information and the product it supports, read the information in Notices on page 423. This edition applies to ersion 5, release 1, modification 0 of IBM LoadLeeler (product numbers 5725-G01, 5641-LL1, 5641-LL3, 5765-L50, and 5765-LLP) and to all subsequent releases and modifications until otherwise indicated in new editions. This edition replaces SC Copyright 1986, 1987, 1988, 1989, 1990, 1991 by the Condor Design Team. Copyright IBM Corporation 1986, US Goernment Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

5 Contents Figures ii Tables ix About this information xi Who should use this information xi Conentions and terminology used in this information xi Prerequisite and related information xii How to send your comments xiii Summary of changes x Part 1. Oeriew of LoadLeeler concepts and operation Chapter 1. What is LoadLeeler? LoadLeeler basics LoadLeeler: A network job management and scheduling system Job definition Machine definition How LoadLeeler schedules jobs How LoadLeeler daemons process jobs The master daemon The Schedd daemon The startd daemon The region manager daemon The resource manager daemon The kbdd daemon The negotiator daemon The LoadLeeler job cycle LoadLeeler job states Consumable resources Consumable resources and Workload Manager 23 Oeriew of reserations Fair share scheduling oeriew Chapter 2. Getting a quick start using the default configuration What you need to know before you begin Using the default configuration files LoadLeeler for Linux quick start Quick installation Quick configuration Quick erification Post-installation considerations Starting LoadLeeler Directory considerations Chapter 3. What operating systems are supported by LoadLeeler? LoadLeeler for AIX and LoadLeeler for Linux compatibility Restrictions for LoadLeeler for Linux Features not supported in LoadLeeler for Linux 36 Restrictions for LoadLeeler for AIX and LoadLeeler for Linux mixed clusters Part 2. Configuring and managing the LoadLeeler enironment Chapter 4. Configuring the LoadLeeler enironment The master configuration file Setting the LoadLeeler user Setting the configuration source Oerriding the shared memory key File-based configuration Database configuration option Understanding remotely configured nodes Using the configuration editor Modifying configuration data Defining LoadLeeler administrators Defining a LoadLeeler cluster Defining LoadLeeler machine characteristics.. 59 Defining security mechanisms Defining usage policies for consumable resources 65 Gathering job accounting data Managing job status through control expressions 72 Tracking job processes Querying multiple LoadLeeler clusters Handling switch-table errors Proiding additional job-processing controls through installation exits Chapter 5. Defining LoadLeeler resources to administer Defining machines Planning considerations for defining machines. 90 Machine_group stanza format and keyword summary Machine substanza format and keyword summary Machine stanza format and keyword summary 91 Default alues for machine_group and machine stanzas Examples of machine_group and machine stanzas 92 Dynamic adapter discoery LoadLeeler adapter and node status monitoring.. 94 Defining classes Using limit keywords Allowing users to use a class Class stanza format and keyword summary.. 97 Examples: Class stanzas Defining user substanzas in class stanzas Copyright IBM Corp. 1986, 2012 iii

6 Examples: Substanzas Defining users User stanza format and keyword summary Examples: User stanzas Defining groups Group stanza format and keyword summary 104 Examples: Group stanzas Defining clusters Cluster stanza format and keyword summary 104 Examples: Cluster stanzas Defining regions Region stanza format and keyword summary 106 Examples: Region stanzas Chapter 6. Performing additional administrator tasks Setting up the enironment for parallel jobs Scheduling considerations for parallel jobs Steps for reducing job launch oerhead for parallel jobs Steps for allowing users to submit interactie POE jobs Setting up a class for parallel jobs Striping when some networks fail Setting up a parallel master node Using the BACKFILL scheduler Tips for using the BACKFILL scheduler Example: BACKFILL scheduling Data staging Configuring LoadLeeler to support data staging Using an external scheduler Replacing the default LoadLeeler scheduling algorithm with an external scheduler Customizing the configuration file to define an external scheduler Example: Retrieing specific information Example: Changing scheduler types Preempting and resuming jobs Oeriew of preemption Planning to preempt jobs Steps for configuring a scheduler to preempt jobs Configuring LoadLeeler to support reserations 127 Steps for configuring reserations in a LoadLeeler cluster Steps for integrating LoadLeeler with the Workload Manager LoadLeeler support for checkpointing jobs Checkpoint keyword summary Planning considerations for checkpointing jobs 136 Additional planning considerations for checkpointing MetaCluster HPC jobs on AIX Checkpoint and restart limitations Submitting a MetaCluster HPC checkpoint job to LoadLeeler job_1.cmd - A checkpointable job command file 138 Using the llckpt command to checkpoint a job step Restarting a job step from a checkpoint Making periodic checkpoints i LoadLeeler: Using and Administering Using the ckpt_dir and ckpt_subdir keywords 143 Remoing old checkpoint files Using the ckpt_execute_dir keyword Initiating a checkpoint using the ll_ckpt() API 146 LoadLeeler scheduling affinity support Configuring LoadLeeler to use scheduling affinity LoadLeeler multicluster support Configuring a LoadLeeler multicluster LoadLeeler Blue Gene support Configuring LoadLeeler Blue Gene support 155 Blue Gene reseration support Blue Gene fair share scheduling support Blue Gene heterogeneous memory support Blue Gene preemption support Using fair share scheduling Fair share scheduling keywords Reconfiguring fair share scheduling keywords 161 Example: three groups share a LoadLeeler cluster Example: two thousand students share a LoadLeeler cluster Querying information about fair share scheduling Resetting fair share scheduling Saing historic data Restoring saed historic data Procedure for recoering a job spool Configuring and using island scheduling Energy aware job support S3 state support Part 3. Submitting and managing LoadLeeler jobs Chapter 7. Building and submitting jobs Building a job command file Using multiple steps in a job command file Examples: Job command files Editing job command files Defining resources for a job step Submitting jobs requesting data staging Working with coscheduled job steps Submitting coscheduled job steps Determining priority for coscheduled job steps 178 Supporting preemption of coscheduled job steps 179 Coscheduled job steps and commands and APIs 179 Termination of coscheduled steps Using bulk data transfer Preparing a job for checkpoint/restart Preparing a job for preemption Submitting a job command file Job state monitoring Submitting a job using a submit-only machine 184 Working with parallel jobs Step for controlling whether LoadLeeler copies enironment ariables to all executing nodes.. 185

7 Ensuring that parallel jobs in a cluster run on the correct leels of PE and LoadLeeler software Task-assignment considerations Submitting jobs that use striping Running interactie POE jobs Debugging interfaces between POE and LoadLeeler Running MPICH Running Open MPI Running Intel MPI jobs Running embarassingly parallel jobs Examples: Building parallel job command files 197 Obtaining status of parallel jobs Obtaining allocated host names Building and submitting MPICH2 and serial interactie jobs Working with reserations Types of reserations Understanding the flexible job step Understanding the reseration life cycle Creating new reserations Submitting jobs to run under a reseration Remoing bound jobs from the reseration Querying existing reserations Modifying existing reserations Canceling existing reserations Reserations with floating resources Submitting jobs requesting scheduling affinity Submitting and monitoring jobs in a LoadLeeler multicluster Steps for submitting jobs in a LoadLeeler multicluster enironment Working with energy aware jobs Submitting and monitoring Blue Gene jobs Chapter 8. Managing submitted jobs 223 Querying the status of a job Working with machines Displaying currently aailable resources Setting and changing the priority of a job Example: How does a job's priority affect dispatching order? Placing and releasing a hold on a job Canceling a job Checkpointing a job Chapter 9. Example: Using commands to build, submit, and manage jobs Part 4. LoadLeeler interfaces reference Chapter 10. Configuration keyword reference Configuration keyword syntax Numerical and alphabetical constants Mathematical operators bit support for configuration file keywords and expressions Configuration keyword descriptions User-defined keywords LoadLeeler ariables Variables to use for setting dates Variables to use for setting times Chapter 11. Administration keyword reference Administration file structure and syntax Stanza characteristics Syntax for limit keywords bit support for administration file keywords 297 Administration keyword descriptions Chapter 12. Job command file reference Job command file syntax Serial job command file Parallel job command file Syntax for limit keywords bit support for job command file keywords 334 Job command file keyword descriptions Job command file ariables Run-time enironment ariables Job command file examples Part 5. Appendixes Appendix A. Troubleshooting LoadLeeler Frequently asked questions Why won't LoadLeeler start? Why won't my job run? Why won't my parallel job run? Why won't my checkpointed job restart? Why won't my submit-only job run? Why does a job stay in the Pending (or Starting) state? What happens to running jobs when a machine goes down? Why does llstatus indicate that a machine is down when llq indicates a job is running on the machine? Why won't my job run on a cluster with both AIX and Linux machines? Why won't my jobs run that were directed to an idle pool? What happens if the central manager isn't operating? How do I recoer resources allocated by a Schedd machine? Why can't I find a core file on Linux? Why am I seeing inconsistencies in my llfs output? Why don't I see my job when I issue the llq command? Contents

8 What happens if errors are found in my configuration or administration file? Why is my flexible reseration not actiated? 403 Why was my energy aware job rejected? Other questions Troubleshooting in a multicluster enironment How do I determine if I am in a multicluster enironment? How do I determine how my multicluster enironment is defined and what are the inbound and outbound hosts defined for each cluster? Why is my multicluster enironment not enabled? How do I find log messages from my multicluster-defined installation exits? Why won't my remote job be submitted or moed? Why did the CLUSTER_REMOTE_JOB_FILTER not update the job with all of the statements I defined? How do I find my remote job? Why won't my remote job run? Why does llq -X all show no jobs running when there are jobs running? Troubleshooting adapter aailability Troubleshooting in a Blue Gene enironment Why do all of my Blue Gene jobs fail een though llstatus shows that Blue Gene is present? 409 Why does llstatus show that Blue Gene is absent? Why did my Blue Gene job fail when the job was submitted to a remote cluster? Why does llmkres or llchres return "Insufficient resources to meet the request" for a Blue Gene reseration when resources appear to be aailable? Helpful hints Scaling considerations Hints for running jobs Hints for using machines History files and Schedd Getting help from IBM Appendix B. LoadLeeler port usage 417 Accessibility features for LoadLeeler 421 Accessibility features Keyboard naigation IBM and accessibility Notices Trademarks Glossary Index i LoadLeeler: Using and Administering

9 Figures 1. Example of a LoadLeeler cluster LoadLeeler job steps Multiple roles of machines High-leel job flow Job is submitted to LoadLeeler LoadLeeler authorizes the job LoadLeeler prepares to run the job LoadLeeler starts the job LoadLeeler completes the job How control expressions affect jobs Multicluster Example Job command file with multiple steps Job command file with multiple steps and one executable Job command file with arying input statements Using LoadLeeler ariables in a job command file Job command file used as the executable Striping oer multiple networks Striping oer a single network When the primary central manager is unaailable Multiple central managers Copyright IBM Corp. 1986, 2012 ii

10 iii LoadLeeler: Using and Administering

11 Tables 1. Conentions xii 2. Major topics in LoadLeeler: Using and Administering Topics in the LoadLeeler oeriew LoadLeeler daemons startd determines whether its own state permits a new job to run Job state descriptions and abbreiations Roadmap of tasks for LoadLeeler administrators Roadmap of administrator tasks related to using or modifying the LoadLeeler configuration Roadmap for defining LoadLeeler cluster characteristics Default locations for all of the files and directories Log control statements Roadmap of configuration tasks for securing LoadLeeler operations Roadmap of tasks for gathering job accounting data Collecting account data - modifying configuration keywords Roadmap of administrator tasks accomplished through installation exits Roadmap of tasks for modifying the LoadLeeler administration file Types of limit keywords Enforcing job step limits Setting limits Roadmap of additional administrator tasks Roadmap of BACKFILL scheduler tasks Roadmap of tasks for using an external scheduler Effect of LoadLeeler keywords under an external scheduler Roadmap of tasks for using preemption Preemption methods for which LoadLeeler automatically resumes preempted jobs Preemption methods for which administrator or user interention is required Roadmap of reseration tasks for administrators Roadmap of tasks for checkpointing jobs Deciding where to define the directory for staging executables Multicluster support subtasks and associated instructions Multicluster support related topics Subtasks for configuring a LoadLeeler multicluster Blue Gene subtasks and associated instructions Blue Gene related topics and associated information Blue Gene configuring subtasks and associated instructions Learning about building and submitting jobs Roadmap of user tasks for building and submitting jobs Standard files for the fie job steps Checkpoint configurations Valid combinations of task assignment keywords are listed in each column node and total_tasks Blocking Unlimited blocking Roadmap of tasks for reseration owners and users Reseration states, abbreiations, and usage notes Instructions for submitting a job to run under a reseration Submitting and monitoring jobs in a LoadLeeler multicluster Roadmap of user tasks for managing submitted jobs How LoadLeeler handles job priorities Configuration subtasks Administration file subtasks Notes on 64-bit support for administration file keywords Summary of possible alues set for the en_copy keyword in the administration file Sample user and group settings for the max_reserations keyword Job command file subtasks Notes on 64-bit support for job command file keywords mcm_affinity_options default alues Why your job might not be running Why your job might not be running Troubleshooting running jobs when a machine goes down LoadLeeler default port usage Copyright IBM Corp. 1986, 2012 ix

12 x LoadLeeler: Using and Administering

13 About this information Attention: For LoadLeeler Version 5 Release 1, changes apply to Linux. Disclaimer: The functions or features found herein may not be aailable on all operating systems or platforms and do not indicate the aailability of these functions or features within the IBM product or future ersions of the IBM product. The deelopment, release, and timing of any future features or functionality is at IBM's sole discretion. IBM's plans, directions, and intent are subject to change or withdrawal without notice at IBM's sole discretion. The information mentioned is not a commitment, promise, or legal obligation to delier any material, code or functionality. The information may not be incorporated into any contract and it should not be relied on in making a purchasing decision. Release notes: For the latest release notes, go to the Fix Central website ( support/fixcentral/?productgroup0=ibm/fcpower&productgroup1=ibm/clustersoftware &productgroup2=ibm/power/ibm). For more information about Fix Central, see Locating LoadLeeler software fixes and updates in Fix Central on page xiii. IBM LoadLeeler proides arious ways of scheduling and managing applications for best performance and most efficient use of resources. LoadLeeler manages both serial and parallel jobs oer a cluster of machines or serers, which may be desktop workstations, dedicated serers, or parallel machines. This information describes how to configure and administer this cluster enironment, and to submit and manage jobs that run on machines in the cluster. Who should use this information This book is intended for two separate audiences: Personnel who are responsible for installing, configuring and managing the LoadLeeler cluster enironment. These people are called LoadLeeler administrators. LoadLeeler administratie tasks include: Setting up configuration and administration files Maintaining the LoadLeeler product Setting up the distributed enironment for allocating batch jobs Users who submit and manage serial and parallel jobs to run in the LoadLeeler cluster. Both LoadLeeler administrators and general users should be experienced with the UNIX commands. Administrators also should be familiar with: Cluster system management techniques such as SMIT, as it is used in the AIX enironment Networking and NFS or AFS protocols Conentions and terminology used in this information Throughout the IBM LoadLeeler product information: LoadLeeler for Linux on x86 Architecture includes: IBM System x Intelligent Cluster Copyright IBM Corp. 1986, 2012 xi

14 IBM System serers with Adanced Micro Deices (AMD) Opteron or Intel Extended Memory 64 Technology (EM64T) processors References to Schedd are also referred to as job manager. Table 1. Conentions Conention bold bold underlined constant width italic Usage Bold words or characters represent system elements that you must use literally, such as commands, flags, path names, directories, file names, alues, and selected menu options. Bold underlined keywords are defaults. These take effect if you do not specify a different keyword. Examples and information that the system displays appear in constant-width typeface. Italic words or characters represent ariable alues that you must supply. Italics are also used for information unit titles, for the first use of a glossary term, and for general emphasis in text. <key> Angle brackets (less-than and greater-than) enclose the name of a key on the keyboard. For example, <Enter> refers to the key on your terminal or workstation that is labeled with the word Enter. \ In command examples, a backslash indicates that the command or coding example continues on the next line. For example: mkcondition -r IBM.FileSystem -e "PercentTotUsed > 90" \ -E "PercentTotUsed < 85" -m d "FileSystem space used" {item} [item] <Ctrl-x> item... Braces enclose a list from which you must choose an item in format and syntax descriptions. Brackets enclose optional items in format and syntax descriptions. The notation <Ctrl-x> indicates a control character sequence. For example, <Ctrl-c> means that you hold down the control key while pressing <c>. Ellipses indicate that you can repeat the preceding item one or more times. In syntax statements, ertical lines separate a list of choices. In other words, a ertical line means Or. In the left margin of the document, ertical lines indicate technical changes to the information. Prerequisite and related information The LoadLeeler publications are: AIX Installation Guide, SC Linux Installation Guide, SC Using and Administering, SC Diagnosis and Messages Guide, SC Command and API Reference, SC Resource Manager, SC To access all LoadLeeler documentation, refer to the IBM Cluster Information Center ( which contains the most recent LoadLeeler documentation in PDF and HTML formats. xii LoadLeeler: Using and Administering

15 How to send your comments A LoadLeeler Documentation Updates file also is maintained on this Web site. The LoadLeeler Documentation Updates file contains updates to the LoadLeeler documentation. These updates include documentation corrections and clarifications that were discoered after the LoadLeeler information units were published. Both the current LoadLeeler books and earlier ersions of the library are also aailable in PDF format from the IBM Publications Center ( e-business/linkweb/publications/serlet/pbi.wss). To easily locate a book in the IBM Publications Center, supply the book's publication number. The publication number for each of the LoadLeeler books is listed after the book title in the preceding list. Locating LoadLeeler software fixes and updates in Fix Central To locate LoadLeeler software fixes and updates, do the following: 1. Go to the Fix Central website ( &productgroup2=ibm/power/ibm). 2. In the Product Group pull-down menu, under Software, select Cluster software. 3. In the Select from Cluster software pull-down menu, choose LoadLeeler. 4. In the Installed Version pull-down menu, select the ersion you hae installed, for example, In the Platform pull-down menu, select the platform, for example, All. 6. Click Continue and select an option on the Identify fixes panel. For example, select Browse for fixes and click Continue again. 7. On the Select fixes panel, select fixes to download or, to iew the Restrictions list, click on Readme, next to the appropriate fix and open the Known Limitations section. Your feedback is important in helping us to produce accurate, high-quality information. If you hae any comments about this book or any other LoadLeeler documentation, send your comments by to: mhrcfs@us.ibm.com Include the book title and order number, and, if applicable, the specific location of the information you hae comments on (for example, a page number or a table number). For technical information and to exchange ideas related to high performance computing, go to: HPC Central ( hpccentral/hpc+central) HPC Central Technical Forum ( forum.jspa?forumid=1056) About this information xiii

16 xi LoadLeeler: Using and Administering

17 Summary of changes Version 5 Release 1 The following sections summarize changes to the IBM LoadLeeler product and library. Within each information unit in the library, a ertical line to the left of text and illustrations indicates technical changes or additions made to the preious edition of the information. Attention: For LoadLeeler Version 5 Release 1, changes apply to Linux. Disclaimer: The functions or features found herein may not be aailable on all operating systems or platforms and do not indicate the aailability of these functions or features within the IBM product or future ersions of the IBM product. The deelopment, release, and timing of any future features or functionality is at IBM's sole discretion. IBM's plans, directions, and intent are subject to change or withdrawal without notice at IBM's sole discretion. The information mentioned is not a commitment, promise, or legal obligation to delier any material, code or functionality. The information may not be incorporated into any contract and it should not be relied on in making a purchasing decision. Release notes: For the latest release notes, go to the Fix Central website ( support/fixcentral/?productgroup0=ibm/fcpower&productgroup1=ibm/clustersoftware &productgroup2=ibm/power/ibm). For more information about Fix Central, see Locating LoadLeeler software fixes and updates in Fix Central on page xiii. Changes to LoadLeeler for Linux Version 5 Release 1 product and library include: New information: A new job command file keyword, first_node_tasks, has been added to allow users to request a different task count for the first node. The alue specified for tasks_per_node would apply to all nodes except the first node. Support for LoadLeeler as the scheduler for Blue Gene /Q has been added. - A new Blue Gene command called llbgctl has been added. You can use the llbgctl command to control Blue Gene system resources. - A new command called llbgstatus has been added that returns Blue Gene system status information. For more information, see LoadLeeler: Command and API Reference. LoadLeeler now permits its execute directory to be located in olatile storage, such as RAM disk, where the contents may not exist following the reboot of an execute node. To aoid the potential performance impact of using a shared file system, configure the execute directory in either local disk, or in RAM disk. Support for Workload Manager (WLM) on LoadLeeler for Linux on POWER (RHEL 6.2) and on x86_64 (SLES 11) has been added. A new affinity interface is introduced into LoadLeeler. The new interface is used by IBM Parallel Enironment (PE) Runtime Edition on the x86 platform. Copyright IBM Corp. 1986, 2012 x

18 Island scheduling is introduced in LoadLeeler. New configuration parameters permit administrators to define islands within a LoadLeeler cluster. You can now request that your jobs run within one island or across multiple islands. Support for running Intel Message Passing Interface (MPI) and Open MPI jobs as batch LoadLeeler jobs has been added. The llrun command has been added to allow you to run an interactie command that uses resources allocated by LoadLeeler. LoadLeeler proides support for running embarrassingly parallel jobs without using an MPI runtime process manager. For more information, see the following: - LoadLeeler: Command and API Reference - LoadLeeler: Using and Administering Support for the energy function on the x86_64 idataplex Sandy Bridge processor for SUSE Linux Enterprise Serer (SLES) 11 includes the following: - The ability to run jobs in a lower CPU frequency. To minimize energy consumption with minimal performance degradation, LoadLeeler proides the job energy policy tag to allow you to set the acceptable performance degradation for the job. - Job energy reports can be generated from the accounting data by using the llsummary command. The job energy consumption is calculated after the job finishes. - An administrator can switch idle nodes to standby state to sae energy. This can be done either manually or LoadLeeler can do it automatically. You can wake up the idle nodes that are in standby mode to run the workload as needed. To support the energy function, the following hae been added: - Four new commands: llrchgmstat, llreinit, llrqetag, llrrmetag. See LoadLeeler: Resource Manager for more information about these commands. - Two new subroutines: llr_change_node_powerstate and llr_remoe_energy_tags. See LoadLeeler: Resource Manager for more information about these subroutines. For additional information, see the following: - LoadLeeler: Command and API Reference - LoadLeeler: Using and Administering - LoadLeeler for Linux: Installation Guide Support for checkpoint/restart on Red Hat Enterprise Linux (RHEL) 6.2, which includes the following: - The system administrator or user can now set the checkpoint interal on a per job basis. - Task migration, which proides a method to moe portions of a running parallel job from the current allocated set of nodes to a new set of nodes. To support task migration, the following hae been added: The llmigrate command The ll_migrate_job_step subroutine For more information, see the following: - LoadLeeler: Command and API Reference - LoadLeeler: Resource Manager - LoadLeeler: Using and Administering xi LoadLeeler: Using and Administering

19 Support for multiple endpoints has been added. The new endpoints keyword is supported on a class basis for all jobs submitted in that class, as well as an option on the network keyword. For more information, see the following: - LoadLeeler: Using and Administering - LoadLeeler: Resource Manager Support for running LoadLeeler on Blue Gene/Q has been added on LoadLeeler for Linux on POWER (RHEL 6.2). For more information, see the following: - LoadLeeler: Command and API Reference - LoadLeeler: Using and Administering - LoadLeeler: Resource Manager Changed information: The llrctl command is changed to be consistent with the LoadLeeler scheduler command, llctl. The same keywords supported in the llctl command are now supported in the llrctl command. The drain jobmgr and resume jobmgr keywords are no longer supported in the llrctl command. The following table summarizes the Blue Gene/Q terminology changes: BG/L and BG/P (preious terms) Partitions Base partitions Wires Node cards mpirun, submit, mpiexec BG/Q terms Blocks Midplanes Cables Node boards runjob Deleted information: Support for using irtual IP (VIP) addresses with checkpoint or restart has been remoed, so all references to VIPs or the VIP serer hae been remoed from LoadLeeler publications. The ALLOC_EXCLUSIVE_CPU_PER_JOB configuration file keyword has been deprecated and has been remoed. The following Blue Gene job command file keywords hae been deprecated and remoed: - bg_connection (replaced with bg_connectiity) - bg_partition (replaced with bg_block) Summary of changes xii

20 xiii LoadLeeler: Using and Administering

21 Part 1. Oeriew of LoadLeeler concepts and operation Setting up IBM LoadLeeler inoles defining machines, users, jobs, and how they interact, in such a way that LoadLeeler is able to run jobs quickly and efficiently. Once you hae a basic understanding of the LoadLeeler product and its interfaces, you can find more details in the topics listed in Table 2. Table 2. Major topics in LoadLeeler: Using and Administering To learn about: Read the following: Performing administrator tasks Part 2, Configuring and managing the LoadLeeler enironment, on page 37 Performing general user tasks Part 3, Submitting and managing LoadLeeler jobs, on page 169 Using LoadLeeler interfaces Part 4, LoadLeeler interfaces reference, on page 229 Copyright IBM Corp. 1986,

22 2 LoadLeeler: Using and Administering

23 Chapter 1. What is LoadLeeler? LoadLeeler is a job management system that allows users to run more jobs in less time by matching the jobs' processing needs with the aailable resources. LoadLeeler schedules jobs, and proides functions for building, submitting, and processing jobs quickly and efficiently in a dynamic enironment. Figure 1 shows the different enironments to which LoadLeeler can schedule jobs. Together, these enironments comprise the LoadLeeler cluster. LoadLeeler cluster IBM Power Systems running AIX Submit-only workstations IBM Intelligent Cluster running AIX and Linux As Figure 1 also illustrates, a LoadLeeler cluster can include submit-only machines, which allow users to hae access to a limited number of LoadLeeler features. Throughout all the topics, the terms workstation, machine, node, and operating system instance (OSI) refer to the machines in your cluster. In LoadLeeler, an OSI is treated as a single instance of an operating system image. If you are unfamiliar with the LoadLeeler product, consider reading one or more of the introductory topics listed in Table 3: Table 3. Topics in the LoadLeeler oeriew To learn about: Read the following: Using the default configuration for getting a quick start Specific products and features that are required for or aailable through the LoadLeeler enironment IBM x86 Serer running Linux Figure 1. Example of a LoadLeeler cluster Chapter 2, Getting a quick start using the default configuration, on page 29 Chapter 3, What operating systems are supported by LoadLeeler?, on page 35 Copyright IBM Corp. 1986,

24 LoadLeeler basics LoadLeeler has arious types of interfaces that enable users to create and submit jobs and allow system administrators to configure the system and control running jobs. These interfaces include: Control files that define the elements, characteristics, and policies of LoadLeeler and the jobs it manages. These files are the configuration file, the administration file, and job command file. Optionally, a configuration database can replace the configuration and administration files. The command line interface, which gies you access to basic job and administratie functions. An application programming interface (API), which allows application programs written by users and administrators to interact with the LoadLeeler enironment. The commands and APIs permit different leels of access to administrators and users. User access is typically restricted to submitting and managing indiidual jobs, while administratie access allows setting up system configurations, job scheduling, and accounting. Users create job command files that instruct the system on how to process information. Each job command file consists of keywords followed by the user defined association for that keyword. For example, the keyword executable tells LoadLeeler that you are about to define the name of a program you want to run. Therefore, executable = longjob tells LoadLeeler to run the program called longjob. After creating the job command file, you inoke LoadLeeler commands to monitor and control the job as it moes through the system. LoadLeeler monitors each job as it moes through the system using process control daemons. Howeer, the administrator maintains ultimate control oer all LoadLeeler jobs by defining job classes that control how and when LoadLeeler will run a job. In addition to setting up job classes, the administrator can also control how jobs moe through the system by specifying the type of scheduler. LoadLeeler has seeral different scheduler options that start jobs using specific algorithms to balance job priority with aailable machine resources. When LoadLeeler administrators are configuring clusters and when users are planning jobs, they need to be aware of the machine resources aailable in the cluster. These resources include items like the number of CPUs and the amount of memory aailable for each job. Because resource aailability will ary oer time, LoadLeeler defines them as consumable resources. LoadLeeler: A network job management and scheduling system A network job management and job scheduling system, such as LoadLeeler, is a software program that schedules and manages jobs that you submit to one or more machines under its control. LoadLeeler accepts jobs that users submit and reiews the job requirements. LoadLeeler then examines the machines under its control to determine which machines are best suited to run each job. 4 LoadLeeler: Using and Administering

25 Job definition LoadLeeler schedules your jobs on one or more machines for processing. The definition of a job, in this context, is a set of job steps. For each job step, you can specify a different executable (the executable is the part of the job that gets processed). You can use LoadLeeler to submit jobs which are made up of one or more job steps, where each job step depends upon the completion status of a preious job step. For example, Figure 2 illustrates a stream of job steps: 1. Copy data from tape 2. Check exit status Job job command file Q Q Q Job step 1 Job step 2 Job step 3 exit status = x 1. Process data 2. Check exit status exit status = x exit status = y exit status = y Format and print results End program Figure 2. LoadLeeler job steps Each of these job steps is defined in a single job command file. A job command file specifies the name of the job, as well as the job steps that you want to submit, and can contain other LoadLeeler statements. LoadLeeler tries to execute each of your job steps on a machine that has enough resources to support executing and checkpointing each step. If your job command file has multiple job steps, the job steps will not necessarily run on the same machine, unless you explicitly request that they do. You can submit batch jobs to LoadLeeler for scheduling. Batch jobs run in the background and generally do not require any input from the user. Batch jobs can either be serial or parallel. A serial job runs on a single machine. A parallel job is a program designed to execute as a number of indiidual, but related, processes on one or more of your system's nodes. When executed, these related processes can communicate with each other (through message passing or shared memory) to exchange data or synchronize their execution. For parallel jobs, LoadLeeler interacts with Parallel Operating Enironment (POE) to allocate nodes, assign tasks to nodes, and launch tasks. Machine definition For LoadLeeler to schedule a job on a machine, the machine must be a alid member of the LoadLeeler cluster. A cluster is the combination of all of the different types of machines that use LoadLeeler. Chapter 1. What is LoadLeeler? 5

26 To make a machine a member of the LoadLeeler cluster, the administrator has to install the LoadLeeler software onto the machine and identify the central manager (described in Roles of machines ). Once a machine becomes a alid member of the cluster, LoadLeeler can schedule jobs to it. Roles of machines Each machine in the LoadLeeler cluster performs one or more roles in scheduling jobs. Roles performed in scheduling jobs by each machine in the LoadLeeler cluster are as follows: Job Manager Machine: When a job is submitted, it gets placed in a queue managed by a job manager machine. This machine contacts another machine that seres as the central manager for the entire LoadLeeler cluster. This job manager machine asks the central manager to find a machine that can run the job, and also keeps persistent information about the job. Some job manager machines are known as public job manager machines, meaning that any LoadLeeler user can access them. These machines schedule jobs submitted from submit-only machines. Central Manager Machine: The role of the central manager is to examine the job's requirements and find one or more machines in the LoadLeeler cluster that will run the job. Once it finds the machine(s), it notifies the scheduling machine. Executing Machine: The machine that runs the job is known as the executing machine. Resource Manager Machine: The role of the resource manager is to collect status from all executing and job manager machines in the cluster. Region Manager Machine: The role of the region manager is to monitor node and adapter status by receiing heartbeats from the executing machines it manages. Submitting Machine: This type of machine is known as a submit-only machine. It participates in the LoadLeeler cluster on a limited basis. Although the name implies that users of these machines can only submit jobs, they can also query and cancel jobs. The submit-only machine feature allows workstations that are not part of the LoadLeeler cluster to submit jobs to the cluster. Keep in mind that one machine can assume multiple roles, as shown in Figure 3 on page 7. 6 LoadLeeler: Using and Administering

27 LoadLeeler cluster Central manager Resource manager Job manager Region manager Submit-only machines Executing machine Job manager Executing machine Figure 3. Multiple roles of machines Machine aailability There may be times when some of the machines in the LoadLeeler cluster are not aailable to process jobs. For instance, when the owners of the machines hae decided to make them unaailable. This ability of LoadLeeler to allow users to restrict the use of their machines proides flexibility and control oer the resources. Machine owners can make their personal workstations aailable to other LoadLeeler users in seeral ways. For example, you can specify that: The machine will always be aailable The machine will be aailable only between certain hours The machine will be aailable when the keyboard and mouse are not being used interactiely. Owners can also specify that their personal workstations neer be made aailable to other LoadLeeler users. How LoadLeeler schedules jobs LoadLeeler consists of the following two components: the resource manager, which is responsible for managing all machine and job resources, and the scheduler, which is responsible for scheduling jobs on the resources proided by the resource manager. When a user submits a job, LoadLeeler examines the job command file to determine what resources the job will need. LoadLeeler determines which machine, or group of machines, is best suited to proide these resources, then LoadLeeler dispatches the job to the appropriate machines. To aid this process, LoadLeeler uses queues. A job queue is a list of jobs that are waiting to be processed. When a user submits a job to LoadLeeler, the job is entered into an internal database, which resides on one of the machines in the LoadLeeler cluster, until it is ready to be dispatched to run on another machine. Chapter 1. What is LoadLeeler? 7

28 Once LoadLeeler examines a job to determine its required resources, the job is dispatched to a machine to be processed. A job can be dispatched to either one machine, or in the case of parallel jobs, to multiple machines. Once the job reaches the executing machine, the job runs. Jobs do not necessarily get dispatched to machines in the cluster on a first-come, first-sere basis. Instead, LoadLeeler examines the requirements and characteristics of the job and the aailability of machines, and then determines the best time for the job to be dispatched. LoadLeeler also uses job classes to schedule jobs to run on machines. A job class is a classification to which a job can belong. For example, short running jobs may belong to a job class called short_jobs. Similarly, jobs that are only allowed to run on the weekends may belong to a class called weekend. The system administrator can define these job classes and select the users that are authorized to submit jobs of these classes. You can specify which types of jobs will run on a machine by specifying the types of job classes the machine will support. LoadLeeler also examines a job's priority to determine when to schedule the job on a machine. A priority of a job is used to determine its position among a list of all jobs waiting to be dispatched. The LoadLeeler job cycle on page 16 describes job flow in the LoadLeeler enironment in more detail. How LoadLeeler daemons process jobs LoadLeeler has its own set of daemons that control the processes moing jobs through the LoadLeeler cluster. The LoadLeeler daemons are programs that run continuously and control the processes that moe jobs through the LoadLeeler cluster. A master daemon (LoadL_master) runs on all machines in the LoadLeeler cluster and manages other daemons. Machine and adapter configuration and status changes are now detected by the region manager and the startd daemons. Table 4 summarizes these daemons, which are described in further detail in topics immediately following the table. Table 4. LoadLeeler daemons Daemon Description Part of Resource Manager component LoadL_master Referred to as the master Yes daemon. Runs on all machines in the LoadLeeler cluster and manages other daemons. LoadL_schedd Referred to as the Schedd daemon or the job manager. Receies jobs from the llsubmit command and manages them on machines selected by the negotiator daemon (as defined by the administrator). Yes Part of Scheduler component Yes No 8 LoadLeeler: Using and Administering

29 Table 4. LoadLeeler daemons (continued) Daemon Description Part of Resource Manager component LoadL_startd Referred to as the startd Yes daemon. The startd daemon performs the following functions: Monitors job and machine resources on local machines Generates local adapter configuration Forwards information to the negotiator, resource manager, and region manager daemons Sends heartbeats to the region manager daemon Part of Scheduler component No The startd daemon spawns the starter process (LoadL_starter) which manages running jobs on the executing machine. LoadL_region_mgr Referred to as the region manager daemon. Detects and monitors machine and adapter status from the startd machines it manages and sends this information to the LoadL_negotiator and LoadL_resource_mgr daemons. Yes No LoadL_resource_mgr Referred to as the resource manager daemon. Collects all machine updates from the executing machine. Generates eents to the scheduler for changes in the machine status. Maintains a list of all jobs managed by the resource manager. Responds to query requests for machine, job, and cluster information. Yes No LoadL_kbdd Referred to as the keyboard daemon. Monitors keyboard and mouse actiity. Yes No LoadL_negotiator Referred to as the negotiator daemon or central manager. Monitors the status of each job and machine in the cluster. Responds to queries from llstatus and llq commands. Runs on the central manager machine. No Yes The master daemon The master daemon runs on eery machine in the LoadLeeler cluster, except the submit-only machines. The real and effectie user ID of this daemon must be root. Chapter 1. What is LoadLeeler? 9