Tivoli Workload Scheduler LoadLeveler. Installation GI

Transcription

1 Tivoli Workload Scheduler LoadLeveler Installation Guide Version 3 Release 5 GI

2

3 Tivoli Workload Scheduler LoadLeveler Installation Guide Version 3 Release 5 GI

4 Note! Before using this information and the product it supports, be sure to read the information in Notices on page 45. Sixth Edition (November 2008) This edition applies to version 3, release 5, modification 0 of IBM Tivoli Workload Scheduler LoadLeveler (product numbers 5765-E69 and 5724-I23) and to all subsequent releases and modifications until otherwise indicated in new editions. This edition replaces GI Significant changes or additions to the text and illustrations are indicated by a vertical line () to the left of the change. IBM welcomes your comments. A form for readers comments may be provided at the back of this publication, or you can send your comments to the address: International Business Machines Corporation Department 58HA, Mail Station P South Road Poughkeepsie, NY United States of America FAX (United States & Canada): FAX (Other Countries): Your International Access Code IBMLink (United States customers only): IBMUSM10(MHVRCFS) Internet mhvrcfs@us.ibm.com If you want a reply, be sure to include your name, address, and telephone or FAX number. Make sure to include the following in your comment or note: v Title and order number of this publication v Page number or topic related to your comment When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. Copyright International Business Machines Corporation 1986, US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

5 Contents About this information v Who should read this information v Conventions and terminology used in this information v Prerequisite and related information......vi IBM System Blue Gene Solution documentation vi How to send your comments vii Summary of changes ix Chapter 1. Introduction Chapter 2. Pre-installation planning... 3 Installation overview About the TWS LoadLeveler filesets and RPMs..3 Configuration planning General installation considerations Migration considerations TWS LoadLeveler migration considerations...7 Hardware, software, and disk requirements....9 Hardware requirements Software requirements Disk space requirements Configuration example Chapter 3. Installing TWS LoadLeveler for AIX Decide on directories for TWS LoadLeveler for AIX 15 Log in as root on TWS LoadLeveler for AIX...15 Create the TWS LoadLeveler for AIX loadl group name Create the TWS LoadLeveler for AIX loadl user ID 16 Change the TWS LoadLeveler for AIX user ID..17 Install the TWS LoadLeveler for AIX image Run the TWS LoadLeveler for AIX initialization script llinit Make the TWS LoadLeveler for AIX graphical user interface resource file available (optional) Update the TWS LoadLeveler for AIX PATH environment variable Make the TWS LoadLeveler for AIX man pages available (optional) Repeat the appropriate steps for each TWS LoadLeveler for AIX machine Configure your TWS LoadLeveler for AIX system 22 Chapter 4. Installing TWS LoadLeveler for Linux Decide on directories for TWS LoadLeveler for Linux Log in as root on TWS LoadLeveler for Linux...25 Create the TWS LoadLeveler for Linux loadl group name Create the TWS LoadLeveler for Linux loadl user ID 25 Change the TWS LoadLeveler for Linux user ID 26 Install the TWS LoadLeveler for Linux RPMs...26 Run the TWS LoadLeveler for Linux initialization script llinit Make the TWS LoadLeveler for Linux graphical user interface resource file available (optional)...30 Update the TWS LoadLeveler for Linux PATH environment variable Make the TWS LoadLeveler for Linux man pages available (optional) Repeat the appropriate steps for each TWS LoadLeveler for Linux machine Configure your TWS LoadLeveler for Linux system 32 Chapter 5. Installing submit-only TWS LoadLeveler for AIX Submit-only TWS LoadLeveler for AIX planning considerations TWS LoadLeveler for AIX disk space requirements Set up the submit-only TWS LoadLeveler for AIX user ID and group Install the submit-only TWS LoadLeveler for AIX image Update your submit-only TWS LoadLeveler for AIX configuration Update the submit-only TWS LoadLeveler for AIX cluster configuration Install submit-only TWS LoadLeveler for AIX on other machines in the pool Chapter 6. Installing submit-only TWS LoadLeveler for Linux Submit-only TWS LoadLeveler for Linux planning considerations Submit-only TWS LoadLeveler for Linux disk space requirements Set up the submit-only TWS LoadLeveler for Linux user ID and group Install the submit-only TWS LoadLeveler for Linux RPMs Update your submit-only TWS LoadLeveler for Linux configuration Update the submit-only TWS LoadLeveler for Linux cluster configuration Install submit-only TWS LoadLeveler for Linux on other machines in the pool Accessibility features for TWS LoadLeveler Accessibility features Keyboard navigation IBM and accessibility Copyright IBM Corp. 1986, 2008 iii

6 Notices Trademarks Index Glossary iv TWS LoadLeveler: Installation Guide

7 About this information This information will help you install, configure, and migrate IBM Tivoli Workload Scheduler (TWS) LoadLeveler. It includes concepts and instructions pertaining to: v Installing and configuring TWS LoadLeveler for AIX and Linux v Installing and configuring submit-only TWS LoadLeveler for AIX and Linux Who should read this information This information is intended for system administrators responsible for installing, configuring, and maintaining the TWS LoadLeveler system. It assumes the administrators have a working knowledge of the AIX and Linux operating systems. Conventions and terminology used in this information Throughout the TWS LoadLeveler product information: v TWS LoadLeveler for Linux Multiplatform includes: IBM System servers with Advanced Micro Devices (AMD) Opteron or Intel Extended Memory 64 Technology (EM64T) processors IBM System x servers IBM BladeCenter Intel processor-based servers IBM Cluster 1350 Note: IBM Tivoli Workload Scheduler LoadLeveler is supported when running Linux on non-ibm Intel-based and AMD hardware servers. Supported hardware includes: Servers with Intel 32-bit and Intel EM64T Servers with AMD 64-bit technology v Note that in this information: LoadLeveler is also referred to as Tivoli Workload Scheduler LoadLeveler and TWS LoadLeveler. Switch_Network_Interface_For_HPS is also referred to as HPS or High Performance Switch. Table 1 describes the typographic conventions used in this information. Table 1. Summary of typographic conventions Typographic Usage Bold v Bold words or characters represent system elements that you must use literally, such as commands, flags, and path names. v Bold words also indicate the first use of a term included in the glossary. Italic v Italic words or characters represent variable values that you must supply. Constant width v Italics are also used for book titles and for general emphasis in text. Examples and information that the system displays appear in constant width typeface. [ ] Brackets enclose optional items in format and syntax descriptions. Copyright IBM Corp. 1986, 2008 v

8 Table 1. Summary of typographic conventions (continued) Typographic Usage { } Braces enclose a list from which you must choose an item in format and syntax descriptions. A vertical bar separates items in a list of choices. (In other words, it means or. ) < > Angle brackets (less-than and greater-than) enclose the name of a key on the keyboard. For example, <Enter> refers to the key on your terminal or workstation that is labeled with the word Enter.... An ellipsis indicates that you can repeat the preceding item one or more times. <Ctrl-x> The notation <Ctrl-x> indicates a control character sequence. For example, <Ctrl-c> means that you hold down the control key while pressing <c>. \ The continuation character is used in coding examples in this information for formatting purposes. Prerequisite and related information The Tivoli Workload Scheduler LoadLeveler publications are: v Installation Guide, GI v Using and Administering, SA v Diagnosis and Messages Guide, GA To access all TWS LoadLeveler documentation, refer to the IBM Cluster Information Center, which contains the most recent TWS LoadLeveler documentation in PDF and HTML formats. This Web site is located at: A TWS LoadLeveler Documentation Updates file also is maintained on this Web site. The TWS LoadLeveler Documentation Updates file contains updates to the TWS LoadLeveler documentation. These updates include documentation corrections and clarifications that were discovered after the TWS LoadLeveler books were published. Both the current TWS LoadLeveler books and earlier versions of the library are also available in PDF format from the IBM Publications Center Web site located at: To easily locate a book in the IBM Publications Center, supply the book s publication number. The publication number for each of the TWS LoadLeveler books is listed after the book title in the preceding list. IBM System Blue Gene Solution documentation Table 2 on page vii lists the IBM System Blue Gene Solution publications that are available from the IBM Redbooks Web site at the following URLs: vi TWS LoadLeveler: Installation Guide

9 Table 2. IBM System Blue Gene Solution documentation Blue Gene System Publication Name URL Blue Gene /P Blue Gene /L IBM System Blue Gene Solution: Blue Gene/P System Administration IBM System Blue Gene Solution: Blue Gene/P Safety Considerations IBM System Blue Gene Solution: Blue Gene/P Application Development sg html redp4257.html sg html Evolution of the IBM System Blue Gene Solution redp4247.html IBM System Blue Gene Solution: System Administration sg html Blue Gene/L: Hardware Overview and Planning redp4257.html IBM System Blue Gene Solution: Application Development sg html Unfolding the IBM eserver Blue Gene Solution redp4247.html How to send your comments Your feedback is important in helping us to produce accurate, high-quality information. If you have any comments about this book or any other TWS LoadLeveler documentation: v Send your comments by to: mhvrcfs@us.ibm.com Include the book title and order number, and, if applicable, the specific location of the information you have comments on (for example, a page number or a table number). v Fill out one of the forms at the back of this book and return it by mail, by fax, or by giving it to an IBM representative. To contact the IBM cluster development organization, send your comments by to: cluster@us.ibm.com About this information vii

10 viii TWS LoadLeveler: Installation Guide

11 Summary of changes The following sections summarize changes to the IBM Tivoli Workload Scheduler (TWS) LoadLeveler product and TWS LoadLeveler library for each new release or major service update for a given product version. Within each information unit in the library, a vertical line to the left of text and illustrations indicates technical changes or additions made to the previous edition of the information. Changes to TWS LoadLeveler for this release or update include: v New information: Recurring reservation support: - The TWS LoadLeveler commands and APIs have been enhanced to support recurring reservation. - Accounting records have been enhanced to have recurring reservation entries. - The new recurring job command file keyword will allow a user to specify that the job can run in every occurrence of the recurring reservation to which it is bound. Data staging support: - Jobs can request data files from a remote storage location before the job executes and back to remote storage after it finishes execution. - Schedules data staging at submit time or just in time for the application execution. Multicluster scale-across scheduling support: - Allows a large job to span resources across more than one cluster v Scale-across scheduling is a way to schedule jobs in the multicluster environment to span resources across more than one cluster. This feature allows large jobs that request more resources than any single cluster can provide to combine the resources from more than one cluster and run large jobs on the combined resources, effectively spanning resources across more than one cluster. v Allows utilization of fragmented resources from more than one cluster Fragmented resources occur when the resources available on a single cluster cannot satisfy any single job on that cluster. This feature allows any size job to take advantage of these resources by combining them from multiple clusters. Enhanced WLM support: - Integrates TWS LoadLeveler with AIX Workload Manager (WLM) virtual memory and the large page resource limit support. - Enforces virtual memory and the large page limit usage of a job. - Reports statistics for virtual memory and the large page limit usage. - Dynamically changes virtual memory and the large page limit usage of a job. Enhanced adapter striping (sn_all) support: - Submits jobs to nodes that have one or more networks in the failed (NOTREADY) state provided that all of the nodes assigned to the job have more than half of the networks in the READY state. Copyright IBM Corp. 1986, 2008 ix

12 - A new striping_with_minimum_networks configuration keyword has been added to the class stanza to support striping with failed networks. Enhanced affinity support: - Task affinity support has been enhanced on nodes that are booted in single threaded (ST) mode and on nodes that do not support simultaneous multithreading (SMT). NetworkID64 for Mellanox adapters on Linux systems with InfiniBand support: - Generates unique NetworkID64 IDs for adapter ports that are connected to the same switch and have the same IP subnet address. This ensures that ports that are connected to the same switch, but are configured with different IP subnet addresses, will get different NeworkID64 values. v Changed information: This is the last release that will provide the following functions: - The Motif-based graphical user interface xloadl. The function available in xloadl has been frozen since TWS LoadLeveler and there are no plans to update this GUI with any new function added to TWS LoadLeveler after that level. - The IBM BladeCenter JS21 with a BladeCenter H chassis interconnected with the InfiniBand Host Channel Adapters connected to a Cisco InfiniBand SDR switch. - The IBM Power System 575 (Model ) and IBM Power System 550 (Model A) interconnected with the InfiniBand Host Channel Adapter and Cisco switch. - The High Performance Switch. If you have a mixed TWS LoadLeveler cluster and need to run your job on a specific operating system or architecture, you must define the requirements keyword statement in your job command file specifying the desired Arch or OpSys. For example: Requirements: (Arch == "RS6000") && (OpSys == "AIX53") v Deleted information: The following function is no longer supported and the information has been removed: The scheduling of parallel jobs with the default scheduler (SCHEDULER_TYPE=LL_DEFAULT) The min_processors and max_processors keywords The RSET_CONSUMABLE_CPUS option for the rset_support configuration keyword and the rset job command file keyword The API functions: - ll_get_nodes - ll_free_nodes - ll_get_jobs - ll_free_jobs - ll_start_job Red Hat Enterprise Linux 3 The llctl purgeschedd function has been replaced by the llmovespool function. The lldbconvert function is no longer needed for migration and the lldbconvert command is not included in TWS LoadLeveler 3.5. x TWS LoadLeveler: Installation Guide

13 Chapter 1. Introduction Installation descriptions encompass a range of product and processor options. This topic describes how to install: v IBM Tivoli Workload Scheduler (TWS) LoadLeveler for AIX (AIX 6.1 and AIX 5.3), Version 3 Release 5 Modification 0, product number 5765-E69 or 5724-I23 v TWS LoadLeveler for Linux Version 3 Release 5.0, product number 5724-I23, running on the following: Red Hat Enterprise Linux (RHEL) 4 and RHEL 5 SUSE Linux Enterprise Server (SLES) 9 and SLES 10 Note: IBM Tivoli Workload Scheduler LoadLeveler is supported when running Linux on non-ibm Intel-based and AMD hardware servers. Supported hardware includes: Servers with Intel 32-bit and Intel EM64T Servers with AMD 64-bit technology References to TWS LoadLeveler Version 3 Release 5 apply to the latest version of TWS LoadLeveler, which is x, where x represents the latest service update. At the time this information unit was published, the current service update was To find out which release of TWS LoadLeveler you currently are running, issue the llctl version command. You can obtain AIX and Linux service updates by accessing the IBM Tivoli Workload Scheduler LoadLeveler Support and downloads for AIX and Linux Web site at: Copyright IBM Corp. 1986,

14 2 TWS LoadLeveler: Installation Guide

15 Chapter 2. Pre-installation planning Before installing TWS LoadLeveler, there are planning activities that you must first consider. This topic explains the things you should be aware of before you install TWS LoadLeveler. Installation overview Before installing TWS LoadLeveler, take these steps to make the necessary considerations and decisions. Complete the following steps to install TWS LoadLeveler: 1. Ensure prerequisites are installed. (See Software requirements on page 10.) 2. Make configuration decisions, including: v Decide which machine will act as the central manager, and decide which machines will be backup central managers v Decide on the locations of the home and local directories for the TWS LoadLeveler user ID v Decide which directories will be shared directories, and which will not be shared. 3. Set up the TWS LoadLeveler user ID and group ID for all of the machines in the TWS LoadLeveler cluster. 4. Install the TWS LoadLeveler for AIX filesets or the TWS LoadLeveler for Linux RPM packages and the mandatory PTF. 5. Run the initialization script. 6. Update the configuration files. 7. Repeat the installation process for each workstation in the TWS LoadLeveler cluster. About the TWS LoadLeveler filesets and RPMs Filesets comprise the TWS LoadLeveler for AIX product, while RPMs comprise the TWS LoadLeveler for Linux product. Table 3 shows the filesets that comprise the TWS LoadLeveler for AIX product. It also shows their associated AIX directories after installation. Note that lang refers to the abbreviated name for a language locale, such as en_us. Table 3. TWS LoadLeveler for AIX filesets and their associated directories Fileset Description Associated directory LoadL.full TWS LoadLeveler /usr/lpp/loadl/full LoadL.so Submit-only TWS LoadLeveler /usr/lpp/loadl/so LoadL.msg.lang Messages and man pages for the supported languages Various subdirectories under /usr LoadL.tguides Configuration tasks /usr/lpp/loadl/codebase LoadL.loc.license Software License Agreement /usr/swlag/lang Copyright IBM Corp. 1986,

16 The TWS LoadLeveler for AIX product is available through the TWS LoadLeveler installation image on CD. At times, the TWS LoadLeveler installation image is also given as an AIX file. To view the filesets that are available to you, enter the following AIX command: installp -l -d device where device is: v /cdrom for a CD v The path name of the installation image for a file Table 4 shows the RPMs that comprise the TWS LoadLeveler for Linux product. It also shows their associated Linux directories after installation. Table 4. TWS LoadLeveler for Linux RPMs and their associated directories RPM Description Associated directory LoadL-full-license-Linux_identifier.rpm TWS LoadLeveler license acceptance /opt/ibmll/loadl/lap /opt/ibmll/loadl/sbin LoadL-full-Linux_identifier.rpm TWS LoadLeveler /opt/ibmll/loadl/full /opt/ibmll/loadl/readmes LoadL-so-license-Linux_identifier.rpm Submit-only TWS LoadLeveler license acceptance /opt/ibmll/loadl/lap /opt/ibmll/loadl/sbin LoadL-so-Linux_identifier.rpm Submit-only TWS LoadLeveler /opt/ibmll/loadl/so /opt/ibmll/loadl/readmes Table 5 shows the supported Linux identifiers: Table 5. Linux platforms and associated identifiers Platform Identifier RHEL 4 on Intel IA bit platforms RH4-X x-0.i386 RHEL 5 on Intel IA bit platforms RH5-X x-0.i386 SLES 9 on Intel IA bit platforms SLES9-X x-0.i386 SLES 10 on Intel IA bit platforms SLES10-X x-0.i386 RHEL 4 on AMD Opteron or Intel EM64T 64-bit platforms RHEL 5 on AMD Opteron or Intel EM64T 64-bit platforms SLES 9 on AMD Opteron or Intel EM64T 64-bit platforms SLES 10 on AMD Opteron or Intel EM64T 64-bit platforms RH4-X86_ x-0.x86_64 RH5-X86_ x-0.x86_64 SLES9-X86_ x-0.x86_64 SLES10-X86_ x-0.x86_64 RHEL 4 on IBM Power systems RHEL4-PPC x-0.ppc64 RHEL 5 on IBM Power systems RHEL5-PPC x-0.ppc64 SLES 9 on IBM Power systems SLES9-PPC x-0.ppc64 SLES 10 on IBM Power systems SLES10-PPC x-0.ppc64 Note: x represents the latest service update version Note to Blue Gene/L users An RPM may apply to your IBM System Blue Gene Solution. 4 TWS LoadLeveler: Installation Guide

17 The following RPM is for the IBM System Blue Gene Solution and applies to SLES 9 on IBM Power systems only: v LoadL-full-lib-SLES9-PPC x-0.ppc.rpm Note to Parallel Operating Environment (POE) users Parallel Operating Environment (POE) users may require a corresponding optional RPM. If you plan to use POE with TWS LoadLeveler for Linux on a 64-bit platform such as RHEL 4 on Opteron, RHEL 4 on IBM Power systems, SLES 9 or SLES 10 on Opteron, or SLES 9 or SLES 10 on IBM Power systems, you must install the corresponding optional RPM. For TWS LoadLeveler, version , the optional RPMs would be: v LoadL-full-lib-RH4-X i386.rpm v LoadL-full-lib-RHEL4-PPC ppc.rpm v LoadL-full-lib-SLES9-X i386.rpm v LoadL-full-lib-SLES9-PPC ppc.rpm v LoadL-full-lib-SLES10-X i386.rpm v LoadL-full-lib-SLES10-PPC ppc.rpm These RPMs contain the 32-bit TWS LoadLeveler libraries for the appropriate 64-bit platforms and are required for the proper operation of POE with TWS LoadLeveler. Installation of these RPMs is not needed if you do not plan to use POE with TWS LoadLeveler: v A 32-bit library RPM can only be installed if the corresponding 64-bit TWS LoadLeveler full product RPM is already installed. v The rpm -ivh command is used to install the library RPM. v On SLES 9 IBM Power systems, if you have already installed LoadL-full-lib-SLES9-PPC x-0.ppc.rpm for Blue Gene/L use, there is no need to reinstall it to enable the POE and TWS LoadLeveler interface. Note to SLES 9 SP3 users The installation of the glibc-update for IBM Power systems (SLES 9 SP3) is mandatory. You must upgrade glibc to , or later compatible release. The glibc update is available at: Configuration planning An integral part of the installation of TWS LoadLeveler is to configure your nodes. TWS LoadLeveler: Using and Administering has full details regarding configuration. Table 6 shows TWS LoadLeveler configuration information. Table 6. TWS LoadLeveler configuration information and default values Configuration information Default value TWS LoadLeveler user ID and group ID loadl is the TWS LoadLeveler user ID and the TWS LoadLeveler group ID The configuration directory The home directory of loadl is the configuration directory The global configuration file LoadL_config is the name of the configuration file Chapter 2. Pre-installation planning 5

18 The keywords LOCAL_CONFIG and ADMIN_FILE in the global configuration file define the full path names of the local configuration and the administration files. The global configuration file specifies system attributes common to all nodes. The administration file specifies attributes specific to individual nodes. The information given in the local configuration file is specific to an individual node. It overrides the corresponding information given in the global configuration file. TWS LoadLeveler assumes a common name space. This means that you need the same user ID and group ID on all machines in a TWS LoadLeveler cluster. The term cluster refers to all machines mentioned in the TWS LoadLeveler configuration file. See TWS LoadLeveler: Using and Administering for planning considerations for defining machines. You can define your own TWS LoadLeveler user ID and group ID. Note, however, that if you use an ID other than loadl, you have to substitute this new ID into all appropriate installation instructions. Also, you must create a file called /etc/loadl.cfg and place it in the local directory /etc of each of the nodes in your TWS LoadLeveler cluster. In the LoadL.cfg file, you specify the following three keywords: LoadLUserid Specifies the TWS LoadLeveler user ID LoadLGroupid Specifies the TWS LoadLeveler group ID LoadLConfig Specifies the full path name of the global configuration file For TWS LoadLeveler on AIX, see Change the TWS LoadLeveler for AIX user ID on page 17 for additional information. For TWS LoadLeveler on Linux, see Change the TWS LoadLeveler for Linux user ID on page 26. General installation considerations When planning your TWS LoadLeveler installation, there are specific activities that you must first consider. v Many of the steps for installing TWS LoadLeveler must be performed as root. Therefore, you need superuser authority on each machine in the cluster. v AIX and Linux commands are case sensitive. Therefore, you must enter commands, directory names, and so on exactly as they appear. v Many of the installation steps contain examples that assume you are using standard TWS LoadLeveler directories and names. If you do not use the standard names, you must modify the examples accordingly. This is especially true for the TWS LoadLeveler local and home directories. v The directory containing the configuration files must be readable by all users who will submit jobs. v To use the llctl start command to start remote machines in a cluster, all TWS LoadLeveler administrator user IDs must have rsh privileges on all machines in the cluster. Check the appropriate AIX or Linux documentation for more information on the files used by rsh, such as $HOME/.rhosts and /etc/hosts.equiv. If you use the LL_RSH_COMMAND keyword to specify using another executable instead of rsh, ensure all appropriate files and necessary permissions have been set up. 6 TWS LoadLeveler: Installation Guide

19 v The LoadL_master binary is installed as a setuid program with the owner set to root. If the release directory will be shared among machines in the cluster using NFS, you must allow all files to be world-readable (by default), or you must export the file system with root permission to all hosts that will be running TWS LoadLeveler. When mounting the file system, do not use the -nosuid flag. v TWS LoadLeveler log files should be stored locally at the nodes for best performance. v TWS LoadLeveler for AIX can be run with cluster security services. If you want to enable cluster security services, see TWS LoadLeveler: Using and Administering for details. v TWS LoadLeveler for Linux does not support cluster security services. Migration considerations Before migrating TWS LoadLeveler, consider the following specific migration-related activities. This topic discusses considerations for migrating to TWS LoadLeveler for AIX 3.5 from a previous TWS LoadLeveler for AIX release and for migrating to TWS LoadLeveler for Linux 3.5 from a previous TWS LoadLeveler for Linux release. Note: If you are running TWS LoadLeveler in a mixed cluster, refer to the appropriate level of documentation for each version of TWS LoadLeveler in the cluster. TWS LoadLeveler migration considerations Make these considerations before migrating to TWS LoadLeveler from a previous TWS LoadLeveler release. Use either of the following two methods to migrate TWS LoadLeveler: 1. Take down all of the machines and migrate the entire cluster at once. 2. Migrate your cluster gradually one machine or a group of machines at a time. This method requires that the first machine in the cluster to be migrated is the central manager. Whichever method you choose, you should make backup copies of the following files: v LoadL_admin, the TWS LoadLeveler administration file v LoadL_config, the TWS LoadLeveler global configuration file v LoadL_config.local, the TWS LoadLeveler local configuration file Note: If process tracking is set and the TWS LoadLeveler kernel extension is already loaded when migration occurs, you might need to reboot the nodes. The process tracking kernel extension is not unloaded. Therefore, if a mismatch in the version is found, the startd daemon will exit. A reboot of the node is needed to unload the currently loaded kernel extension. Migrating TWS LoadLeveler Follow either of these step-by-step instructions depicting common migration scenarios. Chapter 2. Pre-installation planning 7

20 Scenario 1: Migrating an entire TWS LoadLeveler cluster requires that all of the machines in the cluster be migrated together. Scenario 2: Machine-by-machine TWS LoadLeveler migration corresponds to a machine-by-machine migration strategy. Scenario 1: Migrating an entire TWS LoadLeveler cluster: This scenario requires that all of the machines in the cluster be migrated together. 1. Log in as a TWS LoadLeveler administrator. Issue the command llctl -g drain. This will drain all of the Schedd machines so that no new jobs can be submitted to TWS LoadLeveler. This will also drain all of the startd machines so that no jobs already submitted to TWS LoadLeveler can be started. Jobs that are currently running will continue to run. 2. After all of the running jobs have completed and only idle jobs remain in the job queues, issue the command llctl -g stop to stop TWS LoadLeveler on all of the machines in the cluster. 3. Run the llsummary command on the machines that run the Schedd daemons if accounting reports are required. Accounting reports are made by first running the llacctmrg command to create a global history file before running the llsummary command. 4. Perform a backup of all of the files in the TWS LoadLeveler spool directory on all machines that run a Schedd daemon. This step is not essential, but strongly suggested in case something goes wrong during the installation. 5. Install the new version of TWS LoadLeveler using the instructions described in Chapter 3, Installing TWS LoadLeveler for AIX, on page 15 or Chapter 4, Installing TWS LoadLeveler for Linux, on page 25. You will need root permission to perform this operation. Under normal circumstances, the files in the spool directories (job queues, history files, executables of batch jobs, and so on) are not affected by the installation of TWS LoadLeveler. If these files are accidentally deleted during the installation process, the saved files from the previous step can be used to perform the appropriate restore operations. 6. Issue the command llctl -g start to bring up the migrated machines. Scenario 2: Machine-by-machine TWS LoadLeveler migration: This scenario corresponds to a machine-by-machine migration strategy. Being able to migrate one machine at a time implies that the new release of TWS LoadLeveler can coexist with the preceding TWS LoadLeveler release. In other words, it is possible to set up a TWS LoadLeveler cluster consisting of some machines running TWS LoadLeveler 3.5 level of software while the rest of the cluster remains at the TWS LoadLeveler 3.4 software level. Note: If you run in this environment, it is possible that some TWS LoadLeveler commands will display message referring to ** unknown LL_Specification ** multiple times. This message can be ignored. The message will not be displayed after all machines in the cluster are upgraded to the new release. 1. Select the machines that you want to migrate. The central manager machine must be migrated before any other machines can be restarted with the new TWS LoadLeveler release. Log in as a TWS LoadLeveler administrator. Issue the command llctl -h machine_name drain. As a result of this action, no new 8 TWS LoadLeveler: Installation Guide

21 jobs can be submitted to this machine and no new jobs will be started on this machine. Jobs that are currently running will continue to run. Repeat this step for all of the machines to be migrated. 2. After all running jobs on the machines to be migrated have completed and only idle jobs remain in the queues of the Schedd daemons on the machines to be migrated, issue the command llctl -h machine_name stop to stop TWS LoadLeveler on the specified machine. Repeat this action for all of the machines to be migrated. 3. Perform a backup of all of the files in the TWS LoadLeveler spool directory on the machines to be migrated that run the Schedd daemon. This step is not essential, but strongly suggested in case something goes wrong in the next step. 4. Install TWS LoadLeveler on the machines to be migrated using the instructions described in Chapter 3, Installing TWS LoadLeveler for AIX, on page 15 or Chapter 4, Installing TWS LoadLeveler for Linux, on page 25. You will need root permission to perform this operation. Under normal circumstances, the files in the spool directories are not affected by the installation of TWS LoadLeveler. If these files are accidentally deleted during the installation process, the files saved in the previous step can be used to perform the appropriate restore operations. 5. Issue the command llctl -h machine_name start to bring up TWS LoadLeveler 3.5 on the specified machine. Repeat this action for all of the machines to be migrated. At this point, the migrated machines should be running TWS LoadLeveler 3.5 and any jobs carried over will run in the upgraded cluster. New features in TWS LoadLeveler 3.5 cannot be used in a mixed cluster of TWS LoadLeveler 3.4 and 3.5 machines. If you are running TWS LoadLeveler in a mixed cluster, refer to the appropriate level of information for each version of TWS LoadLeveler in the cluster. TWS LoadLeveler APIs: TWS LoadLeveler provides application programming interfaces (APIs) that allow application programs written by customers to use specific data or functions that are part of TWS LoadLeveler. These APIs are enabled in such a way that user programs linked dynamically to the library libllapi.a on AIX or libllapi.so on Linux do not have to be compiled and linked again when an installation migrates from one TWS LoadLeveler release to the next. Note that a user application compiled with an earlier TWS LoadLeveler version is limited to the information specific to that release. Thus, if you have an application using a TWS LoadLeveler API compiled in a previous TWS LoadLeveler release, you do not absolutely have to recompile to use it with latest version of TWS LoadLeveler. However, this application cannot access the new data and functions specific to the latest TWS LoadLeveler release. In order to do this, you may have to make enhancements to the source codes of your application and recompile this application on a TWS LoadLeveler machine using the new version of the llapi.h header file. Hardware, software, and disk requirements Planning for installation requires two main tasks: ensuring you have the correct software and hardware installed, and allocating the correct amount of disk space for directories. Chapter 2. Pre-installation planning 9

22 Hardware Note: You are responsible for licensing and installing these products and integrating them with TWS LoadLeveler. requirements Installation planning requires that you ensure that compatible hardware will be installed. The following topics list the TWS LoadLeveler hardware requirements. TWS LoadLeveler for AIX hardware requirements TWS LoadLeveler for AIX 6.1 and AIX 5.3 are compatible with certain hardware platforms. TWS LoadLeveler for AIX 6.1 and AIX 5.3 runs on: v IBM Power Systems (POWER6, POWER5, and POWER4 ) v IBM BladeCenter Power Architecture servers v IBM Cluster 1600 TWS LoadLeveler for Linux Multiplatform hardware requirements TWS LoadLeveler for Linux Multiplatform supports certain hardware platforms. TWS LoadLeveler for Linux Multiplatform supports the following platforms: v IBM System servers with AMD Opteron or Intel EM64T processors v IBM System x servers v IBM BladeCenter Intel processor-based servers v IBM System Cluster 1350 Note: IBM Tivoli Workload Scheduler LoadLeveler is supported when running Linux on non-ibm Intel-based and AMD hardware servers. Supported hardware includes: v Servers with Intel 32-bit and Intel Extended Memory 64 Technology (EM64T) v Servers with Advanced Micro Devices (AMD) 64-bit technology TWS LoadLeveler for Linux Multiplatform requires: v A minimum of 1 GB of memory v 1 GB of swap space TWS LoadLeveler for Linux on POWER hardware requirements TWS LoadLeveler for Linux on POWER supports certain hardware platforms. TWS LoadLeveler for Linux on POWER supports the following platforms: v IBM Power Systems (POWER6, POWER5, and POWER4) v IBM BladeCenter Power Architecture servers v IBM OpenPower Software TWS LoadLeveler for Linux on POWER requires: v A minimum of 1 GB of memory v 1 GB of swap space requirements Installation planning requires that you ensure that compatible software will be installed. 10 TWS LoadLeveler: Installation Guide

23 The following topics list the TWS LoadLeveler software requirements. TWS LoadLeveler for AIX software requirements TWS LoadLeveler for AIX 6.1 and AIX 5.3 have certain software platform requirements. Table 7 lists the TWS LoadLeveler AIX 6.1 and AIX 5.3 software requirements. Table 7. TWS LoadLeveler for AIX software requirements To use: TWS LoadLeveler for AIX AIX operating systems, including bos.cpr AIX 6.1, TL AIX 5.3, TL Mandatory TWS LoadLeveler Service APAR IZ31336 If you want: Parallel jobs Using: loadleveler/download/aix.html POE Parallel Environment (PE) 5.1 MPICH MPICH or later OpenSSL in a multicluster environment OpenSSL or later RDMA with InfiniBand adapters AIX 6.1: LAPI MPICH-GM On AIX 5.3: MPICH or later and GM or later AIX 5.3: LAPI OpenMP thread binding XL C or C or later RSCT or the llextrpd command XL FORTRAN or later or later (See note 1.) TWS LoadLeveler graphical user interface (xloadl) AIXWindows Environment Java Runtime Environment filesets TaskGuide Runtime Environment fileset (sysmgt.sguide.rte) Notes on the TWS LoadLeveler for AIX software requirements: 1. If you plan to configure TWS LoadLeveler to support the cluster security services component of RSCT, ensure that for each machine in the TWS LoadLeveler cluster, that cluster security services and Host Based Authentication (HBA) are installed and configured. The cluster security services component of RSCT is installed as part of the base AIX installation. HBA should be configured as the cluster security services default and the HBA ctcasd daemon should be running. The Trusted Hosts List (THL) must be updated to include other machines that will need to be authenticated. Chapter 2. Pre-installation planning 11

24 TWS LoadLeveler for Linux software requirements TWS LoadLeveler for Linux has certain software platform requirements. Table 8 lists the TWS LoadLeveler for Linux software requirements. Table 8. TWS LoadLeveler for Linux Multiplatform and TWS LoadLeveler for Linux on POWER software requirements To use: TWS LoadLeveler for Linux Multiplatform Linux operating systems (requires libelf, libxm, and libxp packages) Mandatory TWS LoadLeveler Service If you want: RHEL 4 or RHEL 5 SLES 9 or SLES 10 RPMs for Process tracking RHEL 5, SLES 10: afs (See note 1.) IBM System Blue Gene Solution Parallel jobs Using: OpenSSL in a multicluster environment RDMA with InfiniBand adapters TWS LoadLeveler for Linux on POWER You need to download, build, and install the process tracking kernel module. The source files and installation instructions are available at: kernels.html RHEL 4 or RHEL 5: OpenAFS or later Not applicable Not applicable SLES 9: Blue Gene/L Release 2 or later Not applicable SLES 10: Blue Gene/P Release 2 or later POE Parallel Environment (PE) 5.1 MPICH MPICH or later MPICH-GM On AIX 5.3: MPICH or later and GM or later MVAPICH MVAPICH or later OpenSSL or later Not applicable SLES10: LAPI OpenMP thread binding Not applicable XL C or C or later RSCT or the llextrpd command TWS LoadLeveler graphical user interface (xloadl) Not applicable XL FORTRAN or later RSCT or later IBMJava2-JRE Notes on the TWS LoadLeveler for Linux software requirements: 1. OpenAFS or later is supported on TWS LoadLeveler for Linux nodes. It can be used for token passing and token refresh between Linux and AIX. OpenAFS is not supported on TWS LoadLeveler for AIX nodes. 12 TWS LoadLeveler: Installation Guide

25 2. If you plan to use scheduling affinity, SLES 10 SP2 is required. To use scheduling affinity on SLES 10 SP2, you must create a /dev/cpuset directory using the mkdir /dev/cpuset command. Then mount a cpuset file system to /dev/cpuset using the following command: mount -t cpuset none /dev/cpuset 3. If you plan to include a Blue Gene system in your cluster, ensure that all symbolic links have been created in /usr/lib64 as required in the Blue Gene installation procedure. See the Blue Gene Driver README available at the Blue Gene Software Delivery Web site for details on creating the required symbolic links in /usr/lib You can download the Myrinet software from the following Web site: Disk space requirements Certain directories require specific amounts of disk space. In order for your installation to be successful, ensure that your directories have adequate space available. Table 9 lists disk space requirements for TWS LoadLeveler for AIX facilities. Table 9. Information about TWS LoadLeveler for AIX facilities Directory Size Release directory (/usr/lpp/loadl/full) 45 MB¹ Local directory 15 MB (minimum) Home directory No limits unless same as release or local directory Release directory for submit-only (/usr/lpp/loadl/so) 35 MB Configuration tasks directory (usr/lpp/loadl/codebase) 1 MB ¹ If installing the LoadL.webui sample, add 40 MB for /usr/lpp/loadl/ and 55 MB for /opt/ibm/ll/webui. For more information about this sample, see the README file in samples/llwebui in the release directory. Table 10 shows disk space requirements for TWS LoadLeveler for Linux facilities. Table 10. Information about TWS LoadLeveler for Linux facilities Directory Size Release directory (/opt/ibmll/loadl/full) 30 MB¹ Local directory 20 MB (minimum) Home directory No limits unless same as release or local directory Release directory for submit-only (/opt/ibmll/loadl/so) 22 MB System directory (/usr/lib or /usr/lib64) 24 MB (additional) ¹ If installing the LoadL-webui RPM, add 40 MB for /opt/ibmll/loadl and 60 MB for /opt/ibm/ll/webui. For more information about this sample, see the README file in samples/llwebui in the release directory. Chapter 2. Pre-installation planning 13

26 Configuration example This example shows a TWS LoadLeveler installation on a small pool of workstations connected by a network. Many other configurations are possible. Table 11 demonstrates one way to install TWS LoadLeveler. Table 11. TWS LoadLeveler configuration example TWS LoadLeveler element Example directory or location on central manager machine Machine name mycenman mynode01 Central manager mycenman mycenman TWS LoadLeveler for AIX home directory TWS LoadLeveler for Linux home directory Example directory or location on first node in cluster /u/loadl (exported) /u/loadl (mounted) /home/loadl (exported) /home/loadl (mounted) Local directory /var/loadl/mycenman /var/loadl/mynode01 TWS LoadLeveler for AIX release directory TWS LoadLeveler for Linux release directory Global configuration file LoadL_config (in home directory) /usr/lpp/loadl/full (exported) /usr/lpp/loadl/full (mounted) /opt/ibmll/loadl/full (exported) /opt/ibmll/loadl/full (mounted) LoadL_config (in home directory) One copy for all machines. Local configuration file LoadL_config.local One copy for each machine. Administration file LoadL_admin (in home directory) One copy for all machines. log directory /var/loadl/mycenman/log (in subdirectory of a local directory) spool directory /var/loadl/mycenman/spool (in subdirectory of a local directory) execute directory /var/loadl/mycenman/execute (in subdirectory of the local directory) One copy for all machines. LoadL_config.local (in local directory) One copy for each machine. LoadL_admin (in home directory) One copy for all machines. /var/loadl/mynode01/log (in subdirectory of a local directory) /var/loadl/mynode01/spool (in subdirectory of a local directory) /var/loadl/mynode01/execute (in subdirectory of the local directory) Entries in the TWS LoadLeveler for AIX LoadL_admin file and the LoadL_config file appear in Table 12. For TWS LoadLeveler for Linux on System x and TWS LoadLeveler for Linux on POWER, update RELEASEDIR to RELEASEDIR=/opt/ ibmll/loadl/full in the LoadL_config or LoadL_config.local file. Table 12. Sample LoadL_admin and LoadL_config file entries LoadL_admin file LoadL_config file mycenman: type=machine central_manager=true mynode01: type=machine central_manager=false RELEASEDIR=/usr/lpp/LoadL/full LOCAL_CONFIG=$(tilde)/$(host)/LoadL_config.local LOG=/var/loadl/$(host)/log SPOOL=/var/loadl/$(host)/spool EXECUTE=/var/loadl/$(host)/execute 14 TWS LoadLeveler: Installation Guide

27 Chapter 3. Installing TWS LoadLeveler for AIX You can install TWS LoadLeveler for AIX from the product media. Use this information to configure your TWS LoadLeveler environment. The System Management Interface Tool (SMIT) is suggested for performing the installation, but you can also use the command line to install TWS LoadLeveler. These installation instructions assume that the TWS LoadLeveler user ID is loadl. See Change the TWS LoadLeveler for AIX user ID on page 17 for information on using a user ID other than loadl as the TWS LoadLeveler user ID. Decide on directories for TWS LoadLeveler for AIX Decide on and record the locations of your home and local directories. Use the information in Installation overview on page 3 to decide on the location of the home and local directories. Once you decide, write the locations in the Actual Directory column in Table 13. Table 13. TWS LoadLeveler for AIX directory location checklist TWS LoadLeveler element Example directory Actual directory Local directory /var/loadl Home directory /u/loadl Release directory /usr/lpp/loadl/full /usr/lpp/loadl/full Name of central manager machine mycenman Important: The local directory must be unique to each machine in the cluster and must not be shared by any other machines in the cluster. Also, each machine must have its own TWS LoadLeveler spool, execute, and log directories. Log in as root on TWS LoadLeveler for AIX Unless otherwise indicated, all of the steps performed during TWS LoadLeveler installation should be done from the root user ID. Create the TWS LoadLeveler for AIX loadl group name Create the group name for every node in the TWS LoadLeveler cluster. The procedure in Table 14 on page 16 creates the group name only for the node that you are currently logged into. It must be repeated for every node in the TWS LoadLeveler cluster. Copyright IBM Corp. 1986,

28 Table 14. Procedure for creating the loadl group name If using: Do this: SMIT TYPE smit SELECT Security and Users SELECT Groups SELECT Add a Group ENTER PRESS The appropriate information in each field. For example: Group name: loadl Administrative group: true The group name and ID must be the same on each machine in the cluster. Ok PRESS Done and then select System Management from the Return to: list. mkgroup Enter the following command: If using SMITTY, press F3 (cancel) until you return to the System Management window. mkgroup -a loadl Create the TWS LoadLeveler for AIX loadl user ID Create the user ID for every node in the TWS LoadLeveler cluster. The procedure in Table 15 on page 17 creates the user ID only for the node that you are currently logged into. It must be repeated for every node in the TWS LoadLeveler cluster. 16 TWS LoadLeveler: Installation Guide