Platform LSF concepts and terminology

Transcription

1 Platform LSF concepts and terminology Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 7.0

2 Unit objectives After completing this unit, you should be able to: Define Platform LSF terminology Describe LSF architecture

3 What is Platform LSF Platform LSF (load sharing facility) is a suite of distributed resource management products that: Connects computers into a cluster (or grid) Monitors loads of systems Distributes, schedules, and balances workload Controls access and load by policies Analyzes the workload Provides transparent access to all available resources

4 Without Platform LSF Which node can run my job or task? In a distributed environment (hundreds of hosts): Monitoring and controlling of resources is complex Work load is silo -based Resource usage imbalance Users perceive a lack of resources.

5 Without Platform LSF: Example I don t have enough resources to do my work Job Job 1 2 Job Job Patterns: 1. Users over utilize popular/well known hosts Users perceive a lack of resources 2. New hosts go under utilized Administrators perceive excessive resources. Overall resources usage is poor and customer satisfaction is reduced.

6 With Platform LSF Now, Platform LSF will run my job or task on the best node available! Virtual pool of computing resources managed by Platform LSF In a Platform LSF environment (hundreds of hosts): Monitoring and controlling of resources is simple Work load is balanced Resource usage is balanced Users perceive a plethora of resources. All nodes are grouped into a cluster

7 With Platform LSF - Example With Platform LSF, I now have enough resources to do my work! Patterns: 1 Job Job 1. Users jobs are spread out across cluster nodes resulting in resources not being over utilized 2. New hosts become utilized as soon as LSF is installed and started on the host. Overall resource usage is optimized and customer satisfaction is increased. Virtual pool of computing resources managed by Platform LSF 2 All nodes are grouped into a cluster Job Job

8 Key Platform LSF objectives Provide the means to create a powerful computer system made up of many smaller systems to increase productivity and lower operating costs. Match limited supply of resources with demand.

9 Platform LSF benefits Cost management Spend money where it counts - increase profits. Time management Reduce resource idle time. Productivity management Improve personnel productivity, design, and product quality Increase job throughput Reduce time-to-market and costs. Resource management Maximize license utilization Maximize resource usage Maximize sharing of resources.

10 Platform LSF architecture Enterprise applications Application Application Application Application Application Services in Real-time Application Servers Ondemand Heterogeneous enterprise resources Network Servers Licenses Data Storage bandwidth

11 Platform LSF terminology (1 of 2) Cluster A collection of TCP/IP networked hosts running Platform LSF. Master host A cluster requires a master host. This is the first host installed. The master host controls the rest of the hosts in the grid. Master candidates Master failover host. Server host A host within the cluster that submits and executes jobs and tasks. Client host A host within the cluster that only submits jobs and tasks.

12 Platform LSF terminology (2 of 2) Execution host The host that executes the job or task. Submission host Job Queue Job slot The host from which a job or task is submitted. A command submitted to Platform LSF batch. Can take more than one Job slot. A network-wide holding place for jobs which implements different job scheduling and control policies. The basic unit of processor allocation in Platform LSF. Can be more than one per physical processor.

13 Platform LSF terminology diagram ( 1 of 2 ) Cluster Client host Master host Server host Acts as a submission host Acts as a submission host and execution host Can be CLOSED to not allow execution or submission Acts as a submission host and execution host Can also be a master candidate

14 Platform LSF terminology diagram (2 of 2 ) Cluster Job Job Job Submission host Job submitted to LSF from this host LSF client or LSF server Master host Job placed in queue Passed to scheduler Dispatched to the execution host with an open job slot LSF master Execution host Execution host receives the job Begins execution of job or task LSF server

15 Platform LSF overview 1 1. Individual machines are grouped into a cluster to be managed by Platform LSF One machine in the cluster is selected as the master of Platform LSF. 3. Each slave machine in the cluster collects its own vital signs periodically and reports them back to the master. 4. Users then submit their jobs to Platform LSF and the master makes a decision on where to run the job based on the collected vital signs.

16 Platform LSF job submission and control Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 7.0

17 Unit objectives After completing this unit, you should be able to: Submit jobs Manipulate jobs such as stopping, resuming and terminating View detailed job information

18 Setting up the Platform LSF environment Setup the Platform LSF environment before using Platform LSF commands. C-shell type shells (csh, tcsh) % source /YZGROUP/ibm/lsf/conf/cshrc.lsf Bourne type shells (sh, bash, ksh, and so on.) $. /YZGROUP/ibm/lsf/conf/profile.lsf Check after setting up environment: % env grep -i lsf MANPATH=/YZGROUP/ibm/lsf/10.1/man: LSF_SERVERDIR=/YZGROUP/ibm/lsf/10.1/linux2.6-glibc2.3-x86_64/etc LSF_LIBDIR=/YZGROUP/ibm/lsf/10.1/linux2.6-glibc2.3-x86_64/lib LD_LIBRARY_PATH=/tools/user01/lsf/9.1/linux2.6-glibc2.3-x86_64/lib PATH=/YZGROUP/ibm/lsf/10.1/linux2.6-glibc2.3-x86_64/etc:/YZGROUP/ ibm/lsf/10.1/linux2.6-glibc2.3-x86_64/bin:/bin:/usr/bin:/usr/ local/bin LSF_BINDIR=/YZGROUP/ibm/lsf/10.1/linux2.6-glibc2.3-x86_64/bin LSF_ENVDIR=/YZGROUP/ibm/lsf/conf

19 Job submission and control commands bsub [options] command [cmdargs] bjobs [-a][-j jobname][-u usergroup -u all][ ] jobid bhist [-a][-j jobname][-u usergroup -u all][ ] jobid bbot/btop [jobid "jobid[index_list] ] [position] bkill [-J jobname] [-m] [-u ] [-q] [-s signalvalue] bmod [bsub_options] jobid bpeek [-f] jobid bstop/bresume jobid bswitch destination_queue jobid

20 Platform LSF job submission commands (1 of 4) bsub [commonly used options] -n # - number of CPUs required for the job -o filename - redirect stdout, stderr, and resource usage -oo filename information of the job to the specified output file - same as -o, but overwrite file if it exists -e filename - redirect stderr to the specified error file -eo filename - same as -e, but overwrite file if it exists -i filename - use the specified file as standard input for the job -q qname - submits the job to the specified queue -m hname - select host(s) or host group; keywords all and others can be used -J jobname - assigns the specified name to the job

21 Platform LSF job submission commands (2 of 4) bsub [options] -Q [exit_code] [EXCLUDE(exit_code)] - Success exit code -L login_shell - Initializes the execution environment using the specified login shell -n number - if PARALLEL_SCHED_BY_SLOT=Y in lsb.params, then specify number of job slots, not processors -g jobgroup - submit job to specified group -sla serviceclass - submit job to specified service class -W runlimit - if ABS_RUNLIMIT=Y uses wall clock time -app - application profiling

22 Platform LSF job submission commands (3 of 4) bsub [setting limits] -C core_limit - set the per-process core file size limit (KB) -c cpu_time - limit the total CPU time for the job ([HH:]mm) -cwd dir_path - specify the current working directory for the job -W runlimit - set the run time limit for the job ([HH:]MM) -We run_time - specifies an estimation run time for the job ([HH:]MM)

23 Platform LSF job submission commands (4 of 4) bsub [options] -B - send when the job is dispatched -H - hold the job in the PSUSP state after submission -N - job header only when job completes -b begin_time - dispatch the job after the specified time with year and time -G user_group - associate job with specified user group (fairshare) -t term_time - specify the job termination deadline with year and time -u mail_user - send to the specified address

24 Platform LSF interactive job submission bsub -I submit an interactive batch job -Ip -Is submit an interactive batch job with pseudo-tty support submit an interactive batch job and create a pseudo-tty with shell mode support Examples: [lsfuser@host1 conf]$ bsub -I hostname Job <212> is submitted to default queue <interactive>. <<Waiting for dispatch...>> <<Starting on host2>> host2

25 Platform LSF interactive: SSH support Four bsub options handle interactive jobs using SSH connection: -IS - submit interactive job with SSH -ISp - submit interactive pseudo terminal job with SSH -ISs - submit interactive pseudo terminal shell job with SSH -IX - used to submit interactive X-window job with SSH SSH port forwarding is used to create a secured channel between the submission host and the execution host for X-Window jobs. Do not need to run xhost + command on the X-server host. The X-server host can handle connections through the SSH channel. Use the command netstat -an and ps ef to check the SSH channel of the interactive job.

26 bsub: Condensed host notations bsub -m host[1-11,24]+1 host[12-23]+2 job1 The (+) indicates that host[12-23] are the most preferred and host[1-11] and host24 are the least preferred. Commands that can use the notation: bsub brun bmod brestart brsvadd brsvmod bswitch bjobs bhist bacct brsvs bmig bpeek

27 bsub 4k jobname bsub J `pwd` command Allows user to use CWD or PWD as the job name # bsub J`pwd` sleep 100 Job <4>, Job Name </tools/home/john/regression/chip1>, User <awu>, Project <default>, Status <RUN>, Queue <normal>, Command <sleep 100> Thu Apr 17 17:48:43: Submitted from host <ibm01.platform.com>, CWD <$HOME/john/regression/tool/data>; Thu Apr 17 17:51:55: Started on <ibm01.lsf.platform.com>, Execution Home </home/john>, Execution CWD </home/john/regression/tool/data>; Thu Apr 17 17:52:12: Resource usage collected. MEM: 2 Mbytes; SWAP: 5 Mbytes; NTHREAD: 1 PGID: 30400; PIDs: 30400

28 Available methods for submitting jobs By script or command % cd /home/user/project_dir % bsub q parallel a fluent n 4./my_fluent_launcher.sh By job spooling % bsub < spoolfile Interactively % bsub bsub> #BSUB q parallel n 4 bsub> #BSUB a fluent bsub> cd /home/user/project_dir bsub>./my_fluent_launcher.sh bsub> ^D Job <1234> submitted to queue <parallel> Using Platform Application Center (PAC). Note: PAC is a separate product. Example spoolfile #BSUB q parallel #BSUB n 4 a fluent cd /home/user/project_dir./my_fluent_launcher.sh

29 Platform LSF interactive task submission lsrun Submits a task to Platform LSF for execution Uses the Platform LSF base system only (no queues policies) Task will run even if hosts are closed Tasks will run immediately (no scheduling delay) Tasks will not run when a host is busy Can be disable with configuration change Note: As an LSF Administrator you DO NOT want users running this command to submit jobs as it circumvents the scheduling policies! Disable by setting LSF_DISABLE_LSRUN=y in lsf.conf

30 Platform LSF job termination bkill Bulk job termination--new bkill -b option for terminating a bulk set of jobs from the system. This eases administration and decreases the time to delete a large number of jobs. Example: To terminate all jobs of the current user % bkill b 0

31 Viewing submitted job information bjobs Can display parallel jobs and condensed host groups in an aggregate format -a Display information about jobs in all states (including finished jobs) -A Display summarized information about job arrays -d Display information about jobs that finished recently -l -w Display information in long or wide format -p Display information about pending jobs -r Display information about running jobs -g job_group Display information about jobs in specified group -J job_name Display information about specified job or array -m host_list Display information about jobs on specified hosts or groups -P project_name Display information about jobs in specified project -q queue_name Display information about jobs in specified queue -u user_name Display information about jobs for specified users/groups -sum Display information about jobs that are unfinished

32 Job name: Wild card "*" -J Option Supports multiple wildcards in any position (beginning, middle, and end) of a job and job array names Examples: -J joba* -J job*a -J "*A -J * - To match all job -J "myar*aya[1]

33 View submitted job information example % bjobs u all a JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 1233 user1 DONE normal training8 training1 *sortname Nov 21 10: user1 RUN priority training8 training1 *verilog Nov 21 10: user2 PEND night training9 *sortfile Nov 21 10: user2 PEND normal training9 *sortname Nov 21 10:04 % bjobs -u all -J "*sort*: JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 1233 user1 DONE normal training8 training1 *sortname Nov 21 10: user2 PEND night training9 *sortfile Nov 21 10: user2 PEND normal training9 *sortname Nov 21 10:04

34 View detailed submitted job information % bjobs -l 1235 Job <1235>, User <user2>, Project <default>, Status <PEND>, Queue <night>, Command <night_job> Wed Nov 21 10:03:51: Submitted from host <host9>, CWD <$HOME>; Pending reasons: The queue is inactivated by its time windows; Scheduling parameters: r15s r1m r15m ut pg io ls it tmp swp mem loadsched Resource Requirement Details ( LSF ): RESOURCE REQUIREMENT DETAILS: Combined: select[type == local] order[r15s:pg] Effective: -

35 Viewing historical job information bhist -a Display information about all jobs (overrides d, -p, -r, and -s) -b -l -w Display information in brief, long, or wide format -d Display information about finished jobs -p Display information about pending jobs -s Display information about suspended jobs -t Display job events chronologically -C -D -S -T start_time Display information about completed, end_time dispatched, submitted, or all jobs in specified time window -P project Display information about jobs belonging to specified project -q queue Display information about jobs submitted to specified queue -u username all Display information about jobs submitted by user

36 View historical job information example (1 of 2) % bhist Summary of time in seconds spent in various states: JOBID USER JOB_NAME PEND PSUSP RUN USUSP SSUSP UNKWN TOTAL 2299 alfred *eep alfred *eep alfred *eep alfred *eep

37 View historical job information example (2 of 2) % bhist -l 2302 Job <2302>, User <lsfuser>, Project <default>, Command <sleep 100> Wed Mar 30 13:53:44: Submitted from host <host1>, to Queue <normal>,cwd<$home>; Mon Mar 18 13:30:37: Dispatched to <host3>, Effective RES_REQ <select[type ==local] order[r15s:pg] >; Wed Mar 30 13:53:47: Starting (Pid 24595); Wed Mar 30 13:53:52: Running with execution home </home/lsfuser>, Execution CWD </home/lsfuser>, Execution Pid <24595>; Wed Mar 30 13:55:32: Done successfully. The CPU time used is 0.0 seconds; Wed Mar 30 13:55:32: Post job process done successfully; MEMORY USAGE: MAX MEM: 3 Mbytes; AVG MEM: 3 Mbytes Summary of time in seconds spent in various states by Wed Mar 30 13:55:32 PEND PSUSP RUN USUSP SSUSP UNKWN TOTAL

38 Manipulating jobs options bbot btop bkill bmod bpeek bstop bresume bswitch moves a pending job to the bottom of the queue. moves a pending job to the top of the queue. sends a signal to kill, suspend or resume unfinished jobs (use a job ID of 0 to kill all your jobs). New scalability improvements resulting in improved performance and user experience. modifies job submission options of a job. displays the stdout and stderr of an unfinished job. suspend unfinished jobs. resumes one or more suspended jobs. switches unfinished jobs to another queue.

39 Platform LSF job submission examples (1 of 2) Example #1 % bsub /home/lsf/scripts/sleeper Job <1234> is submitted to default queue <normal> % bstop 1234 Job <1234> is being stopped % bresume 1234 Job < 1234> is being resumed Example #2 % bsub q night m "host1 host2" night_job Job <1235> is submitted to queue <night> % bmod m "hostgroupa" 1235 Parameters of job <1235> are being changed

40 Platform LSF job submission examples (2 of 2) Example #3 % bsub i "/home/user1/in.dat" -o "/home/user1/out.dat" e "/home/user1/error.dat" long_job Job <1236> is submitted to default queue <normal> % bswitch night 1236 Job <1236> is switched to queue <night> Example #4 % bsub P Research J Projection1 project_job Job <1238> is submitted to default queue <normal> % btop 1238 Job <1238> has been moved to position 1 from top % bbot 1238 Job <1238> has been moved to position 1 from bottom

41 Job arrays A job array is a Platform LSF structure that allows a sequence of jobs to: Share the same executable Have different input files Be controlled and monitored as a single job unit. Once the job array has been submitted, the individual elements of the array are independently scheduled and dispatched. Examples: regression testing, rendering, and file conversion.

42 Creating job arrays (1 of 2) Syntax: bsub J "AName[index, ]" i in.%i appbin Where: J "AName[ ]" names and creates the job array index can be start[-end[:step]] where: start specifies an individual index or range of indices end specifies the end of a range of indices step specifies the value to increment indices in a range i in.%i is the input file to be used and %I is the array index reference value appbin is the application to be executed

43 Creating job arrays (2 of 2) By default, a maximum number of 1000 elements can be submitted at one time using a job array. To allow a greater number of elements in a job array submission, in lsb.params: MAX_JOB_ARRAY_SIZE = integer where integer can be from 1 to

44 Submitting job arrays examples (1 of 3) Example #1 % bsub J "test[1-1000]" i "in.%i" o "out.%j.%i" appa Job <104> submitted to default queue <normal> The %I is the index value of the job array and %J is the jobid assigned for the submitted job array. Example output files: out.104.1, out.104.2, out.104.3, out.104.4, Example #2 % bsub J "test[1-100:3]" i "in.%i" o "out.%j.%i appa Job <105> submitted to default queue <normal> Step through the index values by iteration of three. Example output files: out.105.1, out.105.4, out.105.7, out

45 Submitting job arrays examples (2 of 3) Example #3 % bsub J "test[1-10,45,90]" i "in.%i o "out.%i.%j" appa Job <106> submitted to default queue <normal> Specific index values can be specified. Example output files: out.104.1, out.104.2, out.104.3,, out , out

46 Submitting job arrays examples (3 of 3) Example #4 % bsub J "test[1-200]%2" i "input.%i" appa Job <108> submitted to default queue <normal> Only 2 elements from this job array may run concurrently in the cluster at one time. [lsfuser@host1 ~]$ bjobs -A 108 JOBID ARRAY_SPEC OWNER NJOBS PEND DONE RUN EXIT SSUSP USUSP PSUSP 108 test[1-2 lsfuser Example output files: out.104.1, out.104.2, out.104.3,, out , out

47 Monitoring job arrays View all elements of the job array % bjobs 104 JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 104 user1 RUN normal training8 training7 test[1] Jun 6 15: user1 RUN normal training8 training5 test[2] Jun 6 15: user1 PEND normal training8 test[3] Jun 6 15:40 View a single element of the job array % bjobs "104[90]" JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 104 user1 PEND normal training8 test[90] Jun 6 15:40 View the summary status of the job array % bjobs A 104 JOBID ARRAY_SPEC OWNER NJOBS PEND DONE RUN EXIT SSUSP USUSP PSUSP 104 test user

48 Terminating job arrays To terminate all elements of an array: % bkill 104 Job <104>: Operation is in progress To terminate an individual element: % bkill "104[90]" Job <104[90]> is being terminated To terminate a group of elements: % bkill "104[1-10,75,90]" Job <104[1]> is being terminated Job <104[10]> is being terminated Job <104[75]> is being terminated Job <104[90]> is being terminated

49 Resource Requirement String Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 7.0

50 Resource requirements Each job or task can have its own resource requirements. Server hosts that match the resource requirements of the job are considered as candidates for execution. Can be set in queues and/or for individual jobs.

51 Resource requirement string (1 of 2) Platform LSF Job level resource requirement: bsub R Queue level and Application profile resource requirement: RES_REQ Describes the resources required by a job or task. Used for mapping tasks and jobs onto execution hosts.

52 Resource requirement string (2 of 2) A resource requirement string is divided into the following sections: Selection - select[selection_string] Ordering - order[order_string] Usage - rusage[rusage_string] Locality - span[span_string] Same - same[same_string] Compute unit - cu[cu_string] Affinity - affinity[affinity_string] Note: cu[] and affinity[] are covered in the advanced LSF class. The span and same sections are specifically used for parallel jobs.

53 Selection string (1 of 2) A logical expression built from a set of resource names. Specifies the characteristics of a server host to be considered as a potential execution host. Evaluated for each host If the expression evaluates to true, then that host is considered a candidate.

54 Selection string (2 of 2) The select keyword can be omitted if the selection section is first in the resource requirement string. The default selection string for execution type commands such as bsub or lsrun is: select[type==local] The default selection string for query type commands such as lsload is: select[type==any] The default selection string for query and execution type commands such as bsub or lsrun on a client host is: select[type==any]

55 Selection string examples (1 of 2) $ bsub R "select[type==any && swp>=300 && \ mem>500]" job1 Select a candidate execution host of any type which has at least 300 MB of available swap and more than 500 MB of available memory. $ lsload R "select[type==local && cpuf<18.0] Displays all candidate execution hosts of the same type as the submission host which has a CPU factor less than 18.0.

56 Selection string examples (2 of 2) $ bsub R (ut<0.50 && ncpus==2) \ (ut<0.75 && ncpus==4) job2 Select a candidate execution host the CPU utilization is less than 0.50 and the number of CPUs is two, or the CPU utilization is less than 0.75 and the number of CPUS is four. $ bsub R "type==sunsol && swap>300 \ type==hppa && swap>400" task1 Select a candidate execution host where the type is SUNSOL and more than 300 MB of available swap or where the type is HPPA and has more than 400 MB of available swap.

57 Order string Allows candidate execution hosts to be sorted according to the value of resources. The first index is the primary sort key, the second is your secondary sort key, and so on. Hosts are ordered from best to worst on the given index or indices. If defined at the job and queue level, job level takes precedent. The default order string used is order[r15s:pg].

58 Order string examples (1 of 2) $ bsub R "select[type==any && swp>=300 && \ mem>500] order[mem]" job1 Order the candidate execution hosts from the highest to lowest amount of available memory. $ lsload R "select[type==local && cpuf < 18.0] \ order[cpuf] Order the candidate execution hosts from highest to lowest CPU factor. If the order is not specified, order of the candidate execution hosts will use the default order string of [r15s:pg].

59 Order string examples (2 of 2) $ bsub R select[ut<0.8 && mem>200] \ order[r1m:ut:-mem]" small_mem_job.sh Order the candidate execution hosts from lowest to highest one minute run queue length, from lowest to highest CPU utilization, and from lowest to highest amount of available memory.

60 Resource usage string Specifies resource reservations for jobs on execution hosts. Ignored when running interactive tasks. By default no resources are reserved. If rusage is defined at the job level and queue level, the job level takes precedence. Keywords duration and decay can be used. If a job can run with more than one rusage string, it is possible to specify multiple strings with an OR operator and have Platform LSF pick the first one that matches.

61 Resource usage string examples (1 of 3) $ bsub R "select[type==any && swap>=300 && \ mem>500] order[swap:mem] \ rusage[swap=300,mem=500]" job1 On the selected execution host, reserve 300 MB of swap space and 500 MB of memory for the duration of the job. $ bsub R rusage[mem=500:app_lic_v2=1 \ mem=400:app_lic_v1.5=1]" job1 Job will use 500 MB with app_lic_v2, or 400 MB with app_lic_v1.5. Resource reservation is ignored for interactive tasks (that is: lsload, lsrun).

62 Resource usage string examples (2 of 3) $ bsub R select[ut<0.50 && ncpus==2] \ rusage[ut=0.50:duration=20:decay=1] job2 On the selected execution host, reserve 50% of cpu utilization and linearly decay the amount of cpu utilization reserved over the duration of the period. $ bsub R "select[type == SUNSOL && mem > 300] \ rusage[mem=300:duration=1h]" job3 On the selected execution host, reserve 300 MB of memory for one hour.

63 Resource usage string examples (3 of 3) $ bsub -q reserve R rusage[res1=4,tmp=1000] Job1 % bjobs lp Job <215>, User <john>, Project <default>, Status <PEND>, Queue <reserve>, Command <job1> Thu Jul 24 07:12:16: Submitted from host <delpe07>, CWD </home/john>, Requested Resources <rusage[res1=4, tmp=1000]>; Thu Jul 24 07:12:21: Reserved <1> job slot on host <delpe07>; Thu Jul 24 07:12:21: Reserved <581> megabyte tmp on host <581M*delpe07>; Thu Jul 24 07:12:21: Reserved <2> res1

64 Span string Specifies the locality of a parallel job. Supported options: span[hosts=1] which indicates that all processors allocated to this job must be on the same execution host. span[ptile=n] which indicates that up to n processors on each execution host should be allocated to the job. span[ptile=![,hosttype:n] uses the predefined maximum job slot limit in lsb.hosts (MXJ per host type/model) as the value for other host model or type, other then those host type is specified. When defined at both job-level and queue-level, the job-level definition takes precedence.

65 Span string examples $ bsub n 16 R "select[ut<0.15] order[ut] \ span[hosts=1]" parallel_job1 All processors required to complete this job must reside on the same execution host. $ bsub n 16 R "select[ut<0.15] order[ut] \ span[ptile=2]" parallel_job2 Up to two CPUs per execution host can be used to execute this job therefore at least eight execution hosts are required to complete this job.

66 Same string Used to specify that all processes of a parallel job must run on hosts with the same resources. The parallel job scheduler plugin must be installed to use this option. When defined at both job-level and queue-level, both requirements are combined to allocate processors. Any static resource can be specified.

67 Same string examples $ bsub n 64 R "select[type==sgi6 type==sol7] \ same[type]" parallel_job1 Run all parallel processes on the same host type, either SGI6 or SOL7, but not both. $ bsub n 64 R "select[type==any] \ same[type:model]" parallel_job2 Run all parallel processes on the same host type and model.

68 Multiple resource requirements Support for multiple resource requirement strings (-R) options. The administrator can easily change resource requirements. In job level submission: bsub -R "select[swp > 15]" -R select[hpux] order[r15m] -R rusage[mem=100] R order[ut] -R same[type] -R rusage[tmp=50:duration=60] -R same[model] myjob Platform LSF merges the multiple -R options into one string and selects a host that meets all of the resource requirements. The number of -R option sections is unlimited. Up to a maximum of 512 characters for the entire string per R option.

69 Support multiple resource requirements (1 of 3) Merge job level, app level, and queue level resource requirement strings. Queue level define: RES_REQ = rusage[r1=v1 R2=V2 R3=V3 R4=V4] Job level resource requirements: # bsub -R "rusage[r5=v5:duration=60] Expected merge result: rusage [ R5=V5:duration=60, R1=V1 R5=V5:duration=60, R2=V2 R5=V5:duration=60, R3=V3 R5=V5:duration=60, R4=V4 ]

70 Support multiple resource requirements (2 of 3) Multiple R options support commands: bsub bmod Supports order, rusage, select, and same. Multiple span clauses are not supported.

71 Support multiple resource requirements (3 of 3) Example: Queue level resource definition: RES_REQ: rusage[swp=5000 mem=100] Job level resource definition: bsub -R "rusage[tmp=50 mem=50:duration=60]" -q test job1 Job <2> is submitted to queue <test_resreq> # bjobs l 2 Job <2>, User <usr1>, Project <default>, Status <RUN>, Queue <test>, Command <job1> Tue Apr 10 03:37:52: Submitted from host <lsfhost1>, CWD </>, Requested Resources <rusage[tmp=50 mem=50:duration=60]>; Tue Apr 10 03:37:58: Started on <lsfhost2>, Execution Home </home/usr1>, Execution CWD </>, Execution rusage <[tmp=50.00,mem=100.00]>; RESOURCE REQUIREMENT DETAILS: Combined: select[type == local] order[r15s:pg] rusage[tmp=50.00:swp= t mp=50.00:mem= mem=50.00:duration=1h:decay=0,swp= mem=50.00:duration=1h:decay=0]

72 Resources requirements priorities If application level resource requirements are not defined, job-level and queue-level resource requirements are merged. If application-level resource requirements are defined, joblevel requirements usually take precedence. Specifically: The select sections from the job, application profile, and queue must all be satisfied. The job-level order section overrides the application profile section, which overrides the queue-level section. The job-level rusage section takes precedence. Any rusage requirements defined in the application profile that are not already specified at the job level are then merged. Any queue-level requirements are then merged with that result. The job-level span section overrides the application profile span section, which overrides a queue-level section. The same section from the job, application profile, and queue must all be satisfied.

73 Platform LSF cluster query commands Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 7.0

74 Unit objectives After completing this unit, you should be able to: View cluster, resource and queue information View job information including resource usage, history and output View host and user groups

75 Platform LSF cluster query commands UNIX/Linux manual pages exist for all commands # man commandname LIM commands lsid lsinfo [-l][-m -M][-r][-t][resource_name...] lshosts [-w -l][-r "res_req"][host_name]... lsload [-l][-n -E][-n num_hosts][-r "res_req"] [host_name...] lsmon [-N -E][-n num_hosts][-r "res_req"][-i interval] [-L file_name][host_name...] Batch commands bhosts [-e -l -w][-x][-x][-r "res_req"][host_name host_group]... bmgroup [-r][-l][-w][host_group...] bqueues [-w -l -r][-m host_name -m all][queue_name...] busers [use rname... user group... all] bugroup [-l][-r][-w][user_group ]

76 View cluster and master names Display LSF version number, cluster name, and master host # lsid IBM Platform LSF Standard , Feb Copyright International Business Machines Corp, US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. My cluster name is cluster01 My master name is host1.ibm.edu Display load sharing configuration information # lsinfo RESOURCE_NAME TYPE ORDER DESCRIPTION r15s Numeric Inc 15-second CPU run queue length mem Numeric Dec Available memory (Mbytes) ncpus Numeric Dec Number of CPUs ndisks Numeric Dec Number of local disks mg Boolean N/A Management hosts TYPE_NAME UNKNOWN_AUTO_DETECT DEFAULT LINUX MODEL_NAME CPU_FACTOR ARCHITECTURE Intel_EM64T x15_6789_intelrxeon

77 Server host configuration information Display static resource information about hosts. # lshosts HOST_NAME type model cpuf ncpus maxmem maxswp server RESOURCES host1.ibm.e X86_64 Intel_EM G 5.7G Yes (mg) host2 X86_64 Intel_EM G 5.7G Yes () host3 X86_64 Intel_EM G 5.7G Yes () host4 X86_64 Intel_EM G 5.7G Yes () Display detailed static resource information about a particular host # lshosts -l host1 HOST_NAME: host1.ibm.edu type model cpuf ncpus ndisks maxmem maxswp maxtmp rexpri server nprocs ncores nthreads X86_64 Intel_EM64T G 5.7G 5799M 0 Yes RESOURCES: (mg) RUN_WINDOWS: (always open) LOAD_THRESHOLDS: r15s r1m r15m ut pg io ls it tmp swp mem

78 Server host load information Display load information for hosts # lsload HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem host1.ibm.edu ok % M 5.7G 3.5G host2 ok % M 5.7G 3.5G host3 ok % M 5.7G 3.5G host4 ok % M 5.7G 1.1G Display load information without truncation (including disk I/O) # lsload -l HOST_NAME status r15s r1m r15m ut pg io ls it tmp swp mem host2 ok % M 5.7G 3.5G host3 ok % M 5.7G 3.5G host4 ok % M 5.7G 1.1G host1.ibm.edu ok % M 5.7G 3.5G Display current load information with periodic updates # lsmon Hostname: host1.ibm.edu Refresh rate: 10 secs HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem host2 ok % M 5.7G 3.5G host3 ok % M 5.7G 3.5G host1.ibm.edu ok % M 5.7G 3.5G host4 ok % M 5.7G 1.1G

79 Batch server host information Display hosts and their static and dynamic resources # bhosts HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV host1.ibm.edu ok host2 ok host3 ok host4 ok Display detailed resource information about a particular host # bhosts -l host1 HOST host1.ibm.edu STATUS CPUF JL/U MAX NJOBS RUN SSUSP USUSP RSV DISPATCH_WINDOW ok CURRENT LOAD USED FOR SCHEDULING: r15s r1m r15m ut pg io ls it tmp swp mem slots Total % M 5.7G 3.5G 2 Reserved % M 0M 0M - LOAD THRESHOLD USED FOR SCHEDULING: r15s r1m r15m ut pg io ls it tmp swp mem loadsched loadstop

80 Queue information Display information about queues # bqueues QUEUE_NAME PRIO STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SUSP priority 43 Open:Active night 40 Open:Active chkpnt_rerun_qu 40 Open:Active normal 30 Open:Active interactive 30 Open:Active idle 20 Open:Active Display detailed information about queues # bqueues -l normal QUEUE: normal -- For normal low priority jobs, running only if hosts are lightly loaded. This is the default queue. PARAMETERS/STATISTICS PRIO NICE STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SSUSP USUSP RSV Open:Active Interval for a host to accept two jobs is 0 seconds SCHEDULING PARAMETERS r15s r1m r15m ut pg io ls it tmp swp mem loadsched loadstop SCHEDULING POLICIES: NO_INTERACTIVE USERS: all HOSTS: all

81 User and user group information Display user information By default busers will display the user information for the current user Specify all to see all user and group information # busers all USER/GROUP JL/P MAX NJOBS PEND RUN SSUSP USUSP RSV default user user usergroupa Display user group member information # bugroup GROUP_NAME usergroupa usergroupb USERS user2 user3 UnixAdminGrp NISgroup NTgroup