Fixed Bugs for IBM Platform LSF Version 9.1.3

Transcription

1 Fixed Bugs for IBM LSF Version Release Date: July The following bugs have been fixed in LSF Version between 8 October 2013 and 21 July 2014: Date The preemption calculation was refined for shared resources to improve the preemption performance and throughput of the whole cluster. Component mbschd, schmod_preemption.so Throughput of the cluster is diminished when it takes a long time to get small jobs preempted Date The parameter MAX_EVENT_STREAM_SIZE cannot limit the size of the lsb.status file. After this fix, either the oldest lsb.stream.timestamp file or the oldest lsb.status.timestamp file will be deleted and then a new file will be written when the number for MAX_EVENT_STREAM_FILE_NUMBER is reached., liblsbstream.so The lsb.status file grows to a very large size and causes a storage space issue. When the storage system does not work well, the LSF cluster may become unavailable Date Duplicate data is logged in the lsb.status file which causes a PK violation in PA.

2 Cluster Admin sees many PK violations on Analytics side Date When running a large number of short jobs (for example, sleep 3) on a Windows host, some jobs show as exited even though they ran successfully. Component res.exe Windows jobs run successfully, but LSF reports that some small jobs have exited Date To allow for enabling and disabling the sourcing of LSB_SUB_PARAM_FILE without causing an error, the internal parameter LSB_SUB_ADDITIONAL_REMOVAL_FROM_PARAMFILE (in lsf.conf) is used. When LSB_SUB_ADDITIONAL_REMOVAL_FROM_PARAMFILE is set as Y or y, LSB_SUB_ADDITIONAL should be removed from LSB_SUB_PARAM_FILE and exported as an environment variable. By default, LSB_SUB_ADDITIONAL is not removed and the setting of LSB_SUB_ADDITIONAL_REMOVAL_FROM_PARAMFILE is incorrectly regarded as N or n. Component bsub, mesub A script using LSB_SUB_ADDITIONAL no longer works in LSF Date POE will core dump if a user specifies LSB_PJL_TASK_GEOMETRY in the submission script for a job and the number of task groups in the task geometry is not equal to the number of execution hosts allocated by LSF. Component permapi.so Linux

3 POE jobs run inside an LSF core dump Date When using bsub -M to submit an exclusive job with a memory limit so large that it exceeds the execution host s maximum physical memory, an out of memory (OOM) condition occurs. Component sbatchd Linux\UNIX A host becomes unavailable when the job exceeds the host s memory Date The XDR buffer used by the Master LIM to send resource information to the remote cluster is set to a fixed size. If the number of resources on a cluster is increased so that the resource information size is larger than the buffer, connections between the member clusters will break. In this fix, the buffer size is calculated dynamically based on resource configuration in a cluster. Component lim MultiCluster fails if too many resources are defined in lsf.shared Date Jobs occasionally get stuck with the pending reason New job is waiting for scheduling. This may occur when trying to brun a job when the job is already scheduled. Component mbschd Some jobs are stuck in a pending state until noticed by administrators or users complain to administrators.

4 Date When cgroup accounting features are enabled (set by LSB_MEMLIMIT_ENFORCE=y, LSF_PROCESS_TRACKING=Y, or LSF_LINUX_CGROUP_ACCT=Y in lsf.conf), the job may be terminated by a memory limit (set by bsub -M or MEMLIMIT in lsb.queues). The memory usage accounting incorrectly shows that the job exceeds the memory limit, so the job is terminated. Component sbatchd Linux The job is terminated by MEMLIMIT unexpectedly when cgroup memory accounting is enabled Date When no decay value is defined for the queue, a job submitted with decay=0" will get rejected, even though this is a valid value. For example: bsub R "rusage[mem=300:decay=0]" A job with a valid rusage string is rejected Date mbatchd restarts, replays the events file, and core dumps if you use bsub to submit a new job and the job's resource requirement string (such as order[ ] ) is longer than 511 characters. mbatchd core dumps and the LSF batch system is unavailable.

5 Date If an interactive job is submitted that tries to pass the job's input data to bsub using a pipe ( ), bsub will not forward the input data to the job correctly. For example, if the command used is echo "exit" bsub -Ip tcsh -s, the "exit" string will not be passed to tcsh -s and the job will not exit. Component bsub Linux/Solaris Some interactive jobs do not run Date The LSF/Clearcase integration utility daemons.wrap does not show the actual Clearcase view name (ccview) in the error log. Component daemons.wrap Error messages in the log are not sufficient to diagnose problems Date mbatchd hangs when there are duplicate job events with the same jobid in the lsb.events file. The duplicate events are caused by the failover, for example, when the network is operating erratically. mbatchd does not response and the LSF batch system is unavailable Date

6 When the nofile limit is set to unlimited, LSF daemons take a long time to start and the cluster behaves abnormally. With this fix applied, setting the nofile limit higher than 65535, an INFO level message will be logged for lim, res, mbatchd, and sbatchd stating: The nofile limit is in excess of This may cause performance issues with your cluster. Component lim, res, mbatchd, sbatchd Cluster becomes unusable Date After a socket error occurs on Windows hosts, subsequent jobs cannot run and the following error message appears in the child sbatchd log: sbdchild: starting mode=-s handles=664:600 rcvjobfile: chanread_() failed. A socket operation has failed: Socket operation on non-socket. execjob: Job < > <494999>failed in rcvjobfile_(), Unknown error. Component sbatchd.exe Windows subsequent jobs remain pending on the host and sbatchd must be restarted to process the jobs Date For jobs submitted using bsub n, the bstop command does not work as designed. After using bstop, the bjobs output shows the job in SUSPEND status. However, the job is still running if checked by a system command. Component sbatchd A job is not stopped with bstop, even though bjobs shows the job is in SUSPEND status.

7 Date An application level checkpoint array element in a job array remains pending after a brestart command. Restarting a checkpoint array job fails and the job remains pending Date If JOB_DEP_LAST_SUB (in lsb.params) is set to 0, there are warning messages indicating that the parameter will be ignored. JOB_DEP_LAST_SUB is set to 1 by default. JOB_DEP_LAST_SUB in lsb.params cannot be disabled Date When QJOB_LIMIT is defined in lsb.queues, jobs submitted to the queue occasionally cannot be dispatched even if there are available slots. Component mbschd, schmod_limit.so Some jobs in a queue cannot be scheduled even if there are slots available Date When LSB_MIXED_PATH_ENABLE (in lsf.conf) is set to Y and bsub is used with a long command name, the job may exit with the incorrect status. Component sbatchd

8 Some jobs using long command names may exit with the incorrect status Date During job cleanup, the effective user ID for sbatchd is changed to root. When the user s home directory is mounted from the NFS server and configured without root privilege, the job post script will fail. Component sbatchd Linux.lsbatch/* files cannot be cleaned up Date PREEMPT_JOBTYPE=BACKFILL does not work. When configuring a low priority queue with a job to preempt a backfill job, LSF does not allow the job (which may not have a run limit) to preempt the backfill job because it may delay the start of the job in the high priority queue. An administrator cannot set up a queue that would preempt a backfill queue under certain conditions Date Windows hosts cannot execute esub.bat. Component bsub.exe Windows Cannot use esub.bat on Windows hosts.

9 Date Messages for lsb.acct file rotation and deletion are not logged. No lsb.acct file rotation info message for users Date In MultiCluster environments, bjobs -pac -l jobid does not show the PREDICTEDREMAINTIME column. Component Mbatchd, bjobs The PAC Multi-cluster feature is missing the PREDICTEDREMAINTIME value in the GUI Date When frequently using bhosts to check the host status, it causes the child mbatchd to perform a core dump. Child mbatchd core dumps and does not respond to b* query commands Date Remote jobid information is unavailable in bjobs output. Therefore, Application Center is unable to show the remote jobid of a job in a remote cluster. Component bjobs

10 Application Center cannot show the remote jobid from bjobs output Date Due to a script error, elim.gpfshost cannot be started. Therefore, local host GPFS information (gtotalin and gtotalout) cannot be collected and reported. Component elim.gpfshost Linux elim.gpfshost exits when loading host GPFS information Date When starting dynamic hosts on a particular subnet they should join the cluster automatically and start taking jobs. LSF adds them to the correct host group but they never leave closed_lim status in bhosts, even if all LSF daemons on them are restarted. Component lim Dynamic batch execution hosts cannot be added to a cluster if the package size of the slave lim configuration that needs to send to the master lim is bigger than the MTU Date No error is recorded in the lim log when lim does not have permission to open lsf.conf or lsf.cluster. Component lim lim error message is confusing or absent, making it difficult to debug a problem Date

11 A job that is not configured with application level rerun, but is in a rerun queue (that is, a "rerunnable" job) will no longer be rerunnable after an Administrator runs badmin reconfig. Rerunnable jobs are no longer rerunnable after an Administrator runs badmin reconfig Date When issuing the commands lsadmin resrestart/shutdown or lsadmin resdebug -o, esub is called unnecessarily. Component lsadmin Running lsadmin resrestart/shutdown or lsadmin resdebug -o takes a longer time than necessary Date LSF does not remove the file hostaffinityfile after the job is finished. Component sbatchd Linux The hostaffinityfile temp file must be deleted manually Date When a MPI job is running across nodes, the run limit is not handled correctly with signal SIGUSR2. Component blaunch Linux

12 MPI jobs are killed prematurely instead of a graceful shutdown Date In a MultiCluster lease mode environment, submitting a job from an LSF host to an LSF9.1.2 host causes an mbatchd core dump on LSF all Jobs cannot be submitted from LSF8.0.1 to LSF9.1.2 in MultiCluster lease mode Date Occasionally, the mbschd log contains the error Cannot connect to the mbatchd:, even when all processes are functioning correctly. Error message gives the impression that the cluster is not functioning well Date Value of the PER_PROJECT parameter (in lsb.resources) is limited to 59 characters. Value of PER_PROJECT is limited to 59 characters Date mbatchd generates some duplicate records in the lsb.status file.

13 The lsb.status file grows very quickly and impacts the PA reports due to a PK violation Date When a job is run with both bkill and brequeue at the same time, a memory error occurs causing an mbatchd core dump. mbatchd core dumps and LSF batch system is unavailable Date LSF randomly calculates an incorrect memory usage for MPI jobs. Component sbatchd Linux The correct memory usage for MPI jobs is not available Date The value of PREDICTEDSTARTTIME in the output of bjobs -l -pac jobid contains a redundant <>. Component bjobs Linux Pending jobs cannot be viewed in PAC Date

14 Scheduler performance suffers after using bmod to modify a job group with a limit when there are many jobs in the group. Component mbschd, schmod_limit.so Jobs cannot be dispatched Date Performance of scheduler with resource reservation suffers when there are many pending jobs. Component mbschd, schmod_reserve.so, schmod_parallel.so Scheduler performance degradation Date If there are multiple *.swtag files under $LSF_TOP/properties/version, the multiple FUSERGRP and FXUSR entries cause patchinstall to fail. Component patchinstall Linux\UNIX Patch cannot be installed by patchinstall Date Large memory leak with pim on Mac OS X Component pim Mac OS X Jobs will pend since memory is not available

15 Date When using the bswitch command to switch a running job to another queue, the JOBS limit defined in lsb.resources is bypassed. Linux2.6-glibc2.3-x86_64 A running job in a low priority queue can be switched to a high priority queue and exceed the JOBS limit defined on the high priority queue Date lsadmin ckconfig or reconfig does not show an error message when Begin ClusterAdmins is missing in the lsf.cluster file. Component lim, lsadmin The cluster is unavailable after a restart and an error message indicating the problem is not available Date Problems when using lsmake to build customized Android 4.3 code: (1) Building native Android 4.3 using lsmake is very slow. (2) Cannot build HTC customized Android code on two hosts using lsmake. (3) Slots are occupied by lsmake, leading to low slot usage. The following options have been added to the command lsmake to accommodate this feature: --max-cross-host-level <number> If lsmake enters <number> level, do not distribute the task to other hosts. The default value is a large integer. --no-block-shell-mode Perform "shell" tasks without blocking mode. Without this parameter, blocking mode is used. Component lsmake, lsmakerm

16 lsmake cannot be used and performs worse than gmake when building Android 4.3 code Date When using the badmin reconfig command, the fatal error: Master host <host_name> is not defined in the Host section of the lsb.hosts file is reported if a short host name is defined in LSF_MASTER_LIST (in lsf.conf) and a long host name is defined in /etc/hosts and /etc/sysconfig/network. badmin reconfig quits with a fatal message Date When the DNS is responding slowly, mbatchd responds to b* commands slowly as well, because it attempts to resolve host group names. mbatchd responds to b* commands slowly Date The startup script hostsetup does not work on hosts with Mac OS X 10.8 and higher and LSF daemons do not start up after reboot. Component hostsetup Mac OS X LSF has to be restarted manually after every restart to configure the LSF startup script even though hostsetup has been run Date

17 Job information should be exported as environment variables to eexec, so that eexec does not have to issue bjobs or bhist to get job information. Component sbatchd Poor job submission and execution performance with GOLD integration Date mbatchd generates some duplicate records in the lsb.status file. lsb.status file grows very quickly and impact the PA report feature due to a PK violation Date Using the bsub -cwd option or setting DEFAULT_JOB_CWD in lsb.params can unintentionally overwrite the value of the LS_SUBCWD environment variable. Component sbatchd The original submission directory is lost in the execution environment Date A child mbatchd core dump occurs when ENABLE_EVENT_STREAM is enabled, CONDENSE_PENDING_REASONS is set to ALL, and the system fails to write to the pendingreasons.<cluster_name> file for any reason (for example, there is no space left for the directory set by PENDING_REASONS_TMP_DIR or the file has been deleted).

18 Child mbatchd core dump occurs but there is no observable impact to the LSF cluster Date The mbatchd log includes the message RB_rusageUpdate() Scheduler doesn't scheduled job on host xxx, but SBD reports job usage info from that host when resized jobs are run and hosts are released on completion. Confusing warning message Date A job is put into a pending state with the pending reason Job has a specified start time if both the -b and -R requirements have not been met. However, the job's pending reason will be changed if there is another job pending due to the same -R option. Component mbschd The job pending reason is incorrect Date An mbatchd core dump occurs when upgrading LSF from version to 9.1.2, if sbatchd is restarted first, then mbatchd is restarted., sbatchd mbatchd core dump occurs and LSF batch system is unavailable.

19 Date The operator does not work for shared resources in rusage with parallel jobs even if there are free resources available for a sibling resource requirement. Component mbschd, schmod_parallel.so Parallel jobs with a sibling resource requirement cannot be dispatched even if there are enough resources in the cluster Date A job submitted with a file limit is killed if the information is sent to the res log but it has already reached the file limit. Component res Jobs submitted to LSF cluster are killed incorrectly Date When LSB_KRB_TGT_FWD=Y and LSB_AFS_JOB_SUPPORT=Y are configured in lsf.conf, res causes an FD leak which eventually means there are no FDs available. Component res Linux\UNIX LSF jobs fail to renew AFS tokens Date The error System error 109 has occurred is returned when using the net stop command on Windows to stop res or sbatchd. Component res, lim

20 Windows Scripts cannot be used to manage daemons Date An sbatchd core dump occurs when AIX is upgraded to the TL and service pack and jobs are submitted. Component sbatchd AIX sbatchd core dumps and LSF reports the job is exited Date Using the LSF API to read a streaming file causes a memory leak. Component liblsbstream.so ALL A memory leak causes the application s memory usage to grow exponentially Date A resource requirement is not changed by bmod after a checkpoint job is restarted. Component sbatchd Checkpoint jobs restart with a wrong RES_REQ Date

21 In a mixed MultiCluster, forward environment with MultiCluster lease mode, mbatchd core dumps when it restarts. mbatchd core dumps and LSF batch system is not available Date When DJOB_ENV_SCRIPT (in lsb.queues) is configured and the openmpi_rankfile.sh script is used, the file created by the script cannot be accessed. The openmpi_rankfile.sh script is missing an environment. Component res openmpi_rankfile.sh Linux\UNIX The blaunch/openmpi integration is not complete for the CPU binding Date Before a job is finished, bpeek fails to display the job output and generates an unclear error message: ls_rstat: File operation failed: No such file or directory. Component bpeek An unclear error message is generated by bpeek Date elim.mic contains a hard coded library path to libmicmgmt.so. If libmicmgmt.so is not installed in the default directory, elim.mic exits. Component elim.mic Linux

22 elim.mic exits and no GUP information is collected Date If sbatchd is not responding or is unavailable when mbatchd attempts to send modification to it, sbatchd will never receive the modification information. Component mbschd, sbatchd Changing a running job s run time limit with bmod does not take effect when sbatchd is unavailable Date Loading the Kerberos library fails and causes sbatchd to core dump when running jobs if LSB_KRB_TGT_FWD=Y is set but LSB_KRB_TGT_DIR is not configured in lsf.conf. Component sbatchd Linux\UNIX LSF jobs fail to renew AFS tokens Date After migration to a new host, a job array element remains in a pending state. A long-running checkpoint job fails to start in LSF after migration to a new host Date An mbatchd core dump occurs, caused by a memory overflow, when using bsub -f with %J and the file path is larger than 256 characters.

23 mbatchd core dumps and LSF batch system is not available Date When running patchinstall outside the installation directory, patchinstall fails with the error Error: Unable to access jar file lap/lapapp.jar. Component patchinstall Linux/Unix patchinstall fails to install patches when it is issued outside the installation directory Date LSF does not recognize a user group and gives a warning message when running badmin reconfig if a user group cannot be searched by LDAP due to an LDAP size limitation. Linux\UNIX User group is not recognized by LSF Date Problem with guarantee and preemption features. High priority, guaranteed consumer jobs remain pending even when resource requirements are met. Component mbschd, schmod_default.so Pending jobs do not run even if resource requirements are ment.

24 Date In a MultiCluster environment, the error message MCB_encodeMcbMsg: xdr_func() failed is frequently found incorrectly placed in the mbatchd log. The execution cluster cannot send job usage information to the submission cluster Date If Intel MPI is installed in a directory other than the default /opt directory, Intel MPI jobs using the PAM/TASK starter framework cannot run unless the MPI_TOPDIR is manually changed in the Intel MPI wrapper. Component intelmpi_wrapper Linux The intelmpi_wrapper must be modified manually if the Intel MPI location is not the default directory Date An XDR error message MCB_channelJobDecsnToRemote: MCB_sendDispDecsnToCluster() failed occurs in mbatchd, and the job cannot be forwarded to a remote cluster. Jobs are not forwarding to the remote cluster in MultiCluster lease mode Date When checkpoint jobs are submitted by a script, the command brestart -W does not work and restarted jobs cannot be terminated by RUNLIMIT. Component sbatchd, erestart

25 Linux After running brestart, the job cannot exit and keeps being "Checkpoint initiated" and "Checkpoint succeeded" iteratively Date When executing a job containing multiple tasks, task RES calculates an incorrect XDR size and causes an XDR encoding error. Component res Jobs that are launched by blaunch fail to execute Date In MultiCluster lease mode, an uninitialized variable sometimes causes an mbschd core dump. Component mbschd schmod_default.so Job dispatch fails Date The RES log contains an incorrect spelling for "acquire". Component res, mbschd Typo in an error message of the RES log file Date

26 The command combination bkill -r -J <job_name> does not work as expected. Component bkill bkill -r -J <job_name> does not kill jobs as expected Date In MultiCluster lease mode, a buffer overflow occurs when the lease.state.file is too large, causing an mbatchd core dump. mbatchd core dumps and the LSF batch system is not available Date When the first line of a bsub job script is larger than characters a core dump occurs. Component bsub Linux2.6-glibc2.3-x86_64 bsub core dumps and the job is not submitted Date When mbschd encounters an error on a job, other jobs do not get scheduled and remain stuck. Component mbschd Jobs are pending until badmin reconfig is issued.

27 Date In MultiCluster forward mode, a parallel job submitted with the same RES_REQ is blocked. Component mbschd, schmod_mc.so Jobs are stuck in pending status on submission cluster Date When the lsb_submit() API is used to submit jobs for both a parent and child process, jobs submitted for the child process do not process LSB_SUB_MODIFY_FILE and LSB_SUB_MODIFY_ENVFILE properly. Component liblsf.a, liblsb.so, libbat.a, libbat.so, lsbatch.h, lsf.h esub does not work with the lsb_submit() API Date When a shared resource is configured for the cluster, vemkd reports a warning message: lsfinit: resource <resource_name> is being used by multiple hosts. It cannot be used in a resource requirement expression. Component vemkd Error messages in the vemkd log file with even with correct configuration Date

28 When preemption and guaranteed SLA are both enabled, mbschd will take a long time to finish one scheduling cycle, especially when there are tens of thousands of pending jobs. Component mbschd mbschd performance issue causes low job throughput of the LSF cluster Date A core dump occurs when using bsub to submit a job, and the specified command and its arguments contain multiple quotations. Component bsub Some jobs cannot be submitted Date After enabling LSB_QUERY_ENH (in lsf.conf), if there are many bhosts requests and there is an affinity host in the cluster or affinity is enabled in the cluster, the query child mbatchd will core dump repeatedly. The core dump is caused by a "thread unsafe function" Child mbatchd core dumps when b* query commands do not work Date When using the brestart command with GOLD Integration jobs, the command fails due to some missing job information such as a project name or jobid. Component brestart

29 Linux GOLD integration does not work well with brestart Date When a long pre-execution job is run, but the job is killed before the pre-execution portion is finished, the environment variable LSF_JOB_EXECUSER cannot be retrieved with eexec. Component sbatchd Linux Many GOLD reservations will not be released, if gcharge fails and a job is killed Date Occasionally esub does not work when LSF_TMPDIR is set to a shared file system because the temp file for one job is overriden by another job. Component bsub Incorrect or missing job submission options are set in esub Date mbatchd exits when the license error Unable to contact LIM occurs. Error message is confusing Date

30 A job that is running on another host occasionally fails when the bpeek command is used in a mixed cluster environment. Component bpeek bpeek occasionally does not work Date For exclusive jobs, the reserved job cannot be backfilled. Component mbschd, schmod_reserve.so Short jobs cannot use backfill slots. Copyright and trademark information Copyright IBM Corporation 2014 U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at