Fixed Bugs for IBM Platform LSF Version

Transcription

1 Fixed Bugs for IBM LSF Version Release Date: July 2013 The following bugs have been fixed in LSF Version since March 2013 until June 24, 2013: Date The full pending reason cannot be displayed when using badmin diagnose for a job array with a job array ID (not a job array element). Component mbschd Customers cannot see the full pending reason when diagnosing a whole job array Date A job s resource usage is not equal to the sum of the resource usage for each execution host. Component sbatchd Resource usage information is incorrect Date In Windows, all job processes would exit but the job would still occasionally show running status in bjobs if JOB_INCLUDE_POSTPROC=Y. Component sbatchd.exe Windows Job status is wrong, preventing customers from performing further operations on that job and reducing cluster usability.

2 Date mbschd memory leaks when many jobs spend a long time pending and the following conditions occur: LSB_TIME_RESERVE_NUMJOBS and LSF_HPC_EXTENSIONS are configured in lsf.conf RESOURCE_RESERVE_PER_SLOT is configured in lsb.params Component mbschd schmod_reserve.so mbschd memory leaks and uses up customer host memory Date elim.gpu reports an incorrect value after setting the compute mode to exclusive. Component elim.gpu nvjob UNIX / Linux gpu scheduling is incorrect Date The slot limit defined for users in the lsb.resources exclude list is incorrect if a user does not exist in lsb.users. Component mbatchd A user s job cannot be dispatched properly.

3 Date Permission errors occur when LSF handles.msgs directories without the no_root_squash property of the NFS export directory. Component mbatchd Many useless files are left in system and waste space Date If backfill and reservation features are enabled and too many jobs need to be backfilled, mbschd takes a long time to try to backfill the jobs that have reserved resources. Component schmod_reserve.so mbschd responds slowly Date When LSB_CHECK_JOB_PID_REUSE_ON_REBOOT=Y in lsf.conf, jobs stay in running status after the execution host restarts. Component sbatchd Jobs are always in running status and occupy slot(s).

4 Date Jobs fail on a Windows host if the execution directory contains a space. Component sbatchd.exe Windows Customers cannot run jobs on a Windows host if the execution directory contains a space Date If a large scale, long-running and cpu/network intensive parallel job is run by the pam-ts framework, then occasionally a pam res action timeout error occurs and the job is killed, even though the job was started properly. Component pam The job is terminated prematurely even though it was started properly Date blaunch job does not exit when one of its execution hosts fails. Component res Part of the job does not run and there is no indication to make the user aware of this.

5 Date LSB_SUB_COMMANDNAME has no slash before special characters while sourcing LSB_SUB_PARM_FILE in esub. Component bsub libbat.a libbat.so liblsf.a liblsf.so Customers cannot source LSB_SUB_PARM_FILE because spooled commands are not correctly parsed in the LSB_SUB_COMMANDNAME variable Date When LSF_TS_REG_TASK_GROUP=y in lsf.conf, the job hangs if the leading task is signaled but other tasks are alive. Component res TaskStarter Linux Customers cannot run their tests. The behavior is different from standalone POE Date When one execution host is rebooted or hung, the blaunch process hangs in the head node. Component res Jobs hang.

6 Date The hostsetup script should allow dynamic hosts to install system startup scripts. Component hostsetup UNIX The hostsetup does not support dynamic hosts Date egroup works with badmin hstartup, but cannot print hosts after using badmin hrestart and badmin mbdrestart. Component daemons.wrap The host cannot be used in egroup and puts jobs in lost_and_found queues Date melim log does not go to syslog if LSF_LOGDIR is undefined. Component melim Linux Melim log will go to /tmp if LSF_LOGDIR is undefined.

7 Date bkill cannot kill a rerunnable job. The job is continually requeued. Component mbatchd Customers cannot kill a rerunnable job Date The user has a running job with a set runtime limit. Before the runtime limit is reached, the user increases the runtime limit with bmod. However, this modification is removed after restarting mbatchd and lim, and the job is killed. Component mbatchd The job is terminated with the wrong runtime limit after restarting mbatchd and lim Date When using bsub -b to specify a past time, the turnaround time is calculated from the time specified by the user to the end time instead of from the submit time. Component bacct Customer's reporting program is impacted since bacct reports the job's turnaround time as very long even though it only ran for a few minutes.

8 Date If a customer defines GSLA and slot_share in the same queue and the queue uses up its portion of the slot_share, the reservation for jobs according to GSLA does not take effect. Component mbschd schmod_default.so Jobs cannot reserve resources as expected Date Jobs fail when the customer appends to the %PATH% environmental variable and then submits a job using only the executable name. The execution host does not use the submission host's %PATH% value. Component sbatchd.exe Windows Jobs fail when the execution host does not use the submission host's %PATH% value Date The -N option for brsvadd removes the system reservation for hosts in the unavailable state. Component mbatchd Cannot use brsvadd N for system reservation.

9 Date In an LSF multihead environment, some jobs with SLA stay in the pending state when the cluster is busy and there is a mismatch of resource allocation from EGO. Component mbatchd lib2vemkd.so Some jobs with SLA stay in the pending state until the cluster is restarted Date Support for parameter LSF_FORWARD_USER_TGT is required for the Kerberos feature. Component mbatchd, sbatchd, res, bsub, libbat.so, libbat.a LSF_FORWARD_USER_TGT is not supported Date When a job is submitted with a swap requirement, the job remains in pending status even though bhosts shows there are enough resources on the host for the swap requirement. Component mbatchd Linux/UNIX Jobs cannot run even with enough swap resources on the host.

10 Date bsub incorrectly considers \&\& in an execution command line to be "&&", which makes it impossible for customers to run jobs when they intentionally use the \&\& operator. Component bsub UNIX Jobs fail when the execution command line contains \&\& Date Poor scheduler performance occurs under the following conditions: 1. The cluster contains many exclusive hosts and most of the jobs have span[ptile] in the resource requirement. 2. Jobs have span[ptile] in the resource requirement and are pending on hosts in a guaranteed host pool. 3. If the backfill and reservation features are enabled and too many jobs need to be backfilled, mbschd takes a long time trying to backfill the jobs that have reserved resources. Component mbschd schmod_reserve.so schmod_default.so schmod_parallel.so Job dispatch rate is low Date Disabling job with LSB_JOB_REPORT_MAIL=N is inconsistent when jobs run using a different login shell. Component sbatchd UNIX Customers cannot disable job for specific users by setting LSB_JOB_REPORT_MAIL=N when jobs are submitted under a different login shell.

11 Date In a mixed cluster, the sbatchd log entries, cwdjob: Cannot access job" and "lsbatchdir(): Cannot access job do not provide useful information. Component sbatchd Useless messages appear in sbatchd log Date Currently mbatchd uses chanenqueue_ to en-queue job status sent to bld, which could create a huge backlog for mbatchd. As such, customers could observe very slow job status updates in bld. This can also enlarge memory consumption in mbatchd, thus further degrading mbatchd performance. One solution is to change chanenqueue_ to chanenqueue2_, which can speed up communication between mbatchd and bld. Component mbatchd Low license usage and inaccurate token usage reports occur when using LSF with LS Date badmin hhist <hostname> shows the history for other hosts instead of the specified host. Component badmin badmin hhist <hostname> shows the wrong hosts.

12 Date $TMPDIR is set and changed which is incorrect behavior when LSB_SET_TMPDIR is not defined. Component sbatchd $TMPDIR is wrongly set which affects customer s applications Date POE crashes if LSB_TASK_GEOMETRY is set and the number of tasks per node are uneven. Component esub.poe esub.tvpoe mpirun.lsf permapi.so Linux Jobs cannot run. POE crashes Date The enhancement for mbatchd log needs more detailed information when using bjgroup, bpost, bread and bstatus. This log enhancement is intended for easier troubleshooting. Component mbatchd Not enough information is provided in the enhanced mbatchd log.

13 Date mbschd performance is poor when a job group limit works with a job array that was modified by bmod. Component mbschd schmod_limit.so Customer s jobs cannot be scheduled properly and LSF throughput is poor Date blaunch fails when the RES port is defined in /etc/services (or NIS) rather than lsf.conf Component blaunch UNIX blaunch fails.

14 Date A very large value for mem or swap produces incorrect output format for lsload. Component blaunch lsload output format is messed up Date If you submit many jobs to queues that specify the span[ptile] resource requirement string and resource reservation, mbschd may become very busy and consume 100% cpu. Component schmod_parallel.so Can not schedule jobs Date Job submission fails if esub.poe is called and a job script with a for loop is called. Component esub.poe Linux Job fails.

15 Date A job exits with code 1 when its full output file path name is longer than 260 characters on Windows. Component sbatchd.exe Windows Jobs fail when their full output file path name is longer than 260 characters Date In daemon wrapper mode, xdr_jobcard() may fail with buffer overflow. In that case, the job cannot start or be cleaned up, depending on when the xdr error happened. Component sbatchd Linux Jobs fail Date When res fails to connect to the NIOS port, it may core dump. Component res res core dump.

16 Date MBD hangs with duplicate event logging enabled when LSB_SHAREDIR becomes unavailable. Component mbatchd Linux MBD hangs Date Hosts are no longer governed by guarantee policies when an Advance Reservation is created on the host. Component mbschd Resources on hosts with Advance Reservations are not reserved for guarantees, and usage of these hosts by a Service Class does not contribute to that Service Class's use of the pool Date lim core dumps on the master candidate host after switching the master host. Component lim lim dies on master candidate hosts.

17 Date blaunches running concurrently in the background cause error messages such as, lsb_launch(): Failed while waiting for tasks to finish. Component res blaunch output contains many error messages Date The Kerberos integration does not work if the host name has upper case characters. Component eauth.krb5 Kerberos integration does not work in the cluster Date Jobs with small run limits and span[ptile=x] should run on reserved hosts with the advance reservation feature enabled. Instead, these jobs remain in PEND state in the cluster. Component schmod_parallel.so Hosts are unusable.

18 Copyright and trademark information Copyright IBM Corporation 2013 U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at