IBM Platform LSF 9.1.3

Transcription

1 IBM Platform LSF Global Product Portfolio Manager, IBM Platform LSF Family 1

2 IBM Platform LSF Family Key Drivers Unceasing demand for Compute Scalability and Throughput Node counts are steadily growing and are becoming more dense From Single core per socket, to 18-20, and 72 is on the horizon From Single job per node now multiple jobs per node Accelerators Need for isolation and containment Job volumes continue to increase Manageability and usability Need to be able to operate efficiently at scale Need to provide useful feedback to the user on when and why Need to be more SLA focused reduce the complexity Power is an issue Cost and limits Data awareness/affinity Secure Collaboration and Remote Visualization Operational Management HPC is business critical 2

3 IBM Platform LSF Family IBM Platform Analytics IBM Platform Application Center Platform MPI IBM Platform RTM Platform LSF IBM Platform Process Manager Hadoop Connector MapReduce Accelerator Data Manager IBM Platform Dynamic Cluster Scheduling Extensions IBM Platform Session Scheduler IBM Platform License Scheduler 3 Docker Connector Elastic Storage (GPFS)

4 IBM Platform LSF Family Releases 0 9.1Major Release Stable Production Branch, Fixes Only Every 4 months x (Jan, May, Sept) Every 4 months x (Nov, Mar, July) Next Major Release Codename : Nevis 0 4 1H2013 2H2013 1H2014 2H2014 1H2015 2H2015 Add-on s follow roughly the same timeline

5 9.1 Highlights LSF 9.1 Scalability & Performance New threaded query subsystem significantly improving performance Scheduling cycle decreased with memory optimisations Usability & Manageability Clearer reporting of resource usage Fast detection of hung hosts/jobs Leverage Linux kernel cgroup for process tracking and accounting Query Diagnostics Kerberos Refresh Scheduling Alternative/time based resource requirements Enhanced SLA scheduling SLA scheduling unit extended to core+memory packages Numerous multicluster enhancements LS 9.1 New Project mode to supplement cluster mode Flexible hierarchal project definition Better handling of parallel jobs where each rank checks out a license Removed the need for LSF restarts when the LS configuration changes PAC 9.1 Increased scalability and performance Configurable information retention periods Support for additional LSF fields Extended Web Services interface Addition remote visualization support DCV, EoD Enhanced integration with Platform Analytics RTM 9.1 Support for GPFS moniitoring Refreshed Alarms module allowing new alarms to be quickly defined Enhanced filtering and drilldown capabilities Support for LSF GSLA and Dynamic Cluster New ELIM templates to allow automatic graphing of LSF ELIM information PPM 9.1 Extended support for user variables and flow controls Flow version control Support for non-lsf systems 4Q2011 1Q2012 2Q2012 3Q2012 4Q2012 1Q2013 5

6 9.1.1 Highlights LSF Usability Ability to customize the default output of bjobs Ability to pass arguments to esubs Manageability Linux OOM Killer support Flexible clean period for DONE and EXITed jobs New Queue Host Limit Aggressive cleanup on host failure Scheduling Cgroup enforcement of cpu affinity and memory use Support preemption & backfill with affinity Enhancements to blaunch performance and reliability Updated HW Support ARMv7 & Cray XC-30 Energy Aware Scheduling (Phase 1) LA Set core/node frequency, Suspend to S3, Energy Reporting PAC Support for DCV session sharing using Support for X.509 authentication Updated browser support: IE, Firefox and Chrome. Support for PAC on PowerLinux Various usability enhancements Product usage tracking for the LSF Family Analytics This release adds formal support for IBM Platform Symphony It includes updates to the data loaders and new Symphony specific workbooks LSF Manageability Ability to set the default order Ability to set env vars in application profiles Scheduling New CPU Affinity New Memory Affinity New IBM PE integration on x86 (Power 3Q) PMPI Various performance enhancements Shared memory DNS caching at startup GPUDirect Non blocking collectives Free PMPI Community Edition 1Q2013 2Q2013 3Q2013 4Q2013 6

7 LSF Highlights LSF Usability & Manageability Updates to DRMAA, Perl and Python API s Support for host based/parallel pre/post exec for node health checks Enhanced Kerberos Support for parallel jobs New ACL based security to control who is allowed to see which jobs Scheduling Compound resource scheduling extended to support a global same[] clause Updates to Nvidia GPGPU and Intel Phi support Block allocations Energy Aware Scheduling including frequency prediction Further blaunch improvements for parallel job launching Updated HW Support zlinux and ARM refresh PMPI Community Edition LS Combine related features into packages for ease of use. Run lmstat through ssh for remote clusters General performance enhancements New LS Basic Edition Support for LSF Advanced Edition PAC Upgrade to IBM Java 7 Various usability and functional enhancements Ability to pass parameters from PAC to Platform Analytics for report generation Preview release of PAC on IBM Websphere SiteMinder Support Analytics Support for Platform Symphony Updates to Tableau & Vertica Updated/refresh workbooks. FNM, Updated Loaders RTM New Heuristic plugin providing a holistic view on cluster health, throughput and job failures. New Grid Host History plugin Enhancements to the License plugin Support for Power Linux PPM Support for SiteMinder Marshalling of flow level stdout/stderr Misc of enhancements Support for Power Linux 1Q2013 2Q2013 3Q2013 4Q2013 7

8 Overview The v9.1.3 release of the Platform LSF Family continues our commitment to delivering the best of breed workload management solutions. There are over 75 customer driven enhancements across the family, including: Enhanced scheduling and improved manageability capabilities in LSF. Support for the Reprise License Manager in LSF, License Scheduler and RTM. Application Center now leverages IBM WebSphere for higher performance, as well as scale out and high availability. Numerous enhancements in Process Manager to support our market focus in Life Sciences and Genomics Processor support IBM Power8 (Linux, AIX), ARM 64-bit (Linux) IO optimizations with GPFS/ROMIO and GPU Direct support in Platform MPI New add-ons and extensions for Hadoop & Data Management 8

9 LSF Highlights LSF Change how exclusive jobs are scheduled and accounted for Discrete slot counts for parallel jobs Enhanced hostfile support Auto purge of orphaned dependent jobs Node Based Advanced Reservations Control of Env Var propagation Improved handling of job control failures Remote proclimit considered in MC forwarding LS Support for non-flexlm licenses (RLM) PMPI Mellanox Connect IB/DCT support to increase scale GPU-Direct RDMA Improved Torus Scale Support (Mellanox ACM) Improved ROMIO/GPFS support (MPI IO optimization) Improved Shmem performance PPM Enhanced flow completion criteria Execute multiple revisions of the same flow Group permissions & Execution Account PAC Migrate to WebSphere Performance, Scale out & HA,. Enhanced field dependencies & hide form sections Job data file actions Online editors RTM Extended License Dashboard and non FlexLM support perfmon support Additional alarming templates Native Phi Support for LSF 9 and PMPI 1Q2014 ARMv8 Support 2Q2014 3Q2014 4Q2014 1Q2015 Connector for Hadoop Power 8 BE Docker Integration MapReduce Accelerator Data Manager Power 8 LE KNL Support

10 Overview of IBM Platform LSF Easier to Manage You can now account for the actual resources allocated (and blocked) by a job, rather than just the resources it requested, providing better visibility into utilization. You can restrict users to submitting jobs of specific sizes improving overall utilization. Orphaned Dependent Jobs can be automatically cleaned up, maintaining performance. New options for controlling how memory limits are applied and enforced. Improved handling of common failures Support for RLM in addition to FlexLM Support for ARMv8, Power8 10

11 Requested and Allocated Resources Many organizations base their fairshare allocations and accounting on the resources the user requested. In the example below, the red job is spanning 8 hosts, and using 4 cores per host. The accounting information will show 32 cores in use, fairshare will be based on 32 cores. Now consider the same job, submitted for exclusive execution. What should the accounting show? 32 or 128? 11

12 Auto Clean up of Orphan Dependent Jobs LSF Job dependencies are a key part of many customers workflows. Consider the simple example below: Done(A) Job A Exit(A) Done(B) Job B Exit(B) Done(E) Job E Exit(E) Job C Job D Job F Job G If A fails and E succeeds then F will run, but B,C,D & G will never run - they are orphans and will remain pending forever with Dependency condition invalid or never satisfied. The user could go back and fix whatever caused A to fail, rerun it, and then B etc would execute. In we have introduced: An administrator option to purge orphan jobs after a grace period this gives the user some time to fix failed jobs or bad dependencies. A user option to delete jobs as soon as they become orphans 12

13 Smart Memory Limit Enforcement The Smart Memory Limit allows a job to use more memory (or swap) than it asked for, as long as there is no contention. 64GB 64GB A overuse Threshold C overuse 32GB 32GB C A B overuse B 13 JobA reserved 32GB of Memory. However, it tries to use 48GB With traditional LSF or cgroup memory enforcement, the job would be killed. Smart Memory Limit would allow this because there is no contention. When there is contention (memory or swap threshold), the worst offender will be terminated. JobB asked for 10GB but it using 16GB JobC asked for 16GB but is using 40GB Thus JobC will be killed

14 Handling Failures MultiCluster PROCLIMIT/TASKLIMIT MultiCluster makes numerous checks when deciding which cluster to forward to. However, one check that wasn t made was whether the remote PROCLIMIT for the queue or application profile would allow the job to run. For example, the local cluster may allow a 256 way Fluent job, but the remote cluster may only allow a 64 way Fluent job. Not checking this in advance resulted in the job being returned to the submission cluster. In this now checked in advanced, resulting in more efficient scheduling. Pre-Execution Failures When a pre-exec repeatedly fails, the job will be suspended and left for the user to fix (if they remember!) In there is now a queue/application option to have the job exit instead. Checking of Job Control return codes Job controls are expected to put the job in the correct state before returning. In some cases this does not happen with the result that the LSF state and real job state are out of sync. To alleviate this, we will now check the return codes of the job control if zero, then the action has succeeded, if non-zero it has failed and LSF will not change the job state. 14

15 Overview of IBM Platform LSF Easier to Use New online help for bsub and bjobs helping users help themselves. Increased flexibility when creating advance reservations. You can now create them per node (rather than per slot), and give access to a group of users (rather than individual users) If you have to run the same job repeatedly on the exact same hosts and layout, you can now specify exact job topology via a host file. Fine grained control of how the job environment is propagated. 15

16 Platform LSF Family Add-on s 16

17 IBM Platform Application Center Platform Application Center s principal focus is ease of use, and is driven by enterprise IM clients and by Life Sciences Easier to Manage Greater performance, scalability and high availability through replacing Apache Tomcat with IBM WebSphere. Enterprise Identity management with support for CA SiteMinder Support for RHEL7 (with MariaDB) on x86 and Power 8 Easier to Use Support for online file editing of job submission scripts and data files removes the need to repeatedly download-edit-upload-test. Support for concurrently viewing multiple output files, avoiding the need to view them sequentially. Easier to Customize Simplified submission templates through extended support for field dependencies, layout persistence, and custom file actions. Users can be restricted from creating copies of existing templates enabling a more controlled and standardised environment. 17

18 Why move to IBM WebSphere? Improved web portal performance and scalability Support more concurrent users Better performance (30-60% faster) More stable under heavy load than tomcat WebSphere Liberty Profile is embedded in Application Center No requirement to purchase WebSphere seperately. Provides enterprise level HA Leverage WebSphere clustering for scale out and fail over Leverage prebuilt components from WebSphere For example: SiteMinder support 100 concurrent users, 1.4 million active jobs, 4 million historical jobs 5k hosts, mysql 5.6; PAC servers: Redhat 6.4, 2CPU, 32 Cores, 128G memory Submit/Query 18 test case: 100 clients, each client submit 10 jobs continually, each job with 1 file attachment(size 1k)

19 Superior Repeatable Performance = Better Business Outcomes Benchmarks designed to simulate workloads typically found in high throughput environments at small scale (128 cores) 12x less work 150x less work 6x less work 19

20 Summary of IBM Platform Process Manager Product direction is strongly influenced by the adoption of workflow in Industrial Manufacturing and in Life Sciences Easier to Use Faster time to results through array element to element dependencies. Reduce wasted computation through enhanced flow completion criteria. S1 S2 S3 Reduce errors through the use of parameter files. Users now have control over which version of a particular flow is triggered allowing them to compare different versions or verify old results. Easier to Manage Ability to delegate flow submission, and management. Support for automatic archiving of flow execution history enabling 20 traceability over much longer periods of time.

21 Platform RTM Platform RTM s direction is driven by a broad range of clients spanning many industries. The main foci for V9.1.3 were: Make it easier to get started with Platform RTM Platform LSF performance monitoring Collect additional Platform LSF data Support RLM Platform RTM also adds support for POWER8 on RHEL6.5/SLES

22 RTM 9.1.3: Guided Help Page level help for each link in the Console tab Link to Getting Started 22

23 RTM Cluster Performance Tracking To aid in the tracking of cluster performance, the output of badmin perfmon and badmin showstatus can be collected periodically. RTM can also periodically submit and track a test job for test job for end to end performance monitoring. 23

24 Platform MPI Two Key Performance Enhancements: ROMIO Support for Elastic Storage ROMIO is a parallel I/O library for MPI applications Now also available IBM Parallel Environment (PE) Significantly enhanced performance for I/O intensive applications. GPU-Direct RDMA New NVIDIA functionality Delivers significantly enhanced performance for GPU enabled applications MBytes/sec Message Size 24

25 New Add-on s and Extensions Data Manager for LSF Docker Integration Hadoop Connector MapReduce Accelerator for LSF 25

26 Integration with In short, Docker is a lightweight Linux container technology built on top of LXC and cgroup. Think of it as a Virtual Machine, without the overheads or size of a VM, that installs a complete application stack, on demand. All environment/library/distro dependencies can be encapsulated within the container so two applications that would have conflicting library requirements can now share the same host 26

27 Integration with Hot topic in big data for installing applications on demand in the cloud. Significant interest in Life Sciences, Software Development and other industries for encapsulating workflows and installing all components on demand [gecko1]$ cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.2 (Santiago) [gecko1]$ bsub m gecko1 -Is -a "docker(panxun/bwa)" /bin/sh Job <402> is submitted to default queue <interactive>. <<Waiting for dispatch...>> <<Starting on gecko1>> $ hostname Docker_SL-402 $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu LTS" LSF OpenBeta & Whitepaper 27

28 28

29 29

30 For further information, please contact: Bill McMillan Global Product Portfolio Manager IBM Platform Computing LSF Family of Products Tel:

31 31