Minerva User Group 2018

Transcription

1 Minerva User Group 2018 Patricia Kovatch Bhupender Thakur, PhD Francesca Tartaglione, MS Dansha Jiang, PhD Eugene Fluder, PhD Hyung Min Cho, PhD Lili Gai, PhD Jan 18, 2018

2 Outline Welcome and general comments 2017 Accomplishments 2017 Minerva usage 2017 outages and known issues Survey results and discussion Q&A Road map for 2018 Compute and storage upgrade OS upgrade and package rebuild Demeter data science cluster Cloud services such as VM, Spark and containers (Docker/Shifter) Documentation and tutorial sessions Q&A; Floor walk 2

3 Welcome and general comments

4 2017 Accomplishments GPFS upgrade Upgraded Minerva file system from release 3.5 to 4.1 which introduced improvements and bug fixes. IBM ESS storage installation and data migration Installed 6 PB of IBM Elastic Storage Server to provide additional and faster storage to Minerva users and started users data migration to the new pool. TSM upgrade Successfully upgraded TSM software from v6.1 to v8.1, as well as tape library and tape drives firmware of our archive storage system for better performance and reliability. New database server deployment and data migration Installed and configured a new database server (7.3 SSD disk space, Centos 7.3, MariaDB v10.1). Migrated the users and their data from the old system that was reaching EOL. New compute nodes including high-mem node and GPU nodes Installed and configured a new high-mem node ( GHz, 1.7 TB memory). Job tested and now in production. Installed and configured a new GPU node which equipped with 4*P100 Nvidia GPUs. Job tested on this node and will open to users in 01/

5 2017 Accomplishments - continued Minerva OS upgrade Started the preparations for the cluster operating system update from CentOS 6.3 to CentOS 7 (OS image setup, packages rebuild, testing). Consulting services Provided consulting for users private nodes purchases. User support Continued to support Minerva users through ticketing system (closed more than 2,304 tickets in 2017) and in person meetings. Packages installation Installed more than 100 packages to satisfy users requests and needs (1,299 total packages and growing). Accomplished a new round of allocations Allocated 228 users projects for a total of 2.6 PB on BODE and 3.7 PB on ESS storage. New storage purchase New Flash file system 260 TB for metadata and small files. New ESS file system 3.7 PB to be added to the existing storage. Collaboration accounts for groups 5

6 2017 Minerva Usage

7 2017 Minerva usage summary Accounts Number of new users 422 Number of active users in Number of total users Number of project groups 1,735 (600 external users) 267 Compute Number of jobs run 28,672,579 CPU-hours utilized 61,208,294 hours Storage High-speed storage used Archival storage used 5.1 PB (77% utilization) 6.8 PB (13.7 PB total including offsite copy) System Number of maintenance sessions 3 planned / 4 unplanned (99% uptime) 7

8 Jobs and CPU-hours break down by compute resource: Compute # Jobs CPU-hour Utilization Manda 14,709,127 28,183,931 88% Mothra 8,369,565 11,467,197 93% BODE 8,994,398 7,636,607 79% ,215 2, ,166 28,672,579 61,208,294 Hi-memory node GPU nodes Total: 91% 20% increase in compute cycles used compared to

9 CPU-hours break down by project Avner Schlessinger Gabriel Hoffman Rui Chang Marta Filizola Hardik Shah Gaurav Pandey Eimear Kenny 9

10 Job Mix 10

11 High-speed storage (Orga) What Minerva s storage processes on a daily basis: # of reads and writes Storage FLASH (Small File/Metadata) Total (GSS+DDN+FLASH) Size Reads/Day Writes/Day Reads/day Writes/day 150 TB ~1.05 billion ~0.6 billion PB ~1.8 billion ~1 billion ~500 TB ~175 TB 11

12 Storage usage breakdown by research project Milind Mahajan Production Bioinformatics Group 22,532,449 files Gabriel Hoffman Common Mind Psychiatry 170,553,026 files Alison Goate LOAD 30,060,193 files Bin Zhang Adineto 52,056,583 files Milind Mahajan Genomics Core Facility 208,067,608 files Gabriel Hoffman Psychiatry 19,167,326 files Lisa Edelman Hardik Shah (Robert Sebra) Pacbio SmrtPortal Genetics and Genomic Sciences 26,342,148 files CCSQXT 19,398,482 files Michael Marin MMAAAS 344,530 files Total: 5.1 PB used by 1,452,320,208 files from 267 projects. Bin Zhang AMPADWGS 131,182 files 12

13 Archive storage Current archive storage usage Archived data 6.85 PB (270,461,256 files) Total data with offsite copy PB Number of tapes used 8,673 Statistics of 2017 Archived data in ,834 TB # of archive operations 23,065 # of archive users 106 Retrieved data 7.5 % # of retrieve operations 7,917 # of retrieve users 63 13

14 Archival storage occupancy by users 14

15 2017 Outages and Known issues

16 2017 Minerva Outages Summary Planned / unplanned Date Duration GPFS issue, long waiters and file system not accessible Unplanned 04/26/2017 ~6 h ESS recovery groups lost causing long waiters and file system issues Unplanned 08/02/2017 ~4 h ESS node communication problems causing long waiters and file system issues Unplanned 08/05/2017 ~4 h Two ESS nodes went arbitrating and several servers in unknown state Unplanned 08/08/2017 ~3 h The new version of GPFS ( ) was installed on all Minerva nodes and storage servers Planned 08/22/2017 ~8 h Upgraded GPFS on GSS, worked on GOLD and DB1 Planned 09/19/2017 ~8 h Generated new keys for GPFS, tested ESS system on orga compute nodes, upgraded minerva2, added fixes for LSF Planned 11/07/2017 ~8 h TSM upgrade which includes, Firmware upgrade on tape library, OS upgrade on hpctsm1, tsm upgrade on both server nodes and client nodes. Planned 11/06/2017 ~7 days Total 99% uptime 16

17 LSF Job Scheduler Overload in Dec Caused by certain pipelines submitting several (tens of thousands) short jobs (~minutes) query the LSF too often. "Batch system concurrent query limit exceeded... retrying in 1 second(s)." - Resolved by communicating with users to understand and optimize the script. We also put a limit on a couple of user s jobs. 17

18 2017 Survey Results

19 Survey results and discussion We asked four questions: Q1: Overall, how satisfied are you with queue structure, compute and storage resources? Q2: Please rate current software environment (packages and services such as database, web, container etc): Q3: Please rate your satisfaction with operations (documentation, ticket system, responsiveness of staff,...): Q4: General suggestions for service improvement. We received 42 responses and 52 comments. 19

20 Survey results and discussion Q1: Overall, how satisfied are you with queue structure, compute and storage resources? Comments: Short of compute resources: long waiting time for job to start; queues are crowded. Storage issues: Not enough space in Scratch and longer time before deletion; file-system not stable. Better queue support: the compute resource can be occupied by one group only; different queues for many small jobs and high CPU jobs. Memory usage exceed in login nodes; need more interactive nodes. 20

21 Survey results and discussion Q2: Please rate current software environment (packages and services such as database, web, container etc): Comments: Any omissions in the packages are quickly installed, but need to keep up-to-date. OS upgrade with services such as containers, VM, Hadoop, Spark and HDFS. 21

22 Survey results and discussion Q3: Please rate your satisfaction with operations (documentation, ticket system, responsiveness of staff,...): Comments: Lack of documentation and examples, documentations are outdated. Training for new users. 22

23 Summary Thank you for your feedback! Actions we took and/or are taking: Upgrading storage and compute nodes. Upgrading the OS and rebuilding packages. Deploying the Demeter data science cluster for the community Making cloud services available such as VMs, Spark and containers (Docker/Shifter). Updating the documentation and increasing the number of tutorial sessions. Please continue to provide feedback at any time via our ticketing system or talk to us directly. 23

24 2018 Minerva Roadmap

25 Drivers for our 2018 Roadmap Hardware reaching end of life (no vendor will support anymore) Storage component Flash (Stores all metadata for the file system) 50 TB, EOL in Q1. Ramsan (Stores tiny files which see frequent use) 100 TB, EOL in Q1. GSS (The default location for all data and sees highest use) 2.9 PB, unsupported in Q1. DDN10k (The oldest and the smallest storage tier) 1.2 PB EOL in Q1. Computes Manda compute nodes, EOL in Q4. Outdated OS/Software stack Minerva OS and base packages CentOS 6.4 is out of support. Older base compilers and libraries block newer packages. Newer MPI/OFED stack will require dependent packages to be rebuilt. Feedback from user survey 25

26 Storage and file system upgrade plan Client side: GPFS upgrade from v3.5 to v4.2 Manda Mothra/Bode Login nodes Login nodes Mgmt nodes Mgmt nodes Compute nodes Compute nodes Infiniband (IB) network 1G Ethernet network File system side: Orga ESS Data Pool (6PB) Data Transfer GSS Data Pool (3PB) DDN 12K Data Pool (4PB) DDN 10K Data Pool (1.5PB) Flash Data Pool (160TB)

27 Current file system upgrade status Storage upgrade completed in 2017: After the last town hall we setup a separate GPFS cluster to: Identify with the vendors (IBM and Mellanox) what went wrong during the first integration. Solve the remaining issues (mainly related to the Infiniband network). Extensively test and stress the ESS file system to be sure the new storage is stable. Currently: We gradually integrated the ESS storage into orga. We started the data migration from the DDN10 (~790 TB) and GSS (~2.2 PB) to the ESS pool (currently migrated ~27% of data). In future: Complete the current file-system upgrade (ETA 2018/03). Remove GSS and DDN10k from Orga after data migration completion. Upgrade client cluster and file system cluster to GPFS v

28 New storage purchase and integration In Dec 2017, we purchased: New Flash file system 260 TB for metadata and small files. New ESS file system 3.7 PB to be added to the existing storage. Integration schedule: We will work on a plan with IBM to integrate the new ESS storage and replace the metadata tier. ETA: April-May 2018 (need to coordinate with IBM). Storage tier after the upgrade Size NewFlash (Stores all metadata for the file system) 132 TB NewRamsan (Stores tiny files which see frequent use) 132 TB ESS (IBM storage where the current data is being migrated to) 5.1 PB NewESS 3.7 PB Total storage available after the upgrade: 8.8 PB 28

29 Compute upgrades for 2018 Login nodes New set of public login nodes to replace overloaded minerva2. Compute upgrade Newer racks of high density nodes to replace Manda compute partition. We will tailor the nodes based on usage (more memory? more cores? more GPUs? ) Infrastructure upgrade: New management and infrastructure nodes. New network switches. Dedicated data transfer nodes. Additional VMs for web services. A migration path to EDR+ Infiniband fabric. If you have special requirements for compute resources, please let us know, we are happy to work with you. 29

30 OS upgrade and package rebuild Centos 7 image is being tested on new login nodes: data2. A new set of load-balanced login nodes will be made available to users for CentOS 7 testing. We are currently rebuilding the software packages on a test cluster, which we will open for early user testing in 02/2018. The new pool of compute nodes (due this year) will be installed with Centos 7 and newer packages. We will update the rest of the computes partition, i.e., manda, mothra and bode, after the integration of the new compute nodes. We need your help to test the new OS and packages! 30

31 Demeter data science cluster The Demeter cluster has been run by Hammerbacher Lab and is being transferred to HPC. It is a Hadoop cluster with 80+ nodes, 3 PB storage space as an hdfs file system. the Demeter cluster is not under maintenance from any vendor, so when compute nodes or storage fail, they cannot be replaced. We are in the process of upgrading it to make it available to you to determine the demand for this type of cluster. We plan to open it for early user testing in April If you have an Apache/Spark pipeline which can benefit from this resource, let us know! 31

32 Cloud services - VM, containers, and other services Containers We will provide containers via Shifter or Singularity on Minerva by April This will be part of the new OS stack on the compute nodes. We will be adding additional GPU nodes (with newer GPUs) to support containers. VMs We are considering migrating Minerva user websites to multiple VMs to support multiple package requirements (would like your feedback). We are considering support for user VMs going forward (would like your feedback). Database We will provide support for MongoDB user databases going forward. 32

33 Documentation and training For most recent announcement and updates: Four training sessions will be offered this year. Two sets of training sessions in spring and fall. Topics include Introduction to Minerva and LSF job scheduler. Introduction to Scientific Computing BSR1015 is now a two credit course with an expanded lab. It s being taught this spring by Anthony Costa, PhD. Documentation update on the website ( Join our mail-list: hpcusers@mssm.edu Follow us on Minerva user group meetings will be scheduled as needed. We will refresh the website by March. We will add newer pages/articles as needed over next 3-6 months. We will provide additional training material (including slides) online. We are also considering a new ticket system/knowledge base (please give us feedback). 33

34 HPC Roadmap Finish ESS migration New ESS and Flash storage deployment CentOS 7 upgrade and package rebuild Cloud technology deployment and testing New Compute nodes Demeter data science cluster reinstall and deployment Compute node early testing Documentation and web page updates Spring training Fall training 34

35 Question and comments

36 Thank you!