Introduction to High Performance Computing and an Statistical Genetics Application on the Janus Supercomputer. Purpose

Transcription

1 Introduction to High Performance Computing and an Statistical Genetics Application on the Janus Supercomputer Daniel Yorgov Department of Mathematical & Statistical Sciences, University of Colorado Denver The tutorial scripts are available at Google yorgov links Purpose The purpose of this talk is to help you get running readily available software on Janus. I assume you are familiar how to use Linux shell and you have a local server to start from. There are many ways to approach the same step or task. In my tutorial example I aim for the simplest. 1

2 Outline 1. A Quick Introduction to High Performance Computing (HPC) 2. Meet Janus Supercomputer 3. Tutorial Example: Setting up and Running a GWAS with plink 4. Important Considerations Why Use a Supercomputer? Supercomputers give you the opportunity to solve problems that are too complex for a smaller machine. - Might take hours, days, weeks, months, years - If you use a supercomputer, might only take minutes, hours, days, or weeks Some supercomputer nodes have large amount of memory, e.g. 256Gb, so some jobs that will not fit on a smaller machine can run there 2

3 What is a Supercomputer? Many modern supercomputers are clustersmade up of many smaller computers Each smaller computer is called a node Each node has many cores and shared memory (12 cores on Janus) Within a supercomputer, all these different nodes talk to each other through a communications network (on Janus InfiniBand) At your local server, you can run software directly. At the supercomputer you ask a scheduler to start task(s) on one or many nodes. Job Scheduling On a supercomputer, jobs are scheduled rather than just run instantly at the command line; you need a scheduler. Janus switched to the Slurm scheduler (open source) SLURM = Simple Linux Utility for Resource Management sbatch--qos janus script.sh QoSjanushas a job time limit 24 hours and you can request nodes (up to 5760 cores). 3

4 Janus Supercomputer Janus Supercomputer building; Janus has 16,416 CPU cores - CU Denver contributed funds to Janus. - 17% dedicated utilization of Janus for CU Denver (downtown and AMC). ~16.3 mln. CPU core hours My Credentials I am simply a Janus user, so I share my understanding as such. - I was a big user, ~ 6 mln. hrs., in several hundred hours in the last 2 years - Most core hours: search for a binary code using my own C code, highly optimized to fit in Level 2 cache, hardware 64-bit bitwise operations, algorithmic tricks, etc. - StatGen software like Shapeit and Impute2 Lately, I mostly use a local cluster with 200 cores. 4

5 Janus Supercomputer 1368 compute nodes (Dell C6100) - 12 cores per node so 16,428 total cores - No battery backup of the compute nodes - Infiniband network between nodes TB of usable Lustrebased scratch storage; GB/s max throughput RedHat Linux and currently Slurm scheduler Additional computing resources at Boulder: hi-mem nodes, long-running jobs nodes, GPU nodes Initial Steps to Use RC Systems Apply for an RC account Get a One-Time Password device (OTP) Startup allocation of 50K core hours granted immediately. Additional core hours require a proposal. You may be able to use the existing CU Denver allocation. 5

6 Different Node Types at Janus Login nodes - This is where you are when you log in to login.rc.colorado.edu. - code editing, minor compiling, job submission - I immediately sshto a compile node Compile nodes janus-compile1-4 - No heavy computations, interactive jobs, or long running processes - Script or code editing, compiling - job submission Compute/batch nodes - This is where jobs that are submitted through the scheduler run - Intended for heavy computations Storage Spaces Home Directories/home/user Not high performance; not for direct computational output - 2 GB quota Project Spaces /projects/user Not high performance; can be used to store or share programs, input files, maybe small data files GB quota LustreScratch Filesystem /lustre/janus_scratch/user No hard quotas - Files created more than 180 days in the past may be purged at any time. Copy your results 6

7 Levels of Parallelism 1. No shared memory usage (each program instance is one thread and uses one core and some of the memory at the node). Split your job in many chunks across cores and nodes. 2. OpenMPshared memory parallelism (one program uses many threads and some/all of the cores on a node). You can use all cores and all of the node memory with a single program call. 3. MPI based parallelism (message passing between different instances of the same code on different machines) Tutorial Example 1. A quick GWAS with plink 2. Download plink 3. Download a toy dataset (413 cases and 5209 controls 4,013,155 markers). 4. Run 22 chromosomes in parallel on 2 Janus nodes (11 chromosomes per node) 5. Get the results back to a local server. 7

8 Important Considerations. Failed Runs Check for failed runs. - Runs will fail for a variety of reasons. Make sure you check which runs failed. - Scheduler logs are useful for back-tracking. - However, I would recommend saving (or creating) log files at the end of a job and checking those log files. - Just re-run those jobs that failed. Important Considerations. Memory Memory main constrain at Janus for many applications. ~20.1 Gb per node. - Make sure that the nodes you are choosing have enough memory for all program instances. - That is, adjust how many program instances you will run per node. - Janus nodes have no swap, if you hit the memory limit, job(s) at this node will fail. 8

9 Important Considerations. Memory Example: Impute2 project: ~ 1400 independent runs on different chromosomal chunks. No multi-threading. - Stochastic peak memoryusage(since the marker density varies per chromosomal chunk) - I checked a few random chunks for peak memory usage, e.g.: 6.1, 7.1, 7.3, 8.3, 10.2 in Gb. - Run just 2 Impute2 instances per node - For billing purposes you still use the full node. So 2 threads for 6 hours will be billed 6 hours*12 =72 core hours (although 10 cores were idle at the node). - Still a few Impute2 runs (out of ~1400) failed due to hitting memory limit per node Important Considerations. Memory. You may need to run only one concurrent computation per node due to memory constrains. If memory is still an issue then use different Research Computing type of nodes, like himem or crestone If the program you are using has multi-threading, run just one instance per node requesting 12 threads from the program. All compute cores in that node will be used (hopefully most of the time). 9

10 Important Considerations. Disk Usage - Understand how the disk I/O for your application works. - Have larger input and output files in /lustre/janus_scratch. - Janus Lustreis tuned for large parallel I/O operations. Creating, reading, writing, or removing many small files simultaneously can cause performance problems. - Don t put more than 10,000 files in a single directory. - Avoid ls l in a large directory and avoid shell wildcard expansions ( * ) in large directories. - Users occasionally halt Janus by overwhelming Lustre Acknowledgements Pete Ruprecht, Senior HPC Analyst at Research Computing CU Boulder for numerous advises and sharing slides Research Computing support at rc-help@colorado.edu Jan Mandel Professor and Chair, Department of Mathematical and Statistical Sciences for his advice in setting up some of my computations 10

11 Thank you! 11