An Introduction to Gauss. Paul D. Baines University of California, Davis November 20 th 2012

Transcription

1 An Introduction to Gauss Paul D. Baines University of California, Davis November 20 th 2012

2 What is Gauss? * * 12 node compute cluster (2 x 16 cores per node) * 1 TB storage per node * ~ 11 TB storage on head node * 64GB RAM per node * Total 416 cores (inc. head node)

3 What is Gauss good for? * Running large numbers of independent jobs * Running long- running jobs * Running jobs involving parallel computing * Running large- memory jobs

4 What Gauss is not designed for * Running simple, fast jobs (just use your laptop) * Running interactive R sessions * Running GPU- based calculations

5 Gauss Overview * Create your public/private key (see Wiki for details) * Provide CSE with your public key and campus username (via to help@cse.ucdavis.edu) * Log in to Gauss via ssh: (e.g., ssh X username@gauss.cse.ucdavis.edu) * When you ssh into Gauss, you log in to the head node * If you just directly type R at the command line, you will be running R on the head node * (Please do not do this é!) * To use the compute nodes you submit jobs via SLURM * SLURM manages which jobs runs on which nodes

6 Gauss Structure Head Node SLURM Compute Node 1 Compute Node 2 Compute Node 3 Compute Node Compute Node 12

7 SLURM Basics Important commands to know: * sbatch (submit a job to Gauss) * sarray (submit an array job to Gauss) * squeue (check the status of running jobs) * scancel (cancel a job) Examples (more detailed examples later): squeue # view all running jobs squeue u pdbaines # check all jobs scancel u pdbaines # cancel all of pdbaines jobs scancel # cancel job 19213

8 Resource Allocation on Gauss * The compute resources (CPU s, memory) are shared across all Gauss users. * When users submit jobs, SLURM allocates resources. * You must be sure to request sufficient resources (e.g., cores, memory) for your jobs to run * Resource requests are made when submitting your job (via your sbatch or sarray scripts) * Resources are allocated as per user requests, but strict limits are not enforced * If you use more memory than you requested it can ~massively~ slow down yours (and others) jobs! * To check the memory usage of your jobs you can use the myjobs command (see examples later)

9 Gauss Etiquette * Gauss is a shared resource your bad code can (potentially) ruin someone elses simulation! * Test your code thoroughly before running large jobs * Make sure you request the correct amount of resources for your jobs * Regularly check memory usage for long- running jobs * Be considerate of others!

10 Aside: Linux Basics * To use Gauss you need to know some basic Linux commands (these work on a Mac terminal too) * You should already be, or quickly get, familiar with the following commands: ls, cd, cp, mv, rm, pwd, cat, tar, grep * It helps if you learn how to use a command line editor such as vim or nano. (hint: use vim J )

11 Ways to use Gauss: Example 1 Bob has been given a large dataset by a collaborator and told to analyze it in. The dataset is large and the job will take about 3 days to complete so he doesn t want to use his laptop! Bob can submit the job on Gauss, and keep on working on other stuff in the meantime.

12 Example 1 cont Code files: bob_example_1.r bob_example_1.sh To submit: sbatch bob_example_1.sh

13 Example 1 Code: SLURM script

14 Allocating Resources * How do you know how much memory to request? Run small trial jobs! * Use the myjobs command e.g., pdbaines@gauss:~/examples/example_3$ myjobs Tue Nov 20 10:27:45 PST pdbaines has jobs running on: c0-11 jobs for pdbaines on c0-11 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND pdbaines ? R 10:25 3:12 R pdbaines ? R 10:25 3:12 R pdbaines ? R 10:25 3:12 R pdbaines ? R 10:25 3:12 R VSZ and RSS give a rough indication of how much memory your job is using (in Kb) e.g., The above R jobs are using ~ Mb each.

15 Ways to use Gauss: Example 2 Bob has been given 3 more datasets to analyze by his collaborator (or three new analyses to perform on the same dataset). He just needs to set up the same thing as example 1 multiple times.

16 Example 2 cont Code files: bob_example_2a.r, bob_example_2b.r, bob_example_2c.r bob_example_2a.sh, bob_example_2b.sh, bob_example_2c.sh, To submit: sbatch bob_example_2a.sh sbatch bob_example_2b.sh sbatch bob_example_2b.sh

17 Example 2 Code: SLURM script

18 Ways to use Gauss: Example 3 Bob has developed a new methodology for analyzing super- complicated data. He wants to run a simulation to prove to the world how awesome his method is compared to his competitors methods. He decides to simulate 100 datasets, and analyze each of them with his method, and his competitors methods. This is done using an array job.

19 Example 3 cont * Bob writes an R script to randomly generate and analyze one dataset at a time * He would like to run the script 100 times on Gauss * To do this, he write a shell script to submit to SLURM * Each run must use a different random seed, o/w he will analyze the same dataset 100 times! * He will also need to write an R script to combine the results from all 100 jobs * He will also need a shell script to submit the post- processing portion of the analysis * (Note: I have described this process in detail on the Gauss page of the CSE Wiki: support:systems:gauss)

20 Example 3 cont Code files: bob_example_3.r Bob_post_process.R To submit: sarray bob_example_3.sh sbatch bob_post_process.sh

21 Example 3: SLURM script

22 Example 3: Modified R Code

23 Retrieving your results * To copy results back from Gauss to your laptop: * Archive them e.g., tar cvzf all_results.tar.gz my_results/ Copy them by either using a file transfer (sftp) program, or, just use the command line (Linux/Mac users) e.g., scp myusername@gauss.cse.ucdavis.edu:~/all_results.tar.gz./

24 More Advanced Usage * Gauss can be setup to run parallel computing jobs using MPI, OpenMP etc. * SLURM submit files need to be modified to specify number of tasks, CPU s, memory per CPU etc. * New (free) software can be installed on Gauss at your request by ing help@cse

25 References Pre- requisite Linux skills: * Gauss/SLURM Links: * support:general:security:ssh#moving_and_copying_keys * * * * * *