Using ITaP clusters for large scale statistical analysis with R. Doug Crabill Purdue University

Transcription

1 Using ITaP clusters for large scale statistical analysis with R Doug Crabill Purdue University

2 Topics Running multiple R jobs on departmental Linux servers serially, and in parallel Cluster concepts and terms Use cluster to run same R program many times Use cluster to run same R program many times with different parameters Running jobs on an entire cluster node

3 Invoking R in a batch mode R CMD BATCH vs Rscript Rscript t.r > t.out Rscript t.r > t.out & # run in background nohup Rscript t.r > t.out & background, even if logout # run in Can launch several such jobs with & s simultaneously, but best to stick with between 2 and 8 jobs per departmental server. Invoking manually doesn t scale well for running dozens of jobs

4 Launching several R jobs serially Create a file like run.sh that contains several Rscript invocations. The first job will run until it completes, then the second job will run, etc. Rscript t.r > tout.001 Rscript t.r > tout.002 Rscript t.r > tout.003 Rscript t.r > tout.004 Rscript t.r > tout.005 Rscript t.r > tout.006 Rscript t.r > tout.007 Rscript t.r > tout.008 Do NOT use & at the end or it could crash server

5 Creating the run.sh script programatically Extra cool points for creating the run.sh script using R > sprintf("rscript t.r > tout.%03d", 1:8) [1] "Rscript t.r > tout.001" "Rscript t.r > tout.002" "Rscript t.r > tout.003" [4] "Rscript t.r > tout.004" "Rscript t.r > tout.005" "Rscript t.r > tout.006" [7] "Rscript t.r > tout.007" "Rscript t.r > tout.008" > write(sprintf("rscript t.r > tout.%03d", 1:8), "run.sh")

6 Invoking run.sh sh run.sh # run every job in dorun.sh one at a time # run every job in dorun.sh one at a time, and keep running even if logout nohup sh run.sh & nohup xargs -d '\n' -n1 -P4 sh -c < run.sh & # run every job in run.sh keeping 4 jobs running simultaneously keep running even if logout

7 Supercomputers and clusters Supercomputer = A collection of computer nodes in a cluster managed by scheduler Node = one computer in a cluster (dozens to hundreds) Core = one CPU core in a Node (often 8 to 48 cores per node) Front end = one or more computers used for launching jobs on the cluster PBS / Torque is the scheduling software. PBS is like the maître d, seating groups of various sizes for varying times at available tables, with a waiting list, a bar, reservations, bad customers that spoil the party

8 ITaP / RCAC clusters Conte was fastest supercomputer on any academic campus in the world when built June Has Intel Phi coprocessors Carter Has NVIDIA GPU-accelerated nodes Hansen, Rossmann, Coates Scholar uses part of Carter, for instructional use Hathi Hadoop Radon Accessible by all researchers on campus Anecdotes <queue patriotic music>

9 More info on radon cluster then select Computation->Radon Sub- Cluster Number of Nodes Radon-D 30 Processo rs per Node Two 2.33 GHz Quad- Core Intel E5410 Cores per Node Memory per Node Read the User s guide on left sidebar Intercon nect Theoretic al Peak TeraFLO PS 8 16 GB 1 GigE 58.2

10 Logging into radon Get accounts on the RCAC website previously mentioned or ask me Make an SSH connection to radon.rcac.purdue.edu From Linux (or Mac terminal), type this to log into one of the cluster front ends (as user dgc): ssh X radon.rcac.purdue.edu l dgc Do not run big jobs on the front ends! They are only used for submitting jobs to the cluster and light testing and debugging

11 File storage on radon Home directory quota is ~10GB (type myquota ) Can be increased to 100GB via Boiler Backpack settings at Scratch storage of around 1TB per user. This directory differs per user, and is accessible by the $RCAC_SCRATCH environment variable: radon-fe01 ~ $ cd $RCAC_SCRATCH radon-fe01 /scratch/radon/d/dgc $ All nodes can see all files in home and scratch

12 Software on radon List of applications installed on radon can be found in the users guide previously mentioned The module command is used to load software packages for use by the current login session module avail available module load r #See the list of applications # Add R Must be included as part of every R job to be run on the cluster

13 PBS scheduler commands qstat # See list of jobs in the queue qstat u dgc # See list of jobs in the queue submitted by dgc qsub jobname.sh # Submit jobname.sh to run on the cluster qdel JOBIDNUMBER # delete a previously submitted job from the queue

14 Simple qsub submission file Qsub accepts command line arguments, or embedded comments that are ignored by the script, but honored by qsub. radon-fe01 ~/cluster $ cat myjob.sh #!/bin/sh -l #PBS -l nodes=1:ppn=1 #PBS -l walltime=00:10:00 cd $PBS_O_WORKDIR /bin/hostname radon-fe01 ~/cluster $ qsub myjob.sh radon-adm.rcac.purdue.edu The JOBID of this particular job is

15 Viewing status and the results Use qstat or qstat -u dgc to check job status Output of job goes to myjob.sh.o Errors from job goes to myjob.sh.e Inconvenient to collect the results from a dynamically named file like myjob.sh.o Best to write output to a filename of your choosing by writing directly to filenames of your choosing in your R program or directing the output to a file in your job submission file

16 Our first R job submission Say we want to run the R program t.r on radon: summary(1 + rgeom(10^7, 1/1000 )) Create R1.sh with contents: #!/bin/sh -l #PBS -l nodes=1:ppn=1 #PBS -l walltime=00:10:00 cd $PBS_O_WORKDIR module add r Rscript t.r > out1 Submit using qsub R1.sh

17 Let s do that 100 times Using our R1.sh file as a template, create files prog001.sh through prog100.sh, changing the output file for each job to be out.nnn. In R: s<-scan("r1.sh", what='c', sep="\n") sapply(1:100, function(i) { s[6]=sprintf("rscript t.r > out.%03d", i); write(s, sprintf("prog%03d.sh", i)); }) write(sprintf("qsub prog%03d.sh", 1:100), "runall.sh") Submit all 100 jobs by typing sh x runall.sh Generating files using bash instead (all on one line): for i in `seq -w 1 100`; do (head -6 R1.sh; echo "Rscript t.r > out.$i") > prog$i.sh; echo "qsub prog$i.sh"; done > runall.sh

18 Coupon collector problem I want to solve the coupon collector problem with large parameters but it will take much too long on a single computer (around 2.5 days): sum(sapply(1:10000, function(y) {mean(1 + rgeom(10^8, y/10000))})) The obvious approach is to break it into 10,000 smaller R jobs and submit them to the cluster. Better to break it into 250 jobs, each operating on 40 numbers. Create an R script that accepts command line arguments to process many numbers at a time. Estimate walltime carefully!

19 Coupon collector R code t2.r, read arguments into args, process each args <- commandargs(true) sapply(as.integer(args), function(y) {mean(1 + rgeom(10^8, y/10000))}) Can test via: Rscript t2.r # Change reps from 10^8 to 10^5 for test Generate 250 scripts with 40 arguments each s<-scan("r2.sh", what='c', sep="\n") sapply(1:250, function(y) { s[6]=sprintf("rscript t2.r %s > out.%03d", paste((y*40-39):(y*40), collapse=" "), y); write(s, sprintf("prog%03d.sh", y)); }) write(sprintf("qsub prog%03d.sh", 1:250), "runall.sh")

20 Coupon collector results Output is in the files out.001 through out.250: radon-fe00 ~/cluster/r2done $ cat out.001 [1] [8] [15] [22] [29] [36] It s hard to read 250 files with that stupid leading column. UNIX tricks to the rescue! sum(scan(pipe("cat out* colrm 1 5"))) # works for small indexes only sum(scan(pipe("cat out* sed -e 's/.*]//'"))) # works for all index sizes Cha-ching!

21 Using all cores on a single node When running your job on a single core of a node shared with strangers, some may misbehave and use too much RAM or CPU. Solution is to request entire nodes, and fill them with just your jobs so you never share a node with anyone else. Job submission file should include: #PBS -l nodes=1:ppn=8 Forces PBS to exclusively schedule a node for you. If it is a single R job, you are using just one core! Must use xargs or a similar trick to launch 8 simultaneous R jobs. Only submit 1/8 th the jobs.

22 All cores example one #!/bin/sh -l #PBS -l nodes=1:ppn=8 #PBS -l walltime=00:30:00 cd $PBS_O_WORKDIR module add r Rscript t3.r >out1 & Rscript t3.r >out2 & Rscript t3.r >out3 & Rscript t3.r >out4 & Rscript t3.r >out5 & Rscript t3.r >out6 & Rscript t3.r >out7 & Rscript t3.r >out8 & wait

23 All cores example two #!/bin/sh -l #PBS -l nodes=1:ppn=8 #PBS -l walltime=00:30:00 cd $PBS_O_WORKDIR module add r xargs -d '\n' -n1 -P8 sh -c < batch1.sh Where batch1.sh contains (could be > 8 lines!): Rscript t3.r >out1 Rscript t3.r >out2 Rscript t3.r >out3 Rscript t3.r >out4 Rscript t3.r >out5 Rscript t3.r >out6 Rscript t3.r >out7 Rscript t3.r >out8

24 Thanks! Thanks to Prof. Mark Daniel Ward for all his help with the examples used in this talk! URL for these notes: (copy and paste works poorly with PDF!)