Guillimin HPC Users Meeting October 20, 2016

Transcription

1 Guillimin HPC Users Meeting October 20, 2016 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

2 Please be kind to your fellow user meeting attendees Limit to two slices of pizza per person to start please And please recycle your pop cans. Thank you! 2

3 Outline Compute Canada News System Status Software Updates Training News Special Topic Job Workflow and Pipelines 3

4 Compute Canada News (Reminder) 2017 Resource Allocation Competitions More information here: esources/resource-allocation-competitions/ Competition Information Sessions (slides): esources/resource-allocation-competitions/#2017 Process Opens Due Fast Track application (by invitation only) Early October Nov 9, 2016 RRG and RPP full application Early October Nov 24, 2016 Announcement & Implementation of awards Early March 2017 Mid April

5 Compute Canada News At which point should you request Resources for Research Groups (RRG) on Guillimin: More than 2 times the default CPU allocation: RAS: Default priority level: 30 core*years Resource name: Guillimin - phase 1 or phase 2 More than 1 GPU*year or 1 MIC*year RAS: small fraction of GPU or MIC More than 5 times the default storage allocation: RAS: Default project space allocation : 1 TB (up to 5 TB on demand, but not guaranteed) Resource name: DataSTAR, total 2 PB for RAC 2017 Storage space on tape (archive or backup) RAS: only Home directories are saved in backup system Resource name: DataSTAR, Guillimin - phase 2 5

6 Storage Status Sunday Oct. 16: locked files on /gs Caused long delays on /sb and /lb Job scheduling has been stopped Monday Oct. 17: logs investigation Half login nodes were down Tuesday Oct. 18: -> hardware problem, -> action plan Wednesday Oct. 19 (until midnight): Blocked access to all login nodes Stopped GPFS on all servers (including VMs) Proceeded with the hardware replacement Restarted GPFS on all nodes. Validated through tests Opened access to login nodes Job scheduling has been reactivated 6

7 Storage Status Space Management /gs is quite full: 98% used, 70TB free (Oct. 20) Improved scratch cleanup: supports special characters For better space management we continue to progressively migrate cold data from disk to tape Metadata remains on disk Users can still access their files through usual methods, but with an increased latency Storage space is a precious resource - manage it wisely! Delete temporary files, compress large files not frequently accessed, tar many smaller files into collections, 7

8 New Software Installations Check module spider modulename how to load: gperftools/2.5 (originally Google Performance Tools, for tcmalloc) pandas/ python (package for data analysis) JupyterHub/0.4.1-Python (for workshops using Jupyter) QuantumESPRESSO/6.0 (electronic structure/materials modelling) R/3.3.1 (language for statistical computing) OpenFOAM/4.0 (Computational Fluid Dynamics) DOLFIN/ Python (for FEniCS: Finite Elements) PETSc/3.7.3-Python (PDE solvers) SLEPc/3.7.2-Python (Eigenvalue Problem Computations) 8

9 Training News All upcoming events: calculquebec.eventbrite.ca Oct Software Carpentry (U. Montreal) Oct Introduction to Intel Xeon Phi (McGill U.) Nov. 8 - Easy GPU programming with OpenACC (U. Laval) Recently completed: Sept Introduction to Adv. Research Comp. (McGill U.) Oct. 6 - Parallel Programming with MPI (McGill U.) Oct Python performant (U. Laval) Oct Introduction to R (McGill. U.) Oct Introduction to GPU / CUDA (U. de Montreal) All materials from previous workshops are available online: wiki.calculquebec.ca/w/formations/en All user meeting presentations online at 9

10 User Feedback and Discussion Questions? Comments? We value your feedback. Contact us at: Guillimin Operational News for Users Status Pages (all CQ systems) Follow us on Twitter 10

11 Job Workflow and Pipelines October 20, 2016 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

12 Outline Simple Job Script Job Script Generator Using the Job Name Using More Environment Variables Using STDIN Job Dependencies Submitting jobs from a job At the end of a job Sequence of Parallel Matlab Jobs Other tools Job Arrays GNU-Parallel 12

13 How to Scale from a Simple Job Script #!/bin/bash #PBS -l nodes=1:ppn=1 #PBS -l walltime=00:10:00 #PBS -A xyz-123-aa #PBS -N JobTest cd $PBS_O_WORKDIR module load gcc/4.9.1./your_app arg1 arg2 < data/input.txt > output.txt $ qsub script.sh $ cat script.sh qsub 13

14 Job Script Generator Need to submit multiple jobs Multiple data Multiple parameter combinations Main script (job script generator) For loop to generate a script and to submit the script Job script template With PBS options Uses environment variables to execute: A specific task On a specific dataset With specific parameters 14

15 Using the Job Name ### main_script.sh for i in $(ls data/*.dat) do qsub -N $(basename $i.dat) script.sh done ### script.sh #!/bin/bash #PBS... cd $PBS_O_WORKDIR./your_app < data/$pbs_jobname.dat > res/$pbs_jobname.out 15

16 Using More Environment Variables ### main_script.sh for i in $(seq 1 10); do for j in $(seq 31 36); do qsub -o $i_$j.log -e $i_$j.err -v I=$i,J=$j script.sh done done ### script.sh #!/bin/bash #PBS -N IJ_job cd $PBS_O_WORKDIR./your_app -i $I -j $J > res/$i_$j.out 16

17 Using STDIN ### main_script.sh for i in $(seq 1 10); do for j in $(seq 31 36); do echo -e '#!/bin/bash' \ "\ncd \$PBS_O_WORKDIR" \ "\nexport OMP_NUM_THREADS=4 ;" \ "./your_app -i $i -j $j > res/$i_$i.out" qsub \ -l procs=12 -l pmem=5700m -l walltime=03:00:00 \ -A qzx-813-aa -N IJ_job -o $i_$j.log -e $i_$j.err done done 17

18 Job Dependencies Useful when a workflow needs different amounts of resources for different steps JID=$(qsub cleandata.sh) LIST="$JID" for i in $(seq 1 10); do TID=$(qsub -v I=$i -W depend=afterok:$jid process.sh) LIST="$LIST:$TID" done qsub -W depend=afterok:$list genresults.sh 18

19 Submitting jobs from a job ### Near the end of the job script./app arg1 arg2 > output.log if [ $? -eq 0 ]; then qsub nextjob.sh else echo Current job failed because... fi 19

20 Sequence of Parallel Matlab Jobs ### In jobscript.sh submitnextjob() { currindex=$(echo $1 grep -oe '[0-9]{3}$') nextindex=$(printf '%03d\n' $(echo $currindex + 1 bc)) # If less than 16, submit next job after 5 seconds if [ "$nextindex" \< "016" ]; then nextjobname=$(echo $1 sed -e "s/[0-9]\{3\}$/$nextindex/g") sleep 5 echo qsub -N $nextjobname jobscript.sh fi } submitnextjob $PBS_JOBNAME & matlab < input.m > output.log ### Submit first job: qsub -N case001 jobscript.sh 20

21 Job Arrays #!/bin/bash #PBS -j oe #PBS -o outputfiles/ #PBS -t cd $PBS_O_WORKDIR/folder$PBS_ARRAYID for i in $(seq 1 10) do IND=$(echo "10 * $PBS_ARRAYID + $i" bc) echo Global task $IND / 6000 cd task$ind mpiexec -n 128../../app data.dat > out.log cd.. done 21

22 GNU-Parallel #PBS -l procs=12 #... cd $PBS_O_WORKDIR myfunc() {./app < data/$1 > res/$(basename $1.dat).log echo "done simulation on $1" } export -f myfunc module add gnu-parallel parallel myfunc {} ::: $(ls data/*.dat) 22

23 Conclusion No more than 3000 jobs in queue The pipeline must be robust against failed job: make sure to identify which jobs have failed #PBS -t 345,456,567, qalter -W depend=afterok:$jobid03 \ $JOBID02 For help: guillimin@calculquebec.ca 23