Introduction to W&M. Feb 12, 2019 Eric J. Walter Manager of HPC

Transcription

1 Introduction to W&M th Laura Hild System Engineer Feb 12, 2019 Eric J Walter Manager of HPC Jay Kanukurthy Applications Analyst

2 Using HPC OR Important topics: Getting an account Linux command line / text editors Logging into the clusters Selecting software How to use filesystems efficiently How to use to use the batch system Compiling / installing your own applications HPC ticket system mail: hpc-help@wmedu Using HPC

3 Intel Broadwell /sciclone/home20 bora 11T bo01-bo55 hi01hi07 20/ /224 Opteron Seoul /sciclone/home10 27T sciclonewmedu vx29vortex vx01vx28 vx36 12/432 Intel Nehalem hu01hurricane hu12 147T 8/384 storm Opteron Magny-Cours ha01ha38 wi01wi26 8/304 16/416 ice01 48 ice02 32 statwmedu /sciclone/scr10 breeze /sciclone/scr20 twister 86T 73T Opteron Santa Rosa rain W&M W&M W&M W&M /sciclone/pscr wh01wh52 Opteron Shanghai W&M W&M W&M W&M Cluster Diagram /sciclone/scr30 ra115-ra186 17T tornado or W&M VPN /sciclone/data10 360T tempest 4/288 Intel Knights Landing / Xeon Phi meltemi mlt001-mlt100 64/6400 /sciclone/scr-mlt 74T mistral proj gale

4 Logging In Must use Secure Shell client (SSH) Linux / Mac built-in (terminal) Windows SSH Secure Shell Client / PuTTY [ewalter@particle ~]$ ssh hurricanesciclonewmedu Password: Last login: Tue Feb 2 13:57: from particlehpcwmedu William and Mary Information Technology / SciClone Cluster 1 [hurricane] Questions to ask yourself: Am I on or off campus? If you are off-campus log into statwmedu first using your W&M username and password Is my username the same as my current machine? If it is different use: ssh <username>@<host><domain> Do I need graphics? If yes, then log in with -X

5 Start-up Files & Modules Start-up files control global and cluster specific settings Using start-up files is the recommended way to select software for particular cluster $PLATFORM variable: 11 [vortex] echo $PLATFORM rhel6-opteron This means that startup is controlled by cshrcrhel6-opteron for vortex

6 Modules and Start-up 12 [vortex] module avail /usr/local/modules/modulefiles acml/531/gcc mongo/2612 acml/531/open64 mpfr/313 acml/531/pgi mpi4py/200/gcc-520 acml-int64/531/gcc mpi4py/300/gcc-520 acml-int64/531/open64 multiwell-2017/gcc acml-int64/531/pgi mvapich2-ib/22/gcc-520 acml-mp/531/gcc mvapich2-ib/22/intel acml-mp/531/open64 mvapich2-ib/22/pgi-177 acml-mp/531/pgi mvapich2-tv-ib/22/gcc-520 acml-mp-int64/531/gcc mvapich2-tv-ib/22/intel acml-mp-int64/531/open64 mvapich2-tv-ib/22/pgi [vortex] echo $PLATFORM rhel6-opteron 14 [vortex] less cshrcrhel6-opteron # # Environment configuration for RHEL 6x / Opteron environment # # # # VORTEX # module load isa/seoul module load espresso/61/intel isa controls which software stack is used 16 [vortex] ml Currently Loaded Modulefiles: 1) modules 4) isa/seoul 2) maui/r156-gres 5) intel/2017 3) torque/6111 6) openmpi/211/intel 7) espresso/61/intel Must start new session (log out and back in) to load new start-up modules - online module help

7 Files and I/O There are multiple files-systems available Some are for ongoing / project storage data, homexx Some are for running jobs scrxx, pscr Only data/homexx backed up Use local scratch when possible (every node has some) Users are responsible for using disk space responsibly!! Misuse of file-systems can disturb other jobs and can result in a administrative action! Don't use data10 for job writes or large job reads Use scratch space for jobs (90 day purge)

8 Transferring Files hpcwmedu Using HPC Files & Filesystems Transferring Files Each file-system has a server that runs it For direct access you are STRONGLY encouraged to use the recommended node eg : Logged into bora ; cd d into data10 ; transfer off-site Do this from tempest since files won't have to hop through bora to get off-site /sciclone/home20 11T /sciclone/data10 360T bora tempest Remote site Good 1 hop Bad 2 hops Globus - We have direct endpoints for most file-systems

9 Sharing Files & Folders : Permissions I see for more information 33 [hurricane] pwd /sciclone/home04/ewalter print working directory list directory 34 [hurricane] ls -ld results drwx ewalter hpcf 512 Jun 16 14:37 results directory user group other r-read w-write x-execute associated associated user group **directories need to be x to be entered/passed through 35 [hurricane] ls -l total 3 -rw ewalter -rw ewalter -rw ewalter results list files in directory hpcf 194 Jun 16 14:37 wwdat hpcf 194 Jun 16 14:37 yydat hpcf 194 Jun 16 14:37 zzdat

10 Common Permissions Tasks I Change the permissions of a directory chmod: [hurricane] ls -ld VASP drwx ewalter hpcf 512 Apr 25 [hurricane] chmod go+rx VASP 26 [hurricane] ls -ld VASP drwxr-xr-x 4 ewalter hpcf 512 Apr 27 [hurricane] chmod o-rx VASP 28 [hurricane] ls -ld VASP drwxr-x--- 4 ewalter hpcf 512 Apr 29 [hurricane] chmod g-rx VASP 30 [hurricane] ls -ld VASP drwx ewalter hpcf 512 Apr VASP VASP VASP VASP Change the permissions of a directory and everything under it chmod -R: 32 [hurricane] ls -ld VASP drwx ewalter hpcf 512 Apr 33 [hurricane] ls -l VASP total rw ewalter hpcf rw ewalter hpcf drwx ewalter hpcf [hurricane] chmod -R 4 Apr Apr Aug 2014 VASP potpaw_lda52targz potpaw_pbe52targz 6 00:01 vasp53 g+rx VASP 35 [hurricane] ls -ld VASP drwxr-x--- 4 ewalter hpcf 512 Apr 36 [hurricane] ls -l VASP total rw-r ewalter hpcf rw-r ewalter hpcf drwxr-x--- 2 ewalter hpcf Apr Apr Aug 2014 VASP potpaw_lda52targz potpaw_pbe52targz 6 00:01 vasp53 Use this command if you want to allow group access to your home, scrxx, and dataxx directories

11 Common Permissions Tasks II How do I change the initial permissions that files and folders are given when created: You need to edit your cshrc file in your home directory and add: umask umask 077 umask 027 umask 022 files get -rw folders get drwx files get -rw-r----- folders get drwxr-x--- files get -rw-r--r-- folders get drwxr-xr-x HPC default umask is 077 What groups are I in?: groups 52 [hurricane] groups ewalter ewalter : hpcf wmall hpcstaff www seadas vasp sysadmin wm hpcadmin wheel hpsmh My primary (default) group is hpcf and the rest are secondary Change a group associated with a file or directory: chgrp 54 [hurricane] ls -ld project drwx ewalter hpcf 512 Aug 18 20:47 project 55 [hurricane] chgrp hpcstaff project 56 [hurricane] ls -ld project drwx ewalter hpcstaff 512 Aug 18 20:47 project

12 Software and Compilers There are many software packages available on the HPC systems! Ways to find out whether a package is available Check the modules on a particular cluster with: module avail Look at software web page ( Install it yourself hpc-help@wmedu We encourage users to install their own software in their home directory if possible We will also do it for you or at least help, but we get LOTS of request so try not to abuse Compilers: Intel, PGI, GNU MPI libraries: Intel, mvapich2,openmpi mvp2run wrapper for all three with extra functionality (node load checking)

13 Using the Batch System HPC uses Torque (PBS) to schedule and run jobs Nodes are selected via the node type qsub submits the job to the batch system node type 27 [vortex] qsub -I -l walltime=30:00 -l nodes=1:vortex:ppn=12 qsub: waiting for job to start qsub: job ready 1 [vx01] python progpy JobID Interactive job puts you on a node ready to work There are many node types The default node type is simply the sub-cluster name vortex It is also possible to select certain subsets within a cluster or a collection of subclusters x5672 any hurricane or whirlwind node c18b only large memory vortex nodes See online documentation or send to hpc-help@wmedu for more information DO NOT RUN JOBS ON FRONT-END/LOGIN MACHINES!

14 Using the Batch System II You can also submit a batch job which does not run interactively First you must write a batch script: 34 [hurricane] cat run #!/bin/tcsh #PBS -N test #PBS -l nodes=1:x5672:ppn=8 #PBS -l walltime=0:10:00 #PBS -j oe cd $PBS_O_WORKDIR python progpy >& progout #!/bin/tcsh -N -l -j interpret the following in tcsh syntax name of the job job specifications (walltime ; nodespec) combine stderr and stdout cd $PBS_O_WORKDIR cd to where I submitted the job /aout run the job 35 [hurricane] qsub run 148 [vortex] more testo Warning: no access to tty (Bad file descriptor) Thus no job control in this shell tput: No value for $TERM and no -T specified most widely used batch commands qsub submit job qdel delete job qstat list jobs qsu list my jobs

15 Using the Batch System III MATLAB example 107 [hurricane] more run #!/bin/tcsh #PBS -N test #PBS -l nodes=1:x5672:ppn=8 #PBS -l walltime=12:00:00 #PBS -j oe #PBS -q matlab must add -q matlab for matlab jobs load matlab module (if needed) cd $PBS_O_WORKDIR module load matlab redirect stdout and stderr matlab file for stdout and stderr -nodisplay -r "readmatrix" >& OUT 108 [hurricane] head readmatrixm tic %parpool(8) syms a b c d; meshpoints = meshgenerator(); eigfile = fopen('eigfiletxt', 'wt'); count = 1; count2 = 1; %set(0, 'CurrentFigure', 1); %plot3(0,0,0,''); %grid on

16 Where to get help? HPC webpage: HPC ticket system mail: Using the ticket system is useful since it is monitored by three of us WE'RE HERE TO HELP!

17 Q&A