Computing with the Moore Cluster

Transcription

1 Computing with the Moore Cluster Edward Walter An overview of data management and job processing in the Moore compute cluster.

2 Overview Getting access to the cluster Data management Submitting jobs (MPI based) Writing parallel applications Submitting jobs (parallelizing serial jobs) Misc optimized libraries

3 Getting Access Login via SSH (secure shell) Kerberos integrated Use your SCS password Uses public key authentication internally Keys automatically created on first login Use blank password for keys Used internally to send jobs to compute nodes Connections from CMU networks only Use VPN or relay through linux.gp from off campus

4 Initial login screen: $ ssh test1@warp It doesn't appear that you have set up your ssh key. This process will make the files: /home/test1/.ssh/id_rsa.pub /home/test1/.ssh/id_rsa /home/test1/.ssh/authorized_keys Generating public/private rsa key pair. Enter file in which to save the key (/home/test1/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/test1/.ssh/id_rsa. Your public key has been saved in /home/test1/.ssh/id_rsa.pub. The key fingerprint is: f6:58:6c:a4:f4:af:49:0a:66:b3:0d:e6:af:b6:f4:9d test1@warp.hpc1.cs.cmu.edu [test1@warp ~]$

5 Data Storage Local home directories /home/username Lustre /lustre/username AFS /afs/cs.cmu.edu/user/username NFS /nfs/servername

6 Data local home directories Path = /home/username Globally visible / available throughout cluster Consistent path on all nodes Shared via NFS over gigabit network 282 GB of shared space available 102 GB currently used 160 GB currently free

7 Data - lustre Path = /lustre/username Globally visible / available throughout cluster Consistent path on all nodes Communication over InfiniBand (16 GB/s) Distributed filesystem (very fast for large files) 12x RAID arrays for data 50 TB of shared space available 16 TB in use 34 TB available

8 Data - AFS Path = /afs/cs.cmu.edu/user/username Visible on frontend only (not on compute nodes) Communication over gigabit Normal AFS quotas and ACLs apply

9 Data - NFS Path = /nfs/servername/ Globally visible / available throughout cluster Consistent path on all nodes Communication over gigabit network NATed connection on compute nodes

10 Data moving data sets From Windows WinSCP - Cygwin - SCP (command line) - From Linux SCP (command line) cp from NFS or AFS rsync (over ssh) tar (piped over ssh) for large data sets

11 Data moving - Windows Login to warp.hpc1.cs.cmu.edu Confirm the server key (one time only) Drag and drop files and folders

12 Data moving - Linux SCP (from another machine) scp file username@warp:/home/username/file cp (from warp) cp /afs/cs.cmu.edu/user/username/file file rsync (from warp) rsync -av -e ssh user@remotehost:/file file tar over ssh (large folders) tar -C /folder -clf -. ssh user@remotehost 'tar -C /folder -xpf -'

13 Submitting Jobs - PBS PBS = Portable Batch System We run the torque version of PBS PBS commands qsub jobfile submits a job (described in the jobfile) showq lists current jobs in the queue and their status qstat similar to showq with less info qstat -f jobnumber gives detailed job info qdel jobnumber delete a job from the queue

14 Job types MPI (parallel execution) Custom written software Uses MPI libraries and/or compilers Serial (manually parallelized) Shell scripts Launch individual programs in parallel

15 MPI job example Requires MPI aware program file Example (run qsub jobfile to submit the job) #!/bin/bash #PBS -l walltime=0:10:0 #PBS -l nodes=4 echo starting mpiexec some-mpi-app echo ending PBS options determine Number of nodes / cores Resource limits (run time etc)

16 MPI another example job #!/bin/csh #PBS -l nodes=16:ppn=8 cd /home/ewalter/example /opt/mpich/intel/bin/mpirun -np 128 -machinefile nodes mpiexample #PBS tells the job manager that job parameters follow -l nodes=16:ppn=8 use 16 compute nodes with 8 cores per node mpirun run this program as an mpi program -np 128 require 128 cpu cores to run this program (wait until they re available) -machinefile nodes only use the machines listed in nodes mpiexample the actual mpi program to be run run qsub jobfile to submit the job to PBS

17 MPI programming hello world #include <iostream.h> #include "/opt/mpich/intel/include/mpi2c++/mpi++.h" int main(int argc, char **argv) { MPI::Init(argc, argv); int rank = MPI::COMM_WORLD.Get_rank(); int size = MPI::COMM_WORLD.Get_size(); cout << "Hello World! I am " << rank << " of " << size << endl; MPI::Finalize(); return 0; }

18 MPI programming hello world pt 2 #include "/opt/mpich/intel/include/mpi2c++/mpi++.h" Include the mpi libraries you want to link/build against Different libraries have different benefits / penalties Libraries should match the compiler your using i.e. They should be built by the same compiler as your code Multiple MPIs available

19 Compilers Available compilers gcc typically gcc or g++ gfortran typically gfortran or f90 Intel C++ typically icc Intel Fortran typically ifort SunStudio 12 (look in /opt/sunstudio12/bin) C, C++ typically suncc Fortran typically sunf90 (depending on fortran version)

20 MPI Libaries OpenMPI (1.2.x) Built with GCC and ICC available now Currently in development (not stale) MPICH(1.2.x) Built with GCC and ICC MPICH 2 (1.0.x) Built with GCC Stale (last update 3 years ago) MVAPICH 2 (coming soon) Optimized for Infiniband Built in support for Lustre (faster IO)

21 Serial Jobs Require shell based logic for data managent Example (run qsub jobfile to submit the job) jobfile: #!/bin/csh #PBS -l nodes=16:ppn=8 pbsdsh -c 16 -u sleeper.sh -l nodes=16:ppn=8 run on 16 nodes and require 8 cores per node pbsdsh PBS application to manage parallel job distribution -c 16 = run on 16 machines -u = run on unique hostnames sleeper.sh = program to run NOTE: PBS is buggy sometimes things don t work as expected (ie -u flag) test and verify expected behavior

22 Serial program built for parallel runs sleeper.sh #!/bin/sh DATE=`date` echo $DATE > $HOSTNAME for i in `seq 1 10` ; do echo $i >> $HOSTNAME sleep 2 done Use shell variables to manage data $HOSTNAME is unique to the executing machine Can also use timestamps etc for output file names Reviewing timestamps in example output files shows parallel execution

23 Serial apps that run in parallel Matlab? Maya - depends on execution flags and renderer Python depending on included libraries Others?

24 Misc libraries for High Performance Math Kernal Libraries Intel (part of Intel compiler suite) look in /opt/intel BLAS look in /usr/lib64 and /usr/lib Sun (part of Sun Studio 12) look in /opt/sun/sunstudio12/??? FFTW and FFTW2 Python SciPY, NumPY Python-ctypes GSL, PyGSL Misc others (look and ask if missing)