ACEnet for CS6702 Ross Dickson, Computational Research Consultant 29 Sep 2009

Transcription

1 ACEnet for CS6702 Ross Dickson, Computational Research Consultant 29 Sep 2009

2 What is ACEnet? Shared resource......for research computing... physics, chemistry, oceanography, biology, math, engineering, CS,......across Atlantic Canada six clusters ~12 technical personnel plus admin staff

3 Getting Connected Logging in ssh Secure Shell Moving Data Around sftp scp Secure Copy Secure File Transfer Protocol both provided with Mac OS X or Linux ssh -X username@brasdor.ace-net.ca sftp username@placentia.ace-net.ca Windows? Use PuTTY and WinSCP just google for them

4 Hardware Processors: AMD Opterons, 2.4 GHz thru 3.0 GHz Cluster Total Cores RAM/ node Cores/ node Interconnect Best use Brasdor Gb 4 GigE serial Courtenay Gb 4 GigE development Fundy , 16 IB shared mem Glooscap >>248 4 Gb 4, 16 GigE any Mahone Myr MPI Placentia >> , 16 IB, GigE any GigE = Gigabit Ethernet IB = InfiniBand Myr = Myrinet

5 File systems Each cluster has own storage array Move data explicitly between clusters sftp or scp Filesystems: /home NFS, individual, Gb /globalscratch NFS, individual, 238 Gb /nqs NFS, common, multi-tb /scratch/tmp node-local, various sizes See for more

6 Node architectures SunFire X4100 & X2200 2x dual-core Opterons 4-way HP similar SunFire X4600 8x dual-core Opterons 16-way X6440 blades 4x quad-core Opterons 16-way no local disk, just small flash memory coming into service this autumn

7

8 Parallel programming Compilers support OpenMP except Gnu v3 MPI implementation is Open MPI release mpicc is wrapper for PGI pgcc mpicc wraps pgcc,...etc. see for more info

9 MPI 'Hello, world' #include <stdio.h> #include <mpi.h> int main (int argc, char *argv[]) { int rank, size; MPI_Init (&argc, &argv); MPI_Comm_rank (MPI_COMM_WORLD, &rank); MPI_Comm_size (MPI_COMM_WORLD, &size); printf( "This is process %d of %d.\n", rank, size ); MPI_Finalize(); return 0; }

10 Testing 'Hello, world' > mpicc mpihello.c -o mpihello > mpirun -np 2 mpihello This is process 0 of 2. This is process 1 of 2. > Testing on head node restricted to <15 CPU mins Longer jobs must use scheduler Sun Grid Engine 'SGE' (or 'N1GE')

11 Testing the scheduler > cat parscript #$ -cwd #$ -l h_rt=1:0:0 #$ -pe ompi* 4-16 mpirun mpihello > qsub parscript Your job 5691 ("parscript") has been submitted > qstat...

12 Grid Engine commands qsub [options] script submit a job e.g. qsub -l h_rt=3600 runme.sh qstat is my job running? done? qsum * how busy is the cluster? showq * how far am I from the head of the waiting list? * Local scripts, no man pages, no options --- see wiki.ace-net.ca

13 qsub flags script directives Directives in script... #$ -cwd #$ -l h_rt=1:0:0./myprog <file.in >file.out...equivalent to arguments to qsub qsub -cwd script Command line trumps script Right-most (last) takes precedence

14 Job parameters, general -l h_rt=hh:mm:ss hard limit on run time REQUIRED - allows scheduler to plan ahead -l test=true (opt) run on high-availability test nodes must have h_rt 1:0:0 -l h_vmem=memory (opt) memory per parallel slot -cwd (opt) run in current working directory

15 Job parameters, parallel -pe parallel_env n_procs parallel_env one of ompi* openmp n_procs can be range or list, e.g. 4,8,12 -l mx_endpoints=1 only needed at Mahone.ace-net.ca requests Myrinet endpoints one per job slot

16 Basic Scheduling Algorithm Don't interrupt running jobs Assign priorities to waiting jobs * When resources come free: Run highest priority job that can be satisfied by available resources Iterate until all resources are committed or all jobs have been checked Wait for new resources or new jobs * How? See

17 Parallel job starvation Jobs scheduled when resources are available Scheduler tries to fill all available resources Small jobs (B) can starve wide jobs (A) if the small jobs never stop coming: slot slot qsub A Time > B(0) B(2) B(4) B(6) B(8) A B(1) B(3) B(5) B(7) B(9)

18 Solution: Reservation qsub -R y jobscript causes scheduler to reserve resources for highest-priority jobs. qsub -R y A B(0) B(2) A res B(1) B(3) >...but it's costly, so you have to ask explicitly.

19 Two more SGE commands qalter [options] jobid change job parameters between submission and dispatch e.g. qalter -R y jobid qdel jobid

20 Where to go for help man command standard Linux online manual ACEnet User Wiki ACEnet User Guide Research/user support personnel