Practical Introduction to

Transcription

1 Outline of the workshop Practical Introduction to Bart Oldeman, Calcul Québec McGill HPC Theoretical / practical introduction Parallelizing your serial code What is OpenMP? Why do we need it? How do we run OpenMP codes (on the Guillimin cluster)? How to program with OpenMP? Program structure Basic functions Examples 1 2 Outline of the workshop Practical exercises on Guillimin Login, setup environment, launch OpenMP code Analyzing and running examples Modifying and tuning OpenMP codes 3 Parallelizing your serial code Models for parallel computing (as an ordinary user sees it...) Implicit Parallelization minimum work for you Threaded libraries (MKL, ACML, GOTO, etc...) Compiler directives (OpenMP) Good for desktops and shared memory machines Explicit Parallelization work is required! You tell what should be done on what CPU Low-level option for shared memory machines: POSIX Threads (pthreads) Solution for distributed clusters (MPI: shared nothing!) 4

2 OpenMP Shared Memory API Open Multi-Processing: An Application Program Interface for multi-threaded programs in a shared-memory environment. Consists of Compiler directives Runtime library routines Environment variables Allows for relatively simple incremental parallelization. Not distributed, but can be combined with MPI (hybrid). 5 Shared memory approach Thread 0 Private memory Shared memory Thread 1 Private memory Most memory is shared by all threads. Each thread also has some private memory: variables explicitly declared private, local variables in functions and subroutines. 6 OpenMP: fork/join model Master Thread Serial Region Parallel Region Serial Region Parallel Region Serial Region Worker Threads Fork Join Fork Join Master + Workers = Team Implementations use thread pools so worker threads sleep from join to fork. 7 What is OpenMP for a user? What is OpenMP for a user? OpenMP is NOT a language! OpenMP is NOT a compiler or specific product OpenMP is a de-facto industry standard, a specification for an Application Program Interface (API). You use its directives, routines, and environment variables. You compile and link your code with specific flags. History: version 1.0 (1997), 2.5 (2005), 3.0 (2008), 3.1 (2011), 4.0 (2013). Different implementations : GCC (since version 4.2), Intel, PGI, Visual C++, Solaris Studio,... 8

3 Basic features of OpenMP program Include basic definitions (#include <omp.h>, INCLUDE omp lib.h, or USE omp lib). Parallel region declared by a directive of the form #pragma omp parallel (C) or!$omp PARALLEL (Fortran), declaring which variables are private. Optional: code only compiled for OpenMP: use OPENMP preprocessor symbol (C) or!$ prefix (Fortran). 9 Fortran PROGRAM hello!$ USE omp lib IMPLICIT NONE INTEGER rank, size rank = 0 size = 1 Example: Hello from N cores!$omp PARALLEL PRIVATE(rank, size)!$ size = omp get num threads()!$ rank = omp get thread num() WRITE(*,*) Hello from processor,& rank, of, size!$omp END PARALLEL END PROGRAM hello C #include <stdio.h> #ifdef OPENMP #include <omp.h> #endif int main (int argc, char * argv[]) int rank = 0, size = 1; #ifdef OPENMP #pragma omp parallel private(rank, size) #endif #ifdef OPENMP rank = omp get thread num(); size = omp get num threads(); #endif printf("hello from processor %d" " of %d\n", rank, size ); return 0; 10 POSIX Threads Hello from N cores // pthreads.c #include <stdio.h> #include <pthread.h> #define SIZE 4 void *hello(void *arg) printf( "Hello from processor %d of %d\n", *(int *)arg, SIZE ); return NULL; int main(int argc, char* argv[]) int i, p[size]; pthread t threads[size]; for (i = 1; i < SIZE; i++) /* Fork threads */ p[i] = i; pthread create(&threads[i], NULL, hello, &p[i]); p[0] = 0; hello(&p[0]); /* thread 0 greets as well */ for (i = 1; i < SIZE; i++) /* Join threads. */ pthread join(threads[i], NULL); return 0; 11 Compiling your OpenMP code NOT defined by the standard A special compilation flag must be used. On the Guillimin cluster: module add gcc gcc -fopenmp hello.c -o hello gfortran -fopenmp hello.f90 -o hello module add ifort icc icc -openmp hello.c -o hello ifort -openmp hello.f90 -o hello module add pgi pgcc -mp hello.c -o hello pgfortran -mp hello.f90 -o hello 12

4 Running your OpenMP code Important: environment variable OMP NUM THREADS. export OMP NUM THREADS=4./hello Hello from processor 2 of 4 Hello from processor 0 of 4 Hello from processor 3 of 4 Hello from processor 1 of 4 unset OMP NUM THREADS pgcc -mp hello.c -o hello./hello Hello from processor 0 of 1 gcc -fopenmp hello.c -o hello./hello Hello from processor 3 of 8 (...) 13 Running your OpenMP code On your laptop or desktop, just compile and run your code as above. On Guillimin cluster, use batch system to submit non-trivial OpenMP jobs! Example: hello.pbs: #!/bin/bash #PBS -l nodes=1:ppn=2 #PBS -l walltime=00:05:00 #PBS -V #PBS -N hello cd $PBS O WORKDIR export OMP NUM THREADS=2./hello > hello.out Submit your job: guillimin> msub -q class hello.pbs 14 Exercise 1: Log in to Guillimin, setting up the environment 1) Log in to Guillimin: ssh username@guillimin.clumeq.ca 2) Check for loaded software modules: guillimin> module list 3) See all available modules: guillimin> module av 4) Load necessary modules: guillimin> module add ifort icc 5) Check loaded modules again 15 Exercise 2: Hello program, compilation 1) Copy all files to your home directory: guillimin> cp /software/workshop/openmp/*. 2) Compile your code: guillimin> ifort -openmp hello.f90 -o hello guillimin> icc -openmp hello.c -o hello 16

5 Exercise 2: Hello, submitting your job 3) View the file hello.pbs : #!/bin/bash #PBS -l nodes=1:ppn=2 #PBS -l walltime=00:05:00 #PBS -V #PBS -N hello cd $PBS O WORKDIR export OMP NUM THREADS=2./hello > hello.out Exercise 2: Hello, submitting your job 4) Submit your job: guillimin> msub -q class hello.pbs 5) Check the job status: guillimin> showq -u <username> 6) Check the output (hello.out) Exercise 2: Hello, compile and run Alternatively, using interactive msub, or on your own Mac/Linux/Cygwin/MSYS computer: 1) Interactive login: msub -q class -I -l nodes=1:ppn=2,walltime=3:00:00 or create and then copy all files to a directory: yourlaptop> scp username@guillimin.clumeq.ca:/software/workshop/openmp/*. 2) Compile your code: > gfortran -fopenmp hello.f90 -o hello > gcc -fopenmp hello.c -o hello 3) Run your code: > # can use any value here > # defaults to number of cores > export OMP NUM THREADS=2 >./hello 19 OpenMP directives Format: sentinel directive [clause,] where sentinel is #pragma omp or!$omp. Examples: #pragma omp parallel (C),!$OMP PARALLEL,!$OMP END PARALLEL (Fortran): Parallel region construct. #pragma omp for: A workshare construct that makes a loop parallel (!$OMP DO in Fortran). #pragma omp parallel for: A combined construct: defines a parallel region that only contains the loop. #pragma omp barrier: A synchronization directive: all threads wait for each other here. 20

6 OpenMP clauses Examples: Data scope: private and shared.!$omp parallel private(i) shared(x) The variable i is private to the thread but the variable x is shared with all other threads. Default: all variables shared except loop variables (C: outer, Fortran:all), and variables declared inside block.!$omp parallel default(shared) private(i) All variables are shared except i. nowait in #pragma omp for nowait A loop where threads do not wait for each other upon completion. 21 OpenMP important library routines int omp get max threads(void); Get maximum number of threads used here. void omp set num threads(int); Set number of threads for next parallel region. int omp get thread num(void); Get current thread number in parallel region. int omp get num threads(void); Get number of threads in parallel region. double omp get wtime(void); Portable wall clock timing routine. More exist, for example for locks and nested regions. 22 OpenMP important environment variables OMP NUM THREADS Sets the maximum number of threads used. OMP SCHEDULE Used for run-time scheduling. More exist, for example to control nested parallelism. 23 parallel for void addvectors(const int *a, const int *b, int *c, int n) int i; #pragma omp parallel for for (i = 0; i < n; i++) Here i is automatically made private because it is the loop variable. All other variables are shared. Loop split between threads, for example with two threads, for n=10, thread 0 does index 0 to 4 and thread 1 does index 5 to 9. 24

7 parallel for Eliminate overhead from fork and join (in practise: synchronization) by using just one parallel region for two vector additions: void addvectors(const int *a, const int *b, int *c, int n) int i; #pragma omp for for (i = 0; i < n; i++)... #pragma omp parallel addvectors(a, b, c, n); addvectors(b, c, d, n); 25 parallel for, nowait and barrier void addvectors(const int *a, const int *b, int *c, int n) int i; #pragma omp for nowait for (i = 0; i < n; i++)... #pragma omp parallel addvectors(a, b, c, n); #pragma omp barrier addvectors(b, c, d, n/2); addvectors(e, f, g, n); omp for by default implies a barrier where all threads wait at the end of the loop. Eliminate synchronization overhead using nowait But need to add explicit barrier (dependency)! 26 parallel for: if clause The if clause allows conditional parallel regions: if n is too small, the overhead is not worth it: void addvectors(const int *a, const int *b, int *c, int n) int i; #pragma omp for for (i = 0; i < n; i++)... #pragma omp parallel if (n > 10000) addvectors(a, b, c, n); 27 parallel for: scheduling parallel for: scheduling schedule(static, 10000) allocates chunks of loop iterations to every thread: void addvectors(const int *a, const int *b, int *c, int n) int i; #pragma omp for schedule(static, 10000) for (i = 0; i < n; i++) Use dynamic instead of static to dynamically assign threads, if one finishes it is assigned the next chunk. Useful for unequal work within iterations. guided instead of dynamic: chunk sizes decrease as less work is left to do. runtime: use OMP SCHEDULE environment variable.28

8 parallel: manual scheduling (SPMD style) SPMD=Single Program Multiple Data, like in MPI. void addvectors(const int *a, const int *b, int *c, int n) for (i = 0; i < n; i++)... int tid, nthreads, low, high; #pragma omp parallel default(shared) private(tid, nthreads,\ low, high) tid = omp get thread num(); nthreads = omp get num threads(); low = (n * tid) / nthreads; high = (n * (tid + 1)) / nthreads; addvectors(&a[low], &b[low], &c[low], high-low); Calculate which thread does which loop iterations. Note: no barrier. 29 SPMD vs. worksharing Worksharing (omp for/omp do) is easiest to implement. SPMD (do work based on thread ID) may give better performance but is harder to implement. SPMD like in MPI: Instead of using large shared arrays, use smaller arrays private to threads: mark all (non-read-only) global and persistent (static/save) variables threadprivate, and communicate using buffers and barriers. Fewer cache misses using more private data may give better performance. More advanced topic: see Advanced OpenMP workshop in June. 30 sections (SPMD construct) #pragma omp parallel sections #pragma omp section addvectors(a, b, c, n); #pragma omp section printf("hello world!\n"); #pragma omp section printf("i may or may not be the third thread\n"); The sections are individual code blocks that are distributed over the threads. More flexible alternative (OpenMP 3.0): omp task, useful when traversing dynamic data structures (lists, trees, etc.). 31 workshare (Fortran) integer a(10000), b(10000), c(10000), d(10000)!$omp PARALLEL!$OMP WORKSHARE c(:) = a(:) + b(:)!$omp END WORKSHARE NOWAIT!$OMP WORKSHARE d(:) = a(:)!$omp END WORKSHARE NOWAIT!$OMP END PARALLEL Array assignments in Fortran are distributed among threads like loops. Note that nowait in Fortran comes at the end. 32

9 Exercise 3: Modifying Hello Let s ask each CPU to do its own computation... IF (rank == 0) THEN a=sqrt(2.0) b=0.0 WRITE(*,*) a,b=,a,b, on proc,rank END IF IF (rank == 1) THEN a=0.0 b=sqrt(3.0) WRITE(*,*) a,b=,a,b, on proc,rank END IF Exercise 4: Modifying Hello Do the same thing, now using omp sections Recompile as before and submit to the queue... Recompile as before and submit to the queue Race conditions (Data races) innerprod.c, innerprod.f90 #pragma omp parallel for private(i) shared(ip,a,b) ip += a[i] * b[i]; Problem: could be internally run as: int register = ip; register += a[i] * b[i]; ip = register; Threads may sum to their private CPU registers at the same time and overwrite ip, losing the addition from the other threads! 35 Race conditions a=1,2, b=3,4, inner product 1*3+2*4=11, two threads. ip register0 register1 tid=0 int register = ip; 0 0 unknown tid=1 int register = ip; tid=0 register += a[0] * b[0]; tid=1 register += a[1] * b[1]; tid=0 ip = register; tid=1 ip = register; Wrong result: ip=8. But may just as well be: ip=3: non-deterministic. 36

10 Solution: critical section Solution: atomic section #pragma omp parallel for private(i) shared(ip,a,b) #pragma omp critical ip += a[i] * b[i]; critical makes sure only one thread can run the (compound) statement at a time Problem: slow! #pragma omp parallel for private(i) shared(ip,a,b) #pragma omp atomic ip += a[i] * b[i]; atomic is like critical but can only apply to a specific memory location. Less general but less overhead Solution: local summing #pragma omp parallel private(i,localip) shared(ip,a,b) local #pragma omp for nowait localip += a[i] * b[i]; #pragma omp atomic ip += localip; Still needs atomic but only times the number of threads, not times N, greatly reducing overhead and improving performance. 39 Solution: local summing using array int *localips = malloc(omp get max threads()*sizeof(*localips)); #pragma omp parallel private(i,tid) shared(ip,a,b,localips) tid = omp get thread num(); localips[tid] = 0; #pragma omp for localips[tid] += a[i] * b[i]; #pragma omp single for (i = 0; i < omp get num threads(); i++) ip += localips[i]; One single thread does the final summing. Could also use omp master here, to force the master thread to do the final summing. omp master works like if (tid==0). 40

11 Solution: local summing using array int *localips = malloc(omp get max threads()*sizeof(*localips)); #pragma omp parallel private(i,tid) shared(ip,a,b,localips) tid = omp get thread num(); localips[tid] = 0; #pragma omp for localips[tid] += a[i] * b[i]; #pragma omp single for (i = 0; i < omp get num threads(); i++) ip += localips[i]; master and single are especially useful for I/O. Problem here: false cache sharing of the array localips. 41 Solution: local summing using array int *localips = malloc(omp get max threads()*sizeof(*localips)); #pragma omp parallel private(i,tid,localip) shared(ip,a,b,localips) tid = omp get thread num(); local #pragma omp for nowait localip += a[i] * b[i]; #pragma omp barrier localips[tid] = localip; #pragma omp single for (i = 0; i < omp get num threads(); i++) ip += localips[i]; Minimizes false sharing, moving it outside the loop. Could also use padding in localips, but need to know size of L1 cache (e.g. 64 bytes). 42 Easiest solution: use reduction #pragma omp parallel for reduction(+:ip) ip += a[i] * b[i]; Reduction is the most straightforward solution. Caveat: only works on scalars, and cannot control rounding errors caused by floating point calculations. For vectors use one of the other methods. 43 Exercise 5: Compilation error See omp bug.c or omp bug.f90, courtesy of Blaise Barney, Lawrence Livermore National Laboratory and find the compilation error in: #pragma omp parallel for \ shared(a,b,c,chunk) \ private(i,tid) \ schedule(static,chunk) tid = omp get thread num(); for (i=0; i < N; i++) printf("tid= %d i= %d c[i]= %f\n", tid, i, c[i]); /* end of parallel for construct */ 44

12 Exercise 6: Computing π Consider pi collect.c(f90), π = 4 arctan 1 = 4 Let s add timings: i=0 ( 1) i 1 2i + 1 = double t1, t2 t1 = omp get wtime();... t2 = omp get wtime(); printf("time = %.16f\n", t2-t1); Try some of the other alternatives (atomic, critical) to a reduction and measure the performance. 45 Exercise 7: Matrix multiplication See the file mm.c or mm.f90. Make the initialization and multiplication parallel and measure the speedup. 46 Further information: The standard itself, news, development, links to tutorials: Intel tutorial on YouTube (from Tim Mattson): New: OpenMP 4.0: thread affinity, SIMD, accelerators (GPUs, coprocessors), in Intel compilers 14.0, future GCC 4.9. Questions? Write the guillimin support team at guillimin@calculquebec.ca 47