COSC 6374 Parallel Computation Introduction to OpenMP Some slides based on material by Barbara Chapman (UH) and Tim Mattson (Intel) Edgar Gabriel Fall 2015 OpenMP Provides thread programming model at a high level. The user does not need to specify all the details Especially with respect to the assignment of work to threads Creation of threads User makes strategic decisions Compiler figures out details 1
OpenMP Overview OpenMP is a shared memory model. Threads communicate by sharing variables. Unintended sharing of data causes race conditions: race condition: when the program s outcome changes as the threads are scheduled differently. To control race conditions: Use synchronization to protect data conflicts. Synchronization is expensive so: Change how data is accessed to minimize the need for synchronization. Syntax details Most of the constructs in OpenMP are compiler directives. For C and C++: Directives are pragmas with the form: #pragma omp construct [clause [clause] ] Include file: #include <omp.h> For Fortran: the directives are comments and take one of the forms: Fixed form C$OMP construct [clause [clause] ] Free form (but works for fixed form too)!$omp construct [clause [clause] ] The OpenMP lib module: use omp_lib 2
Structured blocks (C/C++) Most OpenMP constructs apply to structured blocks. Structured block: a block with one point of entry at the top and one point of exit at the bottom. The only branches allowed are STOP statements in Fortran and exit() in C/C++. In C/C++: a block is a single statement or a group of statements between brackets id = omp_get_thread_num(); res[id] = do_work(id); #pragma omp for for(i=0; i<n; i++) res[i] = big_calc(i); A[i] = B[i] + res[i]; Structured Block Boundaries int id = omp_get_thread_num(); more: res[id] = do_big_job(id); if(!conv(res[id]) goto more; printf( All done \n ); A structured block int id = omp_get_thread_num(); more: res[id] = do_big_job(id); if(conv(res[id]) goto done; goto more; done: if(!really_done()) goto more; Not a structured block 3
Parallel Regions Threads are created using omp parallel pragma Each thread executes a copy of the code within the structured block How many threads are created? Environment variable to set no. of threads Runtime function System default used if no additional information given double A[1000]; // do some work printf( all done\n ); Parallel Regions Fork-join model of OpenMP Threads are created at the beginning of a parallel region and destroyed at the end of the parallel region (conceptually) Sequential execution double A[1000]; Parallel execution pooh(0,a) pooh(1,a) pooh(2,a) pooh(3,a) Sequential execution printf( all done\n ); Threads wait here for all threads to finish before proceeding (i.e. a barrier) 4
A multi-threaded Hello world program Starting point: sequential hello world int main(int argc, char **argv) int ID = 0; printf( hello(%d), ID); printf( world(%d) \n, ID); A multi-threaded Hello world program #include omp.h int main( int argc, char **argv) int ID= omp_get_thread_num(); printf( hello(%d), ID); printf( world(%d) \n, ID); return 0; Sample output hello(1) hello(0) world(1) world(0) hello (3) hello(2) world(3) world(2) 5
OpenMP Library routines Modify/Check the number of threads omp_set_num_threads() omp_get_num_threads() omp_get_thread_num() omp_get_max_threads() Are we in a parallel region? omp_in_parallel() How many processors in the system? omp_num_procs() Example: vector add operation Sequential code for( i=0; i<n; i++) a[i] = a[i] + b[i]; OpenMP parallel version int id = omp_get_thread_num(); int Nthrds = omp_get_num_threads(); int istart = id * N / Nthrds; int iend = (id+1) * N / Nthrds; int i; for( i=istart; i<iend; i++) a[i] = a[i] + b[i]; 6
Example: vector add operation All variables declared inside of the parallel region are considered to be private to each thread Each thread has its own copy of the variable Variables declared outside of a parallel region are shared amongst threads Unless explicitly changed by the user If istart and iend are not coordinated cautiously, it can lead to cache coherence problems e.g. a[iend] on thread with id=x and a[istart] on the thread with id=x+1 are in the same cache line OpenMP work-sharing constructs The for work-sharing construct splits up loop iterations among the threads #pragma omp for for ( i=0; i<n; i++ ) neat_stuff(i); By default, there is a barrier at the end of the omp for. Use the nowait clause to turn off the barrier, e.g. #pragma omp for nowait nowait is useful between two consecutive, independent omp for loops. 7
Example: vector add operation #pragma omp for for ( i=0; i<n; i++ ) a[i] = a[i] + b[i]; Much simpler code than the previous OpenMP version Loop variable i is automatically declared to be private on each thread Does not define how the loop iterations are distributed among the threads Can use the schedule clause to influence the work distribution, e.g. #pragma omp for schedule(static) OpenMP for construct The schedule clause affects how loop iterations are mapped onto threads schedule(static [,chunk]) Deal-out blocks of iterations of size chunk to each thread. schedule(dynamic[,chunk]) Each thread grabs chunk iterations off a queue until all iterations have been handled. schedule(guided[,chunk]) Threads dynamically grab blocks of iterations. The size of the block starts large and shrinks down to size chunk as the calculation proceeds. schedule(runtime) Schedule and chunk size taken from the OMP_SCHEDULE environment variable. 8
OpenMP work-sharing constructs The sections work-sharing construct gives a different structured block to each thread. By default, there is a barrier at the end of the omp sections. Use the nowait clause to turn off the barrier. #pragma omp sections #pragma omp section X_calculation(); #pragma omp section y_calculation(); #pragma omp section z_calculation(); OpenMP work-sharing constructs The single construct denotes a block of code that is executed by only one thread. A barrier is implied at the end of the single block. do_many_things(); #pragma omp single exchange_boundaries(); do_many_other_things(); 9
Combined parallel/work-share constructs OpenMP shortcut: Put the parallel and the workshare on the same line double res[max]; int i; #pragma omp for for ( i=0; i<max; i++) res[i] = huge(); double res[max]; int i; for for ( i=0; i<max; i++) res[i] = huge(); These are equivalent Data Environment: Default storage attributes Shared Memory programming model: Most variables are shared by default Global variables are SHARED among threads C: File scope variables, static Fortran: COMMON blocks, SAVE variables, MODULE variables But not everything is shared... Stack variables in sub-programs called from parallel regions are PRIVATE Automatic variables within a statement block are PRIVATE. 10
Data Sharing Examples double A[10]; int index[10]; work(index); printf(%d\n,index[0]); void work (int * index) float temp[10]; int count; return; A and index are shared by all threads. temp is local to each thread A, index temp[10] temp[10] temp[10] A, index * Third party trademarks and names are the property of their respective owner. Data Environment: Changing storage attributes One can selectively change storage attributes constructs using the following clauses* SHARED PRIVATE FIRSTPRIVATE THREADPRIVATE All the clauses on this page apply to OpenMP construct NOT to the entire region. The value of a private inside a parallel loop can be transmitted to a global value outside the loop with: LASTPRIVATE The default status can be modified with: DEFAULT (PRIVATE SHARED NONE) 11
Private Clause private(var) creates a local copy of var for each thread The value is uninitialized Private copy is not storage-associated with the original The original is undefined at the end int var = 13; for private (var) for ( j=0; j<1000; j++ ) var = var + j; printf ( %d\n, var ); Each thread gets its own var which are however not initialized Regardless of initialization, var is undefined at the end of the parallel region Firstprivate Clause Firstprivate is a special case of private. Initializes each private copy with the corresponding value from the master thread. int var = 13; for firstprivate (var) for ( j=0; j<1000; j++ ) var = var + j; printf ( %d\n, var ); Each thread gets its own var with an initial value of 13 Regardless of initialization, var is undefined at the end of the parallel region 12
Lastprivate Clause Lastprivate passes the value of a private from the last iteration to a global variable. int var = 13; for firstprivate (var) lastprivate(var) for ( j=0; j<1000; j++ ) var = var + j; printf ( %d\n, var ); Each thread gets its own var with an initial value of 13 var is defined as its value at the last sequential iteration (i.e. for j=999) OpenMP: A data environment test Consider this example of PRIVATE and FIRSTPRIVATE int A=1, B=1, C=1; private(b) firstprivate (C) Are A,B,C local to each thread or shared inside the parallel region? What are their initial values inside and after the parallel region? Inside this parallel region... A is shared by all threads; equals 1 B and C are local to each thread. B s initial value is undefined C s initial value equals 1 Outside this parallel region... The values of B and C are undefined. 13
Default Clause Note that the default storage attribute is DEFAULT(SHARED) (so no need to use it) To change default: DEFAULT(PRIVATE) each variable in static extent of the parallel region is made private as if specified in a private clause mostly saves typing DEFAULT(NONE): no default for variables in static extent. Must list storage attribute for each variable in static extent Only the Fortran API supports default(private). C/C++ only has default(shared) or default(none). OpenMP Reduction Combines an accumulation operation across threads: reduction (op : list) Inside a parallel or a work-sharing construct: A local copy of each list variable is made and initialized depending on the op (e.g. 0 for + ). Compiler finds standard reduction expressions containing op and uses them to update the local copy. Local copies are reduced into a single value and combined with the original global value. The variables in list must be shared in the enclosing parallel region. 14
Example from last lecture slightly modified var = 0; for ( j=0; j<1000; j++ ) var = var + j; printf( %d\n, var ); int var = 0; for firstprivate (var) lastprivate(var) for ( j=0; j<1000; j++ ) var = var + j; printf ( %d\n, var ); Parallel code shown here does not lead to the same result as sequential solution Code in last lecture was used to explain demonstrate behavior of firstprivate and lastprivate clauses Example from last lecture cont. E.g. 2 threads, assuming thread 0 has j = 0 -> 499 thread 1 has j = 500 ->999 Lastprivate clause ensure that the variable listed contains the value of the last sequential iteration (i.e. j=1000) In parallel code, var = 999 j=500 In the sequential code, var = j 999 j=0 j 15
OpenMP reduction solution var = 0 for ( j=0; j<1000; j++ ) var = var + j printf( %d\n, var ); Parallel code leading to the identical result as the sequential version: var = 0 for reduction(+:var) for ( j=0; j<1000; j++ ) var = var + j; printf ( %d\n, var); Reduction operands/initial-values A range of associative operands can be used with reduction: Initial values are the ones that make sense mathematically. Operand Initial value + 0 * 1-0.AND..OR..NEQV..EQV. MIN* MAX*.true..false..false..true. Largest pos. number Most neg. number Operand Initial value iand All bits on ior 0 ieor 0 & ~0 0 ^ 0 && 1 0 * Min and Max are not available in C/C++ 16
OpenMP: Synchronization High level synchronization: critical atomic barrier ordered Low level synchronization flush locks (both simple and nested) Synchronization critical Only one thread at a time can enter a critical region. int val=13; int lval=0; #pragma omp for for ( i=0; i<1000; i++ ) lval = lval + i; #pragma omp critical val = val + lval; 17
Synchronization - barrier Barrier: Each thread waits until all threads arrive. Remember: implicit barriers at the end of work-sharing constructs and parallel regions shared (A,C) private(id) id = omp_get_thread_num(); A[id] = big_calc1(id); #pragma omp barrier #pragma omp for for(i=0;i<n;i++) C[i]=big_calc3(I,A); OpenMP Library Routines To use a known, fixed number of threads used in a program, tell the system that you don t want dynamic adjustment of the number of threads, set the number of threads, check the number of threads you got. Note: the system may give you fewer threads than requested. If the precise number of threads matters, test for it and respond accordingly. 18
OpenMP Library Routines int num_threads; omp_set_dynamic( 0 ); omp_set_num_threads( omp_num_procs() ); int id=omp_get_thread_num(); #pragma omp single num_threads = omp_get_num_threads(); do_lots_of_stuff(id); OpenMP Environment Variables Control how omp for schedule(runtime) loop iterations are scheduled. OMP_SCHEDULE schedule[, chunk_size] Set the default number of threads to use. OMP_NUM_THREADS int_literal How to set environment variables Depends on the shell that you use E.g. using bash (default on shark) export OMP_NUM_THREADS 8 Or adding export OMP_NUM_THREADS=8 in the.bashrc file in your home directory and logging in freshly 19
Example: Numerical Integration Mathematically, we know that: 4.0 1 4.0 (1+x 2 ) 0 dx = 2.0 We can approximate the integral as a sum of rectangles: N F(x i ) x i = 0 0.0 X 1.0 Where each rectangle has width x and height F(x i ) at the middle of interval i. PI Program: an example static long num_steps = 100000; double step; int main ( int argc, char **argv) int i; double x, pi, sum = 0.0; step = 1.0/(double) num_steps; for (i=0;i<= num_steps; i++) x = (i+0.5)*step; sum = sum + 4.0/(1.0+x*x); pi = step * sum; return 0; 20
First solution: using parallel regions only static long num_steps = 100000; int i, id, nthreads; double x, pi, sum[num_threads]; Promote scalar to an array dimensioned by number of threads to avoid race condition. double step = 1.0/(double) num_steps; omp_set_num_threads(num_threads); private (i, id, x) id = omp_get_thread_num(); #pragma omp single nthreads = omp_get_num_threads(); for (i=id, sum[id]=0.0;i< num_steps; i=i+nthreads) x = (i+0.5)*step; sum[id] += 4.0/(1.0+x*x); for(i=0, pi=0.0; i<nthreads; i++) pi += sum[i] * step; Performance will suffer due to false sharing of the sum array. Second solution: using critical region int i, id, nthreads; double x, pi, sum; step = 1.0/(double) num_steps; omp_set_num_threads(num_threads); private (i, id, x, sum) id = omp_get_thread_num(); #pragma omp single nthreads = omp_get_num_threads(); for (i=id, sum=0.0;i< num_steps; i=i+nthreads) x = (i+0.5)*step; sum += 4.0/(1.0+x*x); #pragma omp critical pi += sum * step; No array, so no false sharing. Note: this method of combining partial sums doesn t scale very well. 21
Third solution: parallel for with a reduction int i; double x, pi, sum = 0.0; step = 1.0/(double) num_steps; omp_set_num_threads(num_threads); For good OpenMP implementations, reduction is more scalable than critical. for private(x) reduction(+:sum) for (i=0;i<= num_steps; i++) x = (i+0.5)*step; sum = sum + 4.0/(1.0+x*x); pi = step * sum; In practice, you set number of threads by setting the environment variable, OMP_NUM_THREADS 22