MULTICORE PROGRAMMING WITH OPENMP

Transcription

1 MULTICORE PROGRAMMING WITH OPENMP Dott. Alessandro Capotondi

2 Outline and Goals Understand OpenMP directives usage Manage data parallelism and load balancing using OpenMP Understand Overheads Compare to Amdahl s predictions

3 Disclaimer (Unfortunally) This is not an OpenMP course, so it will not cover all the aspect related to topic. If you want to elaborate in deep your knowledge of parallel programming using OpenMP here you can find some additional material (this is not required for the successful pass of this exam) Tutorials: Reference Card Books

4 Programming model: OpenMP De-facto standard for the shared memory programming model A collection of compiler directives, library routines and environment variables Easy to specify parallel execution within a serial code Targets several types of parallelism (data, task, heterogeneous) Popular in high-end embedded Requires special support in the compiler Generates calls to threading libraries (e.g. pthreads)

5 Recap

6 Recap

7 Recap

8 Fork/Join Parallelism Sequential program Parallel program Initially only master thread is active Master thread executes sequential code Fork: Master thread creates or awakens additional threads to execute parallel code Join: At the end of parallel code created threads are suspended upon barrier synchronization

9 Pragmas Pragma: a compiler directive in C or C++ Stands for pragmatic information A way for the programmer to communicate with the compiler Compiler free to ignore pragmas: original sequential semantic is not altered Syntax: #pragma omp <rest of pragma>

10 Components of OpenMP a subset of the directives Directives Parallel regions #pragma omp parallel Work sharing #pragma omp for #pragma omp sections Synchronization #pragma omp barrier #pragma omp critical #pragma omp atomic Clauses Thread Forking/Joining omp_parallel_start() omp_parallel_end() Loop scheduling Thread IDs Data scope attributes private shared reduction Loop scheduling static dynamic Runtime Library omp_get_thread_num() omp_get_num_threads()

11

12 Outlining parallelism The parallel directive Fundamental construct to outline parallel computation within a sequential program Code within its scope is replicated among threads Defers implementation of parallel execution to the runtime (machinespecific, e.g. pthread_create) A sequential program....is easily parallelized int main() #pragma omp parallel printf ( \nhello world! ); int main() omp_parallel_start(&parfun, ); parfun(); omp_parallel_end(); int parfun( ) printf ( \nhello world! );

13 #pragma omp parallel Code originally contained within the scope of the pragma is outlined to a new function within the compiler int main() omp_parallel_start(&parfun, ); parfun(); omp_parallel_end(); int main() #pragma omp parallel printf ( \nhello world! ); int parfun( ) printf ( \nhello world! );

14 #pragma omp parallel The #pragma construct in the main function is replaced with function calls to the runtime library int main() omp_parallel_start(&parfun, ); parfun(); omp_parallel_end(); int main() #pragma omp parallel printf ( \nhello world! ); int parfun( ) printf ( \nhello world! );

15 #pragma omp parallel First we call the runtime to fork new threads, and pass them a pointer to the function to execute in parallel int main() omp_parallel_start(&parfun, ); parfun(); omp_parallel_end(); int main() #pragma omp parallel printf ( \nhello world! ); int parfun( ) printf ( \nhello world! );

16 #pragma omp parallel Then the master itself calls the parallel function int main() omp_parallel_start(&parfun, ); parfun(); omp_parallel_end(); int main() #pragma omp parallel printf ( \nhello world! ); int parfun( ) printf ( \nhello world! );

17 #pragma omp parallel Finally we call the runtime to synchronize threads with a barrier and suspend them int main() omp_parallel_start(&parfun, ); parfun(); omp_parallel_end(); int main() #pragma omp parallel printf ( \nhello world! ); int parfun( ) printf ( \nhello world! );

18 #pragma omp parallel Data scope attributes int main() int id; int a = 5; #pragma omp parallel id = omp_get_thread_num(); if (id == 0) printf ( Master: a = %d., a*2); else printf ( Slave: a = %d., a); Call runtime to get thread ID: Every thread sees a different value Master and slave threads access the same variable a A slightly more complex example

19 #pragma omp parallel Data scope attributes int main() int id; int a = 5; #pragma omp parallel id = omp_get_thread_num(); if (id == 0) printf ( Master: a = %d., a*2); else printf ( Slave: a = %d., a); Call runtime to get thread ID: Every thread sees a different value Master and slave threads access the same variable a A slightly more complex example

20 #pragma omp parallel Data scope attributes What is the view of memory among different threads in a parallel region? int main() int id; int a = 5; #pragma omp parallel shared (a) private (id) id = omp_get_thread_num(); if (id == 0) printf ( Master: a = %d., a*2); else printf ( Slave: a = %d., a); Insert code to retrieve the address of the shared object from within each parallel thread Allow symbol privatization: Each thread contains a private copy of this variable A slightly more complex example

21 #pragma omp parallel Data scope attributes int main() int id; int a = 5; #pragma omp parallel shared (a) private (id) id = omp_get_thread_num(); if (id == 0) printf ( Master: a = %d., a*2); else printf ( Slave: a = %d., a); Insert code to retrieve the address of the shared object from within each parallel thread Allow symbol privatization: Each thread contains a private copy of this variable A slightly more complex example

22 LET S TRY IT! ALESSANDRO CAPOTONDI ALESSANDRO.CAPOTONDI@UNIBO.IT GIUSEPPE TAGLIAVINI GIUSEPPE.TAGLIAVINI@UNIBO.IT

23 ODROID XU Exynos5 Octa Cortex -A15 1.4Ghz quad core and Cortex -A7 quad core CPUs PowerVR SGX544MP3 GPU (OpenGL ES 2.0, OpenGL ES 1.1 and OpenCL 1.1 EP) 2Gbyte LPDDR3 RAM USB 3.0 Host x 1, USB 3.0 OTG x 1, USB 2.0 Host x 4 HDMI 1.4a output Type-D connector emmc 4.5 Flash Storage Full Fledged Ubuntu OS support

24 DO YOU TRUST ME? Somewhere in the undergrounds of Bologna

25 SUBMIT YOUR RESULTS You could submit your results and questions at Include: Student(s) Name and ID(s) Attach the SpreadSheet! Solutions will be published on the course website in a few weeks!

26 SETUP FOR THE EXERCISES Login to Odroid ssh X stud_usr@<board-ip> Execise files are already loaded on the board cd /home/stud_usr/hsdes17 List the files using the command ls Take a look at test.c. Different exercises are #ifdef-ed gedit test.c & To compile and execute the desired one make clean all run e MYOPTS="-DEX1" (Execise 1) make clean all run e MYOPTS="-DEX2" (Execise 2)

27 EX 1 OpenMP Hello World! #pragma omp parallel num_threads (?) printf "Hello world, I m thread??" Use parallel directive to create multiple threads Each thread executes the code enclosed within the scope of the directive Use runtime library functions to determine thread ID All SPMD parallelization is based on this approach

28 More data sharing clauses firstprivate copyin, private storage lastprivate Copyout, private storage

29 Sharing work among threads The for directive The parallel pragma instructs every thread to execute all of the code inside the block If we encounter a for loop that we want to divide among threads, we use the for pragma #pragma omp for

30 #pragma omp for #pragma omp for can be placed everywhere inside a parallel construct, or combined with it, as in the example int main() #pragma omp parallel for for (i=0; i<10; i++) a[i] = i; int main() omp_parallel_start(&parfun, ); parfun(); omp_parallel_end(); int parfun( ) int LB = ; int UB = ; for (i=lb; i<ub; i++) a[i] = i;

31 #pragma omp for int main() omp_parallel_start(&parfun, ); parfun(); omp_parallel_end(); int main() #pragma omp parallel for for (i=0; i<10; i++) a[i] = i; int parfun( ) int LB = ; int UB = ; for (i=lb; i<ub; i++) a[i] = i;

32 #pragma omp for int main() omp_parallel_start(&parfun, ); parfun(); omp_parallel_end(); int main() #pragma omp parallel for for (i=0; i<10; i++) a[i] = i; int parfun( ) int LB = ; int UB = ; for (i=lb; i<ub; i++) a[i] = i;

33 The schedule clause Static Loop Partitioning #pragma omp for schedule(static) for (i=0; i<12; i++) a[i] = i; Es. 12 iterations (N), 4 threads (Nthr) DATA CHUNK C = ceil ( N ) Nthr 3 iterations thread Useful for: Simple, regular loops Iterations with equal duration Iteration space Thread ID (TID) LOWER BOUND LB = C * TID UPPER BOUND UB = min [C * ( TID + 1) ], N

34 The schedule clause Static Loop Partitioning #pragma omp for schedule(static) for (i=0; i<12; i++) a[i] = i; Es. 12 iterations (N), 4 threads (Nthr) DATA CHUNK C = ceil ( N ) Nthr 3 iterations thread Useful for: Simple, regular loops Iterations with equal duration Iteration space Thread ID (TID) LOWER BOUND LB = C * TID UPPER BOUND UB = min [C * ( TID + 1) ], N

35 The schedule clause Static Loop Partitioning #pragma omp for schedule(static) for (i=0; i<12; i++) a[i] = i; int start = rand(); int count = 0; Iteration space while (start++ < 256) count++; a[count] = foo(); UNBALANCED workloads

36 The schedule clause Dynamic Loop Partitioning #pragma omp for schedule(static) schedule(dynamic) for (i=0; i<12; i++) int start = rand(); int count = 0; Iteration space while (start++ < 256) count++; a[count] = foo();

37 The schedule clause Dynamic Loop Partitioning Runtime environment Iteration space Work queue int parfun( ) int LB, UB; GOMP_loop_dynamic_next(&LB, &UB); for (i=lb; i<ub; i++)

38 The schedule clause Dynamic Loop Partitioning Iteration space BALANCED workloads 9

39 Parallelization granularity Iteration chunking Fine-grain Parallelism Best opportunities for load balancing, but.. Small amounts of computational work between parallelism computation stages Low computation to parallelization ratio High parallelization overhead Coarse-grain Parallelism Harder to load balance efficiently, but.. Large amounts of computational work between parallelism computation stages High computation to parallelization ratio Low parallelization overhead

40 The schedule clause Dynamic Loop Partitioning #pragma omp for schedule(dynamic, 1) for (i=0; i<12; i++) int start = rand(); int count = 0; Iteration space while (start++ < 256) count++; a[count] = foo();

41 The schedule clause Dynamic Loop Partitioning Runtime environment Iteration space Work queue int parfun( ) int LB, UB; GOMP_loop_dynamic_next(&LB, &UB); for (i=lb; i<ub; i++) WORK

42 The schedule clause Dynamic Loop Partitioning Iteration space

43 The schedule clause Dynamic Loop Partitioning Iteration space chunking overhead

44 The schedule clause Dynamic Loop Partitioning #pragma omp for schedule(dynamic, 2) for (i=0; i<12; i++) int start = rand(); int count = 0; Iteration space while (start++ < 256) count++; a[count] = foo();

45 The schedule clause Dynamic Loop Partitioning Runtime environment Iteration space Work queue int parfun( ) int LB, UB; GOMP_loop_dynamic_next(&LB, &UB); for (i=lb; i<ub; i++) WORK

46 The schedule clause Dynamic Loop Partitioning Iteration space

47 The schedule clause Dynamic Loop Partitioning Iteration space smaller chunking overhead

48 More scheduling clauses schedule (guided[, chunk]) Threads dynamically grab blocks of iterations. The size of the block starts large and shrinks down to size chunk as the calculation proceeds. schedule (runtime[, chunk]) Schedule and chunk size taken from the OMP_SCHEDULE environment variable (or the runtime library for OpenMP 3.0)

49 LET S TRY IT! ALESSANDRO CAPOTONDI ALESSANDRO.CAPOTONDI@UNIBO.IT GIUSEPPE TAGLIAVINI GIUSEPPE.TAGLIAVINI@UNIBO.IT

50 EX 2A STATIC LOOP #pragma omp parallel for \ num_threads(nthreads) schedule(static) for (uint i=0; i<niters; i++) work (W); /* (implicit) SYNCH POINT */ T0 T1 T2 T With schedule(static) M iterations are statically assigned to each thread, where: M = NITERS/NTHREADS Small overhead: loop indexes are computed according to thread ID Optimal scheduling if workload is balanced

51 EX 2A STATIC LOOP #pragma omp parallel for \ num_threads(nthreads) schedule(static) for (uint i=0; i<niters; i++) work (W); /* (implicit) SYNCH POINT */ 2A Use the following configurations: NITERS = 1024 NTHREADS= 1,2,4,8,16 W = Objectives: Collect execution time in a excel sheet for the various configurations Comment on the results

52 EX 2B DYNAMIC LOOP #pragma omp parallel for \ num_threads (NTHREADS) schedule (dynamic, M) for (uint i=0; i<niters; i++) work (W); /* (implicit) SYNCH POINT */ T0 T1 T2 T OVERHEAD CHUNK Iterations are dynamically assigned to threads in groups of M = NITERS/NTHREADS Same parallelization of schedule(static) Coarse granularity Overhead only at beginning and end of loop

53 EX 2B DYNAMIC LOOP T0 T1 T2 T #pragma omp parallel for \ num_threads (NTHREADS) \ schedule(dynamic, 1) for (uint i=0; i<niters; i++) work (W); /* (implicit) SYNCH POINT */ Iterations are dynamically assigned to threads one iteration per chunk Finest granularity Overhead at every iteration OVERHEAD CHUNK (size = 1 iter)

54 EX 2B DYNAMIC LOOP #pragma omp parallel for \ num_threads (NTHREADS) schedule (dynamic, M) for (uint i=0; i<niters; i++) work (W); /* (implicit) SYNCH POINT */ 2a, 2b Use the following configurations: NITERS = 1024 NTHREADS = 1,2,4,8,16 M = NITERS/NTHREADS, 1 W = Objectives: Collect execution time in a excel sheet for the various configurations Comment on the results

55 EX 2C DYNAMIC LOOP #pragma omp parallel for \ num_threads (NTHREADS) schedule (dynamic, M) for (uint i=0; i<niters; i++) work (W); /* (implicit) SYNCH POINT */ 2c Use the following configurations: NITERS = NTHREADS = 1,2,4,8,16 M = NITERS/NTHREADS, 1 W = 10 Objectives: Collect execution time in a excel sheet for the various configurations Comment on the results

56 EX 2D DYNAMIC LOOP (HOMEWORKS) Study the impact of chunk size for varying sizes of of the workload W Use the following configurations: NITERS = 1024*256 NTHREADS = 4 M = = 32, 16, 8, 4, 1 W = 1, 10, 100, 1000 Objectives: Collect execution time in a excel sheet for the various configurations Comment on the results HINT: Plot NORMALIZED execution time to the fastest scheduling option for a given value of W

57 EX 3 - UNBALANCED LOOP T0 T1 T2 T3 #pragma omp parallel for \ num_threads (NTHREADS) \ schedule (dynamic, NITERS/NTHREADS) SYNCH POINT for (uint i=0; i<niters; i++) work((i>>2) * W); /* (implicit) SYNCH POINT */ Iterations have different duration Using coarse-grained chunks (NITERS/NTHREADS) creates unbalanced work among the threads Due to the barrier at the end of parallel region, all threads have to wait for the slowest one

58 EX 3 - UNBALANCED LOOP T0 T1 T2 T3 #pragma omp parallel for \ num_threads (NTHREADS) \ schedule (dynamic, 1) for (uint i=0; i<niters; i++) work((i>>2) * W); /* (implicit) SYNCH POINT */ Iterations have different duration SPEEDUP Using fine-grained chunks (the finest is creates balanced work among the threads The overhead for dynamic scheduling is amortized by the effect of workbalancing on the overall loop duration

59 EX 3 - UNBALANCED LOOP #pragma omp parallel for \ num_threads (NTHREADS) schedule (dynamic, M) for (uint i=0; i<niters; i++) work((i>>2) * W); /* (implicit) SYNCH POINT */ Exercise 3a, 3b Use the following configurations: NITERS = 128 NTHREADS = 4 M = 32, 16, 8, 4, 1 W = Objectives: Create a new excel sheet plotting results (execution time) for static and dynamic schedules (with chunk size M) Comment on results

60 #pragma omp barrier Most important synchronization mechanism in shared memory fork/join parallel programming All threads participating in a parallel region wait until everybody has finished before computation flows on This prevents later stages of the program to work with inconsistent shared data It is implied at the end of parallel constructs, as well as for and sections (unless a nowait clause is specified)

61 #pragma omp critical Critical Section: a portion of code that only one thread at a time may execute We denote a critical section by putting the pragma #pragma omp critical in front of a block of C code

62 π-finding code example double area, pi, x; int i, n; #pragma omp parallel for private(x) \ shared(area) for (i=0; i<n; i++) x = (i + 0.5)/n; area += 4.0/(1.0 + x*x); pi = area/n;

63 Race condition Ensure atomic updates of the shared variable area to avoid a race condition in which one process may race ahead of another and ignore changes

64 Race condition (Cont d) Thread A reads into a local register Thread B reads into a local register Thread A updates area with Thread B ignores write from thread A and updates area with time

65 π-finding code example double area, pi, x; int i, n; #pragma omp parallel for private(x) shared(area) for (i=0; i<n; i++) x = (i +0.5)/n; #pragma omp critical area += 4.0/(1.0 + x*x); pi = area/n; #pragma omp critical protects the code within its scope by acquiring a lock before entering the critical section and releasing it after execution

66 Correctness, not performance! As a matter of fact, using locks makes execution sequential To dim this effect we should try use fine grained locking (i.e. make critical sections as small as possible) A simple instruction to compute the value of area in the previous example is translated into many more simpler instructions within the compiler! The programmer is not aware of the real granularity of the critical section

67 Correctness, not performance! As a matter of fact, using locks makes execution sequential To dim this effect we should try use fine grained locking (i.e. make critical sections as small as possible) A simple instruction to compute the This value is a of dump area in of the previous example is translated into many intermediate more simpler instructions within the compiler! representation of the The programmer is not aware of the program real granularity within the of the critical section compiler

68 Correctness, not performance! As a matter of fact, using locks makes execution sequential To dim this effect we should try use fine grained locking (i.e. make critical sections as small as possible) A simple instruction to compute the value of area in the previous example is translated into many more simpler instructions within the compiler! The programmer is not aware of the real granularity of the critical section

69 Correctness, not performance! As a matter of fact, using locks makes execution sequential To dim this effect we should try use fine grained locking (i.e. make critical sections as small as possible) A simple instruction to compute the value of area in the call runtime to acquire lock previous example is translated into many more simpler instructions within the compiler! The programmer is not aware Lock-protected of the real granularity of the critical section operations (critical section) call runtime to release lock

70 π-finding code example double area, pi, x; int i, n; #pragma omp parallel for \ private(x) \ shared(area) for (i=0; i<n; i++) x = (i +0.5)/n; Parallel #pragma omp critical area += 4.0/(1.0 + x*x); Sequential pi = area/n; Waiting for lock

71 Correctness, not performance! A programming pattern such as area += 4.0/(1.0 + x*x); in which we: Fetch the value of an operand Add a value to it Store the updated value is called a reduction, and is commonly supported by parallel programming APIs OpenMP takes care of storing partial results in private variables and combining partial results after the loop

72 Correctness, not performance! double area, pi, x; int i, n; #pragma omp parallel for private(x) shared(area) for (i=0; i<n; i++) x = (i +0.5)/n; reduction(+:area) area += 4.0/(1.0 + x*x); pi = area/n; The reduction clause instructs the compiler to create private copies of the area variable for every thread. At the end of the loop partial sums are combined on the shared area variable

73 Correctness, not performance! double area, pi, x; int i, n; #pragma omp parallel for private(x) shared(area) for (i=0; i<n; i++) x = (i +0.5)/n; reduction(+:area) area += 4.0/(1.0 + x*x); pi = area/n; The reduction clause instructs the compiler to create private copies of the area variable for every thread. At the end of the loop partial sums are combined on the shared area variable

74 Correctness, not performance! double area, pi, x; int i, n; #pragma omp parallel for private(x) shared(area) for (i=0; i<n; i++) x = (i +0.5)/n; reduction(+:area) area += 4.0/(1.0 + x*x); pi = area/n; The reduction clause instructs the compiler to create private copies of the area variable for every thread. At the end of the loop partial sums are combined on the shared area variable

75 More worksharing constructs The master directive The master construct denotes a structured block that is only executed by the master thread. The other threads just skip it (no synchronization is implied). #pragma omp parallel do_many_things(); #pragma omp master exchange_boundaries(); #pragma omp barrier do_many_other_things();

76 More worksharing constructs The single directive The single construct denotes a block of code that is executed by only one thread (not necessarily the master thread). A barrier is implied at the end of the single block (can remove the barrier with a nowait clause). #pragma omp parallel do_many_things(); #pragma omp single exchange_boundaries(); do_many_other_things();

77 Recap exercise!

78 Recap exercise! Solution (1)

79 Recap exercise! Solution (2)

80 What we learned? Create Parallelism Exploit data-parallelisms using OpenMP Correctness is not performance efficiency! How to measure parallelization effectiveness and efficiency?

81 LET S TRY IT! ALESSANDRO CAPOTONDI ALESSANDRO.CAPOTONDI@UNIBO.IT GIUSEPPE TAGLIAVINI GIUSEPPE.TAGLIAVINI@UNIBO.IT

82 EX 4 AMDAHL S EFFECT A program has a workload composed of NITERS repetitions of the function work. This function is invoked with a parameter W that specifies its duration. Of these NITERS repetitions, only P% can be executed in parallel. Use the following configurations: NITERS = NTHREADS = 4 P = 0%, 10%, 50%, 90% of NITERS W = 1, 1000 Objective Compute on a new spreadsheet the estimated speedup according to the Amdahl s law Measure the actual speedup Comment on results. What is the effect of the overheads? SPEEDUP EX3 0.00% 10.00% 50.00% 90.00% P Speedup (W=1) Speedup (W=10000) Ideal

83 SPMD VS MPMD Recall.. SPMD (single program, multiple data) Processors execute the same stream of instructions over different data #pragma omp for MPMD (multiple program, multiple data) Processors execute different streams of instructions over (possibly) different data #pragma omp sections #pragma omp task

84 Recap: TASK parallelism in OpenMP 2.5 The sections directive The for pragma allows to exploit data parallelism in loops OpenMP 2.5 also provides a directive to exploit task parallelism #pragma omp sections

85 Task Parallelism Example int main() v = alpha(); w = beta (); y = delta (); x = gamma (v, w); z = epsilon (x, y)); printf ( %f\n, z);

86 Task Parallelism Example int main() #pragma omp parallel sections v = alpha(); w = beta (); #pragma omp parallel sections y = delta (); x = gamma (v, w); z = epsilon (x, y)); printf ( %f\n, z);

87 Task Parallelism Example int main() #pragma omp parallel sections v = alpha(); w = beta (); #pragma omp parallel sections y = delta (); x = gamma (v, w); z = epsilon (x, y)); printf ( %f\n, z);

88 Task Parallelism Example int main() #pragma omp parallel sections #pragma omp section v = alpha(); #pragma omp section w = beta (); #pragma omp parallel sections #pragma omp section y = delta (); #pragma omp section x = gamma (v, w); z = epsilon (x, y)); printf ( %f\n, z);

89 Task Parallelism Example int main() v = alpha(); w = beta (); y = delta (); x = gamma (v, w); z = epsilon (x, y)); printf ( %f\n, z);

90 Task Parallelism Example int main() #pragma omp parallel sections v = alpha(); w = beta (); y = delta (); x = gamma (v, w); z = epsilon (x, y)); printf ( %f\n, z);

91 Task Parallelism Example int main() #pragma omp parallel sections v = alpha(); w = beta (); y = delta (); x = gamma (v, w); z = epsilon (x, y)); printf ( %f\n, z);

92 Task Parallelism Example int main() #pragma omp parallel sections #pragma omp section v = alpha(); #pragma omp section w = beta (); #pragma omp section y = delta (); x = gamma (v, w); z = epsilon (x, y)); printf ( %f\n, z);

93 Task parallelism The sections directive allows a very limited form of task parallelism All tasks must be statically outlined in the code What if a functional loop (while) body is identified as a task? Unrolling? Not feasible for high iteration count What if recursion is used?

94 Task parallelism Why? Example: list traversal OpenMP v2.5 EXAMPLE void traverse_list (List l) Element e ; #pragma omp parallel private ( e ) for ( e = e first; e; e = e next ) #pragma omp single nowait process ( e ) ; Awkward! Poor performance Not composable

95 Task parallelism EXAMPLE Why? Example: tree traversal void traverse_tree (Tree *tree) #pragma omp parallel sections #pragma omp section if ( tree left ) traverse_tree ( tree left ); #pragma omp section if ( tree right) traverse_tree ( tree right); process (tree); OpenMP v2.5 Too many parallel regions Extra overheads Extra synchronizations Not always well supported

96 Task parallelism Better solution for those problems Main addition to OpenMP 3.0a Allows to parallelize irregular problems unbounded loops recursive algorithms producer/consumer schemes... Ayguadé et al., The Design of OpenMP Tasks, IEEE TPDS March 2009

97 Task parallelism The OpenMP tasking model Creating tasks Data scoping Synchronizing tasks Execution model

98 What is an OpenMP task? Tasks are work units which execution may be deferred they can also be executed immediately! Tasks are composed of: code to execute data environment Initialized at creation time internal control variables (ICVs)

99 Task directive #pragma omp task [ clauses ] structured block Each encountering thread creates a task Packages code and data environment Highly composable. Can be nested inside parallel regions inside other tasks inside worksharing constructs (for, sections)

100 List traversal with tasks Why? Example: list traversal EXAMPLE void traverse_list (List l) Element e ; for ( e = e first; e; e = e next ) #pragma omp task process ( e ) ; What is the scope of e?

101 Task data scoping Data scoping clauses shared(list) private(list) firstprivate(list) data is captured at creation default(shared none)

102 Task data scoping when there are no clauses.. If no clause Implicit rules apply e.g., global variables are shared Otherwise... firstprivate shared attribute is lexically inherited

103 List traversal with tasks EXAMPLE int a ; void foo ( ) int b, c ; #pragma omp parallel shared(c) int d ; #pragma omp task int e ; a = shared b = firstprivate c = shared d = firstprivate e = private Tip default(none) is your friend Use it if you do not see it clear

104 List traversal with tasks EXAMPLE void traverse_list (List l) Element e ; for ( e = e first; e; e = e next ) #pragma omp task process ( e ) ; e is firstprivate

105 List traversal with tasks EXAMPLE void traverse_list (List l) Element e ; for ( e = e first; e; e = e next ) #pragma omp task process ( e ) ; how we can guarantee here that the traversal is finished?

106 Task synchronization Barriers (implicit or explicit) All tasks created by any thread of the current team are guaranteed to be completed at barrier exit Task barrier #pragma omp taskwait Encountering task suspends until child tasks complete Only direct childs, not descendants!

107 List traversal with tasks EXAMPLE void traverse_list (List l) Element e ; for ( e = e first; e; e = e next ) #pragma omp task process ( e ) ; #pragma omp taskwait All tasks guaranteed to be completed here

108 Task execution model Task are executed by a thread of the team that generated it Can be executed immediately by the same thread that creates it Parallel regions in 3.0 create tasks! One implicit task is created for each thread So all task-concepts have sense inside the parallel region Threads can suspend the execution of a task and start/resume another

109 Task parallelism Why? Example: list traversal EXAMPLE CAREFUL! Multiple traversal of the same list List l; #pragma omp parallel traverse_list (l); void traverse_list (List l) Element e ; for ( e = e first; e; e = e next ) #pragma omp task process ( e ) ;

110 Task parallelism Why? Example: list traversal Single traversal EXAMPLE List l; #pragma omp parallel #pragma omp single traverse_list (l); One thread enters single and creates all tasks All the team cooperates executing them void traverse_list (List l) Element e ; for ( e = e first; e; e = e next ) #pragma omp task process ( e ) ;

111 Task parallelism In case task is within a regular counted loop an alternative is to parallelize task creation among threads EXAMPLE /* A DIFFERENT EXAMPLE */ Multiple traversals Multiple threads create tasks All the team cooperates executing them #pragma omp parallel Myfunc (); void Myfunc () int i; #pragma omp for for (i=lb; i<ub; i++) #pragma omp task process ( i ) ;

112 LET S TRY IT! ALESSANDRO CAPOTONDI ALESSANDRO.CAPOTONDI@UNIBO.IT GIUSEPPE TAGLIAVINI GIUSEPPE.TAGLIAVINI@UNIBO.IT

113 EX 5 TASK PARALLELISM WITH SECTIONS void sections() work( ); printf("%hu: Done with first elaboration!\n, ); work( ); printf("%hu: Done with second elaboration!\n, ); work( ); printf("%hu: Done with third elaboration!\n", ); work( ); printf("%hu: Done with fourth elaboration!\n", ); a) Distribute workload among 4 threads using SPMD parallelization I. Get thread id II. Use if/else or switch/case to differentiate workload b) Implement the same workload partitioning with sections directive

114 EX 6 TASK PARALLELISM WITH TASK void tasks() unsigned int i; for(i=0; i<4; i++) work((i+1)* ); printf("%hu: Done with elaboration\n", ); a) Distribute workload among 4 threads using task directive I. Same program as before II. But we had to manually unroll the loop to use sections b) Compare performance and ease of coding with sections

115 EX 7 TASK PARALLELISM WITH TASK void tasks() unsigned int i; Modify the EX6 exercise code as indicated on this slide for(i=0; i<1024; i++) work( ); printf("%hu: Done with elaboration\n", ); b) Parallelize the loop with task directive Use single directive to force a single processor to create tasks c) Parallelize the loop with single directive Use nowait clause to allow for parallel execution Compare performance and ease of coding of the two solutions

116 Overview of major 4.0 additions Device constructs SIMD constructs Cancellation Task dependences and task groups Thread affinity control User-defined reductions Initial support for Fortran 2003 Support for array sections (including in C and C++) Sequentially consistent atomics Display of initial OpenMP internal control variables

117 When are tasks guaranteed to complete or at the end of taskgroup construct #pragma omp taskgroup #pragma omp task do_tasky_stuff() Might create nested tasks wait at end of construct until all tasks created in the construct, including descendants, have completed.

118 Task dependencies #pragma omp task depend(type:list) where type is in, out or inout and list is a list of variables. list may contain subarrays: OpenMP 4.0 includes a syntax for C/C++ in: the generated task will be a dependent task of all previously generated sibling tasks that reference at least one of the list items in an out or inout clause out or inout: the generated task will be a dependent task of all previously generated sibling tasks that reference at least one of the list items in an in, out or inout clause

119 Task dependencies example #pragma omp task depend (out:a)... //writes a #pragma omp task depend (out:b)... //writes b #pragma omp task depend (in:a,b)... //reads a and b The first two tasks can execute in parallel The third task cannot start until the first two are complete

120 OpenMP 4.0 provides support for a wide range of devices Use target directive to offload a region #pragma omp target [clause [[,] clause] ] Creates new data environment from enclosing device data environment Clauses support data movement and conditional offloading device supports offload to a device other than default Does not assume copies are made memory may be shared with host Does not copy if present in enclosing device data environment Does not copy if present in enclosing device data environment if supports running on host if amount of work is small Other constructs support device data environment target data places map list items in device data environment target update ensures variable is consistent in host and device

121 Several other device constructs support full-featured code Use target declare directive to create device versions #pragma omp declare target Can be applied to functions and global variables teams directive creates multiple teams in a target region #pragma omp teams [clause [[,] clause] ] Work across teams only synchronized at end of target region Useful for GPUs (corresponds to thread blocks) Use distribute directive to run loop across multiple teams #pragma omp distribute [clause [[,] clause] ] Several combined/composite constructs simplify device use

122 Example: OpenMP support for devices Jacobi iteration #pragma omp target data map(a, Anew) while (err>tol && iter < iter_max) err = 0.0; #pragma target teams #pragma omp parallel for reduction(max:err) for(int j=1; j< n-1; j++) for(int i=1; i<m-1; i++) Anew[j][i] = 0.25* (A[j][i+1] + A[j][i-1]+ A[j-1][i] + A[j+1][i]); err = max(err,abs(anew[j][i] A[j][i])); #pragma omp target teams #pragma omp parallel for for(int j=1; j< n-1; j++) for(int i=1; i<m-1; i++) A[j][i] = Anew[j]i]; iter ++; Copy A back out to host but only once Create a data region on the device. Map A and Anew onto the target device The "target teams construct tells the compiler to pick the number of teams which translates to thread blocks for CUDA.

123 OpenMP 4.0 provides portable SIMD constructs Use simd directive to indicate a loop should be SIMDized #pragma omp simd [clause [[,] clause] ] Execute iterations of following loop in SIMD chunks Region binds to the current task, so loop is not divided across threads SIMD chunk is set of iterations executed concurrently by a SIMD lanes Creates a new data environment Clauses control data environment, how loop is partitioned safelen(length) limits the number of iterations in a SIMD chunk linear lists variables with a linear relationship to the iteration space aligned specifies byte alignments of a list of variables private, lastprivate, reduction, collapse - usual meanings

124 The declare simd construct generates SIMD functions #pragma omp simd notinbranch float min (float a, float b) return a < b? a : b; #pragma omp simd notinbranch float distsq(float x, float y) return (x y) (x y); Compile library and use functions in a SIMD loop void minex (float *a, float *b, float *c, float *d) #pragma omp parallel for simd for (i = 0; i < N; i++) d[i] = min (distsq(a[i], b[i]), c[i]); Creates implicit tasks of parallel region Divides loop into SIMD chunks Schedules SIMD chunks across implicit tasks Loop is fully SIMDized by using SIMD versions of functions

125 Exercise Considering the follow pseudo code. int main() for(int i=0; i < 128; ++i) Work1(); #pragma omp parallel for schedule(static) num_threads(128) for(int i=0; i < 1024; ++i) Work2(); return 0;

126 Exercise 1. Compute the ideal speedup expected, considering: a fixed execution time of 1ms for the function Work1() and 8ms for the function Work2(), and zero overheads for the control flow of the loops and zero costs for the parallelization. 2. Write, for each thread, the set of indexes i, that they will execute on the parallel for loop, according to the schedule policy specified. (E.g. Thread 0: i=0,1...; Thread 1: i=...) 3. Compute the speedup expected considering also the following overheads: barriers take 8ms, parallel creation takes 32ms; loop overhead takes 1ms.

127 Solution (1) 1. Compute the ideal speedup expected, considering: a fixed execution time of 1ms for the function Work1() and 8ms for the function Work2(), and zero overheads for the control flow of the loops and zero costs for the parallelization. SSSSSSSSSSSSSSSSSSll tttttttt (nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnee tttttttt + ppppppppppppppppppppppppppee tttttttt) #cccccccccc 128 1mmmm mmmm 128 1mmmm = 128 8mmmm 8320mmmm 192mmmm 43.33x

128 Solution (2) 2. Write, for each thread, the set of indexes i, that they will execute on the parallel for loop, according to the schedule policy specified. (E.g. Thread 0: i=0,1...; Thread 1: i=...) Chuck_size = 1024 / 128 = 8 iterations Thread 0 = i from 0 to 7 Thread 1 = i from 8 to 15 Thread 2 = i from 16 to 23 Thread 128 = i from 1016 to 1023

129 Solution (3) 3. Compute the speedup expected considering also the following overheads: barriers take 8ms, parallel creation takes 32ms; loop overhead takes 1ms. SSSSSSSSSSSSSSSSSSll tttttttt (nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnee tttttttt + ppppppppppppppll ooooooooooooooooo + bbbbbbbbbbbbrr ooooooooooooooooo + ppppppppppppppppppppppppppee tttttttt #cccccccccc + iiiiiiiiiiiiiiiiiiss ooooooooooooooooo ) 128 1mmmm mmmm 128 1mmmm + 32mmmm + 8mmmm (8mmmm + 1mmmm) = 8320mmmm 240mmmm 34.66x

130 Conclusions What we learned about OpenMP: Control and dispatch parallelism using OpenMP interface Distribute the work among different threads Implement different parallelization scheme such as data-parallelism (SPMD) or task-parallelism (MPMD) Investigate performance efficiency Novel directions of OpenMP