Massimo Bernaschi Istituto Applicazioni del Calcolo Consiglio Nazionale delle Ricerche.

Size: px

Start display at page:

Download "Massimo Bernaschi Istituto Applicazioni del Calcolo Consiglio Nazionale delle Ricerche."

Cecilia Kelley
5 years ago
Views:

1 Massimo Bernaschi Istituto Applicazioni del Calcolo Consiglio Nazionale delle Ricerche

2 OpenMP by example } Dijkstra algorithm for finding the shortest paths from vertex 0 to the other vertices in an N-vertex undirected graph. The array G is assumed to contain the one-hop distances between vertices. Done = {0} # vertices checked so far NewDone = None # currently checked vertex NonDone = {1,2,...,N-1} # vertices not checked yet for J = 0 to N-1 Dist[J] = G(0,J) # initialize shortest-path lengths for Step = 1 to N-1 find J such that Dist[J] is min among all J in NonDone transfer J from NonDone to Done NewDone = J for K = 1 to N-1 if K is in NonDone # check if there is a shorter path from 0 to K through NewDone # than our best so far Dist[K] = min(dist[k],dist[newdone]+g[newdone,k]) 2

3 Dijkstra Animated Example

4 Dijkstra Animated Example

5 Dijkstra Animated Example

6 Dijkstra Animated Example

7 Dijkstra Animated Example

8 Dijkstra Animated Example

9 Dijkstra Animated Example

10 Dijkstra Animated Example

11 OpenMP (Open spec. for Multi Processing) } OpenMP is not a computer language } Rather it works in conjunction with existing languages such as standard Fortran or C/C++ } Application Programming Interface (API) } } that provides a portable model for parallel applications Three main components: } Compiler directives } Runtime library routines } Environment variables 11

12 OpenMP Parallelization } OpenMP is directive based } code (can) work without them } OpenMP can be added incrementally } OpenMP only works in shared memory } multi-socket nodes, multi-core processors } OpenMP hides the calls to a threads library } less flexible, but much less programming } Caution: write access to shared data can easily lead to race conditions and incorrect data 12

13 #include <omp.h> #include <stdio.h> Simple C OpenMP Program int main ( ) { } printf("starting off in the sequential world.\n"); #pragma omp parallel { printf("hello from thread number %d\n", omp_get_thread_num() ); } printf("back to the sequential world.\n"); return 0; To compile with gcc add the fopenmp flag 13

14 OpenMP Parallelization Thread-based Parallelism Explicit Parallelism Fork-Join Model Compiler Directive Based Dynamic Threads 14

15 Getting Started with OpenMP } OpenMP s constructs fall into 5 categories: } } } } } Parallel Regions Work sharing Data Environment (scope) Synchronization Runtime functions/environment variables } OpenMP is essentially the same for both Fortran and C/C++ 15

16 Directives Format } A directive is a special line of source code with meaning only to certain compilers. } A directive is distinguished by a sentinel at the start of the line. } OpenMP sentinels are: } Fortran:!$OMP (or C$OMP or *$OMP) } C/C++: #pragma omp 16

17 OpenMP: Parallel Regions } The parallel directive leaves it up to the programmer as to how to partition the work. } For example, to create a 4-thread parallel region: } each thread calls foo(id,a) for ID = 0 to 3 Each thread redundantly executes the code within the structured block thread-safe routine: A routine that performs the intended function even when executed concurrently (by more than one thread) double A[1000]; omp_set_num_threads(4); #pragma omp parallel { int ID =omp_get_thread_num(); foo(id,a); } printf( All Done\n ); 17

18 OpenMP: Parallel Regions double A[1000]; omp_set_num_threads(4); A single copy of A is shared between all threads. foo(0,a); foo(1,a); foo(2,a); foo(3,a); Threads wait here for all printf( All Done\n ); threads to finish before proceeding (i.e. barrier). 18

19 How many threads? The number of threads in a parallel region is determined by the following factors: } } } } Use of the omp_set_num_threads() library function Setting of the OMP_NUM_THREADS environment variable The implementation default Threads are numbered from 0 (master thread) to N-1. 19

20 OpenMP runtime library OMP_GET_NUM_THREADS() returns the current # of threads. OMP_GET_THREAD_NUM() - returns the id of this thread. OMP_SET_NUM_THREADS(n) set the desired # of threads. OMP_IN_PARALLEL() returns.true. if inside parallel region. OMP_GET_MAX_THREADS() - returns the # of possible threads. 20

21 Memory footprint Thread 1! Thread 2! Thread 3! PC! Private data! PC! Private data! PC! Private data! Shared data! 21

22 Memory footprint Program Private data Shared data Thread 1 Thread 2 load a add a 1 store a load a add a 1 store a 22

23 Variable Scooping } All existing variables still exist inside a parallel region } by default SHARED between all threads } But work sharing requires private variables } } PRIVATE clause to OMP PARALLEL directive } Index variable of a worksharing loop } All declared local variables within a parallel region } The FIRSTPRIVATE clause would initialize the private instances with the contents of the shared instance Be aware of the sharing nature of static variables 23

24 Scope Issues } In #pragma omp parallel " { int startv, endv, // start, end vertices for this thread step, // whole procedure goes nv steps " mymv, // vertex which attains that value." " the pragma comes before the declaration of the local variables. That means that all of them are local to each thread, i.e. not shared by them. But if a work sharing directive comes within a function but after declaration of local variables, those variables are shared in common among the threads! 24

25 Scope Issues } Keep in mind that variables which are global to the program (in the C/C++ sense) are automatically global to all threads. } To modify the default behaviour of local variables, there is the private keyword. For instance #pragma omp parallel private(x,y)" would make x and y nonshared even if they were declared above the directive line. } The firstprivate variant allow to initialize x and y with the values that were shared before the directive. } The lastprivate variant updates variables using value from last iteration. } In general, the value of the private copies is uninitialized. 25

26 single and barrier directives } With the single directive just one thread executes some code within a parallel or other work sharing block. } There is no control on which thread executes the region marked as single. } Besides the explicit barrier defined by the corresponding barrier directive, there is an implicit barrier at the end of each } single, parallel, for and sections block. This behaviour can be overridden via the nowait clause, #pragma omp for nowait " but be very careful with it! 26

27 OpenMP: How do Threads Interact? } } } } OpenMP is a shared memory model. } Threads communicate by sharing variables. Unintended sharing of data can lead to race conditions: } race condition: when the program s outcome changes as the threads are scheduled differently. To control race conditions: } Use synchronization to protect data conflicts. Synchronization is expensive so: } Change how data is stored to minimize the need for synchronization. 27

28 OpenMP: How do Threads Interact? Note that updates to shared variables: (e.g. a = a + 1;) are not atomic! If two threads try to do this at the same time, one of the updates may get overwritten. 28

29 Program Private data Shared data Thread 1 Thread 2 load a add a 1 store a load a add a 1 store a 29

30 Critical } Only one thread at a time can enter a critical section. #include <omp.h> main() { int x; x = 0; #pragma omp parallel shared(x) { #pragma omp critical x = x + 1; } /* end of parallel section */ 30

31 Atomic } atomic is a special case of a critical section that can be used for certain simple statements #include <omp.h> int count; void Tick() { // Replace lock annotations } #pragma omp atomic count = count+1; The eligible C operators for the atomic clause are ++, --, +=, *=, <<=, &=, = 31

32 Work-Sharing Constructs } Divides the execution of the enclosed code region among the members of the team that encounter it. } Work-sharing constructs do not launch new threads. } No implied barrier upon entry to a work sharing construct. } However, there is an implied barrier at the end of the work sharing construct (unless nowait is used). 32

33 Work Sharing Constructs - example Sequential code OpenMP // Region OpenMP Parallel Region and a worksharing for construct for(i=0;i<n;i++) { a[i] = a[i] + b[i];} #pragma omp parallel { int id, i, Nthrds, istart, iend; id = omp_get_thread_num(); Nthrds = omp_get_num_threads(); istart = id * N / Nthrds; iend = (id+1) * N / Nthrds; for(i=istart;i<iend;i++) {a[i]=a[i]+b[i];} } #pragma omp parallel #pragma omp for schedule(static) for(i=0;i<n;i++) { a[i]=a[i]+b[i];} 33

34 Exploiting Loop Level Parallelism Loop level Parallelism: parallelize only loops. Iterations must be indipendent, i.e., one iteration cannot depend on the result of another. Easy to implement Highly readable code Less than optimal performance (sometimes) Most often used 34

35 } In general in a loop on i, there is no predicting which threads will do which values of i, in which order (but see later). } OpenMP treats the loop index as private even if by context it would be shared. } If there are nested loops, by default the pragma applies only to the outer loop. } It is possible to insert another for pragma inside, to parallelize the inner loop(s). } Starting with OpenMP version 3.0, one can use the collapse clause: #pragma omp parallel for collapse(2) to specify two levels of nesting in the assignment of threads to tasks. 35 Exploiting Loop Level Parallelism

36 Exploiting Loop Level Parallelism Parallel Loop Directives } C\C++ for loop directive } #pragma omp for } This directive does not create a team of threads but assume there has already been a team forked. } If not inside a parallel region shortcuts can be used. } #pragma omp parallel for 36

37 Exploiting Loop Level Parallelism Parallel Loop Directives /2 } These are equivalent to a parallel construct followed immediately by a worksharing construct. #pragma omp parallel for Same as #pragma omp parallel... #pragma omp for 37

38 How is OpenMP Typically Used? OpenMP is mostly used to parallelize loops Split-up this loop between multiple threads void main() { double Res[1000]; for(int i=0;i<1000;i++) { do_huge_comp(res[i]); void main() { double Res[1000]; #pragma omp parallel for for(int i=0;i<1000;i++) { do_huge_comp(res[i]); } } Sequential program } } Parallel program 38

39 39 How is OpenMP Typically Used?

40 schedule(sta6c [,chunk]) Iterations are divided evenly among threads If chunk is specified, divides the work into chunk sized parcels If there are N threads, each thread does every N th chunk of work. #pragma omp parallel for schedule(static,3) for (int j = 1; j<=36; j++) { work (j) } 40

41 schedule(dynamic [,chunk]) Divides the workload into chunk sized parcels. As a thread finishes one chunk, it grabs the next available chunk. Default value for chunk is one. More overhead, but potentially better load balancing. #pragma omp parallel for schedule(dynamic,1) for (int j = 1; j<=36; j++) { work (j) } 41

42 } The Clause SCHEDULE (type [,chunk]) The schedule clause effects how loop iterations are mapped onto threads 42 schedule(static [,chunk]) } Deal-out blocks of iterations of size chunk to each thread schedule(dynamic [,chunk]) } Each thread grabs chunk iterations off a queue until all iterations have been handled schedule(guided [,chunk]) } Threads dynamically grab blocks of iterations. The size of the block starts large and shrinks down to size chunk as the calculation proceeds schedule(runtime) } Schedule and chunk size taken from the OMP_SCHEDULE environment variable schedule(auto) } Schedule is left up to the runtime to choose (does not have to be any of the above).

43 The Clause SCHEDULE (type [,chunk]) Note that setting the chunk size in the schedule() clause is a compile-time operation If you wish to have the chunk size set at run time, call omp set schedule() in conjunction with the runtime clause.... n = atoi(argv[1]); int chunk = atoi(argv[2]); omp_set_schedule(omp_sched_static, chunk); #pragma omp parallel {..." #pragma omp for schedule(runtime) } for (i = 1; i < n; i++) {... } Or set the OMP_SCHEDULE environment variable. 44

44 The Clause SCHEDULE (type [,chunk]) } What is a good value for the chunk? } } } Large chunks are good, due to there being less overhead but If chunk sizes are large, then toward the end of the work, some threads may be working on their last chunks while others have finished and are now idle It would be nice to have large chunks at the beginning of the loop, to reduce the overhead, but smaller chunks at the end. That is the rationale of the guided clause. 45

45 No Wait Clauses nowait: if specified then threads do not synchronise at the end of the parallel loop. } Note that the nowait clause is incompatible with a simple parallel region meaning that using the composite directives will not allow you to use the nowait clause. 46

46 Suppose to have int z; #pragma omp for reduction(+:z) OpenMP: reduction clause for(i=0;i<n;i++) z+=x[i]; the threads will share the work but there will be independent copies of z maintained for each thread, each initialized to 0 before the loop begins When the loop is entirely done, the values of z from the various threads will be summed, of course in an atomic manner. 47

47 } OpenMP: Reduction(op : list) The variables in list must be shared in the enclosing parallel region. } Inside a parallel or a worksharing construct: } } } } A local copy of each list variable is made and initialized depending on the op (e.g., 0 for + ; 1 for * ) pair wise op is updated on the local value Local copies are reduced into a single global copy at the end of the construct. One can specify several reduction variables to the right of the colon, separated by commas. 48

48 OpenMP: A Reduction Example #include <omp.h> #define NUM_THREADS 2 void main () { int i; double ZZ, func(), sum=0.0; } omp_set_num_threads(num_threads); #pragma omp parallel for reduction(+:sum) private(zz) for (i=0; i< 1000; i++){ ZZ = func(i); sum = sum + ZZ; } 49

OpenMP: reduction } Using the reduction clause is not only convenient but also efficient because by maintaining separate copies until the loop is done, the number of serializing atomic actions

49 OpenMP: reduction } Using the reduction clause is not only convenient but also efficient because by maintaining separate copies until the loop is done, the number of serializing atomic actions is reduced. Otherwise we had to use, for instance, critical or atomic clauses. In C/C++, you can use reduction with +, -, *, &,, && and and ^ min and max operators for C starting on version

50 if CLAUSE We can make the parallel region directive itself conditional. Fortran: IF (scalar logical expression) C/C++: if (scalar expression) #pragma omp parallel if (tasks > 1000) { while(tasks > 0) donexttask(); } 51

51 Exercise: compute PI Integrate, i.e determine area under function numerically using slices of h * f(x) at midpoints 52

52 } 53 Exercise: compute PI //#include <omp.h> //uncomment if you plan to use openmp #include <stdlib.h> #include <stdio.h> #include <math.h> #define MINARG 2 int main(int argc, char *argv[]) { int i, num_steps; double pi, x, step, sum=0.0; if(argc<minarg) { fprintf(stderr,"usage: %s num_steps\n",argv[0]); exit(1); } num_steps=atoi(argv[1]); step=1.0/num_steps; for(i=0;i<num_steps; i++) { x=(i+0.5)*step; sum=sum+4.0/(1.0+x*x); } pi=step*sum; printf("pi=%12.9f (expected %12.9f)\n",pi,acos(-1.0)); return 0; To compile with gcc add the fopenmp flag

53 54 Exercise: compute PI

54 } Compute PI: why poor scaling? If independent data elements happen to sit on the same cache line, each update will cause the cache lines to slosh back and forth between threads (remember false sharing ) (possible) Solution: pad arrays so elements you use are on distinct cache lines. Alternatives? 55

55 } Suppose you have: int a=1, b=1, c=1; OpenMP: Data sharing test #pragma omp parallel private(a) firstprivate(c) } Are a, b, c local to each thread or shared inside the parallel region? } What are their initial values inside and values after the parallel region? 56

56 OpenMP: simple list traversal } How can we run in parallel a list traversal like p=head; while (p) { process(p); } p = p->next; The loop worksharing construct only works with loops for which the number of loop iterations can be represented by a closed-form expression at compiler time. While loops are not covered. 57

57 Simple list traversal: a naïve solution while (p!= NULL) { p = p->next; count++; } p = head;" for(i=0; i<count; i++) { parr[i] = p; p = p->next; } #pragma omp parallel { #pragma omp for schedule(static,1) for(i=0; i<count; i++) } processwork(parr[i]); 58

58 Simple list traversal: a C++ STL solution std::vector<node *> nodelist;" for (p = head; p!= NULL; p = p->next) nodelist.push_back(p); int j = (int)nodelist.size();" #pragma omp parallel for schedule(static,1) for (int i = 0; i < j; ++i) processwork(nodelist[i]); 59

59 OpenMP: task directive } New to OpenMP 3.0. The basic idea is to set up a task queue. } When a thread encounters a task directive, it arranges for some thread to execute the associated block at some time. } The task might not execute right away; it may have to wait for some thread to become free after finishing another task. } We could set up our own work queue, as a shared variable, and write our code so that in an atomic way: } whenever a thread finished a unit of work, it would delete the head of the queue; } whenever a thread generated a unit of work, it would add it to the queue. 60

60 61 } Tasks are composed of: } Code to execute } Data environment OpenMP: task directive } Internal Control Variables (ICV) } Task definitions: } Task construct: task directive plus structured block } Task region: the dynamic sequence of instructions produced by the execution of a task by a thread

61 } } OpenMP: task directive Tasks are guaranteed to be complete at thread barriers: #pragma omp barrier Or task barriers #pragma omp taskwait " 62

62 OpenMP: task and data scoping } A task s private variables are undefined outside the task " 63

63 OpenMP: task and data scoping (solution) 64

64 OpenMP: list traversal with tasks #pragma omp parallel #pragma omp single { for(p=head; p; p=p->next) #pragma omp task } process(p); Does this work fine? What is the value of p? 65

65 OpenMP: list traversal with tasks #pragma omp parallel #pragma omp single { For(p=head ; p ; p=p->next) #pragma omp task firstprivate(p) } process(p); 66

66 67 OpenMP: list traversal with tasks

67 68 OpenMP: advantages of using tasks

68 OpenMP: master construct } The master construct denotes a structured block that is only executed by the master thread. } The other threads just skip it #pragma omp parallel { do_many_things(); #pragma omp master { exchange_boundaries(); } #pragma omp barrier do_many_other_things(); } 69

69 OpenMP: sections construct } The sections worksharing construct gives a different structured block to each thread #pragma omp parallel { #pragma omp sections {" #pragma omp section x_calculation(); #pragma omp section y_calculation(); #pragma omp section } } z_calculation(); 70

70 OpenMP: flush directive } OpenMP takes a relaxed consistency approach to the problems of memory consistency } } OpenMP forces updates to memory at all synchronization points, i.e., at: } barrier } entry/exit to/from critical } entry/exit to/from ordered } entry/exit to/from parallel } exit from parallel for } exit from parallel sections } exit from single In between synchronization points, one can force an update to a variable x via the flush pragma: #pragma omp flush (x) The efficiency of the flush operations depends on the platform. The flush operation does not actually synchronize different threads. It just ensures that a thread s values are made consistent with main memory. 71

71 OpenMP: locks functions } There are still some instances in which lock variables may be useful. OpenMP does provide for locks: } Declare the locks to be of type omp_lock_t } omp set lock() to lock the lock } omp_unset_lock() to unlock } omp_test_lock() try to lock (non-blocking) A lock implies a memory fence (a flush ) of all thread visible variables 72

on the top-left All known OpenMP implementations use a thread pool so full

72 OpenMP: what the compiler does } } An OpenMP c o m p i l e r generates code logically analogous to that on the right, given an OpenMP pragma such as that on the top-left All known OpenMP implementations use a thread pool so full cost of threads creation and destruction is not incurred for reach parallel region. 73

73 OpenMP: fine tuning } Looking at the original Dijkstra code we could (try to) eliminate the critical section. } in each iteration, the threads compute their local minimum distance values md and mv, and then update the global values md and mv in a critical section } we could have the threads store their values mymd and mymv in a global array mymins, with each thread using a separate pair of locations within that array, and then at the end of the iteration we could have just one task scan through mymins and update md and mv. } In general, synchronization is expensive so } Change how data is accessed to minimize the need for synchronization. } Other ideas: } Apply refinements like the schedule modifier } Check for possible false-sharing issues. 74

Shared Memory Programming Paradigm!

Shared Memory Programming Paradigm! Ivan Girotto igirotto@ictp.it Information & Communication Technology Section (ICTS) International Centre for Theoretical Physics (ICTP) 1 Multi-CPUs & Multi-cores NUMA