2

Transcription

1 1

2 2

3 3

4 4

5 5

6 Code transformation Every time the compiler finds a #pragma omp parallel directive creates a new function in which the code belonging to the scope of the pragma itself is moved The directive is replaced with a call to a runtime function that is responsible for forking new threads (in a threadbased implementation) or for loading parallel code onto the slave processors Once the parallel region has been executed threads/processors need to synchronize. 6

7 GCC PARSER DUMP The parser splits the parallel for directive in two separate directives It recognizes which variables must be shared and which can be private to each processor 7

8 GCC OPENMP EXPANSION DUMP struct struct { { int[10] int[10] * * a; a; int[10] int[10] * * b; b; int[10] int[10] * * c; c; } }.omp_data_o.3;.omp_data_o.3; Make shared data visible to all processors/threads call runtime to join workers call runtime to wake-up slave threads and run parallel code on them pointer to parallel function call parallel function on master thread Es #1 - pthreads Current implementation of GCC OpenMP runtime environment (libgomp) is basically a wrapper around the pthreads library. Master forks new worker threads with a call to the runtime function GOMP_parallel_start After parallel region master joins workers with a call to the runtime function GOMP_parallel_end 8

9 Es #1 - pthreads Current implementation of GCC OpenMP runtime environment (libgomp) is basically a wrapper around the pthreads library. If num_threads = 0 determine number of worker threads Master forks new worker threads with a call to the runtime function GOMP_parallel_start Fork worker threads After parallel region master joins workers with a call to the runtime function GOMP_parallel_end Wait for all threads to be ready before starting parallel region Es #1 - pthreads Current implementation of GCC OpenMP runtime environment (libgomp) is basically a wrapper around the pthreads library. If num_threads = 0 We need to synchronize determine threads with number a barrier of worker at threads Master forks new worker the end threads of a parallel with region a call to the runtime function GOMP_parallel_start Join worker threads and Fork worker threads After parallel region master joins workers with a call to suspend them the runtime function GOMP_parallel_end Wait for all threads to be ready before starting parallel region 9

10 Es #2 - MPARM runtime library void main() { initenv(); if if (cpuid == == MASTER) { // // gather workers on on barrier start(); // // release workers } else { // // spin until work provided parallel_routine(); // // spin until work provided } } void doall() { // // release workers parallel_routine(); // // gather workers on on barrier } // // Synchronization facilities // // Lock Implementation // // Barrier Implementation parallel code void parallel_routine() { for (i=n*cpuid/nprocs; i<n*(cpuid+1)/nprocs; i++) for (j=i; j<n; j++) A[i][j] = 1.0; } int start() { // sequential code for (i=0; i<n; i++) for do_all(); (j=i; j<n; j++) A[i][j] = 1; // sequential code } 10

11 11

12 GCC OPENMP EXPANSION DUMP struct struct { { int[10] int[10] * * a; a; int[10] int[10] * * b; b; int[10] int[10] * * c; c; } }.omp_data_o.3;.omp_data_o.3; Make shared data visible to all processors/threads Replace Replaceuses usesof of shared sharedvariables with withcorresponding field fieldin in shared shareddata data struct struct 12

13 GCC OPENMP EXPANSION DUMP Call Callruntime runtimetoto determine determinenumber number of of threads threads Call Callruntime runtimetoto determine determinethread threadidid Create work sharing by splitting loop iterations between threads GCC OPENMP EXPANSION DUMP COMPUTE LOWER AND UPPER BOUNDS FOR EACH THREAD HOW? It depends on what the schedule clause specifies (see after) Initialize induction variable to lower bound Create work sharing by splitting loop iterations between threads Check termination condition on upper bound 13

14 14

15 15

16 16

17 Variable tmp is only accessible by the master thread. It has an initial value that slaves need to know Slave threads will have a private copy of tmp initialized with this value 17

18 Variable tmp is only accessible by the master thread. It has an initial value that slaves The initneed valueto isknow made visible to slaves through the shared data struct Slave threads will have a private copy of tmp initialized with this value Variable tmp is only accessible by the master thread. It has an initial value that slaves The initneed valueto isprivate know made copy of tmp is visible to slaves through initialized the with this value shared data struct Slave threads will have a private copy of tmp initialized with this value 18

19 Variable tmp is only accessible by the master thread. Its value will be written by the last thread that works on its private copy 19

20 After all iterations have been executed.. Variable tmp is only accessible by the master thread. Its value will be written by the last thread that works on its private copy local value of tmp is copied into the shared data struct.. After all iterations have been and through this copied executed.. into the master s copy of tmp Variable tmp is only accessible by the master thread. Its value will be written by the last thread that works on its private copy local value of tmp is copied into the shared data struct.. 20

21 21

22 22

23 23

24 THIS IS DONE AT EVERY LOOP ITERATION! call runtime to acquire lock Lock-protected operations call runtime to release lock 24

25 25

26 26

27 UNOPTIMIZED CODE Shared Sharedvariable variableisis updated updatedat at every every iteration. iteration. This Thisisis NOT NOT necessary necessary Shared Sharedvariable variableisis only only updated updatedat at the the end end of of the the loop, loop, when whenits itsfinal value value is isknown UNOPTIMIZED CODE Shared Sharedvariable variableisis updated updatedat at every every iteration. iteration. This Thisisis NOT NOT necessary necessary This Thisisis a single single atomic atomic write. write. Target Target architecture architecture may mayprovide providesuch suchan an instruction instruction sync_fetch_and_add(&.omp_data_i->area, area); Shared Sharedvariable variableisis only only updated updatedat at the the end end of of the the loop, loop, when whenits itsfinal value value is isknown 27

28 28

29 Check condition to determine whether to parallelize the loop or not If it is true set NTHR = 0, otherwise set it to 1 Pass NTHR to runtime: if it equals 1 only one thread will execute it. 29

30 30

31 USING THE schedule CLAUSE A parallel region has at least one barrier at its end, and may have additional barriers within it At each barrier the other members of the team must wait for the last thread to arrive To minimize this wait time shared work should be distributed so that all threads arrive at the barrier at about the same time The choice of a schedule for a for construct is also determined by characteristics of the memory system (presence of caches, uniform access times, etc.) USING THE schedule CLAUSE A parallel region ASSIGNING has at SAME leastiterations one barrier TO at SAME its end, and may have additional THREADS barriers MAY IMPROVE withindata REUSE At each barrier #pragma the other omp members parallel of the team must wait for the last thread to arrive { To minimize this#pragma wait time ompshared for schedule work (static) should be distributed so that for all (i=0; threads i<n; i++) arrive at the barrier at about the same time a[i] = work1(i); The choice of a #pragma schedule omp for a schedule for construct (static) is also determined by characteristics for (i=0; i<n; i++) of the memory system (presence of caches, uniform access times, etc.) } if (i>=k) a[i] = work2(i); 31

32 32

33 SPLITTING LOOP ITERATIONS schedule (static) The static schedule is appropriate for a parallel region containing a single for construct, with each iteration requiring the same amount of work Loop boundaries are statically determined at compile time. No interaction with the runtime is required. Es. N=10, Nthr=4. DATA CHUNK TID C = ceil ( N ) Nthr 3 vector elements LB = C * TID UB = MIN { [C * ( TID + 1) ], N} NOTE: There may be LOAD IMBALANCE between threads (i.e. they operate on data chunks of different sizes) GCC OPENMP EXPANSION DUMP Compute chunk C = ceil (N/Nthr) Compute LB = C * TID Compute UB = min [C * (TID + 1), N] 33

34 schedule (static, C) IN GENERAL: Small chunks allow finer grained control on workload Specifying a size for data chunks the loop is statically split between threads in an interleaved fashion 34

35 SPLITTING LOOP ITERATIONS schedule (dynamic, C) The dynamic schedule is appropriate for the case of a for construct with the iterations requiring varying, or even unpredictable, amounts of work Iterations are assigned one at a time to threads as they become available. This requires a strict cooperation with the runtime Runtime overhead can be reduced by specifying a chunk size k greater than 1, so that threads are assigned k at a time until fewer than k remain 35

36 SPLITTING LOOP ITERATIONS schedule (dynamic, C) Call runtime Iteration step The dynamic schedule is appropriate for the case of a forchunk construct sizewith the iterations requiring varying, or even unpredictable, amounts of work Iterations are assigned one at a time to threads as they become available. This requires a strict cooperation with the runtime Runtime overhead can be reduced by specifying a chunk size k greater than 1, so that threads Absolute are assigned lower and k at upper a time bounds until fewer than k remain Retrieve lower and upper bounds for current thread s first iteration SPLITTING LOOP ITERATIONS schedule (dynamic, C) The dynamic schedule is appropriate for the case of a for construct with the iterations requiring varying, or even unpredictable, amounts of work Iterations are assigned one at a time to threads as they become available. This requires a strict cooperation with the runtime Runtime overhead can be reduced by specifying a chunk size k greater than 1, so that threads Execute are loop assigned body kover at a time this iteration until fewer space.. than k remain Retrieve lower and upper bounds for current thread s first iteration..then compute next chunk s iteration space 36

37 SPLITTING LOOP ITERATIONS schedule (guided, C) The guided schedule is appropriate for the case in which the threads may arrive at varying times at a for construct with each iterations requiring about the same amounts of work This can happen if, for example, the for construct is preceded by one or more for constructs with nowait clauses The interaction with the runtime works much like the dynamic schedule, but the size of chunks is computed dividing remaining iterations among threads, and considering C as a minimum size for the chunk 37

38 38

39 39

40 40

41 41

42 42

43 43

44 44

45 45

46 46

47 47

48 ..specifying number of sections.. Call runtime to obtain consecutive scheduling ids switch these ids to make execution jump to the code corresponding to the relative #pragma omp section 48

49 ..specifying number of sections.. Call runtime to obtain consecutive scheduling ids switch these ids to make execution jump to the code corresponding to the relative #pragma omp section 49

50 50