Fundamentals of OmpSs

Size: px

Start display at page:

Download "Fundamentals of OmpSs"

Rosalind O’Brien’
5 years ago
Views:

1 Fundamentals of OmpSs Tasks and Dependences Xavier Teruel New York, June 2013

2 AGENDA: Fundamentals of OmpSs Tasking and Synchronization Data Sharing Attributes Dependence Model Other Tasking Directives Clauses Taskwait Synchronization Outlined Task Syntax Memory Regions, Nesting and Dependences Memory regions and dependences Nested tasks and dependences Using dependence sentinels Paralllel Programming Methodology 2

3 TASKING AND SYNCHRONIZATION

execute to completion independent of other tasks Can be declared inlined or outlined Creating tasks: task directive #pragma omp task [

4 Task Directive Task description Computation unit. Amount of work (granularity) may vary in a wide range (μsecs to msecs or even seconds), may depend on input arguments, Once started can execute to completion independent of other tasks Can be declared inlined or outlined Creating tasks: task directive #pragma omp task [ default( ) ] [ shared( ) ] [ private( ) ] [ firstprivate( ) ] [ in(...) ] [ out(...) ] [ inout (...) ] [ concurrent (...) ] [ commutative ( ) ] [ priority( ) ] [ label( ) ] [ if( ) ] [ final( ) ] [untied] { structured-block

5 Data Sharing Attributes Task directive (and data sharing clauses) #pragma omp task [ default( ) ] [ shared( ) ] [ private( ) ] [ firstprivate( ) ] { structured-block Data Sharing Attributes default( private firstprivate shared none ) shared(var-list): shares the original symbol private(var-list): privatizes the original symbol firstprivate(var-list): privatizes symbol and capture value

6 Data Sharing Attributes (example) Vector initialization using a task int main( int argc, char *argv[] ) { int i, N = 100; int X[100]; #pragma omp task private(i) shared(x) \\ firstprivate(n) out(x) for (i=0; i < N; i++) X[i] = i ; #pragma omp taskwait on(x) for (i=0; i < size; i++) printf( (%d),x[i]);

..) ] { structured-block Data dependences in(var-list): task will read the contents of the variables out(var-list): task

7 Dependences: in( ), out( ), inout( ) ~ 1 Task directive (and dependence clauses) #pragma omp task [ in(...) ] [ out(...) ] [ inout (...) ] { structured-block Data dependences in(var-list): task will read the contents of the variables out(var-list): task will write into the variable s contents inout(var-list): task will read first and then write #pragma omp task out(x) x = 5; #pragma omp task in(x) printf("%d \n", x ) ; #pragma omp task inout(x) x = x + 1; antidependence printf(x) x=5 x=x+1

Dependences: in( ), out( ), inout( ) ~ 2 Tasks and non taskified

task directives: synchronization required (task / taskwait on) int

$task in(x) printf( first : %d\n", x ) ; #pragma omp task inout(x)$ $second : %d\n", x ) ; Synchronization with previous tasks needed.$

8 Dependences: in( ), out( ), inout( ) ~ 2 Tasks and non taskified code Before task directives: it is already executed no problem After task directives: synchronization required (task / taskwait on) int foo () { int x = 0; This code is already executed before creating other tasks x=5 #pragma omp task out(x) x = 5; printf(x) #pragma omp task in(x) printf( first : %d\n", x ) ; #pragma omp task inout(x) antidependence x=x+1 x = x + 1; #pragma omp task in(x) printf( second : %d\n", x ) ; Synchronization with previous tasks needed. x = {0 5 6? printf(x)

$omp task out(x) x = 5; #pragma omp task in(x) printf( first : %d\n", x ) ;$ $in(x) printf( second : %d\n", x y ) ; antidependence printf(x) x=5 x=x+1$

9 Dependences: in( ), out( ), inout( ) ~ 3 Removing user s program (anti)-dependences User can also rename some variables to break dependences Only possible with WaR and WaW dependences int foo () { int x = 0; #pragma omp task out(x) x = 5; #pragma omp task in(x) printf( first : %d\n", x ) ; #pragma omp task inout(x) in(x) out(y) x y = x + 1; #pragma omp task in(y) in(x) printf( second : %d\n", x y ) ; antidependence printf(x) x=5 x=x+1 printf(x) Rename x to y

10 Dependences: concurrent ~ 1 Task directive (and concurrent clause) #pragma omp task [ concurrent(...) ] { structured-block concurrent(var-list): task can run in parallel with other concurrent s Usually employed as concurrent updates (inout) on var-list items Following dependences on var-item will wait for all concurrent tasks Task may require additional synchronization (atomic, mutex, )

$Dependences: concurrent ~ 2 Concurrent example void foo() { int local, total=0 ; for (int j=0; j<n; j+=bs) { #pragma omp task private(local) \\ in(vec[j;bs]) concurrent(total) { local = 0 ; for$

11 Dependences: concurrent ~ 2 Concurrent example void foo() { int local, total=0 ; for (int j=0; j<n; j+=bs) { #pragma omp task private(local) \\ in(vec[j;bs]) concurrent(total) { local = 0 ; for (int i=0; i<bs; i++) local += vec[i] ; #pragma omp atomic total += local ; #pragma omp task in(total) printf ( TOTAL is %d\n, total) ; N=12 BS= Task #1 Task #2 Task #3 printf

12 Dependences: commutative ~ 1 Task directive (and commutative clause) #pragma omp task [ commutative(...) ] { structured-block commutative(var-list): tasks can run in any order (but not in parallel) Usually employed as non-ordered updates (inout) on var-list items Following dependences on var-item will wait for all commutative tasks Task DO NOT require additional synchronization (atomic, mutex, )

$j+=bs) { #pragma omp task private(local) \\ in(vec[j;bs]) commutative(total) { local = 0 ; for (int$ $i=0; i<bs; i++) local += vec[i] ; total += local ; #pragma omp task in(total) printf ( TOTAL is %d\n,$

13 Dependences: commutative ~ 2 Commutative example void foo() { int local, total=0 ; for (int j=0; j<n; j+=bs) { #pragma omp task private(local) \\ in(vec[j;bs]) commutative(total) { local = 0 ; for (int i=0; i<bs; i++) local += vec[i] ; total += local ; #pragma omp task in(total) printf ( TOTAL is %d\n, total) ; N=12 BS= Task #a Task #1 Task #b Task #c Task #2 Task #3 Task #d printf

14 Task Directive: task priorities Task Directive (and priority clause) #pragma omp task [ priority( ) ] { structured-block priority(value): the higher the best All ready tasks are inserted in an ordered ready queue Once a thread becomes idle gets one of the highest priority tasks User must choose a priority scheduler (in order to honor them) 16

15 Task Directive: labeling tasks Task Directive (and label clause) #pragma omp task [ label( ) ] { structured-block label(identifier): Used for instrumentation purposes Tasks will be named by this identifier instead of compiler given name 17

16 Task Directive: if and final clauses Task Directive (and if clause) #pragma omp task [ if( ) ] { structured-block if(expr): If expression evaluates to false, do not create the task Throttle (cutoff) mechanism to control task creation Task code will be executed immediately (not deferred) Task Directive (and final clause) #pragma omp task [ final( ) ] { structured-block final(expr): If expression evaluates to false, do not create more tasks in the enclosed task context 18

Task Directive: if and final clauses (example) Creating (or not)

{ for (int i=0; i<n; i++) { #pragma omp task if ( size[i] >

{4,23,84 * {54,78,,16 * * * {83,12 {34,32,92,64 {24,29,,27 Using

fibonacci ( int n ) { if ( n < 2 ) return n ; #pragma omp task

task final ( n < CUTOFF ) int y = fibonacci ( n 2 ) ; return x +

17 Task Directive: if and final clauses (example) Creating (or not) a task using if clause void foo(int *size, int **vector, int N) { for (int i=0; i<n; i++) { #pragma omp task if ( size[i] > MIN_SIZE ) compute(vector[i], size[i]) ; * {4,23,84 * {54,78,,16 * * * {83,12 {34,32,92,64 {24,29,,27 Using final tasks as cutoff mechanism to control granularity void fibonacci ( int n ) { if ( n < 2 ) return n ; #pragma omp task final ( n < CUTOFF ) int x = fibonacci ( n 1 ) ; #pragma omp task final ( n < CUTOFF ) int y = fibonacci ( n 2 ) ; return x + y ;

Task Directive: untied Task Directive (and untied clause) #pragma omp task [ final( ) ] { structured-block Task-Pool untied: Task will not be tied to any thread Tied tasks: (default) once the task is

18 Task Directive: untied Task Directive (and untied clause) #pragma omp task [ final( ) ] { structured-block Task-Pool untied: Task will not be tied to any thread Tied tasks: (default) once the task is executed (for the first time) in one thread it will be tied to this thread, and (if suspended) will be only resumed in this thread. Untied tasks: can be suspended and resumed in any thread (thread switch) Global Thread Team 20

$taskwait [ on( ) ] Taskwait with no clause (wait for all created/children tasks) void traverse_queue ( Queue q ) { Element e ; for ( e$ = q->first; e ; e = e->next ) { #pragma omp task process ( e ) ; #pragma omp taskwait return; process return process process process

= q->first; e ; e = e->next ) { #pragma omp task process ( e ) ; #pragma omp taskwait return; process return process process process

19 Taskwait Synchronization ~ 1 Taskwait directive Suspends the current task until children / dependences are completed #pragma omp taskwait [ on( ) ] Taskwait with no clause (wait for all created/children tasks) void traverse_queue ( Queue q ) { Element e ; for ( e = q->first; e ; e = e->next ) { #pragma omp task process ( e ) ; #pragma omp taskwait return; process return process process process process Without taskwait the subroutine will return immediately after spawning the tasks allowing the calling function to continue spawning tasks

$printf( first : %d\n", x ) ; #pragma omp task in(x) out(y) y = x + 1 ;$ $#pragma omp task in(y) printf( second : %d\n", y ) ; #pragma omp taskwait$

20 TODO: Taskwait Synchronization ~ 2a Taskwait on data dependences on(var-list): list of variables to wait-on before proceed int foo () { int x = 0 ; #pragma omp task out(x) x = 5 ; #pragma omp task in(x) printf( first : %d\n", x ) ; #pragma omp task in(x) out(y) y = x + 1 ; #pragma omp task in(y) printf( second : %d\n", y ) ; #pragma omp taskwait return ; We don t want to create a task here int x = 0 return printf(x) printf(x) x=5 y=x+1

$in(x) printf( first : %d\n", x ) ; #pragma omp taskwait on(x) y = x + 1 ; #pragma omp task in(y) printf( second : %d\n", y ) ; #pragma omp taskwait return ; int x = 0 printf(x) y = x + 1 return$

21 TODO: Taskwait Synchronization ~ 2b Taskwait on data dependences on(var-list): list of variables to wait-on before proceed int foo () { int x = 0 ; #pragma omp task out(x) x = 5 ; #pragma omp task in(x) printf( first : %d\n", x ) ; #pragma omp taskwait on(x) y = x + 1 ; #pragma omp task in(y) printf( second : %d\n", y ) ; #pragma omp taskwait return ; int x = 0 printf(x) y = x + 1 return printf(x) x=5

22 Task Directive in Functions Task directive attached to a function definition/declaration #pragma omp task [ default( ) ] [ shared( ) ] [ private( ) ] [ firstprivate( ) ] [ in(...) ] [ out(...) ] [ inout (...) ] [ concurrent (...) ] [ commutative ( ) ] [ priority( ) ] [ label( ) ] [ if( ) ] [ final( ) ] [untied] { function-definition or { function-declaration All Function invocations become a task Data sharing attributes not needed anymore Scalar parameters firstprivate equivalent Memory pointers parameters shared equivalent Function local variables private equivalent Remaining clauses keep its meaning

$*argv[] ) { int i, N = 100; int X[100]; #pragma omp task private(i) shared(x) \\ firstprivate(n) out(x) for (i=0; i < N; i++) X[i] = i ; Function s tasks$ #pragma omp task out(y) void foo ( int Y[], int size ) { int j; for (j=0; j < size; j++) Y[j] = j; int main( int argc, char *argv[] ) { int i, N = 100; int

#pragma omp task out(y) void foo ( int Y[], int size ) { int j; for (j=0; j < size; j++) Y[j] = j; int main( int argc, char *argv[] ) { int i, N = 100; int

23 Task Directive in Functions (example) Outline code s tasks Private Locals variables Shared Pointers, arrays Firstprivate Arguments int main( int argc, char *argv[] ) { int i, N = 100; int X[100]; #pragma omp task private(i) shared(x) \\ firstprivate(n) out(x) for (i=0; i < N; i++) X[i] = i ; Function s tasks #pragma omp task out(y) void foo ( int Y[], int size ) { int j; for (j=0; j < size; j++) Y[j] = j; int main( int argc, char *argv[] ) { int i, N = 100; int X[100]; foo (X, N) ; #pragma omp taskwait on(x) for (i=0; i < size; i++) printf( (%d),x[i]); #pragma omp taskwait on(x) for (i=0; i < size; i++) printf( (%d),x[i]);

24 MEMORY REGIONS, NESTING AND DEPENDENCES

$..) ] { structured-block Data dependences {in out inout(region-list): task will read/write the specified region int A[8][8] ; #pragma omp task out(a[0;4][0;3]) m_init_1(a,0,0,3,2, 8 ) ; #pragma omp$

25 Region Dependences: in( ), out( ), inout( ) ~ 1 Task directive (and region dependence clauses) #pragma omp task [ in(...) ] [ out(...) ] [ inout (...) ] { structured-block Data dependences {in out inout(region-list): task will read/write the specified region int A[8][8] ; #pragma omp task out(a[0;4][0;3]) m_init_1(a,0,0,3,2, 8 ) ; #pragma omp task out(a[2;5][4;3]) m_init_2(a,2,4,6,6, 8) ; #pragma omp task in(a[5;2][1;4]) m_print(a,5,1,6,4, 8) ; #pragma omp taskwait return ; init_1 return init_2 print

26 Region Dependences: in( ), out( ), inout( ) ~ 2 Indicating 1-dim matrix region Single element region in/out(a[i]) Argument is the element A[i] First element and size in/out(a[i;size]) A[i;size] is a block of size elements starting in element A[i] First element can be omitted (default is 0) Size can be omitted if it is known Range of elements in/out(a[i:k]) A[i:k] is a block from element A[i] to element A[k] (both included) Lower bound can be omitted (default is 0) Upper bound can be omitted (default is Size-1) Multiple ways to specify the same region Clause in/out(a[i:i+bs-1]) is equivalent to in/out(a[i;bs])

27 Region Dependences: in( ), out( ), inout( ) ~ 3 Examples using 1-dim regions (whole array) int A[N]; #pragma omp task in(a) // whole array used to compute dependences int A[N]; #pragma omp task in(a[0:n-1]) // whole array used to compute dependences int A[N]; #pragma omp task in(a[0;n]) // whole array used to compute dependences Examples using 1-dim regions (array section) int A[N]; #pragma omp task in(a[0:3]) // first 4 elements of the array used to compute dependences int a[n]; #pragma omp task in(a[0;4]) // first 4 elements of the array used to compute dependences

Region Dependences: in( ), out( ), inout( ) ~ 4 Indicating n-dim matrix regions Single element region in/out(a[i][j] ) First element and size in/out(a[i;s1][j;s2] ) A[i;s1][j;s2] is a 2-dim block of

28 Region Dependences: in( ), out( ), inout( ) ~ 4 Indicating n-dim matrix regions Single element region in/out(a[i][j] ) First element and size in/out(a[i;s1][j;s2] ) A[i;s1][j;s2] is a 2-dim block of s1xs2 elements starting in element A[i][j] Range of elements in/out(a[i:k][j:l] ) A[i:k][j:l] is a 2-dim block from element A[i][j] to element A[k][l] (whole area included) Examples using n-dim regions int A[N][M]; #pragma omp task in(a[2:4][3:6]) // 3 x 4 block of A starting at A[2][3] int A[N][M]; #pragma omp task in(a[2;3][3;4]) // 3 x 4 block of A starting at A[2][3] 0 1 i k j 4 5 l 7 3 s1 s2

29 Nested Tasks and Dependences Nested Tasks Tasks creating tasks themselves Hierarchical Dependence Task Graph An specific dependence domain per task Several dependence task graphs at the same time Task creation dependences (in, out, inout) registered in parent domain Task synchronization (taskwait on) registered in the current domain Parent task probably need to wait for its children taskwait Different level tasks share the same resources When ready, queued into the same ready queues No priority differences between parent tasks and its children

Nested Tasks and Dependences (example) Using nested tasks: Matrix Multiply #pragma omp task

#pragma omp task in([n]a, [N]B) inout([n]c) void dgemm ( float (*A)[N], float (*B)[N], float

k+=bs) block_dgemm( &A[i][k*BS], &B[k][j*BS], &C[i][j*BS] ); #pragma omp taskwait return;

30 Nested Tasks and Dependences (example) Using nested tasks: Matrix Multiply #pragma omp task in([bs][bs]a, [BS][BS]B) inout([bs][bs]c) void block_dgemm (float *A, float *B, float *C); #pragma omp task in([n]a, [N]B) inout([n]c) void dgemm ( float (*A)[N], float (*B)[N], float (*C)[N] ) { for (int i=0; i< N; i+=bs) for (int j=0; j< N; j+=bs) return for (int k=0; k< N; k+=bs) block_dgemm( &A[i][k*BS], &B[k][j*BS], &C[i][j*BS] ); #pragma omp taskwait return; main() { dgemm(a,b,c); dgemm(d,e,f); dgemm(c,f,g); #pragma omp taskwait return; return C F return G return

Dependence Sentinels Mechanism to handle complex dependences when difficult to specify proper in/out clauses when trying to avoid using memory regions (larger data-structures) when want to specify

31 Dependence Sentinels Mechanism to handle complex dependences when difficult to specify proper in/out clauses when trying to avoid using memory regions (larger data-structures) when want to specify artificial dependences (introduced by users) To be avoided if possible! #pragma omp task out(*sentinel) void foo (..., int *sentinel); #pragma omp task in(*sentinel) void bar (..., int *sentinel); foo void main () { int dummy; foo (..., &dummy); bar (..., &dummy) bar 33

32 Parallel Programming Methodology Correctness in sequential program (starting point) Detecting program hotspots (profile) targets Gradually increment taskification (concurrence level) Not many changes at the same time Test every included task with forced single thread in-order execution Use extra taskwaits to force certain levels of serialization Gradually increase execution complexity Single thread in-order execution: --schedule-queue=fifo Single thread out-of-order execution --schedule-queue=lifo Increment number of threads (parallelism) Test performance 34

33 Hands-on Exercises: Machine Description Minotauro (system overview) 2 Intel E5649 6C at 2.53 GHz 2 M2090 NVIDIA GPU Cards 24 GB of Main memory Peak Performance: TFlops 250 GB SSD as local storage 2 Infiniband QDR (40 Gbit each) Top 500 Ranking (11/2012): 35

34 Hands-on Exercises: Methodology Exercises in: ompss-exercises-mt.tar.gz 02-basics_on_ompss dot_product complete dependence clauses, synchronization multisort debuggin errors on task execution matmul parallelize kernel partition = debug reservation = summer-school (at job script: run-x.sh) Paraver configuration files Organized in directories tasks: task related information runtime: runtime internals events scheduling: task-graph and scheduling cuda: specific to CUDA implementations (GPU devices) data-transfer: specific for data send and receive operations mpi-ompss: specific to MPI + OmpSs hybrid parallelization 36

35 Thank you! For further information please contact 37

OpenMP Tasking Model Unstructured parallelism

www.bsc.es OpenMP Tasking Model Unstructured parallelism Xavier Teruel and Xavier Martorell What is a task in OpenMP? Tasks are work units whose execution may be deferred or it can be executed immediately!!!