OpenMP at Twenty Past, Present, and Future. Michael Klemm Chief Executive Officer OpenMP Architecture Review Board

Transcription

1 OpenMP at Twenty Past, Present, and Future Michael Klemm Chief Executive Officer OpenMP Architecture Review Board

2 Outline 20 Years OpenMP Redux The Past The Present The Future 2

3 Definitions The Past: OpenMP < 3.0 The Present: OpenMP 3.0 and OpenMP < 5.0 The Future: OpenMP 5.0 3

4 OpenMP Roadmap OpenMP has a well-defined roadmap: 5-year cadence for major releases One minor release in between (At least) one Technical Report (TR) with feature previews in every year TR6 OpenMP 5.0 TR7* OpenMP 5.x TR8* TR9* OpenMP 6.0 Nov 17 Nov 18 Nov 19 Nov 20 Nov 21 Nov 22 Nov 23 * Numbers assigned to TRs may change if additional TRs are released. 4

5 Levels of Parallelism in OpenMP 4.5 Cluster Coprocessors/Accelerators Node Socket Core Hyper-Threads Superscalar Pipeline Vector Group of computers communicating through fast interconnect Special compute devices attached to the local node through special interconnect Group of processors communicating through shared memory Group of cores communicating through shared cache Group of functional units communicating through registers Group of thread contexts sharing functional units Group of instructions sharing functional units Sequence of instructions sharing functional units Single instruction using multiple functional units 7

6 The Past (or: Stuff you shouldn t be doing no more!) 8

7 OpenMP Worksharing #pragma omp parallel { #pragma omp for for (i = 0; i<n; i++) { fork distribute work barrier #pragma omp for for (i = 0; i< N; i++) { distribute work barrier join 9

8 barrier OpenMP Worksharing/2 double a[n]; double l,s = 0; #pragma omp parallel for reduction(+:s) \ private(l) schedule(static,4) s=0 s =0 s =0 s =0 s =0 distribute work for (i = 0; i<n; i++) { l = log(a[i]); s += s s += s s += l; s += s s = s 10

9 The Present (or: Modern OpenMP) Single Instruction Multiple Data

10 In a Time before OpenMP 4.0 Programmers had to rely on auto-vectorization or to use vendor-specific extensions Programming models (e.g., Intel Cilk Plus) Compiler pragmas (e.g., #pragma vector) Low-level constructs (e.g., _mm_add_pd()) #pragma omp parallel for #pragma vector always #pragma ivdep for (int i = 0; i < N; i++) { a[i] = b[i] +...; You need to trust your compiler to do the right thing. 12

11 Example: SIMD Version of Scalar Product void sprod(float *a, float *b, int n) { float sum = 0.0f; #pragma omp for simd reduction(+:sum) for (int k=0; k<n; k++) sum += a[k] * b[k]; return sum; parallelize Thread 0 Thread 1 Thread 2 vectorize 13

12 SIMD Function Vectorization #pragma omp declare simd float min(float a, float b) { return a < b? a : b; #pragma omp declare simd float distsq(float x, float y) { return (x - y) * (x - y); vec8 min_v(vec8 a, vec8 b) { return a < b? a : b; vec8 distsq_v(vec8 x, vec8 y) { return (x - y) * (x - y); void example() { #pragma omp parallel for simd for (i=0; i<n; i++) { d[i] = min(distsq(a[i], b[i]), c[i]); vd = min_v(distsq_v(va, vb, vc))

13 The Present (or: Modern OpenMP) Task-based Programming

14 Ragged Fork/Join Traditional worksharing can lead to ragged fork/join patterns void example() { compute_in_parallel(a); compute_in_parallel_too(b); cblas_dgemm(, A, B, ); 16

15 Traditional Worksharing Worksharing constructs do not compose well Pathological example: parallel dgemm in MKL void example() { #pragma omp parallel { compute_in_parallel(a); compute_in_parallel_too(b); // dgemm is either parallel or sequential, // but has no orphaned worksharing cblas_dgemm(cblasrowmajor, CblasNoTrans, CblasNoTrans, m, n, k, alpha, A, k, B, n, beta, C, n); Writing such code either oversubscribes the system, yields bad performance due to OpenMP overheads, or needs a lot of glue code to use sequential dgemm only for sub-matrixes 17

16 (Modern) Task-based Execution Model Supports unstructured parallelism unbounded loops while ( <expr> ) {... recursive functions void myfunc( <args> ) {...; myfunc( <newargs> );...; Several scenarios are possible: single creator, multiple creators, nested tasks (tasks & worksharing) All threads in the team are candidates to execute tasks Example: #pragma omp parallel #pragma omp master while (elem!= NULL) { #pragma omp task compute(elem); elem = elem->next; Parallel Team Task pool 18

17 Task Synchronization w/ Dependencies int x = 0; #pragma omp parallel #pragma omp single { #pragma omp task std::cout << x << std::endl; #pragma omp taskwait OpenMP 3.1 int x = 0; #pragma omp parallel #pragma omp single { #pragma omp task depend(in: x) std::cout << x << std::endl; OpenMP 4.0 #pragma omp task x++; #pragma omp task depend(inout: x) x++; OpenMP 3.1 t1 t2 Task s creation time Task s execution time OpenMP 4.x t1 t2 19

18 Example: Cholesky Factorization void cholesky(int ts, int nt, double* a[nt][nt]) { for (int k = 0; k < nt; k++) { ts // Diagonal Block factorization nt potrf(a[k][k], ts, ts); // Triangular systems for (int i = k + 1; i < nt; i++) { #pragma omp task trsm(a[k][k], a[k][i], ts, ts); #pragma omp taskwait // Update trailing matrix for (int i = k + 1; i < nt; i++) { for (int j = k + 1; j < i; j++) { #pragma omp task dgemm(a[k][i], a[k][j], a[j][i], ts, ts); #pragma omp task syrk(a[k][i], a[i][i], ts, ts); #pragma omp taskwait nt ts OpenMP 3.1 ts ts void cholesky(int ts, int nt, double* a[nt][nt]) { for (int k = 0; k < nt; k++) { // Diagonal Block factorization #pragma omp task depend(inout: a[k][k]) potrf(a[k][k], ts, ts); // Triangular systems for (int i = k + 1; i < nt; i++) { #pragma omp task depend(in: a[k][k]) depend(inout: a[k][i]) trsm(a[k][k], a[k][i], ts, ts); // Update trailing matrix for (int i = k + 1; i < nt; i++) { for (int j = k + 1; j < i; j++) { #pragma omp task depend(inout: a[j][i]) depend(in: a[k][i], a[k][j]) dgemm(a[k][i], a[k][j], a[j][i], ts, ts); #pragma omp task depend(inout: a[i][i]) depend(in: a[k][i]) syrk(a[k][i], a[i][i], ts, ts); OpenMP

19 Example: saxpy Operation blocking for (i = 0; i<size; i+=1) { A[i]=A[i]*B[i]*S; taskloop for (i = 0; i<size; i+=ts) { UB = SIZE < (i+ts)? SIZE : i+ts; for (ii=i; ii<ub; ii++) { A[ii]=A[ii]*B[ii]*S; for (i = 0; i<size; i+=ts) { UB = SIZE < (i+ts)? SIZE : i+ts; #pragma omp task private(ii) \ firstprivate(i,ub) shared(s,a,b) for (ii=i; ii<ub; ii++) { A[ii]=A[ii]*B[ii]*S; #pragma omp taskloop grainsize(ts) for (i = 0; i<size; i+=1) { A[i]=A[i]*B[i]*S; With a manual transformation it is difficult to determine grain 1 single iteration too fine whole loop no parallelism Apply blocking techniques taskloop: improved programmability 21

20 Example: Sparse CG w/ taskloop #pragma omp parallel #pragma omp single for (iter = 0; iter < sc->maxiter; iter++) { precon(a, r, z); vectordot(r, z, n, &rho); beta = rho / rho_old; xpay(z, beta, n, p); matvec(a, p, q); vectordot(p, q, n, &dot_pq); alpha = rho / dot_pq; axpy(alpha, p, n, x); axpy(-alpha, q, n, r); sc->residual = sqrt(rho) * bnrm2; if (sc->residual <= sc->tolerance) break; rho_old = rho; void matvec(matrix *A, double *x, double *y) { //... #pragma omp taskloop private(j,is,ie,j0,y0) \ grain_size(grainsz) for (i = 0; i < A->n; i++) { y0 = 0; is = A->ptr[i]; ie = A->ptr[i + 1]; for (j = is; j < ie; j++) { j0 = index[j]; y0 += value[j] * x[j0]; y[i] = y0; //... 22

21 The Present (or: Modern OpenMP) Heterogeneous Programming for Coprocessors

22 Device Model OpenMP 4.0 supports accelerators/coprocessors Device model: One host Multiple accelerators/coprocessors of the same kind Coprocessors Host 24

23 Execution Model The target construct transfers the control flow to the target device Transfer of control is sequential and synchronous The transfer clauses control direction of data flow Array notation is used to describe array length The target data construct creates a scoped device data environment Does not include a transfer of control The transfer clauses control direction of data flow The device data environment is valid through the lifetime of the target data region Use target update to request data transfers from within a target data region 25

24 Example #pragma omp target data device(0) map(alloc:tmp[:n]) map(to:input[:n)) map(from:res) { #pragma omp target device(0) #pragma omp parallel for for (i=0; i<n; i++) tmp[i] = some_computation(input[i], i); update_input_array_on_the_host(input); #pragma omp target update device(0) to(input[:n]) #pragma omp target device(0) #pragma omp parallel for reduction(+:res) for (i=0; i<n; i++) res += final_computation(input[i], tmp[i], i) host target host target host 26

25 Multi-level Device Parallelism int main(int argc, const char* argv[]) { float *x = (float*) malloc(n * sizeof(float)); float *y = (float*) malloc(n * sizeof(float)); // Define scalars n, a, b & initialize x, y #pragma omp target data map(to:x[0:n]) { #pragma omp target map(tofrom:y) #pragma omp teams num_teams(num_blocks) num_threads(bsize) all do the same #pragma omp distribute for (int i = 0; i < n; i += num_blocks){ workshare (w/o barrier) #pragma omp parallel for for (int j = i; j < i + num_blocks; j++) { workshare (w/ barrier) y[j] = a*x[j] + y[j]; 27

26 Multi-level Device Parallelism/2 int main(int argc, const char* argv[]) { float *x = (float*) malloc(n * sizeof(float)); float *y = (float*) malloc(n * sizeof(float)); // Define scalars n, a, b & initialize x, y #pragma omp target map(to:x[0:n]) map(tofrom:y) { #pragma omp teams distribute parallel for \ num_teams(num_blocks) num_threads(bsize) for (int i = 0; i < n; ++i){ y[i] = a*x[i] + y[i]; 28

27 The Future

28 New OpenMP Features Task reductions New task dependencies OpenMP tools interface OpenMP debugging interface concurrent construct 30

29 Task Reductions Task reductions extend traditional reductions to arbitrary task graphs Extend the existing task and taskgroup constructs Also work with the taskloop construct int res = 0; node_t* node = NULL;... #pragma omp parallel { #pragma omp single { #pragma omp taskgroup task_reduction(+: res) { while (node) { #pragma omp task in_reduction(+: res) \ firstprivate(node) { res += node->value; node = node->next; 31

30 New Task Dependencies int x = 0, y = 0, res = 0; #pragma omp parallel #pragma omp single { #pragma omp task depend(out: res) //T0 res = 0; T1 T0 T2 #pragma omp task depend(out: x) //T1 long_computation(x); #pragma omp task depend(out: y) //T2 short_computation(y); #pragma omp task depend(in: x) res += x; depend(mutexinoutset: depend(inout: res) //T3res) //T3 T3 T3 T4 T4 #pragma omp task depend(in: y) res += y; depend(inout: depend(mutexinoutset: res) //T4res) //T4 T5 #pragma omp task depend(in: res) //T5 std::cout << res << std::endl; 32

31 OpenMP Tools Interface (OMPT) Provide interfaces for first-party tools to attach to the OpenMP implementation for performance analysis and correctness checking OpenMP Program Register callbacks 2 1 st Party Tool (same process) 4 Querying further runtime information ompt_start_tool ompt_initialize 1 OpenMP 3 Runtime Library (RTL) Callbacks on OpenMP events Application Address Space and Threads 33

32 OpenMP Tools Interface (OMPT) Developer view Implementation view 34

33 OpenMP Debugging Interface (OMPD) Provide interfaces for third-party tools to inspect the current state of OpenMP execution. OpenMP Program Debugger OpenMP Runtime Library Application Threads 1 Request OpenMP state Result Attach Callback 3 OMPD DLL Debugger OMPD DLL loaded into debugger and callbacks registered Request symbol addresses and read memory 2 35

34 OpenMP Debugging Interface (OMPD) Segmentation fault occurs in thread 100; is this an OpenMP thread? Debugger gets: Address space Load OMPD DLL Initialize OMPD Initialize process Is thread in a device? Yes Request thread handle for thread identifier No Initialize device Returned OK? Yes Thread 100 is an OpenMP thread No Debugger gets: Thread handle Debugger gets: Device Address space 36

35 concurrent Construct Assert to the compiler that all iterations of a loop are free of loop-carried dependencies and may execute in any order. Syntax: #pragma omp concurrent [clause [ [,] clause] ] for-loops!$omp concurrent [clause [ [,] clause] ] do-loops [!$omp end concurrent] 37

36 concurrent Construct Existing loop constructs are tightly bound to execution model: #pragma omp for for (i=0; i<n;++i) { #pragma omp simd for (i=0; i<n;++i) { #pragma omp taskloop for (i=0; i<n;++i) { fork generate tasks distribute work barrier join taskwait The concurrent construct is meant to let the OpenMP implementation pick choose the right parallelization scheme. 38

37 Simplifying Multi-level Device Parallelism int main(int argc, const char* argv[]) { float *x = (float*) malloc(n * sizeof(float)); float *y = (float*) malloc(n * sizeof(float)); // Define scalars n, a, b & initialize x, y #pragma omp target map(to:x[0:n]) map(tofrom:y) { #pragma omp teams concurrent distribute parallel for \ for num_teams(num_blocks) (int i = 0; i < n; ++i){ num_threads(bsize) for y[i] (int= i a*x[i] = 0; i + < y[i]; n; ++i){ y[i] = a*x[i] + y[i]; 39

38 OpenMP 5.0 Miscellaneous (Planned) Features Release/acquire semantics for the memory model Support for memory allocators Task affinity support Improved multiple device support Improved base language support Fortran 2003 C11, C++11, C++14 lots of corrections and clarifications 40

39 Adverts: Engage with the Community 41

40 UK OpenMP Users Conference May 2018 St Catherine s College, Oxford, UK Call for Submissions is now open Tech presentations, research, tutorials, posters Additional Information: Organised by: Conference Chair: Simon McIntosh-Smith, Prof. in High Performance Computing, University of Bristol Program Co-Chair: Dr Mark Bull, Architect (Research and Training), EPCC Program Co-Chair: Jim Cownie, Principal Engineer, Intel Corporation (UK) Ltd. 42

41 OpenMPCon & IWOMP 2018 Tentative dates: OpenMPCon: Sep Tutorials: Sep 26 IWOMP: Sep Location: Barcelona, Spain (?) 43

42 OpenMP Book 44

43 The Last Slide OpenMP 5.0 will be a major leap forward Well-defined interfaces for tools New ways to express parallelism OpenMP is a modern directive-based programming model Multi-level parallelism supported (coprocessors, threads, SIMD) Task-based programming model is the modern approach to parallelism 45

44 Visit for more information 46