Session 4: Parallel Programming with OpenMP
|
|
|
- Harry Parks
- 6 years ago
- Views:
Transcription
1 Session 4: Parallel Programming with OpenMP Xavier Martorell Barcelona Supercomputing Center
2 Agenda Agenda 10:00-11:00 OpenMP fundamentals, parallel regions 11:00-11:30 Worksharing constructs 11:30-12:00 Break 12:00-12:15 Synchronization mechanisms in OpenMP 12:15-13:00 Practical: heat diffusion 13:00-14:00 Lunch 14:00-14:20 Programming using a hybrid MPI/OpenMP approach 14:20-17:00 Practical: heat diffusion Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
3 Part I OpenMP fundamentals, parallel regions Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
4 Outline OpenMP Overview The OpenMP model Writing OpenMP programs Creating Threads Data-sharing attributes Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
5 Outline OpenMP Overview OpenMP Overview The OpenMP model Writing OpenMP programs Creating Threads Data-sharing attributes Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
6 What is OpenMP? OpenMP Overview It s an API extension to the C, C++ and Fortran languages to write parallel programs for shared memory machines Current version is 3.1 (July 2011) is open for public comments Supported by most compiler vendors Intel, IBM, PGI, Sun, Cray, Fujitsu, HP, GCC... Maintained by the Architecture Review Board (ARB), a consortium of industry and academia Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
7 A bit of history OpenMP Overview OpenMP Fortran 1.0 OpenMP C/C OpenMP Fortran 1.1 OpenMP Fortran 2.0 OpenMP C/C OpenMP 2.5 OpenMP 3.0 OpenMP Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
8 OpenMP Overview Advantages of OpenMP Mature standard and implementations Standardizes practice of the last 20 years Good performance and scalability Portable across architectures Incremental parallelization Maintains sequential version (mostly) High level language Some people may say a medium level language :-) Supports both task and data parallelism Communication is implicit Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
9 OpenMP Overview Disadvantages of OpenMP Communication is implicit Flat memory model Incremental parallelization creates false sense of glory/failure No support for accelerators (...yet, maybe in 4.0) No error recovery capabilities (...yet, 4.0) Difficult to compose Lacks high-level algorithms and structures Does not run on clusters Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
10 Outline The OpenMP model OpenMP Overview The OpenMP model Writing OpenMP programs Creating Threads Data-sharing attributes Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
11 OpenMP at a glance The OpenMP model OpenMP components Constructs Compiler OpenMP Executable OpenMP API Environment Variables OpenMP Runtime Library ICVs OS Threading Libraries CPU CPU CPU CPU CPU CPU SMP Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
12 The OpenMP model Execution model Fork-join model OpenMP uses a fork-join model The master thread spawns a team of threads that joins at the end of the parallel region Threads in the same team can collaborate to do work Master Thread Nested Parallel Region Parallel Region Parallel Region Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
13 The OpenMP model Memory model OpenMP defines a relaxed memory model Threads can see different values for the same variable Memory consistency is only guaranteed at specific points Luckily, the default points are usually enough Variables can be shared or private to each thread Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
14 Outline Writing OpenMP programs OpenMP Overview The OpenMP model Writing OpenMP programs Creating Threads Data-sharing attributes Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
15 Writing OpenMP programs OpenMP directives syntax In Fortran Through a specially formatted comment: s e n t i n e l c o n s t r u c t [ clauses ] where sentinel is one of:!$omp or C$OMP or *$OMP in fixed format!$omp in free format In C/C++ Through a compiler directive: #pragma omp c o n s t r u c t [ clauses ] OpenMP syntax is ignored if the compiler does not recognize OpenMP Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
16 Writing OpenMP programs OpenMP directives syntax In Fortran Through a specially formatted comment: s e n t i n e l c o n s t r u c t [ clauses ] where sentinel is one of:!$omp or C$OMP or *$OMP in fixed format!$omp in free format In C/C++ Through a compiler directive: #pragma omp c o n s t r u c t [ clauses ] OpenMP syntax is ignored if the compiler does not recognize We llopenmp be using C/C++ syntax through this tutorial Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
17 Writing OpenMP programs Headers/Macros C/C++ only omp.h contains the API prototypes and data types definitions The _OPENMP is defined by the OpenMP enabled compilers Allows conditional compilation of OpenMP Fortran only The omp_lib module contains the subroutine and function definitions Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
18 Writing OpenMP programs Structured Block Definition Most directives apply to a structured block: Block of one or more statements One entry point, one exit point No branching in or out allowed Terminating the program is allowed Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
19 Outline Creating Threads OpenMP Overview The OpenMP model Writing OpenMP programs Creating Threads Data-sharing attributes Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
20 Creating Threads The parallel construct Directive #pragma omp parallel [ clauses ] s t r u c t u r e d block where clauses can be: num_threads(expression) if(expression) shared(var-list) private(var-list) firstprivate(var-list) default(none shared private firstprivate ) reduction(var-list) We ll see it later Coming shortly! Only in Fortran Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
21 Creating Threads The parallel construct Specifying the number of threads The number of threads is controlled by an internal control variable (ICV) called nthreads-var When a parallel construct is found a parallel region with a maximum of nthreads-var is created Parallel constructs can be nested creating nested parallelism The nthreads-var can be modified through the omp_set_num_threads API called the OMP_NUM_THREADS environment variable Additionally, the num_threads clause causes the implementation to ignore the ICV and use the value of the clause for that region Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
22 The parallel construct Creating Threads Avoiding parallel regions Sometimes we only want to run in parallel under certain conditions E.g., enough input data, not running already in parallel,... The if clause allows to specify an expression. When evaluates to false the parallel construct will only use 1 thread Note that still creates a new team and data environment Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
23 Putting it together Creating Threads Example void main ( ) { #pragma omp parallel... omp_set_num_threads ( 2 ) ; #pragma omp parallel... #pragma omp parallel num_threads ( random()%4+1) if (N>=128)... } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
24 Putting it together Creating Threads Example void main ( ) { #pragma omp parallel... An unknown number of threads here. Use OMP_NUM_THREADS omp_set_num_threads ( 2 ) ; #pragma omp parallel... #pragma omp parallel num_threads ( random()%4+1) if (N>=128)... } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
25 Putting it together Creating Threads Example void main ( ) { #pragma omp parallel... omp_set_num_threads ( 2 ) ; #pragma omp parallel... A team of two threads here #pragma omp parallel num_threads ( random()%4+1) if (N>=128)... } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
26 Putting it together Creating Threads Example void main ( ) { #pragma omp parallel... omp_set_num_threads ( 2 ) ; #pragma omp parallel... #pragma omp parallel num_threads ( random()%4+1) if (N>=128)... A team of [1..4] threads here } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
27 Creating Threads API calls Other useful routines int omp_get_num_threads() int omp_get_thread_num() int omp_get_num_procs() int omp_get_max_threads() double omp_get_wtime() Returns the number of threads in the current team Returns the id of the thread in the current team Returns the number of processors in the machine Returns the maximum number of threads that will be used in the next parallel region Returns the number of seconds since an arbitrary point in the past Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
28 Outline Data-sharing attributes OpenMP Overview The OpenMP model Writing OpenMP programs Creating Threads Data-sharing attributes Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
29 Data-sharing attributes Data environment A number of clauses are related to building the data environment that the construct will use when executing shared private firstprivate default threadprivate lastprivate reduction Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
30 Data-sharing attributes Data-sharing attributes Shared When a variable is marked as shared, the variable inside the construct is the same as the one outside the construct In a parallel construct this means all threads see the same variable but not necessarily the same value Usually need some kind of synchronization to update them correctly OpenMP has consistency points at synchronizations Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
31 Data-sharing attributes Data-sharing attributes Example i n t x =1; #pragma omp parallel shared ( x ) num_threads ( 2 ) { x ++; p r i n t f ( "%d\n", x ) ; } p r i n t f ( "%d\n", x ) ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
32 Data-sharing attributes Data-sharing attributes Example i n t x =1; #pragma omp parallel shared ( x ) num_threads ( 2 ) { x ++; p r i n t f ( "%d\n", x ) ; } p r i n t f ( "%d\n", x ) ; Prints 2 or 3 (three printfs in total) Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
33 Data-sharing attributes Data-sharing attributes Private When a variable is marked as private, the variable inside the construct is a new variable of the same type with an undefined value In a parallel construct this means all threads have a different variable Can be accessed without any kind of synchronization Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
34 Data-sharing attributes Data-sharing attributes Example i n t x =1; #pragma omp parallel private ( x ) num_threads ( 2 ) { x ++; p r i n t f ( "%d\n", x ) ; } p r i n t f ( "%d\n", x ) ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
35 Data-sharing attributes Data-sharing attributes Example i n t x =1; #pragma omp parallel private ( x ) num_threads ( 2 ) { x ++; p r i n t f ( "%d\n", x ) ; } p r i n t f ( "%d\n", x ) ; Can print anything (twice, same or different) Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
36 Data-sharing attributes Data-sharing attributes Example i n t x =1; #pragma omp parallel private ( x ) num_threads ( 2 ) { x ++; p r i n t f ( "%d\n", x ) ; } p r i n t f ( "%d\n", x ) ; Prints 1 Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
37 Data-sharing attributes Data-sharing attributes Firstprivate When a variable is marked as firstprivate, the variable inside the construct is a new variable of the same type but it is initialized to the original value of the variable In a parallel construct this means all threads have a different variable with the same initial value Can be accessed without any kind of synchronization Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
38 Data-sharing attributes Data-sharing attributes Example i n t x =1; #pragma omp parallel firstprivate ( x ) num_threads ( 2 ) { x ++; p r i n t f ( "%d\n", x ) ; } p r i n t f ( "%d\n", x ) ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
39 Data-sharing attributes Data-sharing attributes Example i n t x =1; #pragma omp parallel firstprivate ( x ) num_threads ( 2 ) { x ++; p r i n t f ( "%d\n", x ) ; } p r i n t f ( "%d\n", x ) ; Prints 2 (twice) Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
40 Data-sharing attributes Data-sharing attributes Example i n t x =1; #pragma omp parallel firstprivate ( x ) num_threads ( 2 ) { x ++; p r i n t f ( "%d\n", x ) ; } p r i n t f ( "%d\n", x ) ; Prints 1 Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
41 Data-sharing attributes Data-sharing attributes What is the default? Static/global storage is shared Heap-allocated storage is shared Stack-allocated storage inside the construct is private Others If there is a default clause, what the clause says none means that the compiler will issue an error if the attribute is not explicitly set by the programmer Otherwise, depends on the construct For the parallel region the default is shared Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
42 Data-sharing attributes Data-sharing attributes Example i n t x, y ; #pragma omp parallel private ( y ) { x = y = #pragma omp parallel private ( x ) { x = y = } } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
43 Data-sharing attributes Data-sharing attributes Example i n t x, y ; #pragma omp parallel private ( y ) { x = x is shared y = #pragma ompy parallel is private private ( x ) } { } x = y = Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
44 Data-sharing attributes Data-sharing attributes Example i n t x, y ; #pragma omp parallel private ( y ) { x = y = #pragma omp parallel private ( x ) { x = x is private y = } y is shared } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
45 Data-sharing attributes Threadprivate storage The threadprivate construct #pragma omp t h r e a d p r i v a t e ( var l i s t ) Can be applied to: Global variables Static variables Class-static members Allows to create a per-thread copy of global variables threadprivate storage persist across parallel regions if the number of threads is the same Threadprivate persistence across nested regions is complex Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
46 Threaprivate storage Data-sharing attributes Example char foo ( ) { s t a t i c char b u f f e r [ BUF_SIZE ] ; #pragma omp t h r e a d p r i v a t e ( b u f f e r )... } return b u f f e r ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
47 Threaprivate storage Data-sharing attributes Example char foo ( ) { s t a t i c char b u f f e r [ BUF_SIZE ] ; #pragma omp t h r e a d p r i v a t e ( b u f f e r )... Creates one static copy of buffer per thread } return b u f f e r ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
48 Threaprivate storage Data-sharing attributes Example char foo ( ) { s t a t i c char b u f f e r [ BUF_SIZE ] ; #pragma omp t h r e a d p r i v a t e ( b u f f e r )... Now foo can be called by multiple threads at the same time } return b u f f e r ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
49 Threaprivate storage Data-sharing attributes Example char foo ( ) { s t a t i c char b u f f e r [ BUF_SIZE ] ; #pragma omp t h r e a d p r i v a t e ( b u f f e r ) }... return b u f f e r ; foo returns correct address to caller Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
50 Part II Worksharing constructs Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
51 Outline The worksharing concept Loop worksharing Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
52 Outline The worksharing concept The worksharing concept Loop worksharing Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
53 The worksharing concept Worksharings Worksharing constructs divide the execution of a code region among the threads of a team Threads cooperate to do some work Better way to split work than using thread-ids Lower overhead than using tasks But, less flexible In OpenMP, there are four worksharing constructs: single loop worksharing section workshare Restriction: worksharings cannot be nested Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
54 Outline Loop worksharing The worksharing concept Loop worksharing Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
55 Loop worksharing Loop parallelism The for construct #pragma omp for [ clauses ] for ( i n i t expr ; t e s t expr ; inc expr ) where clauses can be: private firstprivate lastprivate(variable-list) reduction(operator:variable-list) schedule(schedule-kind) nowait collapse(n) ordered Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
56 Loop worksharing The for construct How it works? The iterations of the loop(s) associated to the construct are divided among the threads of the team Loop iterations must be independent Loops must follow a form that allows to compute the number of iterations Valid data types for induction variables are: integer types, pointers and random access iterators (in C++) The induction variable(s) are automatically privatized The default data-sharing attribute is shared It can be merged with the parallel construct: #pragma omp parallel for Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
57 The for construct Loop worksharing Example void foo ( i n t m, i n t N, i n t M) { i n t i ; #pragma omp parallel for private ( j ) for ( i = 0; i < N; i ++ ) for ( j = 0; j < M; j ++ ) m[ i ] [ j ] = 0; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
58 The for construct Loop worksharing Example void foo ( i n t m, i n t N, i n t M) { i n t i ; New created threads cooperate to execute all the iterations of the loop #pragma omp parallel for private ( j ) for ( i = 0; i < N; i ++ ) for ( j = 0; j < M; j ++ ) m[ i ] [ j ] = 0; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
59 The for construct Loop worksharing Example void foo ( i n t m, i n t N, i n t M) { i n t i ; #pragma omp parallel for private ( j ) for ( i = 0; i The < N; i variable i ++ ) is automatically privatized for ( j = 0; j < M; j ++ ) m[ i ] [ j ] = 0; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
60 The for construct Loop worksharing Example void foo ( i n t m, i n t N, i n t M) { i n t i ; #pragma omp parallel for private ( j ) for ( i = 0; i < N; i ++ ) for ( j = 0; j Must < M; bej explicitly ++ ) privatized m[ i ] [ j ] = 0; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
61 The for construct Loop worksharing Example void foo ( std : : vector <int > &v ) { #pragma omp parallel for for ( std : : vector <int > : : i t e r a t o r i t = v. begin ( ) ; i t < v. end ( ) ; i t ++ ) i t = 0; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
62 The for construct Loop worksharing Example void foo ( std : : vector <int > &v ) { #pragma omp parallel for random access iterators for ( std : : vector <int > : : i t e r a t o r i t = v. begin (and pointers) ( ) ; are valid i t < v. end ( ) ; types i t ++ ) i t = 0; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
63 The for construct Loop worksharing Example void foo ( std : : vector <int > &v ) { #pragma omp parallel for for ( std : : vector <int > : : i t e r a t o r i t = v. begin ( ) ; i t < v. end ( ) ;!= cannot be used in the test expression i t ++ ) i t = 0; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
64 Loop worksharing Removing dependences Example x = 0; for ( i = 0; i < n ; i ++ ) { } v [ i ] = x ; x += dx ; Each iteration x depends on the previous one. Can t be parallelized Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
65 Loop worksharing Removing dependences Example x = 0; for ( i = 0; i < n ; i ++ ) { } x = i dx ; v [ i ] = x ; But x can be rewritten in terms of i. Now it can be parallelized Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
66 Loop worksharing The lastprivate clause When a variable is declared lastprivate, a private copy is generated for each thread. Then the value of the variable in the last iteration of the loop is copied back to the original variable A variable can be both firstprivate and lastprivate Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
67 Loop worksharing The reduction clause A very common pattern is where all threads accumulate some values into a single variable E.g., n += v[i], our heat program,... Using critical or atomic is not good enough Besides being error prone and cumbersome Instead we can use the reduction clause for basic types Valid operators are: +, -, *,,, &, &&,^, min, max User-defined reductions coming soon... The compiler creates a private copy that is properly initialized At the end of the region, the compiler ensures that the shared variable is properly (and safely) updated We can also specify reduction variables in the parallel construct Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
68 The reduction clause Loop worksharing Example i n t vector_sum ( i n t n, i n t v [ n ] ) { i n t i, sum = 0; #pragma omp parallel for reduction ( + : sum) { for ( i = 0; i < n ; i ++ ) sum += v [ i ] ; } return sum ; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
69 The reduction clause Loop worksharing Example i n t vector_sum ( i n t n, i n t v [ n ] ) { i n t i, sum = 0; #pragma omp parallel for reduction ( + : sum) { Private copy initialized here to the identity value for ( i = 0; i < n ; i ++ ) sum += v [ i ] ; } Shared variable updated here with the partial values of each thread return sum ; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
70 Loop worksharing The schedule clause The schedule clause determines which iterations are executed by each thread If no schedule clause is present then is implementation defined There are several possible options as schedule: STATIC STATIC,chunk DYNAMIC[,chunk] GUIDED[,chunk] AUTO RUNTIME Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
71 Loop worksharing The schedule clause Static schedule The iteration space is broken in chunks of approximately size N/num threads. Then these chunks are assigned to the threads in a Round-Robin fashion Static, N schedule (Interleaved) The iteration space is broken in chunks of size N. Then these chunks are assigned to the threads in a Round-Robin fashion Characteristics of static schedules Low overhead Good locality (usually) Can have load imbalance problems Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
72 Loop worksharing The schedule clause Dynamic, N schedule Threads dynamically grab chunks of N iterations until all iterations have been executed. If no chunk is specified, N = 1. Guided, N schedule Variant of dynamic. The size of the chunks deceases as the threads grab iterations, but it is at least of size N. If no chunk is specified, N = 1. Characteristics of dynamic schedules Higher overhead Not very good locality (usually) Can solve imbalance problems Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
73 Loop worksharing The schedule clause Auto schedule In this case, the implementation is allowed to do whatever it wishes Do not expect much of it as of now Runtime schedule The decision is delayed until the program is run through the sched-nvar ICV. It can be set with: The OMP_SCHEDULE environment variable The omp_set_schedule() API call Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
74 Loop worksharing The nowait clause When a worksharing has a nowait clause then the implicit barrier at the end of the loop is removed This allows to overlap the execution of non-dependent loops/tasks/worksharings Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
75 The nowait clause Loop worksharing Example #pragma omp for nowait for ( i = 0; i < n ; i ++ ) v [ i ] = 0; #pragma omp for for ( i = 0; i < n ; i ++ ) a [ i ] = 0; First and second loop are independent, so we can overlap them Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
76 The nowait clause Loop worksharing Example #pragma omp for nowait for ( i = 0; i < n ; i ++ ) v [ i ] = 0; #pragma omp for for ( i = 0; i < n ; i ++ ) a [ i ] = 0; Side note: you would better fuse the loops in this case Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
77 The nowait clause Loop worksharing Example #pragma omp for nowait for ( i = 0; i < n ; i ++ ) v [ i ] = 0; #pragma omp for for ( i = 0; i < n ; i ++ ) a [ i ] = v [ i ] v [ i ] ; First and second loops are dependent! No guarantees that the previous iteration is finished Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
78 Loop worksharing The nowait clause Exception: static schedules If the two (or more) loops have the same static schedule and all have the same number of iterations Example #pragma omp for schedule ( static, M) nowait for ( i = 0; i < n ; i ++ ) v [ i ] = 0; #pragma omp for schedule ( static, M) for ( i = 0; i < n ; i ++ ) a [ i ] = v [ i ] v [ i ] ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
79 Loop worksharing The collapse clause Allows to distribute work from a set of n nested loops Loops must be perfectly nested The nest must traverse a rectangular iteration space Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
80 Loop worksharing The collapse clause Allows to distribute work from a set of n nested loops Loops must be perfectly nested The nest must traverse a rectangular iteration space Example #pragma omp for collapse ( 2 ) for ( i = 0; i < N; i ++ ) for ( j = 0; j < M; j ++ ) foo ( i, j ) ; i and j loops are folded and iterations distributed among all threads. Both i and j are privatized Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
81 Break Coffee time! :-) Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
82 Part III Basic Synchronizations Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
83 Outline Thread barriers Exclusive access Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
84 Why synchronization? Mechanisms Threads need to synchronize to impose some ordering in the sequence of actions of the threads. OpenMP provides different synchronization mechanisms: barrier critical atomic taskwait ordered locks Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
85 Outline Thread barriers Thread barriers Exclusive access Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
86 Thread Barrier Thread barriers The barrier construct #pragma omp barrier Threads cannot proceed past a barrier point until all threads reach the barrier AND all previously generated work is completed Some constructs have an implicit barrier at the end E.g., the parallel construct Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
87 Barrier Thread barriers Example #pragma omp parallel { foo ( ) ; #pragma omp barrier bar ( ) ; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
88 Barrier Thread barriers Example #pragma omp parallel { foo ( ) ; #pragma omp barrier bar ( ) ; } Forces all foo occurrences too happen before all bar occurrences Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
89 Barrier Thread barriers Example #pragma omp parallel { foo ( ) ; #pragma omp barrier bar ( ) ; } Implicit barrier at the end of the parallel region Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
90 Outline Exclusive access Thread barriers Exclusive access Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
91 Exclusive access Exclusive access The critical construct #pragma omp critical [ ( name ) ] s t r u c t u r e d block Provides a region of mutual exclusion where only one thread can be working at any given time. By default all critical regions are the same, but you can provide them with names Only those with the same name synchronize Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
92 Critical construct Exclusive access Example i n t x =1; #pragma omp parallel num_threads ( 2 ) { #pragma omp critical x ++; } p r i n t f ( "%d\n", x ) ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
93 Critical construct Exclusive access Example i n t x =1; #pragma omp parallel num_threads ( 2 ) { #pragma omp critical x ++; Only one thread at a time here } p r i n t f ( "%d\n", x ) ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
94 Critical construct Exclusive access Example i n t x =1; #pragma omp parallel num_threads ( 2 ) { #pragma omp critical x ++; Only one thread at a time here } p r i n t f ( "%d\n", x ) ; Prints 3! Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
95 Critical construct Exclusive access Example i n t x =1, y =0; #pragma omp parallel num_threads ( 4 ) { #pragma omp critical ( x ) x ++; #pragma omp critical ( y ) y ++; } Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
96 Critical construct Exclusive access Example i n t x =1, y =0; #pragma omp parallel num_threads ( 4 ) { #pragma omp critical ( x ) x ++; #pragma omp critical ( y ) y ++; } Different names: One thread can update x while another updates y Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
97 Exclusive access Exclusive access The atomic construct #pragma omp atomic expression Provides an special mechanism of mutual exclusion to do read & update operations Only supports simple read & update expressions E.g., x += 1, x = x - foo() Only protects the read & update part foo() not protected Usually much more efficient than a critical construct Not compatible with critical Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
98 Atomic construct Exclusive access Example i n t x =1; #pragma omp parallel num_threads ( 2 ) { #pragma omp atomic x ++; } p r i n t f ( "%d\n", x ) ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
99 Atomic construct Exclusive access Example i n t x =1; #pragma omp parallel num_threads ( 2 ) { #pragma omp atomic x ++; Only one thread at a time updates x here } p r i n t f ( "%d\n", x ) ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
100 Atomic construct Exclusive access Example i n t x =1; #pragma omp parallel num_threads ( 2 ) { #pragma omp atomic x ++; } p r i n t f ( "%d\n", x ) ; Prints 3! Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
101 Atomic construct Exclusive access Example i n t x =1; #pragma omp parallel num_threads ( 2 ) { #pragma omp critical x ++; #pragma omp atomic x ++; } p r i n t f ( "%d\n", x ) ; Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
102 Atomic construct Exclusive access Example i n t x =1; #pragma omp parallel num_threads ( 2 ) { #pragma omp critical x ++; #pragma omp atomic x ++; } p r i n t f ( "%d\n", x ) ; Different threads can update x at the same time! Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
103 Atomic construct Exclusive access Example i n t x =1; #pragma omp parallel num_threads ( 2 ) { #pragma omp critical x ++; #pragma omp atomic x ++; } p r i n t f ( "%d\n", x ) ; Prints 3,4 or 5 :( Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
104 Part IV Practical: OpenMP heat diffusion Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
105 Outline Heat diffusion Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
106 Outline Heat diffusion Heat diffusion Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
107 Heat diffusion Before you start Enter the OpenMP directory to do the following exercises Session4.1-exercise contains the serial version of the Heat application you can use SsGrind on heat-tareador to determine parallelism, and observe diferences among the three algorithms and then annotate the application with OpenMP Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
108 Heat diffusion Description of the Heat Diffusion app Hands-on Parallel loops The file solver.c implements the computation of the Heat diffusion 1 Annotate the jacobi, redblack, and gauss functions with OpenMP 2 Execute the application with different numbers of processors compare the results evaluate the performance Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
109 Break Bon appétit!* *Disclaimer: actual food may differ from the image! :-) Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
110 Part V Programming using a hybrid MPI/OpenMP approach Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
111 Outline MPI+OpenMP programming Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
112 Outline MPI+OpenMP programming MPI+OpenMP programming Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
113 MPI+OpenMP programming Distributed- vs Shared- Memory Programming Distributed-memory programming Separate processes Private variables are unaccessible from others Point-to-point and collective communication Implicit synchronization Shared-memory programming Multiple threads share same address space Explicit synchronization Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
114 MPI+OpenMP programming Hybrid programming Combining MPI+OpenMP Distributed algorithms spread over nodes Shared memory for computation within each node Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
115 MPI+OpenMP programming Opportunities When to use MPI+OpenMP Starting from OpenMP and moving to clusters with MPI Starting from MPI and exploiting further parallelism inside each node Improvements OpenMP can solve MPI imbalance Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
116 MPI+OpenMP programming Alternatives MPI + computational kernels in OpenMP Use OpenMP directives to exploit parallelism between communication phases OpenMP parallel will end before new communication calls MPI inside OpenMP constructs Call MPI from within for-loops, or tasks MPI needs to support multi-threaded mode Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
117 MPI+OpenMP programming Compiling MPI+OpenMP MPI compiler driver needs the proper OpenMP option mpicc -openmp mpicc -fopenmp Also useful mpicc -show <your command line options and files> It displays the full command line executed by mpicc to compile your program Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
118 Part VI Practical: MPI+OpenMP heat diffusion Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
119 Outline MPI+OpenMP Heat diffusion Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
120 Outline MPI+OpenMP Heat diffusion MPI+OpenMP Heat diffusion Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
121 Before you start MPI+OpenMP Heat diffusion Enter the Session4.2-exercise directory to do the following exercises Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
122 MPI+OpenMP Heat diffusion Description of the Heat Diffusion app Hands-on Parallel loops The file solver.c implements the computation of the Heat diffusion 1 Use MPI to distribute the work across nodes 2 Annotate the jacobi, redblack, and gauss functions with OpenMP tasks 3 Execute the application with different numbers of nodes/processors, and compare the results Xavier Martorell (BSC) PATC Parallel Programming Workshop November 26-30, / 91
A brief introduction to OpenMP
A brief introduction to OpenMP Alejandro Duran Barcelona Supercomputing Center Outline 1 Introduction 2 Writing OpenMP programs 3 Data-sharing attributes 4 Synchronization 5 Worksharings 6 Task parallelism
EPL372 Lab Exercise 5: Introduction to OpenMP
EPL372 Lab Exercise 5: Introduction to OpenMP References: https://computing.llnl.gov/tutorials/openmp/ http://openmp.org/wp/openmp-specifications/ http://openmp.org/mp-documents/openmp-4.0-c.pdf http://openmp.org/mp-documents/openmp4.0.0.examples.pdf
OpenMP Fundamentals Fork-join model and data environment
www.bsc.es OpenMP Fundamentals Fork-join model and data environment Xavier Teruel and Xavier Martorell Agenda: OpenMP Fundamentals OpenMP brief introduction The fork-join model Data environment OpenMP
Overview: The OpenMP Programming Model
Overview: The OpenMP Programming Model motivation and overview the parallel directive: clauses, equivalent pthread code, examples the for directive and scheduling of loop iterations Pi example in OpenMP
OpenMP - II. Diego Fabregat-Traver and Prof. Paolo Bientinesi WS15/16. HPAC, RWTH Aachen
OpenMP - II Diego Fabregat-Traver and Prof. Paolo Bientinesi HPAC, RWTH Aachen fabregat@aices.rwth-aachen.de WS15/16 OpenMP References Using OpenMP: Portable Shared Memory Parallel Programming. The MIT
Introduction to OpenMP. OpenMP basics OpenMP directives, clauses, and library routines
Introduction to OpenMP Introduction OpenMP basics OpenMP directives, clauses, and library routines What is OpenMP? What does OpenMP stands for? What does OpenMP stands for? Open specifications for Multi
Programming with Shared Memory PART II. HPC Fall 2012 Prof. Robert van Engelen
Programming with Shared Memory PART II HPC Fall 2012 Prof. Robert van Engelen Overview Sequential consistency Parallel programming constructs Dependence analysis OpenMP Autoparallelization Further reading
Topics. Introduction. Shared Memory Parallelization. Example. Lecture 11. OpenMP Execution Model Fork-Join model 5/15/2012. Introduction OpenMP
Topics Lecture 11 Introduction OpenMP Some Examples Library functions Environment variables 1 2 Introduction Shared Memory Parallelization OpenMP is: a standard for parallel programming in C, C++, and
Shared Memory Parallelism - OpenMP
Shared Memory Parallelism - OpenMP Sathish Vadhiyar Credits/Sources: OpenMP C/C++ standard (openmp.org) OpenMP tutorial (http://www.llnl.gov/computing/tutorials/openmp/#introduction) OpenMP sc99 tutorial
Advanced C Programming Winter Term 2008/09. Guest Lecture by Markus Thiele
Advanced C Programming Winter Term 2008/09 Guest Lecture by Markus Thiele Lecture 14: Parallel Programming with OpenMP Motivation: Why parallelize? The free lunch is over. Herb
Lecture 4: OpenMP Open Multi-Processing
CS 4230: Parallel Programming Lecture 4: OpenMP Open Multi-Processing January 23, 2017 01/23/2017 CS4230 1 Outline OpenMP another approach for thread parallel programming Fork-Join execution model OpenMP
Introduction to OpenMP
Introduction to OpenMP Ricardo Fonseca https://sites.google.com/view/rafonseca2017/ Outline Shared Memory Programming OpenMP Fork-Join Model Compiler Directives / Run time library routines Compiling and
OpenMP 2. CSCI 4850/5850 High-Performance Computing Spring 2018
OpenMP 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning Objectives
Introduction to OpenMP
Introduction to OpenMP Le Yan Scientific computing consultant User services group High Performance Computing @ LSU Goals Acquaint users with the concept of shared memory parallelism Acquaint users with
COMP4300/8300: The OpenMP Programming Model. Alistair Rendell. Specifications maintained by OpenMP Architecture Review Board (ARB)
COMP4300/8300: The OpenMP Programming Model Alistair Rendell See: www.openmp.org Introduction to High Performance Computing for Scientists and Engineers, Hager and Wellein, Chapter 6 & 7 High Performance
COMP4300/8300: The OpenMP Programming Model. Alistair Rendell
COMP4300/8300: The OpenMP Programming Model Alistair Rendell See: www.openmp.org Introduction to High Performance Computing for Scientists and Engineers, Hager and Wellein, Chapter 6 & 7 High Performance
https://www.youtube.com/playlist?list=pllx- Q6B8xqZ8n8bwjGdzBJ25X2utwnoEG
https://www.youtube.com/playlist?list=pllx- Q6B8xqZ8n8bwjGdzBJ25X2utwnoEG OpenMP Basic Defs: Solution Stack HW System layer Prog. User layer Layer Directives, Compiler End User Application OpenMP library
ECE 574 Cluster Computing Lecture 10
ECE 574 Cluster Computing Lecture 10 Vince Weaver http://www.eece.maine.edu/~vweaver vincent.weaver@maine.edu 1 October 2015 Announcements Homework #4 will be posted eventually 1 HW#4 Notes How granular
Programming with Shared Memory PART II. HPC Fall 2007 Prof. Robert van Engelen
Programming with Shared Memory PART II HPC Fall 2007 Prof. Robert van Engelen Overview Parallel programming constructs Dependence analysis OpenMP Autoparallelization Further reading HPC Fall 2007 2 Parallel
Introduction to Standard OpenMP 3.1
Introduction to Standard OpenMP 3.1 Massimiliano Culpo - m.culpo@cineca.it Gian Franco Marras - g.marras@cineca.it CINECA - SuperComputing Applications and Innovation Department 1 / 59 Outline 1 Introduction
OpenMP. António Abreu. Instituto Politécnico de Setúbal. 1 de Março de 2013
OpenMP António Abreu Instituto Politécnico de Setúbal 1 de Março de 2013 António Abreu (Instituto Politécnico de Setúbal) OpenMP 1 de Março de 2013 1 / 37 openmp what? It s an Application Program Interface
Introduction to OpenMP.
Introduction to OpenMP www.openmp.org Motivation Parallelize the following code using threads: for (i=0; i
OpenMP I. Diego Fabregat-Traver and Prof. Paolo Bientinesi WS16/17. HPAC, RWTH Aachen
OpenMP I Diego Fabregat-Traver and Prof. Paolo Bientinesi HPAC, RWTH Aachen fabregat@aices.rwth-aachen.de WS16/17 OpenMP References Using OpenMP: Portable Shared Memory Parallel Programming. The MIT Press,
Data Environment: Default storage attributes
COSC 6374 Parallel Computation Introduction to OpenMP(II) Some slides based on material by Barbara Chapman (UH) and Tim Mattson (Intel) Edgar Gabriel Fall 2014 Data Environment: Default storage attributes
Allows program to be incrementally parallelized
Basic OpenMP What is OpenMP An open standard for shared memory programming in C/C+ + and Fortran supported by Intel, Gnu, Microsoft, Apple, IBM, HP and others Compiler directives and library support OpenMP
OpenMP - Introduction
OpenMP - Introduction Süha TUNA Bilişim Enstitüsü UHeM Yaz Çalıştayı - 21.06.2012 Outline What is OpenMP? Introduction (Code Structure, Directives, Threads etc.) Limitations Data Scope Clauses Shared,
Shared Memory Parallelism using OpenMP
Indian Institute of Science Bangalore, India भ रत य व ज ञ न स स थ न ब गल र, भ रत SE 292: High Performance Computing [3:0][Aug:2014] Shared Memory Parallelism using OpenMP Yogesh Simmhan Adapted from: o
Barbara Chapman, Gabriele Jost, Ruud van der Pas
Using OpenMP Portable Shared Memory Parallel Programming Barbara Chapman, Gabriele Jost, Ruud van der Pas The MIT Press Cambridge, Massachusetts London, England c 2008 Massachusetts Institute of Technology
Shared Memory Programming with OpenMP
Shared Memory Programming with OpenMP (An UHeM Training) Süha Tuna Informatics Institute, Istanbul Technical University February 12th, 2016 2 Outline - I Shared Memory Systems Threaded Programming Model
HPC Practical Course Part 3.1 Open Multi-Processing (OpenMP)
HPC Practical Course Part 3.1 Open Multi-Processing (OpenMP) V. Akishina, I. Kisel, G. Kozlov, I. Kulakov, M. Pugach, M. Zyzak Goethe University of Frankfurt am Main 2015 Task Parallelism Parallelization
CSL 860: Modern Parallel
CSL 860: Modern Parallel Computation Hello OpenMP #pragma omp parallel { // I am now thread iof n switch(omp_get_thread_num()) { case 0 : blah1.. case 1: blah2.. // Back to normal Parallel Construct Extremely
OpenMP examples. Sergeev Efim. Singularis Lab, Ltd. Senior software engineer
OpenMP examples Sergeev Efim Senior software engineer Singularis Lab, Ltd. OpenMP Is: An Application Program Interface (API) that may be used to explicitly direct multi-threaded, shared memory parallelism.
Parallelising Scientific Codes Using OpenMP. Wadud Miah Research Computing Group
Parallelising Scientific Codes Using OpenMP Wadud Miah Research Computing Group Software Performance Lifecycle Scientific Programming Early scientific codes were mainly sequential and were executed on
Module 10: Open Multi-Processing Lecture 19: What is Parallelization? The Lecture Contains: What is Parallelization? Perfectly Load-Balanced Program
The Lecture Contains: What is Parallelization? Perfectly Load-Balanced Program Amdahl's Law About Data What is Data Race? Overview to OpenMP Components of OpenMP OpenMP Programming Model OpenMP Directives
Parallel Processing Top manufacturer of multiprocessing video & imaging solutions.
1 of 10 3/3/2005 10:51 AM Linux Magazine March 2004 C++ Parallel Increase application performance without changing your source code. Parallel Processing Top manufacturer of multiprocessing video & imaging
Parallel programming using OpenMP
Parallel programming using OpenMP Computer Architecture J. Daniel García Sánchez (coordinator) David Expósito Singh Francisco Javier García Blas ARCOS Group Computer Science and Engineering Department
OpenMP Algoritmi e Calcolo Parallelo. Daniele Loiacono
OpenMP Algoritmi e Calcolo Parallelo References Useful references Using OpenMP: Portable Shared Memory Parallel Programming, Barbara Chapman, Gabriele Jost and Ruud van der Pas OpenMP.org http://openmp.org/
Introduction to OpenMP
Introduction to OpenMP Le Yan HPC Consultant User Services Goals Acquaint users with the concept of shared memory parallelism Acquaint users with the basics of programming with OpenMP Discuss briefly the
Multithreading in C with OpenMP
Multithreading in C with OpenMP ICS432 - Spring 2017 Concurrent and High-Performance Programming Henri Casanova (henric@hawaii.edu) Pthreads are good and bad! Multi-threaded programming in C with Pthreads
CS 470 Spring Mike Lam, Professor. OpenMP
CS 470 Spring 2018 Mike Lam, Professor OpenMP OpenMP Programming language extension Compiler support required "Open Multi-Processing" (open standard; latest version is 4.5) Automatic thread-level parallelism
Introduction to. Slides prepared by : Farzana Rahman 1
Introduction to OpenMP Slides prepared by : Farzana Rahman 1 Definition of OpenMP Application Program Interface (API) for Shared Memory Parallel Programming Directive based approach with library support
OpenMP. OpenMP. Portable programming of shared memory systems. It is a quasi-standard. OpenMP-Forum API for Fortran and C/C++
OpenMP OpenMP Portable programming of shared memory systems. It is a quasi-standard. OpenMP-Forum 1997-2002 API for Fortran and C/C++ directives runtime routines environment variables www.openmp.org 1
CS 470 Spring Mike Lam, Professor. OpenMP
CS 470 Spring 2017 Mike Lam, Professor OpenMP OpenMP Programming language extension Compiler support required "Open Multi-Processing" (open standard; latest version is 4.5) Automatic thread-level parallelism
1 of 6 Lecture 7: March 4. CISC 879 Software Support for Multicore Architectures Spring Lecture 7: March 4, 2008
1 of 6 Lecture 7: March 4 CISC 879 Software Support for Multicore Architectures Spring 2008 Lecture 7: March 4, 2008 Lecturer: Lori Pollock Scribe: Navreet Virk Open MP Programming Topics covered 1. Introduction
Parallel Computing Parallel Programming Languages Hwansoo Han
Parallel Computing Parallel Programming Languages Hwansoo Han Parallel Programming Practice Current Start with a parallel algorithm Implement, keeping in mind Data races Synchronization Threading syntax
UvA-SARA High Performance Computing Course June Clemens Grelck, University of Amsterdam. Parallel Programming with Compiler Directives: OpenMP
Parallel Programming with Compiler Directives OpenMP Clemens Grelck University of Amsterdam UvA-SARA High Performance Computing Course June 2013 OpenMP at a Glance Loop Parallelization Scheduling Parallel
COSC 6374 Parallel Computation. Introduction to OpenMP. Some slides based on material by Barbara Chapman (UH) and Tim Mattson (Intel)
COSC 6374 Parallel Computation Introduction to OpenMP Some slides based on material by Barbara Chapman (UH) and Tim Mattson (Intel) Edgar Gabriel Fall 2015 OpenMP Provides thread programming model at a
Parallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 17 Shared-memory Programming 1 Outline n OpenMP n Shared-memory model n Parallel for loops n Declaring private variables n Critical
Parallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 17 Shared-memory Programming 1 Outline n OpenMP n Shared-memory model n Parallel for loops n Declaring private variables n Critical
EE/CSCI 451 Introduction to Parallel and Distributed Computation. Discussion #4 2/3/2017 University of Southern California
EE/CSCI 451 Introduction to Parallel and Distributed Computation Discussion #4 2/3/2017 University of Southern California 1 USC HPCC Access Compile Submit job OpenMP Today s topic What is OpenMP OpenMP
Introduction to OpenMP
Introduction to OpenMP Le Yan Objectives of Training Acquaint users with the concept of shared memory parallelism Acquaint users with the basics of programming with OpenMP Memory System: Shared Memory
Multi-core Architecture and Programming
Multi-core Architecture and Programming Yang Quansheng( 杨全胜 ) http://www.njyangqs.com School of Computer Science & Engineering 1 http://www.njyangqs.com Programming with OpenMP Content What is PpenMP Parallel
CMSC 714 Lecture 4 OpenMP and UPC. Chau-Wen Tseng (from A. Sussman)
CMSC 714 Lecture 4 OpenMP and UPC Chau-Wen Tseng (from A. Sussman) Programming Model Overview Message passing (MPI, PVM) Separate address spaces Explicit messages to access shared data Send / receive (MPI
OpenMP. Application Program Interface. CINECA, 14 May 2012 OpenMP Marco Comparato
OpenMP Application Program Interface Introduction Shared-memory parallelism in C, C++ and Fortran compiler directives library routines environment variables Directives single program multiple data (SPMD)
Parallel and Distributed Programming. OpenMP
Parallel and Distributed Programming OpenMP OpenMP Portability of software SPMD model Detailed versions (bindings) for different programming languages Components: directives for compiler library functions
Programming Shared Memory Systems with OpenMP Part I. Book
Programming Shared Memory Systems with OpenMP Part I Instructor Dr. Taufer Book Parallel Programming in OpenMP by Rohit Chandra, Leo Dagum, Dave Kohr, Dror Maydan, Jeff McDonald, Ramesh Menon 2 1 Machine
Distributed Systems + Middleware Concurrent Programming with OpenMP
Distributed Systems + Middleware Concurrent Programming with OpenMP Gianpaolo Cugola Dipartimento di Elettronica e Informazione Politecnico, Italy cugola@elet.polimi.it http://home.dei.polimi.it/cugola
Shared Memory programming paradigm: openmp
IPM School of Physics Workshop on High Performance Computing - HPC08 Shared Memory programming paradigm: openmp Luca Heltai Stefano Cozzini SISSA - Democritos/INFM
OpenMP: Open Multiprocessing
OpenMP: Open Multiprocessing Erik Schnetter June 7, 2012, IHPC 2012, Iowa City Outline 1. Basic concepts, hardware architectures 2. OpenMP Programming 3. How to parallelise an existing code 4. Advanced
[Potentially] Your first parallel application
![[Potentially] Your first parallel application](/thumbs/71/65941908.jpg "[Potentially] Your first parallel application")
[Potentially] Your first parallel application Compute the smallest element in an array as fast as possible small = array[0]; for( i = 0; i < N; i++) if( array[i] < small ) ) small = array[i] 64-bit Intel
OpenMP. Diego Fabregat-Traver and Prof. Paolo Bientinesi WS16/17. HPAC, RWTH Aachen
OpenMP Diego Fabregat-Traver and Prof. Paolo Bientinesi HPAC, RWTH Aachen fabregat@aices.rwth-aachen.de WS16/17 Worksharing constructs To date: #pragma omp parallel created a team of threads We distributed
OpenMP Introduction. CS 590: High Performance Computing. OpenMP. A standard for shared-memory parallel programming. MP = multiprocessing
CS 590: High Performance Computing OpenMP Introduction Fengguang Song Department of Computer Science IUPUI OpenMP A standard for shared-memory parallel programming. MP = multiprocessing Designed for systems
Introduction to OpenMP. Martin Čuma Center for High Performance Computing University of Utah
Introduction to OpenMP Martin Čuma Center for High Performance Computing University of Utah mcuma@chpc.utah.edu Overview Quick introduction. Parallel loops. Parallel loop directives. Parallel sections.
Little Motivation Outline Introduction OpenMP Architecture Working with OpenMP Future of OpenMP End. OpenMP. Amasis Brauch German University in Cairo
OpenMP Amasis Brauch German University in Cairo May 4, 2010 Simple Algorithm 1 void i n c r e m e n t e r ( short a r r a y ) 2 { 3 long i ; 4 5 for ( i = 0 ; i < 1000000; i ++) 6 { 7 a r r a y [ i ]++;
15-418, Spring 2008 OpenMP: A Short Introduction
15-418, Spring 2008 OpenMP: A Short Introduction This is a short introduction to OpenMP, an API (Application Program Interface) that supports multithreaded, shared address space (aka shared memory) parallelism.
CS 470 Spring Mike Lam, Professor. Advanced OpenMP
CS 470 Spring 2018 Mike Lam, Professor Advanced OpenMP Atomics OpenMP provides access to highly-efficient hardware synchronization mechanisms Use the atomic pragma to annotate a single statement Statement
OpenMP Application Program Interface
OpenMP Application Program Interface DRAFT Version.1.0-00a THIS IS A DRAFT AND NOT FOR PUBLICATION Copyright 1-0 OpenMP Architecture Review Board. Permission to copy without fee all or part of this material
OpenMP Overview. in 30 Minutes. Christian Terboven / Aachen, Germany Stand: Version 2.
OpenMP Overview in 30 Minutes Christian Terboven 06.12.2010 / Aachen, Germany Stand: 03.12.2010 Version 2.3 Rechen- und Kommunikationszentrum (RZ) Agenda OpenMP: Parallel Regions,
Parallel Programming using OpenMP
1 OpenMP Multithreaded Programming 2 Parallel Programming using OpenMP OpenMP stands for Open Multi-Processing OpenMP is a multi-vendor (see next page) standard to perform shared-memory multithreading
Parallel Programming using OpenMP
1 Parallel Programming using OpenMP Mike Bailey mjb@cs.oregonstate.edu openmp.pptx OpenMP Multithreaded Programming 2 OpenMP stands for Open Multi-Processing OpenMP is a multi-vendor (see next page) standard
Alfio Lazzaro: Introduction to OpenMP
First INFN International School on Architectures, tools and methodologies for developing efficient large scale scientific computing applications Ce.U.B. Bertinoro Italy, 12 17 October 2009 Alfio Lazzaro:
Parallel Programming with OpenMP. CS240A, T. Yang
Parallel Programming with OpenMP CS240A, T. Yang 1 A Programmer s View of OpenMP What is OpenMP? Open specification for Multi-Processing Standard API for defining multi-threaded shared-memory programs
Amdahl s Law. AMath 483/583 Lecture 13 April 25, Amdahl s Law. Amdahl s Law. Today: Amdahl s law Speed up, strong and weak scaling OpenMP
AMath 483/583 Lecture 13 April 25, 2011 Amdahl s Law Today: Amdahl s law Speed up, strong and weak scaling OpenMP Typically only part of a computation can be parallelized. Suppose 50% of the computation
CS 5220: Shared memory programming. David Bindel
CS 5220: Shared memory programming David Bindel 2017-09-26 1 Message passing pain Common message passing pattern Logical global structure Local representation per processor Local data may have redundancy
Chip Multiprocessors COMP Lecture 9 - OpenMP & MPI
Chip Multiprocessors COMP35112 Lecture 9 - OpenMP & MPI Graham Riley 14 February 2018 1 Today s Lecture Dividing work to be done in parallel between threads in Java (as you are doing in the labs) is rather
Shared memory parallel computing
Shared memory parallel computing OpenMP Sean Stijven Przemyslaw Klosiewicz Shared-mem. programming API for SMP machines Introduced in 1997 by the OpenMP Architecture Review Board! More high-level than
Shared memory programming
CME342- Parallel Methods in Numerical Analysis Shared memory programming May 14, 2014 Lectures 13-14 Motivation Popularity of shared memory systems is increasing: Early on, DSM computers (SGI Origin 3000
Introduction [1] 1. Directives [2] 7
![Introduction [1] 1. Directives [2] 7](/thumbs/74/71246585.jpg "Introduction [1] 1. Directives [2] 7")
OpenMP Fortran Application Program Interface Version 2.0, November 2000 Contents Introduction [1] 1 Scope............................. 1 Glossary............................ 1 Execution Model.........................
Parallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 17 Shared-memory Programming Outline OpenMP Shared-memory model Parallel for loops Declaring private variables Critical sections Reductions
OPENMP OPEN MULTI-PROCESSING
OPENMP OPEN MULTI-PROCESSING OpenMP OpenMP is a portable directive-based API that can be used with FORTRAN, C, and C++ for programming shared address space machines. OpenMP provides the programmer with
CS691/SC791: Parallel & Distributed Computing
CS691/SC791: Parallel & Distributed Computing Introduction to OpenMP 1 Contents Introduction OpenMP Programming Model and Examples OpenMP programming examples Task parallelism. Explicit thread synchronization.
Shared Memory Programming with OpenMP (3)
Shared Memory Programming with OpenMP (3) 2014 Spring Jinkyu Jeong (jinkyu@skku.edu) 1 SCHEDULING LOOPS 2 Scheduling Loops (2) parallel for directive Basic partitioning policy block partitioning Iteration
OPENMP TIPS, TRICKS AND GOTCHAS
OPENMP TIPS, TRICKS AND GOTCHAS Mark Bull EPCC, University of Edinburgh (and OpenMP ARB) markb@epcc.ed.ac.uk OpenMPCon 2015 OpenMPCon 2015 2 A bit of background I ve been teaching OpenMP for over 15 years
A Short Introduction to OpenMP. Mark Bull, EPCC, University of Edinburgh
A Short Introduction to OpenMP Mark Bull, EPCC, University of Edinburgh Overview Shared memory systems Basic Concepts in Threaded Programming Basics of OpenMP Parallel regions Parallel loops 2 Shared memory
An Introduction to OpenMP
An Introduction to OpenMP U N C L A S S I F I E D Slide 1 What Is OpenMP? OpenMP Is: An Application Program Interface (API) that may be used to explicitly direct multi-threaded, shared memory parallelism
OpenMP C and C++ Application Program Interface Version 1.0 October Document Number
OpenMP C and C++ Application Program Interface Version 1.0 October 1998 Document Number 004 2229 001 Contents Page v Introduction [1] 1 Scope............................. 1 Definition of Terms.........................
Parallel Programming
Parallel Programming OpenMP Nils Moschüring PhD Student (LMU) Nils Moschüring PhD Student (LMU), OpenMP 1 1 Overview What is parallel software development Why do we need parallel computation? Problems
Jukka Julku Multicore programming: Low-level libraries. Outline. Processes and threads TBB MPI UPC. Examples
Multicore Jukka Julku 19.2.2009 1 2 3 4 5 6 Disclaimer There are several low-level, languages and directive based approaches But no silver bullets This presentation only covers some examples of them is
Shared memory programming model OpenMP TMA4280 Introduction to Supercomputing
Shared memory programming model OpenMP TMA4280 Introduction to Supercomputing NTNU, IMF February 16. 2018 1 Recap: Distributed memory programming model Parallelism with MPI. An MPI execution is started
OpenMP: Open Multiprocessing
OpenMP: Open Multiprocessing Erik Schnetter May 20-22, 2013, IHPC 2013, Iowa City 2,500 BC: Military Invents Parallelism Outline 1. Basic concepts, hardware architectures 2. OpenMP Programming 3. How to
OpenMP. Dr. William McDoniel and Prof. Paolo Bientinesi WS17/18. HPAC, RWTH Aachen
OpenMP Dr. William McDoniel and Prof. Paolo Bientinesi HPAC, RWTH Aachen mcdoniel@aices.rwth-aachen.de WS17/18 Loop construct - Clauses #pragma omp for [clause [, clause]...] The following clauses apply:
OPENMP TIPS, TRICKS AND GOTCHAS
OPENMP TIPS, TRICKS AND GOTCHAS OpenMPCon 2015 2 Directives Mistyping the sentinel (e.g.!omp or #pragma opm ) typically raises no error message. Be careful! Extra nasty if it is e.g. #pragma opm atomic
COMP4510 Introduction to Parallel Computation. Shared Memory and OpenMP. Outline (cont d) Shared Memory and OpenMP
COMP4510 Introduction to Parallel Computation Shared Memory and OpenMP Thanks to Jon Aronsson (UofM HPC consultant) for some of the material in these notes. Outline (cont d) Shared Memory and OpenMP Including
Mango DSP Top manufacturer of multiprocessing video & imaging solutions.
1 of 11 3/3/2005 10:50 AM Linux Magazine February 2004 C++ Parallel Increase application performance without changing your source code. Mango DSP Top manufacturer of multiprocessing video & imaging solutions.
High Performance Computing: Tools and Applications
High Performance Computing: Tools and Applications Edmond Chow School of Computational Science and Engineering Georgia Institute of Technology Lecture 2 OpenMP Shared address space programming High-level
by system default usually a thread per CPU or core using the environment variable OMP_NUM_THREADS from within the program by using function call
OpenMP Syntax The OpenMP Programming Model Number of threads are determined by system default usually a thread per CPU or core using the environment variable OMP_NUM_THREADS from within the program by
Introduction to OpenMP
Introduction to OpenMP Xiaoxu Guan High Performance Computing, LSU April 6, 2016 LSU HPC Training Series, Spring 2016 p. 1/44 Overview Overview of Parallel Computing LSU HPC Training Series, Spring 2016
!OMP #pragma opm _OPENMP
Advanced OpenMP Lecture 12: Tips, tricks and gotchas Directives Mistyping the sentinel (e.g.!omp or #pragma opm ) typically raises no error message. Be careful! The macro _OPENMP is defined if code is
Shared Memory Programming Paradigm!
Shared Memory Programming Paradigm! Ivan Girotto igirotto@ictp.it Information & Communication Technology Section (ICTS) International Centre for Theoretical Physics (ICTP) 1 Multi-CPUs & Multi-cores NUMA
An Introduction to OpenMP
Dipartimento di Ingegneria Industriale e dell'informazione University of Pavia December 4, 2017 Recap Parallel machines are everywhere Many architectures, many programming model. Among them: multithreading.