An Introduction to OpenMP

Transcription

1 Dipartimento di Ingegneria Industriale e dell'informazione University of Pavia December 4, 2017

2 Recap Parallel machines are everywhere Many architectures, many programming model. Among them: multithreading. Multithreading is the natural programming model for multicore and SMP machines (desktop, server, embedded, cluster): All processors share the same memory Threads in a process see the same address space Multithreading is (correctly) perceived to be hard! Lots of expertise necessary, dependencies, granularity, balancing... Non-deterministic behavior makes it hard to debug

3 Outline 1 Introduction Motivation What is? 2 3 Complete Example Summary

4 Motivation What is? Outline 1 Introduction Motivation What is? 2 3 Complete Example Summary

5 Motivation What is? Ideal Parallel Programming Ideal Parallel Programming Paradigm 1 Start with any algorithm Embarrassing parallelism is helpful, but not necessary 2 Implement serially, ignoring: Data Races Threading Syntax 3 Test and Debug 4 Automatically (magically?) parallelize Expect linear speedup

6 Motivation What is? Threading Programming Paradigm Threading Programming Paradigm 1 Start with a parallel algorithm 2 Implement, keeping in mind: Data races Threading Syntax 3 Test & Debug 4 Debug 5 Debug

7 Motivation What is? Problem Parallelize the following code using threads: i n t main ( ) { f o r ( i n t i =0; i <16; i ++) p r i n t f ( " H e l l o, World! \ n" ) ; } A lot of work to do a simple thing Dierent threading APIs: Windows: CreateThread UNIX: pthread_create Problems with the code: Dierent code for serial and parallel version Is it possible to check for number of processors?

8 Motivation What is? Motivation - Threading Library v o i d S a y H e l l o ( v o i d f o o ) { p r i n t f ( " H e l l o, w o r l d! \ n" ) ; r e t u r n NULL ; } i n t main ( ) { pthread_attr_t a t t r ; pthread_t t h r e a d s [ 1 6 ] ; i n t tn ; p t h r e a d _ a t t r _ i n i t (& a t t r ) ; p t h r e a d _ a t t r _ s e t s c o p e (& a t t r, PTHREAD_SCOPE_SYSTEM) ; } f o r ( tn =0; tn <16; tn++) { p t h r e a d _ c r e a t e (& t h r e a d s [ tn ], &a t t r, SayHello, NULL) ; } f o r ( tn =0; tn <16 ; tn++) { p t h r e a d _ j o i n ( t h r e a d s [ tn ], NULL) ; } r e t u r n 0 ;

9 Motivation What is? Motivation - Threading Library II Thread libraries are hard to use PThreads has many library calls for initialization, synchronization, thread creation, condition variables, etc. Programmer must code with multiple threads in mind between threads introduces a new dimension of program correctness Wouldn't it be nice to write serial programs and somehow parallelize them automatically? can parallelize many serial programs with relatively few annotations that specify parallelism and independence is a small API that hides cumbersome threading calls with simpler directives

10 Motivation What is? Motivation - i n t main ( ) { // Do t h i s p a r t i n p a r a l l e l p r i n t f ( " H e l l o, World! \ n" ) ; } r e t u r n 0 ;

11 Motivation What is? Motivation - i n t main ( ) { omp_set_num_threads ( 1 6 ) ; // Do t h i s p a r t i n p a r a l l e l #pragma omp p a r a l l e l { p r i n t f ( " H e l l o, World! \ n" ) ; } } r e t u r n 0 ;

12 Motivation What is? Parallel Programming Programming Paradigm 1 Start with a parallelizable algorithm Embarrassing parallelism is good, loop-level parallelism is necessary 2 Implement serially, mostly ignoring: Data Races Threading Syntax 3 Test and Debug 4 Annotate the code with parallelization (and sync) directives Hope for linear speedup 5 Test and Debug

13 Motivation What is? Outline 1 Introduction Motivation What is? 2 3 Complete Example Summary

14 Motivation What is? What is? What is? Portable, threaded, shared-memory, standard programming specication with light syntax Talks, examples, forums, etc. Last release: 4.5 (November 2015) Requires compiler support (C or Fortran) Exact behavior depends on implementation! High-level API Preprocessor (compiler) directives ( ~ 80% ) Library Calls ( ~ 19% ) Environment Variables ( ~ 1% )

15 Motivation What is? A Programmer's View of will: Allow a programmer to separate a program into serial regions and parallel regions rather than T concurrent threads. Hide stack management Provide synchronization constructs will not: Parallelize (or detect!) dependencies Guarantee speedup Provide freedom from data races

16 Motivation What is? Methodology Parallelization with is an optimization process. Proceed with care: Start with a working program, then add parallelization Measure the changes after every step Remember Amdahl's law. Use the prolertools available

17 Outline Introduction 1 Introduction Motivation What is? 2 3 Complete Example Summary

18 Syntax Introduction Most of the constructs of are pragmas #pragma omp c o n s t r u c t [ c l a u s e [ c l a u s e ]... ] An construct applies to a structural block (open with {, close with }) In addition: Several omp_<something> function calls Several OMP_<something> environment variables

19 Controlling Behavior Function calls and similar environment variables omp_set_dynamic(int) omp_set_num_threads(int) omp_get_num_threads() omp_get_num_procs() omp_get_thread_num() omp_set_nested(int) omp_in_parallel() omp_get_wtime()

20 Programming Model Introduction Thread-like fork/join model Arbitrary number of logical thread creation/ destruction events Fork and join are implicit Serial regions by default, annotate to create parallel regions Using #pragmas Generic parallel regions, loops, sections

21 Thread Identication Introduction Master Thread Thread with ID=0 Only thread that exists in sequential regions Some special directives aect only the master thread (like master)

22 Nested Threading Introduction Fork/Join can be nested Nesting complication handled transparently at compile-time Independent of the number of threads actually running Avoid if possibile: possible performance problems

23 Data/Control Parallelism Data parallelism Threads perform similar functions, guided by thread identier Control parallelism Threads perform diering functions One thread for I/O, one for computation, etc...

24 Outline Introduction 1 Introduction Motivation What is? 2 3 Complete Example Summary

25 Parallel Regions I Introduction Main construct: #pragma omp parallel Denes a parallel region over structured block of code Threads are created as parallel pragma is crossed Threads block at end of region (implicit barrier)

26 Parallel Regions II Introduction double D[ ] ; #pragma omp p a r a l l e l { i n t i ; double sum = 0 ; f o r ( i =0; i <1000; i ++) sum += D[ i ] ; p r i n t f ( " Thread %d computes %f \n", omp_thread_num ( ), sum ) ; } Executes the same code as many times as there are threads How many threads do we have? omp_set_num_threads(n) What is the use of repeating the same work n times in parallel? Can use omp_thread_num() to distribute the work between threads. D is shared between the threads, i and sum are private

27 Implementing Work Sharing Sequential code f o r ( i n t i =0; i <N; i ++) { a [ i ]=b [ i ]+ c [ i ] ; } (Semi) manual parallelization #pragma omp p a r a l l e l { i n t i d = omp_get_thread_num ( ) ; i n t Nthr = omp_get_num_threads ( ) ; i n t i s t a r t = i d N/ Nthr ; i n t i e n d = ( i d +1) N/ Nthr ; f o r ( i n t i=i s t a r t ; i <i e n d ; i ++) { a [ i ]=b [ i ]+ c [ i ] ; } } Automatic parallelization #pragma omp p a r a l l e l f o r s c h e d u l e ( s t a t i c ) { f o r ( i n t i =0; i <N; i ++) { a [ i ]=b [ i ]+ c [ i ] ; } }

28 Work Sharing: For Introduction Used to assign each thread an independent set of iterations Threads must wait at the end Can combine the directives: #pragma omp parallel for Only simple kinds of for loops: Only one signed integer variable Initialization: var=init Comparison: var op last op: <, >, <=, >= Increment: var++, var--, var+=incr, var-=incr, etc.

29 Problems of #parallel for Load balancing If all the iterations execute at the same speed, the processors are used optimally If some iterations are faster than others, some processors may get idle, reducing the speedup We don't always know the distribution of work, may need to re-distribute dynamically Granularity Thread creation and synchronization takes time Assigning work to threads on per-iteration resolution may take more time than the execution itself! Need to coalesce the work to coarse chunks to overcome the threading overhead Trade-o between load balancing and granularity!

30 Controlling Granularity #pragma omp parallel if (expression) Can be used to disable parallelization in some cases (when the input is determined to be too small to be benecially multithreaded) #pragma omp num_threads (expression) Control the number of threads used in this parallel region

31 Schedule Clause: Controlling Work Distribution schedule(static [, chunksize]) Default: chunks of approximately equivalent size, one to each thread If more chunks than threads: assigned in round-robin to the threads Why might we want to use chunks of dierent size? schedule(dynamic [, chunksize]) Threads receive chunk assignments dynamically Default chunk size = 1 (why?) schedule(guided [, chunksize]) Start with large chunks Threads receive chunks dynamically Chunk size reduces exponentially, down to chunksize

32 Graphic Scheduling Introduction

33 Scheduling Example Introduction The function TestForPrime (usually) takes little time But can take long, if the number is a prime indeed #pragma omp p a r a l l e l f o r s c h e d u l e???? f o r ( i n t i = s t a r t ; i <= end ; i += 2 ) { i f ( TestForPrime ( i ) ) gprimesfound++; } Solution: use dynamic, but with chunks

34 Scheduling Example Introduction The function TestForPrime (usually) takes little time But can take long, if the number is a prime indeed #pragma omp p a r a l l e l f o r s c h e d u l e???? f o r ( i n t i = s t a r t ; i <= end ; i += 2 ) { i f ( TestForPrime ( i ) ) gprimesfound++; } Solution: use dynamic, but with chunks

35 Work Sharing: Sections answer1 = long_computation_1 ( ) ; answer2 = long_computation_2 ( ) ; i f ( answer1!= answer2 ) {... } How to parallelize? These are just two independent computations! #pragma omp s e c t i o n s { #pragma omp s e c t i o n answer1 = long_computation_1 ( ) ; #pragma omp s e c t i o n answer2 = long_computation_2 ( ) ; } i f ( answer1!= answer2 ) {... }

36 Outline Introduction 1 Introduction Motivation What is? 2 3 Complete Example Summary

37 Introduction Shared memory communication Threads cooperates by accessing shared variables Denition: Any variable that is seen by two or more threads is shared Any variable that is seen by one thread only is private Race conditions possible Use synchronization to protect from conicts Change how data is stored to minimize the synchronization

38 Data Visibility Introduction Shared Memory programming model Most variables (including locals) are shared by default { } i n t sum = 0 ; #pragma omp p a r a l l e l f o r f o r ( i n t i =0; i <N; i ++) sum += i ; Global variables are shared Some variables can be private Automatic variables inside the statement block Automatic variables in the called functions Variables can be explicitly declared as private. In that case, a local copy is created for each thread

39 Overriding Storage Attributes private: A copy of the variable is created for each thread. No connection between the original variable and the private copies Can achieve the same using variables inside { } i n t i ; #pragma omp p a r a l l e l f o r \ p r i v a t e ( i ) f o r ( i =0; i <n ; i ++) {... }

40 Overriding Storage Attributes II rstprivate: Same, but the initial value is copied from the main copy lastprivate: Same, but the last value is copied to the main copy i n t i d x =1; i n t x = 1 0 ; #pragma omp p a r a l l e l f o r \ f i r s p r i v a t e ( x ) \ l a s t p r i v a t e ( i d x ) f o r ( i =0; i <n ; i ++) { i f ( data [ i ]==x ) i d x = i ; }

41 Threadprivate Introduction Similar to private, but dened per variable Declaration immediately after variable denition. Must be visible in all translation units. Persistent between parallel sections Can be initialized from the master's copy with #pragma omp copyin More ecient than private, but a global variable! Example: i n t data [ ] ; #pragma omp t h r e a d p r i v a t e ( data )... #pragma omp p a r a l l e l f o r c o p y i n ( data ) f o r (... )

42 Outline Introduction 1 Introduction Motivation What is? 2 3 Complete Example Summary

43 Introduction X = 0 ; #pragma omp X = X+1; p a r a l l e l What should the result be (assuming 2 threads)? 2 is the expected answer But can be 1 with unfortunate interleaving assumes that the programmer knows what he is doing Regions of code that are marked to run in parallel are independent If access collisions are possible, it is the programmer's responsibility to insert protection

44 Mechanisms Many of the existing mechanisms for shared programming Nowait (turn synchronization o!) Single/Master execution Critical sections, Atomic updates Ordered Barriers Flush (memory subsystem synchronization) Reduction (special case)

45 Single/Master Introduction #pragma omp single Only one of the threads will execute the following block of code The rest will wait for it to complete Good for non-thread-safe regions of code (such as I/O) Must be used in a parallel region Applicable to parallel for sections

46 Single/Master II Introduction #pragma omp master The following block will be executed by the master thread No synchronization involved Applicable only to parallel sections #pragma omp p a r a l l e l { d o _ p r e p r o c e s s i n g ( ) ; #pragma omp s i n g l e read_input ( ) ; #pragma omp master notify_input_consumed ( ) ; } d o _ p r o c e s s i n g ( ) ;

47 Critical Sections Introduction #pragma omp critical [name] Standard critical section functionality Critical sections are global in the program Can be used to protect a single resource in dierent functions Critical sections are identied by the name All the unnamed critical sections are mutually exclusive throughout the program All the critical sections having the same name are mutually exclusive between themselves i n t x = 0 ; #pragma omp p a r a l l e l s h a r e d ( x ) { #pragma omp c r i t i c a l x++; }

48 Atomic Execution Introduction Critical sections on the cheap Protects a single variable update Can be much more ecient (a dedicated assembly instruction on some architectures) #pragma omp atomic update_statement Update statement is one of: var= var op expr, var op= expr, var++, var. The variable must be a scalar The operation op is one of: +, -, *, /, ^, &,, <<, >> The evaluation of expr is not atomic!

49 Ordered Introduction #pragma omp ordered statement Executes the statement in the sequential order of iterations Example: #pragma omp p a r a l l e l f o r f o r ( j =0; j <N; j ++) { i n t r e s u l t = heavy_computation ( j ) ; #pragma omp o r d e r e d p r i n t f ( " computation(%d ) = %d\n", j, r e s u l t ) ; }

50 Barrier synchronization #pragma omp barrier Performs a barrier synchronization between all the threads in a team at the given point. Example: #pragma omp p a r a l l e l { i n t r e s u l t = heavy_computation_part1 ( ) ; #pragma omp atomic sum += r e s u l t ; #pragma omp b a r r i e r heavy_computation_part2 ( sum ) ; }

51 Explicit Locking Introduction Can be used to pass lock variables around Can be used to implement more involved synchronization constructs Functions: omp_init_lock(), omp_destroy_lock(), omp_set_lock(), omp_unset_lock(), omp_test_lock() The usual semantics Use #pragma omp ush to synchronize memory Avoid it: very easy to screw up!

52 Reduction Introduction f o r ( j =0; j <N; j ++) { sum = sum+a [ j ] b [ j ] ; } How to parallelize this code? sum is not private, but accessing it atomically is too expensive Have a private copy of sum in each thread, then add them up Use the reduction clause! #pragma omp parallel for reduction(+: sum)any associative operator must be used: +, -,,, *, etc. The private value is initialized automatically (to 0, 1, ~0... )

53 Overhead Lost time waiting for locks Prefer to use structures that are as lock-free as possible! Use parallelization granularity which is as large as possible #pragma omp p a r a l l e l { #pragma omp c r i t i c a l {... }... }

54 Complete Example Summary Outline 1 Introduction Motivation What is? 2 3 Complete Example Summary

55 Complete Example Summary Numerical Integration Mathematically, we know that: (1 + x 2 ) dx = π We can approximate the integral as a sum of rectangles: N i=0 F (x i ) x π Where each rectangle has width x and height F (x i ) at the middle of interval i.

56 Complete Example Summary Serial Code s t a t i c long num_steps =100000; double step, p i ; v o i d main ( ) { i n t i ; double x, sum = 0. 0 ; } s t e p = 1. 0 / ( double ) num_steps ; f o r ( i =0; i < num_steps ; i ++){ x = ( i +0.5) s t e p ; sum = sum / ( x x ) ; } p i = s t e p sum ; p r i n t f ( " Pi = %f \n", p i ) ; Parallelize the numerical integration code using What variables can be shared? What variables need to be private? What variables should be set up for reductions?

57 Complete Example Summary Parallel Code s t a t i c long num_steps =100000; double step, p i ; v o i d main ( ) { i n t i ; double x, sum = 0. 0 ; s t e p = 1. 0 / ( double ) num_steps ; #pragma omp p a r a l l e l f o r \ p r i v a t e ( x ) r e d u c t i o n (+:sum ) } f o r ( i =0; i < num_steps ; i ++){ x = ( i +0.5) s t e p ; sum = sum / ( x x ) ; } p i = s t e p sum ; p r i n t f ( " Pi = %f \n", p i ) ; Parallelization code is a one-liner! sum is a reduction, hence shared, variable i is private since it is the loop variable

58 Complete Example Summary Outline 1 Introduction Motivation What is? 2 3 Complete Example Summary

59 Complete Example Summary Summary : A framework for code parallelization Available for C++ and FORTRAN Based on a standard Implementations from a wide selection of vendors Easy to use Write (and debug!) code rst, parallelize later Parallelization can be incremental Parallelization can be turned o at runtime or compile time Code is still correct for a serial machine

60 Complete Example Summary Limitations requires compiler support Sun, Intel, GCC... Embedded system? does not parallelize dependencies Often does not detect dependencies Nasty race conditions still exist! is not guaranteed to divide work optimally among threads Programmer-tweakable with scheduling clauses Still lots of rope available

61 Appendix For Further Reading For Further Reading I Clay Breshears The art of Concurrency O'Really, Blaise Barney Introduction to,