Introduction to OpenMP

Transcription

1 MPSoC Architectures OpenMP Aberto Bosio, Associate Professor UM Microeectronic Departement Introduction to OpenMP What is OpenMP? Open specification for Muti-Processing Standard API for defining muti-threaded sharedmemory programs Taks, exampes, forums, etc. High-eve API Preprocessor (compier) directives ( ~ 80% ) Library Cas ( ~ 19% ) Environment Variabes ( ~ 1% ) 1

2 A Programmer s View of OpenMP OpenMP is a portabe, threaded, shared-memory programming specification with ight syntax Exact behavior depends on OpenMP impementation! Requires compier support (C or Fortran) OpenMP wi: Aow a programmer to separate a program into seria regions and parae regions, rather than T concurrentyexecuting threads. Hide stack management Provide synchronization constructs OpenMP wi not: Paraeize (or detect!) dependencies Guarantee speedup Provide freedom from data races Introduction Outine Motivating exampe Parae Programming is Hard OpenMP Programming Mode Easier than PThreads Microbenchmark Performance Comparison vs. PThreads Discussion specomp 2

3 Current Parae Programming 1. Start with a parae agorithm 2. Impement, keeping in mind: Data races Synchronization Threading Syntax 3. Test & Debug 4. Debug 5. Debug Motivation Threading Library void* SayHeo(void *foo) printf( "Heo, word!\n" ); return NULL; int main() pthread_attr_t attr; pthread_t threads[16]; int tn; pthread_attr_init(&attr); pthread_attr_setscope(&attr, PTHREAD_SCOPE_SYSTEM); for(tn=0; tn<16; tn++) pthread_create(&threads[tn], &attr, SayHeo, NULL); for(tn=0; tn<16 ; tn++) pthread_join(threads[tn], NULL); return 0; 3

4 Motivation Thread ibraries are hard to use P-Threads/Soaris threads have many ibrary cas for initiaization, synchronization, thread creation, condition variabes, etc. Programmer must code with mutipe threads in mind Synchronization between threads introduces a new dimension of program correctness Motivation Woudn t it be nice to write seria programs and somehow paraeize them automaticay? OpenMP can paraeize many seria programs with reativey few annotations that specify paraeism and independence OpenMP is a sma API that hides cumbersome threading cas with simper directives 4

5 Better Parae Programming 1. Start with some agorithm Embarrassing paraeism is hepfu, but not necessary 2. Impement seriay, ignoring: Data Races Synchronization Threading Syntax 3. Test and Debug 4. Automaticay (magicay?) paraeize Expect inear speedup Motivation OpenMP int main() // Do this part in parae printf( "Heo, Word!\n" ); return 0; 5

6 Motivation OpenMP int main() omp_set_num_threads(16); // Do this part in parae #pragma omp parae printf( "Heo, Word!\n" ); return 0; OpenMP Parae Programming 1. Start with a paraeizabe agorithm Embarrassing paraeism is good, oop-eve paraeism is necessary 2. Impement seriay, mosty ignoring: Data Races Synchronization Threading Syntax 3. Test and Debug 4. Annotate the code with paraeization (and synchronization) directives Hope for inear speedup 5. Test and Debug 6

7 Programming Mode - Threading Seria regions by defaut, annotate to create parae regions Generic parae regions Paraeized oops Sectioned parae regions Fork Thread-ike Fork/Join mode Arbitrary number of ogica thread creation/ destruction events Join Programming Mode - Threading int main() // seria region printf( Heo ); // parae region #pragma omp parae printf( Word ); // seria again printf(! ); Fork Join Heo WordWordWordWord! 7

8 Programming Mode Nested Threading Fork/Join can be nested Nesting compication handed automagicay at compie-time Independent of the number of threads actuay running Fork Fork Join Join Programming Mode Thread Identification Master Thread Thread with ID=0 Ony thread that exists in sequentia regions Depending on impementation, may have specia purpose inside parae regions Some specia directives affect ony the master thread (ike master) 0 Fork Join 0 8

9 Exampe int main() int tid, nthreads; omp_set_num_threads(16); // Do this part in parae #pragma omp parae private(nthreads, tid) printf( "Heo, Word!\n" ); /* Obtain and print thread id */ tid = omp_get_thread_num(); if (tid == 0) nthreads = omp_get_num_threads(); printf("i'm the master, Number of threads = %d\n", nthreads); Programming Mode Data/Contro Paraeism Data paraeism Threads perform simiar functions, guided by thread identifier Fork Contro paraeism Threads perform differing functions - One thread for I/O, one for computation, etc Join 9

10 Programming mode: Summary Memory Mode Shared memory communication Threads cooperates by accessing shared variabes The sharing is defined syntacticay Any variabe that is seen by two or more threads is shared Any variabe that is seen by one thread ony is private Race conditions possibe Use synchronization to protect from conficts Change how data is stored to minimize the synchronization 10

11 Structure Programming Mode Concurrent Loops OpenMP easiy paraeizes oops No data dependencies between iterations! Preprocessor cacuates oop bounds for each thread directy from seria source #pragma omp parae for for( i=0; i < 25; i++ ) printf( Foo ); 11

12 The probem Executes the same code as many times as there are threads How many threads do we have? omp_set_num_threads(n) What is the use of repeating the same work n times in parae? Can use omp_thread_num() to distribute the work between threads. D is shared between the threads, i and sum are private Programming Mode Concurrent Loops 12

13 Programming Mode Concurrent Loops Programming Mode Concurrent Loops Load baancing If a the iterations execute at the same speed, the processors are used optimay If some iterations are faster than others, some processors may get ide, reducing the speedup We don't aways know the distribution of work, may need to re-distribute dynamicay Granuarity Thread creation and synchronization takes time Assigning work to threads on per-iteration resoution may take more time than the execution itsef! Need to coaesce the work to coarse chunks to overcome the threading overhead Trade-off between oad baancing and granuarity! 13

14 Controing Granuarity #pragma omp parae if (expression) Can be used to disabe paraeization in some cases (when the input is determined to be too sma to be beneficiay mutithreaded) #pragma omp num_threads (expression) Contro the number of threads used for this parae region Programming Mode Loop Scheduing schedue cause determines how oop iterations are divided among the thread team static([chunk]) divides iterations staticay between threads - Each thread receives [chunk] iterations, rounding as necessary to account for a iterations - Defaut [chunk] is cei( # iterations / # threads ) dynamic([chunk]) aocates [chunk] iterations per thread, aocating an additiona [chunk] iterations when a thread finishes - Forms a ogica work queue, consisting of a oop iterations - Defaut [chunk] is 1 guided([chunk]) aocates dynamicay, but [chunk] is exponentiay reduced with each aocation 14

15 Programming Mode Loop Scheduing Exampe The function TestForPrime (usuay) takes itte time But can take ong, if the number is a prime indeed Soution: use dynamic, but with chunks 15

16 Work sharing: Sections Sections The SECTIONS directive is a non-iterative work-sharing construct. It specifies that the encosed section(s) of code are to be divided among the threads in the team. Independent SECTION directives are nested within a SECTIONS directive. Each SECTION is executed once by a thread in the team. Different sections may be executed by different threads. It is possibe that for a thread to execute more than one section if it is quick enough and the impementation permits such. 16

17 Exampe #incude <omp.h> #define N 1000 main () int i; foat a[n], b[n], c[n], d[n]; /* Some initiaizations */ for (i=0; i < N; i++) a[i] = i * 1.5; b[i] = i ; Exampe #pragma omp parae shared(a,b,c,d) private(i) #pragma omp sections #pragma omp section for (i=0; i < N; i++) c[i] = a[i] + b[i]; #pragma omp section for (i=0; i < N; i++) d[i] = a[i] * b[i]; /* end of sections */ /* end of parae section */ 17

18 Data Sharing Shared Memory programming mode Most variabes are shared by defaut We can define a variabe as private // Do this part in parae #pragma omp parae private(nthreads, tid) printf( "Heo, Word!\n" ); if (tid == 0). Programming Mode Data Sharing Parae programs often empoy two types of data Shared data, visibe to a threads, simiary named Private data, visibe to a singe thread (often stack-aocated) PThreads: Goba-scoped variabes are shared Stack-aocated variabes are private int bigdata[1024]; void* foo(void* bar) int tid; #pragma omp parae \ shared ( bigdata ) \ private ( tid ) OpenMP: shared variabes are shared private variabes are private /* Cac. here */ 18

19 Programming Mode Data Sharing private: A copy of the variabe is created for each thread. No connection between the origina variabe and the private copies Can achieve the same using variabes inside Int i; #pragma omp parae for private(i) for (i=0; i<n; i++)... Programming Mode Data Sharing Firstprivate: Same, but the initia vaue is copied from the main copy Lastprivate: Same, but the ast vaue is copied to the main copy 19

20 Thread private Simiar to private, but defined per variabe Decaration immediatey after variabe definition. Must be visibe in a transation units. Persistent between parae sections Can be initiaized from the master's copy with #pragma omp copyin More efficient than private, but a goba variabe! Synchronization What shoud the resut be (assuming 2 threads)? X=0; #pragma omp parae X = X+1; 20

21 Synchronization 2 is the expected answer But can be 1 with unfortunate intereaving OpenMP assumes that the programmer knows what he is doing Regions of code that are marked to run in parae are independent If access coisions are possibe, it is the programmer's responsibiity to insert protection Synchronization Many of the existing mechanisms for shared programming OpenMP Synchronization Nowait (turn synchronization off!) Singe/Master execution Critica sections, Atomic updates Ordered Barriers Fush (memory subsystem synchronization) Reduction (specia case) 21

22 Singe/Master #pragma omp singe Ony one of the threads wi execute the foowing bock of code The rest wi wait for it to compete Good for non-thread-safe regions of code (such as I/O) Must be used in a parae region Appicabe to parae for sections Singe/Master #pragma omp master The foowing bock wi be executed by the master thread No synchronization invoved Appicabe ony to parae sections #pragma omp parae do_preprocessing () ; #pragma omp singe read_input () ; #pragma omp master notify_input_consumed () ; do_processing () ; 22

23 Critica Sections #pragma omp critica [name] Standard critica section functionaity Critica sections are goba in the program Can be used to protect a singe resource in different functions Critica sections are identified by the name A the unnamed critica sections are mutuay excusive throughout the program A the critica sections having the same name are mutuay excusive between themseves Critica Sections int x=0; #pragma omp parae shared(x) #pragma omp critica x++; 23

24 Ordered #pragma omp ordered statement Executes the statement in the sequentia order of iterations Exampe: #pragma omp parae for ordered for (j=0; j<n; j++) int resut = j*j; #pragma omp ordered printf ("computation(%d) = %d\n",j, resut ) ; Barrier synchronization #pragma omp barrier Performs a barrier synchronization between a the threads in a team at the given point. Exampe: #pragma omp parae int resut = heavy_computation_part1 () ; #pragma omp atomic sum += resut ; #pragma omp barrier heavy_computation_part2 (sum) ; 24

25 Expicit Locking Can be used to pass ock variabes around (unike critica sections!) Can be used to impement more invoved synchronization constructs Functions: omp_init_ock(), omp_destroy_ock(), omp_set_ock(), omp_unset_ock(), omp_test_ock() The usua semantics Use #pragma omp fush to synchronize memory Consistency Vioation? 25

26 Consistency Vioation? Reduction How to paraeize this code? for (j=0; j<n; j++) sum = sum+a[j] b[j]; sum is not private, but accessing it atomicay is too expensive Have a private copy of sum in each thread, then add them up Use the reduction cause! #pragma omp parae for reduction(+: sum) An operator must be used: +, -, *... 26

27 Synchronization Overhead Lost time waiting for ocks Prefer to use structures that are as ock-free as possibe! Summary OpenMP is a compier-based technique to create concurrent code from (mosty) seria code OpenMP can enabe (easy) paraeization of oopbased code Lightweight syntactic anguage extensions OpenMP performs comparaby to manuay-coded threading Scaabe Portabe Not a siver buet for a appications 27

28 More Information OpenMP officia site A handy OpenMP tutoria Another OpenMP tutoria and reference Backup Sides Syntax, etc 28

29 OpenMP Syntax Genera syntax for OpenMP directives #pragma omp directive [cause ] CR Directive specifies type of OpenMP operation Paraeization Synchronization Etc. Causes (optiona) modify semantics of Directive OpenMP Syntax PARALLEL syntax #pragma omp parae [cause ] CR structured_bock Ex: Output: Heo! (T=4) #pragma omp parae printf( Heo!\n ); // impicit barrier Heo! Heo! Heo! 29

30 OpenMP Syntax DO/for Syntax (DO-Fortran, for-c) #pragma omp for [cause ] CR for_oop Ex: #pragma omp parae #pragma omp for private(i) shared(x) \ schedue(static,x/n) for(i=0;i<x;i++) printf( Heo!\n ); // impicit barrier Note: Must reside inside a parae section OpenMP Syntax More on Causes private() A variabe in private ist is private to each thread shared() Variabes in shared ist are visibe to a threads Impies no synchronization, or even consistency! schedue() Determines how iterations wi be divided among threads schedue(static, C) Each thread wi be given C iterations - Usuay T*C = Number of tota iterations schedue(dynamic) Each thread wi be given additiona iterations as-needed - Often ess efficient than considered static aocation nowait Removes CS Architecture impicit Seminarbarrier from end of bock 30

31 OpenMP Syntax PARALLEL FOR (combines parae and for) #pragma omp parae for [cause ] CR for_oop Ex: #pragma omp parae for shared(x)\ private(i) \ schedue(dynamic) for(i=0;i<x;i++) printf( Heo!\n ); Exampe: AddMatrix // omp- Fies: (Makefie) addmatrix.c paraeized matrixmain.c printmatrix.c // non-omp // non-omp 31

32 OpenMP Syntax ATOMIC syntax #pragma omp atomic CR simpe_statement Ex: #pragma omp parae shared(x) #pragma omp atomic x++; // impicit barrier CRITICAL syntax OpenMP Syntax structured_bock Ex: #pragma omp parae shared(x) #pragma omp critica #pragma omp critica CR // ony one thread in here // impicit barrier 32

33 ATOMIC vs. CRITICAL OpenMP Syntax Use ATOMIC for simpe statements Can have ower overhead than CRITICAL if HW atomics are everaged (impementation dep.) Use CRITICAL for arger expressions May invove an unseen impicit ock OpenMP Syntax MASTER ony Thread 0 executes a bock #pragma omp master CR structured_bock SINGLE ony one thread executes a bock #pragma omp singe CR structured_bock No impied synchronization 33

34 OpenMP Syntax BARRIER #pragma omp barrier CR Locks Locks are provided through omp.h ibrary cas omp_init_ock() omp_destroy_ock() omp_test_ock() omp_set_ock() omp_unset_ock() OpenMP Syntax FLUSH #pragma omp fush CR Guarantees that threads views of memory is consistent Why? Reca OpenMP directives Code is generated by directives at compie-time - Variabes are not aways decared as voatie - Using variabes from registers instead of memory can seem ike a consistency vioation Synch. Often has an impicit fush - ATOMIC, CRITICAL 34

35 OpenMP Syntax Functions omp_set_num_threads() omp_get_num_threads() omp_get_max_threads() omp_get_num_procs() omp_get_thread_num() omp_set_dynamic() omp_[init destroy test set unset]_ock() Function for the environment omp_set_dynamic(int) omp_set_num_threads(int) omp_get_num_threads() omp_get_num_procs() omp_get_thread_num() omp_set_nested(int) omp_in_parae() omp_get_wtime() 35