Chap. 6 Part 3. CIS*3090 Fall Fall 2016 CIS*3090 Parallel Programming 1

Transcription

1 Chap. 6 Part 3 CIS*3090 Fall 2016 Fall 2016 CIS*3090 Parallel Programming 1

2 OpenMP popular for decade Compiler-based technique Start with plain old C, C++, or Fortran Insert #pragmas into source file You guide compiler in parallelizing loops and spawning threads under the hood Implemented by Gnu and commercial compilers Threads will be pthreads or Windows threads according to OS that spawn onto cores Compiler options en/disable OpenMP #pragmas Rule about #pragmas: if compiler doesn t recognize, must disregard without error Fall 2016 CIS*3090 Parallel Programming 2

3 What OpenMP is NOT It s NOT the compiler finding the parallelism for you! That s largely a vain hope You re telling compiler what to parallelize Easy to implement Peril-L style of pseudocode Fall 2016 CIS*3090 Parallel Programming 3

4 People like it because You skip the pain/agony of pthreads programming Key advantage: If #pragmas disabled, program compiles/runs serially, so you always have correct P=1 version to compare with! Highly suitable for work flow of parallelizing existing serial app Accounts for lots of situations Fall 2016 CIS*3090 Parallel Programming 4

5 Caution needed Deceptively easy to use! Often don t end up writing much code But very easy to introduce all the problems of concurrent programming thru the back door, w/o being alert shared-mem corruption, race conditions leading to erratic results, deadlocks OpenMP programmers may start out naive Fall 2016 CIS*3090 Parallel Programming 5

6 Still needs intelligence Wringing out the performance load balancing by selecting appropriate granularity of work load reducing false-sharing OpenMP evolving to be more powerful More complex constructs, with tricks and subtleties We ll only scratch the surface! Fall 2016 CIS*3090 Parallel Programming 6

7 OpenMP in a word Authors: When a little parallelism is enough Less flexibility than with pthreads programming Fall 2016 CIS*3090 Parallel Programming 7

8 Applicability For shared-memory systems In non-shared memory, can use as a lower layer of fine-grained parallelism Deploy your Pilot processes on individual computer nodes Let each multicore computer run OpenMP Most clustered computers are themselves multicore (e.g., mako 8/node, orca 24!) Easy to combine Pilot & OpenMP Fall 2016 CIS*3090 Parallel Programming 8

9 Goal is best of both worlds Coarse-grained parallelism with Pilot Highly scalable to take advantage of 100s nodes Fine-grained parallelism with OpenMP Only makes sense with shared memory Can still be exploited on each node Price is added complexity of hybrid program Fall 2016 CIS*3090 Parallel Programming 9

10 Compiler directives C/C++: #pragma omp Fortran:!$OMP, C$OMP, *$OMP 1st character denotes a comment line so acts like ignorable #pragma $OMP denotes the directive Fall 2016 CIS*3090 Parallel Programming 10

11 Basic operation easily viewed in example (Fig. 6.28) If take out #pragmas, goes back to serial counting of 3 s in array count_p var Start with ln 9 #pragma omp for This is just Peril-L s forall! for spawns threads instead of iterations pthread_create & work func are effectively generated for you by compiler pthread_join effect also generated Fall 2016 CIS*3090 Parallel Programming 11

12 Figure 6.28 schedule(static) Copyright 2009 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12

13 Optional clauses on for directive schedule(static): static scheduling Static work allocation predetermines responsibility for portions of the work Compiler will generate the code to do this based on no. of loop iterations & threads at run time private(i): make variable thread-local one copy/thread (otherwise would be shared, creating havoc) Fall 2016 CIS*3090 Parallel Programming 13

14 Not controlled in example How many threads will be spawned? Generally, follows scalable parallelism principle of #threads = #CPUs (default) Work then divides up on data-parallel basis reasonable load balancing Can do dynamic scheduling/allocation Fall 2016 CIS*3090 Parallel Programming 14

15 Controlling no. of threads 3 ways: num_threads clause omp_set_num_threads() function #include <omp.h> OMP_NUM_THREADS environment var. Set in shell before running program Overridden by omp_set_num_threads() or num_threads in program Fall 2016 CIS*3090 Parallel Programming 15

16 How much magic? Are limitations on acceptable for statements See Code Spec 6.21 for restrictions If too complicated, compiler can t figure out how to divide the work Fall 2016 CIS*3090 Parallel Programming 16

17 Parallel region (ln 5) for directive inside parallel region Allocates work to threads (more info later) Also parallel for directive As convenience, combines omp parallel and omp for Useful if the only reason you re making a parallel region is a single for loop omp for without enclosing parallel region executes serially Fall 2016 CIS*3090 Parallel Programming 17

18 Figure 6.28 schedule(static) Copyright 2009 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 18

19 Taking control over local vs. shared mem Already saw private(i) = thread-local shared( ) Notifies that those variables are accessed by multiple threads Here, array and length would be defined as global variables (static storage) private(count_p) Again, one copy/thread (but scope is wider than for s i) Fall 2016 CIS*3090 Parallel Programming 19

20 omp critical (ln 17) Designates critical section for updating global variable Compiler automagically creates mutex to protect it since you declared it shared Compiler won t know to set up critical sections if you don t tell it! Fall 2016 CIS*3090 Parallel Programming 20

21 Can see some pitfalls already Might forget to privatize a variable that shouldn t be global Threads overwriting the one copy Might forget to make critical sections to modify shared global variable Race conditions Fall 2016 CIS*3090 Parallel Programming 21

22 What really happens at omp parallel directive Good explanations: When a program reaches a parallel Creates a team of threads One is master = thread #0 The code block is effectively spun off into a pthreads function that all threads execute Can designate code that only master executes Fall 2016 CIS*3090 Parallel Programming 22

23 End of parallel code block Implied barrier at the end All threads effectively join with master Only the master thread continues execution past this point Barrier avoided by nowait clause Useful if sailing from one parallel for block into another (saves taking down the team and creating the threads again) Fall 2016 CIS*3090 Parallel Programming 23

24 Efficient reductions count=0; #pragma omp parallel for reduction(+:count) for (i=0; i<length; i++) { count += (array[i]==3)? 1 : 0; } With reduction(operator,list of vars), can avoid making critical section Why duplicate operator in reduction(+:) and the count += statement?? Remember rule that without #pragma, serial code must produce same result! Fall 2016 CIS*3090 Parallel Programming 24

25 Atomic clause Another way to avoid critical section: atomic clause More efficient if single operation being applied to shared variable Likely will use HW instruction with atomic execution semantics (non-interruptible) Much faster than un/locking mutex Fall 2016 CIS*3090 Parallel Programming 25

26 Task parallel constructs So far, only seen data parallelism implied by parallel for Can use omp sections { section{} section{} } to designate task code to run in parallel (p 199) As always, without #pragmas, the tasks would run in sequence Fall 2016 CIS*3090 Parallel Programming 26

27 Parallel sections #pragma omp sections { #pragma omp section { } Task_A(); #pragma omp section { } Task_B(); #pragma omp section { } Task_C(); } Fall 2016 CIS*3090 Parallel Programming 27

28 Next time Continue with OpenMP Will be using Intel training slides (going over some ground we touched on) Will do hands-on in lab Thu. (workbook) Fall 2016 CIS*3090 Parallel Programming 28