EE/CSCI 451: Parallel and Distributed Computation

Transcription

1 EE/CSCI 451: Parallel and Distributed Computation Lecture #7 2/5/2017 Xuehai Qian University of Southern California 1

2 Outline From last class Shared memory programing model Scalability of a parallel solution Today A simple model of shared memory parallel computation Example shared memory programs OpenMP OpenMP programming model OpenMP directives Examples 2

3 Announcements HW #1 due on Wednesday HW #2 out on Wednesday, due next Wednesday (Feb, 14 th ) Programming HW #2 due today 3

4 A Simple Model of Shared Address Space Parallel Machine (PRAM) (1) 1 unit of time = Local access shared memory access Synchronous model Parallel time = total number of cycles Ps programming model? Asynchronous shared memory?? Shared memory 0 p 1 p processors 4

5 Random Access Machine (RAM) Random Access Access to any memory location (Direct access) Time Access to memory 1 unit of time Arithmetic/Logic operation 1 unit of time Serial time complexity T s (n) Example: Merge Sort PRAM (2) T s (n) = O(n log n) 1 unit of time Processor Memory 5 2/5/2018 5

6 PRAM (3) Parallel Random Access Machine (PRAM) Shared memory Memory p processors P 0 P 1 P p 1 Random Access Machines, executing in parallel using a common clock 6 2/5/2018 6

7 PRAM (4) Parallel Random Access Machine (PRAM) Use shared memory for communication between processors Synchronous Model (Global clock) Time Access to any memory location 1 unit of time Execute one instruction in each processor (arithmetic/logic operation) 1 unit of time 7 2/5/2018 7

8 PRAM (5) PRAM is a synchronous model clock 0 Shared memory p 1 Each processor is a RAM Local program acting on data in shared memory 1 unit of time For all i, i-th instruction in the execution sequence is executed in the i-th cycle by all the processors 8 Synchronous execution

9 PRAM (6) PRAM features p = # of Processors In each cycle, p locations can be accessed (in parallel) All communication overheads ignored Ideal shared memory In one cycle in each processor Read (from shared memory) Compute (using data in the processor) Write (into shared memory) 1 cycle The code executed by different processors can be different Lock step execution p operations in parallel 9

10 Adding on PRAM (1) Simple shared memory algorithm for adding n numbers Output = σ n 1 i=0 A i in A(0) A(0) A(n 1) 10

11 For n = 8 i = iteration # Adding on PRAM (2) Key Idea end of i = 0 i = 1 i = Active processors = Processor index = {0, 2, 4, 6} {0, 4} {0} {0, 1, 2, 3} 2 {0, 1} 4 {0} 8 11

12 Adding on PRAM (3) Example: n = 8 Active processors Processor # (j) iteration # (i) ~~~~~~~~~ ~~~~~~~~~ (time) Total # of additions performed = n 2 + n Total # of additions performed = n 1 Total # of additions performed = Total # of additions performed by a serial program 12

13 Algorithm Program in processor j, 0 j n Do i = 0 to log 2 n 1 Adding on PRAM (4) If j = k 2 i+1, for some k N then A j A j + A(j + 2 i ) A(0) 3. end Note: A is shared among all the processors Synchronous operation [For ex. all the processors execute instruction 2 during the same cycle, log 2 n time] N = set of natural numbers = {0, 1, } Parallel time = O(log n) A(n 1) 13

14 Adding on PRAM (5) ith iteration: data which are 2 i+1 distance apart added n 2i+1 partial results produced A(0) 2 i+1 A(n 1) 14

15 Addition on PRAM, p < n processors 1. Add within block of size n p 2. Apply Algorithm 1 n/p T p = Parallel Time = O( n + log p) p T s = Serial Time = O(n) n p 15 15

16 Synchronous and Asynchronous Clock 0 y Synchronous (e.g. PRAM) Do i = 0 to 499 y y + A(i) End y x + y OUTPUT y STOP x 0 Do i = 500 to 999 x x + A(i) End STOP Asynchronous (e.g. Ps) y 0 Do i = 0 to 499 y y + A(i) End Barrier y x + y x 0 Do i = 500 to 999 x x + A(i) End Barrier

17 Addition: Ps model? Instruction execution NOT synchronized Thread j Do i = 0 to log 2 n 1 end If j = k 2 i+1, for same k N then A j A j + A(j + 2 i ) BARRIER Correct output? How to measure time? Number of active s in iteration i = 2n 2 i+1 Complete each iteration before proceeding to next iteration Within each iteration s may execute asynchronously 17

18 Example: n = 8 Total # of s = n Addition using Ps Active s (s doing arithmetic operations) Thread # (j) iteration # (i) ~~~~~~~~~ Barrier ~~~~~~~~~ (time) 18

19 Addition using Ps, p < n s (1) Thread j Each is allocated n Τp data values 0 j < p Compute B j sum of values allocated to it BARRIER i is a local variable to each Do i = 0 to log 2 p 1 If j = k 2 i+1, for some k N then B j B j + B j + 2 i BARRIER end A n/p B(0) B(p 1) 19

20 Addition using Ps, p < n s (2) There are p s All s execute in parallel Correct output is produced for all inputs independent of execution speed Note: Threads can execute asynchronously Variable access latency to shared variables okay 20

21 OpenMP (Open Multi-Processing) Application programming interface (API) for shared memory parallel programming A portable (and scalable) programming model (can be used with C, C++, Fortran and various parallel platforms) Outcome of standardization efforts Consists of compiler directives, library routines, and environment variables Higher level of abstraction than Ps 21

22 Two Uses of OpenMP Problem (e.g., Sort) Parallel Program Directly using OpenMP Serial Program Compile Runtime System system Parallelization 1. Profile serial code 2. Identify parallelization opportunities 3. Insert parallelization directives Target Platform 22

23 OpenMP vs. Ps OpenMP Application Developer OpenMP program Compiler Run time system Operating system Ps Programmer Hardware platform 23

24 OpenMP Programming Model (1) Shared memory, s based parallelism Explicit parallelism Explicit (not automatic) programming model, offering the programmer full control over parallelization Parallelization can be taking a serial program and inserting compiler directives Underlying hardware may or may not provide hardware support for shared memory 24

25 OpenMP Programming Model (2) Directive based parallel programming 1 #pragma omp directive [clause list] 2 /* structured block */ Structured block = a section of code which is grouped together OpenMP directives are not part of a programming language Can be included in various programming languages (e.g., C, C++, Fortran) 25

26 OpenMP Programming Model (3) Fork - Join model: Fork: The master creates a team of parallel s Join: When the team s complete the statements in the parallel region, they synchronize and terminate, leaving only the master Fork Join Fork Join 26

27 OpenMP Directives (1) OpenMP executes serially until parallel directive parallel region construct Same code will be executed by multiple s Fundamental OpenMP parallel construct Example #pragma omp parallel [clause list] { task(); } 27 Master Team Team Thread task() task() task() Clause list specifies parameters of parallel execution # of s, private variables, shared variables Implied Barrier

28 OpenMP Directives (2) Work-Sharing Constructs Divide the execution of the enclosed code region among the members Implied barrier at the end of a work sharing construct Types of Work-Sharing Constructs: for - shares iterations of a loop across the team. Represents a type of "data parallelism". sections - breaks work into separate, discrete sections. Each section is executed by a. Represents a type of "functional parallelism". 28

29 OpenMP Directives (3) Loop parallelism Most programs have loops Rich source for parallelization/optimization 90/10 rule Many scheduling strategies and compile time and run time optimizations 29

30 OpenMP Directives (4) for directive example: N N matrix multiplication, C = A B Serial for (i=0; i<n; i++) { } for (j=0; j<n; j++) { } c(i,j) = 0; for (k=0; k<n; k++) c(i, j) += a(i, k) b(k, j); Static scheduling of loops #pragma omp parallel num_s (4) #pragma omp for schedule(static) for (i=0; i<n; i++) { for (j=0; j<n; j++) { c(i,j) = 0; for (k=0; k<n; k++) c(i, j) += a(i, k) b(k, j); } } Static scheduling splits the iteration space into equal chunks of size (specified in the clause list) and assigns them to s in a round-robin fashion 30

31 OpenMP Directives (5) Iteration Space 3 level nested loop for matrix multiplication dim = 128 A for (i = 0; i < dim; i++) { for (j = 0; j < dim; j++) { c(i,j) = 0; for (k = 0; k < dim; k++) c(i, j) += a(i, k) * b(k, j); } } 128 B C 31

32 OpenMP Directives (6) Iteration Space 3 level nested loop for matrix multiplication dim = 128, 4 s #prama omp parallel num_s (4) #pragma omp for schedule(static) for (i = 0; i < dim; i++) { for (j = 0; j < dim; j++) { c(i,j) = 0; for (k = 0; k < dim; k++) c(i, j) += a(i, k) * b(k, j); } } Note: Iteration space can be partitioned in many ways compiler optimizations 32

33 OpenMP Directives (7) Scheduling iteration space Iterations map s Static: partition into equal sized chunks, map at compile time Dynamic: partition into equal sized chunks, run time system assigns them to s (for ex. as the s become idle) Guided: Vary the chunk size as the (loop) computation proceeds to ensure load balance among the s Runtime: scheduling decisions left to the run time system 33

34 OpenMP Directives (8) for directive example (Data/Loop Parallelism): N N matrix multiplication: C = A B #pragma omp parallel num_s(4) { #pragma omp for for (i=0; i< N; i++) Calculate_block() } C Master Team Team Thread Team Thread i (0, N 4 1) i (N 4, N 2 1) i ( N 2, 3N 4 1) i ( 3N 4, N 1) 34

35 OpenMP Directives (9) sections example (Task Parallelism): N N matrix multiplication: C = A B #pragma omp parallel num_s(4) { #pragma omp section Caculate_block0; #pragma omp section Caculate_block1; #pragma omp section Caculate_block2; #pragma omp section Caculate_block3; } Master 0 2 Team C 1 3 Team Team 35

36 Example (1) 5-field classification Router Packet Routing table Source IP Destination IP Source Port Destination Port Protocol Action / / TCP Act / / TCP Act / / UDP Act 2 36

37 #pragma omp parallel num_s (5) { } #pragma omp section search_sip(); #pragma omp section search_dip(); #pragma omp section search_sp(); #pragma omp section search_dp(); #pragma omp section search_prot(); #pragma omp barrier #pragma omp single Merge(); Example (2) Example: 5-field packet classification Master SIP Merge() 37 Team DIP Team SP Team DP Team Prot

38 #pragma omp parallel num_s (2) { #pragma omp section } Sort(0, N 2 1); #pragma omp section Sort( N, N 1); 2 #pragma omp barrier #pragma omp single Merge(0, N 1); More Examples (1) Sort an N-element Array 0 Master Sort(0, N 2 1); N 1 N 2 2 Team Sort( N, N 1); 2 N 1 Merge(0, N 1); 38

39 More Examples (2) Find min. element in an N-element Array Serial Program min = A[0]; For i = 1 to N 1 if ( min > A[i] ) min = A[i]; End OpenMP Program #pragma omp parallel num_s(2) { #pragma omp section } min1 = Min (0, N 2 1); #pragma omp section min2 = Min ( N, N 1); 2 #pragma omp barrier #pragma omp single min = Min (min1, min2); 39

40 Summary OpenMP Simple high level abstraction for shared address space programming Directives to the compiler Specify: # of s, conditions for parallelization, local/global variables Serial code incremental parallelization Loop parallelization, task parallelization Static analysis parallel code that executes with the aid of a run time system Run time system: data management, layout, communication, mapping, optimizations, 40

41 Backup Slides 41

42 OpenMP Programming Model Fork/Join can be nested: Nesting complication handled automatically at compile-time Independent of the number of s actually running Fork Fork Join Join 42

43 OpenMP Programming Model Master Thread Thread with ID=0 Only that exists in sequential regions Depending on implementation, may have special purpose inside parallel regions Some special directives affect only the master 0 Fork Join 0 43

44 OpenMP Programming Model (3) Execute serially until parallel directive The directive is responsible for creating a group of s The directive defines the structured block that each executes The which encounters the directive becomes the master of this group of s 44

45 OpenMP Programming Model (5) Fork - Join model example: Serial segment Serial #pragma omp parallel num_s(4) Parallelizable segment Serial segment Barrier Parallel Serial #pragma omp parallel num_s(2) Parallelizable segment Serial segment Parallel Serial 45

46 OpenMP Programming Model (6) General Structure // serial segment #pragma omp parallel [clause list ] /* structured block */ // rest of serial segment Each created by this directive executes the structured block specified by the parallel directive 46

47 for directive example: OpenMP Directives vector-add: c[0:29] = a[0:29] + b[0:29] #pragma omp parallel num_s (3) { #pragma omp for { for (i=0; i < 30; i++) c[i] = a[i] + b[i]; } } Master for (i=0; i < 10; i++) c[i] = a[i] + b[i]; #for directive Team for (i=11; i < 21; i++) c[i] = a[i] + b[i]; Team Thread for (i=21; i < 30; i++) c[i] = a[i] + b[i]; 47 Implied Barrier Note: this is an example; different partitioning is possible, depending on compiler optimizations, user inputs, etc.

48 reduction clause OpenMP Directives Specifies how multiple local copies of a variable in different s are combined into a single copy at the master when s exit Usage: reduction (operator: variable list) Operator: +, *, -, &, ^, && and Example #pragma omp parallel reduction (+: sum) num_s(8) { } /* compute local sums here */ /* sum here contains sum of all local instances of sums */ 48

49 OpenMP Directives sections example: Task parallelism vector-add: c[0:29] = a[0:29] + b[0:29] vector-sub: d[0:29] = a[0:29] b[0:29] #pragma omp parallel num_s (2) { #pragma omp section for (i=0; i < 30; i++) c[i] = a[i] + b[i]; #pragma omp section for (i=0; i < 30; i++) d[i] = a[i] b[i]; } 2 tasks Master for (i=0; i < 30; i++) c[i] = a[i] + b[i]; # Sections directive Team for (i=0; i < 30; i++) d[i] = a[i] b[i]; Implied Barrier 49

50 OpenMP Directives sections example: N N matrix multiplication: C = A B #pragma omp parallel num_s(4) { #pragma omp section Caculate_block0; #pragma omp section Caculate_block1; #pragma omp section Caculate_block2; #pragma omp section Caculate_block3; } Master 0 2 Team C 1 3 Team Team 50

51 OpenMP Directives single directive specifies a computation within a structured block to be performed by a single #pragma omp parallel num_s (2) { #pragma omp single { task(); } } Master # single directive Team task(); idle Implied Barrier 51

52 barrier directive OpenMP Directives Synchronizes all s in the team When a barrier directive is reached, a will wait at that point until all other s have reached that barrier All s resume executing in parallel the code that follows the barrier Example: #pragma omp parallel num_s(2) { task_a(); #pragma omp barrier task_b(); } Implied Barrier 52 Master task_a() task_b() Team task_a() task_b()

53 Data Handling Shared adrs. space programming Communication is implicit Local, global variables An experienced developer can optimize performance: Variable initialized and used by a private variable Shared data across s repeatedly accessed (read) copy the variable at creation Shared data accessed (modified) by multiple s explore local op. + global op. (rewrite the code) Multiple s manipulating different portions of large data partition data and make them private 53