Shared-memory Programming

Transcription

1 Shared-memory Programming Introduction to High Performance Computing Systems (CS1645) Esteban Meneses

2 Programming a Supercomputer Languages: Fortran, C/C++, Python. Nodes Cores Accelerators Supercomputer Libraries: linear algebra, scientific methods, data mining. 3-tier parallelism, hybrid architecture. 2

3 Node Parallelism One process per node. Private memory. Communication Message Passing Interface (MPI) send(m) between nodes: Message-passing. Basic operations: send, receive. Collective operations. M receive(m) 3

4 Core Parallelism One thread per core. Shared-memory. Synchronization between cores: OpenMP #pragma omp parallel for! for(i=0;i<n;i++)! x[i] += y[i] Basic operations: fork, join. Bulk-synchronous parallel (BSP). 4

5 Accelerator Parallelism Massively parallel thread architecture. Shared memory. Offload mechanism OpenACC #pragma acc kernels! for(i=0;i<n;i++)! x[i] = foo(i) to spawn kernel on accelerator. Manage memory copy operations. 5

6 Multithreading Source: David Whittaker 6

7 Shared-memory Systems Instruction Streams single multiple MISD MIMD SISD SIMD single multiple Data Streams CPU CPU CPU CPU Interconnect Memory There is a single shared address space for all processors. All processors share the same view of memory. 7

8 Types of Shared-memory Systems Uniform Memory Access (UMA): memory is equally accessible to all processors with the same performance (bandwidth and latency). Cache-coherent Non Uniform Memory Access (ccnuma): memory is physically distributed but appears as a single address space. Performance (bandwidth and latency) is different for local and remote memory access. Copies of the same cache line may reside in different caches. Cache coherence protocols guarantees consistency all time (for UMA and ccnuma). Cache coherence protocols do not alleviate parallel programming for shared-memory architectures. Source: Prof. Dr. G. Wellein 8

9 OpenMP Open Multi-Processing. An implementation of multithreading. Standard API for shared-memory programming. OpenMP Architecture Review Board (ARB) published OpenMP standard 1.0 in Adopted by most important operating systems and supercomputer vendors. Based on pragmas, programmer annotations for the compiler. Embraces incremental parallelization philosophy. Serial counterpart is always available. 9

10 Bulk Synchronous Parallelism (BSP) Bridging model developed by Leslie Valiant in Model components: Processors. Network. Synchronization hardware. Computation proceeds by executing supersteps: Concurrent computation. Communication. Barrier synchronization. Concurrency level expands and shrinks. Source: 10

11 Two Types of Parallelism Data parallelism (SIMD): parallel execution of the same code stream on a supercomputer. Each node may process a different data portion. Task parallelism (MIMD): distribution of different code streams on a supercomputer. Each node may process a different data portion. Both are possible in OpenMP. 11

12 Example Grading a set of exams: 40 exams, 4 questions, 4 graders. Data parallelism: 10 exams per grader Task parallelism: 1 question per grader. #1 #2 #3 #

13 OpenMP Standard Defines an Application Program Interface (API) which provides a portable and scalable model for developers of shared memory parallel applications (instead of PThreads). A library with some simple functions. A few program directives (pragmas = hints to the compiler). A few environment variables. A compiler translates OpenMP functions and directives to PThread calls. Program begins with a master thread. Threads are forked when specified by directives. 13

14 Parallel Regions Defined with #pragma omp parallel: #include <omp.h>! int main(){!! print( The output:\n );!! #pragma omp parallel // multi-threaded section!! {!!! printf( Hello World\n );!! }!! // resumes serial section! } The number of spawned threads can be specified through: An environment variable: setenv OMP_NUM_THREADS 8 An API function: void omp_set_num_threads(int number);! Get number of threads: int get_num_threads(); 14

15 Private Variables The format of a pragma in C/C++is: #pragma omp name_of_directive [optional_clauses]! private is a clause that declares variables that are local to the spawned threads. A similar clause, shared, is used to declare variables that are not local (the default). #include <omp.h>! int main(){!! int id, np;!! #pragma omp parallel private(id, np)!! {!!! np = omp_get_num_threads();!!! id = omp_get_thread_num();!!! printf( Hello from thread %d, out of %d threads\n, id, np);!! }! } 15

16 Running jobs on Supercomputers Source: 16

17 Running OpenMP Programs Compiler flag: icpc -openmp pgm.cpp Include file: #include <omp.h> Environment variable for number of threads: OMP_NUM_THREADS Useful functions: omp_set_num_threads(int t) sets number of active threads. omp_get_wtime() gets current wall clock time in seconds. omp_in_parallel() returns true if program is running in parallel; false otherwise omp_get_num_threads() returns the number of active threads. omp_get_thread_num() returns the thread ID. 17

18 Number of Threads The num_threads clause can be added to a parallel directive: # pragma omp parallel num_threads (thread_count) It allows the programmer to specify the number of threads that should execute the following block. There may be system-defined limitations on the number of threads that a program can start. The OpenMP standard does not guarantee that this will actually start thread_count threads. Most current systems can start hundreds or even thousands of threads. Unless we are trying to start a lot of threads, we will almost always get the desired number of threads. 18

19 Critical Sections and Barriers A critical section is executed by one thread at a time; (name) is optional. Critical sections with different names can be executed simultaneously. #pragma omp parallel shared(sum) {!!! value = foo();!! #pragma omp critical(name)! {!! sum = sum + value;! }! } All the threads in the active team must reach the barrier point before any can continue: # pragma omp barrier 19

20 Example Using the trapezoidal rule to compute the integral of a function. h = (b-a)/n! approx = (f(a)+f(b))/2;! for(i=1; i<=n-1; i++){!! xi = a + i*h;!! approx += f(xi);! }! approx = h*approx; Source: 20

21 Example (cont.) We identify two types of tasks: Computation of the areas of individual trapezoids. Adding the areas of trapezoids. There is no communication among the tasks in the first collection, but each task in the first collection communicates with task in the second. We assume that there would be many more trapezoids than cores. So we aggregate tasks by assigning a contiguous block of trapezoids to each thread. Source: An Introduction to Parallel Programming by Peter Pacheco 21

22 Example (cont.) double f(double x); /* Function we're integrating */! void Trap(double a, double b, int n, double* global_result_p);!! int main(int argc, char* argv[]) {! double global_result = 0.0; /* Store result in global_result */! double a, b; /* Left and right endpoints */! int n; /* Total number of trapezoids */! int thread_count;!!!! if (argc!= 2) Usage(argv[0]);! thread_count = strtol(argv[1], NULL, 10);! printf("enter a, b, and n\n");! scanf("%lf %lf %d", &a, &b, &n);! if (n % thread_count!= 0) Usage(argv[0]);! #pragma omp parallel num_threads(thread_count)! Trap(a, b, n, &global_result);! printf("with n = %d trapezoids, our estimate\n", n);! printf("of the integral from %f to %f = %.14e\n",! a, b, global_result);! return 0;! } /* main */ Source: An Introduction to Parallel Programming by Peter Pacheco 22

23 Example (cont.) void Trap(double a, double b, int n, double* global_result_p) {! double h, x, my_result;! double local_a, local_b;! int i, local_n;! int my_rank = omp_get_thread_num();! int thread_count = omp_get_num_threads();!! h = (b-a)/n;! local_n = n/thread_count;! local_a = a + my_rank*local_n*h;! local_b = local_a + local_n*h;! my_result = (f(local_a) + f(local_b))/2.0;! for (i = 1; i <= local_n-1; i++) {! x = local_a + i*h;! my_result += f(x);! }! my_result = my_result*h;!! #pragma omp critical! *global_result_p += my_result;! Source: An Introduction to } /* Trap */ Parallel Programming by Peter Pacheco 23

24 Exercise Implement in OpenMP an efficient parallel version of matrix-matrix multiplication. for (i=0; i<n; i++)! for (j=0; j<n; j++)! for (k=0; k<n; k++)! C[i][j] += A[i][k]*B[k][j];! 24

25 Solution for (i=0; i<n; i++)! for (j=0; j<n; j++)! for (k=0; k<n; k++)! C[i][j] += A[i][k]*B[k][j];! #pragma omp parallel private(np,id)! {! np = omp_get_num_threads();! id = omp_get_thread_num();! for (i=id*n/np; i<(id+1)*n/np; i++)! for (j=0; j<n; j++)! for (k=0; k<n; k++)! C[i][j] += A[i][k]*B[k][j];! } 25

26 Embarrassingly Parallel A problem that has a trivial parallel solution: Simple distribution of work is enough to parallelize. No significant communication complexity. No significant synchronization complexity. Should we feel embarrassed to come up with an easy parallel solution? Of course not. Embarrassingly parallel Delightfully parallel. 26

27 For-loop pragma Automatically distributes iterations in the for-loop to active threads. Syntax: #pragma omp for! for(i=0; i<n; i++){!...! } Usually used inside a parallel region. 27

28 Example: Adding arrays a and b #define N 1024! #define N_THREADS 16! main(){!! int chunk, a[n], b[n], c[n];!! omp_set_num_threads(n_threads);!! chunk = N / N_THREADS;!! #pragma omp parallel shared(a,b,c) private(i, id)!! {!!! id = omp_get_thread_num();!!! for (i=id*chunk; i<(id+1)*chunk; i++)!!!! c[i] = a[i] + b[i];!! }! } #pragma omp parallel shared(a,b,c) private(i, id)! {!! #pragma omp for!! for (i=0; i<n; i++)!!! c[i] = a[i] + b[i];! } 28

29 Combining parallel and for pragmas Programmer needs to consider data dependencies between iterations. #pragma omp parallel shared(a,b,c) private(i)! {! #pragma omp for! for(i=0; i<n; i++)! c[i] = a[i] + b[i];! } #pragma omp parallel for shared(a,b,c) private(i)! for(i=0; i<n; i++)! c[i] = a[i] + b[i];! 29

30 Data Dependencies Example: fibo[0] = fibo[1] = 1; for (i=2; i<10; i++)! fibo[i] = fibo[i 1] + fibo[i 2]; fibo[0] = fibo[1] = 1; # pragma omp parallel for num_threads(2)! for (i=2; i<10; i++)! fibo[i] = fibo[i 1] + fibo[i 2]; Source: An Introduction to Parallel Programming by Peter Pacheco 30

31 Format of parallel for loops Initialization Condition Iteration index = start index < end! index <= end! index >= end! index > end index++ ++index index--! index += inc! index -= inc! index = index + incr! index = incr + index! index = index - incr Variable index has to be either integer or pointer type. Expressions for start, end, and inc have to be compatible with index and should not be changed during the loop. Variable index is made private by default. 31

32 Odd-Even Sort Parallel equivalent to Bubble Sort. If n processors sort n numbers, the worst time is O(n/2). Alternates between n/2 odd and even phases. Each processor holds one number. Odd phase: odd-numbered processors compare and swap their number with neighbor. Even phase: even-numbered processors compare and swap their number with neighbor

33 Odd-Even Sort Program void OddEvenSort(int A[], int n){!! for(int i = 0; i < n; ++i){!!! if(i & 1){!!!! for(int j = 2; j < n; j+=2)!!!!! if (a[j] < a[j-1])!!!!!! swap(a[j-1], a[j]);!!! } else {!!!! for(int j = 1; j < n; j+=2)!!!!! if (a[j] < a[j-1])!!!!!! swap(a[j-1], a[j]);!!! }!! }! } 33

34 Exercise Transform the following loops into valid OpenMP parallel for loops. for(i=0; i<n; i++){!! x = a[i] + b[i];!! for(j=i; j<n; j++){!!! k++;!!! c[i][j] = x + d[k][j];!! }! } for(i=0; i<n; i++){!! x = a[i] + b[i];!! #pragma omp parallel for!! for(j=i; j<n; j++){!!! c[i][j] = x + d[k-i+j+1][j];!! }!! k += n-i;! } 34

35 Exercise (cont.) for(i=0; i<n; i++){!! a[i] = b[i] + c[i][j];!! t = a[i] + 2;!! b[i] = t * 2;!! c[i][j] = t * 3;!! s += t;! } #pragma omp parallel for private(t)! for(i=0; i<n; i++){!! a[i] = b[i] + c[i][j];!! t = a[i] + 2;!! b[i] = t * 2;!! c[i][j] = t * 3;!! #pragma omp critical!! s += t;! } 35

36 Exercise (cont.) for(i=0; i<n; i++){!! for(j=0; j<m; j++){!!! a[j] = a[j] + b[k];!!! k++;!! }!! k++;!! for(j=0; j<m; j++)!!! c[i][j] = a[j] + 1;! } for(i=0; i<n; i++){!! #pragma omp parallel for!! for(j=0; j<m; j++){!!! a[j] = a[j] + b[k+j];!! }!! k += m + 1;!! #pragma omp parallel for!! for(j=0; j<m; j++)!!! c[i][j] = a[j] + 1;! } #pragma omp parallel for! for(i=0; i<n; i++){!! for(j=0; j<m; j++){!!! a[j] = a[j] + b[k+n*(m+1)+j];!! }!! for(j=0; j<m; j++)!!! c[i][j] = a[j] + 1;! }! k += n*(m + 1); 36

37 Scheduling iterations of for loops Used to specify distribution of iterations to threads. Syntax: #pragma omp parallel for schedule(type,chunksize) The type can be: Static: iterations are distributed among threads before the loop is executed. Dynamic or guided: iterations are assigned to threads during the execution of the loop. Auto: the compiler or runtime system determines the proper schedule. Runtime: the environment variable OMP_SCHEDULE is used to determine schedule at runtime. chunksize is a positive integer. 37

38 Static scheduling The iteration space is divided into chunks and distributed to threads in a round-robin fashion. T1 T2 T3 T4 T1 T2 T3 T4 Example, 24 iterations: Chunksize T1 T2 T3 T4 1 0,4,8,12,16,20 1,5,9,13,17,21 2,6,10,14,18,22 3,7,11,15,19,23 2 0,1,8,9,16,17 2,3,9,10,18,19 4,5,11,12,20,21 6,7,13,14,22,23 3 0,1,2,12,13,14 3,4,5,15,16,17 6,7,8,18,19,20 9,10,11,21,22,23 4 0,1,2,3,16,17,18,19 4,5,6,7,20,21,22,23 8,9,10,11 12,13,14,15 38

39 Dynamic Scheduling Iterations are divided into chunks and assigned to threads dynamically. Each thread executes a chunk, and when done, requests another chunk from the runtime system. Although each chunk contains the same number of iterations, chunks may have different execution times. Example: sum = 0.0;! for(i=0; i<=n; i++)! sum += f(i) double f(int i){!! int j, start = i*(i+1), finish = start+i;!! double return_val = 0.0;!! for(j=start; j<=finish; j++){!!! return_val += sin(j);!! }!! return return_val;! } 39

40 Guided Scheduling T1 T2 T3 T4 T4 T3 T3 T2 T1 Similar to dynamic scheduling, except that chunk size decreases as the execution progresses. At the dynamic scheduling decision, the chunk size is equal to 1/p of the remaining iterations, where p is the number of threads. Can specify the smallest chunksize (except possibly the last). The default smallest chunksize is 1. 40

41 Exercise What would be the assignment of iterations to threads for a guided schedule in a parallel for loop with 1000 iterations and 2 threads. The sequence in which threads 0 and 1 request more work is given by: 0,1,1,1,0,1,0,1,1,1,1. Chunk Thread [1-500] 0 [ ] 1 [ ] 1 [ ] 1 [ ] 0 [ ] 1 [ ] 0 [ ] 1 [ ]

42 Data Scoping Types of variables: Shared. Private. Firstprivate: private but initialized to its value before entering the region. Lastprivate: private but on exit, value in master is last value in thread. Default: default type for all variables in the thread. Reduction: variables declared in enclosing context, are private in the parallel sections, but reduced upon exit using the specified operator (e.g. +, *, -, &,, ˆ, &&, ). 42

43 Example Internal product of two vectors a and b. main(){!! int i, n;!! float a[100], b[100], result;!! n = 100;!! result = 0.0;!! for(i=0; i<n; i++){!!! a[i] =... ; b[i] =... ;!! }!! #pragma omp parallel for default(shared) private(i) reduction(+:result)!! for (i=0; i < n; i++)!!! result = result + (a[i] * b[i]);!! printf("final result= %f\n",result);! } 43

44 Exercise Use a loop pragma to implement the matrix multiplication algorithm in OpenMP. for (i=0; i<n; i++)! for (j=0; j<n; j++)! for (k=0; k<n; k++)! C[i][j] += A[i][k]*B[k][j];! #pragma omp parallel for! for (i=0; i<n; i++)! for (j=0; j<n; j++)! for (k=0; k<n; k++)! C[i][j] += A[i][k]*B[k][j];! for (i=0; i<n; i++)! #pragma omp parallel for! for (j=0; j<n; j++)! for (k=0; k<n; k++)! C[i][j] += A[i][k]*B[k][j];! 44

45 Exercise Implement in OpenMP the parallel sum algorithm. shared float v[n];! int value;!! for(i=1; i<=log(n); i++)! if(id mod 2 i == 2 i -1)! value = v[id] + v[id-2 i-1 ] barrier! v[id] = value! barrier 45

46 Exercise (cont.) #pragma omp parallel private(i, nthreads, tid, value, stages)! {!!!! // sum local entries into value!! // getting thread id and number of threads! tid = omp_get_thread_num();! nthreads = omp_get_num_threads();!! stages = (int) log2(nthreads);! for(i=1; i<=stages; i++){! if(tid % (int)pow(2,i) == (int)pow(2,i)-1){! value = vector[tid] + vector[tid-(int)pow(2,i-1)];! }! #pragma omp barrier! vector[tid] = value;! #pragma omp barrier! }! } 46

47 Exercise (cont.) #pragma omp parallel for reduction(+:sum)! for(int i=0; i<n; i++){!! sum += vector[i];! } 47

48 Sections Define independent pieces of code. May combine the parallel and sections pragmas (as with for). If there are more threads than sections, then some are idle. If there are fewer threads than sections, then some sections are serialized. #define N 1000! main(){!! int i;!! float a[n], b[n], c[n];!! for (i=0; i < N; i++)!!! a[i] = b[i] =... ;!! #pragma omp parallel shared(a,b,c) private(i)!! {...!!! #pragma omp sections!!! {!!!! #pragma omp section!!!! {!!!!! for (i=0; i < N/2; i++)!!!!!! c[i] = a[i] + b[i];!!!! }!!!! #pragma omp section!!!! {!!!!! for (i=n/2; i < N; i++)!!!!!! c[i] = a[i] + b[i];!!!! }!!! } /* end of sections */!!...!! } /* end of parallel section */! } 48

49 Single and Master The single directive serializes a section of code within a parallel region. More convenient and efficient than terminating a parallel region and starting it later. Typically used to serialize a small section of the code that s not thread safe. Threads in the team that do not execute the single directive, wait at the end of the enclosed code block, unless a nowait clause is specified. The master directive is the same as the single directive except that: The serial code is executed by the master thread. The other threads skip the master section, but do not wait for the master thread to finish executing it. 49

50 Atomic A critical section composed of one statement (expression) of the form: x++, x--, ++x, x, or x <op> = <expression>; <op> can be +,*,-,/,&,^,,<<, or >>! Compiler will use hardware atomic instructions. Is more efficient than using the critical section directive. Example: #pragma omp parallel for shared(sum)! for(i = 0; i < n; i++){!! value = f(a[i]);!! #pragma omp atomic!! sum = sum + value;! } 50

51 Task Parallelism Pragma task spawns a task (non-blocking). Pragma taskwait wait for all spawned tasks. Pragmas for task parallelism must be used inside a parallel section. int fibo(int n){! int fn1, fn2;! if (n<2)! return n;! else {! #pragma omp task! fn1 = fibo(n-1);! #pragma omp task! fn2 = fibo(n-2);! #pragma omp taskwait! return fn1 + fn2;! }! }!! int main() {! #pragma omp parallel shared(n)! {! #pragma omp single!! fibo(n);! }! } 51

52 Summary OpenMP is the de facto standard for sharedmemory programming in HPC. It is a wrapper for multithreading libraries (such as PThreads). OpenMP combines directives to the compiler and parallelization by the programmer. OpenMP embraces incremental parallelization: The sequential version of an OpenMP program is always available. 52

53 Acknowledgements Lecture notes by Rami Melhem. An Introduction to Parallel Programming by Peter S. Pacheco. Morgan-Kaufmann,