Parallel Programming

Transcription

1 Parallel Programming Lecture delivered by: Venkatanatha Sarma Y Assistant Professor MSRSAS-Bangalore 1

2 Session Objectives To understand the parallelization in terms of computational solutions. To understand the difference between designing an algorithm as a sequential solution and as a parallel solution To understand how to estimate Time and Space complexity for a given parallel algorithm To be aware of provisions for parallelism provided by a programming language 2

3 Introduction We have worked with sequential logic so far Today, multi core processors are common GPUs can also join in and help in computation Is it not a waste of resources if we do not use these idle computational elements? Can we make all algorithms work in parallel? What is the effect on data? 3

4 Which Algorithms Can Be Parallelized Not all algorithms can be parallelized Parallel logic requires different types of data Shared All execution blocks have access Private or Local Only one execution block has access Parallel logic also requires synchronization 4

5 Comparison of Two Algorithms Binary Search Divides problem space by half at each execution Quick Sort Requires access to complete problem space at all times Can be parallelized but complexity remains same Divides problem space at each iteration Finds location of pivot at each iteration Does not require access to all elements 5

6 Types of Data 6

7 Shared Data When executing a parallel program, there is a possibility that data can be shared among parallel Threads of execution The programmer or the language must ensure sanity of this data Such data is treated as a Critical Section Programming languages provide mechanisms to programmer such as semaphores (C) and Monitors (synchronized in Java) to deal with such sections of data 7

8 Private Data Data that is not shared Local information such as temporary variables Data available to all the threads as a copy of original data Loop invariant data that is not written to but is only read from Most variables that we use fall in to this category No need to worry about sanity of this set 8

9 Synchronization One thread has to finish before other starts its work Producer/Consumer problems All threads have to finish before continuing Parallelized blocks of logic arranged in sequence Merge Sort 9

10 Parallelization of Quick Sort 10

11 Recap Sorting of a list of numbers in ascending order Selection of Pivot Generating High List and Low List Recursively applying the algorithm on high list and low list 11

12 Example List: [4,11,14,17,22,63,65,79,89,95] Chosen Pivot: 79 High List: [89,95] Low List: [4,11,14,17,22,63,65] Execution FLow 12

13 Parallellization Method 1: Implement parallelization while sorting high list and low list Randomly choose a pivot from one of the threads and broadcast it to every thread Each process divides its unsorted list into two lists: Low List and High List based on the pivot Each process in the upper half of the process list sends its low list to a partner process in the lower half of the process list and receives a high list in return 13

14 Parallellization Now, one group of threads have high list, and the other threads have the low list. Thereafter, the threads can work on their high list and low list to split it in to high high list or high low list and low high list or low low list. After log P recursions, every thread has an unsorted list of values completely disjoint from the values held by the other processes. The largest value on process i will be smaller than the smallest value held by process i + 1. Each process can sort its list using sequential quicksort 14

15 Parallellization Execution FLow In Memory 15

16 Parallellization Issues in efficiency: Who chooses median? The data set is not distribited equally if load is not balanced Solution Do one iteration of quicksort Find local median and select the best median Data can now be evenly distributed 16

17 Parallellization Method 2: Hyper Quick Sort (Selecting median after linear Quicksort on local set) Apply quicksort in each thread Select median of medians and broadcast across threads Create low list and high list If Partner thread exists, swap with partner thread Repeat from step 1 Else Merge all lists, the complete list is sorted 17

18 Parallellization Execution FLow of Hyper Quick Sort 18

19 Parallellization Issues in efficiency: The data set can still not be distribited equally Communication overhead due to swapping and exchange of pivots Solution Do not swap! Have dataset relating to each thread moved to that thread in one go 19

20 Method 3: Parallel Sorting by Regular Sampling [PSRS](Selecting median from regularly sampled set) Apply quicksort in each thread Select possible median at every nth element in each thread Apply quicksort on gathered samples Select P-1 pivots from samples at regular intervals Broadcast pivots Each thread must partition its data in to P partitions Distribute ith partition of data to i th thread Merge data in every thread with local quick sort Parallellization 20

21 Parallellization Execution FLow of PSRS 21

22 OpenMP 22

23 OpenMP Checking for OpenMP installation dpkg --status <package> libgomp1 32 bit x86 installtion lib32gomp1 32 support on x64 installation lib64gomp1 64 support on x64 installation Check if compiler is installed dpkg --status gcc 23

24 Pre-Processor Directives #pragma omp pragmas_name [clauses] pragma omp Directive to preprocessor States that the code following is parallel code pragmas_name Directive indicating the type of block that follows clauses Optional attributes such as data specifications 24

25 Pre-Processor Directives #pragma omp pragmas_name [clauses] pragma omp Directive to preprocessor States that the code following is parallel code pragmas_name Directive indicating the type of block that follows clauses Optional attributes such as data specifications 25

26 #include "stdio.h" #include <omp.h> Simple Parallel Program int main(int argc, char *argv[]) { #pragma omp parallel { int NCPU,tid,NPR,NTHR; /* get the total number of CPUs/cores available for OpenMP */ NCPU = omp_get_num_procs(); /* get the current thread ID in the parallel region */ tid = omp_get_thread_num(); /* get the total number of threads available in this parallel region */ NPR = omp_get_num_threads(); /* get the total number of threads requested */ NTHR = omp_get_max_threads(); /* only execute this on the master thread! */ if (tid == 0) { printf("%i : NCPU\t= %i\n",tid,ncpu); printf("%i : NTHR\t= %i\n",tid,nthr); printf("%i : NPR\t= %i\n",tid,npr); } printf("%i : hello multicore user! I am thread %i out of %i\n",tid,tid,npr); } return(0); } 26

27 Output of Program 1 Output: $gcc -g -O0 -fopenmp hello-openmp.c -o hello-openmp.out $export OMP_NUM_THREADS=`grep 'processor' /proc/cpuinfo wc -l ` $./hello-openmp.out 1 : hello multicore user! I am thread 1 out of 8 0 : NCPU = 2 0 : NTHR = 2 0 : NPR = 2 0 : hello multicore user! I am thread 0 out of 8 Pragma omp parallel: #pragma omp parallel [clauses] Clauses: private(list) shared(list) default(shared none) firstprivate(list) reduction(operator: list) copyin (list) if (scalar_expression) 27

28 Clauses for Parallel Pragma Private #pragrma omp parallel private(i,j) Make a copy of i and j that are local to the thread Must be initialized within the block and used only in block Shared #pragrma omp parallel shared(k,l) Shared between threads Only one area in memory Default #pragrma omp parallel default(none) Every variable must be declared with private,shared,etc #pragrma parallel default(shared) Every variable not listed is a shared variable Firstprivate #pragrma omp parallel firstprivate(m,n) Copy values that existed before parallel block, rest is like private 28

29 Clauses for Parallel Pragma Reduction #pragrma omp parallel reduction(+: sum) Same as private except for Initialization: Based on operator Post parallel block: Reduced using the operator and initial value Copyin #pragrma omp parallel copyin(a) Copy values from master thread If #pragma omp parallel if(a>0) Run in parallel based on a condition Multiple clauses can be used with space in between 29

30 Other Pragmas Critical #pragrma omp critical Following is operation on shared data Allow only one thread at a time Master #pragrma omp master Following must be executed only in the master thread Will be executed only once Single #pragrma omp single Only one thread should execute the following code Waits for all threads to complete till here before continuing (barrier) For #pragrma omp for schedule (dynamic, chunk) Execute following for loop in parallel Divide single loop variable value in chunks Allocate variable chunks to threads on demand Barrier #pragrma omp barrier Wait here till all threads finish execution 30

31 OMP C code Demonstration 31

32 Summary Parllalization cannot be applied on all algorithms There are many ways to parallelize a single algorithm Shared data management is essential to parallelization OpenMP is a library can be used to create parallel programs in C Complexity estimation of parallel algorithms are based on thread count 32

33 References Quinn, M., (2003) Parallel Programming in C with MPI and OpenMP, 1st edn, McGraw- Hill 33