Divide and Conquer Algorithms. Sathish Vadhiyar

Transcription

1 Divide and Conquer Algorithms Sathish Vadhiyar

2 Introduction One of the important parallel algorithm models The idea is to decompose the problem into parts solve the problem on smaller parts find the global result using individual results Works naturally and works well for parallelization

3 Introduction Various models Recursive sub-division: Has a division and computation phase, then a merge phase. E.g., merge sort Local compute merge/coordinate local compute. E.g., following algorithms

4 Recursive sub-division: Merge sort (you know already) Solving tri-diagonal systems

5 Parallel solution of linear system with special matrices Tridiagonal Matrices a1 h1 g2 a2 h2 x1 x2 b1 b2 g3 a3 h3 x3 b3 =.. gn an. xn. bn In general: g i x i-1 + a i x i + h i x i+1 = b i Substituting for x i-1 and x i+1 in terms of {x i-2, x i } and {x i, x i+2 } respectively: G i x i-2 + A i x i + H i x i+2 = B i

6 Tridiagonal Matrices A1 H1 A2 H2 G3 A3 H3 G4 A4 H4 x1 x2 x3.. = B1 B2 B3.. Gn-2 An xn Bn Reordering:

7 Tridiagonal Matrices A2 H2 x2 B2 G4 A4 H4 x4 B4 A1 H1 G3 A3 H3 Gn An. xn x1 x3. =. Bn B1 B3. Gn-3 An-1 xn-1 Bn-1

8 Tridiagonal Systems Thus the problem of size n has been split into even and odd equations of size n/2 This is odd even reduction For parallelization, each process can divide the problem into subproblems of smaller size and solve the subproblems This is divide-and-conquer technique

9 Tridiagonal Systems - Parallelization At each stage one representative process of the domain of processes is chosen This representative performs the odd-even reduction of problem i to two problems of size i/2 The problems are distributed to 2 representatives n n/ n/ n/

10 Local compute merge local compute Prefix Computations Sample sort

11 Parallel Algorithm: Prefix computations on arrays Array X partitioned into subarrays Local prefix sums of each subarray calculated in parallel Prefix sums of last elements of each subarray written to a separate array Y Prefix sums of elements in Y are calculated. Each prefix sum of Y is added to corresponding block of X Divide and conquer strategy

12 Example ,3,6 4,9,15 7,15,24 6,15,24 6,21,45 Passing last elements to a processor 1,3,6,10,15,21,28,36,45 Divide Local prefix sum Computing prefix sum of last elements on the processor Adding global prefix sum to local prefix sum in each processor

13 Lessons Learned.. Has local computations Global communication/coordination Back to local computations

14 Sample Sort

15 Parallel Sorting by Regular Sampling (PSRS) 1. Each processor sorts its local data 2. Each processor selects a sample vector of size p-1; kth element is (n/p * (k+1)/p) 3. Samples are sent and merge-sorted on processor 0 4. Processor 0 defines a vector of p-1 splitters starting from p/2 element; i.e., kth element is p(k+1/2); broadcasts to the other processors

16 Example

17 PSRS 5. Each processor sends local data to correct destination processors based on splitters; all-to-all exchange 6. Each processor merges the data chunk it receives

18 Step 5 Each processor finds where each of the p-1 pivots divides its list, using a binary search i.e., finds the index of the largest element number larger than the jth pivot At this point, each processor has p sorted sublists with the property that each element in sublist i is greater than each element in sublist i-1 in any processor

19 Step 6 Each processor i performs a p-way merge-sort to merge the ith sublists of p processors

20 Example Continued

21 Analysis The first phase of local sorting takes O((n/p)log(n/p)) 2 nd phase: Sorting p(p-1) elements in processor 0 O(p 2 logp 2 ) Each processor performs p-1 binary searches of n/p elements plog(n/p) 3 rd phase: Each processor merges (p-1) sublists Size of data merged by any processor is no more than 2n/p (proof) Complexity of this merge sort 2(n/p)logp Summing up: O((n/p)logn)

22 Analysis 1 st phase no communication 2 nd phase p(p-1) data collected; p-1 data broadcast 3 rd phase: Each processor sends (p-1) sublists to other p-1 processors; processors work on the sublists independently

23 Analysis Not scalable for large number of processors Merging of p(p-1) elements done on one processor; processors require 16 GB memory

24 Sorting by Random Sampling An interesting alternative; random sample is flexible in size and collected randomly from each processor s local data Advantage A random sampling can be retrieved before local sorting; overlap between sorting and splitter calculation

25 Sources/References On the versatility of parallel sorting by regular sampling. Li et al. Parallel Computing Parallel Sorting by regular sampling. Shi and Schaeffer. JPDC Highly scalable parallel sorting. Solomonic and Kale. IPDPS 2010.