CSL 730: Parallel Programming

Transcription

1 CSL 73: Parallel Programming

2 General Algorithmic Techniques Balance binary tree Partitioning Divid and conquer Fractional cascading Recursive doubling Symmetry breaking Pipelining 2

3 PARALLEL ALGORITHM TECHNIQUES: BALANCED BINARY TREE

4 Reduction n operands => log n steps Total work = O(n) How do you map? Balance Binary tree technique

5 Reduction n operands => log n steps How do you map? n/2 i processors per step step i: if!(id%2 i )

6 Reduction n operands => log n steps Only have p processors per step Agglomerate and Map

7 Prefix Sums P[] = x[] For i = 1 to n-1 P[i] = P[i-1] + x[i]

8 Recursive Prefix Sums P[] = x[] For i = 1 to n-1 P[i] = P[i-1] + x[i] Prefix Sum

9 Recursive Prefix Sums prefixsums(p, x, (:n]) { parallel for i in (:n/2] y[i] = OP(x[2*i], x[2*i+1]) prefixsum(z, y, (:n/2]) P[] = x[] parallel for i in [1:n] if(i&1) P[i] = z[i/2] else P[i] = OP(z[i/2-1 ], x[i]) } Prefix Sum Or OP -1 (z[i/2], x[i]), if op inver@ble

10 Prefix Sums P[] = x[] For i = 1 to n-1 P[i] = P[i-1] + x[i] S(:n/2] S(n/2:n] S(n/2:3n/4]

11 Prefix Sums P[] = x[] For i = 1 to n-1 P[i] = P[i-1] + x[i] S(:n/2] S(:n/2] S(n/2:n] S(:3n/4]

12 Non-recursive Prefix Sums parallel for i = to n B[][i] = A[i] for h = 1 to log n parallel for i in :n/2 h B[h][i] = B[h-1][2i] OP B[h-1][2i+1] for h = log n to C[h][] = B[h][] parallel for i in 1:n/2 h Odd i: C[h][i] = C[h+1][i/2] Even i: C[h][i] = C[h+1][(i/2-1] OP B[h][i]

13 Prefix Sums: Data flow up B[3][] B[2][] B[2][1] B[1][] B[1][1] B[1][2] B[1][3] B[][] B[][1] B[][2] B[][3] B[][4] B[][5] B[][6] B[][7]

14 Prefix Sums: Data flow down C[3][] = B[3][] C[2][] C[2][1] C[1][] C[1][1] C[1][2] C[1][3] C[][] C[][1] C[][2] C[][3] C[][4] C[][5] C[][6] C[][7] O(n) work, O(log

15 Processor Mapping P P P1 P P P1 P1 P P P P P1 P1 P1 P1

16 Balanced Tree Approach Build binary tree on the input Hierarchically divide into groups and groups of groups.. Traverse tree upwards/downwards Useful to think of tree network topology Only for algorithm design Later map sub-trees to processors 16

17 PARALLEL ALGORITHM TECHNIQUES: PARTITIONING

18 Merge Sorted Sequences (A,B) Determine Rank of each element in A U B Rank(x, A U B) = Rank(x, A) + Rank(x, B) Only need to compute the rank in the other list, if A and B are each sorted already Find Rank(A, B), and similarly Rank(B, A) Find Rank by binary search O(log n) time O(n log n) work 18

19 Towards Optimal Merge (A,B) Partition A and B into log n sized blocks Select from B, elements i * log n, i :n/log n Rank each selected element of B in A Binary search Merge pairs of sub-sequences If A i = log(n), Sequential merge in time O(log(n) ) Otherwise, partition A i into log n blocks And Recursively subdivide B i into sub-sub-sequences

20 Towards Optimal Merge (A,B) Total time is O(log(n)) Total work is O(n) 2

21 Towards Optimal Merge (A,B) Partition A and B into log n sized blocks Select from B, elements i * log n, i = :n/log n Rank each selected element of B in A Binary search Merge pairs of sub-sequences If A i = log(n), Sequential merge in time O(log(n) ) Otherwise, partition A i into log n sized blocks And (similarly) subdivide B i into sub-sub-sequences Complexity: O(log(n)) time O(n) work Should we expect bejer?

22 Fast Merge (A,B) Partition A and B into n blocks Select from B, elements i n, i (: n] Rank each selected element of B in A Parallel search using n processors each search Similarly rank n selected elements from A in B Recursively merge pairs of sub-sequences Total time: T(n) = O(1)+T( n) = O(log log n) Total work: W(n) = O(n)+ n W( n) = O(n log log n) Fast but still need to reduce work Not oplmal yet

23 Optimal Merge (A,B) Use the fast, but non-optimal, algorithm on small enough subsets Subdivide A and B into blocks of size log log n A 1, A 2,.. B 1, B 2,.. Select first element of each block A = p 1, p 2.. B = q 1, q 2.. Now merge loglogn sized blocks n/loglogn times

24 Optimal Merge (A,B) A llg n A n/llg n merge B n/llg n B llg n llg n llg: log log 24

25 Optimal Merge (A,B) A llg n A n/llg n merge B n/llg n B llg n llg n 25

26 Optimal Merge (A,B) 1.Merge A and B find Rank(A :B ), Rank(B :A ) using fast non-optimal algorithm Time = O(log log n) Work = O(n) 2.Compute Rank(A :B) and Rank(B :A) llg If Rank(p i, B) is r i, p i lies in block B ri Sequentially Time = O(log log n) Work = O(n) 3.Compute ranks of remaining elements Sequentially llg Time = O(log log n) Work = O(n) llg

27 Quick Sort Choose the pivot Select median? Subdivide into two groups Group sizes linearly related with high probability Sort each group independently

28 QuickSort Algorithm QuickSort(int A[], int first, int last) { } Select random m in [first:last] parallel for i in [first:last] flag[i] = A[i] < A[m]; // A[m] is pivot Split(A); // Separate flag values and 1, A[m] goes to k // Use Prefix Sum Quicksort A[first:k-1] and A[k+1:last]

29 Quick Sort Choose the pivot Select median? Subdivide into two groups Group sizes linearly related with high probability Sort each group independently Expected O(log n ) rounds Time per round = O(log n) Total work = O(n log n) with high probability

30 Hyper Quick Sort Partition into p groups Sort each group independently O(n/p log n) Choose median of one of the groups Partition each group into less and more set Binary search of the median: (log n) Separate into low and high Merge p/2 less and p/2 more pairs Each sequentially: O(n/p) Now we have p/2 less lists and p/2 more lists Partition and recurse Total time = O(n/p log n) with high probability

31 Partitioning Approach Break into p roughly equal sized problems Solve each sub-problem Preferably, independently of each other Focus on subdividing into independent parts 31

32 PARALLEL ALGORITHM TECHNIQUES: DIVIDE AND CONQUER

33 Merge Sort Partition data into two halves Assign half the processors to each half If only one processor remains, sequentially sort Sort each half Merge results T(n) = T(n/2) + O(log log n) W(n) = 2 W(n/2) + O(n) More on this later

34 Convex Hull Upper Hull Lower Hull

35 Convex Hull p i

36 Convex Hull

37 Convex Hull X

38 PARALLEL ALGORITHM TECHNIQUES: ACCELERATED CASCADING

39 Min-find Input: array with n numbers Algorithm A1 using O(n 2 ) processors: parallel for i in (:n] M[i]:= parallel for i,j in (:n] if i j && C[i] < C[j] M[j]=1 parallel for i in (:n] if M[i]= min = A[i] Not optimal: O(n 2 ) work

40 Optimal Min-find Balanced Binary tree O(log n) time O(n) work => Optimal Use Accelerated cascading Make the tree branch much faster Number of children of node u = n u Where n u is the number of leaves in u s subtree Works if the operation at each node can be performed in O(1)

41 From n 2 processors to n n A1 A1 A1 A1 A1 A1 A1 A1 A1 A1 A1 Step 1: Partition into disjoint blocks of size Step 2: Apply A1 to each block Step 3: Apply A1 to the results from the step 2 n

42 From n n processors to n 1+1/4 A2 A2 A2 A2 A2 A2 A2 A2 A2 A2 A2 Step 1: Partition into disjoint blocks of size Step 2: Apply A2 to each block Step 3: Apply A2 to the results from the step 2 n 1/2. n 3/4 n 3/4

43 n 2 -> n 1+1/2 -> n 1+1/4 -> n 1+1/8 -> n 1+1/2k.. ~n 1+ ℇ Algorithm A k takes O(1) time with processors Does it? Algorithm A k+1 1. Partition input array C (size n) into disjoint blocks of size n 1/2 each 2. Solve for each block in parallel using algorithm A k 3. Re-apply A k to the results of step 3: n/n 1/2 minima n 1/2i = O(1) at leaf Doubly logarithmic-depth tree n log log n work, log log n time

44 Min-Find Review Constant-time algorithm O(n 2 ) work O(log n) Balanced Tree Approach O(n) work Optimal O(loglog n) Doubly-log depth tree Approach O(n loglog n) work Degree is high at the root, reduces going down #Children of node u = (#nodes in tree rooted at u) Depth = O(log log n)

45 Accelerated Cascading Solve recursively Start bojom-up with the oplmal algorithm unll the problem sizes is smaller Switch to fast (non-oplmal algorithm) A few small problems solved fast but non-workoplmally Min Find: OpLmal algorithm for lower loglog n levels Then switch to O(n loglog n)-work algorithm n work, log log n time

46 First 1 problem Input: n-bit vector Output: minimum index of a 1-bit Algorithm: Divide into n blocks of n bits each For block i, B[i] = 1 if there is a 1 in the block Arbitrary CRCW, O(n) work, O(1) time Find the first j such that B[j] = 1 O(n) work, O(1) time Find first 1 in the j th block O(n) work, O(1) time 46

47 All Nearest Smaller Value For each element A[i] in a vector find the largest j<i, such that A[j] < A[i] Algorithm: Find left match: For each i For each j < i, B[i,j] = (A[j] < A[i]) O(n 2 ) work For each B[i] find largest j<i with B[i,j] = 1 47

48 Prefix Minima For each element A[i] in a vector find the minimum of all A[j], j < i Algorithm: Find ANSV For each A[i] O(n 2 ) work O(1) Lme For each A[j], j<i PrefixMin[i] = A[j] where j is the largest index such that A[j] has no smaller value on its left m+ m+ m+ m+ m b a 48

49 Prefix Minima 1.Partition into n blocks of n each 2.Find prefix min for each block j recursively 3.Consider B = m 1.. m n where m j is the minima of block j Find prefix min of B in constant time 4.Minimum of A[1]..A[i], A[i] in block j is the minimum of m 1..m j-1 and A[i] s block prefix minima (computed in step 2) O(1) Lme O(n) work O(n log log n) work, O(log log n) time 49

50 PARALLEL ALGORITHM TECHNIQUES: RECURSIVE DOUBLING

51 List Ranking Given a linked list, compute distance of each node from the end: Initialize: if(next[i] == null) rank[i]=; else rank[i] = 1; log n times: parallel for i in [:n) if(next[i]!= null) rank[i] = rank[i] + rank[next[i]] next[i] = next[next[i]] 51

52 Find Roots in a Forest p(i): parent of node i Do in parallel p(i) = p(p(i)) Stop if p(i) = i Time: log(height) Work: n log(height) 9 7 9

53 Find Roots in a Forest p(i): parent of node i Do in parallel p(i) = p(p(i)) Stop if p(i) = i Time: log(height) Work: n log(height) 9 4 7

54 Find Roots in a Forest p(i): parent of node i Do in parallel p(i) = p(p(i)) Stop if p(i) = i Time: log(height) Work: n log(height) 9 1

55 Pointer Jumping Progressively push computation to all elements at a given distance Doubling the distance at each step After k steps the computation has been performed for elements within a distance of 2 k Applies to array, list, tree

56 PARALLEL ALGORITHM TECHNIQUES: SYMMETRY BREAKING

57 Cycle Coloring c[i]!= c[j] if e[i] = j e[i] = The next vertex from i O(n) SequenLal algorithm Parallel: Make local decision But all verlces look the same locally Break symmetry Put verlces in idenlfiable classes

58 3-Coloring Use vertex index to break symmetry But too many indices Algorithm: Initialize c[i] = i Iterate, reducing #colors each time parallel for all i do Let k = Least significant bit in which c[i] and c[e[i]] differ Set c[i] = 2k + bit k of c[i] 58

59 3-Coloring k = 2; 2 nd bit = => New color = 2*2 + = k = ; th bit = 1=> New color = 2* + 1 = 1 Why does this produce a coloring?

60 3-Coloring #new colors = O(log (#old colors)) Time O(log*n) Work O(n log*n) But can t reduce the number of colors below 6 t bits for i => log t + 1 bits for iteration i+1, t > 3 One more steps each to remove three extra colors parallel for all i If(c[i] = extra-color) c[i] = {,1,2} and NOT (c[[e[i]], c[predecessor[i]) 6

61 General Algorithmic Techniques Pipelining Balanced binary tree Divide and conquer Partitioning Accelerated Cascading Pointer doubling Symmetry breaking 61