CS483 Analysis of Algorithms Lecture 03 Divide-n-Conquer

Transcription

1 CS483 Analysis of Algorithms Lecture 03 Divide-n-Conquer Jyh-Ming Lien February 05, 2009 this lecture note is based on Algorithms by S. Dasgupta, C.H. Papadimitriou, and U.V. Vazirani and to the Design and Analysis of Algorithms by Anany Levitin. Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 1

2 In this lecture we will two main topics: Sort and selection Mergesort and quicksort Binary search Closest-pair and convex-hull algorithms of large integers We will approach these problems using the divide-and-conquer technique Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 2

3 Divide and Conquer Divide and Conquer Divide and Conquer s Master Theorem Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 3

4 Divide and Conquer Divide and Conquer Divide and Conquer Divide and Conquer s Master Theorem Divide and conquer was a successful military strategy long before it became an algorithm design strategy Coalition uses divide-conquer plan in Fallujah By Rowan Scarborough and Bill Gertz, THE WASHINGTON TIMES Coalition troops are employing a divide-and-conquer strategy in Fallujah, Iraq, capitalizing on months of pinpointed intelligence to seal off terrorist-held neighborhoods and then attack enemy pockets. : Your CS 483 instructor give you a 50-question assignment today and ask you to turn it in the tomorrow. What should you do? Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 4

5 Divide and Conquer Divide and Conquer Divide and Conquer Divide and Conquer s Master Theorem The most-well known algorithm design strategy: 1. Divide instance of problem into two or more smaller instances 2. Solve smaller instances recursively 3. Obtain solution to original (larger) instance by combining these solutions Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 5

6 Divide and Conquer s Divide and Conquer Divide and Conquer Divide and Conquer s Master Theorem : Given a list A = {2,3,6,4,12,1,7}, compute 7 i=1 A i Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 6

7 Master Theorem Divide and Conquer Divide and Conquer Divide and Conquer s Master Theorem If we have a problem of size n and our algorithm divides the problems into b instances, with a of them needing to be solved. Then we can set up our running time T(n) as: T(n) = at(n/b) + f(n), where f(n) is the time spent on dividing and merging. Master Theorem: If f(n) Θ(n d ), with d 0, then Θ(n d ) if a < b d T(n) = Θ(n d log n) if a = b d Θ(n log b a ) if a > b d s: 1. T(n) = 4T(n/2) + n T(n) = 2. T(n) = 4T(n/2) + n 2 T(n) = 3. T(n) = 4T(n/2) + n 3 T(n) = Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 7

8 Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 8

9 Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Given an array of n numbers, sort the element from small to large. Algorithm 0.1: MERGESORT(A[1 n]) Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 9

10 Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Merge two sorted arrays, B and C and put the result in A Algorithm 0.2: MERGE(B[1 p], C[1 q], A[1 p + q]) Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 10

11 : 24, 11, 91, 10, 22, 32, 22, 3, 7, 99 Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Is Mergesort stable? Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 11

12 Analysis of Merge Sort Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull C worst (n) Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 12

13 Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Given an array of n numbers, sort the element from small to large. Algorithm 0.3: QUICKSORT(A[1 n]) A[1] in the above algorithm is called pivot Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 13

14 : 24, 11, 91, 10, 22, 32, 22, 3, 7, 22 Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Is Quicksort stable? Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 14

15 Analysis of Quicksort Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull C worst (n) C best (n) C avg (n) Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 15

16 Why is Quicksort quicker? Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Because quicksort allows very fast in-place partition Algorithm 0.4: PARTITION(A[a b]) Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 16

17 The Selection Problem Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Similar to partition in quicksort! Find the k-th smallest element in an array A with n unique elements Algorithm 0.5: SELECT(A[1 n], k) The algorithm above will work well for A with unique elements. How do you change to make it work for more general cases? Time complexity: Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 17

18 Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Imagine that you are placed in an unknown building and you are given a room number, you need to find your CS 483 instructor. What will you do? : Very efficient algorithm for searching in sorted array : find 70 in {3, 14, 27, 31, 39, 42, 55, 70, 74, 81, 85, 93, 98} Efficient search in even in high dimensional unknown space : Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 18

19 Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Given a sorted array A of n numbers, find a key K in A Algorithm 0.6: BINARYSEARCH(A[1 n], K) Binary search is in fact a bad (degenerate) example of divide-and-conquer Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 19

20 Analysis of C worst (n) C best (n) C avg (n) Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer note 1 of slide 19

21 Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Find the closest distance between points in a given point set Algorithm 0.7: CP(P[1 n]) comment: P is a set n points Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 20

22 Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Find the closest distance between points in a given point set Algorithm 0.8: COMBINE(c, P, P 1, P 2, d) What is the time complexity? Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 21

23 Convex Hull Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Here we consider a divide-and-conquer algorithm called quickhull Quickhull is similar to quicksort why? Observations (given a point set P in 2-d): The leftmost and rightmost points in P must be part of the convex hull The furthest point away from any line must be part of the convex hull Points in the triangle formed by any three points in P will not be part of the convex hull Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 22

24 Quickhull Analysis of Merge Sort Analysis of Quicksort Why is Quicksort quicker? The Selection Problem Convex Hull Quickhull Qhull Algorithm 0.9: QHULL(P[1 n]) comment: P is a set n points Animation: ah/alg anim/version1/quickhull.html Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 23

25 Analysis of Quickhull Worst case: Best case: Avg case: Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer note 1 of slide 23

26 Representing polynomial Polynomial evaluation Horner s Rule A closer look Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 24

27 Representing polynomial Polynomial evaluation Horner s Rule A closer look What is the time complexity of multiplying two integers using the algorithms we learned in elementary schools? : how do you compute this: ? Is there a better way of multiplying two intergers than this elementary-school method? Carl Friedrich Gauss ( ) discovered that AB = (a10 n 2 + b)(c10 n 2 + d) = : how do you compute this: ? Carl Friedrich Gauss Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 25

28 Representing polynomial Polynomial evaluation Horner s Rule A closer look Divid-and-conquer interger multiplication Algorithm 0.10: M(A[1 n], B[1 n]) What is the time complexity? Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 26

29 Representing polynomial Polynomial evaluation Horner s Rule A closer look Strassen s Matrix : [ ] [ ] [ ] C11 C 12 A11 A = 12 B11 A 12 C 21 C 22 A 21 A 22 B 21 B 22 [ ] m1 + m = 4 m 5 + m 7 m 3 + m 5 m 2 + m 4 m 1 + m 3 m 2 + m 6 m 1 = (A 11 + A 22 )(B 11 + B 22 ) m 2 = (A 21 + A 22 )B 11 m 3 = A 11 (B 12 B 22 ) m 4 = A 22 (B 21 B 11 ) m 5 = (A 11 + A 12 )B 22 m 6 = (A 21 A 11 )(B 11 + B 12 ) m 7 = (A 12 A 22 )(B 21 + B 22 ) Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 27

30 Representing polynomial Polynomial evaluation Horner s Rule A closer look What is the time complexity? Do you still remember what the time complexity of the brute-force algorithm is? Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 28

31 Polynomial multiplication Representing polynomial Polynomial evaluation Horner s Rule A closer look two degree-n polynomials: A(x) = a n x n + a n 1 x n a 1 x + a 0 B(x) = b n x n + b n 1 x n b 1 x + b 0 of two degree-n polynomial C(x) = A(x)B(x) = c 2n x 2n + c 2n 1 x 2n c 1 x + c 0 The coefficient c k is: A brute force method for computing C(x) will have time complexity= Can we do better? Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 29

32 Representing polynomial Representing polynomial Polynomial evaluation Horner s Rule A closer look Fact: A degree-n polynomial is uniquely defined by any n + 1 distinct points A degree-n polynomial A(x) can be represented by: We can convert between these two representations: 1.5cm The value representation allows us to develop faster algorithm! We only need 2n + 1 points for C(x) It s easy and efficient to generate these 2n + 1 points from A(x) and B(x) Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 30

33 Representing polynomial Polynomial multiplication Polynomial evaluation Horner s Rule A closer look General idea: 1. Convert A and B to value representation (Evaluation) 2. Perform multiplication to obtain C in value representation 3. Convert C back to coefficient representation (Interpolation) Coefficient representation A(x) = a n x n + a n 1 x n a 1 x + a 0 B(x) = b n x n + b n 1 x n b 1 x + b 0 A(x 0 ),A(x 1 ),...,A(x 2n ) B(x 0 ), B(x 1 ),...,B(x 2n ) Value representation Evaluation O(n log n) O(n 2 ) O(n) C(x i ) = A(x i )B(x i ) C(x) = c 2n x 2n + c 2n 1 x 2n c 1 x + c 0 Interpolation O(n log n) C(x 0 ),C(x 1 ),...,C(x 2n ) Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 31

34 Polynomial evaluation Representing polynomial Polynomial evaluation Horner s Rule A closer look f(x) = a n x n + a n 1 x n a 1 x + a 0 Polynomial evaluation: Given x, compute f(x) Brute force algorithm Algorithm 0.11: F(x) Time complexity of this brute force algorithm? Can we do better? Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 32

35 Horner s Rule Representing polynomial Polynomial evaluation Horner s Rule A closer look Horner s rule f(x) = a n x n + a n 1 x n a 1 x + a 0 = (a n x n 1 + a n 1 x n a 1 )x + a 0 = ( (a n x + a n 1 )x + )x + a 0 Polynomial evaluation using Horner s rule Algorithm 0.12: F(x) Time complexity: : f(x) = 2x 4 x 3 + 3x 2 + x 5 at x = 4 Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 33

36 Representing polynomial Polynomial evaluation Horner s Rule A closer look Basic idea: How we select x i affects the run time. : If we pick ±x 0, ±x 1,..., ±x n/2 1, then A(x i ) and A( x i ) have many overlap x 5 + 2x 4 + 3x 3 + 4x 2 + 5x + 6 = A(x) = When evaluate x i, A(x i ) = When evaluate x i, A( x i ) = What we need is x i such that Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 34

37 Representing polynomial Polynomial evaluation Horner s Rule A closer look Idea: Use : z n = 1 as our x i Background: Complex number z = r(cos(θ) + i sin(θ)) Usually denoted as re iθ or (r, θ) (r 1, θ 1 ) (r 2, θ 2 ) = (r 1 r 2, θ 1 + θ 2 ) Let ω n = cos( 2π n ) + isin(2π n ) = e2πi/n be a complex n-th root of unity Other roots include: ωn, 2 ωn, 3...,ωn n 1, ωn n Properties: ωn j = ωn j+n/2 Therefore, (ωn) j 2 = ( ωn j+n/2 ) 2 Moreover, (ωn) j 2 = ω j n/2 P n i=1 ωi n = 1 ωn n 1 ω n = 0 Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 35

38 Representing polynomial Polynomial evaluation Horner s Rule A closer look s n = 8: Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 36

39 Representing polynomial Polynomial evaluation Horner s Rule A closer look FFT Algorithm 0.13: FFT((a 0, a 1, a 2,..., a n 1 ), ω) Time complexity? Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 37

40 Representing polynomial Polynomial evaluation Horner s Rule A closer look Hardware implementation Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 38

41 Representing polynomial Polynomial evaluation Horner s Rule polynomial interpolation A closer look Convert the values C(x i ) back to coefficients: {c i }=FFT(C(x i ), ω 1) Here is why M n (ω) = Entry (j, k) of M n is ω jk Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 39

42 Representing polynomial Polynomial evaluation Horner s Rule polynomial interpolation A closer look M n (ω) is invertible, i.e., column j and column k are orthogonal proof: Inversion formula M n (ω) 1 = 1 n M n(ω 1 ) proof: Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 40

43 A closer look Representing polynomial Polynomial evaluation Horner s Rule A closer look Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 41

44 Summary Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 42

45 Summary Summary Summary Sort and select Mergesort and quicksort 1 Binary search Closest-pair and convex-hull algorithms of large integers - from Gauss - FFT 1 (Also from Gauss) Divide-n-conquer strategy Advantages of Make problems easier Easy parallelization Disadvantages of Divide-n-conquer strategy Recursion can be slow Subproblems may overlap 1 Named one of 10 best algorithms in last century Analysis of Algorithms CS483 Lecture 03-Divide-n-Conquer 43