Recurrences and Divide-and-Conquer

Transcription

1 Recurrences and Divide-and-Conquer Frits Vaandrager Institute for Computing and Information Sciences 16th November 2017 Frits Vaandrager 16th November 2017 Lecture 9 1 / 35

2 Algorithm Design Strategies We can classify algorithms in terms of the strategy they use to solve a problem: Incremental (e.g. insertion sort) Divide and conquer (e.g. merge sort and quicksort) Greedy (e.g. Dijkstra s algorithm) Dynamic programming (e.g. all-pairs-shortest-path) Randomized algorithms (e.g. randomized quicksort) Frits Vaandrager 16th November 2017 Lecture 9 2 / 35

3 Divide and Conquer The concept refers to a strategy that breaks up existing power structures, and especially prevents smaller power groups from linking up, causing rivalries and fomenting discord among the people. Frits Vaandrager 16th November 2017 Lecture 9 3 / 35

4 Divide and Conquer Frits Vaandrager 16th November 2017 Lecture 9 4 / 35

5 Divide and Conquer Algorithm design strategy: 1 Divide a problem into n sub-problems (smaller instances of the original problem) 2 Conquer the sub-problems trivial for small sizes use the same strategy, otherwise 3 Combine solutions of sub-problems into a solution of the original problem Implementation is typically recursive Frits Vaandrager 16th November 2017 Lecture 9 5 / 35

6 Multiply and Conquer Frits Vaandrager 16th November 2017 Lecture 9 6 / 35

7 Multiply and Conquer Frits Vaandrager 16th November 2017 Lecture 9 7 / 35

8 Multiply and Conquer Thus far there is no multiply and conquer algorithm design strategy... Frits Vaandrager 16th November 2017 Lecture 9 8 / 35

9 Multiply and Conquer Thus far there is no multiply and conquer algorithm design strategy... However, Kuratsuba used divide and conquer to develop a fast multiplication algorithm with complexity O(n lg 3 ) O(n )! Frits Vaandrager 16th November 2017 Lecture 9 8 / 35

10 Algorithm invented by John von Neumann in 1945 that uses a divide and conquer strategy: Divide vector in two vectors of similar size Conquer: recursively sort the two vectors (trivial for... Frits Vaandrager 16th November 2017 Lecture 9 9 / 35

11 Algorithm invented by John von Neumann in 1945 that uses a divide and conquer strategy: Divide vector in two vectors of similar size Conquer: recursively sort the two vectors (trivial for... size 1) Combine two ordered vectors into a new ordered vector. Auxiliary merge function: 1 The merge function gets as input two ordered sequences A[p...q] and A[q+1...r] 2 As output it returns the sequence A[p...r] ordered Frits Vaandrager 16th November 2017 Lecture 9 9 / 35

12 void merge_sort(int A[], int p, int r) { if (p < r) { /* base case? */ q = (p+r)/2; /* Divide */ merge_sort(a,p,q); /* Conquer */ merge_sort(a,q+1,r); /* Conquer */ merge(a,p,q,r); /* Combine */ } } Frits Vaandrager 16th November 2017 Lecture 9 10 / 35

13 void merge_sort(int A[], int p, int r) { if (p < r) { /* base case? */ q = (p+r)/2; /* Divide */ merge_sort(a,p,q); /* Conquer */ merge_sort(a,q+1,r); /* Conquer */ merge(a,p,q,r); /* Combine */ } } Question: What is the dimension of each sequence in divide step? Frits Vaandrager 16th November 2017 Lecture 9 10 / 35

14 void merge_sort(int A[], int p, int r) { if (p < r) { /* base case? */ q = (p+r)/2; /* Divide */ merge_sort(a,p,q); /* Conquer */ merge_sort(a,q+1,r); /* Conquer */ merge(a,p,q,r); /* Combine */ } } Question: What is the dimension of each sequence in divide step? Initial call on an array with n elements: merge sort(a,1,n). Frits Vaandrager 16th November 2017 Lecture 9 10 / 35

15 Example I Frits Vaandrager 16th November 2017 Lecture 9 11 / 35

16 Example II Array with number of elements not a multiple of 2 Frits Vaandrager 16th November 2017 Lecture 9 12 / 35

17 Remains to do... merge procedure Input: Array A and indices p, q, r such that p q < r Subarrays A[p..q] and A[q + 1..r] are sorted. By restrictions on p, q, r, neither subarray is empty. Output: The two subarrays are merged into a single sorted subarray in A[p..r]. Frits Vaandrager 16th November 2017 Lecture 9 13 / 35

18 Merge Idea behind merging: Think of two piles of cards. Each pile is sorted and placed face-up on a table with the smallest cards on top. We will merge these into a single sorted pile, face-down on the table. A basic step: Choose the smaller of the two top cards. Remove it from its pile, thereby exposing a new top card. Place the chosen card face-down onto the output pile. Repeatedly perform basic steps until one input pile is empty. Once one input pile empties, just take the remaining input pile and place it face-down onto the output pile. Frits Vaandrager 16th November 2017 Lecture 9 14 / 35

19 Merge Idea behind merging: Think of two piles of cards. Each pile is sorted and placed face-up on a table with the smallest cards on top. We will merge these into a single sorted pile, face-down on the table. A basic step: Choose the smaller of the two top cards. Remove it from its pile, thereby exposing a new top card. Place the chosen card face-down onto the output pile. Repeatedly perform basic steps until one input pile is empty. Once one input pile empties, just take the remaining input pile and place it face-down onto the output pile. What is the complexity of this algorithm? Frits Vaandrager 16th November 2017 Lecture 9 14 / 35

20 Merge Each basic step should take constant time, since we check just the two top cards. There are n basic steps, since each basic step removes one card from the input piles, and we started with n cards in the input piles. Therefore, this procedure should take O(n) time. Frits Vaandrager 16th November 2017 Lecture 9 15 / 35

21 Merge procedure void merge(int A[], int p, int q, int r) { int L[MAX], R[MAX]; int n1 = q-p+1; /* length of array A[p..q] */ int n2 = r-q; /* length of array A[q+1..r] */ for (i=1 ; i<=n1 ; i++) L[i] = A[p+i-1]; /* array LEFT */ for (j=1 ; j<=n2 ; j++) R[j] = A[q+j]; /* array RIGHT */ L[n1+1] = MAXINT; R[n2+1] = MAXINT; /* sentinels */ Frits Vaandrager 16th November 2017 Lecture 9 16 / 35

22 Merge procedure void merge(int A[], int p, int q, int r) { int L[MAX], R[MAX]; int n1 = q-p+1; /* length of array A[p..q] */ int n2 = r-q; /* length of array A[q+1..r] */ for (i=1 ; i<=n1 ; i++) L[i] = A[p+i-1]; /* array LEFT */ for (j=1 ; j<=n2 ; j++) R[j] = A[q+j]; /* array RIGHT */ L[n1+1] = MAXINT; R[n2+1] = MAXINT; /* sentinels */ what are the sentinels good for? Frits Vaandrager 16th November 2017 Lecture 9 16 / 35

23 Merge procedure void merge(int A[], int p, int q, int r) { int L[MAX], R[MAX]; int n1 = q-p+1; /* length of array A[p..q] */ int n2 = r-q; /* length of array A[q+1..r] */ for (i=1 ; i<=n1 ; i++) L[i] = A[p+i-1]; /* array LEFT */ for (j=1 ; j<=n2 ; j++) R[j] = A[q+j]; /* array RIGHT */ L[n1+1] = MAXINT; R[n2+1] = MAXINT; /* sentinels */ } i = 1; j = 1; for (k=p ; k<=r ; k++) if (L[i] <= R[j]) { /* put the next smallest A[k] = L[i]; i++; element in the array */ } else { A[k] = R[j]; j++; } Frits Vaandrager 16th November 2017 Lecture 9 16 / 35

24 Merge procedure void merge(int A[], int p, int q, int r) { int L[MAX], R[MAX]; int n1 = q-p+1; /* length of array A[p..q] */ int n2 = r-q; /* length of array A[q+1..r] */ for (i=1 ; i<=n1 ; i++) L[i] = A[p+i-1]; /* array LEFT */ for (j=1 ; j<=n2 ; j++) R[j] = A[q+j]; /* array RIGHT */ L[n1+1] = MAXINT; R[n2+1] = MAXINT; /* sentinels */ } i = 1; j = 1; for (k=p ; k<=r ; k++) if (L[i] <= R[j]) { /* put the next smallest A[k] = L[i]; i++; element in the array */ } else { A[k] = R[j]; j++; } merge executes in Θ(n) time, where n = r p + 1 Frits Vaandrager 16th November 2017 Lecture 9 16 / 35

25 Example merge(a,9,12,16) Frits Vaandrager 16th November 2017 Lecture 9 17 / 35

26 Example c d merge(a,9,12,16) Frits Vaandrager 16th November 2017 Lecture 9 18 / 35

27 Complexity analysis Algorithm with a recursive call: running time can usually be described as a recurrence. Frits Vaandrager 16th November 2017 Lecture 9 19 / 35

28 Complexity analysis Algorithm with a recursive call: running time can usually be described as a recurrence. How does that work? Frits Vaandrager 16th November 2017 Lecture 9 19 / 35

29 Complexity analysis Algorithm with a recursive call: running time can usually be described as a recurrence. How does that work? Let s for a moment assume that the input size is a power of 2. In each division step the sub-arrays have exactly size n/2. Frits Vaandrager 16th November 2017 Lecture 9 19 / 35

30 Complexity analysis Algorithm with a recursive call: running time can usually be described as a recurrence. How does that work? Let s for a moment assume that the input size is a power of 2. In each division step the sub-arrays have exactly size n/2. T (n): time of execution (in the worst case) for a input of size n; If n = 1 then T (n) is constant: T (n) Θ(1). Frits Vaandrager 16th November 2017 Lecture 9 19 / 35

31 Complexity analysis Otherwise: 1 Divide computing middle of vector is done in constant time: Θ(1). 2 Conquer two problems of size n/2 are solved: 2T (n/2). 3 Compose merge function executes in linear time: Θ(n) Frits Vaandrager 16th November 2017 Lecture 9 20 / 35

32 Complexity analysis Otherwise: 1 Divide computing middle of vector is done in constant time: Θ(1). 2 Conquer two problems of size n/2 are solved: 2T (n/2). 3 Compose merge function executes in linear time: Θ(n) Hence: T (n) = { Θ(1) if n = 1 Θ(1) + 2T (n/2) + Θ(n) if n > 1 Frits Vaandrager 16th November 2017 Lecture 9 20 / 35

33 Complexity analysis Otherwise: 1 Divide computing middle of vector is done in constant time: Θ(1). 2 Conquer two problems of size n/2 are solved: 2T (n/2). 3 Compose merge function executes in linear time: Θ(n) Hence: T (n) = { Θ(1) if n = 1 Θ(1) + 2T (n/2) + Θ(n) if n > 1 and now we want to conclude that T (n) Θ(??). Frits Vaandrager 16th November 2017 Lecture 9 20 / 35

34 Merge vs Insertion sort Running ahead: we will show that T (n) Θ(n lg n). Frits Vaandrager 16th November 2017 Lecture 9 21 / 35

35 Merge vs Insertion sort Running ahead: we will show that T (n) Θ(n lg n). Cf. Insertion sort is Θ(n 2 ) and quicksort is O(n 2 ). Frits Vaandrager 16th November 2017 Lecture 9 21 / 35

36 Merge vs Insertion sort Running ahead: we will show that T (n) Θ(n lg n). Cf. Insertion sort is Θ(n 2 ) and quicksort is O(n 2 ). Frits Vaandrager 16th November 2017 Lecture 9 21 / 35

37 Recurrence relations Going back: How did we get n lg n for merge sort? Frits Vaandrager 16th November 2017 Lecture 9 22 / 35

38 Recurrence relations Going back: How did we get n lg n for merge sort? We had { Θ(1) if n = 1 T (n) = Θ(1) + 2T (n/2) + Θ(n) if n > 1 Frits Vaandrager 16th November 2017 Lecture 9 22 / 35

39 Recurrence relations Going back: How did we get n lg n for merge sort? We had { Θ(1) if n = 1 T (n) = Θ(1) + 2T (n/2) + Θ(n) if n > 1 Let us rewrite this as T (n) = { c if n = 1 2T (n/2) + cn if n > 1 Frits Vaandrager 16th November 2017 Lecture 9 22 / 35

40 Recurrence relations Going back: How did we get n lg n for merge sort? We had { Θ(1) if n = 1 T (n) = Θ(1) + 2T (n/2) + Θ(n) if n > 1 Let us rewrite this as T (n) = { c if n = 1 2T (n/2) + cn if n > 1 where c is the largest constant representing both the time required to solve problems of dimension 1 as well as the time per array element of the divide and combine steps. Frits Vaandrager 16th November 2017 Lecture 9 22 / 35

41 Recurrence relations c d For original problem we have cost cn plus cost of sub-problems: Frits Vaandrager 16th November 2017 Lecture 9 23 / 35

42 Recurrence relations c d For original problem we have cost cn plus cost of sub-problems: For each size-n/2 subproblem we have cost cn/2, plus two sub-problems, each costing T (n/4): Frits Vaandrager 16th November 2017 Lecture 9 23 / 35

43 Recurrence relations c d Continue expanding until the problem sizes get down to 1: Each level has cost cn Frits Vaandrager 16th November 2017 Lecture 9 24 / 35

44 Recurrence relations c d Continue expanding until the problem sizes get down to 1: Each level has cost cn There are lg n + 1 levels (height is lg n; proof by induction) Frits Vaandrager 16th November 2017 Lecture 9 24 / 35

45 Recurrence relations c d Continue expanding until the problem sizes get down to 1: Each level has cost cn There are lg n + 1 levels (height is lg n; proof by induction) Total cost is sum of costs at each level: cn(lg n + 1) = cn lg n + cn Frits Vaandrager 16th November 2017 Lecture 9 24 / 35

46 Recurrence relations c d Continue expanding until the problem sizes get down to 1: Each level has cost cn There are lg n + 1 levels (height is lg n; proof by induction) Total cost is sum of costs at each level: cn(lg n + 1) = cn lg n + cn Ignore low-order term: merge sort is in Θ(n lg n) Frits Vaandrager 16th November 2017 Lecture 9 24 / 35

47 How to solve recurrences? First, a technicality: we assumed length of the array was a power of 2. If that s not case, the right formula for the recurrence is: { Θ(1) if n = 1 T (n) = T ( n/2 ) + T ( n/2 ) + Θ(n) if n > 1 Frits Vaandrager 16th November 2017 Lecture 9 25 / 35

48 How to solve recurrences? First, a technicality: we assumed length of the array was a power of 2. If that s not case, the right formula for the recurrence is: { Θ(1) if n = 1 T (n) = T ( n/2 ) + T ( n/2 ) + Θ(n) if n > 1 The solution of a recurrence can be verified using the substitution method, which allows us to prove that indeed the recurrence above has solution T (n) Θ(n lg n) Frits Vaandrager 16th November 2017 Lecture 9 25 / 35

49 Substitution method 1 Guess solution 2 Use induction to find constants and show that solution works Example: Consider the following recurrence { 1 if n = 1 T (n) = 2T (n/2) + n if n > 1 1 Guess: T (n) = n lg n + n (usually done by looking at the recursion tree). 2 Induction: Basis: n = 1 n lg n + n = 1 = T (n) Frits Vaandrager 16th November 2017 Lecture 9 26 / 35

50 Substitution method Inductive step: Inductive hypothesis is that T (k) = k lg k + k for all k < n. We willl use this inductive hypothesis for T (n/2). T (n) = 2T (n/2) + n Frits Vaandrager 16th November 2017 Lecture 9 27 / 35

51 Substitution method Inductive step: Inductive hypothesis is that T (k) = k lg k + k for all k < n. We willl use this inductive hypothesis for T (n/2). T (n) = 2T (n/2) + n = 2(n/2 lg n/2 + n/2) + n induction hyp. Frits Vaandrager 16th November 2017 Lecture 9 27 / 35

52 Substitution method Inductive step: Inductive hypothesis is that T (k) = k lg k + k for all k < n. We willl use this inductive hypothesis for T (n/2). T (n) = 2T (n/2) + n = 2(n/2 lg n/2 + n/2) + n induction hyp. = n lg n/2 + n + n Frits Vaandrager 16th November 2017 Lecture 9 27 / 35

53 Substitution method Inductive step: Inductive hypothesis is that T (k) = k lg k + k for all k < n. We willl use this inductive hypothesis for T (n/2). T (n) = 2T (n/2) + n = 2(n/2 lg n/2 + n/2) + n induction hyp. = n lg n/2 + n + n = n(lg n lg 2) + n + n Frits Vaandrager 16th November 2017 Lecture 9 27 / 35

54 Substitution method Inductive step: Inductive hypothesis is that T (k) = k lg k + k for all k < n. We willl use this inductive hypothesis for T (n/2). T (n) = 2T (n/2) + n = 2(n/2 lg n/2 + n/2) + n induction hyp. = n lg n/2 + n + n = n(lg n lg 2) + n + n = n lg n n + n + n Frits Vaandrager 16th November 2017 Lecture 9 27 / 35

55 Substitution method Inductive step: Inductive hypothesis is that T (k) = k lg k + k for all k < n. We willl use this inductive hypothesis for T (n/2). T (n) = 2T (n/2) + n = 2(n/2 lg n/2 + n/2) + n induction hyp. = n lg n/2 + n + n = n(lg n lg 2) + n + n = n lg n n + n + n = n lg n + n Frits Vaandrager 16th November 2017 Lecture 9 27 / 35

56 Substitution method Generally, we use asymptotic notation: We would write e.g. T (n) = 2T (n/2) + Θ(n). We assume T (n) = O(1) for sufficiently small n. We express the solution by asymptotic notation: T (n) = Θ(n lg n) (or T (n) Θ(n lg n)). We do not worry about boundary cases, nor do we show base cases in the substitution proof. Frits Vaandrager 16th November 2017 Lecture 9 28 / 35

57 Substitution method T (n) is always constant for any constant n. Since we are ultimately interested in an asymptotic solution to a recurrence, it will always be possible to choose base cases that work. When we want an asymptotic solution to a recurrence, we do not worry about the base cases in our proofs. When we want an exact solution, then we have to deal with base cases. For the substitution method: Name the constant in the additive term. Show the upper (O) and lower (Ω) bounds separately. Might need to use different constants for each. Frits Vaandrager 16th November 2017 Lecture 9 29 / 35

58 Another example Take the recurrence T (n) = { 1 if n = 1 2T ( n/2 ) + n if n > 1 We seem to run into a problem if we want to show that T (n) cn lg n: T (1) c1 lg 1 = 0 FALSE! Frits Vaandrager 16th November 2017 Lecture 9 30 / 35

59 Another example Take the recurrence T (n) = { 1 if n = 1 2T ( n/2 ) + n if n > 1 We seem to run into a problem if we want to show that T (n) cn lg n: T (1) c1 lg 1 = 0 FALSE! However, recall that the O-notation only requires that we show that there exist c, n 0 > 0 for which T (n) cn lg n, for all n n 0. Frits Vaandrager 16th November 2017 Lecture 9 30 / 35

60 Another example Hence we just have to replace the base case n = 1 by something else... We take it step by step: let s look at n 0 = 2 Enough to choose c 2. T (2) = 4 c2 lg 2 = 2c Frits Vaandrager 16th November 2017 Lecture 9 31 / 35

61 Another example Hence we just have to replace the base case n = 1 by something else... We take it step by step: let s look at n 0 = 2 T (2) = 4 c2 lg 2 = 2c Enough to choose c 2. But now careful: T (2) = 2T 2/2 + 2 = 2T (1) + 2 = 4 T (3) = 2T 3/2 + 3 = 2T (1) + 3 = 5 T (4) = 2T 4/2 + 4 = 2T (2) + 4 = 12 T (5) = 2T 5/2 + 5 = 2T (2) + 5 = 13 T (6) = 2T 6/2 + 6 = 2T (3) + 6 = 16 Because both T (2) and T (3) depend on T (1) we cannot simply replace the base case n = 1 by n = 2, but we need both n = 2 and n = 3. Frits Vaandrager 16th November 2017 Lecture 9 31 / 35

62 Another example Hence we just have to replace the base case n = 1 by something else... We take it step by step: let s look at n 0 = 2 T (2) = 4 c2 lg 2 = 2c Enough to choose c 2. But now careful: T (2) = 2T 2/2 + 2 = 2T (1) + 2 = 4 T (3) = 2T 3/2 + 3 = 2T (1) + 3 = 5 T (4) = 2T 4/2 + 4 = 2T (2) + 4 = 12 T (5) = 2T 5/2 + 5 = 2T (2) + 5 = 13 T (6) = 2T 6/2 + 6 = 2T (3) + 6 = 16 Because both T (2) and T (3) depend on T (1) we cannot simply replace the base case n = 1 by n = 2, but we need both n = 2 and n = 3. For n = 3, we have T (3) = 5 c3 lg 3 = 4.8c, which also holds for c 2. Frits Vaandrager 16th November 2017 Lecture 9 31 / 35

63 Induction step T (n) = 2T n/2 + n Frits Vaandrager 16th November 2017 Lecture 9 32 / 35

64 Induction step T (n) = 2T n/2 + n 2(c n/2 lg( n/2 )) + n induction hyp. Frits Vaandrager 16th November 2017 Lecture 9 32 / 35

65 Induction step T (n) = 2T n/2 + n 2(c n/2 lg( n/2 )) + n induction hyp. cn lg n/2 + n Frits Vaandrager 16th November 2017 Lecture 9 32 / 35

66 Induction step T (n) = 2T n/2 + n 2(c n/2 lg( n/2 )) + n induction hyp. cn lg n/2 + n = cn(lg n lg 2) + n Frits Vaandrager 16th November 2017 Lecture 9 32 / 35

67 Induction step T (n) = 2T n/2 + n 2(c n/2 lg( n/2 )) + n induction hyp. cn lg n/2 + n = cn(lg n lg 2) + n = cn lg n cn + n Frits Vaandrager 16th November 2017 Lecture 9 32 / 35

68 Induction step T (n) = 2T n/2 + n 2(c n/2 lg( n/2 )) + n induction hyp. cn lg n/2 + n = cn(lg n lg 2) + n = cn lg n cn + n cn lg n Frits Vaandrager 16th November 2017 Lecture 9 32 / 35

69 Master method Provides a cookbook for solving recurrences of the form T (n) = at (n/b) + f (n), Discussed/proved in the bachelor course Complexity. Frits Vaandrager 16th November 2017 Lecture 9 33 / 35

70 Design an efficient algorithm for the following problem: 1 Input: A collection C of n banking cards 2 Output: Are there more than n/2 equivalent cards, that is, cards that correspond to the same account? 3 Only allowed operation on cards is equivalence test, which determines whether two cards are equivalent Frits Vaandrager 16th November 2017 Lecture 9 34 / 35

71 Algorithm Let C be nonempty set of cards. We define procedure P(C) which returns false if there is no card in C that is equivalent to more than n/2 cards (including itself), and returns yes if there is such a card. In the latter case P also returns a card c C and a number k > n/2 such that c is equivalent to k cards in C (including itself). 1 If n = 1 then P(C) returns yes together with card c and number 1. 2 If n > 1 then we divide C into two subsets C 1 and C 2 with sizes n/2 and n/2, respectively. 3 First we run P(C 1 ). If P(C 1 ) returns true and a card c 1 that is equivalent to k 1 cards in C 1, then we test, for each of the cards in C 2, whether or not it is equivalent to c 1 and record the number l 1 of cards in C 2 for which this is the case. If k 1 + l 1 > n/2 then P(C) returns true with card c 1 and number k 1 + l 1. 4 Otherwise we run P(C 2 ). If P(C 2 ) returns true and a card c 2 that is equivalent to k 2 cards in C 2, then we test, for each of the cards in C 1, whether it is equivalent to c 2 and record the number l 2 of cards in C 1 for which this is the case. If k 2 + l 2 > n/2 then P(C) returns true with card c 2 and number k 2 + l 2. 5 Otherwise, P(C) returns false. Frits Vaandrager 16th November 2017 Lecture 9 35 / 35