Design and Analysis of Algorithms 演算法設計與分析. Lecture 7 April 6, 2016 洪國寶

Transcription

1 Design and Analysis of Algorithms 演算法設計與分析 Lecture 7 April 6, 2016 洪國寶 1

2 Course information (5/5) Grading (Tentative) Homework 25% (You may collaborate when solving the homework, however when writing up the solutions you must do so on your own. No typed or printed assignments.) Midterm exam 25% (Open book and notes) April 20, 2016 Final exam 25% (Open book and notes) Class participation 25% 2

3 Outline of the course (1/1) Introduction (1-4) Midterm exam Data structures (10-14) Apr. 20, 2016 Dynamic programming (15) Greedy methods (16) Amortized analysis (17) Advanced data structures (6, 19-21) Graph algorithms (22-25) NP-completeness (34-35) Other topics (5, 31) 3

4 Outline Review Dynamic programming (Cont.) Greedy method Activity-selection algorithm Basic elements of greedy approach Examples where greedy approach does not work Optimal merge pattern 4

5 Review: Development of A Dynamic-Programming Algorithm 1. Characterize the structure of an optimal solution 2. Recursively define the value of an optimal solution 3. Compute the value of an optimal solution in a bottom-up fashion 4. Construct an optimal solution from computed information 5

6 Matrix chain multiplication Given a sequence (chain) <A 1, A 2,, A n > of n matrices to be multiplied, compute the product A 1 A 2 A n in a way that minimizes the number of scalar multiplications. Every way of multiplying the matrices corresponds to a parenthesization. Impractical to check all possible parenthesizations P 1 n) ( n 1 k 1 P( k) P( n k) if if n 1 n 2 6

7 Review: Elements of Dynamic Programming Optimal substructure: an optimal solution to the problem contains within it optimal solutions to subproblems (for DP to be applicable) Overlapping subproblems: the space of subproblems must be small (for algorithm to be efficient) 7

8 Common Pattern in Discovering Optimal Substructure Show a solution to the problem consists of making a choice. Making the choice leaves one or more subproblems to be solved. Suppose that for a given problem, the choice that leads to an optimal solution is available. Given this optimal choice, determine which subproblems ensue and how to best characterize the resulting space of subproblems Show that the solutions to the subproblems used within the optimal solution to the problem must themselves be optimal by using a cut-and-paste technique and prove by contradiction 8

9 Review: Memoization A variation of dynamic programming that often offers the efficiency of the usual dynamic-programming approach while maintaining a top-down strategy Memoize the natural, but inefficient, recursive algorithm Maintain a table with subproblem solutions, but the control structure for filling in the table is more like the recursive algorithm 9

10 Review: LCS Algorithm if X = m, Y = n, then there are 2 m subsequences of X; we must compare each with Y (n comparisons) So the running time of the brute-force algorithm is O(n 2 m ) Notice that the LCS problem has optimal substructure: solutions of subproblems are parts of the final solution. Subproblems: find LCS of pairs of prefixes of X and Y 10

11 Review: LCS Algorithm Running Time LCS algorithm calculates the values of each entry of the array c[m,n] The running time is O(mn) since each c[i,j] is calculated in constant time, and there are mn elements in the array 11

12 Review: Optimal Polygon Triangulation Input: a convex polygon P=(v 0, v 1,, v n-1 ) Output: an optimal triangulation 12

13 Optimal Polygon Triangulation vs. Matrix-chain multiplication Optimal Polygon Triangulation 0 if i = j t[i,j] = min { t[ i, k] t[ k 1, j] w( vi 1v jvk )} i k j Matrix-chain multiplication if i < j m[ i, j] 0 min 1 k j { m[ i, k] m[ k 1, j] p i 1 p k p j if if i i j j 13

14 Dynamic Programming (TSP) Traveling salesperson problem (revisit) - Optimal substructure? (subproblems) g(i,s) = min{c ij + g(j,s-{j})} j S = the length of a shortest path starting at vertex i, going through all vertices in S, and terminating at vertex 1 - Overlapping subproblems? N = g(i,s) = (n-1)2 n-2 14

15 Outline Review Dynamic programming (Cont.) Greedy method Activity-selection algorithm Basic elements of greedy approach Examples where greedy approach does not work Optimal merge pattern 15

16 Algorithm design techniques So far, we ve looked at the following design techniques: - induction (incremental approach) - divide and conquer - augmenting data structures - dynamic programming Coming up: greedy method 16

17 Dynamic Programming VS. Greedy Algorithms Dynamic programming uses optimal substructure in a bottom-up fashion First find optimal solutions to subproblems and, having solved the subproblems, we find an optimal solution to the problem Greedy algorithms use optimal substructure in a top-down fashion First make a choice the choice that looks best at the time and then solving a resulting subproblem 17

18 Greedy Algorithms A greedy algorithm always makes the choice that looks best at the moment The hope: a locally optimal choice will lead to a globally optimal solution For some problems, it works Dynamic programming can be overkill; greedy algorithms tend to be easier to code 18

19 Example: Activity-Selection Formally: Given a set S of n activities s i = start time of activity i f i = finish time of activity i Find max-size subset A of compatible activities: for all i,j A, [s i, f i ) and [s j, f j ) do not overlap

20 Activity Selection: Optimal Substructure Let k be the minimum activity in A (i.e., the one with the earliest finish time). Then A - {k} is an optimal solution to S = {i S: s i f k } In words: once activity k is selected, the problem reduces to finding an optimal solution for activityselection over activities in S compatible with k Proof: if we could find optimal solution B to S with B > A - {k}, Then B U {k} is compatible And B U {k} > A 20

21 Activity Selection: Greedy Choice Property Dynamic programming? Memoize? Activity selection problem also exhibits the greedy choice property: Locally optimal choice globally optimal sol n Thm 16.1: if S is an activity selection problem sorted by finish time, then optimal solution A S such that {1} A Sketch of proof: if optimal solution B that does not contain {1}, can always replace first activity in B with {1} (Why?). Same number of activities, thus optimal. 21

22 Activity Selection: A Greedy Algorithm Intuition is simple: Always pick the activity with the earliest finish time available at the time GAS(S) 1 if S=NIL then return NIL 2 else return {k} U GAS(S ) where k is the activity in S with smallest f, and S = {i S: s i f k } 22

23 GAS(S) Activity Selection: A Greedy Algorithm 1 if S=NIL then return NIL 2 else return {k} U GAS(S ) where k is the activity in S with smallest f, and S = {i S: s i f k } Proof of correctness: use blackboard Use blackboard 23

24 Activity Selection: A LISP program GAS(S) 1 if S=NIL then return NIL 2 else return {k} U GAS(S ) where k is the activity in S with smallest f, and S = {i S: s i f k } (defun gas (l) (print l) (cond ((null l) nil) (T (cons (car l) (gas (filter (nth 1 (car l)) (cdr l))))))) (defun filter (s l) (cond ((null l) nil) ((> s (nth 0 (car l))) (filter s (cdr l))) (T (cons (car l) (filter s (cdr l)))))) 24

25 Activity Selection: A LISP program (cont.) (defvar *activity* '((1 4) (3 5) (0 6) (5 7) (3 8) (5 9) (6 10) (8 11) (8 12) (2 13) (12 14))) #<BUFFERED FILE-STREAM CHARACTER [3]> *activity* ((1 4) (3 5) (0 6) (5 7) (3 8) (5 9) (6 10) (8 11) (8 12) (2 13) (12 14)) [4]> (gas *activity*) ((1 4) (3 5) (0 6) (5 7) (3 8) (5 9) (6 10) (8 11) (8 12) (2 13) (12 14)) ((5 7) (5 9) (6 10) (8 11) (8 12) (12 14)) ((8 11) (8 12) (12 14)) ((12 14)) NIL ((1 4) (5 7) (8 11) (12 14)) The S for each iteration (print l) [5]> (dribble) 25

26 Activity Selection: A Greedy Algorithm If we sort the activities by finish time then the algorithm is simple (iterative instead of recursive): Sort the activities by finish time Schedule the first activity Then schedule the next activity in sorted list which starts after previous activity finishes Repeat until no more activities Note: we don t have to construct S explicitly. Why? 26

27 Greedy-Activity-Selector Assume (wlog) that f 1 f 2 f n Time complexity: O(n) Exercise: Proof of correctness by loop invariant. 27

28 28

29 Activity Selection 29

30 Other greedy choices Greedy-Activity-Selector always pick the activity with the earliest finish time available at the time (smallest f) Some other greedy choices: Largest f Largest/Smallest s Largest/Smallest (f-s) Fewest overlapping Exercise: Which of these criteria result in optimal solutions? 30

31 A Variation of the Problem Instead of maximizing the number of activities we want to schedule, we want to maximize the total time the resource is in use. None of the obvious greedy choices would work: Choose the activity that starts earliest/latest Choose the activity that finishes earliest/latest Choose the longest activity Exercise: Design an efficient algorithm for this variation of activity selection problem. 31

32 Outline Review Dynamic programming (Cont.) Greedy method Activity-selection algorithm Basic elements of greedy approach Examples where greedy approach does not work Optimal merge pattern 32

33 Elements of Greedy Strategy Greedy choice property: An optimal solution can be obtained by making choices that seem best at the time, without considering their implications for solutions to subproblems. Optimal substructure: An optimal solution can be obtained by augmenting the partial solution constructed so far with an optimal solution of the remaining subproblem. 33

34 Steps in designing greedy algorithms 1. Cast the optimization problem as one in which we make a choice and are left with one subproblem to solve. 2. Prove the greedy choice property. 3. Demonstrate optimal substructure. 34

35 Outline Review Dynamic programming (Cont.) Longest Common Subsequence (LCS) Greedy method Activity-selection algorithm Basic elements of greedy approach Examples where greedy approach does not work Optimal merge pattern 35

36 Examples where greedy approach does not work Traveling salesman problem: nearest neighbor, closest pair Matrix-chain multiplication: multiply the two matrices with lowest cost first Knapsack problem: largest values, largest value per unit weight 36

37 Traveling salesman problem Correctness is not obvious (2/7) (nearest neighbor) 37

38 Traveling salesman problem Correctness is not obvious (3/7) 38

39 Matrix-chain multiplication Greedy choice: multiply the two matrices with lowest cost first Example 1: <A 1, A 2, A 3 > (10*100, 100*5, 5*50) Example 2: <A 1, A 2, A 3, A 4, A 5, A 6 > (30*35, 35*15, 15*5, 5*10, 10*20, 20*25) 39

40 Matrix-chain multiplication Example 1: <A 1, A 2, A 3 > (10*100, 100*5, 5*50) ((A 1 A 2 )A 3 ) 10*100*5 + 10*5*50 = = 7500 (A 1 (A 2 A 3 )) 100*5* *100*50 = =

41 Matrix-chain multiplication Example 2: <A 1, A 2, A 3, A 4, A 5, A 6 > (30*35, 35*15, 15*5, 5*10, 10*20, 20*25) A 3 A 4 15*5*10 = 750 Figure 15.3 ((A 1 (A 2 A 3 ))((A 4 A 5 ) A 6 )) 41

42 42

43 Knapsack problem Given some items, pack the knapsack to get the maximum total value. Each item has some weight and some value. Total weight that we can carry is no more than some fixed number W. So we must consider weights of items as well as their value. Item # Weight Value

44 Knapsack Problem A thief breaks into a jewelry store carrying a knapsack. Given n items S = {item 1,,item n }, each having a weight w i and value v i, which items should the thief put into the knapsack with capacity W to obtain the maximum value? 44

45 Knapsack problem There are two versions of the problem: (1) 0-1 knapsack problem and (2) Fractional knapsack problem (1) Items are indivisible; you either take an item or not. Solved with dynamic programming (2) Items are divisible: you can take any fraction of an item. Solved with a greedy algorithm. 45

46 0-1 Knapsack Problem This problem requires a subset A of S to be determined such that i item A i v is maximized subject to The naïve approach is to generate all possible subset of S and find the maximum value, requiring 2 n time. item A i w i W 46

47 Does Greedy Approach Work? Strategy 1: Steal the items with the largest values. E.g. w=[25, 10, 10], v=[$10, $9, $9], W=30. Value is 10, although the optimal value is 18. w 1 =25 w 2 =10 w 3 =10 v 1 = $10 v 2 = $9 v 3 = $9 47

48 Does Greedy Approach Work? Strategy 2: Steal the items with the largest value per unit weight. E.g. w=[5, 20,10], v=[$50, $140, $60], W=30. Value is 190, although optimal would be 200. w 1 =5 w 2 =20 w 3 =10 v 1 =$50 v 2 =$140 v 3 =$60 48

49 NP-hard Problems (Knapsack, Traveling Salesman,...) The two approaches above cannot yield the optimal result for 0-1 knapsack. This problem is NP-hard even when all items are the same kind, that is, item i described by w i only (v i =w i ). Observation: Greedy approach to the fractional knapsack problem yields the optimal solution. E.g. (see 2nd example) (5/10)*60 = 220 where 5 is the remaining capacity of the knapsack 49

50 Dynamic Programming Approach for the Knapsack Problem We can find optimal solution for 0-1 knapsack problem by DP, but not in polynomial-time. Let A be an optimal subset with items from {item 1,,item i } only. There are two cases: 1) A contains item i Then the total value for A is equal to v i plus the optimal value obtained from the first i-1 items, where the total weight cannot exceed W w i 2) A does not contain item i Then the total value for A is equal to that of the optimal subset chosen from the first i-1 items (with total weight cannot exceed W). Q: What are the subproblems? 50

51 Dynamic Programming Approach for the Knapsack Problem A[i][w]= max(a[i-1][w], v i +A[i-1][w-w i ]) if w>w i A[i-1][w] if w<w i The maximum value is A[n][W] Running time is O(nW), not polynomial in n Q: WHY? n W 51

52 Fractional knapsack problem Greedy approach: taking items in order of greatest value per pound Optimal for the fractional version (why?), but not for the 0-1 version 52

53 Outline Review Dynamic programming (Cont.) Longest Common Subsequence (LCS) Greedy method Activity-selection algorithm Basic elements of greedy approach Examples where greedy approach does not work Optimal merge pattern 53

54 Optimal merge pattern 54

55 Optimal merge pattern 55

56 Huffman codes Compact Text Encoding Code that can be decoded Huffman encoding 56

57 Compact Text Encoding The goal is to develop a code that represents a given text as compactly as possible. A standard encoding is ASCII, which represents every character using 7 bits: An English sentence (A) (n) ( ) (E) (n) (g) (l) (i) (s) (h) ( ) (s) (e) (n) (t) (e) (n) (c) (e) This requires 133 bits 17 bytes 57

58 Compact Text Encoding (Cont.) Of course, this is wasteful because we can encode 12 characters in 4 bits: space = 0000 A = 0001 E = 0010 c = 0011 e = 0100 g = 0101 h = 0110 i = 0111 l = 1000 n = 1001 s = 1010 t = 1011 Then we encode the phrase as 0001 (A) 1001 (n) 0000 ( ) 0010 (E) 1001 (n) 0101 (g) 1000 (l) 0111 (i) 1010 (s) 0110 (h) 0000 ( ) 1010 (s) 0100 (e) 1001 (n) 1011 (t) 0100 (e) 1001 (n) 0011 (c) 0100 (e) This requires 76 bits 10 bytes 58

59 Compact Text Encoding (Cont.) An even better code is given by the following encoding: space = 000 A = 0010 E = 0011 s = 010 c = 0110 g = 0111 h = 1000 i = 1001 l = 1010 t = 1011 e = 110 n = 111 Then we encode the phrase as 0010 (A) 111 (n) 000 ( ) 0011 (E) 111 (n) 0111 (g) 1010 (l) 1001 (i) 010 (s) 1000 (h) 000 ( ) 010 (s) 110 (e) 111 (n) 1011 (t) 110 (e) 111 (n) 0110 (c) 110 (e) This requires 65 bits 9 bytes 59

60 Code that can be decoded Fixed-length codes: Every character is encoded using the same number of bits. To determine the boundaries between characters, we form groups of w bits, where w is the length of a character. Examples: ASCII Our first improved code Prefix codes: No character is the prefix of another character. Examples: Fixed-length codes Huffman codes 60

61 Why Prefix Codes? Consider a code that is not a prefix code: a= 01 m = 10 n = 111 o = 0 r = 11 s = 1 t = 0011 Now you send a fan-letter to you favorite movie star. One of the sentences is You are a star. You encode star as Your idol receives the letter and decodes the text using your coding table: = = moron Prefix codes are unambiguous. (See next slide) 61

62 prefix codes are unambiguous It suffices to show that the first character can be decoded unambiguously. We then remove this character and are left with the problem of decoding the first character of the remaining text, and so on until the whole text has been decoded. Why Are Prefix Codes Unambiguous? c Assume that there are two characters c and c' that could potentially be the first characters in the text. Assume that the encodings are x 0 x 1 x k and y 0 y 2 y l. Assume further that k l. c c' Since both c and c' can occur at the beginning of the text, we have x i = y i, for 0 i k; that is, x 0 x 1 x k is a prefix of y 0 y 2 y l, a contradiction. 62

63 Representing a Prefix-Code Dictionary In the example: space = 000 A = 0010 E = 0011 s = 010 c = 0110 g = 0111 h = 1000 i = 1001 l = 1010 t = 1011 e = 110 n = spc 0 A s e n E c g h i l t 63

64 Figure 16.3/ Figure

65 65

66 Huffman code 66

67 The Cost of Huffman Code Let C be the set of characters in the text to be encoded, and let f(c) be the frequency of character c. Let d T (c) be the depth of node (character) c in the tree representing the code. Then is the number of bits required to encode the text using the code represented by tree T. We call B(T) the cost of tree T. Observation: In a tree T representing an optimal prefix code, every internal node has two children. 67

68 Greedy Choice Lemma: There exists an optimal prefix code such that the two characters with smallest frequency are siblings and have maximal depth in T. 68

69 Proof: Let x and y be two such characters, and let T be a tree representing an optimal prefix code. Greedy Choice T Let a and b be two sibling leaves of maximal depth in T. x y Assume w.l.o.g. that f(x) f(y) and f(a) f(b). T' a b This implies that f(x) f(a) and f(y) f(b). Let T' be the tree obtained by exchanging a and x and b and y. (Continued in the next slide.) a x y b 69

70 Greedy Choice The cost difference between trees T and T' is T T' x y a b a b x y 70

71 Optimal Substructure After joining two nodes x and y by making them children of a new node z, the algorithm treats z as a leaf with frequency f(z) = f(x) + f(y). Let C' be the character set in which x and y are replaced by the single character z with frequency f(z) = f(x) + f(y), and let T' be an optimal tree for C'. Let T be the tree obtained from T' by making x and y children of z. We observe the following relationship between B(T) and B(T'): B(T) = B(T') + f(x) + f(y) (Continued in the next slide.) 71

72 72

73 Lemma: If T' is optimal for C', then T is optimal for C. Assume the contrary. Then there exists a better tree T'' for C. Also, there exists a tree T''' at least as good as T'' for C where x and y are sibling leaves of maximal depth. The removal of x and y from T''' turns their parent into a leaf; we can associate this leaf with z. The cost of the resulting tree is B(T''') f(x) f(y) < B(T) f(x) f(y) = B(T'). This contradicts the optimality of B(T'). Hence, T must be optimal for C. 73

74 74

75 Huffman code (Remarks) Assume that the string is generated by a memoryless source, regardless the past, the next character in the string is c with probability f(c). Then Huffman is optimal. Can we do better? 75

76 Huffman code (Remarks) Huffman encodes fixed length blocks. What if we vary them? Huffman uses one encoding throughout a life. What if characteristics change? What if data has structure? For examples: raster images, video Huffman is lossless. Necessary? LZW, MPEG, etc. 76

77 Huffman code (Implementation) Time complexity and data structure: Let S be the set of n weights (nodes). Constructing a Huffamn code based on greedy strategy can be described as following Repeat until S =1 Find two min nodes x and y from S and remove them from S Construct a new node z with weight w(z)=w(x)+w(y) and insert z into S 77

78 Huffman code (Implementation) Why data structures are important? An algorithm for constructing Huffman code (Tree): Repeat until S =1 Find two min nodes x and y from S and remove them from S Construct a new node z with weight w(z)=w(x)+w(y) and insert z into S The time complexity of the algorithm depends on how S is implemented. Data structure for S each iteration total linked list O(n) O(1) O(n^2) Sorted array O(1) O(n) O(n^2)? O(log n) O(log n) O(n log n) 78

79 We will cover heap in Lecture 9 79

80 Greedy method: Recap Greedy algorithms are efficient algorithms for optimization problems that exhibit two properties: Greedy choice property: An optimal solution can be obtained by making locally optimal choices. Optimal substructure: An optimal solution contains within it optimal solutions to smaller subproblems. If only optimal substructure is present, dynamic programming may be a viable approach; that is, the greedy choice property is what allows us to obtain faster algorithms than what can be obtained using dynamic programming. 80

81 Questions? 81