Tutorial 6-7. Dynamic Programming and Greedy

Transcription

1 Tutorial 6-7 Dynamic Programming and Greedy

2 Dynamic Programming Why DP? Natural Recursion may be expensive. For example, the Fibonacci: F(n)=F(n-1)+F(n-2) Recursive implementation memoryless : time= 1 5 n DP implementation: memorial time=o(n), with extra space O(n) or O(1) Basic Idea of DP O (( + ) ) 2 Computing each subproblem only once Find a reverse topological order for the subproblem graph Scheduling the subproblems according to the reverse topological order Record the subproblem solutions for later use

3 Examples Least Cost of Matrix Multiplication Optimal Binary Search Tree Separating Sequence of Word Coin change problem Longest Common Subsequence

4 Longest Common Subsequence(LCS) (CLRS: P350) Given two sequences x[1..m] and y[1..n], find a longest common subsequence. For example, x: A B C B D A B y: B D C A B A LCS(x,y)= BCBA or BCAB or BDAB

5 Brute-force Solution Check every subsequence of x[1..m] to see if it is also a subsequence of y[1..n]. Analysis Checking=O(n) time per subsequence 2 m subsequence of x Worst-case running time=o(n2 m ) =exponential time!

6 DP Solution Simplification Look at the length of a longest-common subsequence Extend the algorithm to find the LCS itself Strategy: Consider prefixes of x and y Define c[i,j]= LCS(x[1..i],y[1..j]) Then c[m,n]= LCS(x,y).

7 (1) DP Equation 0, if i=0 or j=0 ci [, j] = ci [ 1, j 1] + 1, if x[i]=y[j] max{ ci [ 1, j], ci [, j 1]} otherwise According to the equation, the problem can be solved by using recursive: LCS(x,y,i,j) If x[i]=y[j] then c[i,j] LCS(x,y,i-1,j-1)+1; else c[i,j] max{lcs(x,y,i-1,j),lcs(x,y,i,j-1)}; However, in the worst case, the time complexity of recursive implementation is also exponential!

8 (2) Memorization implementation After computing a solution to a subproblem, store it in a table. Subsequent calls check the table to avoid redoing work. LCS(x,y,i,j) if c[i,j]=nil If x[i]=y[j] then c[i,j] LCS(x,y,i-1,j-1)+1; else c[i,j] max{lcs(x,y,i-1,j),lcs(x,y,i,j-1)}; Time=Θ(mn), Space=Θ(mn)

9 (3)no-recursive implementation Idea: compute the table bottom-up LCS-LENGTH(X, Y) 1 m length[x] 2 n length[y] 3 for i 1 to m 4 do c[i, 0] 0 5 for j 0 to n 6 do c[0, j] 0 7 for i 1 to m 8 do for j 1 to n 9 do if xi = yj 10 then c[i, j] c[i -1, j -1] b[i, j] " " 12 else if c[i -1, j] c[i, j -1] 13 then c[i, j] c[i -1, j] 14 b[i, j] " " 15 else c[i, j] c[i, j -1] 16 b[i, j] 17 return c and b Time=Θ(mn), Space=Θ(mn)

10

11 Greedy Strategy Greedy-choice property A globally optimal solution can be arrived at by making a locally optimal (greedy) choice. In other words, when we are considering which choice to make, we make the choice that looks best in the current problem, without considering results from subproblems. Optimal substructure A problem exhibits optimal substructure if an optimal solution to the problem contains within it optimal solutions to subproblems. This property is a key ingredient of assessing the applicability of dynamic programming as well as greedy algorithms.

12 Steps: Cast the optimization problem as one in which we make a choice and are left with one subproblem to solve. Prove that there is always an optimal solution to the original problem that makes the greedy choice, so that the greedy choice is always safe. Demonstrate that, having made the greedy choice, what remains is a subproblem with the property that if we combine an optimal solution to the subproblem with the greedy choice we have made, we arrive at an optimal solution to the original problem.

13 Examples: MST Problem Prim s Algortithm Kruskal s Algorithm Single-Source Shortest Path Problem Dijstra s Algorithm Huffman Codes

14 Interval Scheduling Problem Also known as activity selection problem (CLRS P371) Problem description A resource, i.e. a lecture room, a computer A set of requests {1,2,,n}, the ith request corresponds to an interval of time starting at s(i), finishing at f(i) Two requests i and j are compatible if the request intervals do not overlap The goal is to accept as large a compatible subset as possible. Compatible sets of maximum size will be called optimal.

15 Designing a Greedy Algorithm (1) We can make greedy decision based on different rules, but they do not always give good solutions. (1) earliest starting time: counter example (a) (2) smallest interval : counter example (b) (3) fewest conflicts : counter example (c) (a) (b) (c)

16 Designing a Greedy Algorithm (2) A greedy rule does lead to the optimal solution: The smallest finish time Idea: we should accept the request that finished first, that is, the request i for which f(i) is as small as possible We insure that our resource becomes free as soon as possible while still satisfying one request, in this way, we can maximize the time left to satisfy other requests.

17 Proof the Optimality of Greedy Strategy Let A be the solution from the greedy strategy Let O be the optimal solution In order to prove A is optimal, we should simply show that A = O

18 Let A={i 1,i 2,,i k }, O={j 1,j 2,,j m }, our goal is to prove k=m Assume requests in A and O are ordered in the left-to-right order of the corresponding intervals Claim 1: For all indices r k, we have f(i r ) f(j r ). Proof by induction Base: for r=1, the statement is true; Hypothesis: assume the statement is true for r-1 Induction: in order to prove f(i r ) f(j r ), we show that f(i r ) f(j r ) is impossible: if so, like the following figure, j r will be finished before i r, according to the greedy rule, i r should be selected in A. i r-1 i r j r-1 j r Cliam 2: The greedy algorithm returns an optimal solution Proof by contradiction If A is not optimal, there must be m>k. According to claim 1, f(i k ) f(j k ). Since m>k, there is a request j k+1 in O, it starts after j k ends, hence after i k ends. So j k+1 is compatible to all elements in A and it can be added into A. -a contradiction

19 Analyzing the Greedy Algorithm The greedy algorithm runs in O(nlgn), which equals to the cost of sorting the n requests in order of finishing time.

20 Huffman Codes (CLRS P385) Character-coding problem Prefix Codes: codes in which no codeword is also a prefix of some other codeword

21 Constructing a Huffman Code Huffman invented a greedy algorithm that constructs an optimal prefix code called a Huffman code. HUFFMAN(C) 1 n C // C is a set of characters, each character c has frequency f(c) 2 Q C // Q ia a min-priority queue 3 for i=1 to n -1 4 do allocate a new node z 5 left[z] x EXTRACT-MIN (Q) 6 right[z] y EXTRACT-MIN (Q) 7 f [z] f [x] + f [y] 8 INSERT(Q, z) 9 return EXTRACT-MIN(Q) Return the root of the tree.

22

23 Analysis Assumes Q is implemented as a binary min-heap. For a set C of n characters, the initialization of Q in line 2 can be performed in O (n) time. The for loop in lines 3-8 is executed exactly n - 1 times, and since each heap operation requires time O (lg n), the loop contributes O (n lg n) to the running time. Thus, the total running time of HUFFMAN on a set of n characters is O (n lg n). Correctness of Huffman s algorithm (Please refer to CLRS P )