91.503 Algorithms Dr. Haim Levkowitz Fall 2007 Lecture 4 Tuesday, 25 Sep 2007 Design Patterns for Optimization Problems Greedy Algorithms 1
Greedy Algorithms 2
What is Greedy Algorithm? Similar to dynamic programming Solves optimization problem Optimal Substructure: optimal solution contains optimal solutions to subproblems 3
Greedy Strategy: At each decision point, do best locally What s best right now Choice not depend on evaluating potential future choices solving subproblems Top-down algorithmic structure W/ step, reduce problem to smaller problem 4
Greedy Choice Property: Hope: locally best globally best Greedy algorithms don t always yield optimal solution But sometimes they do We ll see a problem for which they do Then look at some general characteristics when greedy algorithms give optimal solutions 5
Examples Activity Selection Huffman Codes Minimum Spanning Tree (Ch. 23) Dijkstra Shortest Path (Ch. 24) Fractional Knapsack 6
Activity Selection 7
Example Optimization Problem: Activity Selection Problem Instance: n activities: S = {a 1, a 2,..., a n } a i needs resources [s i, f i ) start time: s i s finish time : f i! f i i Activities i, j compatible iff non-overlapping: [ s ) < i! fi [ s j! f j ) Objective: select largest possible set of mutually compatible activities 8 source: 91.503 textbook Cormen, et al.
Other possible objectives Schedule room for longest period Maximize income rental fees. 9
Example: S sorted by finish time: i 1 2 3 4 s i 1 2 4 1 f i 3 5 7 8 5 5 9 6 8 10 7 9 11 8 11 14 9 13 16 10
Maximum-size mutually compatible set: {a 1, a 3, a 6, a 8 } Not unique: also {a 2, a 5, a 7, a 9 } 11
Optimal substructure of activity selection S i j = {a k S : f i s k < f k s j } = activities that start after a i finishes and finish before a j starts Activities in S i j compatible with all activities that finish by f i, and start no earlier than s j 12
To represent entire problem add fictitious activities a 0 = [, 0) a n+1 = [, +1 ) don t care about in a 0, or +1 in a n+1 13
Then S = S 0,n+1 Range for S i j is 0 i, j n + 1. Assume that activities sorted by monotonically increasing finish time: f 0 f 1 f 2 f n < f n+1 Then i j S i j = 14
Proof If there exists a k S ij : f i s k < f k s j < f j f i < f j But i j f i f j Contradiction 15
So only need to worry about S ij with 0 i < j n + 1 All other S ij are Suppose solution to S ij includes a k Have 2 subproblems: S ik (start after a i finishes, finish before a k starts) S kj (start after a k finishes, finish before a j starts) 16
Solution to S ij is (solution to S ik ) {a k } (solution to S kj ) Since a k is in neither subproblem, and the subproblems are disjoint, solution to S = solution to S ik + 1 + solution to S kj 17
If optimal solution to S ij includes a k solutions to S ik and S kj used within this solution must be optimal as well Use usual cut-and-paste argument Let A ij = optimal solution to S ij So A ij = A ik {a k } A kj, assuming: S ij is nonempty, and we know a k 18
Recursive solution Size c[i, j ] = size of max-size subset of mutually compat. activities in S i j i j ==> S i j = c[i, j] = 0 If S i j, suppose we know that a k is in the subset. Then c[i, j ] = c[i, k] + 1 + c[k, j] But don t know which k to use, ==> 19
Why this range of k? Because S ij = {a k S : f i s k < f k s j } a k can t be a i or a j Also need to ensure that a k actually in S ij i < k < j not sufficient on its own to ensure this From here, could continue as dynamicprogramming problem But can simplify life 20
Dynamic-prog sol n greedy Theorem 16.1 (p. 374) Let S ij, and let a m be the activity in S ij with earliest finish time: f m = min { f k : a k S i j }. Then: 1. a m is used in some maximum-size subset of mutually compatible activities of S ij 2. S im =, so that choosing a m leaves S mj as the only nonempty subproblem 21
Proof (2 first) 2. Suppose there is some a k S im f i s k < f k s m < f m f k < f m Then a k S ij and has earlier finish time than f m contradicts choice of a m there is no a k S im S im = 22
Proof (1) 1. Let A ij be a maximum-size subset of mutually compatible activities in S ij. Order activites in A ij in monotonically increasing order of finish time. Let a k be the first activity in A ij. If a k = a m, done (a m is used in a maximum-size subset). Otherwise, construct A ij = A ij {a k } {a m } (replace a k by a m ). 23
Claim Activities in A ij are disjoint Proof Activities in A ij are disjoint, a k is first activity in A ij to finish, f m f k (so a m doesn t overlap anything else in A ij ) apple (claim) 24
Back to Theorem proof Since A ij = A ij and A ij is a maximum-size subset, so is A ij. apple (theorem) 25
What did Theorem 16.1 give us? Before theorem (DP) After theorem (Greedy) # subproblems in optimal sol n 2 1 (other one empty) # choices to consider j i 2 1 (only one w/earliest finish time) 26
Now can solve top down To solve problem S ij, Choose a m S ij w/ earliest finish time: the greedy choice Then solve S mj 27
Subproblems Original problem is S 0,n+1 Suppose our 1st choice is a m1 Then next subproblem is S m1,n+1 Suppose next choice is a m2 Next subproblem is S m2,n+1 And so on Each subproblem is S mi,n+1 i.e., last activities to finish Subproblems chosen have finish times that increase can consider each activity just once in monotonically increasing order of finish time 28
Recursive-Activity-Selector: R-A-S(s, f, i, j) (p. 376) s start, f finish (times as arrays) i, j starting indices of S ij Assume sorted (f mono incr.) if not, sort (O (n lg n)) 29
Greedy (recursive) The Algorithm: R-A-S(s, f, i, j) 1. m i + 1 2. while m < j and s m < f i // find first activity in S ij 3. do m m + 1 4. if m < j 5. then return {a m } R-A-S(s, f, m, j) 6. else return. Initial call: REC-ACTIVITY-SELECTOR(s, f, 0, n) Time is -- fairly obviously -- Θ(n) each activity examined exactly once 30
Demo Alg. Op: Fig. 16.1, p. 377 source: 91.503 textbook Cormen, et al. 31
Greedy (iterative) GREEDY-ACTIVITY-SELECTOR(s, f, n) A {a 1 } i 1 for m 2 to n do if s m f i then A A {a m } i m // a i is most recent addition to A return A Time: Θ(n). 32
Greedy strategy 1. Determine optimal substructure 2. Develop recursive solution 3. Prove that at any stage of recursion one of optimal choices is greedy choice always safe to make greedy choice 4. Show that all but one of subproblems resulting from greedy choice are empty 5. Develop recursive greedy algorithm 6. Convert it to iterative algorithm 33
Can streamline Typical streamlined steps: 1. Cast optimization problem as one in which Make greedy choice Leave just one subproblem to solve 2. Prove: there s always optimal solution that Makes greedy choice greedy choice always safe 3. Show: greedy choice + optimal solution to subproblem Optimal solution to problem 34
Is greedy alg optimal? No general way to tell, but two key ingredients: 1. greedy-choice property, and 2. optimal substructure 35
Greedy-choice property Globally optimal solution can be arrived at by making locally optimal (greedy) choice Dynamic programming Greedy 36
Dynamic programming: Make choice at each step Choice depends on knowing optimal solutions to subproblems Solve subproblems first Solve bottom-up 37
Greedy: Make choice at each step Make choice before solving subproblems Solve top-down Typically show greedy-choice property by what we did for activity selection: Look at globally optimal solution If it includes greedy choice, done Else, modify to include greedy choice another solution just as good Can get efficiency gains from greedy-choice property Preprocess input to put into greedy order Or, if dynamic data, use priority queue 38
Optimal substructure Just show that optimal solution to subproblem, and greedy choice optimal solution to problem 39
Greedy vs. dynamic programming The knapsack problem is good example 40
Knapsack problems 41
0-1 Knapsack (Fig. 16.2 b) Gold ingots Greedy: Maximize value/weight sub optimal sol n Value: $60 $6/lb. 10 Value: $100 $5/lb. 20 30 Value: $120 $4/lb. Total = $220 50 3: 30 => $120 2: 20 => $100 item1 item2 item3 knapsack 42
Fractional Knapsack (Fig. 16-2 c) Gold dust Greedy: Maximize value/weight optimal Value: $60 $6/lb. 10 Value: $100 $5/lb. 20 30 Value: $120 $4/lb. Total = $240 50 3: 20/30 => $80 2: 20 => $100 1: 10 => $60 item1 item2 item3 knapsack 43
FRACTIONAL- KNAPSACK(v,w,W) load 0 i 1 while load < W and i n do if w i W load then take all of item i else take (W load)/ w i of item i add what was taken to load i i + 1 Time: O(n lg n) to sort, O(n) thereafter. 44
Huffman Codes 45
Effective for compressing data 20%-90% savings Greedy alg uses table of frequency of character occurrences Fig. 16.3: table of frequencies Optimal way to represent each character as binary string Variable-length code (compare to fixed-length) Frequent short Infrequent long 46
100k chars 300k bits (fixed) 224k bits (variable) a b c d e f Freq (x 1,000) 45 13 12 16 9 5 Fixed-length 000 001 010 011 100 101 Variablelength 0 101 100 111 1101 1100 47
Prefix codes Only codes w/ no codeword as prefix of other prefix codes Can show that optimal data compression can be achieved w/prefix code no loss of generality by restricting to these 48
Prefix codes (cont.) Simplify decoding E.g., 001011101 parses uniquely as 0. 0. 101. 1101 = aabe Representation: binary tree Leaves are given characters Interpretation: path from root to character 0: go to left child 1: go to right child 49
Example: Fig. 16.4 Leaf label Character:Frequency Internal node label Sum of frequencies of leaves of its subtrees (a) fixed-length code a = 000,, f = 101 (b) optimal prefix code a = 0, b = 101,, f = 1100 50
Example: Fig. 16.4 51
Fig. 16.4 Not binary search trees Leaves need not appear in sorted order Internal nodes do not contain character keys 52
Optimal code Always represented by full binary tree Every nonleaf has 2 children (Exercise 16.3-1) Fixed-length code (16.4 (a)) not full binary tree Codewords beginning 10 None beginning 11 53
Consider only full binary trees Can say that if C is alphabet used All char frequencies positive then tree for optimal prefix code has exactly C leaves One for each letter of alphabet C - 1 internal nodes Exercise B.5-3 54
# bits required to encode file T: tree corresponding to prefix code Compute # bits required to encode a file For each character c in C f(c) = frequency of c in file d T (c) = depth of c s leaf in tree Also length of codeword for c B(T) = Σ c C f(c) d T (c) The cost of tree T 55
Huffman Codes another example 56
Constructing Huffman code: algorithm (p. 388) 57
Alg. steps demo, Fig. 16.5, p. 389 58
Correctness of Huffman s Algorithm Greedy-choice property Lemma 16.2 p. 388 Proof illustration: Fig. 16.6, p. 390 (next slide) Implication: can build optimal tree by Greedy mergers I.e., start merging at lowest frequencies Optimal-substructure property Lemma 16.3 Final correctness: Thm 16.4 (proof immediate from Lemmas 16.2 & 16.3) 59
Huffman Lemma 16.2 proof illust., Fig. 16.6, p. 390 60
Greedy Heuristic If optimization problem does not have greedy choice property, greedy approach may still be useful as heuristic in bounding optimal solution Example: minimization problem Solution Values Upper Bound (heuristic) Optimal (unknown value) Lower Bound 61