16 Greedy Algorithms

Transcription

1 16 Greedy Algorithms Optimization algorithms typically go through a sequence of steps, with a set of choices at each For many optimization problems, using dynamic programming to determine the best choices is overkill; simpler, more efcient algorithms will do A greedy algorithm always makes the choice that looks best at the moment That is, it makes a locally optimal choice in the hope that this choice will lead to a globally optimal solution MAT AADS, Fall Oct An activity-selection problem Suppose we have a set =,,, of proposed activities that wish to use a resource (e.g., a lecture hall), which can serve only one activity at a time Each activity has a start time and a nish time, where < If selected, activity takes place during the halfopen time interval [, ) MAT AADS, Fall Oct

2 Activities and are compatible if the intervals [, ) and [, ) do not overlap I.e., and are compatible if or We wish to select a maximum-size subset of mutually compatible activities We assume that the activities are sorted in monotonically increasing order of nish time: MAT AADS, Fall Oct Consider, e.g., the following set of activities: For this example, the subset,, consists of mutually compatible activities It is not a maximum subset, however, since the subset,,, is larger In fact, it is a largest subset of mutually compatible activities; another largest subset is,,, MAT AADS, Fall Oct

3 The optimal substructure of the activityselection problem Let be the set of activities that start after nishes and that nish before starts We wish to nd a maximum set of mutually compatible activities in Suppose that such a maximum set is, which includes some activity By including in an optimal solution, we are left with two subproblems: nding mutually compatible activities in the set and nding mutually compatible activities in the set MAT AADS, Fall Oct Let = and =, so that contains the activities in that nish before starts and contains the activities in that start after nishes Thus, =, and so the maximum-size set in consists of = + +1activities The usual cut-and-paste argument shows that the optimal solution must also include optimal solutions for and MAT AADS, Fall Oct

4 This suggests that we might solve the activityselection problem by dynamic programming If we denote the size of an optimal solution for the set by [, ], then we would have the recurrence, =, +, +1 Of course, if we did not know that an optimal solution for the set includes activity, we would have to examine all activities in to nd which one to choose, so that 0 if, = max, =, +, +1 if MAT AADS, Fall Oct Making the greedy choice For the activity-selection problem, we need consider only the greedy choice We choose an activity that leaves the resource available for as many other activities as possible Now, of the activities we end up choosing, one of them must be the rst one to nish Choose the activity in with the earliest nish time, since that leaves the resource available for as many of the activities that follow it as possible Activities are sorted in monotonically increasing order by nish time; greedy choice is activity MAT AADS, Fall Oct

5 If we make the greedy choice, we only have to nd activities that start after nishes < and is the earliest nish time of any activity no activity can have a nish time Thus, all activities that are compatible with activity must start after nishes Let = : be the set of activities that start after nishes Optimal substructure: if is in the optimal solution, then an optimal solution to the original problem consists of and all the activities in an optimal solution to the subproblem MAT AADS, Fall Oct Theorem 16.1 Consider any nonempty subproblem, and let be an activity in with the earliest nish time. Then is included in some maximum-size subset of mutually compatible activities of. Proof Let be a max-size subset of mutually compatible activities in, and let be the activity in with the earliest nish time. If =, we are done, since is in a max-size subset of mutually compatible activities of. If, let the set =. The activities in are disjoint because the activities in are disjoint, is the rst activity in to nish, and. Since =, we conclude that is a maximum-size subset of mutually compatible activities of and includes. MAT AADS, Fall Oct

6 We can repeatedly choose the activity that nishes rst, keep only the activities compatible with this activity, and repeat until no activities remain Moreover, because we always choose the activity with the earliest nish time, the nish times of the activities we choose must strictly increase We can consider each activity just once overall, in monotonically increasing order of nish times MAT AADS, Fall Oct A recursive greedy algorithm RECURSIVE-ACTIVITY-SELECTOR(,,, ) while and [] <[] // nd the rst // activity in to nish if 5. return RECURSIVE-ACTIVITY- SELECTOR(,,, ) 6. else return MAT AADS, Fall Oct

7 MAT AADS, Fall Oct An iterative greedy algorithm GREEDY-ACTIVITY-SELECTOR(, ) for 2to 5. if [ [] return MAT AADS, Fall Oct

8 The set returned by the call GREEDY-ACTIVITY-SELECTOR(, ) is precisely the set returned by the call RECURSIVE-ACTIVITY-SELECTOR(,,, ) Both the recursive version and the iterative algorithm schedule a set of activities in () time, assuming that the activities were already sorted initially by their nish times MAT AADS, Fall Oct Huffman codes Huffman codes compress data very effectively savings of 20% to 90% are typical, depending on the characteristics of the data being compressed We consider the data to be a sequence of characters Huffman s greedy algorithm uses a table giving how often each character occurs (i.e., its frequency) to build up an optimal way of representing each character as a binary string MAT AADS, Fall Oct

9 We have a 100,000-character data le that we wish to store compactly We observe that the characters in the le occur with the frequencies given in the table above That is, only 6 different characters appear, and the character occurs 45,000 times Here, we consider the problem of designing a binary character code (or code for short) in which each character is represented by a unique binary string, which we call a codeword MAT AADS, Fall Oct Using a xed-length code, requires 3 bits to represent 6 characters: = 000, = 001,, = 101 We now need 300,000 bits to code the entire le A variable-length code gives frequent characters short codewords and infrequent characters long codewords Here the 1-bit string 0 represents, and the 4-bit string 1100 represents This code requires ( )1,000 = 224,000bits (savings 25%) MAT AADS, Fall Oct

10 Prex codes We consider only codes in which no codeword is also a prex of some other codeword A prex code can always achieve the optimal data compression among any character code, and so we can restrict our attention to prex codes Encoding is always simple for any binary character code; we just concatenate the codewords representing each character of the le E.g., with the variable-length prex code, we code the 3-character le as = MAT AADS, Fall Oct Prex codes simplify decoding No codeword is a prex of any other, the codeword that begins an encoded le is unambiguous We can simply identify the initial codeword, translate it back to the original character, and repeat the decoding process on the remainder of the encoded le In our example, the string parses uniquely as , which decodes to MAT AADS, Fall Oct

11 The decoding process needs a convenient representation for the prex code so that we can easily pick off the initial codeword A binary tree whose leaves are the given characters provides one such representation We interpret the binary codeword for a character as the simple path from the root to that character, where 0 means go to the left child and 1 means go to the right child Note that the trees are not BSTs the leaves need not appear in sorted order and internal nodes do not contain character keys MAT AADS, Fall Oct The trees corresponding to the xed-length code = 000,, = 101 and the optimal prex code = 0, = 101,, = 1100 MAT AADS, Fall Oct

12 An optimal code for a le is always represented by a full binary tree, in which every nonleaf node has two children The xed-length code in our example is not optimal since its tree is not a full binary tree: it contains codewords beginning 10, but none beginning 11 Since we can now restrict our attention to full binary trees, we can say that if is the alphabet from which the characters are drawn and all character frequencies are positive, then the tree for an optimal prex code has exactly leaves, one for each letter of the alphabet, and exactly 1internal nodes MAT AADS, Fall Oct Given a tree corresponding to a prex code, we can easily compute the number of bits required to encode a le For each character, let the attribute. denote the frequency of and let () denote the depth of s leaf () is also the length of the codeword for Number of bits required to encode a le is thus =. () which we dene as the cost of the tree MAT AADS, Fall Oct

13 Constructing a Huffman code Let be a set of characters and each character be an object with an attribute. freq The algorithm builds the tree corresponding to the optimal code bottom-up It begins with leaves and performs 1 merging operations to create the nal tree We use a min-priority queue, keyed on freq, to identify the two least-frequent objects to merge The result is a new object whose frequency is the sum of the frequencies of the two objects MAT AADS, Fall Oct HUFFMAN() for 1to 1 4. allocate a new node 5.. left EXTRACT-MIN() 6.. right EXTRACT-MIN() 7.. freq. freq +. freq 8. INSERT(, ) 9. return EXTRACT-MIN() // return the root of the tree MAT AADS, Fall Oct

14 MAT AADS, Fall Oct To analyze the running time of HUFFMAN, let be implemented as a binary min-heap For a set of characters, we can initialize (line 2) in () time using the BUILD-MIN-HEAP The for loop executes exactly 1times, and since each heap operation requires time (lg ), the loop contributes (lg ), to the running time Thus, the total running time of HUFFMAN on a set of characters is (lg ) Running time reduces to ( lg lg ) by replacing the binary min-heap with a van Emde Boas tree MAT AADS, Fall Oct

15 Correctness of Huffman s algorithm The problem of determining an optimal prex code exhibits the greedy-choice and optimalsubstructure properties Lemma 16.2 Let be an alphabet in which each character has frequency. freq. Let and be two characters in having the lowest frequencies. Then there exists an optimal prex code for in which the codewords for and have the same length and differ only in the last bit. MAT AADS, Fall Oct Lemma 16.3 Let,. freq,, and be as in Lemma Let =,. Dene freq for as for, except that. freq =. freq +. freq. Let be any tree representing an optimal prex code for the alphabet. Then the tree, obtained from by replacing the leaf node for with an internal node having and as children, represents an optimal prex code for. Theorem 16.4 Procedure HUFFMAN produces an optimal prex code. MAT AADS, Fall Oct

16 17 Amortized Analysis We average the time required to perform a sequence of data-structure operations over all the operations performed Thus, we can show that the average cost of an operation is small, even if a single operation within the sequence might be expensive Amortized analysis differs from average-case analysis in that probability is not involved Amortized analysis guarantees the average performance of each operation in the worst case MAT AADS, Fall Oct Aggregate analysis We show that for all, a sequence of operations takes worst-case time () in total In the worst case, average cost, or amortized cost, per operation is therefore ()/ This amortized cost applies to each operation, even when there are several types of operations in the sequence The other two methods we shall study, may assign different amortized costs to different types of operations MAT AADS, Fall Oct

17 Stack operations The fundamental stack operations PUSH(, ) and POP() each takes (1) time Let s consider the cost of each operation to be 1 The total cost of a sequence of PUSH and POP operations is therefore, and the actual running time for operations is therefore () Now we add the stack operation MULTIPOP(, ), which removes the top objects of stack, popping the entire stack if the stack contains fewer than objects MAT AADS, Fall Oct The operation STACK-EMPTY returns TRUEif there are no objects currently on the stack, and FALSE otherwise MULTIPOP(, ) 1. while not STACK-EMPTY() and >0 2. POP 3. 1 The total cost of MULTIPOP is min(, ), and the actual running time is a linear function of this MAT AADS, Fall Oct

18 Let us analyze a sequence of PUSH, POP, and MULTIPOP operations on an initially empty stack The worst-case cost of a MULTIPOP operation is (), since the stack size is at most The worst-case time of any stack operation is (), and hence a sequence of operations costs ( ) This analysis is correct, but the result is not tight Using aggregate analysis, we can obtain a better upper bound that considers the entire sequence of operations MAT AADS, Fall Oct We can pop each object from the stack at most once for each time we have pushed it onto the stack The number of times that POP can be called on a nonempty stack, including calls within MULTIPOP, is at most the number of PUSH operations, which is at most Any sequence of PUSH, POP, and MULTIPOP operations takes a total of () time The average cost of an operation () = (1) MAT AADS, Fall Oct

19 Incrementing a binary counter Consider the problem of implementing a -bit binary counter that counts upward from 0 We use an array [0.. 1] of bits, where. =, as the counter A binary number that is stored in the counter has its lowest-order bit in [0] and its highestorder bit in [ 1], so that = [] 2 MAT AADS, Fall Oct Initially, = 0: = 0for = 0,1,, 1 To add 1 (modulo 2 ) to the value in the counter, we use the following procedure INCREMENT() while <. and = if < MAT AADS, Fall Oct

20 MAT AADS, Fall Oct At the start of each iteration of the while loop (lines 2 4), we wish to add a 1 into position If =1, then adding 1 ips the bit to 0 in position and yields a carry of 1, to be added into position +1on the next iteration of the loop Otherwise, the loop ends, and then, if <, we know that =0, so that line 6 adds a 1 into position, ipping the 0 to a 1 The cost of each INCREMENT operation is linear in the number of bits ipped MAT AADS, Fall Oct

21 A cursory analysis yields a bound that is correct but not tight Single execution of INCREMENT takes time () in the worst case, when array contains all 1s Thus, a sequence of INCREMENT operations on an initially zero counter takes time () Tighten the analysis to yield a worst-case cost of () by observing that not all bits ip each time INCREMENT is called [0] does ip each time INCREMENT is called MAT AADS, Fall Oct The next bit up, [1], ips only every other time a sequence of INCREMENT operations on zero counter causes [1] to ip 2 times Similarly, bit [2] ips only every fourth time, or 4 times in a sequence of INCREMENT operations In general, bit [] ips 2 times in a sequence of INCREMENT operations on an initially zero counter For, bit [] does not exist, and so it cannot ip MAT AADS, Fall Oct

22 The total number of ips in the sequence is thus 2 < 1 2 = by infinite decreasing geometric series =1 (1), where <1 Worst-case time for a sequence of INCREMENT operations on an initially zero counter is therefore () The average cost of each operation, and therefore the amortized cost per operation, is() = (1) MAT AADS, Fall Oct The accounting method Assign differing charges to different operations, some charged more/less than they actually cost When an operation s amortized cost exceeds its actual cost, we assign the difference to specic objects in the data structure as credit Credit can help pay for later operations We can view the amortized cost of an operation as being split between its actual cost and credit that is either deposited or used up Amortized cost of operations may differ MAT AADS, Fall Oct

23 We want to show that in the worst case the average cost per operation is small by analyzing with amortized costs We must ensure that the total amortized cost of a sequence of operations provides an upper bound on the total actual cost of the sequence Moreover, this relationship must hold for all sequences of operations Let be the actual cost of the th operation and its amortized cost, require for all -sequences MAT AADS, Fall Oct Total credit stored in the data structure is the difference between the total amortized cost and the total actual cost, Total credit associated with the data structure must be nonnegative at all times If we ever were to allow the total credit to become negative, then the total amortized cost would not be an upper bound on the total actual cost We must take care that the total credit in the data structure never becomes negative MAT AADS, Fall Oct

24 Stack operations Recall the actual costs of operations; PUSH: 1, POP: 1, and MULTIPOP min(, ), where is the argument of MULTIPOP and is the stack size Let us assign the following amortized costs: PUSH: 2, POP: 0, and MULTIPOP 0 The amortized cost of MULTIPOP is a constant, whereas the actual cost is variable In general, the amortized costs of the operations under consideration may differ from each other, and they may even differ asymptotically MAT AADS, Fall Oct We can pay for any sequence of stack operations by charging the amortized costs We start with an empty stack When we push on the stack, we use 1 unit of cost to pay the actual cost of the push and are left with a credit of 1 The unit stored serves as prepayment for the cost of popping it from the stack When we execute a POP operation, we charge the operation nothing and pay its actual cost using the credit stored in the stack MAT AADS, Fall Oct

25 We can also charge MULTIPOPs nothing To pop the 1 st element, we take 1 unit of cost from the credit and use it to pay the actual cost of a POP operation To pop a 2 nd element, we again have an unit of cost in the credit to pay for the POP operation, Thus, we have always charged enough up front to pay for MULTIPOP operations For any sequence of operations, the total amortized cost is an upper bound on actual cost Since the total amortized cost is (), so is the total actual cost MAT AADS, Fall Oct Incrementing a binary counter The running time is proportional to the number of bits ipped, which we use as our cost Let us charge an amortized cost of 2 units to set a bit to 1 We use 1 unit to pay for the setting of the bit, and we place the other unit on the bit as credit to be used later when we ip the bit back to 0 At any point in time, every 1 in the counter has a unit of credit on it, and thus we can charge nothing to reset a bit to 0 MAT AADS, Fall Oct

26 The amortized cost of INCREMENT: The cost of resetting the bits within the while loop is paid for by the units on the bits that are reset The INCREMENT procedure sets at most one bit (line 6) and therefore the amortized cost of an INCREMENT operation is at most 2 units The number of 1s in the counter never becomes negative, and thus the amount of credit stays nonnegative at all times For INCREMENT operations, the total amortized cost is (), which bounds the total actual cost MAT AADS, Fall Oct The potential method We represent the prepaid work as potential energy, or just potential, which can be released to pay for future operations Associate the potential with the data structure as a whole rather than with specic objects We will perform operations, starting with an initial data structure Let be the actual cost of the th operation and the data structure that results after applying the th operation to data structure MAT AADS, Fall Oct

27 Potential function maps data structure to a real number ( ), the potential of The amortized cost of the th operation with respect to potential function is dened by = ( ( ) The actual cost + the change in potential The total amortized cost of the operations is = ( ( ) = ( ( ) 2 nd equality follows because the terms telescope MAT AADS, Fall Oct If we can dene a potential function so that ( ( ) then the total amortized cost gives an upper bound on the total actual cost In practice, we do not always know how many operations might be performed Therefore, if we require that ( ( ) for all, then we guarantee, as in the accounting method, that we pay in advance We usually just dene ( ) to be 0 and then show that ( 0for all MAT AADS, Fall Oct

28 If the potential difference ( ( ) of the th operation is positive, then the amortized cost represents an overcharge to the th operation, and the potential of the data structure increases negative, then the amortized cost represents an undercharge to the th operation, and the decrease in the potential pays for the actual cost of the operation Different potential functions may yield different amortized costs yet still be upper bounds on the actual costs MAT AADS, Fall Oct Stack operations We dene the potential function on a stack to be the number of objects in the stack For the empty stack, we have =0 Since the number of objects in the stack is never negative, the stack that results after the th operation has nonnegative potential, and thus 0= The total amortized cost of operations with respect to therefore represents an upper bound on the actual cost MAT AADS, Fall Oct

29 If the th operation on a stack containing objects is a PUSH operation, then the potential difference is = +1 =1 The amortized cost of this PUSH operation is = ( ( )=1+1=2 Suppose that the th operation on the stack is MULTIPOP(, ), which causes = min(, ) objects to be popped off the stack The actual cost of the operation is, and the potential difference is = : MAT AADS, Fall Oct Thus, the amortized cost of the MULTIPOP operation is = ( ( )= =0 Similarly, the amortized cost of an ordinary POP operation is 0 The amortized cost of each of the three operations is (1), and thus the total amortized cost of a sequence of operations is () Since ( ( ), the total amortized cost of operations is an upper bound on the total actual cost The worst-case cost of operations is () MAT AADS, Fall Oct

30 Incrementing a binary counter We dene the potential of the counter after the thincrement to be, the number of 1s in the counter after the th operation Suppose that the thincrement operation resets bits The actual cost of the operation is therefore at most +1, since in addition to resetting bits, it sets at most one bit to 1 If =0, then the th operation resets all bits MAT AADS, Fall Oct In this situation = = If >0, then = +1 In either case, +1, and the potential difference is ( ( ( +1) =1 The amortized cost is therefore = ( ( ) =2 If the counter starts at zero, then ( )=0 ( 0for all, the total amortized cost of INCREMENTs is an upper bound on the total actual cost, and so the worst-case cost is () MAT AADS, Fall Oct

31 The potential method gives us a way to analyze the counter even when it does not start at zero The counter starts with 1s, and after INCREMENTs it has 1s, where, Rewrite total actual cost in terms of amortized cost as = ( ) + ( ) We have 2for all MAT AADS, Fall Oct Since ( )= and ( )=, the total actual cost of INCREMENTs is + Note that since, as long as = (), the total actual cost is () I.e., if we execute at least () INCREMENT operations, the total actual cost is (), no matter what initial value the counter contains MAT AADS, Fall Oct