II (Sorting and) Order Statistics

Transcription

1 II (Sorting and) Order Statistics Heapsort Quicksort Sorting in Linear Time Medians and Order Statistics 8 Sorting in Linear Time The sorting algorithms introduced thus far are comparison sorts Any comparison sort must make ( lg ) comparisons in the worst case to sort elements Merge sort and heapsort are asymptotically optimal We examine counting sort, radix sort, and bucket sort that run in linear time MAT AADS, Fall Sep

2 8.1 Lower bounds for sorting We can view comparison sorts abstractly in terms of decision trees a full binary tree that represents the comparisons between elements that are performed by a particular sorting algorithm MAT AADS, Fall Sep The execution of the sorting algorithm corresponds to tracing a simple path from the root of the decision tree down to a leaf An internal node compares When we come to a leaf, the sorting algorithm has established the ordering () () () Any correct sorting algorithm must be able to produce each permutation of its input Each of the! permutations must appear as one of the leaves of the decision tree for a comparison sort to be correct MAT AADS, Fall Sep

3 A lower bound for the worst case Length of the longest path from the root to any of the reachable leaves is the worst-case number of comparisons The worst-case number of comparisons for a given algorithm equals the height of its tree A lower bound on the heights of all decision trees in which each permutation appears as a reachable leaf is a bound on the running time of any comparison sort algorithm MAT AADS, Fall Sep Theorem 8.1 Any comparison sort algorithm requires ( lg ) comparisons in the worst case. Proof It sufces to determine the height of a decision tree in which each permutation appears as a reachable leaf. Consider a decision tree of height with reachable leaves corresponding to a comparison sort on elements. Each of the! permutations of the input appears as some leaf. MAT AADS, Fall Sep

4 Therefore, we have!. Since a binary tree of height has no more than 2 leaves, we have 2 which, by taking logarithms, implies lg(!) since the lg function is monotonically increasing. Furthermore, lg, because lg! = ( lg ) Hence, heapsort and merge sort are asymptotically optimal comparison sorts MAT AADS, Fall Sep Counting sort Assume that each input element is an integer in the range 0 to, for some integer When = (), the sort runs in () time Counting sort determines, for an input element, the number of elements It uses this information to place element directly into its position in the output array E.g., if 17 elements are less than (or equal to), then belongs in output position 18 MAT AADS, Fall Sep

5 In addition to the input array [1.. ] we require two other arrays: [1..]holds the sorted output, and [0..]provides temporary working storage MAT AADS, Fall Sep COUNTING-SORT(,, ) 1. let [0..]be a new array 2. for 0to for 1to. 5. [ [] // now contains the number of elements equal to 7. for 1to 8. + [ 1] 9. // now contains the number of elements 10. for. downto [] MAT AADS, Fall Sep

6 How much time does counting sort require? The for loop of lines 2 3 takes time () the for loop of lines 4 5 takes time () the for loop of lines 7 8 takes time (), and the for loop of lines takes time () Thus, the overall time is ( + ) In practice, we usually use counting sort when we have = (), in which case the running time is () MAT AADS, Fall Sep Counting sort beats the lower bound of lg because it is not a comparison sort In fact, no comparisons between input elements occur anywhere in the code Instead, counting sort uses the actual values of the elements to index into an array An important property of counting sort is that it is stable numbers with the same value appear in the output array in the same order as they do in the input array MAT AADS, Fall Sep

7 8.3 Radix sort Radix sort is used by the card-sorting machines you now nd only in computer museums Radix sort solves the problem of card sorting counter-intuitively by sorting on the least signicant digit rst In order for radix sort to work correctly, the digit sorts must be stable MAT AADS, Fall Sep RADIX-SORT(, ) 1. for 1to 2. use a stable sort to sort array on digit MAT AADS, Fall Sep

8 Lemma 8.3 Given -digit numbers in which each digit can take on up to possible values, RADIX-SORT sorts the numbers in + time if the stable sort takes + time Proof The correctness of radix sort follows by induction on the columns. When each digit is in the range 0 to 1(so that it can take on possible values), and is not too large, counting sort is the obvious choice. Each pass over -digit numbers then takes time +.There are passes, and so the total time for radix sort follows. MAT AADS, Fall Sep Bucket sort Assume that the input is drawn from uniform distribution and has an average-case running time of () Counting and bucket sort are fast because they assumes something about the input Counting sort: the input contains integers in a small range, Bucket sort: the input is drawn from a random process that distributes elements uniformly and independently over the interval [0,1) MAT AADS, Fall Sep

9 Bucket sort divides the interval 0,1 into equal-sized subintervals, or buckets, and then distributes the input numbers into the buckets Since inputs are uniformly and independently distributed over 0,1, we do not expect many numbers to fall into each bucket We sort numbers in each bucket and go through the buckets in order, listing the elements in each The input is an -element array and each element [] in the array satises []<1 The code requires an auxiliary array [0.. 1] of linked lists (buckets) MAT AADS, Fall Sep BUCKET-SORT() 1. let [0.. 1] be a new array for 0to 1 4. make [] an empty list 5. for 1to 6. insert [] into list [ ] 7. for 0to 1 8. sort list [] with insertion sort 9. concatenate the lists 0,1,,[ 1] together in order MAT AADS, Fall Sep

10 MAT AADS, Fall Sep Consider two elements [] and [] Assume wlog that [ [] Since [ [], either element [] goes into the same bucket as [] or it goes into a bucket with a lower index If [] and [] go into the same bucket the for loop of lines 7 8 puts them into the proper order If [] and [] go into different buckets line 9 puts them into the proper order Therefore, bucket sort works correctly MAT AADS, Fall Sep

11 Observe that all lines except line 8 take () time in the worst case We need to analyze the total time taken by the calls to insertion sort in line 8 Let be the random variable denoting the number of elements placed in bucket [] Since insertion sort runs in quadratic time, the running time of bucket sort is = + ( ) MAT AADS, Fall Sep We now compute the expected value of the running time, where we take the expectation over the input distribution Taking expectations of both sides and using linearity of expectation, we have + + MAT AADS, Fall Sep

12 Claim: = 2 1 for = 0,1,, 1 It is no surprise that each bucket has the same value of, since each value in is equally likely to fall in any bucket To prove the claim, dene indicator random variables =I fallsinbucket for = 0,1,, 1and = 1,2,, Thus, = MAT AADS, Fall Sep = = = + = + MAT AADS, Fall Sep

13 Indicator random variable is 1 with probability 1/ and 0 otherwise, therefore E = =1 When, the variables and are independent, and hence = = = MAT AADS, Fall Sep Substituting these two expected values, we obtain = = 1 +( 1) 1 =1+ 1 =2 1 which proves the claim MAT AADS, Fall Sep

14 Using this expected value we conclude that average-case running time for bucket sort is + (21)= Even if the input is not drawn uniformly, bucket sort may still run in linear time As long as the input has the property the sum of the squares of the bucket sizes is linear in the total number of elements, bucket sort will run in linear time MAT AADS, Fall Sep Medians and Order Statistics The th order statistic of a set of elements is the th smallest element E.g., the minimum of a set of elements is the rst order statistic ( =1), and the maximum is the th order statistic ( = ) Amedian is the halfway point of the set When is odd, the median is unique, occurring at =(+ 1)/2 MAT AADS, Fall Sep

15 When is even, there are two medians, occurring at = /2 and = /2+1 Thus, regardless of the parity of, medians occur at =(+ 1)/2 (the lower median) and =(+ 1)/2 (the upper median) For simplicity, we use the median to refer to the lower median MAT AADS, Fall Sep The problem of selecting the th order statistic from a set of distinct numbers We assume that the set contains distinct numbers virtually everything extends to the situation in which a set contains repeated values We formally specify the problem as follows: Input: A set of (distinct) numbers and an integer, with Output: The element that is larger than exactly 1other elements of MAT AADS, Fall Sep

16 We can solve the problem in ( lg ) time by heapsort or merge sort and then simply index the th element in the output array There are faster algorithms First, we examine the problem of selecting the minimum and maximum of a set of elements Then we analyze a practical randomized algorithm that achieves an () expected running time, assuming distinct elements MAT AADS, Fall Sep Minimum and maximum How many comparisons are necessary to determine the minimum of a set of elements? We can easily obtain an upper bound of 1 comparisons examine each element of the set in turn and keep track of the smallest element seen so far. In the following procedure, we assume that the set resides in array, where. = MAT AADS, Fall Sep

17 MINIMUM() 1. min[1] 2. for 2to. 3. if min>[] 4. min[] 5. return min We can, of course, nd the max with 1 comparisons as well MAT AADS, Fall Sep This is the best we can do, since we can obtain a lower bound of 1comparisons Think of any algorithm that determines the minimum as a tournament among the elements Each comparison is a match in the tournament in which the smaller of the two elements wins Observing that every element except the winner must lose at least one match, we conclude that 1 comparisons are necessary to determine the minimum Hence, the algorithm MINIMUM is optimal w.r.t. the number of comparisons performed MAT AADS, Fall Sep

18 Simultaneous minimum and maximum Sometimes, we must nd both the minimum and the maximum of a set of elements For example, a graphics program may need to scale a set of (, ) data to t onto a rectangular display screen or other graphical output device To do so, the program must rst determine the minimum and maximum value of each coordinate MAT AADS, Fall Sep () comparisons is asymptotically optimal: Simply nd the minimum and maximum independently, using 1comparisons for each, for a total of 2 2comparisons In fact, we can nd both the minimum and the maximum using at most 3 /2 comparisons by maintaining both the minimum and maximum elements seen thus far Rather than processing each element of the input by comparing it against the current minimum and maximum, we process elements in pairs MAT AADS, Fall Sep

19 Compare pairs of input elements rst with each other, and then we compare the smaller with the current min and the larger to the current max, at a cost of 3 comparisons for every 2 elements If is odd, we set both the min and max to the value of the rst element, and then we process the rest of the elements in pairs If is even, we perform 1 comparison on the rst 2 elements to determine the initial values of the min and max, and then process the rest of the elements in pairs as in the case for odd MAT AADS, Fall Sep If is odd, then we perform 3 /2 comparisons If is even, we perform 1 initial comparison followed by 3( 2)/2 comparisons, for a total of 2 2 Thus, in either case, the total number of comparisons is at most 3 /2 MAT AADS, Fall Sep

20 9.2 Selection in expected linear time The selection problem appears more difcult than nding a minimum, but the asymptotic running time for both is the same: () A divide-and-conquer algorithm RANDOMIZED-SELECT is modeled after the quicksort algorithm Unlike quicksort, RANDOMIZED-SELECT works on only one side of the partition Whereas quicksort has an expected running time of ( lg ), the expected running time of RANDOMIZED- SELECT is (), assuming that the elements are distinct MAT AADS, Fall Sep RANDOMIZED-SELECT uses the procedure RANDOMIZED-PARTITION ofrandomized- QUICKSORT RANDOMIZED-PARTITION(,, ) 1. RANDOM(, ) 2. exchange [] with [] 3. return PARTITION(,, ) MAT AADS, Fall Sep

21 Partitioning the array PARTITION(,, ) 1. [] for to 1 4. if [ exchange [] with [] 7. exchange [ + 1]with [] 8. return + 1 MAT AADS, Fall Sep MAT AADS, Fall Sep

22 PARTITION always selects an element = [] as a pivot element around which to partition the subarray [..] As the procedure runs, it partitions the array into four (possibly empty) regions At the start of each iteration of the for loop in lines 3 6, the regions satisfy properties, shown above MAT AADS, Fall Sep At the beginning of each iteration of the loop of lines 3 6, for any array index, 1. If, then [ 2. If +11, then > 3. If =, then = Indices between and 1are not covered by any case, and the values in these entries have no particular relationship to the pivot The running time of PARTITION on the subarray [..]is, = +1 MAT AADS, Fall Sep

23 Return the th smallest element of [.. ] RANDOMIZED-SELECT(,,, ) 1. if = 2. return [] 3. RANDOMIZED-PARTITION(,, ) if = // the pivot value is the answer 6. return [] 7. elseif < 8. return RANDOMIZED-SELECT(,, 1, ) 9. else return RANDOMIZED-SELECT(, + 1,, ) MAT AADS, Fall Sep (1) checks for the base case of the recursion Otherwise, RANDOMIZED-PARTITION partitions [..]into two (possibly empty) subarrays [.. 1] and [ +1..]s.t. each element in the former is < each element of the latter (4) computes the # of elements in [..] (5) checks if [] is th smallest element Otherwise, determine in which of the two subarrays the th smallest element lies If <, then the desired element lies on the low side of the partition, and (8) recursively selects it MAT AADS, Fall Sep

24 If >, then the desired element lies on the high side of the partition Since we already know values that are smaller than the th smallest element of [..]the desired element is the ( )th smallest element of [ +1..], which (9) nds recursively MAT AADS, Fall Sep Worst-case running time for RANDOMIZED-SELECT is ( ), even to nd the minimum The algorithm has a linear expected running time, though Because it is randomized, no particular input elicits the worst-case behavior Let the running time of RANDOMIZED-SELECT on an input array [..]of elements be a random variable () We obtain an upper bound on [ ] as follows MAT AADS, Fall Sep

25 The procedure RANDOMIZED-PARTITION is equally likely to return any element as the pivot Therefore, for each such that, the subarray [..]has elements (all less than or equal to the pivot) with probability 1/ For =1,2,,, dene indicator random variables where =Ithesubarray..hasexactlyelements MAT AADS, Fall Sep Assuming that the elements are distinct, we have =1 When we choose [] as the pivot element, we do not know, a priori, 1) if we will terminate immediately with the correct answer, 2) recurse on the subarray [.. 1], or 3) recurse on the subarray [ +1..] This decision depends on where the th smallest element falls relative to [] MAT AADS, Fall Sep

26 Assuming () to be monotonically increasing, we can upper-bound the time needed for a recursive call by that on largest possible input To obtain an upper bound, we assume that the th element is always on the side of the partition with the greater number of elements For a given call of RANDOMIZED-SELECT, the indicator random variable has value 1 for exactly one value of, and it is 0 for all other When =1, the two subarrays on which we might recurse have sizes 1and MAT AADS, Fall Sep We have the recurrence ( ((max( 1, ))+()) = (max( 1, ))+() Taking expected values, we have ( (max( 1, ))+() = [ (max( 1, ))]+() MAT AADS, Fall Sep

27 = [(max( 1, ))]+() = 1 [(max( 1, ))] + () The first Eq. on this slide follows by independence of random variables and max( 1, ) Consider the expression max( 1, ) = 1 if > /2 if /2 MAT AADS, Fall Sep If is even, each term from /2 up to ( 1) appears exactly twice in the summation, and if is odd, all these terms appear twice and /2 appears once Thus, we have [ 2 +() / We can show that = () by substitution In summary, we can nd any order statistic, and in particular the median, in expected linear time, assuming that the elements are distinct MAT AADS, Fall Sep