Introduction to Algorithms 6.046J/18.401J

Transcription

1 Introduction to Algorithms 6.046J/18.401J Menu for Today Show that Θ (n lg n) is the best possible running time for a sorting algorithm. Design an algorithm that sorts in linear time. Hint: maybe the models are different? Lecture 6 Prof. Manolis Kellis Introduction to Algorithms 2/26/2008 L5.2 Comparison sort All the sorting algorithms we have seen so far are comparison sorts: only use comparisons to determine the relative order of elements. E.g., insertion sort, merge sort, quicksort, heapsort. The best worst-case running time that we ve seen for comparison sorting is O(n lg n). Is O(n lg n) the best we can do? Decision trees can help us answer this question. Decision-tree example Sort a 1, a 2,, a n 123 1: Each internal node is labeled i:j for i, j {1, 2,, n}. The left subtree shows subsequent comparisons if a i a j. The right subtree shows subsequent comparisons if a i a j. Introduction to Algorithms 2/26/2008 L5.3 Introduction to Algorithms 2/26/2008 L5.4 Decision-tree example Sort a 1, a 2, a 3 = 9, 4, 6 : 123 1: Each internal node is labeled i:j for i, j {1, 2,, n}. The left subtree shows subsequent comparisons if a i a j. The right subtree shows subsequent comparisons if a i a j. Decision-tree example Sort a 1, a 2, a 3 = 9, 4, 6 : 123 1: Each internal node is labeled i:j for i, j {1, 2,, n}. The left subtree shows subsequent comparisons if a i a j. The right subtree shows subsequent comparisons if a i a j. Introduction to Algorithms 2/26/2008 L5.5 Introduction to Algorithms 2/26/2008 L5.6 1

2 Decision-tree example Sort a 1, a 2, a 3 = 9, 4, 6 : 123 1: Each internal node is labeled i:j for i, j {1, 2,, n}. The left subtree shows subsequent comparisons if a i a j. The right subtree shows subsequent comparisons if a i a j. Introduction to Algorithms 2/26/2008 L5.7 Decision-tree example Sort a 1, a 2, a 3 = 9, 4, 6 : 123 1: Each leaf contains a permutation π(1), π(2),, π(n) to indicate that the ordering a π(1) a π(2) L a π(n) has been established. Introduction to Algorithms 2/26/2008 L5.8 Decision-tree model A decision tree can model the execution of any comparison sort: One tree for each input size n. View the algorithm as splitting whenever it compares two elements. The tree contains the comparisons along all possible instruction traces. The running time of the algorithm = the length of the path taken. Worst-case running time = height of tree. Lower bound for decisiontree sorting Theorem. Any decision tree that can sort n elements must have height Ω(n lg n). Proof. The tree must contain n! leaves, since there are n! possible permutations. A height-h binary tree has 2 h leaves. Thus, n! 2 h. h lg(n!) (lg is mono. increasing) lg ((n/e) n ) (Stirling s formula) = n lg n n lg e = Ω(n lg n). Introduction to Algorithms 2/26/2008 L5.9 Introduction to Algorithms 2/26/2008 L5.10 Lower bound for comparison sorting Corollary. Merge sort is an asymptotically optimal comparison sorting algorithm. Sorting in linear time Counting sort: No comparisons between elements. : A[1.. n], where A[ j] {1, 2,, k}. Key assumption about input distribution: elements come from fixed range 1..k : B[1.. n], sorted Auxiliary storage: C[1.. k]. Key idea: use addressing array C to move elements from A directly into final B position. Introduction to Algorithms 2/26/2008 L5. Introduction to Algorithms 2/26/2008 L5.12 2

3 for 1 s for 3 s for 4 s Auxiliary count Introduction to Algorithms 2/26/2008 L5.13 Introduction to Algorithms 2/26/2008 L5.14 for 1 s for 3 s for 4 s Auxiliary count Introduction to Algorithms 2/26/2008 L Q: Why bother moving elements? Question: if I know the number of 1s, 2s, etc, why not fill out the output array directly? for 1 s for 3 s for 4 s Fill with 1s? Fill with 3s? Fill with 4s? Answer: Each element may have lots of additional associated information: 1,2.. are just keys to compare elements, but entire data structures must move! E.g.: sorting students by dormitory distance to class. Remember more than just distance or dorm name! Introduction to Algorithms 2/26/2008 L5.16 Auxiliary for 1 s for 3 s for 4 s 4 55 Auxiliary for 1 s for 3 s for 4 s Introduction to Algorithms 2/26/2008 L5.17 Introduction to Algorithms 2/26/2008 L5.18 3

4 Auxiliary for 1 s for 3 s for 4 s Auxiliary for 1 s for 3 s for 4 s Introduction to Algorithms 2/26/2008 L5.19 Introduction to Algorithms 2/26/2008 L5.20 Auxiliary a 0 b c for 1 s for 3 s for 4 s d 55 4 Auxiliary for 1 s for 3 s for 4 s Introduction to Algorithms 2/26/2008 L5.21 Introduction to Algorithms 2/26/2008 L5.22 Counting sort pseudocode Running the pseudocode Initialize C array For each value i First pass: count Post-process: s C[i] = {key i} Second pass: moving copy A[j] to B[.] update for i 1 to k do C[i] 0 for i 2 to k do C[i] C[i] + C[i 1] C: Introduction to Algorithms 2/26/2008 L5.23 Introduction to Algorithms 2/26/2008 L5.24 4

5 Loop 1 Loop 2 C: C: for i 1 to k do C[i] 0 Introduction to Algorithms 2/26/2008 L5.25 Introduction to Algorithms 2/26/2008 L5.26 Loop 2 Loop 2 C: C: 00 Introduction to Algorithms 2/26/2008 L5.27 Introduction to Algorithms 2/26/2008 L5.28 Loop 2 Loop 2 C: C: Introduction to Algorithms 2/26/2008 L5.29 Introduction to Algorithms 2/26/2008 L5.30 5

6 Loop 3 Loop 3 C: C: C': C': for i 2 to k do C[i] C[i] + C[i 1] C[i] = {key i} for i 2 to k do C[i] C[i] + C[i 1] C[i] = {key i} Introduction to Algorithms 2/26/2008 L5.31 Introduction to Algorithms 2/26/2008 L5.32 Loop 3 Loop 4 C: C: C': C': for i 2 to k do C[i] C[i] + C[i 1] C[i] = {key i} Introduction to Algorithms 2/26/2008 L5.33 Introduction to Algorithms 2/26/2008 L5.34 Loop 4 Loop 4 C: C: C': C': 44 Introduction to Algorithms 2/26/2008 L5.35 Introduction to Algorithms 2/26/2008 L5.36 6

7 Loop 4 Loop 4 C: 44 C: C': C': Introduction to Algorithms 2/26/2008 L5.37 Introduction to Algorithms 2/26/2008 L5.38 Analysis Θ(k) Θ(n) Θ(k) Θ(n) Θ(n + k) for i 1 to k do C[i] 0 for i 2 to k do C[i] C[i] + C[i 1] Introduction to Algorithms 2/26/2008 L5.39 Running time If k = O(n), then counting sort takes Θ(n) time. But, sorting takes Ω(n lg n) time! Where s the fallacy? Ω(n lg n) time is for comparison sorting. Counting sort is not a comparison sort. (not a single comparison between elmts occurs) Why not always use counting sort? Needs additional assumptions about input 1..k In general case, 1..k may be >> n (eg. 32-bit ints) Introduction to Algorithms 2/26/2008 L5.40 Google interview questions for presidential candidates During his presidential bid, senator Barack Obama is asked a question key to the election (Nov 14, 2007): Larry Schwimmer asks: What s the most efficient way to sort a million 32-bit integers? Without losing a heartbeat, Obama replies: I think the bubble sort would be the wrong way to go What Obama really meant was: Radix Sort, clearly Radix sort Origin: Herman Hollerith s card-sorting machine for the 1890 U.S. Census. (See Appendix.) Digit-by-digit sort. Hollerith s original (bad) idea: sort on most-significant digit first. Good idea: Sort on least-significant digit first with auxiliary stable sort. Introduction to Algorithms 2/26/2008 L5.41 Introduction to Algorithms 2/26/2008 L5.42 7

8 Stable sorting Operation of radix sort Counting sort is a stable sort: it preserves the input order among equal elements Exercise: What other sorts have this property? Introduction to Algorithms 2/26/2008 L5.43 Introduction to Algorithms 2/26/2008 L5.44 Correctness of radix sort Correctness of radix sort Induction on digit position Assume that the numbers are sorted by their low-order t 1digits. Sort on digit t Induction on digit position Assume that the numbers are sorted by their low-order t 1digits. Sort on digit t Two numbers that differ in digit t are correctly sorted. Introduction to Algorithms 2/26/2008 L5.45 Introduction to Algorithms 2/26/2008 L5.46 Correctness of radix sort Analysis of radix sort T(n,b) Induction on digit position Assume that the numbers are sorted by their low-order t 1digits. Sort on digit t Two numbers that differ in digit t are correctly sorted. Two numbers equal in digit t are put in the same order as the input correct order.() Sort n(e.g.10 6 ) integers of b(e.g.32) bits each. Interpreted as having b/r digits, each base-2 r. b-bit integer: r bits r bits r bits r bits r bits r bits d=b/r digits Assume counting sort is the auxiliary stable sort. How many passes should we make? Example: 32-bit word r = 8 b/r =4passes on base-2 8 digits. r = 16 b/r =2passes on base-2 16 digits. Key requirement: Introduction to Algorithms Stable sort subroutine! 2/26/2008 L5.47 Introduction to Algorithms 2/26/2008 L5.48 8

9 Analysis (continued) T ( n, b) = Θ b ( n + 2r ) r b/r passes b bits b/r passes Counting sort Θ(n + k). Range is (1..k = 1..2 r ). r bits Choose r to minimize T(n, b): Increase r to have fewer passes = b/r. Decrease r to have faster counting sort = 2 r. (If r >> lg n, the time grows exponentially). Introduction to Algorithms 2/26/2008 L5.49 Analysis: choosing optimal r T ( n, b) = Θ b ( n + 2r ) r Minimize T(n, b) by differentiating and setting to 0. Or, just observe that we don t want 2 r > n, and there s no harm asymptotically in choosing r as large as possible subject to this constraint. Choosing r = lg n implies T(n, b) = Θ(bn/lg n). For numbers in the range from 0 to n d 1, we have b = d lg n radix sort runs in Θ(dn) time. Introduction to Algorithms 2/26/2008 L5.50 Conclusions In practice, radix sort is fast for large inputs, as well as simple to code and maintain. Example (32-bit numbers): At most 3 passes when sorting 2000 numbers. Merge sort and quicksort do at least lg 2000 = passes. Potential downside: Radix sort displays little locality of reference, and thus a well-tuned quicksort may fare better on modern processors, which feature steep memory hierarchies. Introduction to Algorithms 2/26/2008 L5.51 Appendix: Punched-card technology Herman Hollerith ( ) Punched cards Hollerith s tabulating system Operation of the sorter Origin of radix sort Modern IBM card Web resources on punchedcard technology Return to last slide viewed. Introduction to Algorithms 2/26/2008 L5.52 Herman Hollerith ( ) The 1880 U.S. Census took almost 10 years to process. While a lecturer at MIT, Hollerith prototyped punched-card technology. His machines, including a card sorter, allowed the 1890 census total to be reported in 6 weeks. He founded the Tabulating Machine Company in 19, which merged with other companies in 1924 to form International Business Machines. Introduction to Algorithms 2/26/2008 L5.53 Punched cards Punched card = data record. Hole = value. Algorithm = machine + human operator. Replica of punch card from the 1900 U.S. census. [Howells 2000] Introduction to Algorithms 2/26/2008 L5.54 9

10 Hollerith s tabulating system Pantograph card punch Hand-press reader Dial counters Sorting box Figure from [Howells 2000]. Introduction to Algorithms 2/26/2008 L5.55 Operation of the sorter An operator inserts a card into the press. Pins on the press reach through the punched holes to make electrical contact with mercuryfilled cups beneath the card. Whenever a particular digit value is punched, the lid of the corresponding sorting bin lifts. The operator deposits the card Hollerith Tabulator, Pantograph, Press, and Sorter into the bin and closes the lid. When all cards have been processed, the front panel is opened, and the cards are collected in order, yielding one pass of a stable sort. Introduction to Algorithms 2/26/2008 L5.56 Origin of radix sort Hollerith s original 1889 patent alludes to a mostsignificant-digit-first radix sort: The most complicated combinations can readily be counted with comparatively few counters or relays by first assorting the cards according to the first items entering into the combinations, then reassorting each group according to the second item entering into the combination, and so on, and finally counting on a few counters the last item of the combination for each group of cards. Least-significant-digit-first radix sort seems to be a folk invention originated by machine operators. Modern IBM card One character per column. Produced by the WWW Virtual Punch- Card Server. So, that s why text windows have 80 columns! Introduction to Algorithms 2/26/2008 L5.57 Introduction to Algorithms 2/26/2008 L5.58 Web resources on punchedcard technology Doug Jones s punched card index Biography of Herman Hollerith The 1890 U.S. Census Early history of IBM Pictures of Hollerith s inventions Hollerith s patent application (borrowed from Gordon Bell s CyberMuseum) Introduction to Algorithms 2/26/2008 L