Lecture 5. Coutig Sort / Radix Sort T. H. Corme, C. E. Leiserso ad R. L. Rivest Itroductio to Algorithms, 3rd Editio, MIT Press, 2009 Sugkyukwa Uiversity Hyuseug Choo choo@skku.edu Copyright 2000-2018 Networkig Laboratory
Sortig So Far Isertio Sort Easy to code Fast o small iputs (less tha ~ 50 elemets) Fast o early-sorted iputs O( 2 ) worst case O( 2 ) average (equally-likely iputs) case O( 2 ) reverse-sorted case Algorithms Networkig Laboratory 2/62
Sortig So Far Merge Sort Divide-ad-Coquer Split array i half Recursively sort subarrays Liear-time merge step O( lg ) worst case It does ot sort i place Algorithms Networkig Laboratory 3/62
Sortig So Far Heap Sort It uses the very useful heap data structure Complete biary tree Heap property: paret key > childre s keys O( lg ) worst case It sorts i place Algorithms Networkig Laboratory 4/62
Sortig So Far Quick Sort Divide-ad-Coquer Partitio array ito two subarrays, recursively sort All of first subarray < all of secod subarray No merge step eeded! O( lg ) average case Fast i practice O( 2 ) worst case Naïve implemetatio: worst case o sorted iput Address this with radomized quicksort Algorithms Networkig Laboratory 5/62
How Fast Ca We Sort? We will provide a lower boud, the beat it How do you suppose we ll beat it? First, a observatio: all of the sortig algorithms so far are compariso sorts The oly operatio used to gai orderig iformatio about a sequece is the pairwise compariso of two elemets Theorem: all compariso sorts are ( lg ) Algorithms Networkig Laboratory 6/62
Decisio Trees 5 7 3 5 73 37 35 35 7 Algorithms Networkig Laboratory 7/62
Decisio Trees Decisio trees provide a abstractio of compariso sorts A decisio tree represets the comparisos made by a compariso sort. What do the leaves represet? How may leaves must there be? Algorithms Networkig Laboratory 8/62
Practice Problems What is the smallest possible depth of a leaf i a decisio tree for a compariso sort? Algorithms Networkig Laboratory 9/62
Decisio Trees Decisio trees ca model compariso sorts For a give algorithm: Oe tree for each Tree paths are all possible executio traces What s the logest path i a decisio tree for isertio sort? For merge sort? What is the asymptotic height of ay decisio tree for sortig elemets? Aswer: ( lg ) (Now let s prove it ) Algorithms Networkig Laboratory 10/62
A Lower Boud for Compariso Sortig Theorem: Ay decisio tree that sorts elemets has height ( lg ) What s the miimum umber of leaves? What s the maximum umber of leaves of a biary tree of height h? Clearly the miimum umber of leaves is less tha or equal to the maximum umber of leaves Algorithms Networkig Laboratory 11/62
A Lower Boud for Compariso Sortig So we have! 2 h Takig logarithms: lg (!) h Stirlig s approximatio tells us: Thus:! e h lg e Algorithms Networkig Laboratory 12/62
A Lower Boud for Compariso Sortig So we have h lg e lg lg lg e Thus the miimum height of a decisio tree is ( lg ) Algorithms Networkig Laboratory 13/62
A Lower Boud for Compariso Sorts Thus the time to compariso sort elemets is ( lg ) Corollary: Heapsort ad Mergesort are asymptotically optimal compariso sorts But the ame of this lecture is Sortig i liear time! How ca we do better tha ( lg )? Algorithms Networkig Laboratory 14/62
Sortig i Liear Time Coutig Sort No comparisos betwee elemets! But depeds o assumptio about the umbers beig sorted Basic Idea For each iput elemet x, determie the umber of elemets less tha or equal to x For each iteger i (0 i k), cout how may elemets whose values are i The we kow how may elemets are less tha or equal to i Algorithm Storage A[1..]: iput elemets B[1..]: sorted elemets C[0..k]: hold the umber of elemets less tha or equal to i Algorithms Networkig Laboratory 15/62
Coutig Sort Illustratio (Rage from 0 to 5) Algorithms Networkig Laboratory 16/55
Coutig Sort Illustratio A 2 5 3 0 2 3 0 3 B 0 3 3 B 0 2 3 3 C 1 2 4 5 7 8 C 1 2 3 5 7 8 B 0 0 2 3 3 B 0 0 2 3 3 3 B 0 0 2 3 3 3 8 C 0 2 3 5 7 8 C 0 2 3 4 7 8 C 0 2 32 4 7 7 B 0 0 2 2 3 3 3 8 C 0 2 3 4 7 7 Algorithms Networkig Laboratory 17/62
Coutig Sort (k) () (k) () (+k) Algorithms Networkig Laboratory 18/62
Coutig Sort Algorithms Networkig Laboratory 19/62
Coutig Sort Video Cotet A illustratio of Coutig Sort. Algorithms Networkig Laboratory 20/62
Coutig Sort Algorithms Networkig Laboratory 21/62
Coutig Sort Total time: O( + k) Usually, k = O() Thus coutig sort rus i O() time But sortig is ( lg )! No cotradictio This is ot a compariso sort I fact, there are o comparisos at all! Notice that this algorithm is stable Algorithms Networkig Laboratory 22/62
Radix Sort Ituitively, you might sort o the most sigificat digit, the the secod msd, etc. Problem Lots of itermediate piles of cards (read: scratch arrays) to keep track of Key idea Sort the least sigificat digit first RadixSort(A, d) for i=1 to d StableSort(A) o digit i Algorithms Networkig Laboratory 23/62
Radix Sort Example Iput data: a b a b a c c a a a c b b a b c c a a a c b b a Algorithms Networkig Laboratory 24/62
Radix Sort Example Pass 1: Lookig at rightmost positio. a b a b a c c a a a c b b a b c c a a a c b b a Place ito appropriate pile. a b c Algorithms Networkig Laboratory 25/62
Radix Sort Example Pass 1: Lookig at rightmost positio. b b a Joi piles. c c a c a a b a b a a c a b a a c b b a c a b c Algorithms Networkig Laboratory 26/62
Radix Sort Example Pass 2: Lookig at ext positio. a b a c a a c c a b b a a c b b a b b a c a a c Place ito appropriate pile. a b c Algorithms Networkig Laboratory 27/62
Radix Sort Example Pass 2: Lookig at ext positio. a a c Joi piles. b a c b a b b b a a c b c a a a b a c c a a b c Algorithms Networkig Laboratory 28/62
Radix Sort Example Pass 3: Lookig at last positio. c a a b a b b a c a a c a b a b b a c c a a c b Place ito appropriate pile. a b c Algorithms Networkig Laboratory 29/62
Radix Sort Example Pass 3: Lookig at last positio. Joi piles. a c b a b a b b a b a c c c a a a c b a b c a a a b c Algorithms Networkig Laboratory 30/62
Radix Sort Example Result is sorted. a a c a b a a c b b a b b a c b b a c a a c c a Algorithms Networkig Laboratory 31/62
Radix Sort Example 13 30 30 28 28 56 56 36 36 28 36 56 13 30 56 36 30 28 13 36 36 30 13 28 130 56 56 28 Radix Bis 0 1 2 3 4 5 6 7 8 9 Algorithms Networkig Laboratory 32/62
Radix Sort Example d digits i total Algorithms Networkig Laboratory 33/62
Radix Sort Coutig sort is obvious choice: Sort umbers o digits that rage from 1..k Time: O( + k) Each pass over umbers with d digits takes time O(+k), so total time O(d+dk) Whe d is costat ad k=o(), takes O() time I geeral, radix sort based o coutig sort is Fast Asymptotically fast (i.e., O()) Simple to code Algorithms Networkig Laboratory 34/62
Radix Sort Video Cotet A illustratio of Radix Sort. Algorithms Networkig Laboratory 35/62
Radix Sort Algorithms Networkig Laboratory 36/62
Practice Problems Show how to sort itegers i the rage 0 to 3 1 i O() time. Algorithms Networkig Laboratory 37/62
Medias ad Order Statistics The ith order statistic of a set of elemets is the ith smallest elemet The media is the halfway poit Defie the selectio problem as: Give a set A of elemets, fid a elemet x A that is larger tha x-1 elemets Obviously, this ca be solve i O( lg ). Why? Is it really ecessary to sort all the umbers? Algorithms Networkig Laboratory 38/62
Order Statistics The ith order statistic i a set of elemets is the ith smallest elemet The miimum is thus the 1st order statistic The maximum is the th order statistic The media is the /2 order statistic If is eve, there are 2 medias The lower media (+1)/2 The upper media (+1)/2 How ca we calculate order statistics? What is the ruig time? Algorithms Networkig Laboratory 39/62
Miimum ad Maximum The maximum ca be foud i exactly the same way by replacig the > with < i the above algorithm. Algorithms Networkig Laboratory 40/62
Simultaeous Mi ad Max How may comparisos are eeded to fid the miimum elemet i a set? The maximum? Ca we fid the miimum ad maximum with less tha twice the cost? Yes: Walk through elemets by pairs Compare each elemet i pair to the other Compare the largest to maximum, smallest to miimum Pair (a i, a i+1 ), assume a i < a i+1, the Compare (mi,a i ), (a i+1,max) 3 comparisios for each pair. Total cost: 3 comparisos per 2 elemets O( 3 /2 ) Algorithms Networkig Laboratory 41/62
The Selectio Problem A more iterestig problem is selectio Fidig the i th smallest elemet of a set We will show A practical radomized algorithm with O() expected ruig time A cool algorithm of theoretical iterest oly with O() worst-case ruig time Algorithms Networkig Laboratory 42/62
Radomized Selectio Key idea: use partitio() from quicksort But, oly eed to examie oe subarray This savigs shows up i ruig time: O() Select the i=7 th smallest i - k Algorithms Networkig Laboratory 43/62
Radomized Selectio k A[q] A[q] p q r Algorithms Networkig Laboratory 44/62
Radomized Selectio Aalyzig Radomized-Select() Worst case: partitio always 0:-1 T() = T(-1) + O() = O( 2 ) (arithmetic series) No better tha sortig! Best case: suppose a 9:1 partitio T() = T(9/10) + O() = O() (Master Theorem, case 3) Better tha sortig! What if this had bee a 99:1 split? Algorithms Networkig Laboratory 45/62
Radomized Selectio Aalysis of expected time The aalysis follows that of radomized quicksort, but it s a little differet. Let T() be the radom variable for the ruig time of Radomized-Select o a iput of size, assumig radom umbers are idepedet. For k = 0, 1,, 1, defie the idicator radom variable Algorithms Networkig Laboratory 46/62
Radomized Selectio To obtai a upper boud, assume that the i th elemet always falls i the larger side of the partitio: Algorithms Networkig Laboratory 47/62
Radomized Selectio Take expectatios of both sides T 1 1 Tmaxk, k 1 2 k 0 1 k / 2 T k Let s show that T() = O() by substitutio Algorithms Networkig Laboratory 48/62
Algorithms Networkig Laboratory 49/62 What happeed here? Split the recurrece What happeed here? What happeed here? What happeed here? Radomized Selectio Assume T(k) ck for sufficietly large c: c c c k k c ck k T T k k k k 1 2 2 1 2 1 2 2 1 1 2 1 2 2 2 ) ( 2 ) ( 1 2 1 1 1 1 2 / 1 2 / The recurrece we started with Substitute T() c for T(k) Expad arithmetic series Multiply it out
Radomized Selectio Assume T(k) ck for sufficietly large c: T ( ) c c 2 2 c c c c 4 2 c c c 4 2 c c c 4 2 c (if c is big eough) 1 1 The recurrece so far What Multiply happeed it out here? What Subtract happeed c/2 here? What Rearrage happeed the arithmetic here? What happeed we set out here? to prove Algorithms Networkig Laboratory 50/62
Selectio i Worst-Case Liear Time Radomized algorithm works well i practice What follows is a worst-case liear time algorithm, really of theoretical iterest oly Basic idea: Geerate a good partitioig elemet Call this elemet x Algorithms Networkig Laboratory 51/62
Selectio i Worst-Case Liear Time The algorithm i words: 1. Divide elemets ito groups of 5 2. Fid media of each group ( How? How log? ) 3. Use Select() recursively to fid media x of the /5 medias 4. Partitio the elemets aroud x. Let k = rak(x) 5. if (i == k) the retur x if (i < k) the use Select() recursively to fid ith smallest elemet i first partitio else (i > k) use Select() recursively to fid (i-k)th smallest elemet i last partitio A: x 1 x 2 x 3 x /5 k 1 elemets x x - k elemets Algorithms Networkig Laboratory 52/62
Order Statistics Aalysis Lesser Elemets Media Greater Elemets Oe group of 5 elemets Algorithms Networkig Laboratory 53/62
Order Statistics Aalysis Lesser Medias Media of Medias Greater Medias All groups of 5 elemets (Ad at most oe smaller group) Algorithms Networkig Laboratory 54/62
Order Statistics Aalysis Defiitely Lesser Elemets Defiitely Greater Elemets Algorithms Networkig Laboratory 55/62
Example Fid the 11th smallest elemet i array: A = {12, 34, 0, 3, 22, 4, 17, 32, 3, 28, 43, 82, 25, 27, 34, 2,19,12,5,18,20,33, 16, 33, 21, 30, 3, 47} Divide the array ito groups of 5 elemets 12 34 0 3 22 4 17 32 3 28 43 82 25 27 34 2 19 12 5 18 20 33 16 33 21 30 3 47 Algorithms Networkig Laboratory 56/62
Example Sort the groups ad fid their medias 0 3 12 22 34 3 4 17 28 32 25 27 34 43 82 2 5 12 18 19 16 20 21 33 33 3 30 47 Algorithms Networkig Laboratory 57/62
Example Fid the media of the medias 0 3 12 22 34 2 5 12 18 19 3 4 17 28 32 16 20 21 33 33 3 30 47 25 27 34 43 82 Algorithms Networkig Laboratory 58/62
Example Partitio the array aroud the media of medias (17) First partitio: {12, 0, 3, 4, 3, 2, 12, 5, 16, 3} Pivot: 17 (positio of the pivot is q = 11) Secod partitio: {34, 22, 32, 28, 43, 82, 25, 27, 34, 19, 18, 20, 33, 33, 21, 30, 47} To fid the 6-th smallest elemet we would have to recur our search i the first partitio Algorithms Networkig Laboratory 59/62
Algorithms Networkig Laboratory 60/62 Aalysis of Ruig Time At least half of the medias foud i step 2 are x All but two of these groups cotribute 3 elemets > x 2groups with 3 elemets > x 5 2 1 6 10 3 2 5 2 1 3 At least elemets greater tha x SELECT is called o at most elemets 6 10 7 6 10 3
Aalysis of Ruig Time Step 1: makig groups of 5 elemets takes Step 2: sortig /5 groups i O(1) time each takes Step 3: callig SELECT o /5 medias takes time Step 4: partitioig the -elemet array aroud x takes Step 5: recursig o oe partitio takes T(7/10 + 6) T() = T(/5) + T(7/10 + 6) + O() O() O() T(/5) O() Show that T() = O() Algorithms Networkig Laboratory 61/62
Aalysis of Ruig Time Let s show that T() = O() by substitutio T() c /5 + c (7/10 + 7) + a c/5 + 7c/10 + 7c + a = 9c/10 + 7c + a = c + (-c/10 + 7c + a) c if: -c/10 + 7c + a 0 Algorithms Networkig Laboratory 62/62