Lecture 5. Counting Sort / Radix Sort

Transcription

1 Lecture 5. Coutig Sort / Radix Sort T. H. Corme, C. E. Leiserso ad R. L. Rivest Itroductio to Algorithms, 3rd Editio, MIT Press, 2009 Sugkyukwa Uiversity Hyuseug Choo choo@skku.edu Copyright Networkig Laboratory

2 Sortig So Far Isertio Sort Easy to code Fast o small iputs (less tha ~ 50 elemets) Fast o early-sorted iputs O( 2 ) worst case O( 2 ) average (equally-likely iputs) case O( 2 ) reverse-sorted case Algorithms Networkig Laboratory 2/62

3 Sortig So Far Merge Sort Divide-ad-Coquer Split array i half Recursively sort subarrays Liear-time merge step O( lg ) worst case It does ot sort i place Algorithms Networkig Laboratory 3/62

4 Sortig So Far Heap Sort It uses the very useful heap data structure Complete biary tree Heap property: paret key > childre s keys O( lg ) worst case It sorts i place Algorithms Networkig Laboratory 4/62

5 Sortig So Far Quick Sort Divide-ad-Coquer Partitio array ito two subarrays, recursively sort All of first subarray < all of secod subarray No merge step eeded! O( lg ) average case Fast i practice O( 2 ) worst case Naïve implemetatio: worst case o sorted iput Address this with radomized quicksort Algorithms Networkig Laboratory 5/62

6 How Fast Ca We Sort? We will provide a lower boud, the beat it How do you suppose we ll beat it? First, a observatio: all of the sortig algorithms so far are compariso sorts The oly operatio used to gai orderig iformatio about a sequece is the pairwise compariso of two elemets Theorem: all compariso sorts are ( lg ) Algorithms Networkig Laboratory 6/62

7 Decisio Trees Algorithms Networkig Laboratory 7/62

8 Decisio Trees Decisio trees provide a abstractio of compariso sorts A decisio tree represets the comparisos made by a compariso sort. What do the leaves represet? How may leaves must there be? Algorithms Networkig Laboratory 8/62

9 Practice Problems What is the smallest possible depth of a leaf i a decisio tree for a compariso sort? Algorithms Networkig Laboratory 9/62

10 Decisio Trees Decisio trees ca model compariso sorts For a give algorithm: Oe tree for each Tree paths are all possible executio traces What s the logest path i a decisio tree for isertio sort? For merge sort? What is the asymptotic height of ay decisio tree for sortig elemets? Aswer: ( lg ) (Now let s prove it ) Algorithms Networkig Laboratory 10/62

11 A Lower Boud for Compariso Sortig Theorem: Ay decisio tree that sorts elemets has height ( lg ) What s the miimum umber of leaves? What s the maximum umber of leaves of a biary tree of height h? Clearly the miimum umber of leaves is less tha or equal to the maximum umber of leaves Algorithms Networkig Laboratory 11/62

12 A Lower Boud for Compariso Sortig So we have! 2 h Takig logarithms: lg (!) h Stirlig s approximatio tells us: Thus:! e h lg e Algorithms Networkig Laboratory 12/62

13 A Lower Boud for Compariso Sortig So we have h lg e lg lg lg e Thus the miimum height of a decisio tree is ( lg ) Algorithms Networkig Laboratory 13/62

14 A Lower Boud for Compariso Sorts Thus the time to compariso sort elemets is ( lg ) Corollary: Heapsort ad Mergesort are asymptotically optimal compariso sorts But the ame of this lecture is Sortig i liear time! How ca we do better tha ( lg )? Algorithms Networkig Laboratory 14/62

15 Sortig i Liear Time Coutig Sort No comparisos betwee elemets! But depeds o assumptio about the umbers beig sorted Basic Idea For each iput elemet x, determie the umber of elemets less tha or equal to x For each iteger i (0 i k), cout how may elemets whose values are i The we kow how may elemets are less tha or equal to i Algorithm Storage A[1..]: iput elemets B[1..]: sorted elemets C[0..k]: hold the umber of elemets less tha or equal to i Algorithms Networkig Laboratory 15/62

16 Coutig Sort Illustratio (Rage from 0 to 5) Algorithms Networkig Laboratory 16/55

17 Coutig Sort Illustratio A B B C C B B B C C C B C Algorithms Networkig Laboratory 17/62

18 Coutig Sort (k) () (k) () (+k) Algorithms Networkig Laboratory 18/62

19 Coutig Sort Algorithms Networkig Laboratory 19/62

20 Coutig Sort Video Cotet A illustratio of Coutig Sort. Algorithms Networkig Laboratory 20/62

21 Coutig Sort Algorithms Networkig Laboratory 21/62

22 Coutig Sort Total time: O( + k) Usually, k = O() Thus coutig sort rus i O() time But sortig is ( lg )! No cotradictio This is ot a compariso sort I fact, there are o comparisos at all! Notice that this algorithm is stable Algorithms Networkig Laboratory 22/62

23 Radix Sort Ituitively, you might sort o the most sigificat digit, the the secod msd, etc. Problem Lots of itermediate piles of cards (read: scratch arrays) to keep track of Key idea Sort the least sigificat digit first RadixSort(A, d) for i=1 to d StableSort(A) o digit i Algorithms Networkig Laboratory 23/62

24 Radix Sort Example Iput data: a b a b a c c a a a c b b a b c c a a a c b b a Algorithms Networkig Laboratory 24/62

25 Radix Sort Example Pass 1: Lookig at rightmost positio. a b a b a c c a a a c b b a b c c a a a c b b a Place ito appropriate pile. a b c Algorithms Networkig Laboratory 25/62

26 Radix Sort Example Pass 1: Lookig at rightmost positio. b b a Joi piles. c c a c a a b a b a a c a b a a c b b a c a b c Algorithms Networkig Laboratory 26/62

27 Radix Sort Example Pass 2: Lookig at ext positio. a b a c a a c c a b b a a c b b a b b a c a a c Place ito appropriate pile. a b c Algorithms Networkig Laboratory 27/62

28 Radix Sort Example Pass 2: Lookig at ext positio. a a c Joi piles. b a c b a b b b a a c b c a a a b a c c a a b c Algorithms Networkig Laboratory 28/62

29 Radix Sort Example Pass 3: Lookig at last positio. c a a b a b b a c a a c a b a b b a c c a a c b Place ito appropriate pile. a b c Algorithms Networkig Laboratory 29/62

30 Radix Sort Example Pass 3: Lookig at last positio. Joi piles. a c b a b a b b a b a c c c a a a c b a b c a a a b c Algorithms Networkig Laboratory 30/62

31 Radix Sort Example Result is sorted. a a c a b a a c b b a b b a c b b a c a a c c a Algorithms Networkig Laboratory 31/62

32 Radix Sort Example Radix Bis Algorithms Networkig Laboratory 32/62

33 Radix Sort Example d digits i total Algorithms Networkig Laboratory 33/62

34 Radix Sort Coutig sort is obvious choice: Sort umbers o digits that rage from 1..k Time: O( + k) Each pass over umbers with d digits takes time O(+k), so total time O(d+dk) Whe d is costat ad k=o(), takes O() time I geeral, radix sort based o coutig sort is Fast Asymptotically fast (i.e., O()) Simple to code Algorithms Networkig Laboratory 34/62

35 Radix Sort Video Cotet A illustratio of Radix Sort. Algorithms Networkig Laboratory 35/62

36 Radix Sort Algorithms Networkig Laboratory 36/62

37 Practice Problems Show how to sort itegers i the rage 0 to 3 1 i O() time. Algorithms Networkig Laboratory 37/62

38 Medias ad Order Statistics The ith order statistic of a set of elemets is the ith smallest elemet The media is the halfway poit Defie the selectio problem as: Give a set A of elemets, fid a elemet x A that is larger tha x-1 elemets Obviously, this ca be solve i O( lg ). Why? Is it really ecessary to sort all the umbers? Algorithms Networkig Laboratory 38/62

39 Order Statistics The ith order statistic i a set of elemets is the ith smallest elemet The miimum is thus the 1st order statistic The maximum is the th order statistic The media is the /2 order statistic If is eve, there are 2 medias The lower media (+1)/2 The upper media (+1)/2 How ca we calculate order statistics? What is the ruig time? Algorithms Networkig Laboratory 39/62

40 Miimum ad Maximum The maximum ca be foud i exactly the same way by replacig the > with < i the above algorithm. Algorithms Networkig Laboratory 40/62

41 Simultaeous Mi ad Max How may comparisos are eeded to fid the miimum elemet i a set? The maximum? Ca we fid the miimum ad maximum with less tha twice the cost? Yes: Walk through elemets by pairs Compare each elemet i pair to the other Compare the largest to maximum, smallest to miimum Pair (a i, a i+1 ), assume a i < a i+1, the Compare (mi,a i ), (a i+1,max) 3 comparisios for each pair. Total cost: 3 comparisos per 2 elemets O( 3 /2 ) Algorithms Networkig Laboratory 41/62

42 The Selectio Problem A more iterestig problem is selectio Fidig the i th smallest elemet of a set We will show A practical radomized algorithm with O() expected ruig time A cool algorithm of theoretical iterest oly with O() worst-case ruig time Algorithms Networkig Laboratory 42/62

43 Radomized Selectio Key idea: use partitio() from quicksort But, oly eed to examie oe subarray This savigs shows up i ruig time: O() Select the i=7 th smallest i - k Algorithms Networkig Laboratory 43/62

44 Radomized Selectio k A[q] A[q] p q r Algorithms Networkig Laboratory 44/62

45 Radomized Selectio Aalyzig Radomized-Select() Worst case: partitio always 0:-1 T() = T(-1) + O() = O( 2 ) (arithmetic series) No better tha sortig! Best case: suppose a 9:1 partitio T() = T(9/10) + O() = O() (Master Theorem, case 3) Better tha sortig! What if this had bee a 99:1 split? Algorithms Networkig Laboratory 45/62

46 Radomized Selectio Aalysis of expected time The aalysis follows that of radomized quicksort, but it s a little differet. Let T() be the radom variable for the ruig time of Radomized-Select o a iput of size, assumig radom umbers are idepedet. For k = 0, 1,, 1, defie the idicator radom variable Algorithms Networkig Laboratory 46/62

47 Radomized Selectio To obtai a upper boud, assume that the i th elemet always falls i the larger side of the partitio: Algorithms Networkig Laboratory 47/62

48 Radomized Selectio Take expectatios of both sides T 1 1 Tmaxk, k 1 2 k 0 1 k / 2 T k Let s show that T() = O() by substitutio Algorithms Networkig Laboratory 48/62

49 Algorithms Networkig Laboratory 49/62 What happeed here? Split the recurrece What happeed here? What happeed here? What happeed here? Radomized Selectio Assume T(k) ck for sufficietly large c: c c c k k c ck k T T k k k k ) ( 2 ) ( / 1 2 / The recurrece we started with Substitute T() c for T(k) Expad arithmetic series Multiply it out

50 Radomized Selectio Assume T(k) ck for sufficietly large c: T ( ) c c 2 2 c c c c 4 2 c c c 4 2 c c c 4 2 c (if c is big eough) 1 1 The recurrece so far What Multiply happeed it out here? What Subtract happeed c/2 here? What Rearrage happeed the arithmetic here? What happeed we set out here? to prove Algorithms Networkig Laboratory 50/62

51 Selectio i Worst-Case Liear Time Radomized algorithm works well i practice What follows is a worst-case liear time algorithm, really of theoretical iterest oly Basic idea: Geerate a good partitioig elemet Call this elemet x Algorithms Networkig Laboratory 51/62

52 Selectio i Worst-Case Liear Time The algorithm i words: 1. Divide elemets ito groups of 5 2. Fid media of each group ( How? How log? ) 3. Use Select() recursively to fid media x of the /5 medias 4. Partitio the elemets aroud x. Let k = rak(x) 5. if (i == k) the retur x if (i < k) the use Select() recursively to fid ith smallest elemet i first partitio else (i > k) use Select() recursively to fid (i-k)th smallest elemet i last partitio A: x 1 x 2 x 3 x /5 k 1 elemets x x - k elemets Algorithms Networkig Laboratory 52/62

53 Order Statistics Aalysis Lesser Elemets Media Greater Elemets Oe group of 5 elemets Algorithms Networkig Laboratory 53/62

54 Order Statistics Aalysis Lesser Medias Media of Medias Greater Medias All groups of 5 elemets (Ad at most oe smaller group) Algorithms Networkig Laboratory 54/62

55 Order Statistics Aalysis Defiitely Lesser Elemets Defiitely Greater Elemets Algorithms Networkig Laboratory 55/62

56 Example Fid the 11th smallest elemet i array: A = {12, 34, 0, 3, 22, 4, 17, 32, 3, 28, 43, 82, 25, 27, 34, 2,19,12,5,18,20,33, 16, 33, 21, 30, 3, 47} Divide the array ito groups of 5 elemets Algorithms Networkig Laboratory 56/62

57 Example Sort the groups ad fid their medias Algorithms Networkig Laboratory 57/62

58 Example Fid the media of the medias Algorithms Networkig Laboratory 58/62

59 Example Partitio the array aroud the media of medias (17) First partitio: {12, 0, 3, 4, 3, 2, 12, 5, 16, 3} Pivot: 17 (positio of the pivot is q = 11) Secod partitio: {34, 22, 32, 28, 43, 82, 25, 27, 34, 19, 18, 20, 33, 33, 21, 30, 47} To fid the 6-th smallest elemet we would have to recur our search i the first partitio Algorithms Networkig Laboratory 59/62

60 Algorithms Networkig Laboratory 60/62 Aalysis of Ruig Time At least half of the medias foud i step 2 are x All but two of these groups cotribute 3 elemets > x 2groups with 3 elemets > x At least elemets greater tha x SELECT is called o at most elemets

61 Aalysis of Ruig Time Step 1: makig groups of 5 elemets takes Step 2: sortig /5 groups i O(1) time each takes Step 3: callig SELECT o /5 medias takes time Step 4: partitioig the -elemet array aroud x takes Step 5: recursig o oe partitio takes T(7/10 + 6) T() = T(/5) + T(7/10 + 6) + O() O() O() T(/5) O() Show that T() = O() Algorithms Networkig Laboratory 61/62

62 Aalysis of Ruig Time Let s show that T() = O() by substitutio T() c /5 + c (7/10 + 7) + a c/5 + 7c/10 + 7c + a = 9c/10 + 7c + a = c + (-c/10 + 7c + a) c if: -c/10 + 7c + a 0 Algorithms Networkig Laboratory 62/62