Sorting Algorithms. Dipartimento di Elettronica e Informazione Politecnico di Milano June 5, 2018

Transcription

1 Sorting Algorithms Dipartimento di Elettronica e Informazione Politecnico di Milano nicholas.mainardi@polimi.it June 5, 2018

2 Selection Sort Recall At each iteration, the considered sub-array is composed the elements which have not being sorted yet Thus, at the first iteration, the whole array is considered Each iteration computes the minimum and swaps it with the first element of the sub-array For the next iteration, the sub-array starting at the second element of the current sub-array is considered Repeat until the length of the sub-array is 1 Its complexity is Θ(n 2 ), but it is generally slower than insertion sort since the minimum always requires a whole scan of the sub-array to be computed, while insetion sort is linear if the array is already or nearly sorted

3 Selection Sort Considerations However, O(n) swaps are required instead of O(n 2 ) of insertion sort, since the minimum element is always placed in its final position Can we do it better? Asymptotically not! Indeed, if each element of the array is placed in a different position than its one in the sorted array, then we need to write each element to its correct position, thus at least n writes Nevertheless, we can improve the actual number of writes with another algorithm: cycle sort

4 Cycle Sort High Level Idea For each element, we can compute its final position i with a linear scan of the array, and write it to i (not swapping) Then, we determine the final position of the elements being overwritten in position i, and so on In this way, we write each element only in its final position, performing at most n writes Some technicalities: The first element is duplicated, since we do not overwrite it when we copy it to its final position Thus, when we get the element whose final position is 1, we overwrite it without searching for its final position Instead, we start again the algorithm on the sub-array starting from the second element

5 Cycle Sort Algorithm cyclesort(a[1,..., N]) 1 for start 1 to N 1 2 do write i start 3 item A[start] 4 for i start + 1 to N 5 do if A[i] < item 6 then write i write i if write i = start 8 then continue 9 while A[write i ] = item 10 do write i write i swap(item, A[write i ]) 12 while start write i 13 do write i start 14 for i start + 1 to N 15 do if A[i] < item 16 then write i write i while A[write i ] = item 18 do write i write i swap(item, A[write i ])

6 Cycle Sort Example A = [97, 155, 127, 97, 175, 146, 90, 13, 151, 146, 19, 97] 1 A = [97, 155, 127, 97, 97, 146, 90, 13, 151, 146, 19, 97], item = A = [97, 155, 127, 97, 97, 146, 90, 13, 151, 146, 19, 175], item = 97 3 A = [97, 155, 127, 97, 97, 97, 90, 13, 151, 146, 19, 175], item = A = [97, 155, 127, 97, 97, 97, 90, 146, 151, 146, 19, 175], item = 13 5 A = [13, 155, 127, 97, 97, 97, 90, 146, 151, 146, 19, 175], item = A = [13, 155, 127, 97, 97, 97, 90, 146, 151, 146, 155, 175], item = 19 7 A = [13, 19, 127, 97, 97, 97, 90, 146, 151, 146, 155, 175], item = A = [13, 19, 127, 97, 97, 97, 127, 146, 151, 146, 155, 175], item = 90 9 A = [13, 19, 90, 97, 97, 97, 127, 146, 151, 146, 155, 175], item = A = [13, 19, 90, 97, 97, 97, 127, 146, 151, 146, 155, 175], item = A = [13, 19, 90, 97, 97, 97, 127, 146, 151, 146, 155, 175], item = A = [13, 19, 90, 97, 97, 97, 127, 146, 151, 146, 155, 175], item = A = [13, 19, 90, 97, 97, 97, 127, 146, 151, 146, 155, 175], item = A = [13, 19, 90, 97, 97, 97, 127, 146, 151, 146, 155, 175], item = A = [13, 19, 90, 97, 97, 97, 127, 146, 151, 151, 155, 175], item = A = [13, 19, 90, 97, 97, 97, 127, 146, 146, 151, 155, 175], item = A = [13, 19, 90, 97, 97, 97, 127, 146, 146, 151, 155, 175], item = A = [13, 19, 90, 97, 97, 97, 127, 146, 146, 151, 155, 175]

7 Cycle Sort Complexity analysis First of all, let s take a look at the for loop at lines 4 6 Independently from the structure of the array, this is executed O(N) times Therefore, for sure T (N) Ω(N 2 ), even in the best case where the array is already sorted Can we get worse than O(N 2 )? Let s take a look at the while loop at lines This loop actually writes the elements to their final positions Since we know, by construction, that at most N writes are performed, then this loop is executed O(N) times throughout the whole algorithm, not for every iteration of the external loop Loop body is O(N), due to for loop at lines Thus, the time complexity of the while loop is O(N 2 ) As a conclusion, T (n) = Θ(N 2 )

8 Bubble Sort Really simple idea: compares pair of adjacent elements and sort them, until all pairs are sorted Algorithm bubblesort(a[1,..., N]) 1 swap True 2 while swap 3 do swap False 4 for i 1 to N 1 5 do if A[i] > A[i + 1] 6 then swap(a[i], A[i + 1]) 7 swap True

9 Bubble Sort A = [151, 185, 122, 28, 20, 28, 36, 28, 38, 194, 38, 5] Example 1 A = [151, 122, 28, 20, 28, 36, 28, 38, 185, 38, 5, 194] 2 A = [122, 28, 20, 28, 36, 28, 38, 151, 38, 5, 185, 194] 3 A = [28, 20, 28, 36, 28, 38, 122, 38, 5, 151, 185, 194] 4 A = [20, 28, 28, 28, 36, 38, 38, 5, 122, 151, 185, 194] 5 A = [20, 28, 28, 28, 36, 38, 5, 38, 122, 151, 185, 194] 6 A = [20, 28, 28, 28, 36, 5, 38, 38, 122, 151, 185, 194] 7 A = [20, 28, 28, 28, 5, 36, 38, 38, 122, 151, 185, 194] 8 A = [20, 28, 28, 5, 28, 36, 38, 38, 122, 151, 185, 194] 9 A = [20, 28, 5, 28, 28, 36, 38, 38, 122, 151, 185, 194] 10 A = [20, 5, 28, 28, 28, 36, 38, 38, 122, 151, 185, 194] 11 A = [5, 20, 28, 28, 28, 36, 38, 38, 122, 151, 185, 194]

10 Bubble Sort Complexity The for loop runs N 1 times How many iterations before no more swaps occur? Consider the worst case: the minimum is in the last position At each iteration, it goes back by 1 position Since it is the minimum, it will swap until it gets to the first position N 1 swaps! Time complexity is O(N 2 ) What about the average case? (lower bound is sufficient) Consider the minimum again, and its position i, uniformly chosen from [1,..., N] i 1 swaps are needed to get the minimum in the first position The average complexity is N i=1 N(i 1) N = N N i=1 (i 1) N = N 1 i=0 i Ω(N2 )

11 Bubble Sort/Shaker Sort An optimization The little elements slowly move to the beginning of the array, but the big ones rapidly move to the end Why? Because comparisons are performed from left to right We can move also the little elements quite rapidly if we perform comparisons from right to left too This version of the algorithm is called shakersort What is the worst case now? When the array is reversely ordered, since at each iteration from left to right, the maximum is the first element, thus the only one being moved. Conversely, the minimum is the first element in iterations from right to left Thus, we still need O(n) iterations Still O(N 2 ), but it is surely faster than bubblesort

12 Bogo Sort bogosort is also called stupidsort, since the idea is trading efficiency for simplicity of the algorithm bogosort performs a random permutation of the array, and then it checks if it is sorted Complexity To check if an array is sorted, it is sufficient to verify that i, A[i] < A[i + 1] O(N) The algorithm runs until the correct permutation is found Since there are N! permutations, each of them selected with 1 N! probability, on average the right permutation is performed after N! trials Thus, average case complexity is O(N N!) What about worst case? Since the algorithm is probabilistic, the right permutation may never be found! T (n) =

13 Worst Sort An interesting target for this algorithm: design the worst sorting algorithm possible worstsort is based on badsort badsort badsort(l, k) 1 if k = 0 2 then bubblesort(l) 3 else P allpermutations(l) 4 badsort(p, k 1) 5 return L[1] Where allpermutations(l) returns the list of all possible permutations of L P is a list of lists!

14 Worst Sort Complexity - Analyze Recursion badsort is a recursive procedure: { Ω(n 2 ) if k = 0 T (n, k) = T (n!, k 1) + nn! if k > 0 Lets expand the recursion: T (n, k) = T (n!, k 1)+nn! = T ((n!)!, k 2)+n!(n!)!+nn! = = T (n! (k), 0) + k i=1 n!(i 1) n! (i) ), where n! (k) means that the factorial operator is applied k times (e.g. n! (3) = ((n!)!)!) T (n, 0) = Ω(n 2 ), therefore T (n! (k), 0) = Ω((n! (k) ) 2 ) In conclusion, T (n, k) = Ω((n! (k) ) 2 ) + k i=1 n!(i 1) n! (i) Ω((n! (k) ) 2 )

15 Worst Sort Complexity - Proof T (n, k) c(n! (k) ) 2 Since the recursion is done on k, not on n, the inductive hypothesis is done only on k, since it holds for every n Hp: T (n, k 1) c((n)! (k 1) ) 2 T (n, k) = T (n!, k 1) + nn! c((n!)! (k 1) ) 2 + nn! = c(n! (k) ) 2 + nn! c(n! (k) ) 2

16 Worst Sort From badsort to worstsort How can we worsen the badsort complexity? Deciding the k to be used! In particular, given a generic computable function f : N N, worstsort is defined as badsort(l, f (length(l))) Since for badsort, T (n, k) = Ω((n! (k) ) 2 ), the time complexity of worstsort is: T (n) = Ω((n! (f (n)) ) 2 ) + T f (n), where T f (n) is the time complexity of f Thus, its time complexity cannot be bounded by a computable function! In particular, since this time complexity is a computable function itself, it is bounded by Busy Beaver, which is a monotone increasing not computable function proven to be greater than every possible computable function

17 Shell Sort Idea of the Algorithm Insertion sort poorly performs since a little element found at the end of the array needs n comparisons to be moved at the beginning (and vice versa) Shellsort tries to move little and big elements, respectively, to the beginning or end of the array faster (i.e. with less comparisons) The array is re-arranged in a matrix of h columns, and elements inside each column are sorted, moving little elements to the top of the columns, that is at the beginning of the array The previous step is repeated with decreasing h, till the last step where h = 1 In this last step, the array is already quasi-sorted, thus insertionsort performs well, requiring few comparisons

18 Shell Sort How is h decreased? This is not a trivial question, which greatly impact the performances In its original version, Shell employed h = 2 k, k = N,..., 1 Algorithm shellsort(a[0,..., N 1]) 1 h 2 N 2 while h > 0 3 do for i h to N 1 4 do j i 5 while j h A[j h] > A[j] 6 do swap(a[j h], A[j]) 7 j j h 8 h h 2

19 Shell Sort Example A = [34, 21, 324, 5, 4, 34, 5, 2, 6, 7, 4, 3, 2, 23, 56, 74, 68, 4] k(2 k > N), the loop at line 3 is never executed We start from h = 16:

20 Shell Sort Example - h = Example - h =

21 Shell Sort Example - h = Example - h =

22 Shell Sort Time Complexity Hint: 2 elements found in positions i, j are sorted with gap h if (i mod h = j mod h) if h = 2 k, k = N,..., 1, then if i is even and j is odd, i mod h = j mod h only for h = 1 Thus, when h = 1, each element in an even position has to be compared with an element in an odd position, and vice versa In the worst case, N 2 elements in odd positions are big, while elements in even positions are little N 2 In such a case, the i th little element has to be compared with i 2 odd elements to get to its position The complexity is thus N 2 i=1 i 2 = 1 2 N 2 i=1 i = N 2 ( N 2 +1) 4 O(N 2 )

23 Shell Sort Different sequences for h allows to get better time complexity: Consider h = 2 k 1, k = N,..., 1 Algorithm shellsort(a[0,..., N 1]) 1 h 2 N 1 2 while h > 0 3 do for i h to N 1 4 do j i 5 while j h A[j h] > A[j] 6 do swap(a[j h], A[j]) 7 j j h 8 h h 2 + 1

24 Shell Sort What is the time complexity with such a sequence? Generic Upper Bound for h-sorting An array is said to be h-sorted if i(a[i] a[i + h]), with h being called the gap If the gap is h, all i elements, starting from i = h are analyzed Each element is compared with all the j < i(j mod h), that is with i h elements mod h = i Therefore, a generic intra-column sorting of gap h costs N i=h i h N i=h i h = 1 N N2 h i=h i = O( h ) Note that this is independent from the sequence of h being considered

25 Shell Sort An Intermediate Result Consider an array which is h, h -sorted, with gcd(h, h ) = 1 We can prove that i, j(j i (h 1)(h 1) a[i] a[j] This is basically due to the fact that if an array is both h and h sorted, then all α, β N(the array is αh + βh -sorted) Moreover, if gcd(h, h ) = 1, then x (h 1)(h 1)( α, β(x = αh + βh )) Thus, if k = j i (h 1)(h 1), then the array is k-sorted Therefore, suppose we want to perform an h-sorting of the array, given that the array is already d, d -sorted, with gcd(d, d ) = 1 and h = O(d) For each element i, we need to compare it just with the subsequent (d 1)(d 1) elements, but since the gap is h = O(d), then actually O(d) comparisons are performed As a conclusion, an h-sorting requires O(Nd) comparisons

26 Shell Sort Complexity Analysis - 1 The previous result actually applies to the considered gap sequence Indeed, if we are h k = 2 k 1-sorting, the array has been already been h k+1 = 2 k+1 1, h k+2 = 2 k+2 1 sorted, and gcd(h k+1, h k+2 ) = 1. Moreover, h k h k+2 4 = O(h k+2 ) Thus, so far we got 2 upper bounds, one which is generic, and the other which is specific to certain gap sequences To estimate the complexity, for each h-sorting, we can always choose to use the minimum upper bound between these 2 Which one is the minimum for each h? > Nh N2 > Nh 2 h 2 < N h < N N 2 h Thus, for h N, h-sorting is O(Nh), while for h > N, h-sorting is O( N2 h )

27 Shell Sort Complexity Analysis - 2 Define now h t = max i (2 i 1 N) We use O(Nh k ) for every h k, k t, whilst using O( N2 h k ) for every h k, t < k N First case: t k=0 cnh k = t k=0 cn(2k 1) cn t k=0 (2k ) = cn2 t+1 O(N N) Second case: N k=t c N2 = N h k k=t c N2 2 k 1 2cN2 N k=t t 1 ) = 4cN 2 ( ) 4cN 2 1 t 2 N 2 O(4cN 2 1 t 2 1 = 2cN 2 ( k 2 N 1 2 Thus, the shell sort complexity with gap sequence h = 2 k 1, k = N,..., 1 is O(n 3 2 ) N ) = O(N N)

28 Radix Sort Idea: sort by digit! Each element of the array is expressed in a base b Starting from the least significant digit, the array is sorted by digits of its elements. How? b buckets are used. Atj th iteration, the i th bucket contains the set of elements having the j th digit, starting from the least significant one, equal to i At each iteration, with a single scan of the array, we can place each element in its bucket Then, the content of the buckets is wrote back to the array, starting from bucket 0 up to bucket b In the next iteration, the same procedure is done by considering the next digit to place elements in the buckets (if necessary, assume elements have non significant 0 digits) The array is sorted when there are mo more digits for every element

29 Radix Sort Algorithm radixsort(a) 1 bucket[0,..., b 1] b is the base employed to represent elements of the array 2 digit = 1 3 finished 1 4 while finished > 0 5 do finished 0 6 for i 1 to N 7 do index A[i] digit mod b 8 finished finished + A[i] digit 9 append(bucket[index], A[i]) 10 write 1 11 for index 0 to b 1 12 do l length(bucket[index]) 13 for j 1 to l 14 do A[write] get(bucket[index], 0) 15 remove(bucket[index], 0) 16 write write digit digit b

30 Radix Sort A = [4, 87, 60, 180, 199, 60, 37, 87, 168, 37, 103, 37] Example - First Digit Buckets 0 [60, 180, 60] 3 [103] 4 [4] 7 [87, 37, 87, 37, 37] 8 [168] 9 [199] A = [60, 180, 60, 103, 4, 87, 37, 87, 37, 37, 168, 199]

31 Radix Sort Example - Second Digit Buckets 0 [103, 4] 3 [37, 37, 37] 6 [60, 60, 168] 8 [180, 87, 87] 9 [199] A = [103, 4, 37, 37, 37, 60, 60, 168, 180, 87, 87, 199] Example - Third Digit Buckets 0 [4, 37, 37, 37, 60, 60, 87, 87] 1 [103, 168, 180, 199] A = [4, 37, 37, 37, 60, 60, 87, 87, 103, 168, 180, 199] Last step: Digits are finished, thus algorithm ends

32 Radix Sort Complexity Analysis Complexity of the list management functions: First of all, suppose we have a pointer to the end of the list in each bucket append has O(1) time complexity get and remove are O(j) But, since j is always the first element of the list, get and remove are O(1) in our algorithm The for loop at lines 6 9 runs N times, and its body is O(1) The for loop at lines writes back each element in A, thus it runs N times in total, and its loop body is O(1) How many iterations for the while loop? At each iteration, a different digit is employed for sorting it runs w = log b (max(a)) times The total complexity is thus O(nw)

33 Radix Sort Considerations As for countingsort, the algorithm may be faster or slower than O(n log(n)) comparison sorting methods depending on the constant w It is sufficient that all the elements are different to get a complexity Ω(n log b (n)) (since max(a) N) However, it is possible to change the base b being considered, to get the complexity back to linear, since O(n log n (n)) = O(n) But on the other way, we need b buckets, thus if b is larger, more memory is going to be required It is the usual time/memory tradeoff

34 Summary A comparison Algorithm Complexity (worst) Space Adaptive Insertion O(n 2 ) O(1) O(n) Shell O(n 3 2 ) O(1) O(nlog(n)) Bubble O(n 2 ) O(1) Θ(n) Selection Θ(n 2 ) (O(n) swap) O(1) No Cycle Θ(n 2 ) (O(n) write) O(1) No Merge Θ(nlog(n)) Θ(n) No Quick O(n 2 ) O(log(n)) O(n log(n)) Heap O(n log(n)) O(1) No