Sorting Popular algorithms: Selection sort* Insertion sort* Bubble sort* Quick sort* Comb-sort Shell-sort Heap sort* Merge sort* Counting-sort Radix-sort Bucket-sort Tim-sort Many algorithms for sorting in parallel also exist. 1
Properties of Sorting Algorithms A sorting algorithm is said to sort in place if it rearranges numbers within a single array, with at most a constant number of them stored outside the array at any given time. A sorting algorithm is said to be stable if numbers with the same value appear in the output in the same order as they do in the input. Any given sorting algorithm can be either, neither, or both. 2
Selection Sort Sorting Algorithm #1: Selection sort Very easy to understand and implement. Not very efficient. 3
Selection Sort // Selection sort in Java public static void sort(int[] a){ int minpos, temp; for (int i=0; i<=a.length-2; i++){ // Find the position of the value that belongs in position i minpos = i; for (int j=i+1; j<=a.length-1; j++) if (a[j] < a[minpos]) minpos = j; } } // Swap the values in positions i and min temp = a[i]; a[i] = a[minpos]; a[minpos] = temp; Running time? Θ(n 2 ) best, worst, average 4
Selection Sort 5
Insertion Sort Sorting Algorithm #2: Insertion sort Also very easy to understand and implement. Also not very efficient. 6
Insertion Sort Initial version: // Insertion sort public static void insertionsort(int[] a) { int j; } for (int i=1; i<=a.length-1; i++) { j=i; while (j>=1) { if (a[j] < a[j-1]) { temp=a[j-1]; a[j-1]=a[j]; a[j]=temp; } j=j-1; } } Running time? Θ(n 2 ) best, worst, average Analysis is same as for Selection-sort 7
Insertion Sort Second version: // This one eliminates the boolean variable public static void insertionsort(int[] a) { int j; } for (int i=1; i<=a.length-1; i++) { } j=i; } while ((j>=1) && (a[j]<a[j-1])) { temp=a[j-1]; a[j-1]=a[j]; a[j]=temp; j = j 1; Running time? Θ(n 2 ) worst (list in reverse order) Θ(n) best (list already sorted) 8
Insertion Sort More Technically, assuming the list is already sorted On the i th iteration of the outer loop, as i goes from 1 to a.length-1, the inner loop executes exactly 1 time. This gives a total of n iterations of the inner loop. Again, note that we only counted the number of iterations of the inner loop. 9
Insertion Sort Third version: // Another slight improvement in efficiency public static void insertionsort(int[] a) { int j, v; } for (int i=1; i<=a.length-1; i++) { j=i; v = a[j]; while ((j>=1) && (v<a[j-1])) { a[j]=a[j-1]; j=j-1; } a[j] = v; } Running time? Θ(n 2 ) worst (list in reverse order) Θ(n) best (list already sorted) 10
Bubble Sort Sorting Algorithm #3: Bubble sort Also very easy to understand and implement. Also not very efficient. Several minor variations and enhancements are possible. 11
Bubble Sort Initial version: // Bubble sort public static void bubblesort1(int[] a) { int temp; } for (int i=1; i<=a.length-1; i++) { } for (int j=0; j<a.length-i; j++) { } if (a[j] > a[j+1]) { } temp = a[j]; a[j] = a[j+1]; a[j+1] = temp; Running time? Θ(n 2 ) best, worst, average Analysis is same as for Selection-sort 12
Bubble Sort Second version: (fewer bubbles) // This version stops when a pass occurs with no swaps. public static void bubblesort1(int[] a) { int i, temp; boolean domore; } i = 1; domore = true; while ((i<=a.length-1) && (domore)) { domore = false; for (int j=0; j<a.length-i; j++) if (a[j] > a[j+1]) { temp = a[j]; a[j] = a[j+1]; a[j+1] = temp; domore = true; } i = i + 1; } Running time? Θ(n 2 ) worst (list in reverse order) Θ(n) best (list already sorted) 13
Bubble Sort More Technically, assuming the list is already sorted On the 1st iteration of the outer loop, inner loop executes exactly n times. The outer loops only executes 1. Again, note that we only counted the number of iterations of the inner loop. 14
Quick Sort Sorting Algorithm #4: Quick sort Proposed by C.A.R. Hoare in 1962. Divide-and-conquer algorithm. More efficient than selection, insertion, or bubble sort, on average. Worst case is just as bad - Θ(n 2 ) Very practical. 15
Quick Sort 1. Divide: Partition the array into two subarrays around a pivot x such that elements in lower subarray x elements in upper subarray. 2. Conquer: Recursively sort the two subarrays. 3. Combine: Trivial. Key: Linear-time partitioning subroutine. x xx x x x 16
Quick Sort Partitioning algorithm from the book: PARTITION(A, p, q) A[ p..q] x A[p] pivot = A[p] i p for j p + 1 to q do if A[ j] x then i i + 1 exchange A[i] A[ j] exchange A[ p] A[i] return i 17
Quick Sort 6 10 13 5 8 3 2 11 i j 18
Quick Sort 6 10 13 5 8 3 2 11 i j 19
Quick Sort 6 10 13 5 8 3 2 11 i j 20
Quick Sort 6 10 13 5 8 3 2 11 6 5 13 10 8 3 2 11 i j 21
Quick Sort 6 10 13 5 8 3 2 11 6 5 13 10 8 3 2 11 i j 22
Quick Sort 6 10 13 5 8 3 2 11 6 5 13 10 8 3 2 11 i j 23
Quick Sort 6 10 13 5 8 3 2 11 6 5 13 10 8 3 2 11 6 5 3 10 8 13 2 11 i j 24
Quick Sort 6 10 13 5 8 3 2 11 6 5 13 10 8 3 2 11 6 5 3 10 8 13 2 11 i j 25
Quick Sort 6 10 13 5 8 3 2 11 6 5 13 10 8 3 2 11 6 5 3 10 8 13 2 11 6 5 3 2 8 13 10 11 i j 26
Quick Sort 6 10 13 5 8 3 2 11 6 5 13 10 8 3 2 11 6 5 3 10 8 13 2 11 6 5 3 2 8 13 10 11 i j 27
Quick Sort 6 10 13 5 8 3 2 11 6 5 13 10 8 3 2 11 6 5 3 10 8 13 2 11 6 5 3 2 8 13 10 11 i j 28
Quick Sort 6 10 13 5 8 3 2 11 6 5 13 10 8 3 2 11 6 5 3 10 8 13 2 11 6 5 3 2 8 13 10 11 2 5 3 6 8 13 10 11 i 29
QUICKSORT(A, p, r) if p < r then q PARTITION(A, p, r) QUICKSORT(A, p, q 1) QUICKSORT(A, q+1, r) Initial call: QUICKSORT(A, 1, n) 30
Quick Sort 31
Quick Sort What would partition do in the worst-case? 3 5 8 10 11 13 21 35 i j 32
Quick Sort Recursion Tree T(n) T(n) = T(0) + T(n 1) + cn 33
Worst Case Recursion Tree cn T(0) T(n 1) T(n) = T(0) + T(n 1) + cn 34
Quick Sort Recursion Tree T(n) = T(0) + T(n 1) + cn cn T(0) c(n 1) T(0) T(n 2) 35
Quick Sort Recursion Tree T(n) = T(0) + T(n 1) + cn cn T(0) c(n 1) T(0) c(n 2) T(0) (1) 36
Quick Sort Recursion Tree T(n) = T(0) + T(n 1) + cn cn T(0) c(n 1) T(0) c(n 2) T(0) (1) 37
Quick Sort Recursion Tree cn (1) c(n 1) T(n) = T(0) + T(n 1) + cn h = n (1) c(n 2) (1) (1) T(n) = (n) + (n 2 ) = (n 2 ) 38
Quick Sort Best case analysis: If we get lucky, partition splits the array evenly T(n) = 2T(n/2) + Θ(n) = Θ(nlgn) (same as Merge-Sort) What if the split is 1/10 : 9/10? T(n) = T(n/10) + T(n/10) + Θ(n) = Θ(nlgn) (left as an exercise recursion tree) In fact, any split by a constant proportional amount will lead to Θ(nlgn) 39
Quick Sort What if alternates between best and worst cases: L(n) = 2U(n/2) + Θ(n) U(n) = L(n-1) + Θ(n) Solution: L(n) = 2U(n/2) + Θ(n) = (2L(n/2-1) + Θ(n/2)) + Θ(n) = 2L(n/2-1) + Θ(n) = Θ(nlgn) (left as an exercise) 40
Heap Sort In this context, the term heap has nothing to do with memory organization! Heap properties: Forms an almost-complete binary tree, i.e., completely filled on all levels, except possibly the lowest, which is filled from the left up to some point. The value at any given node is greater than the value of both it s children (max-heap). The root will have the largest value. 16 14 10 8 7 9 3 2 4 1 41
Heap Sort An important operation on heaps is Max-Heapify, which pushes a value down the tree if it violates the heap property. 5 14 10 8 7 9 3 2 4 1 42
Heap Sort In such a case, Max-Heapify will swap the value with the larger of it s two children and then repeat. 5 14 10 8 7 9 3 2 4 1 43
Heap Sort In such a case, Max-Heapify will swap the value with the larger of it s two children and then repeat. 14 5 10 8 7 9 3 2 4 1 44
Heap Sort In such a case, Max-Heapify will swap the value with the larger of it s two children and then repeat. 14 8 10 5 7 9 3 2 4 1 45
Heap Sort Sometimes Max-Heapify will push a value all the way to the leaf-level. 3 14 10 8 7 9 3 2 4 1 46
Heap Sort Sometimes Max-Heapify will push a value all the way to the leaf-level. 14 3 10 8 7 9 3 2 4 1 47
Heap Sort Sometimes Max-Heapify will push a value all the way to the leaf-level. 14 8 10 3 7 9 3 2 4 1 48
Heap Sort Sometimes Max-Heapify will push a value all the way to the leaf-level. 14 8 10 4 7 9 3 2 3 1 49
Heap Sort 1 2 3 4 5 6 7 8 9 10 11 12 16 16 14 10 8 7 9 3 2 4 1 14 10 For the above: A.length = 12 A.heap-size = 10 8 7 9 3 2 4 1 50
Heap Sort The Max-Heapify procedure can then be specified as: 51
Heap Sort 52
53 Heap Sort
Heap Sort The Heap-Sort algorithm works by: Building a heap Removing the value at the root of the heap Replacing the root value with the value in the highest numbered position Re-heapifying the array, starting at the root 54
55 Heap Sort
Heap Sort - Analysis Exercises: 6.1-1 What are the minimum and maximum numbers of elements in a heap of height h? 6.1-2 Show that an n-element heap has height equal to lgn. 6.3-3 Show that there are at most n/2 h+1 nodes at height h in any n- element heap. 56
Heap Sort - Analysis Another series that will be helpful (A.8 in Appendix A): kx k = h=0 x (1 x) 2 57
Heap Sort - Analysis Build-Max-Heap Analysis: One call to Max-Heapify costs O(lgn) time. Build-Max-Heap makes O(n) such calls. Total is O(nlgn) cost. This gives an upper-bound, but one that is not tight. 58
Heap Sort - Analysis A tight upper bound on Build-Max-Heap: Time required by Max-Heapify on a node at height h is O(h) time. Total cost is therefore: lgn h=0 n lgn 2 h+1 O h = O(n h=0 h/2 h ) = O(n h/2 h ) h=0 = O(n) Since, from A.8: σ h=0 h/2 h = 1/2 (1 1 2 )2 = 2 59
Heap Sort - Analysis Heap-sort Analysis: Build-Max-Heap takes O(n) time. n-1 calls to Max-Heapify are then made, each taking O(lgn) time. Total cost is therefore O(nlgn) cost. 60
Priority Queues A priority queue is a data structure for maintaining a set S of elements, each with an associated value called a key. Can be either max-priority or min-priority. Classic applications: Job scheduling (max priority) Event-driven simulation (min time-order) 16 14 10 8 7 9 3 2 4 1 Typically associated objects reside outside the queue, and are connected to priority queue entries via handles, e.g., pointers. 61
Priority Queues Heap Implementation Max-priority queue operations: INSERT(S, x) MAXIMUM(S) EXTRACT-MAX(S) INCREASE-KEY(S, x, k) 62
Priority Queues Heap Implementation short-circuiting 63
Priority Queues Heap Implementation Θ(1) O(lgn) O(lgn) O(lgn) 64
Priority Queues How about using a circular, array-based queue as a priority queue? What is the running time of the operations: INSERT(S, x) MAXIMUM(S) EXTRACT-MAX(S) INCREASE-KEY(S, x, k) 65
Lower Bounds on Sorting A sorting algorithm that only uses comparisons between elements to determine their order is said to be comparison-based. Though out the following we assume the elements to be sorted are unique, consequently tests for equality (=) between elements are unnecessary. The only comparisons used are therefore,,, and. 66
Decision Tree Model The behavior of a particular comparison-based sorting algorithm on a particular length input can be modeled by a decision-tree: node => a single comparison edge => the result of a comparison leaf => a sorted permutation of the input path from root to leaf => a trace of a completed sort on a specific input Insertion-sort on a list of length 3. array position # s 67
Decision Tree Model Note operations other than comparisons are not reflected in the tree. This is significant because we are only focusing on comparisons, and establishing a lower bound on the number of comparisons is sufficient. Also note that paths don t all have the same length why? This is what distinguishes best, worst, and other cases. 68
Comparison-Based Lower Bound Theorem 8.1 Any comparison-based sorting algorithm requires Ω(nlgn) comparisons in the worst case. Proof: Let h be the height of a decision tree, and let n be the length of the input. Then the number of leaves in the decision tree is at most 2 h. Since each permutation of the n input elements must appear on a leaf, we have: n! 2 h Taking log of both sides gives: h lg(n!) = lg(n) + lg(n-1) + + lg(1) lg(n) + + lg(n/2) (n/2)lg(n/2) = Ω(nlgn) 69
Sorting in Linear Time - Counting Sort Counting sort assumes that each of the n input elements is an integer in the range 0 to k, for some integer k. If k is O(n), then count sort runs in Θ(n) time. What is different here? We are limiting, albeit asymptotically, the size of the numbers. Normally we don t do this, i.e., heap-sort, merge-sort, etc. Limiting the size of input numbers can lead to interesting results, in general. 70
Counting Sort COUNTING-SORT(A, B, k) // A - Input array, B - output array, k max number Let C[0..k] be a new array for i = 0 to k C[i] = 0 for j = 1 to A.length C[A[j]] = C[A[j]] + 1 // C[i] now contains the number of elements equal to i. for i = 1 to k C[i] = C[i] + C[i-1] // C[i] now contains the number of elements to i. for j = A.length downto 1 B[C[A[j]]] = A[j] C[A[j]] = C[A[j]]-1 // C[i] now contains the number of // elements < i. 71
Counting Sort Running time for Counting-Sort: First loop is Θ(k) Second loop is Θ(n) Third loop is Θ(k) Forth loop is Θ(n) Total is therefore Θ(n + k) What this means is: Relatively big numbers k dominates. Relatively small numbers n dominates. With comparison-based sorts, k (the max size number) has no effect. If k is O(n), then count sort runs in Θ(n) time. 72
Radix Sort Historically, used by card-sorting machines. IBM punch card 80 x 12 73
Radix Sort If each column has one hold punched in it, then the machine can sort based on a single user-selected column. bin #1 bin #2 card sorter bin #12 74
Radix Sort In a decimal number, each digit position stores 1 of 10 digits. A card can then represent one d-digit number. So how do we sort? bin #1 bin #2 card sorter bin #12 75
Radix Sort Incorrect intuition sort most significant digits first. 3 2 9 4 5 7 6 5 7 8 3 9 4 3 6 7 2 0 3 5 5 3 2 9 3 5 5 4 5 7 4 3 6 6 5 7 7 2 0 8 3 9 3 2 9 7 2 0 4 3 6 8 3 9 3 5 5 4 5 7 6 5 7 7 2 0 3 5 5 4 3 6 4 5 7 6 5 7 3 2 9 8 3 9 76
Radix Sort Correct sort least significant digits first. 3 2 9 4 5 7 6 5 7 8 3 9 4 3 6 7 2 0 3 5 5 7 2 0 3 5 5 5 3 6 4 5 7 6 5 7 3 2 9 8 3 9 7 2 0 3 2 9 4 3 6 8 3 9 3 5 5 4 5 7 6 5 7 3 2 9 3 5 5 4 3 6 4 5 7 6 5 7 7 2 0 8 3 9 77
Radix Sort Sorting each column must be stable (why?) 3 2 9 4 5 7 6 5 7 8 3 9 4 3 6 7 2 0 3 5 5 7 2 0 3 5 5 5 3 6 4 5 7 6 5 7 3 2 9 8 3 9 7 2 0 3 2 9 4 3 6 8 3 9 3 5 5 4 5 7 6 5 7 3 2 9 3 5 5 4 3 6 4 5 7 6 5 7 7 2 0 8 3 9 The proof of correctness is by induction on i, the column #, from the right. The inductive step requires the numbers to be sorted on the first i-1 columns, and that the sort on column i be stable. 78
Radix Sort Code for Radix-sort: Running time: (using counting-sort) Each call to counting-sort takes Θ(n + k) Total time is therefore Θ(d(n + k)) If d is constant and k is O(n), then radix-sort runs in Θ(n) time. But so was counting-sort, so when would we need radix-sort? Food for thought 79
Radix Sort Another Version of Radix-sort: Suppose we are given n b-bit numbers: 10110011 0110 Let r be any positive integer such that r b Break each number into d = b/r digits of r bits each Each resulting digit is an integer in the range 0 to 2 r 1 Use Radix-sort with k = 2 r Running time: Θ((b/r)(n + 2 r )) Isn t this just Θ(d(n + k))? How has this changed anything? 80
Radix Sort Another Version of Radix-sort: Suppose we are given n b-bit numbers: Let r be any positive integer such that r b Break each number into d = b/r digits of r bits each Each resulting digit is an integer in the range 0 to 2 r 1 Use counting sort with k = 2 r Running time: Θ((b/r)(n + 2 r )) Notice the addition of the parameter r in the analysis, which is not part of the input this is something we can choose, or rather, adjust. 81
Radix Sort What is the effect of the choice of r? Given n and b, we want to chose r such that r b and (b/r)(n + 2 r ) is minimized. Case 1: b < lg n Then choose r = b The running time is then Θ((b/r)(n + 2 r )) = Θ((b/b)(n + 2 b )) = Θ(n), which is optimal. Case 2: b lg n The choose r = lgn Thus, the running time is Θ((b/r)(n + 2 r )) = Θ((b/lgn)(n + 2 lgn )) = Θ(bn/lgn), which is optimal. See the book for proof of optimality in this case 82
Radix Sort But why do we care what was wrong with the original version of Radix-sort? Food for thought 83