Sorting. Outline. Sorting with a priority queue Selection-sort Insertion-sort Heap Sort Quick Sort

Transcription

1 Sorting Hiroaki Kobayashi 1 Outline Sorting with a priority queue Selection-sort Insertion-sort Heap Sort Quick Sort Merge Sort Lower Bound on Comparison-Based Sorting Bucket Sort and Radix Sort Hiroaki Kobayashi 2

2 Sorting with a Priority Queue We can use a priority queue to sort a set of comparable elements Insert the elements one by one with a series of insertitem(e, e) operations Remove the elements in sorted order with a series of removemin() operations The running time of this sorting method depends on the priority queue implementation Algorithm PQ-Sort(S, C) Input sequence S, comparator C for the elements of S Output sequence S sorted in increasing order according to C P priority queue with comparator C while S.isEmpty () e S.remove (S. first ()) P.insertItem(e, e) while P.isEmpty() e P.removeMin() S.insertLast(e) Hiroaki Kobayashi 3 Sequence-based Priority Queue Implementation with an unsorted list Implementation with a sorted list Performance: insertitem takes O(1) time since we can insert the item at the beginning or end of the sequence removemin, minkey and minelement take O(n) time since we have to traverse the entire sequence to find the smallest key Performance: insertitem takes O(n) time since we have to find the place where to insert the item removemin, minkey and minelement take O(1) time since the smallest key is at the beginning of the sequence Hiroaki Kobayashi 4

3 Selection-Sort Selection-sort is the variation of PQ-sort where the priority queue is implemented with an unsorted sequence Running time of Selection-sort: Inserting the elements into the priority queue with n insertitem operations takes O(n) time Removing the elements in sorted order from the priority queue with n removemin operations takes time proportional to O(n+(n-1)+ +2+1) = O( i) =O(n(n-1)/2) Selection-sort runs in O(n 2 ) time Hiroaki Kobayashi Insertion-Sort Insertion-sort is the variation of PQ-sort where the priority queue is implemented with a sorted sequence Running time of Insertion-sort: Inserting the elements into the priority queue with n insertitem operations takes time proportional to O( n) =O( i) =O(n(n-1)/2) Removing the elements in sorted order from the priority queue with a series of n removemin operations takes O(n) time Insertion-sort runs in O(n 2 ) time Hiroaki Kobayashi

4 In-place Insertion-sort Instead of using an external data structure, we can implement selection-sort and insertion-sort in-place A portion of the input sequence itself serves as the priority queue For in-place insertion-sort We keep sorted the initial portion of the sequence We can use swapelements instead of modifying the sequence Hiroaki Kobayashi 7 Heap Sort Hiroaki Kobayashi 8

5 What is a heap A heap is a binary tree storing keys at its internal nodes and satisfying the following properties: Heap-Order: for every internal node v other than the root, key(v) key(parent(v)) Complete Binary Tree: let h be the height of the heap for i = 0,, h 1, there are 2 i nodes of depth i at depth h 1, the internal nodes are to the left of the external nodes The last node of a heap is the rightmost internal node of depth h 1 9 Hiroaki Kobayashi last node Height of a Heap Theorem: A heap storing n keys has height O(log n) Proof: (we apply the complete binary tree property) Let h be the height of a heap storing n keys Since there are 2 i keys at depth i = 0,, h 2 and at least one key at depth h 1, we have n h Thus, n 2 h 1, i.e., h log n + 1 depth 0 1 h 2 h 1 keys h 2 1 Hiroaki Kobayashi 10

6 Heaps and Priority Queues We can use a heap to implement a priority queue We store a (key, element) item at each internal node We keep track of the position of the last node For simplicity, we show only the keys in the pictures (2, Sue) (, Pat) (, Mark) (9, Jeff) (7, Anna) Hiroaki Kobayashi 11 Heap-Sort Consider a priority queue with n items implemented by means of a heap the space used is O(n) methods insertitem and removemin take O(log n) time methods size, isempty, minkey, and minelement take time O(1) time Using a heap-based priority queue, we can sort a sequence of n elements in O(n log n) time The resulting algorithm is called heap-sort Heap-sort is much faster than quadratic sorting algorithms, such as insertion-sort and selection-sort Hiroaki Kobayashi 12

7 Insertion into a Heap Method insertitem of the priority queue ADT corresponds to the insertion of a key k to the heap The insertion algorithm consists of three steps Find the insertion node z (the new last node) Store k at z and expand z into an internal node Restore the heap-order property (discussed next) z insertion node z Hiroaki Kobayashi 13 Upheap After the insertion of a new key k, the heap-order property may be violated Algorithm upheap restores the heap-order property by swapping k along an upward path from the insertion node Upheap terminates when the key k reaches the root or a node whose parent has a key smaller than or equal to k Since a heap has height O(log n), upheap runs in O(log n) time z 1 z Hiroaki Kobayashi 14

8 Removal from a Heap Method removemin of the priority queue ADT corresponds to the removal of the root key from the heap The removal algorithm consists of three steps Replace the root key with the key of the last node w Compress w and its children into a leaf Restore the heap-order property (discussed next) w w 2 last node 7 Hiroaki Kobayashi 1 Downheap After replacing the root key with the key k of the last node, the heap-order property may be violated Algorithm downheap restores the heap-order property by swapping key k along a downward path from the root Upheap terminates when key k reaches a leaf or a node whose children have keys greater than or equal to k Since a heap has height O(log n), downheap runs in O(log n) time 7 9 w 9 7 w Hiroaki Kobayashi 1

9 Updating the Last Node The insertion node can be found by traversing a path of O(log n) nodes Go up until a left child or the root is reached If a left child is reached, go to the right child Go down left until a leaf is reached Similar algorithm for updating the last node after a removal Hiroaki Kobayashi 17 Vector-based Heap Implementation We can represent a heap with n keys by means of a vector of length n + 1 For the node at rank i the left child is at rank 2i the right child is at rank 2i + 1 Links between nodes are not explicitly stored The leaves are not represented The cell of at rank 0 is not used Operation insertitem corresponds to inserting at rank n + 1 Operation removemin corresponds to removing at rank n Yields in-place heap-sort 9 0 Hiroaki Kobayashi

10 Merging Two Heaps We are given two two heaps and a key k We create a new heap with the root node storing k and with the two heaps as subtrees We perform downheap to restore the heaporder property Hiroaki Kobayashi 19 Bottom-up Heap Construction We can construct a heap storing n given keys in using a bottom-up construction with log n phases In phase i, pairs of heaps with 2 i 1 keys are merged into heaps with 2 i+1 1 keys 2 i 1 2 i 1 2 i+1 1 Hiroaki Kobayashi 20

11 Example: Bottom-up construction of a heap with 1 keys 1-key heaps key heaps Hiroaki Kobayashi 21 Example (contd.) 3-key heaps Down-heap bubbling Hiroaki Kobayashi 22

12 Example (contd.) 7-key heaps Down-heap bubbling Hiroaki Kobayashi 23 Example (end) 1-key heap Final heap after down-heap bubbling Hiroaki Kobayashi 24

13 Analysis We visualize the worst-case time of a downheap with a proxy path that goes first right and then repeatedly goes left until the bottom of the heap (this path may differ from the actual downheap path) Since each node is traversed by at most two proxy paths, the total number of nodes of the proxy paths is O(n) Thus, bottom-up heap construction runs in O(n) time Bottom-up heap construction is faster than n successive insertions and speeds up the first phase of heap-sort Hiroaki Kobayashi 2 Summary For sorting n elements using heap sort First phase of heap-sort: heap construction O(n) Second phase of heap-sort: n removemin operations O(log n) The heap-sort algorithm sorts a sequence S of of n comparable elements in in O(n log n) time Hiroaki Kobayashi 2

14 Quick-Sort Hiroaki Kobayashi 27 Divide-and-Conquer Strategy Divide-and conquer is a general algorithm design paradigm: Divide: divide the input data S in two or more disjoint subsets S 1, S 2, Recur: solve the subproblems recursively Conquer: combine the solutions for S 1,S 2,, into a solution for S The base case for the recursion are subproblems of constant size Analysis can be done using recurrence equations Divide phase Conquer phase Hiroaki Kobayashi 28

15 Quick-Sort Quick-sort is a randomized sorting algorithm based on the divide-and-conquer paradigm: Divide: pick a random element x (called pivot) and partition S into L elements less than x E elements equal x G elements greater than x Recur: sort L and G Conquer: join L, E and G L x x E x G Hiroaki Kobayashi 29 Partition We partition an input sequence as follows: We remove, in turn, each element y from S and We insert y into L, E or G, depending on the result of the comparison with the pivot x Each insertion and removal is at the beginning or at the end of a sequence, and hence takes O(1) time Thus, the partition step of quick-sort takes O(n) time Algorithm partition(s, p) Input sequence S, position p of pivot Output subsequences L, E, G of the elements of S less than, equal to, or greater than the pivot, resp. L, E, G empty sequences x S.remove(p) while S.isEmpty() y S.remove(S.first()) if y < x L.insertLast(y) else if y = x E.insertLast(y) else { y > x } G.insertLast(y) return L, E, G Hiroaki Kobayashi 30

16 Quick-Sort Tree An execution of quick-sort is depicted by a binary tree Each node represents a recursive call of quick-sort and stores Unsorted sequence before the execution and its pivot Sorted sequence at the end of the execution The root is the initial call The leaves are calls on subsequences of size 0 or Hiroaki Kobayashi 31 Execution Example Pivot selection Hiroaki Kobayashi 32

17 Execution Example (cont.) Partition, recursive call, pivot selection Hiroaki Kobayashi 33 Execution Example (cont.) Partition, recursive call, base case Hiroaki Kobayashi 34

18 Execution Example (cont.) Recursive call,, base case, join Hiroaki Kobayashi 3 Execution Example (cont.) Recursive call, pivot selection Hiroaki Kobayashi 3

19 Execution Example (cont.) Partition,, recursive call, base case Hiroaki Kobayashi 37 Execution Example (cont.) Join, join Hiroaki Kobayashi 38

20 Worst-case Running Time The worst case for quick-sort occurs when the pivot is the unique minimum or maximum element One of L and G has size n 1 and the other has size 0 The running time is proportional to the sum n + (n 1) Thus, the worst-case running time of quick-sort is O(n 2 ) depth 0 time n 1 n 1 n 1 1 Hiroaki Kobayashi 39 Expected Running Time Consider a recursive call of quick-sort on a sequence of size s Good call: the sizes of L and G are each less than 3s/4 Bad call: one of L and G has size greater than 3s/ Good call Bad call A call is good with probability 1/2 1/2 of the possible pivots cause good calls: Bad pivots Good pivots Bad pivots Hiroaki Kobayashi 40

21 Expected Running Time, Part 2 Probabilistic Fact: The expected number of coin tosses required in order to get k heads is 2k For a node of depth i, we expect i/2 ancestors are good calls The size of the input sequence for the current call is at most (3/4) i/2 n Therefore, we have For a node of depth 2log 4/3 n, the expected input size is one The expected height of the quick-sort tree is O(log n) The amount or work done at the nodes of the same depth is O(n) Thus, the expected running time of quick-sort is O(n log n) expected height O(log n) s(a) s(r) s(b) s(c) s(d) s(e) s(f) time per level O(n) O(n) O(n) total expected time: O(n log n) Hiroaki Kobayashi 41 In-Place Quick-Sort Quick-sort can be implemented to run in-place In the partition step, we use replace operations to rearrange the elements of the input sequence such that the elements less than the pivot have rank less than h the elements equal to the pivot have rank between h and k the elements greater than the pivot have rank greater than k The recursive calls consider elements with rank less than h elements with rank greater than k Algorithm inplacequicksort(s, l, r) Input sequence S, ranks l and r Output sequence S with the elements of rank between l and r rearranged in increasing order if l r return i a random integer between l and r x S.elemAtRank(i) (h, k) inplacepartition(x) inplacequicksort(s, l, h 1) inplacequicksort(s, k + 1, r) Hiroaki Kobayashi 42

22 In-Place Partitioning Perform the partition using two indices to split S into L and E U G (a similar method can split E U G into E and G). j k (pivot = ) Repeat until j and k cross: Scan j to the right until finding an element > x. Scan k to the left until finding an element < x. Swap elements at indices j and k j k Hiroaki Kobayashi 43 Merge Sort Hiroaki Kobayashi 44

23 Divide-and-Conquer Strategy Divide-and conquer is a general algorithm design paradigm: Divide: divide the input data S in two disjoint subsets S 1 and S 2 Recur: solve the subproblems associated with S 1 and S 2 Conquer: combine the solutions for S 1 and S 2 into a solution for S The base case for the recursion are subproblems of size 0 or 1 Merge-sort is a sorting algorithm based on the divide-and-conquer paradigm Like heap-sort It uses a comparator It has O(n log n) running time Unlike heap-sort It does not use an auxiliary priority queue It accesses data in a sequential manner (suitable to sort data on a disk) Hiroaki Kobayashi 4 Merge-Sort Merge-sort on an input sequence S with n elements consists of three steps: Divide: partition S into two sequences S 1 and S 2 of about n/2 elements each Recur: recursively sort S 1 and S 2 Conquer: merge S 1 and S 2 into a unique sorted sequence Algorithm mergesort(s, C) Input sequence S with n elements, comparator C Output sequence S sorted according to C if S.size() > 1 (S 1, S 2 ) partition(s, n/2) mergesort(s 1, C) mergesort(s 2, C) S merge(s 1, S 2 ) Hiroaki Kobayashi 4

24 Merging Two Sorted Sequences The conquer step of merge-sort consists of merging two sorted sequences A and B into a sorted sequence S containing the union of the elements of A and B Merging two sorted sequences, each with n/2 elements and implemented by means of a doubly linked list, takes O(n) time Algorithm merge(a, B) Input sequences A and B with n/2 elements each Output sorted sequence of A B S empty sequence while A.isEmpty() B.isEmpty() if A.first().element() < B.first().element() S.insertLast(A.remove(A.first())) else S.insertLast(B.remove(B.first())) while A.isEmpty() S.insertLast(A.remove(A.first())) while B.isEmpty() S.insertLast(B.remove(B.first())) return S Hiroaki Kobayashi 47 Merge-Sort Tree An execution of merge-sort is depicted by a binary tree each node represents a recursive call of merge-sort and stores unsorted sequence before the execution and its partition sorted sequence at the end of the execution the root is the initial call the leaves are calls on subsequences of size 0 or Hiroaki Kobayashi 48

25 Execution Example Partition Hiroaki Kobayashi 49 Execution Example (cont.) Recursive call, partition Hiroaki Kobayashi 0

26 Execution Example (cont.) Recursive call, partition Hiroaki Kobayashi 1 Execution Example (cont.) Recursive call, base case Hiroaki Kobayashi 2

27 Execution Example (cont.) Recursive call, base case Hiroaki Kobayashi 3 Execution Example (cont.) Merge Hiroaki Kobayashi 4

28 Execution Example (cont.) Recursive call,, base case, merge Hiroaki Kobayashi Execution Example (cont.) Merge Hiroaki Kobayashi

29 Execution Example (cont.) Recursive call,, merge, merge Hiroaki Kobayashi 7 Execution Example (cont.) Merge Hiroaki Kobayashi 8

30 Analysis of Merge-Sort The height h of the merge-sort tree is O(log n) at each recursive call we divide in half the sequence, The overall amount or work done at the nodes of depth i is O(n) we partition and merge 2 i sequences of size n/2 i we make 2 i+1 recursive calls Thus, the total running time of merge-sort is O(n log n) depth #seqs 0 1 size n 1 2 n/2 i 2 i n/2 i Hiroaki Kobayashi 9 Summary of Sorting Algorithms Algorithm selection-sort Time O(n 2 ) Notes in-place slow (good for small inputs (< 1K)) insertion-sort quick-sort heap-sort merge-sort O(n 2 ) O(n log n) expected O(n log n) O(n log n) in-place slow (good for small inputs (< 1K)) in-place, randomized fastest (good for large inputs (1K 1M)) in-place fast (good for large inputs (1K 1M)) sequential data access fast (good for huge inputs (> 1M)) Hiroaki Kobayashi 0

31 Sorting Lower Bound Hiroaki Kobayashi 1 Comparison-Based Sorting Many sorting algorithms are comparison based. They sort by making comparisons between pairs of objects Examples: bubble-sort, selection-sort, insertion-sort, heap-sort, merge-sort, quick-sort,... Let us therefore derive a lower bound on the running time of any algorithm that uses comparisons to sort n elements, x 1, x 2,, x n. Is x i < x j? no yes Hiroaki Kobayashi 2

32 Counting Comparisons Let us just count comparisons then. Each possible run of the algorithm corresponds to a root-to-leaf path in a decision tree x i < x j? x a < x b? x c < x d? x e < x f? x k < x l? x m < x o? x p < x q? Hiroaki Kobayashi 3 Decision Tree Height The height of this decision tree is a lower bound on the running time Every possible input permutation must lead to a separate leaf output. If not, some input 4 would have same output ordering as 4, which would be wrong. Since there are n!=1*2* *n leaves, the height is at least log (n!) minimum height (time) x i < x j? x a < x b? x c < x d? log (n!) x e < x f? x k < x l? x m < x o? x p < x q? n! Hiroaki Kobayashi 4

33 The Lower Bound Any comparison-based sorting algorithms takes at least log (n!) time Therefore, any such algorithm takes time at least log ( n!) log n 2 n 2 = ( n / 2)log ( n / 2). That is, any comparison-based sorting algorithm must run in (n log n) time. Hiroaki Kobayashi Bucket-Sort and Radix-Sort 1, c 3, a 3, b 7, d 7, g 7, e B Hiroaki Kobayashi

34 Bucket-Sort Let be S be a sequence of n (key, element) items with keys in the range [0, N 1] Bucket-sort uses the keys as indices into an auxiliary array B of sequences (buckets) Phase 1: Empty sequence S by moving each item (k, o) into its bucket B[k] Phase 2: For i = 0,, N 1, move the items of bucket B[i] to the end of sequence S Analysis: Phase 1 takes O(n) time Phase 2 takes O(n + N) time Bucket-sort takes O(n + N) time Algorithm bucketsort(s, N) Input sequence S of (key, element) items with keys in the range [0, N 1] Output sequence S sorted by increasing keys B array of N empty sequences while S.isEmpty() f S.first() (k, o) S.remove(f) B[k].insertLast((k, o)) for i 0 to N 1 while B[i].isEmpty() f B[i].first() (k, o) B[i].remove(f) S.insertLast((k, o)) Hiroaki Kobayashi 7 Example Key range [0, 9] 7, d 1, c 3, a 7, g 3, b 7, e Phase 1 1, c 3, a 3, b 7, d 7, g 7, e B Phase 2 1, c 3, a 3, b 7, d 7, g 7, e Hiroaki Kobayashi 8

35 Properties and Extensions Key-type Property The keys are used as indices into an array and cannot be arbitrary objects No external comparator Stable Sort Property The relative order of any two items with the same key is preserved after the execution of the algorithm Extensions Integer keys in the range [a, b] Put item (k, o) into bucket B[k a] String keys from a set D of possible strings, where D has constant size (e.g., names of the 0 U.S. states) Sort D and compute the rank r(k) of each string k of D in the sorted sequence Put item (k, o) into bucket B[r(k)] Hiroaki Kobayashi 9 Lexicographic Order A d-tuple is a sequence of d keys (k 1, k 2,, k d ), where key k i is said to be the i-th dimension of the tuple Example: The Cartesian coordinates of a point in space are a 3-tuple The lexicographic order of two d-tuples is recursively defined as follows (x 1, x 2,, x d ) < (y 1, y 2,, y d ) x 1 < y 1 x 1 = y 1 (x 2,, x d ) < (y 2,, y d ) I.e., the tuples are compared by the first dimension, then by the second dimension, etc. Hiroaki Kobayashi 70

36 Lexicographic-Sort Let C i be the comparator that compares two tuples by their i-th dimension Let stablesort(s, C) be a stable sorting algorithm that uses comparator C Lexicographic-sort sorts a sequence of d-tuples in lexicographic order by executing d times algorithm stablesort, one per dimension Lexicographic-sort runs in O(dT(n)) time, where T(n) is the running time of stablesort Algorithm lexicographicsort(s) Input sequence S of d-tuples Output sequence S sorted in lexicographic order for i d downto 1 stablesort(s, C i ) Example: (7,4,) (,1,) (2,4,) (2, 1, 4) (3, 2, 4) (2, 1, 4) (3, 2, 4) (,1,) (7,4,) (2,4,) (2, 1, 4) (,1,) (3, 2, 4) (7,4,) (2,4,) (2, 1, 4) (2,4,) (3, 2, 4) (,1,) (7,4,) Hiroaki Kobayashi 71 Radix-Sort Radix-sort is a specialization of lexicographic-sort that uses bucket-sort as the stable sorting algorithm in each dimension Radix-sort is applicable to tuples where the keys in each dimension i are integers in the range [0, N 1] Radix-sort runs in time O(d( n + N)) Algorithm radixsort(s, N) Input sequence S of d-tuples such that (0,, 0) (x 1,, x d ) and (x 1,, x d ) (N 1,, N 1) for each tuple (x 1,, x d ) in S Output sequence S sorted in lexicographic order for i d downto 1 bucketsort(s, N) Hiroaki Kobayashi 72

37 Radix-Sort for Binary Numbers Consider a sequence of n b-bit integers x = x b 1 x 1 x 0 We represent each element as a b-tuple of integers in the range [0, 1] and apply radix-sort with N = 2 This application of the radix-sort algorithm runs in O(bn) time For example, we can sort a sequence of 32-bit integers in linear time Algorithm binaryradixsort(s) Input sequence S of b-bit integers Output sequence S sorted replace each element x of S with the item (0, x) for i 0 to b 1 replace the key k of each item (k, x) of S with bit x i of x bucketsort(s, 2) Hiroaki Kobayashi 73 Example Sorting a sequence of 4-bit integers Hiroaki Kobayashi 74

38 Priority Queue ADT A priority queue stores a collection of items An item is a pair (key, element) Main methods of the Priority Queue ADT insertitem(k, o) inserts an item with key k and element o removemin() removes the item with smallest key and returns its element Additional methods minkey(k, o) returns, but does not remove, the smallest key of an item minelement() returns, but does not remove, the element of an item with smallest key size(), isempty() Applications: Standby flyers Auctions Stock market Hiroaki Kobayashi 7 Total Order Relation Keys in a priority queue can be arbitrary objects on which an order is defined Two distinct items in a priority queue can have the same key Mathematical concept of total order relation Reflexive property: x x Antisymmetric property: x y y x x = y Transitive property: x y y z x z Hiroaki Kobayashi 7

39 Comparator ADT A comparator encapsulates the action of comparing two objects according to a given total order relation A generic priority queue uses an auxiliary comparator The comparator is external to the keys being compared When the priority queue needs to compare two keys, it uses its comparator Methods of the Comparator ADT, all with Boolean return type islessthan(x, y) islessthanorequalto(x,y) isequalto(x,y) isgreaterthan(x, y) isgreaterthanorequalto(x,y) iscomparable(x) Hiroaki Kobayashi 77