Deliverables. Quick Sort. Randomized Quick Sort. Median Order statistics. Heap Sort. External Merge Sort

Transcription

1 More Sorting

2 Deliverables Quick Sort Randomized Quick Sort Median Order statistics Heap Sort External Merge Sort gdeepak.com 2

3 Quick Sort Divide and conquer algorithm which relies on a partition operation: Choose a pivot, move all smaller elements before it, and greater elements after it. The most complex issue is choosing a good pivot.poor choices of pivots can result in O(n²) performance, but if at each step we choose the median as pivot then it works in O(n log n). Finding median is an O(n) operation on unsorted lists. gdeepak.com 3

4 Quick Sort Quick sort is an unstable and in place. It uses O(log n) additional space due to log n recursive calls. Quick sort makes excellent usage of the memory hierarchy, taking perfect advantage of virtual memory and available caches. It can be easily parallelized due to its divide and conquer nature. Combine Step is trivial in Quick sort. Generally it is twice as fast from Merge Sort. gdeepak.com 4

5 Quick Sort Example gdeepak.com 5

6 Quick Sort-algorithm p starting index of array A r last index of array a q is the position of pivot after partitioning Quicksort(A, p, r) if p < r then q Partition(A, p, r) Quicksort(A, p, q - 1) Quicksort(A, q + 1, r) Partition(A, p, r) x A[r] i p - 1 for j p to r - 1 do if A[j] x then i i + 1 swap A[i] A[j] swap A[i + 1] A[r] return i + 1 gdeepak.com 6

7 Quick Sort example i p j r Partition(A, p, r) p x A[r] i p - 1 for j p to r - 1 do if A[j] x then i i + 1 swap A[i] A[j] swap A[i+1] A[r] return i + 1 gdeepak.com 7

8 Quick Sort example Partition(A, p, r) i p j r x A[r] i p - 1 for j p to r - 1 do if A[j] x then i i + 1 swap A[i] A[j] swap A[i+1] A[r] return i + 1 gdeepak.com 8

9 Quick Sort example Partition(A, p, r) x A[r] i p j r i p - 1 for j p to r - 1 do if A[j] x then i i + 1 swap A[i] A[j] swap A[i+1] A[r] return i + 1 gdeepak.com 9

10 Quick Sort example Partition(A, p, r) r r i p j r r r x A[r] i p - 1 for j p to r - 1 do if A[j] x then i i + 1 swap A[i] A[j] swap A[i+1] A[r] return i + 1 gdeepak.com 10

11 Quick Sort example r 2 p 2 p 2 p 2 p p i 1 3 i 1 3 i 1 3 i 1 3 i j j j 6 4 r r r 4 r r Partition(A, p, r) x A[r] i p - 1 for j p to r - 1 do if A[j] x then i i + 1 swap A[i] A[j] swap A[i+1] A[r] return i + 1 gdeepak.com 11

12 Quick Sort example r r p i j p 2 p 2 p 1 3 i 1 3 i 1 3 i j r r 4 r Partition(A, p, r) x A[r] i p - 1 for j p to r - 1 do if A[j] x then i i + 1 swap A[i] A[j] swap A[i+1] A[r] return i + 1 gdeepak.com 12

13 Quick Sort example r r p 2 p 2 p p i j 1 3 i 1 3 i 1 3 i j r r 4 r Partition(A, p, r) x A[r] i p - 1 for j p to r - 1 do if A[j] x then i i + 1 swap A[i] A[j] swap A[i+1] A[r] return i + 1 gdeepak.com 13

14 Quick Sort example r r p 2 p 2 p p 1 3 i 1 3 i 1 3 i i j r r j 4 r Partition(A, p, r) x A[r] i p - 1 for j p to r - 1 do if A[j] x then i i + 1 swap A[i] A[j] swap A[i+1] A[r] return i + 1 gdeepak.com 14

15 Beauty of Quick Sort Invariants p x x I x >x >x j R x Performance decreases with the presence of large number of duplicates. Need to adopt 3-way Quick Sort. gdeepak.com 15

16 Quick sort complexity T(n) = T(i-1) + T(n-i) +n if ( i=1) then T(n) = T(n-1)+n = O(n 2 ) if i=n/2 then T(n) = 2T(n/2) +n = O(nlogn) gdeepak.com 16

17 Recurrence Tree gdeepak.com 17

18 Special Split What if the split is always ( 1 10 ) and ( 9 10 ) T(n)=T ( 1 10 n)+ T ( 9 10 n)+ Θ n What is the solution to this recurrence? gdeepak.com 18

19 gdeepak.com 19

20 Randomized Idea Randomized-Partition(A, p, r) i Random(p, r) Swap A[r] and A[i] Return Partition(A, p, r) gdeepak.com 20

21 Randomized Quick Sort In addition to the input, During execution, algorithm takes random choices depending on those random numbers. The behaviour can vary if the algorithm is run multiple times on the same input. It Chooses a splitter s such that the number of elements less or greater than pivot is at least n/4 The algorithm randomly chooses a key, and checks whether it is a central splitter or not. gdeepak.com 21

22 Key Idea A partition is good if each of the partitions contains at least 25% of the data What is the probability of a good partition? Half of the elements lie within this range and half outside, so 50% chance, in the example 3,4,5,6 are good splitters gdeepak.com 22

23 Randomized Complexity If it is a central splitter, then the array is split with that key Expected number of trials needed to get a central splitter is constant. Probability that the randomly chosen element is a central splitter is ½ If it is not a central splitter then the element is chosen again gdeepak.com 23

24 How many times partitions On average, how many times will Partition need to be called before be get a good partition? Let E be the number of times Recurrence: 1 E 1 E gdeepak.com 24

25 Randomized Complexity Implication is that expected number of times we need to repeat the process of finding a central splitter is 2 Thus, the expected time complexity is O(n) Worst case size of each partition in j th level recursion is n (3/4) j Number of levels of recursion = log 4/3 n = O(log n). Recurrence Relation of the time complexity: T(n) = 2T(3n/4) + O(n) = O(n log n) gdeepak.com 25

26 Finding i th smallest number in linear time Algorithm for returning i th smallest element in A[p..r] Randomized_Select(A, p, r, i) if p=r then return A[p] q Randomized-partition (A, p, r) k q-p+1 if i=k // pivot value is the answer then return A[q] else if i<k then return Randomized_Select(A, p, q-1, i) else return Randomized_Select(A, q+1, r, i-k) gdeepak.com 26

27 Finding i th smallest number Example P q r A[q] A[q] k Let i= pivot q=4 gdeepak.com 27

28 Finding i th smallest number Analysis Now we are looking for 7-4=3 rd rank in the right list Analysis (assume all are distinct) lucky case: any constant fraction split T(n)= T(9n/10) + θ(n) from master theorem n log 10/9 1 = n 0 = 1 so complexity= θ(n) worst case analysis T(n)=T(n-1)+ θ(n) = θ(n 2 ) gdeepak.com 28

29 Median Order Statistics gdeepak.com 29

30 Worst case linear time 1. Divide the elements into groups of five, where the last group may have less than five elements in case when the input array size is not a multiple of five. 2. Compute median of each group (break ties arbitrarily). It may take 6 comparisons to find middle of 5 elements. 3. Make a recursive call to calculate the median of the medians. Set x to the median. 4. Use x as the pivot and partition. During partitioning whole column has to be moved with the middle element being moved during partitioning. gdeepak.com 30

31 Worst case linear time 5. Compare the remaining elements (whose placement in the two partitions can not be determined by median. 6. If the pivot is not the order statistics that is searched for, recurse on the sub array that contains it Use a bound B to stop recursion: If size of array is less than B then use brute-force search to find desired order statics gdeepak.com 31

32 Steps for finding the median of five numbers 1) Put numbers in an array. 2)Use three comparisons and shuffle around the numbers so that a[1] < a[2], a[4] < a[5], and a[1] < a[4]. 3) If a[3] > a[2], then problem is fairly easy. If a[2] < a[4], median value is smaller of a[3] and a[4]. If not, median value is smaller of a[2] and a[5]. 4) So a[3] < a[2]. If a[3] > a[4], then solution is smaller of a[3] and a[5]. Otherwise, solution is smaller of a[2] and a[4]. gdeepak.com 32

33 Median Order Statistics Example gdeepak.com 33

34 Median Order Statistics-Example gdeepak.com 34

35 Median Order Statistics-Example gdeepak.com 35

36 Analysis-Median order statistics At least half of the medians found are greater than or equal to the median-of-medians x. Thus, at least half of the ceil(n/5) groups contribute 3 elements greater than x, except, possibly, for one group containing less than 5 elements and the group containing x itself. Therefore, the number of elements > x is at least 3n/10 Similarly, the number of elements < x is at least 3n/10 Remaining elements may be compared one by one with the median and may be put in the partitions accordingly. 3n/10>= n/4 therefore the recursive call to Select is executed on <= 3n/4 elements gdeepak.com 36

37 gdeepak.com 37

38 Linear complexity for median order Thus recurrence for running time takes at most T(3n/4) for this step So step wise it becomes T(n) T(n/5) + O(n)+T(3n/4) By Substitution T(n)<= cn we have T(n) <= 1/5 cn + ¾ cn+ O(n) =19cn/20+ O(n) = cn-(cn/20 - O(n)) <=cn if c is chosen large enough to handle both the θ(n) and the initial conditions gdeepak.com 38

39 Heap Binary tree where value of parent is children is Max heap All levels of tree are complete except the last Binary tree where value of parent is children is min heap Maximum(S) - return the largest element in the set gdeepak.com 39

40 Heap Operations ExtractMax(S) Return and remove largest element in the set Insert(S, val) insert val into the set IncreaseElement(S, x, val) increase value of element x to val BuildHeap(A) build a heap from an array of elements gdeepak.com 40

41 Heap Every node in a heap is a heap in itself. Height of node = # of edges on a longest simple path from the node down to a leaf. Height of heap = height of root = O(lg n). A heap can be stored as an array A. Root of tree is A[1] Parent of A[i] = A i/2 Left child of A[i] = A[2i ] Right child of A[i] = A[2i + 1] gdeepak.com 41

42 gdeepak.com 42

43 Heapify-up heapify-up(h, i): if i > 1 then let j=parent(i)= i/2 if the key [H[i]] < key [H[j]] then swap the array entries H[i] and H[j] heapify-up(h, j) endif endif gdeepak.com 43

44 gdeepak.com 44

45 gdeepak.com 45

46 gdeepak.com 46

47 gdeepak.com 47

48 gdeepak.com 48

49 Heapify-down Heapify-down(H, i) let n=length(h) if 2i>n then terminate with H unchanged Else if 2i<n then let left=2i and right = 2i+1 let j be the index that minimizes key[h[left]] and key[h[right]] Else if 2i=n then let j=2i Endif if key [H[j]] < key[h[i]] then swap the array entries H[i] and H[j] Heapify-down(H, j) Endif gdeepak.com 49

50 gdeepak.com 50

51 gdeepak.com 51

52 gdeepak.com 52

53 gdeepak.com 53

54 gdeepak.com 54

55 gdeepak.com 55

56 Building a Max-heap BuildMaxHeap(A) heapsize[a] length[a] for i length A /2 downto 1 do heapify-down(a, i) gdeepak.com 56

57 gdeepak.com 57

58 gdeepak.com 58

59 gdeepak.com 59

60 gdeepak.com 60

61 gdeepak.com 61

62 gdeepak.com 62

63 gdeepak.com 63

64 gdeepak.com 64

65 gdeepak.com 65

66 gdeepak.com 66

67 /6/2012 6:48 PM gdeepak.com 67

68 /6/2012 6:48 PM gdeepak.com 68

69 Complexity of Heapify There can be two strategies, We start building the heap from top ( i.e. from the root and push the elements up) or start building the heap from the bottom mean start pushing the elements down. Our strategy shown was from the bottom, Leaves of the heap are already in the heap position so we need to start from the halfway of the total number of elements. Building the heap from the top means we consider root is at its correct position and then add one more element to the heap and so on. 6/6/2012 7:06 PM gdeepak.com 69

70 Complexity of Heapify Heapify is O(n). To start building heap from the bottom or the top 6/6/2012 6:50 PM Push Up Push Down Elements Level Wise Compariso ns Comparis ons *1=5 0*1= *2=8 1*2= *4=12 2*4= *8=16 3*8= *16=16 4*16= *32=0 5*32= O(n) 258 O(nlogn) gdeepak.com 70

71 Sorting Heapsort(A) BuildMinHeap(A) For i length[a] downto 2 Do exchange A[1] A[i] heapsize[a] heapsize[a] -1 heapify-down(a,1) 6/6/2012 6:48 PM gdeepak.com 71

72 /6/2012 6:48 PM gdeepak.com 72

73 /6/2012 6:48 PM 2 4 gdeepak.com 73

74 /6/2012 6:48 PM 2 1 gdeepak.com 74

75 /6/2012 6:48 PM gdeepak.com 75

76 /6/2012 6:48 PM gdeepak.com 76

77 /6/2012 6:48 PM gdeepak.com 77

78 /6/2012 6:48 PM gdeepak.com 78

79 /6/2012 6:48 PM gdeepak.com 79

80 /6/2012 6:48 PM gdeepak.com 80

81 /6/2012 6:48 PM gdeepak.com 81

82 /6/2012 6:48 PM gdeepak.com 82

83 /6/2012 6:48 PM gdeepak.com 83

84 /6/2012 6:48 PM gdeepak.com 84

85 /6/2012 6:48 PM gdeepak.com 85

86 /6/2012 6:48 PM gdeepak.com 86

87 /6/2012 6:48 PM gdeepak.com 87

88 /6/2012 6:48 PM gdeepak.com 88

89 /6/2012 6:48 PM gdeepak.com 89

90 Height of any heap is O(logn) Heap Sort Complexity Heapify up or heapify down is O(logn) Sorting n elements is O(nlogn) worst case Priority Queue can be best implemented by heap where insertion and deletion will take O(logn) time, while in case of Simple Queues [Insertion: O(n), Deletion O(1)] or [Insertion O(1), Deletion O(n)] 6/6/2012 6:48 PM gdeepak.com 90

91 Comparison of sorting algorithms Sorting Worst Average Best Space In Place Stable Adaptive Bubble Ө(n 2 ) Ө(n 2 ) Ө(n) Ө(n) Yes Yes Yes Selection Ө(n 2 ) Ө(n 2 ) Ө(n 2 ) Ө(n) Yes Yes/No No Insertion Ө(n 2 ) Ө(n 2 ) Ө(n) Ө(n) Yes Yes Yes Shell Ө(nlog 2 n) Ө(nlog 2 n) Ө(n) Ө(n) Yes No Yes Merge Ө(nlogn) Ө(nlogn) Ө(nlogn) Ө(n) No/Yes Yes No/Yes Counting Ө(n+N) Ө(n+N) Ө(n+N) Ө(n+N) No Yes No Radix Ө(nd) Ө(nd) Ө(nd) Ө(nd) No Yes No Bucket Ө(n 2 ) Ө(nlogn) Ө(n) Ө(n) No/Yes Yes/No Yes/No Quick Ө(n 2 ) Ө(nlogn) Ө(nlogn) Ө(n) Yes No/Yes No Heap Ө(nlogn) Ө(nlogn) Ө(n) Ө(n) Yes No No/Yes 6/6/2012 6:48 PM gdeepak.com 91

92 Open Problem In place, Stable comparison based nlogn algorithm. 6/6/2012 6:48 PM gdeepak.com 92

93 External Sorting Mostly used for Large Databases or Large Lists Applied when all records can not fit into primary memory (Remember whole RAM is not available for Program space) Bottleneck is page swaps between hard disk and RAM Required for searching, Queries containing Sort-By clause, Joining of tables, Projection queries, Getting rid of duplicates for Primary Key fields or query results 6/6/2012 6:48 PM gdeepak.com 93

94 Why not Quick Sort Natural First choice will be quick sort-but it is a disaster If one page swap takes 1 million cycles and one page contains 100 records then total number of page swaps will be approximately 10000, total complexity will be nlogn Page Swaps (For average Case) where n is the number of elements. Space is a big issue so we can not use additional space in RAM, but space on Hard Disk may be used. 6/6/2012 6:48 PM gdeepak.com 94

95 Quick Sort example i p j r Partition(A, p, r) p x A[r] i p - 1 for j p to r - 1 do if A[j] x then i i + 1 swap A[i] A[j] swap A[i+1] A[r] return i + 1 6/6/2012 6:48 PM gdeepak.com 95

96 What to do Ideally our goal is that any page should not require more than once. Merge sort is the ideal algorithm for this Run is a sorted sub list of data. If Page size is 16K,We bring unsorted pages one by one from the hard disk, Sort the individual page using quick or other sort. Now use Merge Sort to merge two individual sorted pages into sorted page. 6/6/2012 6:48 PM gdeepak.com 96

97 Unsorted Lying in Hard Disk 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 6/6/2012 6:48 PM gdeepak.com 97

98 Unsorted Lying in Hard Disk 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K QS QS Sorted 16K 16K Merge Sorted 16K 16K Written to a new location in the hard disk or same location depending upon whether we are using in place merge sort or not RAM In each pass of the merge sort every page is being read once and written once. 6/6/2012 6:48 PM gdeepak.com 98

99 Unsorted Lying in Hard Disk 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K QS QS Sorted 16K 16K Merge Sorted 16K 16K RAM 6/6/2012 6:48 PM gdeepak.com 99

100 Unsorted Lying in Hard Disk 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K Sorted 16K 16K RAM Merge Sorted 16K If Right Hand Side page is filled it is written back and we start writing on a new page If Left hand side page completes it is replaced with the next page from the corresponding set 6/6/2012 6:48 PM gdeepak.com 100

101 Partially Sorted Lying in Hard Disk 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K 16K Sorted 16K 16K RAM 6/6/2012 6:48 PM Merge Sorted 16K If Right Hand Side page is filled it is written back and we start writing on a new page If Left hand side page completes it is replaced with the next page from the corresponding set gdeepak.com 101

102 Further improvements Total Passes are LgN, where N is the no of pages, no the number of numbers and Total page swaps are 2NLgN. It can be further reduced, because Disks can read blocks together instead of pages. Instead of starting our merge sort procedure from one page, it can be started from 512 pages. In that case a set of every 512 pages will be sorted by quick sort. 6/6/2012 6:48 PM gdeepak.com 102

103 Important Points Applying both the improvements will bring LgN to a very small value and effectively bringing it close to 2N after applying other small improvements. Actually We generally use double buffering and the time in which one page comes from the hard disk to the RAM, in the same time other page gets sorted, So the Swap time dominates over the sorting time. 6/6/2012 6:48 PM gdeepak.com 103

104 Questions, Comments and Suggestions 6/6/2012 6:48 PM gdeepak.com 104

105 Question 1 The sequence 20, 15, 18, 7, 9, 17, 12, 3, 6, 2 is a maxheap. The sequence 20, 18, 15, 9, 7, 6, 5, 3, 2, 1 is a max-heap. A) True, True B) True, False C) False, True D) False, False 6/6/2012 6:48 PM gdeepak.com 105

106 Question 2 Where an item with the largest key will be stored in a min heap A) at the root B) at any internal node C) at any external node D) at any node in the last level of the tree 6/6/2012 6:48 PM gdeepak.com 106

107 Question 3 If an in-place sorting algorithm is applied on the 2- dimensional sorted array, it will always output an unchanged array in the same sequence. Assume sorting is done based on the age. Name q r s t u v w x y Age A) True, Why B) False, Why 6/6/2012 6:48 PM gdeepak.com 107