Algorithms and Data Structures (INF1) Lecture 7/15 Hua Lu Department of Computer Science Aalborg University Fall 2007
This Lecture Merge sort Quick sort Radix sort Summary We will see more complex techniques than those used by Bubble Sort, Selection Sort and Insertion Sort. 2
Divide and Conquer Divide-and-conquer is an important algorithm design technique for large-size problems. If the problem size is small enough to solve it in a straightforward manner, solve it. Otherwise Divide Divide the problem into two or more disjoint sub-problems Conquer Use divide-and-conquer recursively to solve the sub-problems Combine Take the solutions to the sub-problems and combine these solutions into a solution for the original problem 3
Merge Sort Problem: Sort an array A into non-descending order. Divide If A has at least two elements (nothing needs to be done if A has zero or one elements), remove all the elements from A and put them into two arrays, A 1 and A 2, each containing about half of the elements of S. (i.e. A 1 contains the first n/2 elements and A 2 contains the remaining n/2 elements). Conquer Sort arrays A 1 and A 2 using Merge Sort. Combine Put back the elements into A, by merging the sorted arrays A 1 and A 2 into one sorted array 4
Merge Sort Algorithm Merge-Sort(A, p, p, r) r) if if p < r then then q (p+r)/2 Merge-Sort(A, p, p, q) q) Merge-Sort(A, q+1, q+1, r) r) Merge(A, p, p, q, q, r) r) Merge(A, p, p, q, q, r) r) Take Take the the smallest of of the the two two topmost elements of of arrays A[p..q] and and A[q+1..r] and and put put it it into into an an additional array. Repeat this, this, until until both both arrays are are empty. Copy Copy the the additional array array into into A[p..r]. Merge(A, B) B) Merges two two sorted arrays into into one. one. Time Time complexity O( A + B ). 5
MergeSort (Example) - 1 MergeSort(A, 1, 8) 6
MergeSort (Example) - 2 MergeSort(A, 1, 4) 7
MergeSort (Example) - 3 MergeSort(A, 1, 2) 8
MergeSort (Example) - 4 MergeSort(A, 1, 1) 9
MergeSort (Example) - 5 MergeSort(A, 1, 1) returns. 10
MergeSort (Example) - 6 MergeSort(A, 2, 2) 11
MergeSort (Example) - 7 MergeSort(A, 2, 2) returns 12
MergeSort (Example) - 8 Merge(A, 1, 2, 2) 13
MergeSort (Example) - 9 MergeSort(A, 1, 2) returns 14
MergeSort (Example) - 10 MergeSort(A, 3, 4) 15
MergeSort (Example) - 11 MergeSort(A, 3, 3) 16
MergeSort (Example) - 12 MergeSort(A, 3, 3) returns 17
MergeSort (Example) - 13 MergeSort(A, 4, 4) 18
MergeSort (Example) - 14 MergeSort(A, 4, 4) returns 19
MergeSort (Example) - 15 Merge(A, 3, 4, 4) 20
MergeSort (Example) - 16 MergeSort(A, 3, 4) returns 21
MergeSort (Example) - 17 Merge(A, 1, 3, 4) 22
MergeSort (Example) - 18 MergeSort(A, 1, 4) returns 23
MergeSort (Example) - 19 MergeSort(A, 5, 8) 24
MergeSort (Example) - 20 Merge(A, 5, 7, 8) 25
MergeSort (Example) - 21 MergeSort(A, 5, 8) returns 26
MergeSort (Example) - 22 Merge(A, 1, 5, 8) 27
Merge Sort Summarized To sort n numbers if n=1 done! recursively sort 2 sequences of numbers n/2 and n/2 elements merge 2 sorted lists in Ο(n) time Strategy break problem into similar (smaller) subproblems recursively solve subproblems combine solutions to answer 28
Running time of MergeSort Again the running time can be expressed as a recurrence Tn ( ) solving_trivial_problem if n = 1 = num_pieces Tn ( / subproblem_size_factor) + dividing + combining if n> 1 T ( n ) = 1 if n = 1 2 T ( n / 2 ) + n if n > 1 29
Repeated Substitution Method Let s find the running time of merge sort (let s assume that n=2 b, for some b). T( n) 1 if n = 1 = 2 T( n/2) + n if n > 1 ( ) ( ) Tn ( ) = 2 Tn/2 + n substitute = 2 2 T n/ 4 + n/ 2 + n expand ( ) 2 = 2 Tn ( / 4) + 2 n substitute 2 = 2 (2 Tn ( /8) + n/ 4) + 2 n expand = + 3 2 Tn ( /8) 3 n observe the pattern i i Tn () = 2 Tn (/2) + in Let 2 i =n, i=lg n lgn 2 Tnn ( / ) + nlgn= n+ nlgn = 30
Quick Sort A divide-and-conquer algorithm Divide - the core of quick sort! Pick an element, called a pivot, from the array. Reorder the array so that all elements which are less than the pivot come before the pivot, and so that all elements greater than the pivot come after it (equal values can go either way). After this partitioning, the pivot is in its final position. This is called the partition operation. Conquer Recursively sort the 2 subarrays Combine Trivial since sorting is done in place Characteristics The divide-and-conquer nature is like merge sort, but it does not require an additional array It sorts in-place Very practical, average performance O(n log n) (with small constant factors), but worst case O(n 2 ) 31
Partitioning: Key Step in Quicksort We choose some (any) element p in the array as a pivot We then partition the array into three parts based on pivot Left part, pivot itself, and right part Partition returns the final position/index of p elements p elements greater less than p than or equal to p Then, Quicksort will be recursively executed on both left part and right part Quicksort(A, l, r) If l<r then i:=partition(a, l, r) Quicksort(A, l, i-1) Quicksort(A, i+1, r) p 32
Partition Algorithm Choose an array value (say, the first) to use as the pivot Starting from the left end, find the first element that is greater than or equal to the pivot Searching backward from the right end, find the first element that is less than the pivot Interchange (swap) these two elements Repeat, searching from where we left off, until done Partition(A,left,right):int p:= A[left]; l:=left+1; while l<r do while A[l]<p do l:=l+1; while A[r] p do r:=r-1; if l<r then swap(a, l, r) A[left]:=A[r]; A[r]:=p; return r; r:=right; 33
Example of Partitioning choose pivot: 4 3 6 9 2 4 3 1 2 1 8 9 3 5 6 search: 4 3 6 9 2 4 3 1 2 1 8 9 3 5 6 swap: 4 3 3 9 2 4 3 1 2 1 8 9 6 5 6 search: 4 3 3 9 2 4 3 1 2 1 8 9 6 5 6 swap: 4 3 3 1 2 4 3 1 2 9 8 9 6 5 6 search: 4 3 3124 3 1 2 98965 6 swap: 4 3 3122 3 1 4 98965 6 search: 4 3 312231498965 6 (l > r) swap with pivot: 1 3 312234 4 98965 6 34
Best Case of Quicksort Suppose each partition operation divides the array of size n nearly in half Then the depth of the recursion in log 2 n That s how many times we can halve n until we get n 1s At each level of the recursion, all the partitions at that level do work that is linear in n Each partition is linear over its sub-array All the partitions at one level cover the array Hence in the best case, quicksort has time complexity O(log 2 n) * O(n) = O(n log 2 n) Average case also has this complexity Detail omitted here in this course 35
Best Case Partitioning If we are lucky, Partition splits the array evenly O(lg n) * O(n) = O(n lg n) 36
Worst Case of Quicksort In the worst case, partitioning always divides the size n array into these three parts: A length one part, containing the pivot itself A length zero part, and A length n-1 part, containing everything else We don t recur on the zero-length part Recurring on the length n-1 part requires (in the worst case) recurring to depth n-1 In the worst case, recursion may be n levels deep But the partitioning work done at each level is still n So worst case for Quicksort is O(n) * O(n) = O(n 2 ) When does this happen? When the array is sorted to begin with! 37
Worst Case Partitioning O(n) * O(n) = O(n 2 ) 38
Picking a Better Pivot Before, we picked the first element of each sub-array to use as a pivot If the array is already sorted, this results in O(n 2 ) behavior It s no better if we pick the last element We could do an optimal quicksort (guaranteed O(n log n)) if we always picked a pivot value that exactly cuts the array in half Such a value is called a median half of the values in the array are larger, half are smaller The easiest way to find the median is to sort the array and pick the value in the middle (!) Ironically Random pivot Randomized algorithm of partitioning 39
More Comments on Quicksort Quicksort is the fastest known sorting algorithm Most real-world sorting is done by Quicksort For optimum efficiency, the pivot must be chosen carefully Fortunately, Quicksort is usually O(n log 2 n) However, no matter what you do, there will be some cases where Quicksort runs in O(n 2 ) time If the array is sorted to begin with It is possible to construct other bad cases 40
Radix Sort Every integer number k can be represented by at most d digits in base r (radix) All digits can be stored in an array A[1..d] k becomes A[1]A[2] A[d], where A[i] are digits in base r A[1]: the most significant digit A[d]: the least significant digit Example Decimal system 015, 155, 008, 319, 325, 100, 111: d=3, r=10 For 015: A[1]=0, A[2]=1, A[3]=5 Idea sort by the Least Significant Digit first (LSD) the numbers ended with 0 precede the numbers ended with 1, which precede those ended with 2, and so on so forth sort by the next least significant digit continue this until all numbers have been sorted on all d digits Also can sort by the Most Significant Digit first (MSD) 41
Radix Sort Algorithm RadixSort(q:Queue):Queue for i:=0 to r-1 do // Make a new queue for the corresponding digit, e.g. 0, 1, 2 9 A[i]:=Queue.make(); for i:= d downto 1 do // Loop on all digits of each number while (NOT q.isempty()) do Put each number into a x:= q.dequeue(); queue corresponding to j:=i th digit of x.value from right; its i th digit A[j].enqueue(x); for j:=0 to r-1 do q.append(a[j]); A[j]:=Queue.make(); // Make a new queue for that digit We use queues for simplicity of operation All numbers are stored in a queue q Intermediate queues are used Reassemble all numbers back to the original queue. Now they are sorted w.r.t the i th digits. 42
Radix Sort Example (1) Least significant digit first Numbers to sort: 275, 087, 426, 061, 509, 170, 677, 503 while-loop puts all numbers into different queues corresponding to their last digits for-loop combines all queues back into the original single queue q, in the order of radices 170 061 503 275 426 087 677 509 43
Radix Sort Example (2) 170 061 503 275 426 087 677 509 503 509 426 061 170 275 677 087 061 087 170 275 426 503 509 677 44
Radix Sort Analysis Loop invariant of the main for-loop (for i:=d downto 1 do) The queue elements are sorted according to their last d-i digits Increasing the base r decreases the number of passes d becomes smaller Running time (input size n) d passes over the numbers each pass takes q +r while (NOT q.isempty()) do for j:=0 to r-1 do total: O(( q +r)*d) worst case if q =O(n): O((n+r)*d) O(n) as r and d are constants Remarks Radix sort is not based on comparisons; the values are used as array indices when locating corresponding queues Radix sort is good for sorting long sequences of small numbers Large n, fixed (small) d and r 45
Summary of Sorting Algorithms Average Case Worst Case Remarks Bubble Sort O(n 2 ) Don t use it! Selection Sort O(n 2 ) O(n 2 ) Insertion Sort O(n 2 ) O(n 2 ) Merge Sort O(nlgn) O(nlgn) * Quick Sort O(nlgn) O(n 2 ) ** Radix Sort O((n+r) d) O((n+r) d) *** Favor small or sorted inputs. * Good for sorting on external devices with sequential access. ** The fastest general-purpose sorting algorithm. *** Not based on comparison, good for sorting long sequences of small numbers 46
Next Lecture Trees Basics Rooted trees Binary trees, balanced binary trees Tree traversal 47