Motivation of Sorting

Transcription

1 Sorting 1

2 Motivation of Sorting The term list here is a collection of records. Each record has one or more fields. Each record has a key to distinguish one record with another. For example, the phone directory is a list. Name, phone number, and even address can be the key, depending on the application or need. 2

3 Sorting Two ways to store a collection of records Sequential Non-sequential Assume a sequential list f. To retrieve a record with key f[i].key from such a list, we can do search in the following order: f[n].key, f[n-1].key,, f[1].key => sequential search 3

4 Example of An Element of A Search List class Element { public: int getkey() const {return key;}; void setkey(int k) {key = k;}; private: int key; // other records } 4

5 Sequential Search int SeqSearch (Element *f, const int n, const int k) // Search a list f with key values // f[1].key,, f[n].key. Return I such // that f[i].key == k. If there is no such record, return 0 { int i = n; f[0].setkey(k); while (f[i].getkey()!= k) i--; return i; } 5

6 Search A binary search only takes O(log n) time to search a sequential list with n records. An interpolation scheme relies on a ordered list. 6

7 Sorting Application So far we have seen two uses of sorting An aid in searching A means for matching entries in lists Sorting is used in other applications. Estimated 25% of all computing time is spent on sorting No one sorting method is the best for all initial orderings of the list being sorted. 7

8 Categories of Sorting Method Internal Method: Methods to be used when the list to be sorted is small enough so that the entire sort list can be carried out in the main memory. Insertion sort, quick sort, merge sort, heap sort and radix sort. External Method: Methods to be used on larger lists. 8

9 Insertion Sort Hypothesis: we know how to sort n-1 elements Induction on the n-th element sort n-1 elements put the n-th element in its correct place by scanning the n-1 sorted elements movements: O(n 2 ), comparison: O(n 2 ) improvement use binary search in finding correct place comparison: O(nlogn), movement: O(n 2 ) 9

10 Insertion Sort For N elements A[0], A[n-1], insertion sort consists of (N-1) passes (Pass 1 through N-1). In pass P, A[P] is moved left until its correct place is found among the first (p+1) elements, i.e., A[p] is inserted into the correct place among A[0],..A[p- 1] p... N-2 N Sorted Unsorted

11 Example of Insertion Sort A[0] A[1] A[2] A[3] A[4] A[5] Original after pass afetr pass after pass afetr pass after pass

12 Example of Insertion Sort (Cont.) Detail (for example, doing pass 4 after pass 3) A[0] A[1] A[2] A[3] A[4] A[5] Original after pass afetr pass after pass doing pass Tmp afetr pass after pass

13 Algorithm of Insertion Sort void InsertionSort(ElemenType A[ ], int N) { int j,p; Element Type Tmp; for (P=1; P < N; p++) { Tmp=A[P]; for (j=p; j>0 && A[j-1] > Tmp; j--) A[j]=A[j-1]; A[j]=Tmp; }; } 13

14 Selection Sort Hypothesis: we know how to sort n-1 elements Induction on a special n-th numbers sort n-1 elements select the minimal element from unsorted as the n-th element put in correct place by swapping movement: O(n-1), comparison: O(n 2 ) 14

15 Selection Sort (cont.) For N elements A[0], A[n-1], selection sort consists of (N-1) passes (Pass 0 through N-2). In pass P, select the smallest element from unsorted element (i.e., A[P],..A[N-1]), exchange with A[P] min p... N-2 N Sorted Unsorted exchange 15

16 Example of Selection Sort A[0] A[1] A[2] A[3] A[4] A[5] Original after pass afetr pass after pass afetr pass after pass

17 Example of Selection Sort (Cont.) Detailed (for example, doing pass 3 after pass 2) A[0] A[1] A[2] A[3] A[4] A[5] Original after pass afetr pass after pass doing pass minimun exchange afetr pass after pass

18 Algorithm of Selection Sort void SelectionSort(ElemenType A[ ], int N) { int j,p; Element Type Min; for (P=0; P <= N-2; P++) { Min=P; for (j=p+1; j <= N-1 ; j++) if (A[j] < A[Min]) Min=j; exchange (A[P], A[Min]); }; } 18

19 Bubble Sort For N elements A[0],,A[N-1], Bubble sort consists of (N-1) passes (Pass 0 through N-2). In pass P, adjacent elements in A[P],..,A[N-1] are compared & exchanging if necessary. After pass P, the first P elements have been in correct position. 19

20 Example of Bubble Sort A[0] A[1] A[2] A[3] A[4] A[5] Original after pass afetr pass after pass afetr pass after pass

21 Example of Bubble Sort (Cont.) Detailed (for example, doing pass 0) A[0] A[1] A[2] A[3] A[4] A[5] Original doing pass after pass afetr pass after pass afetr pass after pass

22 Algorithm of Bubble Sort void BubbleSort(ElemenType A[ ], int N) { int j,p; Element Type Min; for (P=0; P <=N-2; P++) { for (j=n-1; j >= P ; j--) if (A[j+1] > A[j]) exchange (A[j+1], A[j]); }; } 22

23 Merge Sort Hypothesis: we know how to sort n/2 elements Induction sort two n/2 elements merge Merge sort merge two sorted lists recursive algorithm Divide-and-conquer strategy drawback: merging step requires additional storage 23

24 Example of Merge Sort A[0] A[1] A[2] A[3] A[4] A[5] A[6] A[7] Original after pass afetr pass after pass

25 Example of Merge A[0] A[1] A[2] A[3] A[4] A[5] A[6] A[7] afetr pass p=0 q=4 4 p=0 q=5 4 6 p=0 q= p=1 q= p=2 q= p=2 q= p=3 q=

26 Algorithm of Merge Sort void MergeSort(ElemenType A[ ], ElementType Tmp[], int Left, Right) { int Center; if (Left < Right) { Center = (Left + Right)/2; MergeSort(A, Tmp, Left, Center); MergeSort(A, Tmp, Center+1, Right); Merge(A, Tmp, Left, Center+1, Right); } 26

27 Algorithm of Merge void Merge( ElementType A[ ], TmpArray[ ], int Lpos, int Rpos, int RightEnd ) { int i, LeftEnd, NumElements, TmpPos; LeftEnd = Rpos - 1; TmpPos = Lpos; NumElements = RightEnd - Lpos + 1; while ( Lpos <= LeftEnd && Rpos <= RightEnd ) if ( A[ Lpos ] <= A[ Rpos ] ) TmpArray[ TmpPos++ ] = A[ Lpos++ ]; else TmpArray[ TmpPos++ ] = A[ Rpos++ ]; while ( Lpos <= LeftEnd ) /* Copy rest of first half */ TmpArray[ TmpPos++ ] = A[ Lpos++ ]; while( Rpos <= RightEnd ) /* Copy rest of second half */ TmpArray[ TmpPos++ ] = A[ Rpos++ ]; for( i = 0; i < NumElements; i++, RightEnd-- ) A[ RightEnd ] = TmpArray[ RightEnd ]; 27 }

28 Analysis of Merge Sort Recurrence Relation: T(1)=1, T(N)=2T(N/2)+N Solution 1 T ( N / 2) N / 2 T ( N / 4) = N / T ( N) = N log N + N = O( N log N) 28

29 Analysis of Merge Sort (Cont.) Recurrence Relation: T(1)=1, T(N)=2T(N/2)+N Solution 2 T ( N) = 4T ( N / 4) + 2N... Using k =logn 29

30 Quick Sort the fastest known sorting algorithm in practice divide-and-conquer recursive strategy basic algorithm 1. Pick an element v as pivot 2. Partition two groups S1 & S2, x S1, x < v, y S2, y > v 3. Recursively quick sort S1 and S2 30

31 Quick Sort (Cont.) Select Pivot Partition Recursive Quicksort

32 void QuickSort(Element* list, const int left, const int right) // Sort records list[left],, list[right] into nondecreasing order on field key. Key pivot = list[left].key is // arbitrarily chosen as the pivot key. Pointers I and j are used to partition the sublist so that at any time // list[m].key pivot, m < I, and list[m].key pivot, m > j. It is assumed that list[left].key list[right+1].key. { if (left < right) { int i = left, j = right + 1, pivot = list[left].getkey(); do { do i++; while (list[i].getkey() < pivot); do j--; while (list[j].getkey() > pivot); if (i<j) InterChange(list, i, j); } while (i < j); InterChange(list, left, j); QuickSort(list, left, j 1); QuickSort(list, j+1, right); } 32

33 Picking the Pivot A wrong way: the first or the last element A safe maneuver: choose pivot randomly with the cost of random number generation. Median-of-Three Partitioning: base choice: the median value (how to find?) estimate: pick three elements randomly and use the median. use the median of the left, right, and center element. 33

34 Quick Sort Example Example 7.3: The input list has 10 records with keys (26, 5, 37, 1, 61, 11, 59, 15, 48, 19). R 1 R 2 R 3 R 4 R 5 R 6 R 7 R 8 R 9 R 10 Left Right [ [ ] 26 [ ] 1 5 [1 5] 11 [19 15] 26 [ ] [19 15] 26 [ ] [ ] [48 37] 59 [61] [61]

35 Improvement of Quicksort Quick sort doesn t perform well for small array Use insertion sort for small array when quick sort recursively. 35

36 Analysis of Quick Sort Recurrence Relation: T(N)=T(i)+T(N-i-1)+cN Worst case(o( N 2 )): T(0)=1, T(N)=T(N-1)+cN, N > 1 Best case(o(n logn)): T(N)=2T(N/2)+cN Average case(o(n logn)): T(N)= T(i)*Avg(T(i))+T(N-i-1)* Avg(T(N-i-1))+cN T ( N ) = [ ] N 1 T ( j cn 2 N ) + j= 0 36

37 General Lower Bound for Sorting Comparison-based Sorting Algorithm O(NlogN) Θ(NlogN) Ω(NlogN) Prove by decision tree 37

38 Decision Tree for Sorting Three Element A<B A<B<C A<C<B B<A<C B<C<A C<A<B C<B<A B<A A<B<C A<C<B C<A<B B<A<C B<C<A C<B<A A<C C<A B<C C<B B<C A<B<C A<C<B C<B C<A<B A<C B<A<C B<C<A C<A C<B<A A<B<C A<C<B B<A<C B<C<A 38

39 Proof of Lower Bound of Sorting A binary tree T of depth d has at most 2 d leaves A binary tree with L leaves have depth at least logl Any comparison-based sorting algorithm requires at least log(n!) comparisons in the worst case. Any comparison-based sorting algorithm requires Ω(NlogN) comparisons. ( log(n!) >= (N/2)*log(N/2) ) 39

40 Heaps Heaps Structure property: complete binary tree * complete binary tree: height logn * array implementation of heap Order property: for each node X, the key in the parent of X is smaller than or equal to the key in X. 40

41

42 Insert

43 void Insert( ElementType X, PriorityQueue H ) { int i; if( IsFull( H ) ) { Error( "Priority queue is full" ); return; } for ( i = ++H->Size; H->Elements[ i / 2 ] > X; i /= 2 ) H->Elements[ i ] = H->Elements[ i / 2 ]; H->Elements[ i ] = X; } 43

44 DeleteMin

45 ElementType DeleteMin( PriorityQueue H ) { int i, Child; ElementType MinElement, LastElement; if ( IsEmpty( H ) ) { Error( "Priority queue is empty" ); return H->Elements[ 0 ]; }; MinElement = H->Elements[ 1 ]; LastElement = H->Elements[ H->Size-- ]; for( i = 1; i * 2 <= H->Size; i = Child ) { Child = i * 2; if ( Child!= H->Size && H->Elements[ Child + 1] < H->Elements[ Child ] ) Child++; if ( LastElement > H->Elements[ Child ] ) H->Elements[ i ] = H->Elements[ Child ]; else break; }; H->Elements[ i ] = LastElement; return MinElement; 45

46 Complexity of Heap Operations Insertion: precolate up, O(logN) DeleteMin: precolate down, O(logN) Can heap be used in sorting? Complexity? 46

47 Heap Sort Algorithm (Increasing order) Step 1: Build heap (Min-heap) Step 2: for (i=0; i <N; i++) DeleteMin; 47

48 Improvement of Heap Sort Drawback of previous heap sort: extra array to store output Algorithm (Increasing order) Step 1: Build heap (Max-heap) Step 2: for (i=0; i <N; i++) DeleteMax; 48

49

50

51

52

53

54

55

56 Building Heap Approaches Top down Hypothesis: array [1..i] is a heap Bottom up Hypothesis: all trees represented by array A[i+1..n] satisfy the heap condition 56

57 57

58 Building Heap (ElementType A[], int N) { int i; for (i=n/2; i >=0; i--) PercDown(A, i, N) } 58

59

60

61

62

63 Bucket Sort Bucket sort allocate sufficient number of buckets & put element in corresponding buckets at the end, scan the buckets in order & collect all elements n elements, ranges from 1 to m m buckets 63

64 Radix Sort Drawback of bucket sort: waste buckets (space) Radix sort use several passes of bucket sort more than one number could fall into the same bucket Two approaches most significant bit (MSB): radixexchange sort least significant bit (LSB): straight-radix sort 64

65 Radix-Exchange Sort Given n elements represented by k-digits hypothesis: we know how to sort elements with left k digits induction use bucket sort 65

66 Radix-Exchange Sort (Cont.) Given (064, 008, 216, 512, 027, 729, 000, 001, 343, 125)

67 Straight-Radix Sort Given n elements represented by k-digits Hypothesis: sort elements with < digits Induction ignore the most significant bit & sort the n elements according to their k-1 least significant bits scan all the elements & use bucket sort on the most significant bit collect all the buckets in order 67

68 Straight-Radix Sort (Cont.) Given (64, 8, 216, 512, 27, 729, 0, 1, 343, 125) (0, 1, 512, 343, 64, 125, 216, 27, 8, 729) (0, 1, 8, 512, 216, 125, 27, 729, 343, 64) 68

69 Straight-Radix Sort (Cont.) (0, 1, 8, 512, 216, 125, 27, 729, 343, 64) (0, 1, 8, 27, 64, 125, 216, 343, 512, 729) 69

70 Algorithm Straight-Radix(X, n, k); Input: X (array of n elements with k digits, base p) Output:X begin all elements are initially in a global queue GQ for i:=1 to d do initialize queue Q[i] to be empty; for i:=k downto 1 do while GQ is not empty do pop x from GQ; d:= the i-th digit of x; insert x into Q[d]; for t:=1 to p do insert Q[t] into GQ; for i:=1 to n do pop X[i] from GQ end 70

71 Summary Of Internal Sorting No one method is best for all conditions Insertion sort: best for small number of N Merge sort Best worse-case behavior Needs more space Quick sort Best average-case behavior Worst case is O(n 2 ) Radix sort depends on the size of the keys and the number of buckets 71

72 External Sorting Some lists are too large to fit in the memory of a computer Some records are stored in the disk Merge sort Segments of the input list are sorted Sorted segments (called runs) are written onto external storage. Runs are merged until only run is left 72

73 Example 7.12 Consider a computer which is capable of sorting 750 records is used to sort 4500 records. Six runs are generated with each run sorting 750 records. Allocate three 250-record blocks of internal memory for performing merging runs. Two for input run 2 runs and the last one is for output. Three factors contributing to the read/write time of disk: seek time latency time transmission time 73

74 Example 7.12 (Cont.) run1 (1 750) run2 ( ) run3 ( ) run4 ( ) run5 ( ) run6 ( ) 74

75 Optimal Merging of Runs When runs are of difference size, it is more important to determine the run merge strategy weighted external path length = 2*3 + 4*3 + 5*2 + 15*1 = 43 weighted external path length = 2*2 + 4*2 + 5*2 + 15*2 = 52 75

76 Huffman Code Assume we want to obtain an optimal set of codes for messages M 1, M 2,, M n+1. Each code is a binary string that will be used for transmission of the corresponding message. At receiving end, a decode tree is used to decode the binary string and get back the message. A zero is interpreted as a left branch and a one as a right branch. If q i is the relative frequency with which message Mi will be transmitted, the expected decoding time is 1 i n+ 1 q i d i where d i is the distance of the external node for message Mi from the root node. 76

77 Huffman Codes (Cont.) The expected decoding time is minimized by choosing code words resulting in a decode tree with minimal weighted external path length M M 3 M 1 M 2 77

78 Huffman Function class BinaryTree { public: BinaryTree(BinaryTree bt1, BinaryTree bt2) { root->leftchild = bt1.root; root->rightchild = bt2.root; root->weight = bt1.root->weight + bt2.root->weight; } private: BinaryTreeNode *root; } void huffman (List<BinaryTree> l) // l is a list of single node binary trees as decribed above { int n = l.size(); // number of binary trees in l for (int i = 0; i < n-1; i++) { // loop n-1 times BinaryTree first = l.deleteminweight(); BinaryTree second = l.deleteminweight(); BinaryTree *bt = new BinaryTree(first, second); l.insert(bt); } } O(nlog n) 78

79 Huffman Tree Example Input: (a) (c) (b) 79

80 Huffman Tree Example (cont d) (e) (d) 80