Unit 6 Chapter 15 EXAMPLES OF COMPLEXITY CALCULATION

Transcription

1 DESIGN AND ANALYSIS OF ALGORITHMS Unit 6 Chapter 15 EXAMPLES OF COMPLEXITY CALCULATION EXAMPLES FROM THE SORTING WORLD Sorting provides a good set of examples for analyzing complexity. Simple sorting by repeatedly selecting the minimum element from an array gives an upper bound of O(n 2 ) for an n-element array. Here we look at other methods for sorting to demonstrate that good algorithm design can improve this bound. Remember that though a typical sorting application actually sorts records, data objects consisting of several components or fields. For example, records representing people might have a combination of last name and first name as the key. 1

2 1) BUCKET SORT In the case of a small range of integers, bucket sort is a good choice. Suppose the range is 0 to N -1. Then we can sort by constructing an array of N items into which the data are placed. The elements of this array are thought of as buckets. The general principle here is called distribution sorting, since we distribute the items into buckets. If the data items are just numbers, then, since all numbers are alike, the buckets can just be counters of the number of times each number occurs. If the data items are general records, then the buckets would need to be containers for a sufficient number of records. A linked list of records would be a good candidate implementation. The advantage of bucket sort is that it runs in O(n) time, because we need to only make one pass through the input data to put all of the data in buckets, then a pass over the buckets to create the sorted data. This analysis assumes that most buckets are nonempty. If there is a larger number of buckets than records and many of the buckets are empty, then the time to scan the buckets could dominate. Our O(n) figure assumes that almost all buckets have something in them. 2) RADIX SORT Radix sort is a variant of bucket sorting which uses fewer buckets by exploiting radix representations of the integer keys. If all the buckets are used, then the time to sort is O(n). The reason we qualify this way is that if there are more buckets than items to be sorted, and many buckets remain empty, handling the number of buckets could dominate the sorting time. If the range of input is large, we can conduct the distribution sort recursively, by dividing the range into sub-ranges, performing an initial distribution, then sorting within the buckets. A variation on this idea is the radix sorting principle. It is applicable when the values are represented as integer numerals in some base or radix. We sort on each digit of the numerals, starting with the least-significant. If the radix is b, then there are b buckets. We repeat this process, progressing towards the most-significant digit. After each distribution, we regroup the items anew taking care to preserve their order from the previous distribution. After the last re-grouping, the items are sorted. 2

3 3) SIMPLE INSERTION SORT Repeat the following process: Begin with a prefix of the array containing just one element as a sorted array. From the remaining elements, choose the next one and find its position in the sorted array. Insert the element by moving the higher elements upward by one. It can be seen that the insertion sort is O(n 2 ) by analyzing the program using step counting. Intuitively, bubble sort and insertion sort get their O(n 2 ) due to sequential nature of their attack on the problem: we have an outer loop that runs n steps and the cost of that loop ranges from 1 to n. If we are to break through O(n 2 ) to a lower value for the upper bound, we must find an approach that is not sequential. Here is where the divide-andconquer principle comes into use. ALGORITHM/CODE SNIPPET void insertionsort(int numbers[], int array_size) { int i, j, index; } for (i=1; i < array_size; i++) { index = numbers[i]; j = i; while ((j > 0) && (numbers[j-1] > index)) { numbers[j] = numbers[j-1]; j = j - 1; } numbers[j] = index; } 3

4 4) QUICK SORT The most popular divide-and-conquer sorting algorithm is quicksort. QuickSort is easier to state recursively: QuickSort: Sorting by divide-and-conquer Basis: If the sequence consists of atmost one element, it is sorted. Recursion: Break a sequence of more than one element into two, as follows: 1) Choose an element from the sequence as a pivot value. All elements less than the pivot value are selected as sub-sequence L and all elements greater than or equal to the pivot value are selected as sub-sequence R. 2) Sort the sub-sequences Land R recursively. Then form the sequence consisting of L (sorted) followed by the pivot, followed by R (sorted). Under ideal circumstances, the dividing phase of quicksort will split the elements into two equallength sub-sequences. In this case, there will be log n levels of recursive calls to quicksort. At each level, O(n) steps must be done to split the array. So the running time is of the order of O(n log n). Unfortunately, this is only in ideal circumstances, although by a probabilistic argument it also represents an average case performance under reasonable assumptions. The worst case performance, however, causes the array to be split unevenly, resulting in a worst case running time of O(n 2 ), which is the worst case for other sorting algorithms. In a worst-case sense, quicksort is not better than any other sorting algorithm. 4

5 ALGORITHM algorithm quicksort(a, lo, hi) if lo < hi then p := partition(a, lo, hi) quicksort(a, lo, p - 1 ) quicksort(a, p + 1, hi) algorithm partition(a, lo, hi) pivot := A[hi] i := lo - 1 for j := lo to hi - 1 do if A[j] < pivot then i := i + 1 swap A[i] with A[j] if A[hi] < A[i + 1] then swap A[i + 1] with A[hi] return i + 1 Worst-case analysis The most unbalanced partition occurs when the pivot divides the list into two sublists of sizes 0 and n 1. This may occur if the pivot happens to be the smallest or largest element in the list, or in some implementations when all the elements are equal. If this happens repeatedly in every partition, then each recursive call processes a list of size one less than the previous list. Consequently, we can make n 1 nested calls before we reach a list of size 1. This means that the call tree is a linear chain of n 1 nested calls. The ith call does O(n i) work to do the partition, and O(n²) so in that case Quicksort takes O(n²) time. 5

6 Best-case analysis In the most balanced case, each time we perform a partition we divide the list into two nearly equal pieces. This means each recursive call processes a list of half the size. Consequently, we can make only log 2 n nested calls before we reach a list of size 1. This means that the depth of the call tree is log 2 n. But no two calls at the same level of the call tree process the same part of the original list; thus, each level of calls needs only O(n) time all together (each call has some constant overhead, but since there are only O(n) calls at each level, this is subsumed in the O(n) factor). The result is that the algorithm uses only O(n log n) time. Average-case analysis To sort an array of n distinct elements, quicksort takes O(n log n) time in expectation, averaged over all n! permutations of n elements with equal probability. We list here three common proofs to this claim providing different insights into quicksort's workings. 5) HEAP SORT-USING A TREE TO SORT Tree Structuring Principle: Rather than dealing with the sequence linearly, try to employ a tree structure to cut sequence traversal needs from n to log n. How about doing insertions within a tree rather than in a linear array, as is done by the simple insertion sort? If there are n elements and we can do each insertion in O(log n) time, we might be able to achieve our goal. We have to work out the details concerning how the insertions can be done so as to maintain the balance of the tree. 6

7 6) MERGE SORT One natural method of sorting is to sort by repeated merging. Our analysis of mergesort is as follows: 1) The time to merge a pair of lists to get an n element list is O(n) since each recursive call "retires" an element to the result list and each element of the result gets retired only once. 2) Thus, the time to merge pairs on a list of lists, the summed total length of which is n, is O(n), since from (1) the time to merge a single pair is proportional to the number of elements in the result. 3) The time to use repeat on a list of n 'l-element lists is the number of times repeat is called recursively times O(n), from (2). 4) The number of times repea t is called is O(log n), since each call encounters a list which is about half as long as the previous call. 5) The time to mergesort an N element list is O(n) + time to repeat an N element list, since mapping an n element list is O(n). From (3) and (4), the time to repeat is O(n log n). Therefore the overall time to mergesort an n element list is O(n log n). ALGORITHM 7

8 8

9 SUMMARY OF COMPLEXITY AND CHARACTERISTICS OF SORTING ALGORITHMS We present in the Table (next slide) the most important characteristics, including time and space complexity, of various well known sorting algorithms. Note that a sorting method is called a stable method if it leaves the original order of items with the same key value undisturbed. This becomes important if several passes are made on a data set with different keys, in order to achieve the effect of a multi-component key. 9

10 Notes: 1) For merge sort, heap sort, quick sort, insertion sort, selection sort, and bubble sort, the input consists of n, possibly real, numbers. 2) For counting sort, the input consists of n integers from interval [1 N]. 3) Radix sort assumes input consisting of n integers in the interval [1 n k ]. 4) Quicksort can be implemented to be stable, but not in place. 5) Sorting based selection has the worst case and the average case running time of Θ(n log n). 6) Heap based selection has the worst case and the average case running time of Θ(n+klogn). COMPLEXITY OF SET OPERATIONS AND MAPPINGS A recurring theme in computer problem solving is the need to deal with various kinds of sets and mappings. Frequently, the need arises for sets of integers, strings, and more complex data types, as well as mappings from a wide variety of types to integers, and so on. A wide variety of different representations for these abstractions is available and they differ in the types of data they handle, the complexity, space requirements, and coding difficulty. Thinking of a set as a class, the following operations are typical: Find: Find out whether a member is in the set. If so, return a locator for it. A locator is a kind of reference to the member. It is used in deleting the member or changing it. Add: Add a new member to the set (assuming it is not present already). Delete: Given a locator for a member, remove the member from the set. New: Create a new set. 10

11 1) SETS IMPLEMENTATION USING AN UNORDERED ARRAY This is perhaps the simplest way to represent a set-as an array, the elements of which are in no particular order. The implementation is similar to that for a stack using an array, with an index indicating the last element in the set. Adding to the set (assuming space is available) is important, essentially the same as a push onto the stack. Finding a member requires a search through the array up until the point the element is found or the end is reached. In the worst case, all elements in the array are examined. Delete, given a locator, is simple: since order is not important, we can fill the hole left by deletion by putting the last element in its place (unless, of course, we wish to maintain the order of insertion, but once order becomes important, we no longer have a set). Letting N represent the current set size and M represent the maximum size, we can see from the above discussion that the following upper bounds hold on the set operations: find: O(N), add: O(1), delete: O(1), new: O(1). A similar discussion and set of bounds holds for using a linked-list to represent a set. 2) BINARY SEARCH PRINCIPLE We have already introduced binary search as an example of divide-and conquer. Here we shall treat it from algorithm complexity angle. Some improvement can be obtained, at the expense of coding complexity, by keeping the members of the set in order. This requires that an order be available for the domain of the set, which is not always the case. By keeping the set elements in order, the technique of binary search becomes available for the find operation. Here is how binary search works for find: Using index computation to find the index of the mid-point of the array, we take a "stab" at that point and compare it with the element to be found. If we have equality, we return the index as the locator. If the element to be found is less than the mid-point, then we search in the half array below the mid-point; if it is greater, we search in the half-array above. This process is repeated until the element is either found, or we are left with an empty array to search. 11

12 Each step of the binary search procedure reduces the number of elements under consideration by half. Therefore, the number of steps is proportional to the number of halving it takes to reduce the original number of elements, N, to less than or equal to 1. This number of steps is, of course, log 2 N. The superiority of binary search for find in an ordered array is to be balanced against the costs of addition and deletion. In order to maintain the ordering, we have to shift elements one way or the other, when we insert or delete. In the worst case, we might have to shift all the elements. The complexity for the ordered array case would thus appear as follows: find: O(log 2 N), add: O(N), delete: O(N), new: O(1). If most of the activity involving the set is of the find type, with few additions or deletions, then binary search on an ordered set is preferred to using an unordered set. 3) BINARY SEARCH TREES Does using an ordered linked-list help in a similar way? If we have a linked structure, additions and deletions usually become less complex. Unfortunately, a linked-list does not provide a good way to do the index computation required for binary search. That is, we would like to have a way to find the mid-point of a list similar to what was available for an array. Without adding additional structure, there is no way to do this. The desire to have something similar to a linked structure on which a binary search can be done has motivated the use of various tree structures, the most obvious of which is the binary search tree. 12

13 A binary search tree enables binary searching on a linked structure by approximating access to successive mid-points, similar to the way that binary search was described. To do this, each node in the structure has two links, one to the elements below the current one and one to the elements above. Figure shows a binary search tree that holds the set {1, 2, 4, 6, 8, 10, 11, 12, 13, 14, 15} The defining property of a binary search tree is that all elements in the left sub-tree of a node are less than the node itself, while all elements in the right sub-tree are greater than or equal to the node. If the binary search tree is balanced, meaning that for a tree with N nodes, the length of the longest path is at most 1 + log 2 N, then the search can be done in time O(log 2 N). We can also insert and delete in time O(log 2 N). However, it is not obvious that we can insert and delete in this amount of time and still maintain the balance needed for O(log 2 N) search. In fact, several extensions of the binary search trees have been developed which maintain this balance. These include AVL-trees, 2-3 trees, trees, red-black trees, and B-trees. 13

14 4) BIT VECTORS If the domain of a set's elements consists of integers, or can be easily mapped into integers (for example, character strings can be considered as large-radix numerals and are thus identifiable as integers), then a fast method of set representation is available. A bit vector is an array with one array element per domain element. The value of an element is either 1, indicating that the corresponding domain element is a member of the set being represented, or 0 indicating that it is not. For example, if the domain is {0,...,15}, then a bit vector representation of the set {1, 2, 4, 6, 8, 10, 11, 12, 13, 14, 15} is as given in Figure. The advantage of a bit vector is that find is extremely fast. We simply index the position corresponding to the element we are trying to find. If the value is 1, the element is in the set, otherwise it is not. By the linear addressing principle, find is O(1) time. Similarly, add and delete are also O(1) time. In a bit vector, the actual elements are not stored, so the representation can be compact in terms of space. However, this compactness occurs only if the elements in the set are relatively dense in the range of possible elements. The amount of space required by a bit vector is O(M) where M is the domain size. If the set is sparse and M is large, much space will be wasted. Also, it requires time proportional to M to initialize a bit vector, compared to time proportional to the size of the set for the other representations studied so far. To summarise, for a bit vector: find: 0(1), add: 0(1), delete: 0(1), new: O(M ). 14

15 5) ANALYSIS OF HASHING You may have studied hashing techniques while studying data structures. Under normal conditions, the time for find using hashing only is at best O(1). This assumes that: 1) The elements of the set are distributed relatively evenly into buckets 2) The size of anyone bucket is bounded by a constant 3) The time to compute the hashing function is bounded by a constant Any of these assumptions can be violated, but there are ways to remedy the situation. For example, the first assumption can be satisfied by devising an appropriate hashing function. The size of buckets are bounded, assuming the first assumption holds, by making the table large enough. If the number of elements in the set is not known in advance, there are ways to extend the table size dynamically. This is not a trivial problem, but is solvable, using techniques such as "extendible hashing. The third assumption is true if there is a bounded number of digits in the representation of each element. Obviously this would not be true if we used arbitrarily long strings, but it is approximately true for many common cases. Since we have to look at every character of the string to be matched anyway, and the time to compute a typical hashing function is bounded by a constant times the length of the string, this time can be considered to be a part of the cost of looking up the elements to be found. 15

16 6) THE TRIE PRINCIPLE The Trie representation is in the form of a tree. Unlike binary search trees, but similar to radix sorting, tries can exploit situations where the data can be represented as numerals. Since the linear addressing principle applies at each level of a trie, the access time to any element of a trie is O(1) if the number of levels is bounded, or O(L) where L is the number of levels, in general. 7) SETS VS. BAGS AND MAPPINGS So far we have emphasized structures for storing sets. However, most of these structures can also be adapted to implement bags and mappings as well. Usually it is a matter of storing additional information with the element inside the representation. For example, with bags we store a number indicating the multiplicity of the element in the bag; the absence of an element implies multiplicity 0. With mappings, the elements stored in the set are the domain values, and with each element in the domain we store the corresponding range value. 16

17 VIDEO REFERENCES Bucket sort Radix Sort Insertion Sort Quick Sort QuickSort.htm Heap Sort Merge Sort THANK YOU Dr. Milan Vachhani Assistant Professor, Sunshine Group of Institutions, Rajkot 17