Lower and Upper Bound Theory. Prof:Dr. Adnan YAZICI Dept. of Computer Engineering Middle East Technical Univ. Ankara - TURKEY

Transcription

1 Lower and Upper Bound Theory Prof:Dr. Adnan YAZICI Dept. of Computer Engineering Middle East Technical Univ. Ankara - TURKEY 1

2 Lower and Upper Bound Theory How fast can we sort? Lower-Bound Theory can help us answer this question. Most of the sorting algorithms are comparison sorts: only use comparisons to determine the relative order of elements. Examples: insertion sort, merge sort, quicksort, heapsort. The best worst-case running time that we ve seen for comparison sorting is O(nlgn). Is O(nlgn) the best we can do? 2

3 Lower and Upper Bound Theory Lower Bound, L(n), is a property of the specific problem, i.e. sorting problem, MST, matrix multiplication, not of any particular algorithm solving that problem. Lower bound theory says that no algorithm can do the job in fewer than L(n) time units for arbitrary inputs. For example, every comparison-based sorting algorithm must take at least L(n), nlgn, time in the worst case. L(n) is the minimum over all possible algorithms, of the maximum complexity. 3

4 Lower and Upper Bound Theory Upper bound theory says that for any arbitrary inputs, we can always sort in time at most U(n). How long it would take to solve a problem using one of the known algorithms with worst-case input gives us a upper bound. U(n) is the minimum over all known algorithms, of the maximum complexity. For example: Finding the maximum element in an unsorted array is U(n), O(n). Both upper and lower bounds are minima over the maximum complexity of inputs of size n. Improving an upper bound means finding a new algorithm with better worst-case performance. The ultimate goal is to make these two functions coincide. When this is done, the optimal algorithm will have L(n) = U(n). 4

5 Examples: Lower Bounds an estimate on a minimum amount needed to multiply two n-by-n matrices of work needed to solve a given problem. number of comparisons needed to find the mean element in a set of n numbers. number of comparisons needed to sort an array of size n. number of comparisons necessary for searching in a sorted array. number of multiplications for n-digit integers. 5

6 Lower Bounds (cont.) Lower bound can be an exact count an efficiency class (Ω) Tight lower bound: there exists an algorithm with the same efficiency as the lower bound Problem Lower bound Tightness sorting Ω(nlogn) yes searching in a sorted array Ω(logn) yes element uniqueness Ω(nlogn) yes n-digit integer multiplication Ω(n) unknown multiplication of n-by-n matrices Ω(n 2 ) unknown 6

7 Methods for Establishing Lower Bounds There are few techniques for finding lower bounds: trivial lower bound information-theory decision tree problem reduction adversary arguments 7

8 Trivial Lower Bound Method 1) Trivial Lower Bounds: For many problems it is possible to easily observe that a lower bound identical to number of inputs (n) exists, (or possibly outputs) to the problem. The method is based on counting the number of items that must be processed in input and generated as output. Note that any algorithm must, at least, read its inputs and write its outputs. Example-1: Multiplication of a pair of nxn matrices requires that 2n 2 inputs be examined and n 2 outputs be computed, and the lower bound for this matrix multiplication problem is therefore Ω(n 2 ). 8

9 Trivial Lower Bound Method Example: Finding maximum (or minimum) of unordered array requires examining each input, so it is Ω(n). A simple counting arguments shows that any comparison-based algorithm for finding the maximum value of an element in a list of size n must perform at least n-1 comparisons for any input. Comment: This method may and may not be useful be careful in deciding how many elements must be processed 9

10 Information Theory The information theory method establishing lower bounds by computing the limitations on information gained by a basic operation and then showing how much information is required before a given problem is solved. This is used to show that any possible algorithm for solving a problem must do some minimal amount of work. The most useful principle of this kind is that the outcome of a comparison between two items yields one bit of information. 10

11 Lower and Upper Bound Theory Example: For the problem of searching an ordered list with n elements for the position of a particular item, <a,b,c,d> > <a,b> <c,d> > > a b c d For any outcome, at least 2 comparisions, therefore, 2 bits of information are necessary. 11

12 Information Theory Example: The lower bound of the problem of searching an ordered list with n elements for the position of a particular item is Θ(lgn). Proof: There are n possible outcomes, input strings In this case lgn comparisons are necessary, If a comparison between two items yields one bit of information, then unique identification of an index in the list requires lgn bits. Therefore, lgn bits of information are necessary to specify one of the n possibilities. 12

13 Information Theory Example-2: By using information theory, the lower bound for the problem of comparison-based sorting problem. input > > > <1,2,3> 111 > <2,1,3> > <1,3,2> <3,1,2> <2,3,1> <3,2,1> 000 Figure: There are 3! = 6 possible permutations of the n input elements, so lgn! bits are required for lgn! comparisons for sorting n things. 13

14 Information Theory Example: By using information theory, the lower bound for the problem of comparison-based sorting problem is Θ(nlgn) Proof: If we only know that the input is orderable, then there are n! possible outcomes each of the n! permutations of n things. Within the comparison-swap model, we can only use comparisons to derive information Since each comparison only yields one bit of information, the question is what the minimum number of bits of information needed to allow n! different outcomes is, which is lgn! bits. 14

15 Information Theory How fast lgn! grow? We can bind n! from above by overestimating every term of the product, and bind it below by underestimating the first n/2 terms. n/2*n/2* *n/2* *2*1 n*(n-1)* *2*1 n* n* *n (n/2) n/2 n! n n ½(nlgn-n) lgn! nlgn ½nlgn- ½ n lgn! nlgn ½nlgn lgn! nlgn This follows that lgn! Θ(nlgn) 15

16 Decision-tree model This method can model the execution of any comparison based problem. One tree for each input size n. View the algorithm as splitting whenever it compares two elements. The decision tree contains the comparisons along all possible instruction traces. The running time of the algorithm = the length of the path taken. 16

17 Decision-tree model Decision tree is a convenient model of algorithms involving comparisons in which: Internal nodes represent comparisons leaves represent outcomes. There must be n internal nodes in all of these trees corresponding to the n possible successful occurrences of x in A. Because every leaf in a valid decision tree must be reachable, the worst-case number of comparisons done by such a tree is the number of nodes in the longest path from the root to a leaf in the decision tree consisting of the comparison nodes. Worst-case running time = the length of the longest path of tree = the height of tree. 17

18 Decision-tree Model for Searching Example: Comparison-based searching problem Proof: Let us consider all possible decision trees which model algorithms to solve the searching problem. FIND(n) is bounded below by the distance of the longest path from the root to a leaf in such a tree. 18

19 Decision-tree model for Searching Height=log 2 n x:a( lg(n+1)/2 ) > x:a( lg(n+1)/4 ) x:a( lg(3n+1)/4 ) >. > x:a(1). x:a( lg(n+1)/4-1) x:a( lg(n+1)/4 +1) x:a(n) > > > > Return Failure Return Failure Return Failure Return Failure A decision tree for a search algorithm 19

20 Decision-tree Model for Search on Sorted Elements 3 < > = Height=log2(7)=3 < > < = > = < = > < > < > = = < = > F 0 F F 2 F F 4 F F 6 F Example decision tree for search on sorted list <0,1,2,3,4,5,6> 20

21 Decision-tree Model for Searching Example: Comparison-based searching problem A decision tree, with n leaves and height h, n 2 h. Indeed, a decision tree of height h with largest number of leaves has all its leaves on the last level. Hence, the largest number of leaves in such a tree is 2 h. FIND(n) = h log (n) Since, n 2 h and. Such a longest path is Θ(lgn). Therefore, the lower bound for comparisonbased searching on ordered input is Θ(lgn). 21

22 Decision-tree model for Searching Return(A(1)) = x:a(1) = Return(A(2)) x:a(2) Height = n x:a(n) = Return(A(n)) Return(A(0)) A decision tree for search on unsorted list The number of comparisons in the worst-case is equal to the height (h) of the algorithm s decision tree. h n and h Ω(n). 22

23 Decision-tree for Search on Unsorted Elements Height = 5 4 = 4 1 = 1 5 = 5 3 = 3 2 = 2 0 = 0 F Example decision tree for search on unsorted list <4,1,5,3,2,0>, that is, h = n. Therefore, the lower bound for comparisonbased searching on unordered input is Θ(n). 23

24 Decision-tree model for Sorting a1,a2,a3 yes a1<a2 no a1a2a3 a2a1a3 yes a2<a3 no yes a1<a3 no a1<a2<a3 yes a1a3a2 a1<a3 no a2<a1<a3 yes a2a3a1 a2<a3 no a1<a3<a2 a3<a1<a2 a2<a3<a1 a3<a2<a1 Each internal node is labelled for i,j element-of {1,2,...,n}. The left subtree shows subsequent comparisons if ai aj. The right subtree shows subsequent comparisons if ai aj. 24

25 Decision-tree model for sorting Each permutation represents a distinct sorted order and the tree muct have n! leaves. Since a binary tree of height h has no more than 2 h leaves, we have n! 2 h. 25

26 26

27 Problem Reduction Prob. 1 (to be solved) (reduction) Prob 2 (solvable by Alg.A) Prob 2 (solvable by Alg.A) (Alg.A) Soln. to Prob. 1 Another elegant means of proving a lower bound on a problem P is to show that problem P is at least as hard as another problem Q with a known lower bound. We need to reduce Q to P (not P to Q). In other words, we should show that an arbitrary instance of problem Q can be transformed (in a reasonably efficient way) to an instance of problem P, so that any algorithm solving P would also solve Q as well. Then a lower bound for Q will be a lower bound for P. Prob. Q (LB is known) (reduction) Prob.P (whose LB to be known) 27

28 Problem Reduction Example: Euclidean minimum spanning tree (MST) problem: Given n points in the Cartesian plane construct an Euclidian MST, whose vertices are the given points. As a problem with a known LB, we use the element uniqueness problem (EUP), whose LB is known as Ω(nlgn). The reduction is as follows: Transform any set x 1,x 2,,x n of n real numbers into a set of points in the Cartesian plane by simply adding 0 as the points s y coordinate: (x 1,0), (x 2,0),,(x n,0). This reduction implies that Ω(nlgn) is a LB for the Euclidian MST problem as well. Solution of EUP can now be found using the solution of Euclidean MST: Let T be any Euclidean MST found for this set of points, (x 1,0), (x 2,0),,(x n,0). Since T must contain a shortest edge, checking whether T contains a zero length edge will answer the question about uniqueness of the given numbers, x 1,x 2,,x n. 28

29 Example for reduction: Convex Hull Example: Computing the convex hull in Computational Geometry is perhaps the most basic, elementary function on a set of points. A Convex Hull is the smallest convex polygon that contains every point of the set S. A polygon P is convex if and only if, for any two points A and B, inside the polygon, then the line segment AB is inside P. 29

30 Example for reduction Convex Hull (cont.) One way to visualize a convex hull is to put a "rubber band" around all the points, and let it wrap as tight as it can. We can think of a rubber balloon for the threedimensional case. 30

31 Example for reduction Convex Hull (cont.) Theorem: The comparison-based sorting problem is transformable (in linear time) to the convex hull problem, therefore; the problem of finding the ordered convex hull of n points in the plane requires Ω(nlogn) time. That is, LB of the problem of convex hull is also Ω(nlogn). Proof: First we introduce a proposition: Proposition (Lower Bounds via Transformability): If a problem A is known to require T(n) time and A is τ(n)- transformable to B (A τ(n) B) then B requires at least T(n) O(τ(n)) time. 31

32 Example for reduction Convex Hull (cont.) Reduction: Given n real sorted numbers x 1, x 2,..., x n, all positive, we determine corresponding points (x 1,x 12 ), (x 2,x 22 ),, (x n,x n2 ). All of these points lie on the parabola y = x 2 and they all together result in a convex hull. Since LB of comparison-based sorting problem is known, Ω(nlogn), the LB of the problem of convex hull is also Ω(nlogn). So, when n real numbers x 1, x 2,..., x n are given, we can show how a convex hull algorithm can be used to sort these numbers with only linear overhead. The convex hull of this set, in standard form, will consist of a list of the points sorted. One pass through the list will enable us to read off the x i in order (in sorted form). 32

33 Example for reduction Convex Hull (cont.) y y = x 2 * * * * * x 1 x 2 x 3 x 4 x 5 x 6 * x Sorted 33

34 Example for reduction Convex Hull (cont.) Proposition (Upper Bounds via Transformability): If a problem B can be solved in T(n) time and problem A is τ(n)- transformable to B (A τ(n) B) then A can be solved in at most T(n) + O(τ(n)) time. 34

35 Oracles and Adversary Arguments Another technique for obtaining LBs consists of making use of an oracle or adversary arguments. Adversary arguments establish (worst-case) lower bounds by dynamically creating inputs to a given algorithm that force the algorithm to perform many basic operations. In order to derive a good LB, the adversary tries its best to cause the algorithm to work as hard as it might. We only require that the adversary gives answers that are consistent and truthful. We also assume that the algorithm uses a basic operation when querying the adversary. Clever adversaries force any given algorithm to make many queries before the algorithm has enough information to output the solution. LBs for the worst-case complexity are then obtained by counting the number of basic operations that the adversary forces the algorithm to perform. 35

36 Oracle and Adversary Arguments Adversary argument: a method of proving a lower bound by playing role of adversary that makes algorithm work the hardest by adjusting input. LBs for the worst-case complexity are then obtained by counting the number of basic operations that the adversary forces the algorithm to perform. Example-1: Guessing a number between 1 and n with yes/no questions. Adversary:Put the number in a larger of the two subsets generated by last question. For example; Adversary (or Oracle) knows your guess, which in the order of <4,7,9,4,8>. So, you put the the numbers in that order, 8 at the end, to force the algorithm the hardest way. That way, output is guessed as 8 only after n-1 guesses. That means the lower bound for this problem is Ω(n). Example-2: Merging two sorted lists of size n a 1 < a 2 < < a n and b 1 < b 2 < < b n Adversary: a i < b j iff i < j. There output: b 1 < a 1 < b 2 < a 2 < < b n < a n requires 2n-1 comparisons of adjacent elements. That means the lower bound for this problem is Ω(n). 36

37 Oracles and Adversary Arguments Example: (Merging Problem) Given the sets A(1:m) and B(1:n), where the items in A and in B are sorted. Investigate lower bounds for algorithms merging these two sets to give a single sorted set. Assume that all of the m+n elements are distinct and A(1)<A(2)< <A(m) and B(1) < B(2) < < B(n). Elementary combinatorics tells us that there are C((m+n), n)) ways that the A s and B s may merge together while still preserving the ordering within A and B. Thus, if we use comparison trees as our model for merging algorithms, then there will be C((m+n), n)) external nodes and therefore log C((m+n), m) comparisons are required by any comparison-based merging algorithm in worst-case. If we let MERGE(m,n) be the minimum number of comparisons need to merge m items with n items then we have the inequality log C((m+n), m) MERGE(m,n) m+n-1. The upper bound and lower bound can get arbitrarily far apart as m gets much smaller than n. 37