Introduction to Asymptotic Running Time Analysis CMPSC 122 I. Comparing Two Searching Algorithms Let's begin by considering two searching algorithms you know, both of which will have input parameters of an array A, an integer n, and a value v, subject to the following specification: PRE: v is initialized, n > 0, and A[1..n] is initialized POST: if there exists an i in 1..n s.t. v = A[i], FCTVAL = i; FCTVAL = -1 otherwise. Here is pseudocode for the linear search: LINEAR-SEARCH(A, n, v) i = 1 while i A.length and A[i] v i = i + 1 // check elements of array until end or we find key if i == A.length + 1 return -1 return i // case that we searched to end of array, didn t find key // case that we found key Here is pseudocode for the binary search: BINARY-SEARCH(A, n, v) found = FALSE high = n low = 1 while low < high and not found high + low mid = 2 // check elements until high and low cross or we find key // find index of midpoint if v = A[mid] found = TRUE if v < A[mid] high = mid 1 low = mid + 1 // found key at mid // key must be in first half, so eliminate second half // key must be in second half, so eliminate first half if found return mid return -1 // case that we found key at mid // case that we did not find key Question: The precondition as stated isn't quite strong enough for binary search. What must we require? Page 1 of 5 Prepared by D. Hogan for PSU CMPSC 122
We're interested in analyzing the running time of these algorithms. Let's do so by counting the number of comparison operations, i.e. =,, <, >,, and. Suppose we have this array: 70 82 85 86 89 90 91 99 100 Count how many comparison operations are necessary for each searching algorithm to search for each of the following values: Linear Search Binary Search 89 91 70 Of course, this is one very specific array, and we should generalize our analysis. Essentially, there are two factors that come into play: the size of the array and luck. As for the latter, there are really three different kinds of analysis of running time we could do: Best-case analysis Worst-case analysis Average-case analysis Their names are so good that we don't really need to write down definitions. Certainly, probability comes into play in assessing how likely it is the best, worst, and average cases happen. To keep analysis simpler, we look at one kind of analysis at a time. Note, then, that: Best-case analysis usually isn't too useful or enlightening. Sometimes the best case renders the algorithm at hand unnecessary, but sometimes the best case does arise naturally and is worth considering. Worst-case analysis is really what we're interested in. It's usually better to do better than what you advertise, so if our performance beats the worst case, that's good anyhow. Average case analysis comes up sometimes, but it often yields the same results as the worst case, so worst-case analysis is usually fine. Average case analysis usually involves probabilistic analysis. Thus, in this course, when talk about running times, assume we're talking about worst-case running times. As for array size, or, in general, input size, it's standard convention in computer science to use n to refer to the size of an input. Question: When does the worst case happen for each of linear search and binary search? Question 2: Using the simplifying assumption that n is an exact power of 2, how many comparisons are necessary in the worst case for n = 8? n = 32? n = 128? Page 2 of 5 Prepared by D. Hogan for PSU CMPSC 122
Finally, as is relevant, I wish to formalize the notation: Because the binary logarithm or base-2 logarithm comes up more often than any other in computer science, computer scientists default to meaning base 2 when talking about logarithms and use the notation lg n to mean the binary logarithm of n. From here on out, I'll use such notation in our notes (and leave other bases of logs entirely to later courses). II. Code Fragments and Elementary Operations So what exactly are we doing when we're analyzing running time? You may think the name is a bit of a misnomer, but we're not really using time as you know it on a clock. Why? Because different hardware and operating systems and other concerns at low levels of abstraction keep introduce factors we don't care about. Instead, we want a theoretically-clean measure that is a function of the input size. For everything we'll do in this course, that's a single value, n. Okay, so what are we counting? It depends: Usually, we want to count elementary operations. Those are addition, subtraction, multiplication, division, modulus, and exponentiation. Usually, assignment statements, array accesses, and comparisons get lumped in this category too. Sometimes, the driving force behind the running time of algorithms is only comparisons. While we could count all elementary operations, it will turn out in these cases that everything doesn't matter, so we just count comparison operations. (This was the case with searching, and it will also turn out to be the case with most, but not all, sorting algorithms.) This is vague, and that's okay. Above all, we should write down what we're assuming does and does not count in doing analysis. This kind of counting, instead of measuring clock time, is machine independent, which is nice. Our analysis focuses on the algorithms alone, not external factors. Furthermore, the number of steps must always be a whole number and the input size must also be a whole number. Each step of an algorithm stands alone; this means our analysis is what we call discrete (as opposed to continuous). Before we go any further, though, it helps to divert and derive a very fundamental counting rule. Question: How many integer values are there between 4 and 7, inclusive? Question 2: How many integer values are there between -2 and 3, inclusive? Question 3: How many integer values are there between m and n, inclusive? So, with this in mind, let's consider several examples, wherein we will, in each case, count the exact number of elementary operations that are done. (Note that many of these code fragments are useless; the point is only to count.) Example 1: for i = 1 to n a = b + c - i Example 2: for i = 0 to n - 1 a = b + c - i Page 3 of 5 Prepared by D. Hogan for PSU CMPSC 122
Example 3: for i = 0 to n - 1 c = a*a / 2 + 1 Example 4: for i = 1 to n / 2 A[i] = i^2 + i/2 + a + 12 Example 5: for i = 1 to n / 2 A[i] = i^2 + a Example 6: for i = 1 to 2n Example 7: for i = 1 to 2n x = a^2-25 y = x + 1 Example 8: for i = 1 to n for i = 1 to n Now, let's compare what we've seen Page 4 of 5 Prepared by D. Hogan for PSU CMPSC 122
III. Theta Notation, Informally Suppose n is particularly large. Compare the expressions (or, more accurately, functions for the running times of the algorithms for input size n) above. It turns out, we're interested how functions grow with growing input sizes. The kind of analysis we'll do is called asymptotic analysis. Informally, we can simplify expressions to something called Θ-notation via the following motto: "Drop leading coefficients and lower order terms." Doing so, all of the examples could then be written as having running times of Θ(n). (Getting just a little bit more formal about it, Θ-notation gives a tight bound on the function that expresses the running time of each algorithm.) Problem: Simplify each of the following expressions to Θ-notation: n 2 3n 2 n 2 + 5 Problem: Simplify each of the following expressions to Θ-notation: n 5 + n 4 + 2 n 5 2 12n 5 + n 2 Homework: Read (in some cases, reread) all of Chapter 2 of Algorithms Unlocked. Using software that allows you to generate graphs (Matlab, Wolfram Alpha, Maple), where n is on the horizontal axis (functions could be translated to use x as n if it's easier) and you are only concerned with the first quadrant, do the following: 1. Consider the expressions we derived for the number of elementary operations for each of the code fragments in II. Pick any five of them and graph them on the same set of axes. 2. In the examples in III, we looked at two different sets of functions and simplified them to Θ-notation. Prepare two graphs, one for each set of functions, where you graph all of the original functions (not the simplified form) together. 3. Comment on your observations from all of the above graphs. 4. Now pick any one function from those you graphed in #1, any one from the first set in #2, and any one from the second set in #2. Graph those three functions together and make observations. 5. Lastly, graph each of the following together and make observations: n 2, log n, 2 n, 12, n, n 3, 50 n Page 5 of 5 Prepared by D. Hogan for PSU CMPSC 122