1.4 ANALYSIS OF ALGORITHMS

Transcription

1 1.4 ANALYSIS OF ALGORITHMS AS people gain experience using computers, they use them to solve difficult problems or to process large amounts of data and are invariably led to questions like these: How long will my program take? Why does my program run out of memory? You certainly have asked yourself these questions, perhaps when rebuilding a music or photo library, installing a new application, working with a large document, or working with a large amount of experimental data. The questions are much too vague to be answered precisely the answers depend on many factors such as properties of the particular computer being used, the particular data being processed, and the particular program that is doing the job (which implements some algorithm). All of these factors leave us with a daunting amount of information to analyze. Despite these challenges, the path to developing useful answers to these basic questions is often remarkably straightforward, as you will see in this section. This process is based on the scientific method, the commonly accepted body of techniques used by scientists to develop knowledge about the natural world. We apply mathematical analysis to develop concise models of costs and do experimental studies to validate these models. Scientific method The very same approach that scientists use to understand the natural world is effective for studying the running time of programs: Observe some feature of the natural world, generally with precise measurements. Hypothesize a model that is consistent with the observations. Predict events using the hypothesis. Verify the predictions by making further observations. Validate by repeating until the hypothesis and observations agree. One of the key tenets of the scientific method is that the experiments we design must be reproducible, so that others can convince themselves of the validity of the hypothesis. Hypotheses must also be falsifiable, so that we can know for sure when a given hypothesis is wrong (and thus needs revision). As Einstein famously is reported to have said ( No amount of experimentation can ever prove me right; a single experiment can prove me wrong ), we can never know for sure that any hypothesis is absolutely correct; we can only validate that it is consistent with our observations. 172

2 1.4 Analysis of Algorithms 173 Observations Our first challenge is to determine how to make quantitative measurements of the running time of our programs. This task is far easier than in the natural sciences. We do not have to send a rocket to Mars or kill laboratory animals or split an atom we can simply run the program. Indeed, every time you run a program, you are performing a scientific experiment that relates the program to the natural world and answers one of our core questions: How long will my program take? Our first qualitative observation about most programs is that there is a problem size that characterizes the difficulty of the computational task. Normally, the problem size is either the size of the input or the value of a command-line argument. Intuitively, the running time should increase with problem size, but the question of by how much it increases naturally comes up every time we develop and run a program. Another qualitative observation for many programs is that the running time is relatively insensitive to the input itself; it depends primarily on the problem size. If this relationship does not hold, we need to take steps to better understand and perhaps better control the running time s sensitivity to the input. But it does often hold, so we now focus on the goal of better quantifying the relationship between problem size and running time. Example. As a running example, we will work with the program ThreeSum shown here, which counts the number of triples in a file of N integers that sum to 0 (assuming that overflow plays no role). This computation may seem contrived to you, but it is deeply related to numerous fundamental computational tasks (for example, see Exercise ). As a test input, consider the file 1Mints.txt from the booksite, which contains 1 million randomly generated int values. The second, eighth, and tenth entries in 1Mints.txt sum to 0. How many more such triples are there in the file? ThreeSum can tell us, but can it do so in a reasonable amount of time? What is the relationship between the problem size N and running time for ThreeSum? As a first experiment, try running ThreeSum on your computer for the files 1Kints.txt, 2Kints.txt, 4Kints.txt, and 8Kints.txt on the public class ThreeSum { public static int count(int[] a) { // Count triples that sum to 0. int N = a.length; int cnt = 0; for (int i = 0; i < N; i++) for (int j = i+1; j < N; j++) for (int k = j+1; k < N; k++) if (a[i] + a[j] + a[k] == 0) cnt++; return cnt; public static void main(string[] args) { int[] a = In.readInts(args[0]); StdOut.println(count(a)); Given N, how long will this program take?

3 174 CHAPTER 1 Fundamentals % more 1Mints.txt booksite that contain the first 1,000, 2,000, 4,000, and 8,000 integers from 1Mints.txt, respectively. You can quickly determine that there are 70 triples that sum to 0 in 1Kints.txt and that there are 528 triples that sum to 0 in 2Kints.txt. The program takes substantially more time to determine that there are 4,039 triples that sum to 0 in 4Kints.txt, and as you wait for the program to finish for 8Kints.txt, you will find yourself asking the question How long will my program take? As you will see, answering this question for this program turns out to be easy. Indeed, you can often come up with a fairly accurate prediction while % java ThreeSum Kints.txt the program is running. Stopwatch. Reliably measuring the exact running time of a given program can be difficult. Fortunately, we are usually happy with estimates. We want to be able to distinguish programs that will finish in a few seconds or a few minutes from those that might require a few days or a few months or more, and we want to know when one program is twice as fast as another for the same task. Still, we need accurate measurements to generate experimental data that we can use to formulate and to check the validity of hypotheses about the relationship between running time and problem size. For this purpose, we use the Stopwatch data type shown on the facing page. Its elapsedtime() method returns the elapsed time since it was created, in seconds. The implementation is based on using the Java system s currenttimemillis() method, which gives the current time in milliseconds, to save the time when the constructor is invoked, then uses it again to compute the elapsed time when elapsedtime() is invoked. 70 % java ThreeSum Kints.txt 528 % java ThreeSum Kints.txt 4039 tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick Observing the running time of a program

4 1.4 Analysis of Algorithms 175 API public class Stopwatch Stopwatch() double elapsedtime() create a stopwatch return elapsed time since creation typical client public static void main(string[] args) { int N = Integer.parseInt(args[0]); int[] a = new int[n]; for (int i = 0; i < N; i++) a[i] = StdRandom.uniform( , ); Stopwatch timer = new Stopwatch(); int cnt = ThreeSum.count(a); double time = timer.elapsedtime(); StdOut.println(cnt + " triples " + time); application % java Stopwatch triples seconds % java Stopwatch triples seconds implementation public class Stopwatch { private final long start; public Stopwatch() { start = System.currentTimeMillis(); public double elapsedtime() { long now = System.currentTimeMillis(); return (now - start) / ; An abstract data type for a stopwatch

5 176 CHAPTER 1 Fundamentals Analysis of experimental data. The program DoublingTest on the facing page is a more sophisticated Stopwatch client that produces experimental data for ThreeSum. It generates a sequence of random input arrays, doubling the array size at each step, and prints the running times of ThreeSum.count() for each input size. These experiments are certainly reproducible you can also run them on your own computer, as many times as you like. When you run DoublingTest, you will find yourself in a predictionverification cycle: it prints several lines very quickly, but then slows down considerably. Each time it prints a line, you find yourself wondering how long it will be until it prints the next line. Of course, since you have a different computer from ours, the actual running times that you get are likely to be different from those shown for our computer. Indeed, if your computer is twice as fast as ours, your running times will be about half ours, which leads immediately to the well-founded hypothesis that running times on different computers are likely to differ by a constant factor. Still, you will find yourself asking the more detailed question How long will my program take, as a function of the input size? To help answer this question, we plot the data. The diagrams at the bottom of the facing page show the result of plotting the data, both on a normal and on a log-log scale, with the problem size N on the x-axis and the running time T(N ) on the y-axis. The log-log plot immediately leads to a hypothesis about the running time the data fits a straight line of slope 3 on the log-log plot. The equation of such a line is lg(t(n )) = 3 lg N + lg a (where a is a constant) which is equivalent to T(N ) = a N 3 the running time, as a function of the input size, as desired. We can use one of our data points to solve for a for example, T(8000) = 51.1 = a , so a = and then use the equation T(N ) = N 3 to predict running times for large N. Informally, we are checking the hypothesis that the data points on the log-log plot fall close to this line. Statistical methods are available for doing a more careful analysis to find estimates of a and the exponent b, but our quick calculations suffice to estimate running time for most purposes. For example, we can estimate the running time on our computer for N = 16,000 to be about = seconds, or about 6.8 minutes (the actual time was seconds). While waiting for your computer to print the line for N = 16,000 in DoublingTest, you might use this method to predict when it will finish, then check the result by waiting to see if your prediction is true.

6 1.4 Analysis of Algorithms 177 program to perform experiments public class DoublingTest { public static double timetrial(int N) { // Time ThreeSum.count() for N random 6-digit ints. int MAX = ; int[] a = new int[n]; for (int i = 0; i < N; i++) a[i] = StdRandom.uniform(-MAX, MAX); Stopwatch timer = new Stopwatch(); int cnt = ThreeSum.count(a); return timer.elapsedtime(); results of experiments % java DoublingTest public static void main(string[] args) { // Print table of running times. for (int N = 250; true; N += N) { // Print time for problem size N. double time = timetrial(n); StdOut.printf("%7d %5.1f\n", N, time); standard plot 50 log-log plot straight line of slope running time T(N) lg(t(n)) K 2K 4K 8K problem size N Analysis of experimental data (the running time of ThreeSum.count()) 1K 2K 4K 8K lg N

7 178 CHAPTER 1 Fundamentals So far, this process mirrors the process scientists use when trying to understand properties of the real world. A straight line in a log-log plot is equivalent to the hypothesis that the data fits the equation T(N ) = a N b. Such a fit is known as a power law. A great many natural and synthetic phenomena are described by power laws, and it is reasonable to hypothesize that the running time of a program does, as well. Indeed, for the analysis of algorithms, we have mathematical models that strongly support this and similar hypotheses, to which we now turn. Mathematical models In the early days of computer science, D. E. Knuth postulated that, despite all of the complicating factors in understanding the running times of our programs, it is possible, in principle, to build a mathematical model to describe the running time of any program. Knuth s basic insight is simple: the total running time of a program is determined by two primary factors: The cost of executing each statement The frequency of execution of each statement The former is a property of the computer, the Java compiler and the operating system; the latter is a property of the program and the input. If we know both for all instructions in the program, we can multiply them together and sum for all instructions in the program to get the running time. The primary challenge is to determine the frequency of execution of the statements. Some statements are easy to analyze: for example, the statement that sets cnt to 0 in ThreeSum.count() is executed exactly once. Others require higher-level reasoning: for example, the if statement in ThreeSum.count() is executed precisely N (N 1)(N 2)/6 times (the number of ways to pick three different numbers from the input array see Exercise 1.4.1). Others depend on the input data: for example the number of times the instruction cnt++ in ThreeSum.count() is executed is precisely the number of triples that sum to 0 in the input, which could range from 0 of them to all of them. In the case of DoublingTest, where we generate the numbers randomly, it is possible to do a probabilistic analysis to determine the expected value of this quantity (see Exercise ). Tilde approximations. Frequency analyses of this sort can lead to complicated and lengthy mathematical expressions. For example, consider the count just considered of the number of times the if statement in ThreeSum is executed: N (N 1)(N 2)/6 = N 3 /6 N 2 /2 N/3

8 1.4 Analysis of Algorithms 179 As is typical in such expressions, the terms after the leading term are relatively small (for example, when N = 1,000 the value of N 2 /2 N/3 499,667 is certainly insignificant by comparison with N 3 /6 166,666,667). To allow us to ignore insignificant terms and therefore substantially simplify the mathematical formulas that we work with, we often use a mathematical device known as the tilde notation (~). This notation allows us to work with tilde approximations, where we throw away low-order terms that complicate formulas and represent a negligible contribution to values of interest: N function 166,666,667 N 3 /6 1,000 Leading-term approximation tilde approximation N (N 1)(N 2)/6 166,167,000 order of growth Definition. We write ~f (N ) to represent any function that, when divided by f (N ), approaches 1 as N grows, and we write g(n ) ~ f (N ) to indicate that g(n )/f (N ) approaches 1 as N grows. N 3 /6 N 2 /2 N/3 ~ N 3 /6 N 3 N 2 /2 N/2 ~ N 2 /2 N 2 lg N + 1 ~ lg N lg N 3 ~ 3 1 Typical tilde approximations order of growth description function constant 1 logarithmic linear linearithmic log N N N log N quadratic N 2 cubic N 3 exponential 2 N Commonly encountered order-of-growth functions For example, we use the approximation ~N 3 /6 to describe the number of times the if statement in ThreeSum is executed, since N 3 /6 N 2 /2 N/3 divided by N 3 /6 approaches 1 as N grows. Most often, we work with tilde approximations of the form g (N) ~af (N ) where f (N ) = N b (log N ) c with a, b, and c constants and refer to f (N ) as the order of growth of g (N ). When using the logarithm in the order of growth, we generally do not specify the base, since the constant a can absorb that detail. This usage covers the relatively few functions that are commonly encountered in studying the order of growth of a program s running time shown in the table at left (with the exception of the exponential, which we defer to CONTEXT). We will describe these functions in more detail and briefly discuss why they appear in the analysis of algorithms after we complete our treatment of ThreeSum.

9 180 CHAPTER 1 Fundamentals Approximate running time. To follow through on Knuth s approach to develop a mathematical expression for the total running time of a Java program, we can (in principle) study our Java compiler to find the number of machine instructions corresponding to each Java instruction and study our machine specifications to find the time of execution of each of the machine instructions, to produce a grand total. This process, for ThreeSum, is briefly summarized on the facing page. We classify blocks of Java statements by their frequency of execution, develop leading-term approximations for the frequencies, determine the cost of each statement, and then compute a total. Note that some frequencies may depend on the input. In this case, the number of times cnt++ is executed certainly depends on the input it is the number of triples that sum to 0, and could range from 0 to ~N 3 /6. We stop short of exhibiting the details (values of the constants) for any particular system, except to highlight that by using constant values t 0, t 1, t 2,... for the time taken by the blocks of statements, we are assuming that each block of Java statements corresponds to machine instructions that require a specified fixed amount of time. A key observation from this exercise is to note that only the instructions that are executed the most frequently play a role in the final total we refer to these instructions as the inner loop of the program. For ThreeSum, the inner loop is the statements that increment k and test that it is less than N and the statements that test whether the sum of three given numbers is 0 (and possibly the statement that implements the count, depending on the input). This behavior is typical: the running times of a great many programs depend only on a small subset of their instructions. Order-of-growth hypothesis. In summary, the experiments on page 177 and the mathematical model on page 181 both support the following hypothesis: Property A. The order of growth of the running time of ThreeSum (to compute the number of triples that sum to 0 among N numbers) is N 3. Evidence: Let T(N ) be the running time of ThreeSum for N numbers. The mathematical model just described suggests that T(N ) ~ an 3 for some machine-dependent constant a; experiments on many computers (including yours and ours) validate that approximation. Throughout this book, we use the term property to refer to a hypothesis that needs to be validated through experimentation. The end result of our mathematical analysis is precisely the same as the end result of our experimental analysis the running time of ThreeSum is ~ a N 3 for a machine-dependent constant a. This match validates both the experiments and the mathematical model and also exhibits more insight about the

10 1.4 Analysis of Algorithms 181 blocks of statements public class ThreeSum { public static int count(int[] a) { A int N = a.length; int cnt = 0; B C D E for (int i = 0; i < N; i++ ) for (int j = i+1; j < N; j++ ) for (int k = j+1; k < N; k++ ) return cnt; if (a[i] + a[j] + a[k] == 0) cnt++; 1 N ~N 2 / 2 ~N 3 / 6 x frequencies of execution public static void main(string[] args) { int[] a = In.readInts(args[0]); StdOut.println(count(a)); inner loop Anatomy of a program s statement execution frequencies statement block time in seconds frequency total time E t 0 x (depends on input) t 0 x D t 1 N 3 /6 N 2 /2 N/3 t 1 (N 3 /6 N 2 /2 N/3) C t 2 N 2 /2 N/2 t 2 (N 2 /2 N/2) B t 3 N t 3 N A t 4 1 t 4 grand total (t 1 /6) N 3 (t 2 /2 t 1 /2) N 2 (t 1 /3 t 2 /2 t 3 ) N t 4 t 0 x tilde approximation ~ (t 1 / 6) N 3 (assuming x is small) order of growth N 3 Analyzing the running time of a program (example)

11 182 CHAPTER 1 Fundamentals program because it does not require experimentation to determine the exponent. With some effort, we could validate the value of a on a particular system as well, though that activity is generally reserved for experts in situations where performance is critical. Analysis of algorithms. Hypotheses such as Property A are significant because they relate the abstract world of a Java program to the real world of a computer running it. Working with the order of growth allows us to take one further step: to separate a program from the algorithm it implements. The idea that the order of growth of the running time of ThreeSum is N 3 does not depend on the fact that it is implemented in Java or that it is running on your laptop or someone else s cellphone or a supercomputer; it depends primarily on the fact that it examines all the different triples of numbers in the input. The algorithm that you are using (and sometimes the input model) determines the order of growth. Separating the algorithm from the implementation on a particular computer is a powerful concept because it allows us to develop knowledge about the performance of algorithms and then apply that knowledge to any computer. For example, we might say that ThreeSum is an implementation of the brute-force algorithm compute the sum of all different triples, counting those that sum to 0 we expect that an implementation of this algorithm in any programming language on any computer will lead to a running time that is proportional to N 3. In fact, much of the knowledge about the performance of classic algorithms was developed decades ago, but that knowledge is still relevant to today s computers. Cost model. We focus attention on properties of algorithms by articulating a cost model that defines the basic operations used by the algorithms we are studying to solve the problem at hand. For example, an appropriate cost model for the 3-sum problem, shown at right, is the number of times we access an array 3-sum cost model. When studying algorithms to solve the 3-sum problem, we count array accesses (the number of times an array entry is accessed, for read or write). entry. With this cost model, we can make precise mathematical statements about properties of an algorithm, not just a particular implementation, as follows: Proposition B. The brute-force 3-sum algorithm uses ~N 3 /2 array accesses to compute the number of triples that sum to 0 among N numbers. Proof: The algorithm accesses each of the 3 numbers for each of the ~N 3 /6 triples. We use the term proposition to refer to mathematical truths about algorithms in terms of a cost model. Throughout this book, we study the algorithms that we consider within

12 1.4 Analysis of Algorithms 183 the framework of a specific cost model. Our intent is to articulate cost models such that the order of growth of the running time for a given implementation is the same as the order of growth of the cost of the underlying algorithm (in other words, the cost model should include operations that fall within the inner loop). We seek precise mathematical results about algorithms (propositions) and also hypotheses about performance of implementations (properties) that you can check through experimentation. In this case, Proposition B is a mathematical truth that supports the hypothesis stated in Property A, which we have validated with experiments, in accordance with the scientific method.

13 184 CHAPTER 1 Fundamentals Summary. For many programs, developing a mathematical model of running time reduces to the following steps: Develop an input model, including a definition of the problem size. Identify the inner loop. Define a cost model that includes operations in the inner loop. Determine the frequency of execution of those operations for the given input. Doing so might require mathematical analysis we will consider some examples in the context of specific fundamental algorithms later in the book. If a program is defined in terms of multiple methods, we normally consider the methods separately. As an example, consider our example program of Section 1.1, BinarySearch. Binary search. The input model is the array a[] of size N; the inner loop is the statements in the single while loop; the cost model is the compare operation (compare the values of two array entries); and the analysis, discussed in Section 1.1 and given in full detail in Proposition B in Section 3.1, shows that the number of compares is at most lg N 1. Whitelist. The input model is the N numbers in the whitelist and the M numbers on standard input where we assume M >> N; the inner loop is the statements in the single while loop; the cost model is the compare operation (inherited from binary search); and the analysis is immediate given the analysis of binary search the number of compares is at most M (lg N 1). Thus, we draw the conclusion that the order of growth of the running time of the whitelist computation is at most M lg N, subject to the following considerations: If N is small, the input-output cost might dominate. The number of compares depends on the input it lies between ~M and ~M lg N, depending on how many of the numbers on standard input are in the whitelist and on how long the binary search takes to find the ones that are (typically it is ~M lg N ). We are assuming that the cost of Arrays.sort() is small compared to M lg N. Arrays.sort() implements the mergesort algorithm, and in Section 2.2, we will see that the order of growth of the running time of mergesort is N log N (see Proposition G in chapter 2), so this assumption is justified. Thus, the model supports our hypothesis from Section 1.1 that the binary search algorithm makes the computation feasible when M and N are large. If we double the length of the standard input stream, then we can expect the running time to double; if we double the size of the whitelist, then we can expect the running time to increase only slightly.

14 1.4 Analysis of Algorithms 185 Developing MATHEMATICal models for the analysis of algorithms is a fruitful area of research that is somewhat beyond the scope of this book. Still, as you will see with binary search, mergesort, and many other algorithms, understanding certain mathematical models is critical to understanding the efficiency of fundamental algorithms, so we often present details and/or quote the results of classic studies. When doing so, we encounter various functions and approximations that are widely used in mathematical analysis. For reference, we summarize some of this information in the tables below. description notation definition fl o o r x largest integer not greater than x ceiling x smallest integer not smaller than x natural logarithm ln N log e N (x such that e x = N) binary logarithm lg N log 2 N (x such that 2 x = N) integer binary logarithm lg N largest integer not greater than lg N (# bits in binary representation of N ) 1 harmonic numbers H N 1 1/2 1/3 1/4... 1/N factorial N! N Commonly encountered functions in the analysis of algorithms description harmonic sum approximation H N = 1 1/2 1/3 1/4... 1/N ~ ln N triangular sum N ~ N 2 /2 geometric sum Stirling s approximation binomial coefficients N = 2N 1 ~ 2N when N = 2 n lg N! = lg 1 lg 2 lg 3 lg 4... lg N ~ N lg N ( N k ) ~ N k /k! when k is a small constant exponential (1 1/x) x ~ 1/e Useful approximations for the analysis of algorithms

15 186 CHAPTER 1 Fundamentals Order-of-growth classifications We use just a few structural primitives (statements, conditionals, loops, nesting, and method calls) to implement algorithms, so very often the order of growth of the cost is one of just a few functions of the problem size N. These functions are summarized in the table on the facing page, along with the names that we use to refer to them, typical code that leads to each function, and examples. Constant. A program whose running time s order of growth is constant executes a fixed number of operations to finish its job; consequently its running time does not depend on N. Most Java operations take constant time. Logarithmic. A program whose running time s order of growth is logarithmic is barely slower than a constant-time program. The classic example of a program whose running time is logarithmic in the problem size is binary search (see BinarySearch on page 47). The base of the logarithm is not relevant with respect to the order of growth (since all logarithms with a constant base are related by a constant factor), so we use log N when referring to order of growth. Linear. Programs that spend a constant amount of time processing each piece of input data, or that are based on a single for loop, are quite common. The order of growth of such a program is said to be linear its running time is proportional to N. Linearithmic. We use the term linearithmic to describe programs whose running time for a problem of size N has order of growth N log N. Again, the base of the logarithm is not relevant with respect to the order of growth. The prototypical examples of linearithmic algorithms are Merge.sort() (see Algorithm 2.4) and Quick.sort() (see Algorithm 2.5). Quadratic. A typical program whose running time has order of growth N 2 has two nested for loops, used for some calculation involving all pairs of N elements. The elementary sorting algorithms Selection.sort() (see Algorithm 2.1) and Insertion.sort() (see Algorithm 2.2) are prototypes of the programs in this classification. Cubic. A typical program whose running time has order of growth N 3 has three nested for loops, used for some calculation involving all triples of N elements. Our example for this section, ThreeSum, is a prototype. Exponential. In ChAPter 6 (but not until then!) we will consider programs whose running times are proportional to 2 N or higher. Generally, we use the term exponential to refer to algorithms whose order of growth is b N for any constant b > 1, even though different values of b lead to vastly different running times. Exponential algorithms are extremely slow you will never run one of them to completion for a large problem. Still, exponential algorithms play a critical role in the theory of algorithms because

16 1.4 Analysis of Algorithms 187 description order of growth typical code framework description example constant 1 a = b + c; statement add two numbers logarithmic log N [ see page 47 ] divide in half binary search linear N double max = a[0]; for (int i = 1; i < N; i++) if (a[i] > max) max = a[i]; loop find the maximum linearithmic N log N [ see Algorithm 2.4 ] divide and conquer mergesort quadratic N 2 for (int i = 0; i < N; i++) for (int j = i+1; j < N; j++) if (a[i] + a[j] == 0) cnt++; double loop check all pairs for (int i = 0; i < N; i++) cubic N 3 for (int j = i+1; j < N; j++) for (int k = j+1; k < N; k++) if (a[i] + a[j] + a[k] == 0) cnt++; triple loop check all triples exponential 2 N [ see chapter 6 ] exhasutive search check all subsets Summary of common order-of-growth hypotheses

17 188 CHAPTER 1 Fundamentals standard plot 500T exponential cubic quadratic time 200T 100T log-log plot time 512T 64T 8T 4T 2T T exponential there exists a large class of problems for which it seems that an exponential algorithm is the best possible choice. cubic quadratic linearithmic linear logarithmic constant 100K 200K 500K problem size linearithmic linear logarithmic constant These classifications are the most common, but certainly not a complete set. The order of growth of an algorithm s cost might be N 2 log N or N 3/2 or some similar function. Indeed, the detailed analysis of algorithms can require the full gamut of mathematical tools that have been developed over the centuries. A great many of the algorithms that we consider have straightforward performance characteristics that can be accurately described by one of the orders of growth that we have considered. Accordingly, we can usually work with specific propositions with a cost model, such as mergesort uses between ½ N lg N and N lg N compares that immediately imply hypotheses (properties) such as the order of growth of mergesort s running time is linearithmic. For economy, we abbreviate such a statement to just say mergesort is linearithmic. The plots at left indicate the importance of the order of growth in practice. The x-axis is the problem size; the y-axis is the running time. These charts make plain that quadratic and cubic algorithms are not feasible for use on large problems. As it turns out, several important problems have natural solutions that are quadratic but clever algorithms that are linearithmic. Such algorithms (including mergesort) are critically important in practice because they enable us to address problem sizes far larger than could be addressed with quadratic solutions. Naturally, we therefore focus in this book on developing logarithmic, linear, and linearithmic algorithms for fundamental problems. 1K 2K 4K 8K 512K problem size Typical orders of growth

18 1.4 Analysis of Algorithms 189 Designing faster algorithms One of the primary reasons to study the order of growth of a program is to help design a faster algorithm to solve the same problem. To illustrate this point, we consider next a faster algorithm for the 3-sum problem. How can we devise a faster algorithm, before even embarking on the study of algorithms? The answer to this question is that we have discussed and used two classic algorithms, mergesort and binary search, have introduced the facts that the mergesort is linearithmic and binary search is logarithmic. How can we take advantage of these algorithms to solve the 3-sum problem? Warmup: 2-sum. Consider the easier problem of determining the number of pairs of integers in an input file that sum to 0. To simplify the discussion, assume also that the integers are distinct. This problem is easily solved in quadratic time by deleting the k loop and a[k] from ThreeSum.count(), leaving a double loop that examines all pairs, as shown in the quadratic entry in the table on page 187 (we refer to such an implementation as TwoSum). The implementation below shows how mergesort and binary search (see page 47) can serve as a basis for a linearithmic solution to the 2-sum problem. The improved algorithm is based on the fact that an entry a[i] is one of a pair that sums to 0 if and only if the value -a[i] is in the array (and a[i] is not zero). To solve the problem, we sort the array (to enable binary search) and then, for every entry a[i] in the array, do a binary search for -a[i] with rank() in BinarySearch. If the result is an index j with j > i, we increment the count. This succinct test covers three cases: An unsuccessful binary search returns -1, so we do not increment the count. If the binary search returns j > i, we have a[i] + a[j] = 0, so we increment the count. If the binary search returns j between 0 and i, we also have a[i] + a[j] = 0 but do not increment the count, to avoid double counting. The result of the computation is precisely the same as the result of the quadratic algorithm, but it takes much less time. The running time of the mergesort is import java.util.arrays; public class TwoSumFast { public static int count(int[] a) { // Count pairs that sum to 0. Arrays.sort(a); int N = a.length; int cnt = 0; for (int i = 0; i < N; i++) if (BinarySearch.rank(-a[i], a) > i) cnt++; return cnt; public static void main(string[] args) { int[] a = In.readInts(args[0]); StdOut.println(count(a)); Linearithmic solution to the 2-sum problem

19 190 CHAPTER 1 Fundamentals proportional to N log N, and the N binary searches each take time proportional to log N, so the running time of the whole algorithm is proportional to N log N. Developing a faster algorithm like this is not merely an academic exercise the faster algorithm enables us to address much larger problems. For example, you are likely to be able to solve the 2-sum problem for 1 million integers (1Mints.txt) in a reasonable amount of time on your computer, but you would have to wait quite a long time to do it with the quadratic algorithm (see Exercise ). Fast algorithm for 3-sum. The very same idea is effective for the 3-sum problem. Again, assume also that the integers are distinct. A pair a[i] and a[j] is part of a triple that sums to 0 if and only if the value -(a[i] + a[j]) is in the array (and not a[i] or a[j]). The code below sorts the array, then does N (N 1)/ 2 binary searches that each take time proportional to log N, for a total running time proportional to N 2 log N. Note that in this case the cost of the sort is insignificant. Again, this solution enables us to address much larger problems (see Exercise ). The plots in the figure at the bottom of the next page show the disparity in costs among these four algorithms for problem sizes in the range we have considered. Such differences certainly motivate the search for faster algorithms. Lower bounds. The table on page 191 summarizes the discussion of this section. An interesting question immediately arises: Can we find algorithms for the 2-sum and 3-sum problems that are substantially import java.util.arrays; public class ThreeSumFast { public static int count(int[] a) { // Count triples that sum to 0. Arrays.sort(a); int N = a.length; int cnt = 0; for (int i = 0; i < N; i++) for (int j = i+1; j < N; j++) if (BinarySearch.rank(-a[i]-a[j], a) > j) cnt++; return cnt; public static void main(string[] args) { int[] a = In.readInts(args[0]); StdOut.println(count(a)); N 2 lg N solution to the 3-sum problem faster than TwoSumFast and ThreeSumFast? Is there a linear algorithm for 2-sum or a linearithmic algorithm for 3-sum? The answer to this question is no for 2-sum (under a model that counts and allows only comparisons of linear or quadratic functions of the numbers) and no one knows for 3-sum, though experts believe that the best possible algorithm for 3-sum is quadratic. The idea of a lower bound on the order of growth of the worst-case running time for all possible algorithms to solve a problem is a very powerful one, which we will

20 1.4 Analysis of Algorithms 191 revisit in detail in Section 2.2 in the context of sorting. Nontrivial lower bounds are difficult to establish, but very helpful in guiding our search for efficient algorithms. algorithm order of growth of running time TwoSum N 2 The examples in this section set the stage for our treatment of algorithms in this book. Throughout the book, our strategy for addressing new problems is the following: Implement and analyze a straighforward solution to TwoSumFast ThreeSum ThreeSumFast N log N N 3 N 2 log N the problem. We usually refer to such solutions, like Summary of running times ThreeSum and TwoSum, as the brute-force solution. Examine algorithmic improvements, usually designed to reduce the order of growth of the running time, such as TwoSumFast and ThreeSumFast. Run experiments to validate the hypotheses that the new algorithms are faster. In many cases, we examine several algorithms for the same problem, because running time is only one consideration when choosing an algorithm for a practical problem. We will develop this idea in detail in the context of fundamental problems throughout the book N 1000 N 3 /2 N 2 lgn ThreeSum array accesses (thousands) TwoSum array accesses (millions) TwoSumFast ThreeSumFast 4N lgn 1K 2K 4K 8K problem size N 1K 2K 4K 8K problem size N Costs of algorithms to solve the 2-sum and 3-sum problems

21 192 CHAPTER 1 Fundamentals Doubling ratio experiments The following is a simple and effective shortcut for predicting performance and for determining the approximate order of growth of the running time of any program: Develop an input generator that produces inputs that model the inputs expected in practice (such as the random integers in timetrial() in DoublingTest. Run the program DoublingRatio given below, a modification of DoublingTest that calculates the ratio of each running time with the previous. Run until the ratios approach a limit 2 b. This test is not effective if the ratios do not approach a limiting value, but they do for many, many programs, implying the following conclusions: The order of growth of the running time is approximately N b. To predict running times, multiply the last observed running time by 2 b and double N, continuing as long as desired. If you want to predict for an input size that is not a power of 2 times N, you can adjust ratios accordingly (see Exercise 1.4.9). As illustrated below, the ratio for ThreeSum is about 8 and we can predict the running times for N = 16,000, 32,000, 64,000 to be 408.8, , seconds, respectively, just by successively multiplying the last time for 8,000 (51.1) by 8. program to perform experiments public class DoublingRatio { public static double timetrial(int N) // same as for DoublingTest (page 177) public static void main(string[] args) { double prev = timetrial(125); for (int N = 250; true; N += N) { double time = timetrial(n); StdOut.printf("%6d %7.1f ", N, time); StdOut.printf("%5.1f\n", time/prev); prev = time; results of experiments % java DoublingRatio predictions

22 1.4 Analysis of Algorithms 193 This test is roughly equivalent to the process described on page 176 (run experiments, plot values on a log-log plot to develop the hypothesis that the running time is an b, determine the value of b from the slope of the line, then solve for a), but it is simpler to apply. Indeed, you can accurately predict preformance by hand when you run DoublingRatio. As the ratio approaches a limit, just multiply by that ratio to fill in later values in the table. Your approximate model of the order of growth is a power law with the binary logarithm of that ratio as the power. Why does the ratio approach a constant? A simple mathematical calculation shows that to be the case for all of the common orders of growth just discussed (except exponential): Proposition C. (Doubling ratio) If T(N) ~ a N b lg N then T(2N)/T(N) ~ 2 b. Proof: Immediate from the following calculation: T(2N)/T(N) = a (2N ) b lg (2N ) / a N b lg N = 2 b (1 + lg 2 / lg N ) ~ 2 b Generally, the logarithmic factor cannot be ignored when developing a mathematical model, but it plays a less important role in predicting performance with a doubling hypothesis. You should consider running doubling ratio experiments for every program that you write where performance matters doing so is a very simple way to estimate the order of growth of the running time, perhaps revealing a performance bug where a program may turn out to be not as efficient as you might think. More generally, we can use hypotheses about the order of growth of the running time of programs to predict performance in one of the following ways: Estimating the feasibility of solving large problems. You need to be able to answer this basic question for every program that you write: Will the program be able to process this given input data in a reasonable amount of time? To address such questions for a large amount of data, we extrapolate by a much larger factor than for doubling, say 10, as shown in the fourth column in the table at the bottom of the next page. Whether it is an investment banker running daily financial models or a scientist running a program to analyze experimental data or an engineer running simulations to test a design, it is not unusual for people to regularly run programs that take several hours to complete,

23 194 CHAPTER 1 Fundamentals so the table focuses on that situation. Knowing the order of growth of the running time of an algorithm provides precisely the information that you need to understand limitations on the size of the problems that you can solve. Developing such understanding is the most important reason to study performance. Without it, you are likely have no idea how much time a program will consume; with it, you can make a back-of-the-envelope calculation to estimate costs and proceed accordingly. Estimating the value of using a faster computer. You also may be faced with this basic question, periodically: How much faster can I solve the problem if I get a faster computer? Generally, if the new computer is x times faster than the old one, you can improve your running time by a factor of x. But it is usually the case that you can address larger problems with your new computer. How will that change affect the running time? Again, the order of growth is precisely the information needed to answer that question. A famous rule of thumb known as Moore s Law implies that you can expect to have a computer with about double the speed and double the memory 18 months from now, or a computer with about 10 times the speed and 10 times the memory in about 5 years. The table below demonstrates that you cannot keep pace with Moore s Law if you are using a quadratic or a cubic algorithm, and you can quickly determine whether that is the case by doing a doubling ratio test and checking that the ratio of running times as the input size doubles approaches 2, not 4 or 8. order of growth of time for a program that takes a few hours for input of size N 2x 10x description function factor factor predicted time for10n predicted time for 10N on a 10x faster computer linear N 2 10 a day a few hours linearithmic N log N 2 10 a day a few hours quadratic N a few weeks a day cubic N 3 8 1,000 several months a few weeks exponential 2 N 2 N 2 9N never never Predictions on the basis of order-of-growth function

24 1.4 Analysis of Algorithms 195 Caveats There are many reasons that you might get inconsistent or misleading results when trying to analyze program performance in detail. All of them have to do with the idea that one or more of the basic assumptions underlying our hypotheses might be not quite correct. We can develop new hypotheses based on new assumptions, but the more details that we need to take into account, the more care is required in the analysis. Large constants. With leading-term approximations, we ignore constant coefficients in lower-order terms, which may not be justifed. For example, when we approximate the function 2 N 2 + c N by ~2 N 2, we are assuming that c is small. If that is not the case (suppose that c is 10 3 or 10 6 ) the approximation is misleading. Thus, we have to be sensitive to the possibility of large constants. Nondominant inner loop. The assumption that the inner loop dominates may not always be correct. The cost model might miss the true inner loop, or the problem size N might not be sufficiently large to make the leading term in the mathematical description of the frequency of execution of instructions in the inner loop so much larger than lower-order terms that we can ignore them. Some programs have a significant amount of code outside the inner loop that needs to be taken into consideration. In other words, the cost model may need to be refined. Instruction time. The assumption that each instruction always takes the same amount of time is not always correct. For example, most modern computer systems use a technique known as caching to organize memory, in which case accessing elements in huge arrays can take much longer if they are not close together in the array. You might observe the effect of caching for ThreeSum by letting DoublingTest run for a while. After seeming to converge to 8, the ratio of running times may jump to a larger value for large arrays because of caching. System considerations. Typically, there are many, many things going on in your computer. Java is one application of many competing for resources, and Java itself has many options and controls that significantly affect performance. A garbage collector or a justin-time compiler or a download from the internet might drastically affect the results of experiments. Such considerations can interfere with the bedrock principle of the scientific method that experiments should be reproducible, since what is happening at this moment in your computer will never be reproduced again. Whatever else is going on in your system should in principle be negligible or possible to control. Too close to call. Often, when we compare two different programs for the same task, one might be faster in some situations, and slower in others. One or more of the considerations just mentioned could make the difference. There is a natural tendency among

25 196 CHAPTER 1 Fundamentals some programmers (and some students) to devote an extreme amount of energy running races to find the best implementation, but such work is best left for experts. Strong dependence on inputs. One of the first assumptions that we made in order to determine the order of growth of the program s running time of a program was that the running time should be relatively insensitive to the inputs. When that is not the case, we may get inconsistent results or be unable to validate our hypotheses. For example, suppose that we modify ThreeSum to answer the question Does the input have a triple that sums to 0? by changing it to return a boolean value, replacing cnt++ by return true and adding return false as the last statement. The order of growth of the running time of this program is constant if the first three integers sum to 0 and cubic if there are no such triples in the input. Multiple problem parameters. We have been focusing on measuring performance as a function of a single parameter, generally the value of a command-line argument or the size of the input. However, it is not unusual to have several parameters. A typical example arises when an algorithm involves building a data structure and then performing a sequence of operations that use that data structure. Both the size of the data structure and the number of operations are parameters for such applications. We have already seen an example of this in our analysis of the problem of whitelisting using binary search, where we have N numbers in the whitelist and M numbers on standard input and a typical running time proportional to M log N. Despite all these caveats, understanding the order of growth of the running time of each program is valuable knowledge for any programmer, and the methods that we have described are powerful and broadly applicable. Knuth s insight was that we can carry these methods through to the last detail in principle to make detailed, accurate predictions. Typical computer systems are extremely complex and close analysis is best left for experts, but the same methods are effective for developing approximate estimates of the running time of any program. A rocket scientist needs to have some idea of whether a test flight will land in the ocean or in a city; a medical researcher needs to know whether a drug trial will kill or cure all the subjects; and any scientist or engineer using a computer program needs to have some idea of whether it will run for a second or for a year.