Algorithm Analysis Page 1 - Algorithm Analysis Dr. Fall 2008
Algorithm Analysis Page 2 Outline Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Big-Oh in Algorithm Analysis Relatives of Big-Oh
Algorithm Analysis Textbook Overview Page 3 Outline Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Big-Oh in Algorithm Analysis Relatives of Big-Oh
Algorithm Analysis Textbook Overview Page 4
Algorithm Analysis Textbook Overview Page 5 Algorithm Design I Part I. Fundamental Tools 1. Algorithm Analysis 1.1 Methodologies for Analyzing Algorithms 1.2 Asymptotic Notation 1.3 A Quick Mathematical Review 1.4 Case Studies in Algorithm Analysis 1.5 Amortization 1.6 Experimentation 1.7 Exercises 2. Basic Data Structures 2.1 Stacks and Queues 2.2 Vectors, Lists, and Sequences 2.3 Trees 2.4 Priority Queues and Heaps 2.5 Dictionaries and Hash Tables
Algorithm Analysis Textbook Overview Page 6 Algorithm Design II 2.6 Java Example: Heap 2.7 Exercises 3. Search Trees and Skip Lists 3.1 Ordered Dictionaries and Binary Search Trees 3.2 AVL Trees 3.3 Bounded-Depth Search Trees 3.4 Splay Trees 3.5 Skip Lists 3.6 Java Example: AVL and Red-Black Trees 3.7 Exercises 4. Sorting, Sets, and Selection 4.1 Merge-Sort 4.2 The Set Abstract Data Type 4.3 Quick-Sort 4.4 A Lower Bound on Comparison-Based Sorting 4.5 Bucket-Sort and Radix-Sort
Algorithm Analysis Textbook Overview Page 7 Algorithm Design III 4.6 Comparison of Sorting Algorithms 4.7 Selection 4.8 Java Example: In-Place Quick-Sort 4.9 Exercises 5. Fundamental Techniques 5.1 The Greedy Method 5.2 Divide-and-Conquer 5.3 Dynamic Programming 5.4 Exercises
Algorithm Analysis Textbook Overview Page 8 Algorithm Design IV Part II. Graph Algorithms 1. Graphs 1.1 The Graph Abstract Data Type 1.2 Data Structures for Graphs 1.3 Graph Traversal 1.4 Directed Graphs 1.5 Java Example: Depth-First Search 1.6 Exercises 2. Weighted Graphs 2.1 Single-Source Shortest Paths 2.2 All-Pairs Shortest Paths 2.3 Minimum Spanning Trees 2.4 Java Example: Dijkstra s Algorithm 2.5 Exercises 3. Network Flow and Matching
Algorithm Analysis Textbook Overview Page 9 Algorithm Design V 3.1 Flows and Cuts 3.2 Maximum Flow 3.3 Maximum Bipartite Matching 3.4 Minimum-Cost Flow 3.5 Java Example: Minimum-Cost Flow 3.6 Exercises Part III. Internet Algorithmics 1. Text Processing 1.1 Strings and Pattern Matching Algorithms 1.2 Tries 1.3 Text Compression 1.4 Text Similarity Testing 1.5 Exercises 2. Number Theory and Cryptography 2.1 Fundamental Algorithms Involving Numbers
Algorithm Analysis Textbook Overview Page 10 Algorithm Design VI 2.2 Cryptographic Computations 2.3 Information Security Algorithms and Protocols 2.4 The Fast Fourier Transform 2.5 Java Example: FFT 2.6 Exercises 3. Network Algorithms 3.1 Complexity Measures and Models 3.2 Fundamental Distributed Algorithms 3.3 Broadcast and Unicast Routing 3.4 Multicast Routing 3.5 Exercises
Algorithm Analysis Textbook Overview Page 11 Algorithm Design VII Part IV. Additional Topics 1. Computational Geometry 1.1 Range Trees 1.2 Priority Search Trees 1.3 Quadtrees and k-d Trees 1.4 The Plane Sweep Technique 1.5 Convex Hulls 1.6 Java Example: Convex Hull 1.7 Exercises 2. NP-Completeness 2.1 P and NP 2.2 NP-Completeness 2.3 Important NP-Complete Problems 2.4 Approximation Algorithms 2.5 Backtracking and Branch-and-Bound 2.6 Exercises
Algorithm Analysis Textbook Overview Page 12 Algorithm Design VIII 3. Algorithmic Frameworks 3.1 External-Memory Algorithms 3.2 Parallel Algorithms 3.3 Online Algorithms 3.4 Exercises
Algorithm Analysis Analysis of Algorithm Page 13 Outline Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Big-Oh in Algorithm Analysis Relatives of Big-Oh
Algorithm Analysis Analysis of Algorithm Page 14 Running Time of Algorithms An algorithm is a step-by-step procedure for solving a problem in a finite amount of time Most algorithms transform input objects into output objects The running time of an algorithm typically grows with the input size The average case running time is often difficult to determine We focus on the worst case running time Easier to analyse Crucial to applications such as games, finance, robotics Sometimes, it is also worth to study the best case running time
Algorithm Analysis Analysis of Algorithm Page 15 Running Time: Example 100 80 Running Time 60 40 20 Best case Average case Worst case 0 1000 2000 3000 4000 5000 Input Size
Algorithm Analysis Analysis of Algorithm Page 16 Problem Classes I There are feasible problems, which can be solved by an algorithm efficiently Examples: sorting a list, draw a line between two points, decide whether n is prime, etc. There are computable problems, which can be solved by an algorithm, but not efficiently Example: finding the shortest solution for large sliding puzzles There are undecidable problems, which can not be solved by an algorithm Example: for a given a description of a program and an input, decide whether the program finishes running or will run forever (halting problem)
Algorithm Analysis Analysis of Algorithm Page 17 Problem Classes II All Problems Computable Problems Feasible Problems
Algorithm Analysis Analysis of Algorithm Page 18 Experimental Studies There are two ways to analyse the running time of an algorithm: Experimental studies Theoretical analysis Experiments usually involve the following major steps: Write a program implementing the algorithm Run the program with inputs of varying size and composition Use a method like System.currentTimeMillis() to get an accurate measure of the actual running time Plot the results
Algorithm Analysis Analysis of Algorithm Page 19 Experimental Studies: Example 10000 9000 8000 Running Time (ms) 7000 6000 5000 4000 3000 2000 1000 0 1000 2000 3000 4000 5000 Input Size
Algorithm Analysis Analysis of Algorithm Page 20 Experimental Setup I To setup an experiment, perform the following steps with care Choosing the question Estimate the asymptotic running time in the average case Test which of two algorithm is faster If an algorithm depends on parameters, find the best values For approximation algorithms, evaluate the quality of the approximation Deciding what to measure Running time (system dependent) Number of memory references Number of primitive (e.g. arithmetic) operations/comparisons
Algorithm Analysis Analysis of Algorithm Page 21 Experimental Setup II Generating test data Enough samples for statistically significant results Samples of varying sizes Representative of the kind of data expected in practice Coding the solution and performing the experiment Reproducible results Keep note of the details of the computational environment Evaluating the test results statistically
Algorithm Analysis Analysis of Algorithm Page 22 The Power Test The power test is a methodology to approximate the growth rate of a polynomial (feasible) algorithm Actual running time: t(n) b n c, b =???, c =??? Samples: (x i, y i ) x i = problem size y i = t(x i ) = actual running time Transform x i into x i = log x i and y i into y i = log y i Plot (x i, y i ) Log-Log-Scale If possible, draw a line through samples polynomial b = the line s y-axis intercept, c = slope of the line If the growth of the points increases super-polynomial If the plotted points converge to a constant sublinear
Algorithm Analysis Analysis of Algorithm Page 23 The Power Test 10 1Mia. = 9 8 7 1Mio. = 6 5 4 1000 = 3 2 1 Super-Polynomial Polynomial b=2, c=5/6 0 0 1 2 3 4 5 6 7 8 9 10 =1000 =1Mio. =1Mia. Sublinear t(n) = 2n 5/6
Algorithm Analysis Analysis of Algorithm Page 24 Limits of Experiments It is necessary to implement the algorithm, which may be difficult Results may not be indicative of the running time on other inputs not included in the experiment The running times may depend strongly on the chosen setting (hardware, software, programming language) In order to compare two algorithms, the same hardware and software environments must be used
Algorithm Analysis Analysis of Algorithm Page 25 Theoretical Analysis Uses a high-level description of the algorithm instead of an implementation Describes the running time as a function of the input size n: T (n) = number of primitive operations Takes into account all possible inputs Allows to evaluate the speed of an algorithm independent of hardware software programming language actual implementation
Algorithm Analysis Pseudo-Code and Primitive Operations Page 26 Outline Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Big-Oh in Algorithm Analysis Relatives of Big-Oh
Algorithm Analysis Pseudo-Code and Primitive Operations Page 27 Pseudo-Code Pseudo-code allows a high-level description of an algorithm More structured than English prose, but less detailed than a program Preferred notation for describing algorithms Hides program design issues and details of a particular language
Algorithm Analysis Pseudo-Code and Primitive Operations Page 28 Pseudo-Code: Example Find the maximum element of an array: Algorithm arraymax(a, n) Input array A of n integers Output maximum element of A currentmax A[0] for i 1 to n 1 do if A[i] > currentmax then currentmax A[i] return currentmax
Algorithm Analysis Pseudo-Code and Primitive Operations Page 29 Details of Pseudo-Code I The rules of using pseudo-code are very flexible, but following some guidelines is recommended Method/function declaration Algorithm method or function(arg[, arg...]) Input... Output... Control flow if... then... [else...] while... do... repeat... until... for... do... Indentation replaces braces {} resp. begin... end
Algorithm Analysis Pseudo-Code and Primitive Operations Page 30 Details of Pseudo-Code II Expressions Assignments: (like = in Java) Equality testing: = (like == in Java) Superscript like n 2 and other mathematical notation is allowed Array access: A[i 1] Method/function call object.method(arg[, arg...]) function(arg[, arg...]) Return value return expression
Algorithm Analysis Pseudo-Code and Primitive Operations Page 31 The Random Access Machine Model To study algorithms in pseudo-code analytically, we need a theoretical model of a computing machine A random access machine consists of a CPU a bank of memory cells, each of which can hold an arbitrary number (of any size) or character The size of the memory is unlimited Memory cells are numbered and accessing any cell in memory takes 1 unit time (= random access) All primitive operations of the CPU take 1 unit time
Algorithm Analysis Pseudo-Code and Primitive Operations Page 32 Primitive Operations The primitive operations of a random access machine are the once we allow in a pseudo-code algorithm Evaluating an expression (arithmetic operations, comparisons) Assigning a value to a variable Indexing into an array Calling a method/function Returning from a method/function Largely independent from the programming language By inspecting the pseudo-code, we can determine the maximum (or minimum) number of primitive operations executed by an algorithm, as a function of the input size
Algorithm Analysis Pseudo-Code and Primitive Operations Page 33 Primitive Operations: Example Algorithm arraymax(a, n) currentmax A[0] for i 1 to n 1 do if A[i] > currentmax then currentmax A[i] {increment counter i} return currentmax # Operations 2 2 + n 2(n 1) 2(n 1) 2(n 1) 1 Hence, the total number of primitive operations of arraymax is T (n) = 7n 1.
Algorithm Analysis Growth Rate and Big-Oh Notation Page 34 Outline Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Big-Oh in Algorithm Analysis Relatives of Big-Oh
Algorithm Analysis Growth Rate and Big-Oh Notation Page 35 Estimating Actual Running Times In reality, the time to execute a primitive operation differs Let a be the time taken by the fastest primitive operation Let b be the time taken by the slowest primitive operation If t(n) denotes the actual running time of the algorithm on a concrete machine for a problem of size n, we get a T (n) t(n) b T (n) For arraymax, this means that the actual running time a (7n 1) t(n) b (7n 1) is bounded by two linear functions
Algorithm Analysis Growth Rate and Big-Oh Notation Page 36 Growth Rates of Running Times Changing the hardware/ software environment Affects the constants a and b Therefore, affects t(n) only by a constant factor Does not alter the growth rate of t(n) Therefore, we can consider the growth rate of T (n) as being the characteristic measure for evaluating an algorithm s running time We will use the terms number of primitive operations and running time interchangeably and denote them by T (n) The growth rate of T (n) is an intrinsic property of the algorithm which is independent of its implementation
Algorithm Analysis Growth Rate and Big-Oh Notation Page 37 Polynomial Growth Rates Most interesting algorithms have a polynomial growth rate Linear: T (n) n Quadratic: T (n) n 2 Cubic: T (n) n 3 In a log-log chart, a polynomial growth rate turns out to be a line (see power test) The slope of the line corresponds to the growth rate The growth rate is not affected by constant factors or lower-order terms Examples 10 2 n + 10000 is linear 10 5 n 2 + 10 8 n is quadratic
Algorithm Analysis Growth Rate and Big-Oh Notation Page 38 Polynomial Growth Rate 10 12 cubic quadratic Running Time T(n) 10 10 10 8 10 6 10 4 linear 10 2 1 1 10 1 10 2 10 3 10 4 10 5 10 6 10 7 Input Size n
Algorithm Analysis Growth Rate and Big-Oh Notation Page 39 Constant Factors 10 12 Running Time T(n) 10 10 10 8 10 6 10 4 10 2 n +10 5 n 10 2 1 1 10 1 10 2 10 3 10 4 10 5 10 6 10 7 Input Size n
Algorithm Analysis Growth Rate and Big-Oh Notation Page 40 Big-Oh Notation The so-called big-oh notation is useful to better compare and classify the growth rates of different functions T (n) Given functions f (n) and g(n), we say that f (n) is O(g(n)), if there is a constant c > 0 and an integer n 0 1 such that Example 1: 2n + 10 is O(n) f (n) c g(n), for n n 0 2n + 10 c n n 10/(c 2), i.e. pick c = 3 and n 0 = 10 Example 2: n 2 is not O(n) n 2 c n n c, which can not be satisfied since c is a constant
Algorithm Analysis Growth Rate and Big-Oh Notation Page 41 Big-Oh Example 1 1000 Running Time T(n) 100 10 2n +10 3n n 1 1 10 100 1000 Input Size n
Algorithm Analysis Growth Rate and Big-Oh Notation Page 42 Big-Oh Example 2 10000 10n n 1000 Running Time T(n) 100 100n n 2 10 1 1 10 100 1000 Input Size n
Algorithm Analysis Growth Rate and Big-Oh Notation Page 43 Big-Oh and Growth Rate The big-oh notation gives an upper bound on the growth rate of a function The statement f (n) is O(g(n)) means that the growth rate of f (n) is no more than the growth rate of g(n) We can use the big-oh notation to order functions according to their growth rate f (n) is O(g(n)) g(n) is O(f (n)) f (n) g(n) Yes No f (n) g(n) No Yes f (n) g(n) Yes Yes In other words, O(f (n)) can be seen as an equivalence class of functions with equal growth rate
Algorithm Analysis Growth Rate and Big-Oh Notation Page 44 Big-Oh Rules If f (n) is a polynomial of degree d, i.e. then f (n) is O(n d ) f (n) = a 0 + a 1 n + a 2 n 2 +... a d n d drop lower-order terms drop constant factors e.g. 3 + 2n 2 + 5n 4 is O(n 4 ) Use the smallest possible equivalence class say 2n is O(n) instead of 2n is O(n 2 ) say 3n + 5 is O(n) instead of 3n + 5 is O(3n)
Algorithm Analysis Big-Oh in Algorithm Analysis Page 45 Outline Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Big-Oh in Algorithm Analysis Relatives of Big-Oh
Algorithm Analysis Big-Oh in Algorithm Analysis Page 46 Asymptotic Algorithm Analysis The asymptotic analysis of an algorithm determines the running time in big-oh notation To perform the asymptotic analysis Find the worst-case number of primitive operations T (n) Express this function with big-oh notation Example: arraymax We determine that algorithm arraymax executes at most T (n) = 7n 1 primitive operations We say that algorithm arraymax runs in O(n) time Since constant factors and lower-order terms are eventually dropped anyhow, we can disregard them when counting primitive operations
Algorithm Analysis Big-Oh in Algorithm Analysis Page 47 Typical Functions in Algorithms Constant: O(1) Logarithmic: O(log n) Linear: O(n) Quadratic: O(n 2 ) Polynomial: O(n d ), d > 0 Exponential: O(b n ), b > 1 Others: O( n), O(n log n) Growth rate order: 1 < log n < n < n < n log n < n 2 < n 3 < < 2 n < 3 n <
Algorithm Analysis Big-Oh in Algorithm Analysis Page 48 Growth Rate Comparison n log n n n n log n n 2 n 3 2 n 2 1 1.4 2 2 4 8 4 4 2 2 4 8 16 64 16 8 3 2.8 8 24 64 512 256 16 4 4 16 64 256 4,096 65,536 32 5 5.7 32 160 1,024 32,768 4.29 10 9 64 6 8 64 384 4,096 262,144 1.84 10 19 128 7 11 128 896 16,384 2,097,152 3.40 10 38 256 8 16 256 2,048 65,536 16,777,216 1.15 10 77 512 9 23 512 4,608 262,144 1.34 10 8 1.34 10 154 1,024 10 32 1,024 10,240 1,048,576 1.07 10 9 1.79 10 308 Atoms in the universe: ca. 10 80
Algorithm Analysis Big-Oh in Algorithm Analysis Page 49 The Importance of Asymptotics 1 operation/µs 256 ops./µs Running Time Maximum Problem Size n New Size 400n 2, 500 150, 000 9, 000, 000 256n 20n log n 4, 096 166, 666 7, 826, 087 256n log n 7+log n 2n 2 707 5, 477 42, 426 16n n 4 31 88 244 4n 2 n 19 25 31 n + 8 1 second 1 minute 1 hour
Algorithm Analysis Big-Oh in Algorithm Analysis Page 50 Math You Need to Review Logarithms/exponentials: a = b c log b a = c a = b log b a Properties of logarithms log b ac = log b a + log b c a log b c = log b a log b c log b a c = c log b a Properties of exponentials b a+c = b a b c b a c = (b a )/(b c ) b ac = (b a ) c = (b c ) a Base change: log b a = log c a log c b and bc = a c log a b Computer science: base b = 2, i.e. log 2 a = log a
Algorithm Analysis Relatives of Big-Oh Page 51 Outline Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Big-Oh in Algorithm Analysis Relatives of Big-Oh
Algorithm Analysis Relatives of Big-Oh Page 52 Relatives of Big-Oh I Big-Omega: we say that f (n) is Ω(g(n)), if there is a constant c > 0 and an integer n 0 1 such that f (n) c g(n), for n n 0 Little-Oh: we say that f (n) is o(g(n)), if there is a constant c > 0 and an integer n 0 1 such that f (n) < c g(n), for n n 0 Little-Omega: we say that f (n) is ω(g(n)), if there is a constant c > 0 and an integer n 0 1 such that f (n) > c g(n), for n n 0
Algorithm Analysis Relatives of Big-Oh Page 53 Relatives of Big-Oh II Big-Theta: we say that f (n) is Θ(g(n)), if there are constants c > 0 and c > 0 and an integer n 0 1 such that c g(n) f (n) c g(n), for n n 0 The intuition behind the big-oh relatives are Big-Oh: f (n) is asymptotically less or equal to g(n) Little-Oh: f (n) is asymptotically strictly less than g(n) Big-Omega: f (n) is asymptotically greater or equal to g(n) Little-Omega: f (n) is asymptotically strictly greater than g(n) Big-Theta: f (n) is asymptotically equal to g(n) The big-theta notation is the most precise description of the growth rate, but the big-oh notation is more commonly used