Computer Science Approach to problem solving If my boss / supervisor / teacher formulates a problem to be solved urgently, can I write a program to efficiently solve this problem???
Polynomial-Time Brute force. For many non-trivial problems, there is a natural brute force search algorithm that checks every possible solution. Typically takes 2 N time or worse for inputs of size N. Unacceptable in practice. N! for stable matching with N men and N women Desirable scaling property. When the input size doubles, the algorithm should only slow down by some constant factor C. There exists constants a > 0 and d > 0 such that on every input of size N, its running time is bounded by a N d steps. Def. An algorithm is poly-time if the above scaling property holds. choose C = 2 d Property: poly-time is invariant over all (non-quantum) computer models. 29
Average/Worst-Case Analysis Worst case running time. Obtain bound on largest possible running time of algorithm on any input of a given size N. Generally captures efficiency in practice. Draconian view, but hard to find effective alternative. For probabilistic algorithms, we take the worst average running time. Average case running time. Obtain bound on running time of algorithm on random input as a function of input size N. Hard (or impossible) to accurately model real instances by random distributions. Algorithm tuned for a certain distribution may perform poorly on other input distributions. 30
Worst-Case Polynomial-Time Def. An algorithm is efficient if its running time is polynomial. Justification: It really works in practice! Although 6.02 10 23 N 20 is technically poly-time, it would be useless in practice. In practice, the poly-time algorithms that people develop almost always have low constants and low exponents. Breaking through the exponential barrier of brute force typically exposes some crucial structure of the problem. Exceptions. Some poly-time algorithms do have high constants and/or exponents, and are useless in practice. Some exponential-time (or worse) algorithms are widely used Primality testing because the worst-case instances seem to be rare. simplex method Unix grep 31
Why It Matters 1000 Big-O Complexity 900 800 Operations 700 600 500 400 300 O(1) O(log n) O(n) O(n log n) O(n 2 ) O(2 n ) 200 O(n!) 100 0 0 10 20 30 40 50 60 70 80 90 100 Elements 32
Why It Matters Note: age of Universe ~ 10 10 years 33
Chapter 2 Basics of Algorithm Analysis Slides by Kevin Wayne. Copyright 2005 Pearson-Addison Wesley. All rights reserved. 34
2.2 Asymptotic Order of Growth
Asymptotic Order of Growth Let f:n R + be a function, we define Upper bounds. O(f) = { g:n R + c R +, n 0 N s.t. n n 0 [ g(n) c f(n) ]. Lower bounds. Ω(f) = { g:n R + c R +, n 0 N s.t. n n 0 [ g(n) c f(n) ]. Tight bounds. Θ(f) = O(f) Ω(f). Ex: T(n) = 32n 2 + 17n + 32. T(n) O(n 2 ), O(n 3 ), Ω(n 2 ), Ω(n), and Θ(n 2 ). T(n) O(n), Ω(n 3 ), Θ(n), or Θ(n 3 ). 36
Notation Abuse of notation. T(n) = O(f(n)). Not transitive: f(n) = 5n 3 ; g(n) = 3n 2 f(n) = O(n 3 ) and g(n) = O(n 3 ) but f(n) g(n). Better notation: T(n) O(f(n)). Acceptable notation: T(n) is O(f(n)). (if scared by!) Meaningless statement. Any comparison-based sorting algorithm requires at least O(n log n) comparisons. Statement doesn't "type-check". Precisely, f(n)=1 O(n log n), therefore "at least one comparison". Use Ω for lower bounds: "at least Ω(n log n) comparisons". "requires at least cn log n comparisons for c>0 and all large enough n". 37
Big-O Notation Limit theorems. Let f,g:n R + be functions, such that l im f(n)/g(n) = n c R+, then f Θ(g), g Θ(f), Θ(f) = Θ(g) l n i m f(n)/g(n) = 0, then f O(g), f Ω(g), O(f) O(g), Ω(g) Ω(f) 38
Properties Let f,g:n R + be functions Transitivity. If f O(g) and g O(h) then f O(h) since O(f) O(g) O(h). If f Ω(g) and g Ω(h) then f Ω(h) since Ω(f) Ω(g) Ω(h). If f Θ(g) and g Θ(h) then f Θ(h) since Θ(f) Θ(g) Θ(h). Additivity. If f O(h) and g O(h) then f + g O(h) since f(n) < c f h(n), g(n) < c g h(n) f(n) + g(n) < (c f +c g ) h(n). If f Ω(h) and g Ω(h) then f + g Ω(h). If f Θ(h) and g O(h) then f + g Θ(h). Consequence: f + g O( max{f,g ) since f + g 2max{f,g. f + g Ω( max{f,g ) since f + g max{f,g. f + g Θ( max{f,g ) since max{f,g f + g 2 max{f,g. 39
Properties Consequence: f + g O( max{f,g ). f + g Ω( max{f,g ). f + g Θ( max{f,g ). max a 1 for i = 2 to n { if (a i > max) max a i min (x 1 - x 2 ) 2 + (y 1 - y 2 ) 2 for i = 1 to n { for j = i+1 to n { d (x i - x j ) 2 + (y i - y j ) 2 if (d < min) min d foreach set S i { foreach other set S j { foreach element p of S i { Θ(n) Θ(n 2 ) Θ(n 3 ) determine whether p also belongs to S j if (no element of S i belongs to S j ) report that S i and S j are disjoint Θ(n 3 ) 40
Asymptotic Bounds for Some Common Functions Polynomials. a 0 + a 1 n + + a d n d Θ(n d ) if a d > 0. Polynomial time. Running time O(n d ) for some constant d independent of the input size n. Logarithms. O(log a n) = O(log b n) for any constants a, b > 0. can avoid specifying the base Logarithms. For every x > 0, log n O(n x ). log grows slower than every polynomial Exponentials. For every r > 1 and every d > 0, n d O(r n ). every exponential grows faster than every polynomial 41
2.4 A Survey of Common Running Times
Linear Time: O(n) Linear time. Running time is proportional to input size. Computing the maximum. Compute minimum of n numbers a 1,, a n. min a 1 for i = 2 to n { if (a i < min) min a i 43
O(n log n) Time O(n log n) time. Arises in divide-and-conquer algorithms. also referred to as linearithmic time Sorting. Mergesort and Heapsort are sorting algorithms that perform O(n log n) comparisons. Closest Points on a line. Given n numbers x 1,, x n, what is the smallest distance x i -x j between any two points? O(n log n) solution. Sort the n numbers. Scan the sorted list in order, identifying the minimum gap between two successive points. 44
O(n log n) Time O(n log n) time. Arises in divide-and-conquer algorithms Median Finding. Given n distinct numbers a 1,, a n, find i such that { j : a j <a i = n-1 / 2 and { j : a j >a i = n-1 / 2. 22 31 44 7 12 19 20 35 3 40 27 3 7 12 19 20 22 27 31 35 40 44 9 4 5 6 7 1 11 2 8 10 3 Straight forward approach. Θ(n log n) arithmetic operations (to sort first). Fundamental question. Can we improve upon this approach? Remark. This algorithm is Ω(n log n) and it seems inevitable in general, but this is just an illusion: Θ(n) is actually possible and optimal 45 45
Quadratic Time: O(n 2 ) Quadratic time. Enumerate all pairs of elements. Closest pair of points. Given a list of n points in the plane (x 1, y 1 ),, (x n, y n ), find the pair that is closest. O(n 2 ) solution. Try all pairs of points. min (x 1 - x 2 ) 2 + (y 1 - y 2 ) 2 for i = 1 to n { for j = i+1 to n { d (x i - x j ) 2 + (y i - y j ) 2 if (d < min) min d don't need to take square roots Remark. This algorithm is Ω(n 2 ) and it seems inevitable in general, but this is just an illusion: Θ(n log n) is actually possible and optimal see chapter 5 46
Quadratic Time: O(n 2 ) First Repetition. Given N numbers a 1,, a N, find the smallest n s. t. there exists 1 i<n such that a i =a n. 22 31 44 7 12 19 22 35 3 40 27 Most natural solutions will be Θ(n 2 ). An O(n log n) algorithm (where n is the location of the first repetition). can be obtained from Red-Black Trees or similar to MergeSort. 47
Quadratic Time: O(n 2 ) Quadratic time. Solve O(n 2 ) independent sub-puzzles each in constant-time. nxnxn Rubik s cube. Given a scrambled nxnxn cube, put it in solved configuration. Remark. This algorithm is Ω(n 2 ) and it seems inevitable in general, but this is just an illusion: Θ(n 2 /log n) is actually possible and optimal 48
Cubic Time: O(n 3 ) Cubic time. Enumerate all triples of elements. Matrix multiplication. Given two nxn matrices of numbers A,B, what is their matrix product C? O(n 3 ) solution. For each entry c ij compute as below. Remark. This algorithm is Ω(n 3 ) and it seems inevitable in general, but this is just an illusion: O(n 2.3728639 ) is actually possible 49
Polynomial Time: O(n k ) Time Independent set of size k. Given a graph, are there k nodes such that no two are joined by an edge? O(n k ) solution. Enumerate all subsets of k nodes. k is a constant foreach subset S of k nodes { if (S is an independent set) report S Check whether S is an independent set = O(k 2 ). " n Number of k element subsets = $ % ' = O(k 2 n k / k!) = O(n k # k& ). n (n 1) (n 2)! (n k +1) k (k 1) (k 2)! (2) (1) nk k! poly-time for k=17, but not practical 50
Exponential Time Independent set. Given a graph, what is the maximum size of an independent set? O(n 2 2 n ) solution. Enumerate all subsets. S* foreach subset S of nodes { if (S is an independent set and S > S* ) update S* S 51
Chapter 2 Basics of Algorithm Analysis Slides by Kevin Wayne. Copyright 2005 Pearson-Addison Wesley. All rights reserved. 52