Big-O Aalysis 1 Defiitio: Suppose that f() ad g() are oegative fuctios of. The we say that f() is O(g()) provided that there are costats C > 0 ad N > 0 such that for all > N, f() Cg(). Big-O expresses a upper boud o the growth rate of a fuctio, for sufficietly large values of. By the defiitio above, demostratig that a fuctio f is big-o of a fuctio g requires that we fid specific costats C ad N for which the iequality holds (ad show that the iequality does, i fact, hold).
Big-O Example Cosider the followig fuctio: 5 5 T ( ) We might guess that: T ( ) 5 for all We could easily verify the guess by iductio: If =, the T() = 17 which is less tha 0, so the guess is valid if =. Assume that for some, T() 5. The: 5 5 5 5 5 5 T 5 5 5 5 ( 1) ( 1) ( 1) 5 5 5 5 by the iductive assumptio 5 10 5 5( 1) Thus, by iductio, T() 5 for all. So, by defiitio, T() is O( ).
Makig the Guess 3 The obvious questio is "how do we come up with the guess i the first place"? Here's oe possible aalysis (which falls a bit short of beig a proof): 5 5 T 5 5 ( ) 5 (replace with ^, subtract ) The middle step seems soud sice if the, substitutig will thus add at least 5 to the expressio, so that subtractig should still result i a larger value.
Big-O Theorems 4 For all the followig theorems, assume that f() is a o-egative fuctio of ad that K is a arbitrary positive costat. Theorem 1: K is O(1) Theorem : A polyomial is O(the term cotaiig the highest power of ) f ( ) 7 4 3 5 1000 is O(7 4 ) Theorem 3: K*f() is O(f()) [i.e., costat coefficiets ca be dropped] g( ) 7 4 is O( 4 ) Theorem 4: If f() is O(g()) ad g() is O(h()) the f() is O(h()). [trasitivity] f ( ) 7 4 3 5 1000 is O( 4 )
Big-O Theorems 5 Theorem 5: Each of the followig fuctios is strictly big-o of its successors: K [costat] log b () [always log base if o base is show] log b () to higher powers 3 larger costats to the -th power! [ factorial] smaller larger f ( ) 3 log( ) is O( log( )) ad O( ) ad O( )
Big-O Theorems 6 Theorem 6: I geeral, f() is big-o of the domiat term of f(), where domiat may usually be determied from Theorem 5. f ( ) 7 3 log( ) 5 1000 is O( ) g( ) 7 3 1000000 is O(3 4 ) h( ) 7( log( )) is O( ) Theorem 7: For ay base b, log b () is O(log()).
Big-Omega 7 I additio to big-o, we may seek a lower boud o the growth of a fuctio: Defiitio: Suppose that f() ad g() are oegative fuctios of. The we say that f() is (g()) provided that there are costats C > 0 ad N > 0 such that for all > N, f() Cg(). Big- expresses a lower boud o the growth rate of a fuctio, for sufficietly large values of. Aalagous theorems ca be proved for big-ω.
Big-Theta 8 Fially, we may have two fuctios that grow at essetially the same rate: Defiitio: Alerative: Suppose that f() ad g() are oegative fuctios of. The we say that f() is (g()) provided that f() is O(g()) ad also that f() is (g()). Suppose that f() ad g() are oegative fuctios of. The we say that f() is (g()) provided that there exist costats C1 > 0, C > 0 ad N > 0 such that, for all > N, C 1 g() f() C g(). If f is (g) the, from some poit o, f is bouded below by oe multiple of g ad bouded above by aother multiple of g (ad vice versa). So, i a very basic sese f ad g grow at the same rate.
Order ad Limits 9 The task of determiig the order of a fuctio is simplified cosiderably by the followig result: Theorem 8: f() is (g()) if lim f ( ) g( ) c where 0 c Recall Theorem 7 we may easily prove it (ad a bit more) by applyig Theorem 8: lim log b ( ) log( ) lim 1 l( b) 1 l() lim l() l( b) l() l( b) The last term is fiite ad positive, so log b () is (log()) by Theorem 8. Corollary: if the limit above is 0 the f() is strictly O(g()), ad if the limit is the f() is strictly (g()).
Order ad Limits 10 The coverse of Theorem 8 is false. However, it is possible to prove: Theorem 9: If f() is (g()) the provided that the limit exists. lim f ( ) g( ) c where 0 c A similar extesio of the precedig corollary also follows.
More Theorems 11 May of the big-o theorems may be stregtheed to statemets about big-: Theorem 10: If K > 0 is a costat, the K is (1). Theorem 11: A polyomial is (the highest power of ). proof: Suppose a polyomial of degree k. The we have: lim a a a k 0 1 k lim k a 0 k a 1 k 1 a k 1 a k a Now a k > 0 sice we assume the fuctio is oegative. So by Theorem 8, the polyomial is ( k ). k QED
More Theorems 1 Theorems 3, 6 ad 7 ca be similarly exteded. Theorem 1: K*f() is (f()) [i.e., costat coefficiets ca be dropped] Theorem 13: I geeral, f() is big- of the domiat term of f(), where domiat may usually be determied from Theorem 5. Theorem 14: For ay base b, log b () is (log()).
Strict Comparisos 13 For coveiece, we will say that: - f is strictly O(g) if ad oly if f is O(g) but f is ot Θ(g) - f is strictly Ω(g) if ad oly if f is Ω(g) but f is ot Θ(g) For example, log is strictly O( ) by Theorem 8 ad its corollary, because: log log 1/ lim lim lim 0 1
Big- Is a Equivalece Relatio 14 Theorem 15: If f() is (f()). [reflexivity] Theorem 16: If f() is (g()) the g() is (f()). [symmetry] Theorem 17: If f() is (g()) ad g() is (h()) the f() is (h()). [trasitivity] By Theorems 15 17, is a equivalece relatio o the set of positive-valued fuctios. The equivalece classes represet fudametally differet growth rates. Algorithms whose complexity fuctios belog to the same class are essetially equivalet i performace (up to costat multiples, which are ot uimportat i practice).
Applicatios to Algorithm Aalysis 15 Ex 1: A algorithm with complexity fuctio 3 T ( ) 5 3 is ( ) by Theorem 10. Ex : A algorithm with complexity fuctio T ( ) 3 log 4log is O( log()) by Theorem 5. Furthermore, the algorithm is also ( log()) by Theorem 8 sice: lim T ( ) log lim 3 4 log 3 For most commo complexity fuctios, it's this easy to determie the big-o ad/or big- complexity usig the give theorems.
Complexity of Liear Storage 16 For a cotiguous list of N elemets, assumig each is equally likely to be the target of a search: - average search cost is (N) if list is radomly ordered - average search cost is (log N) is list is sorted - average radom isertio cost is (N) - isertio at tail is (1) For a liked list of N elemets, assumig each is equally likely to be the target of a search: - average search cost is (N), regardless of list orderig - average radom isertio cost is (1), excludig search time
Most Commo Complexity Classes 17 Theorem 5 lists a collectio of represetatives of distict big-θ equivalece classes: K [costat] log b () [always log base if o base is show] log b () to higher powers 3 larger costats to the -th power! [ factorial] Most commo algorithms fall ito oe of these classes. Kowig this list provides some kowledge of how to compare ad choose the right algorithm. The followig charts provide some visual idicatio of how sigificat the differeces are
Graphical Compariso 18 Commo Growth Curves 100 1000 800 600 400 log log ^ ^3 ^ 10^ 00 0 1 3 4 5 6 7 8 9 10 11 (iput size)
Lower-order Classes 19 For sigificatly large values of, oly these classes are truly practical, ad whether is practical is debated. 10 100 Low-order Curves log 80 60 40 0 0 log 1 3 5 7 9 11 13 15 17 19 1 3 (iput size)
Proof of Theorem 8 0 Theorem 8: f() is (g()) if lim f ( ) g( ) c where 0 c Suppose that f ad g are o-egative fuctios of, ad that lim f ( ) g( ) c where 0 c The, from the defiitio of the limit, for every ε > 0 there exists a N > 0 such that wheever > N: f ( ) c from which we have ( c ) g( ) f ( ) ( c ) g( ) g ( ) Let ε = c/, the we have that: Therefore, by defiitio, f is Θ(g). 3c f ( ) g( ) whece f is ( g) c g ( ) f ( ) whece f is ( g )