Aalysis of Algorithms Ruig Time of a algorithm Ruig Time Upper Bouds Lower Bouds Examples Mathematical facts Iput Algorithm Output A algorithm is a step-by-step procedure for solvig a problem i a fiite amout of time. The ruig time of a algorithm typically grows with the iput size. Average case time is ofte difficult to determie. We focus o the worst case ruig time. Easier to aalyze Crucial to applicatios such as games, fiace ad robotics Ruig Time 120 100 80 60 40 20 0 best case average case worst case 1000 2000 3000 4000 Iput Size Aalysis of Algorithms 1 Aalysis of Algorithms 2 Measurig the Ruig Time Beyod Experimetal Studies How should we measure the ruig time of a algorithm? Approach 1: Experimetal Study t (ms) 60 Experimetal studies have several limitatios: eed to implemet limited set of iputs hardware ad software eviromets. 50 40 30 20 10 Aalysis of Algorithms 3 0 50 100 Aalysis of Algorithms 4
Theoretical Aalysis We eed a geeral methodology that: Uses a high-level descriptio of the algorithm (idepedet of implemetatio). Characterizes ruig time as a fuctio of the iput size. Takes ito accout all possible iputs. Is idepedet of the hardware ad software eviromet. Aalysis of Algorithms 5 Aalysis of Algorithms Primitive Operatios: Low-level computatios idepedet from the programmig laguage ca be idetified i pseudocode. Examples: callig a method ad returig from a method arithmetic operatios (e.g. additio) comparig two umbers, etc. By ispectig the pseudo-code, we ca cout the umber of primitive operatios executed by a algorithm. 6 Example: Algorithm arraymax(a, ): Iput: A array A storig itegers. Output: The maximum elemet i A. curretmax A[0] for i 1 to -1 do if curretmax < A[i] the curretmax A[i] retur curretmax curretmax A[0] for i 1 to -1 do if curretmax < A[i] the curretmax A[i] retur curretmax 1 assigmet -1 check -1 assigmets (if we are ot lucky) 1 retur value Aalysis of Algorithms 7 Aalysis of Algorithms 8
Lookig for the rak of a elemet i A of size sizea i 0 while (A[i] elemet) i i+1 retur i 1 assigmet sizea checks & assigmet (if we are ot lucky) Worst Case Big-Oh (upper boud) give fuctios f() ad g(), we say that f() is O(g()) if ad oly if there are positive costats c ad 0 such that f() c g() for 0 c g() f() Aalysis of Algorithms 9 0 Aalysis of Algorithms 10 prove that f() c g() for all 0 A Example f() = 60 2 + 5 + 1 g() = 2 60 2 + 5 2 + 2 prove that f() c 2 for 1 = 66 2 c = 66 0 = 1 f() c 2 0 O the other had 2 is ot O() because there is o c ad 0 such that: 2 c for 0 ( o matter how large a c is chose there is a big eough that 2 > c ). 2 f() = O( 2 ) Aalysis of Algorithms 11 Aalysis of Algorithms 12 0
O(1) < O(log ) < O() < O( log ) < O( 2 ) < O( 3 ) < O(2 ) 0 2 +1 log() 5 = 2 16 256 1024 log log 0 2 3 3.32 log 1 4 8 10 2 16 256 1024 log 2 64 448 10 200 2 4 256 65 500 1.05 * 10 6 3 8 4 100 16 800 800 1.07 * 10 9 2 4 35 500 11.7 * 10 6 1.80 * 10 308 0 Aalysis of Algorithms 13 Aalysis of Algorithms 14 Asymptotic Notatio (cot.) Note: Eve though it is correct to say 7-3 is O( 3 ), a better statemet is 7-3 is O(), that is, oe should make the approximatio as tight as possible Aalysis of Algorithms 15 Theorem: If g() is O(f()), the for ay costat c >0 g() is also O(c f()) Theorem: O(f() + g()) = O(max(f(), g())) Ex 1: Ex 2: 2 3 + 3 2 = O (max(2 3, 3 2 )) = O(2 3 ) = O( 3 ) 2 + 3 log 7 = O(max( 2, 3 log 7)) = O( 2 ) Aalysis of Algorithms 16
Simple Big Oh Rule: Drop lower order terms ad costat factors 7-3 is O() Other Big Oh Rules: Use the smallest possible class of fuctios Say 2 is O() istead of 2 is O( 2 ) 8 2 log + 5 2 + is O( 2 log ) 12 3 + 5000 2 + 2 4 is O( 4 ) Use the simplest expressio of the class Say 3 +5 is O() istead of 3 +5 is O(3) Aalysis of Algorithms 17 Aalysis of Algorithms 18 Asymptotic Notatio (termiology) Special classes of algorithms: costat: O(1) logarithmic: O(log ) liear: O() quadratic: O( 2 ) cubic: O( 3 ) polyomial: O( k ), k >0 expoetial: O(a ), > 1 Example of Asymptotic Aalysis A algorithm for computig prefix averages The i-th prefix average of a array X is average of the first (i + 1) elemets of X. That is, A[i] = X[0] + X[1] + + X[i] Aalysis of Algorithms 19 Aalysis of Algorithms 20
Example of Asymptotic Aalysis Algorithm prefixaverages1(x, ) Iput array X of itegers Output array A of prefix averages of X #operatios A ew array of itegers for i 0 to 1 do s 0 for j 0 to i do 1 + 2 + + s s + X[j] 1 + 2 + + A[i] s / (i + 1) retur A 1 The ruig time of prefixaverages1 is O(1 + 2 + + ) The sum of the first itegers is ( + 1) / 2 There is a simple visual proof of this fact Thus, algorithm prefixaverages1 rus i O( 2 ) time 7 6 5 4 3 2 1 0 1 2 3 4 5 6 21 Aalysis of Algorithms 22 Aother Example A better algorithm for computig prefix averages: Algorithm prefixaverages2(x): Iput: A -elemet array X of umbers. Output: A -elemet array A of umbers such that A[i] is the average of elemets X[0],..., X[i]. Let X be a array of umbers. # operatios s 0 1 for i 0 to -1 do s s + X[i] A[i] s/(i+ 1) retur array A 1 O() time Aalysis of Algorithms 23 big-omega (lower boud) f() is Ω(g()) if there exist c > 0 ad 0 > 0 such that f() c g() for all 0 (thus, f() is Ω(g()) iff g() is O(f()) ) f() c g() Aalysis of Algorithms 0 24
big-theta is big theta g() is Θ(f()) <===> if g() O(f()) AND f() O(g()) Big Theta Θ otatio allows us to say that two fuctios grow at the same rate, up to costat factors. Whe we say g() is Θ(f()), it meas there are two costats c 1 ad c 2, ad 0 1 such that c 1 f() g() c 2 f(), for 0. Aalysis of Algorithms 25 Aalysis of Algorithms 26 A Example We have see that f() = 60 2 + 5 + 1 is O( 2 ) but 60 2 + 5 + 1 60 2 for 1 So: with c = 60 ad 0 = 1 f() c 2 for all 1 f() is O( 2 ) AND f() is Ω( 2 ) f() is Ω( 2 ) Ituitio for Asymptotic Notatio Big-Oh f() is O(g()) if f() is asymptotically less tha or equal to g() big-omega f() is Ω(g()) if f() is asymptotically greater tha or equal to g() big-theta f() is Θ(g()) if f() is asymptotically equal to g() f() is Θ( 2 ) Aalysis of Algorithms 27 Aalysis of Algorithms 28
Math You Need to Review Logarithms ad Expoets (Appedix A) properties of logarithms: log b (xy) = log b x+ log b y log b (x/y) = log b x-log b y log b x a = alog b x log b a= log x a/log x b properties of expoetials: a (b+c) = a b a c a bc = (a b ) c a b /a c = a (b-c) b = a log a b b c = a c*log a b Aalysis of Algorithms 29 More Math to Review Floor: x = the largest iteger x Ceilig: x = the smallest iteger x Summatios: (see Appedix A) Geometric progressio: (see Appedix A) Aalysis of Algorithms 30 More Math to Review Arithmetic Progressio More Math to Review Geometric Progressio S = Σ di = 0 + d + 2d + + d i=0 = d+(-1)d+(-2)d + + 0 2S = d + d + d + + d = (+1) d S = d/2 (+1) for d=1 S = 1/2 (+1) Aalysis of Algorithms 31 S = Σ r i = 1 + r + r 2 + + r i=0 rs = r + r 2 + + r + r +1 rs - S = (r-1)s = r +1-1 S = (r +1-1)/(r-1) If r=2, S = (2 +1-1) Aalysis of Algorithms 32