Aalysis of Algorithms Iput Algorithm Output A algorithm is a step-by-step procedure for solvig a problem i a fiite amout of time. Ruig Time Most algorithms trasform iput objects ito output objects. The ruig time of a algorithm typically grows with the iput size. Average case time is ofte difficult to determie. We focus o the worst case ruig time. Easier to aalyze Crucial to applicatios such as games, fiace ad robotics Ruig Time 120 100 80 60 40 20 0 best case average case worst case 1000 2000 3000 4000 Iput Size Aalysis of Algorithms 2
Experimetal Studies Write a program implemetig the algorithm Ru the program with iputs of varyig size ad compositio Use a method like System.curretTimeMillis() to get a accurate measure of the actual ruig time Plot the results Time (ms) 9000 8000 7000 6000 5000 4000 3000 2000 1000 0 0 50 100 Iput Size Aalysis of Algorithms 3 Limitatios of Experimets It is ecessary to implemet the algorithm, which may be difficult Results may ot be idicative of the ruig time o other iputs ot icluded i the experimet. I order to compare two algorithms, the same hardware ad software eviromets must be used Aalysis of Algorithms 4
Theoretical Aalysis Uses a high-level descriptio of the algorithm istead of a implemetatio Characterizes ruig time as a fuctio of the iput size,. Takes ito accout all possible iputs Allows us to evaluate the speed of a algorithm idepedet of the hardware/software eviromet Aalysis of Algorithms 5 Pseudocode High-level descriptio of a algorithm More structured tha Eglish prose Less detailed tha a program Preferred otatio for describig algorithms Hides program desig issues Example: fid max elemet of a array Algorithm arraymax(a, ) Iput array A of itegers Output maximum elemet of A curretmax A[0] for i 1 to 1 do if A[i] > curretmax the curretmax A[i] retur curretmax Aalysis of Algorithms 6
Pseudocode Details Cotrol flow if the [else ] while do repeat util for do Idetatio replaces braces Method declaratio Algorithm method (arg [, arg ]) Iput Output Method call var.method (arg [, arg ]) Retur value retur expressio Expressios Assigmet (like = i Java) = Equality testig (like == i Java) 2 Superscripts ad other mathematical formattig allowed Aalysis of Algorithms 7 The Radom Access Machie (RAM) Model A CPU A potetially ubouded bak of memory cells, each of which ca hold a arbitrary umber or character 0 12 Memory cells are umbered ad accessig ay cell i memory takes uit time. Aalysis of Algorithms 8
Seve Importat Fuctios ( 4.1) Seve fuctios that ofte appear i algorithm aalysis: Costat 1 Logarithmic log Liear N-Log-N log Quadratic 2 Cubic 3 Expoetial 2 I a log-log chart, the slope of the lie correspods to the growth rate of the fuctio T ( ) 1E+30 1E+28 Cubic 1E+26 1E+24 Quadratic 1E+22 1E+20 Liear 1E+18 1E+16 1E+14 1E+12 1E+10 1E+8 1E+6 1E+4 1E+2 1E+0 1E+0 1E+2 1E+4 1E+6 1E+8 1E+10 Aalysis of Algorithms 9 Primitive Operatios Basic computatios performed by a algorithm Idetifiable i pseudocode Largely idepedet from the programmig laguage Exact defiitio ot importat (we will see why later) Assumed to take a costat amout of time i the RAM model Examples: Evaluatig a expressio Assigig a value to a variable Idexig ito a array Callig a method Returig from a method Aalysis of Algorithms 10
Coutig Primitive Operatios By ispectig the pseudocode, we ca determie the maximum umber of primitive operatios executed by a algorithm, as a fuctio of the iput size Algorithm arraymax(a, ) # operatios curretmax A[0] 2 for i 1 to 1 do 2 if A[i] > curretmax the 2( 1) curretmax A[i] 2( 1) { icremet couter i } 2( 1) retur curretmax 1 Total 8 2 Aalysis of Algorithms 11 Estimatig Ruig Time Algorithm arraymax executes 8 2 primitive operatios i the worst case. Defie: a = Time take by the fastest primitive operatio b = Time take by the slowest primitive operatio Let T() be worst-case time of arraymax. The a (8 2) T() b(8 2) Hece, the ruig time T() is bouded by two liear fuctios Aalysis of Algorithms 12
Growth Rate of Ruig Time Chagig the hardware/software eviromet Affects T() by a costat factor, but Does ot alter the growth rate of T() The liear growth rate of the ruig time T() is a itrisic property of algorithm arraymax Aalysis of Algorithms 13 Costat Factors The growth rate is ot affected by costat factors or lower-order terms Examples 10 2 + 10 5 is a liear fuctio 10 5 2 + 10 8 is a quadratic fuctio T () 1E+26 1E+24 1E+22 1E+20 1E+18 1E+16 1E+14 1E+12 1E+10 1E+8 1E+6 1E+4 1E+2 1E+0 Quadratic Quadratic Liear Liear 1E+0 1E+2 1E+4 1E+6 1E+8 1E+10 Aalysis of Algorithms 14
Big-Oh Notatio ( 4.2.5) Give fuctios f() ad g(), we say that f() is O(g()) if there are positive costats c ad 0 such that 10,000 1,000 100 3 2+10 f() cg() for 0 Example: 2 + 10 is O() 2 + 10 c (c 2) 10 10/(c 2) Pick c = 3 ad 0 = 10 10 1 1 10 100 1,000 Aalysis of Algorithms 15 Big-Oh Example Example: the fuctio 2 is ot O() 2 c c The above iequality caot be satisfied sice c must be a costat 1,000,000 100,000 10,000 1,000 100 10 1 ^2 100 10 1 10 100 1,000 Aalysis of Algorithms 16
More Big-Oh Examples 7-2 7-2 is O() eed c > 0 ad 0 1 such that 7-2 c for 0 this is true for c = 7 ad 0 = 1 3 3 + 20 2 + 5 3 3 + 20 2 + 5 is O( 3 ) eed c > 0 ad 0 1 such that 3 3 + 20 2 + 5 c 3 for 0 this is true for c = 4 ad 0 = 21 3 log + 5 3 log + 5 is O(log ) eed c > 0 ad 0 1 such that 3 log + 5 c log for 0 this is true for c = 8 ad 0 = 2 Aalysis of Algorithms 17 Big-Oh ad Growth Rate The big-oh otatio gives a upper boud o the growth rate of a fuctio The statemet f() is O(g()) meas that the growth rate of f() is o more tha the growth rate of g() We ca use the big-oh otatio to rak fuctios accordig to their growth rate g() grows more f() grows more Same growth f() is O(g()) Yes No Yes g() is O(f()) No Yes Yes Aalysis of Algorithms 18
Big-Oh Rules If is f() a polyomial of degree d, the f() is O( d ), i.e., 1. Drop lower-order terms 2. Drop costat factors Use the smallest possible class of fuctios Say 2 is O() istead of 2 is O( 2 ) Use the simplest expressio of the class Say 3 + 5 is O() istead of 3 + 5 is O(3) Aalysis of Algorithms 19 Asymptotic Algorithm Aalysis The asymptotic aalysis of a algorithm determies the ruig time i big-oh otatio To perform the asymptotic aalysis We fid the worst-case umber of primitive operatios executed as a fuctio of the iput size We express this fuctio with big-oh otatio Example: We determie that algorithm arraymax executes at most 8 2 primitive operatios We say that algorithm arraymax rus i O() time Sice costat factors ad lower-order terms are evetually dropped ayhow, we ca disregard them whe coutig primitive operatios Aalysis of Algorithms 20
Computig Prefix Averages We further illustrate asymptotic aalysis with two algorithms for prefix averages The i-th prefix average of a array X is average of the first (i + 1) elemets of X: A[i] = (X[0] + X[1] + + X[i])/(i+1) Computig the array A of prefix averages of aother array X has applicatios to fiacial aalysis 35 30 25 20 15 10 5 0 X A 1 2 3 4 5 6 7 Aalysis of Algorithms 21 Prefix Averages (Quadratic) The followig algorithm computes prefix averages i quadratic time by applyig the defiitio Algorithm prefixaverages1(x, ) Iput array X of itegers Output array A of prefix averages of X #operatios A ew array of itegers for i 0 to 1 do s X[0] for j 0 to i do 1 + 2 + + ( 1) s s + X[j] 1 + 2 + + ( 1) A[i] s / (i + 1) retur A 1 Aalysis of Algorithms 22
Arithmetic Progressio The ruig time of prefixaverages1 is O(1 + 2 + + ) The sum of the first itegers is ( + 1) / 2 There is a simple visual proof of this fact Thus, algorithm prefixaverages1 rus i O( 2 ) time 7 6 5 4 3 2 1 0 1 2 3 4 5 6 Aalysis of Algorithms 23 Prefix Averages (Liear) The followig algorithm computes prefix averages i liear time by keepig a ruig sum Algorithm prefixaverages2(x, ) Iput array X of itegers Output array A of prefix averages of X A ew array of itegers s 0 1 for i 0 to 1 do s s + X[i] A[i] s / (i + 1) retur A 1 Algorithm prefixaverages2 rus i O() time #operatios Aalysis of Algorithms 24
Math you eed to Review Summatios Logarithms ad Expoets Proof techiques Basic probability properties of logarithms: log b (xy) = log b x + log b y log b (x/y) = log b x - log b y log b xa = alog b x log b a = log x a/log x b properties of expoetials: a (b+c) = a b a c a bc = (a b ) c a b /a c = a (b-c) b = a log a b b c = a c*log a b Aalysis of Algorithms 25 Relatives of Big-Oh big-omega f() is Ω(g()) if there is a costat c > 0 ad a iteger costat 0 1 such that f() c g() for 0 big-theta f() is Θ(g()) if there are costats c > 0 ad c > 0 ad a iteger costat 0 1 such that c g() f() c g() for 0 Aalysis of Algorithms 26
Ituitio for Asymptotic Notatio Big-Oh f() is O(g()) if f() is asymptotically less tha or equal to g() big-omega f() is Ω(g()) if f() is asymptotically greater tha or equal to g() big-theta f() is Θ(g()) if f() is asymptotically equal to g() Aalysis of Algorithms 27 Example Uses of the Relatives of Big-Oh 5 2 is W ( 2 ) f() is Ω(g()) if there is a costat c > 0 ad a iteger costat 0 1 such that f() c g() for 0 let c = 5 ad 0 = 1 5 2 is W () f() is Ω(g()) if there is a costat c > 0 ad a iteger costat 0 1 such that f() c g() for 0 let c = 1 ad 0 = 1 5 2 is Θ( 2 ) f() is Θ(g()) if it is Ω( 2 ) ad O( 2 ). We have already see the former, for the latter recall that f() is O(g()) if there is a costat c > 0 ad a iteger costat 0 1 such that f() < c g() for 0 Let c = 5 ad 0 = 1 Aalysis of Algorithms 28