Itro to Algorithm Aalysis Aalysis Metrics Slides. Table of Cotets. Aalysis Metrics 3. Exact Aalysis Rules 4. Simple Summatio 5. Summatio Formulas 6. Order of Magitude 7. Big-O otatio 8. Big-O Theorems 9. Complexity Classes 0. Practical Complexity Classes. Big-O Simple Summatio. Big-O Aalysis Rules 3. Big-O Array Summatio 4. Array Summatio (exact cout) 5. Practical Applicatios 6. Hardware Speedup 7. Algorithm Behavior Program Ruig (Executio) Time Factors Machie Speed (ot just CPU speed) Programmig Laguage ad Implemetatio Compiler Code Geeratio (optimizatio) Iput Data Size Time Complexity of Algorithm umber of executed statemets: T() Fuctio of the size of the iput (termed ) Ruig Time Factor Implicatios Compiler code geeratio & processor speed differeces are too great to be used as a basis for impartial algorithm comparisos. Overall system load may cause icosistet timig results, eve if the same compiler ad hardware are used. Hardware characteristics, such as the amout of physical memory ad the speed of virtual memory, ca domiate timig results. I ay case, those factors are irrelevat to the complexity of the algorithm. Algorithm Aalysis == Complexity Aalysis
Exact Aalysis Rules 3 Simple Summatio 4 Whe attemptig a exact time aalysis: Give: for (i = 0; i < -; i++) {. We assume a arbitrary time uit.. Ruig of each of the followig operatios takes time T(): a) assigmet operatios b) I/O operatios c) Boolea operatios d) arithmetic operatios e) fuctio retur 3. Ruig time of a selectio statemet (if, switch) is the time for the coditio evaluatio + the maximum of the ruig times for the idividual clauses i the selectio. 4. Loop executio time is the time for the setup (iitializatio & setup) + the sum, (over the umber of times the is executed), of the body time + time for the check ad update operatios. (Loop setup will iclude the termiatio check o pre-test s.) Always assume that the executes the maximum umber of iteratios possible 5. Ruig time of a fuctio call is T() for setup + the time for ay parameter calculatios + the time required for the executio of the fuctio body. o-executable statemets, (e.g., declaratios), are ot couted. Oly executable statemets are aalyzed. Rules 4 ad a: time before Rules 4 ad a: time o each iter. of outer Outer will execute - times -- this is the exteral sum. Ier will execute i times -- this is the iteral sum. So, the total time T() is give by: For For ease ease of of summatio summatio cout cout repetitios repetitios startig startig at at.. for (j = 0; j < i; j++) { aray[i][j] = 0; for (i = 0; i < -; i++) { Rules 4 & a: time before for (j = 0; j < i; j++) { aray[i][j] = 0; Rules 4 & a: time before Rules 4, c ad d: time 3 o each iteratio of outer T ( ) Summatio from applyig Rule 4 to outer Rules 4, c ad d: time (o each iteratio of ier ) = 3 + Rule a: time o each pass of ier 5 + i= j= Summatio from applyig Rule 4 to ier i 3
Summatio Formulas 5 Order of Magitude 6 Let > 0, let A, B, ad C be costats, ad let f ad g be ay fuctios. The: Fuctio Estimatio Give a algorithm that takes time: f() = 3 + 5 + 00 k = Cf ( k) = C f ( k) ( f ( k) ± g( k)) = f ( k) ± k = k = k = S: factor out costat S: separate summed terms k = g( k) Graphically: 4000 000 ( + ) k = )( + + C = C k = k = k= k = 6 S3: sum of costat S4: sum of k S5: sum of k squared ( ) time 0000 8000 6000 f() ^ 5^ The summatio formulas above ca be used to evaluate the expressios obtaied whe aalyzig the complexity of a algorithm: T ( ) = 3 + = 3+ = 3+ 5 + i= j= ( 5 + 3i) i= 3 () 5 + 3 () i i= i= ( )( ) = 3+ 5( ) + 3 3 7 = + i From aalysis o previous slide. Apply S3 to the ier sum. Apply S ad S to the outer sum. Apply S3 to the first sum, ad apply S4 to the secod sum. Simplify ad combie terms. Algebraically: 4000 000 0 5 9 3 7 5 9 33 37 4 45 49 (iput size) If > 0 the > 00 If > 5 the > 5 Therefore, if > 0 the: f() = 3 + 5 + 00 < 3 + + = 5 So 5 forms a upper boud o f() if is 0 or larger (asymptotic boud). I other words, f() does't grow ay faster tha 5 i the log ru.
Big-O otatio 7 Big-O Theorems 8 Big-O otatio is used to express the asymptotic growth rate of a fuctio. Formally suppose that f() ad g() are fuctios of. The we say that f() is i O(g()) provided that there are costats C > 0 ad > 0 such that for all > the f() < Cg(). We ofte say that f is big-o of g or that f is i O(g). By the defiitio above, demostratig that a fuctio f is big-o of a fuctio g requires that we fid specific costats C ad for which the iequality holds (ad show that the iequality does, i fact, hold). Example: from the previous slide, if > 0 the f() = 3 + 5 + 00 < 5 So, by the defiitio above, f() is i O( ). C = 5 = 0 ote that 5 < 9 (for all ), so we could also coclude that f() is O(9 ). Usually, we re iterested i the tightest (smallest) upper boud, so we d prefer 5 to 9. The aalysis give o slide 4 of this chapter is typical of big-o aalysis, ad is somewhat tricky. I order to simplify our work, we state the followig theorems about big-o: Assume that f() is a fuctio of ad that K is a arbitrary costat. Thm : K is O() Thm : A polyomial is O(the term cotaiig the highest power of ) Thm 3: K*f() is O(f()) [i.e., costat coefficiets ca be dropped] Thm 4: I geeral, f() is big-o of the domiat term of f(), where domiat may usually be determied by the followig list: Complexity Classes smaller costats log b () [always log base if o base is show] log b () to higher powers 3 larger costats to the -th power! [ factorial] larger Thm 5: For ay base b, log b () is O(log()).
Complexity Classes 9 Practical Complexity Classes 0 Commo Growth Curves Low-order Curves 00 0 00 log 000 800 log 80 60 600 log ^ ^3 40 0 400 ^ 0^ 0 3 5 7 9 3 5 7 9 3 log 00 (iput size) 0 Observatios 3 4 5 6 7 8 9 0 (iput size) order & less Algorithms with Order > require FAR more time tha algorithms with Order or less, eve for fairly small iput sizes. For small, there s ot much practical differece betwee Order ad order log. Observatios Eve for moderately small iput sizes, Order algorithms will require FAR more time tha Order log() algorithms. costats of proportioality, (coefficiets & lesser terms), have very little effect for large values of (betwee complexity classes). Large problems with Order > log() caot practically be executed For = 000 (medium problems) algorithms ca still be used
Big-O Simple Summatio Big-O Aalysis Rules Recall: for (i = 0; i < -; i++) { Whe attemptig a approximate big-o time aalysis: for (j = 0; j < i; j++) { aray[i][j] = 0; had a total time complexity T() is give by (slide 4): T ( ) + i = 3 + 5 i= j= 3 3 7 ad that T() reduced to (slide 5): T ( ) = + 3 ow by Theorem : ad the by Theorem 3: 3 T ( ) O ( ) T ( ) O So we d say that the give code fragmet is of order complexity.. We assume a arbitrary time uit.. Ruig of each of the followig type of statemet takes time T(): [omittig the arithmetic operators] a) assigmet statemet b) I/O statemet c) Boolea expressio evaluatio d) fuctio retur 3. Ruig time of a selectio statemet (if, switch) is T() for the coditio evaluatio + the maximum of the ruig times for the idividual clauses i the selectio. 4. Loop executio time is the time for the setup (iitializatio & setup) + the sum, over the umber of times the is executed, of the body time + time for the check ad update operatios. Always assume that the executes the maximum umber of iteratios possible 5. Ruig time of a fuctio call is T() for fuctio setup + the time required for the executio of the fuctio body. Igore idividual Boolea operatios & arithmetic operatios Igore parameter computatios 6. Ruig time of a sequece of statemets is the largest time of ay statemet i the sequece. Of course, this ivolved some uecessary work. Big-O aalysis provides a gross idicatio of the complexity of a algorithm, ad that ca be obtaied without first doig a exact aalysis (as we did o slides 4 ad 5 for this code fragmet). idicates chages from the rules for exact aalysis stated earlier.
Big-O Array Summatio 3 Array Summatio (exact cout) 4 typedef it raytype[]; void sumito(raytype ray, it ) { it i, j, t; i = 0; // a while (i <= ) { // b (outer ) j = t = 0; // c while (j <= i) { // d (ier ) t = t + ray[j]; // e j++; // f ray[i] = t; // g i++; // h Aalysis will deal with the statemets labeled a.. h; executable statemets oly. Ier Loop: Sum from 0..i (or..i+) of the body. Body Rule 6: Maximum of { coditio, e, f Rule : coditio, e, ad f each take So, the ier body takes time O() ad so the ier is i+ O = O i ( + ) = O( i) j= Outer Loop: Sum from 0.. (or..+) of the body. Body Rule 6: Maximum of { coditio, c, ier, g, h Rule : coditio, c, g, ad h each take time So, the outer body takes time O(i) ad so the outer is ( + )( + ) O + i = O = O i= ( ) Fially by Rule 6, the big-o complexity of the fuctio is the maximum of the outer ad statemet, which is O(); so the fuctio is O( ). 3 + + = O typedef it raytype[]; void sumito(raytype ray, it ) { it i, j, t; i = 0; // a while (i <= ) { // b (outer ) j = t = 0; // c while (j <= i) { // d (ier ) t = t + ray[j]; // e j++; // f ray[i] = t; // g i++; // h Aalysis will deal with the statemets labeled a.. h; executable statemets oly. Ier Loop: Sum from 0..i (or..i+) of the body. Body Rule 4: Sum of { coditio, e, f Rule : coditio ad f each take time ; e takes time So, the ier body takes time 4 ad so the time for the ier is i + 4 = 4( i + ) j= Outer Loop: Sum from 0.. (or..+) of the body. Body Rule 6: Sum of { coditio, c, ier cod, ier, g, h Rule : coditio, g, ad h each take time ; c takes time So, the outer body takes 4(i+) + 6 or 4i + 0, ad so the outer plus statemet a is ( ) = + + T i= ( + )( + ) ( 4i + 0) = + 4 + 0( + ) = + 6 +
Practical Applicatios 5 Hardware Speedup 6 Assume: day 00,000 sec. 0 5 sec. (actually 86, 400) Iput size = 0 6 A computer that executes,000,000 Ist/sec C/C++statemet istructios Algorithm Complexity Class Compariso Order: Order: (0 (0 6 6 )) Istructios 0 0 Istructios 0 0 // 0 0 6 6 secs secsto to ru ru 0 0 6 6 secs secsto to ru ru 0 0 6 6 // 0 0 5 5 days days to to ru ru 0 0 days days to to ru ru Iteral Class Comparisos Order: Order: log log 0 0 6 6 log log 0 0 6 6 Istructios 0 0 (( 0 0 6 6 )) = (( 0 0 7 7 )) (( 0 0 7 7 ))// 0 0 6 6 secs secs to to ru ru 0 0 sec sec to to ru ru Withi complexity classes the differeces betwee algorithms due to costats of proportioality, (coefficiets & lesser terms), are ot sigificat eough to warrat reportig except for certai (high usage) applicatios (e.g., sortig, searchig) Does the fact that hardware is always becomig faster hardware mea that algorithm complexity does t really matter? Suppose we could obtai a machie that was capable of executig 0 times as may istructios per secod (so roughly 0 times faster tha the machie hypothesized o the previous slide). How log would the order algorithm take o this machie with a iput size of 0 6? Order: Order: # istructios: (0 (0 6 6 )) = 0 0 # secods to to ru: ru: 0 0 // 0 0 7 7 = 0 0 5 5 # days days to to ru: ru: 0 0 5 5 // 0 0 5 5 = = Impressed? You should t be. That s still day versus 0 secods if a algorithm of order log() were used. What about 00 times faster hardware?.4 hours.
Algorithm Behavior 7 Categories Algorithms must be examied uder differet situatios to correctly determie their efficiecy for accurate comparisos. Best Case Aalysis Assumes the iput, data etc. are arraged i the most advatageous order for the algorithm, i.e. causes the executio of the fewest umber of istructios. E.g., sortig - list is already sorted; searchig - desired item is located at first accessed positio. Worst Case Aalysis Assumes the iput, data etc. are arraged i the most disadvatageous order for the algorithm, i.e. causes the executio of the largest umber of statemets. E.g., sortig - list is i opposite order; searchig - desired item is located at the last accessed positio or is missig. Average Case Aalysis Determies the average of the ruig times over all possible permutatios of the iput data. E.g., searchig - desired item is located at every positio, for each search), or is missig. Caveats Algorithms may have quite differet Orders for the aalysis categories, e.g., O(), O( ), O(log), respectively.