Lecture Notes 6 Itroductio to algorithm aalysis CSS 501 Data Structures ad Object-Orieted Programmig Readig for this lecture: Carrao, Chapter 10 To be covered i this lecture: Itroductio to algorithm aalysis Growth rates Big-O otatio Aalysis of search algorithms Itroductio to algorithm aalysis This lecture we are goig to start discussig the efficiecy of algorithms. This is a importat topic that we will retur to every time that we talk about a ew algorithm. Uderstadig the efficiecy of a algorithm is importat. The speed ad/or resposiveess of a wide-variety of applicatios deped o the efficiecy of the algorithm used i the applicatio. Efficiet algorithms are much more importat tha codig tricks ad optimizatio. Examples of applicatios that rely o efficiet algorithms iclude computer games, productivity software (such as word processig ad spreadsheets), systems you ecouter everyday (such as ATMs ad grocery checkout systems), critical applicatios (life support, air-traffic cotrol), ad Mars rovers. This lecture will be largely theoretical. The techiques that we will talk about form the basis for aalysis that we will perform over ad over agai i this course ad that you will use i future courses. They will allow you to compare algorithms for some problem ad determie which is more efficiet (primarily i terms of the computatio time, but similar techiques ca be applied also to the amout of memory required). This lecture will cocetrate o simple algorithms that you have probably see before, but we will apply the techiques to more advaced algorithms i subsequet lectures. For the most part, we have (so far) cosidered the huma cost of algorithms (readability, modularity, etc.) This impacts how easy it is to geerate, modify, test, ad support programs, rather tha speed or resposiveess for the user. Of course, efficiecy is also importat. You should always cosider the efficiecy whe selectig a particular algorithm to solve a problem. We will discuss the efficiecy of algorithms, rather tha programs. Two differet implemetatios of the same algorithm usually vary little i speed. Clever tricks may gai you a little speed, but the big picture (that is, the efficiecy of the algorithm) is far more importat. We examie the aalysis of algorithms abstractly for aother reaso. If we compare two programs to determie which is faster, the result depeds o which program was coded better, which computer was used to test the programs, ad what data was used for the test. Lookig at the overall efficiecy of a algorithm overcomes these problems. Reasos to aalyze the efficiecy of a algorithm: Aalysis helps choose which solutio to use to solve a particular problem. Experimetatio tells us about oe test case, while aalysis ca tell us about all test cases (performace guaratees). Performace ca be predicted before programmig. If you wait util you ve coded a large project ad the discover that it rus slowly, much time has bee wasted. If you ca tell which parts of your solutio execute quickly ad which parts execute slowly, the you kow which to work o to improve the overall solutio. Growth rates We wat to aalyze the ruig time without kowig the details of the iput. This is doe by measurig the ruig time as a fuctio of the size of the problem (for example, the value of a iteger iput, the umber of elemets i a array, or the umber of odes i a liked list). The size is usually called, although ay variable ca be used to deote this size.
Let s say we have 3 algorithms to solve some problem. Oe takes time proportioal to 100, oe takes time proportioal to 10^2, ad oe takes time proportioal to 2^. These are the growth rates for the algorithms i terms of the size of the problem. Let s examie how log these algorithms take for various problem sizes, assumig that times are all i millisecods. Alg. 1 Alg. 2 Alg. 3 = 5 500 250 32 = 10 1000 1000 1024 = 20 2000 4000 1,048,576 = 50 5000 25,000 1.1259 x 10^15 (over 36,000 years) = 1000 100,000 10,000,000 > 10^300 Algorithm 3 is the fastest for a very small problem, but all of the algorithms are doe i uder a secod. For a slightly larger problem, they are all about the same. As the size of the problem gets bigger, algorithm 3 becomes completely ureasoable ad algorithm 1 looks better ad better. Usually, it is the performace of algorithms for large problems that we are cocered about, sice small problem ca be solved quickly usig most algorithms. The rate of growth is the primary measuremet we will use to aalyze efficiecy. Clearly, the rate of growth is the largest for algorithm 3 ad lowest for algorithm 1. Algorithm 1 is the best to use if you do t kow the size of the problem i advace. We will examie growth rates usig Big-O otatio. Big-O otatio Iformally, if the time required by a algorithm is proportioal to some fuctio f(), the the algorithm s ruig time is said to be O(f()), where f() is the growth-rate fuctio. This is called Big-O otatio due to the O (short for order) at the begiig of the otatio. Sice the ruig time of algorithm 2 is proportioal to ^2, its ruig time is O(^2). O() is usually spoke as big-oh of or simply oh of. Formal defiitio: A fuctio (such as a algorithm s growth rate) f() is O(g()), if there are costats k ad 0 such that f() <= k*g() for all >= 0. Example 1: Algorithm 1 required 100 millisecods to ru. I this case, f() = 100 (we will igore whatever uits are used i the measuremet). Is the ruig time O()? Yes. If we choose k to be 100, the f() <= 100 for all > 0. (Is the ruig time of the algorithm O(^2)? O(log )?) Example 2: Let s say a algorithm requires time 20^2 + 100. Is this O()? No. 20^2 + 100 grows faster tha. No matter what costat you use, there is a for which 20^2 + 100 is bigger tha k*. Is this O(^2)? Yes. If we choose k to be 21 the 20^2+100 is always less tha 21^2 for > 10. Why did I choose those costats? I wated to prove that 20^2 + 100 <= k^2. Oe methodology is to choose a reasoable value of k (we ca see it eeds to be greater tha 20 here) ad solve for (if possible). With k = 21, we get 100 <= ^2 ad this is true for >=10. What Big-O otatio idicates is that, for a sufficietly large problem ( > 0 ), the growth rate is proportioal to (or less tha) the g() i O(g()). Note that, accordig to the defiitio f() = 1 is O(^25), sice it is always less tha or equal to ^25 for >= 1. This does t tell us much. For this reaso, we always choose the fuctio g() to be as small as we ca ad still be true. Ca a fuctio be O(1)? Yes, if the size of the fuctio ca be bouded, o matter what is. A example is f() = 10 -. No matter how large is, this fuctio ca ever get larger tha 10. Therefore, we ca choose k = 10 ad 0 = 0. With these values f() <= k * 1 for all > 0.
Rules for simplifyig Big-O otatio: Igore lower order terms: f(x) = x^2 + 1000x + 1000000 is O(x^2). Note also that it does t matter what variable you use. It is customary to use, but ay variable ca be used. Igore multiplicative costats: f() = 42 is O(). O(f()) + O(g()) = O(f() + g()): If your fuctio has two parts ad the first has complexity f() = 10 ad the secod has complexity g() = ^2. The the total complexity is O(10 + ^2), which is O(^2). O(f()) * O(g()) = O(f() * g()): If your fuctio repeats the ier loop x times ad the ier loop takes time x + 5, the the total complexity is O(x * (x + 5)) = O(x^2). Examples: 8 + 10^3 + 35 is: O(^3) ¼ ^2 + 25 log is: O(^2) 5 + 2^ / 5 is: O(2^) For algorithms, the complexity of the algorithm refers to the rate of growth, so it commo to say that the complexity of a algorithm is O(somethig). The smallest possible complexity for algorithms is O(1), sice at least some costat amout of time must be spet o matter what. The hierarchy of complexities is: O(1): Essetially costat. The complexity ca be bouded, o matter how big is. Example: addig 10 to a iteger, accessig oe array elemet. O(log ): Logarithmic growth. This complexity grows very slowly. The base does t matter. Example: biary search of a sorted list. O(): Liear growth. Example: traversig a liked list with odes, or sequetial search of a array. O( log ): Slightly worse tha liear. Usually occurs whe you break a problem dow ito two problems of half the size that must both be solved. Example: Merge sort. O(^2): Quadratic growth. Grows with the square of the problem size. Ofte two ested loops. Example: Isertio sort. O(^3): Cubic growth. Worse tha quadratic. Ofte three ested loops. Example: Matrix multiplicatio (straightforward approach). O(2^): Expoetial growth. Usually too slow for large problems. Example: The towers of Haoi. O(1) < O(log ) < O() < O( log ) < O(^2) < O(^3) < O(2^) Note that these are ot the oly possible complexities, but they are the most commo. Before we start lookig at particular algorithms, I wat to ote that, while Big-O complexity is very importat, it does ot tell the whole story. As we ca see from the above rules, Big-O hides the costat of proportioality ad lower order terms So, algorithms with the same complexity ca vary by a arbitrary costat factor (the lower order terms are usually less importat for large problems). If some algorithm has a extremely large costat multiplier, it may still be impractical, eve if the complexity is low. If a fuctio f() is O(g()), this says that f() grows o faster tha g() after hidig costats for large eough values of. It does t say that f() grows as fast as g(). There is a differet otatio f() = Ω(g()) (proouced Omega of g of ) that says f() grows at least as fast as g() for large eough values of (ad after hidig costats). If f() is both O(g()) ad Ω(g()), the it is said to be (g()) (theta of g of ).
Aalysis of search algorithms Now, we are goig to aalyze the complexity of two search algorithms: sequetial search ad biary search. We will determie the efficiecy of the algorithms i terms of the umber of items i the list that is beig searched. Whe the ruig time depeds o the data that is give to the algorithm, there are several ways we ca aalyze the ruig time of the algorithm. Best case: The smallest umber of comparisos that ca be achieved with ay iput. This ca be iterestig, but it is ot very useful, i geeral. Worst case: The largest umber of comparisos that ca be achieved with ay iput. This ca be very importat if we wat to guaratee that the algorithm will fiish withi some time. Average case: The average umber of comparisos that are performed. This is usually the most importat case to aalyze, but it is usually the most difficult to determie. If the algorithms performace does ot deped o the data (summig a array, outputtig a liked list), the all of these will be the same. However, the performace of may algorithms depeds o the data give to the algorithm. Oe example is searchig a list. If the item is at the begiig of the list, we fid it right away usig sequetial search. If the item is t i the list, the we have to examie the etire list. Let s cosider the followig sequetial search code. This is a iterative versio of the algorithm we saw previously. // Fid the positio of a item i a list. Returs 1 if the item is ot i the list. template <typeame Comparable> it sequetialsearch(cost vector<comparable> &alist, cost Comparable &item) { for (it i = 0; i < alist.size(); i++) if (item == alist[i]) retur i; retur -1; } We ca measure the complexity of the search i terms of the umber of comparisos made, sice there is a costat amout of other work performed per compariso. We will assume that the comparisos ca be performed i O(1) time, although this is t ecessarily true, i geeral. I the best case, sequetial search fids the item o the first try. I the worst case, sequetial search examies each item i the list ad, thus, iteratios ad comparisos are performed. Therefore, the complexity is O(). The average case is the positio of item i the list (assumig that it is i the list). If item is i the list, let s assume that each positio i the list is equally likely. That is: P[i] = 1/. Assumig item is i the list, the average umber of comparisos is foud by summig the probability of beig at each positio with the umber of comparisos required for each positio: 1 C() = P[ i] i = i i1 i1 1 = i i 1 = 1 ( 1) 2 = ( 1) 2 = O(). If item is t i the list, the the umber of comparisos is always, which is also O(). Say the probability of a item beig i the list is p, the, overall, the average umber of iteratios of the loop is: ( 1) p( 1) p (1 p) 2 2 This is also O(). Note that the complexity does t deped o whether the list is sorted or ot.
If the list is sorted, we ca use biary search, as we saw last lecture. // Biary search usig recursio. // Precoditio: the list must be sorted ito ascedig order. template <typeame Comparable> it biarysearch(cost vector<comparable> &list, cost Comparable &item, it first, it last) { if (first > last) retur 1; it mid = (first + last) / 2; if (item == list[mid]) retur mid; if (item < list[mid]) retur biarysearch(list, item, first, mid - 1); else retur biarysearch(list, item, mid + 1, last); } Example: alist = 2 3 5 7 11 13 17 item = 5 I geeral, how log does the fuctio take? Each iteratio of the while loop requires a costat amout of time - O(1) assumig that the compariso ca be performed i costat time. This meas we ca measure the complexity of the fuctio by coutig the umber of iteratios through the loop. I the best case, we fid it o the first try. The average case is a little tricky for this fuctio ad we wo t compute it i this class, but we ca compute the worst case. We ca defie the umber of iteratios ecessary i the worst case usig a recurrece relatio: F(1) = 1 F() 1 + F(/2) 1 + 1 + F(/4) k + F(/2 k ) The process eds whe /2 k is less tha or equal to 1 ad we ca use the base case F(1): F() 1 + log 2 = O(log ) Usually, whe you ca break the problem i half ad the fid the solutio by solvig oly oe half of the origial problem, the complexity of the algorithm will be O(log ). If you have to solve both halves of the problem, this is ot true, as we will see whe we discuss sortig. It turs out that whe is large, the average case is early the same as the worst case, although it is slightly lower. How much better is biary search tha sequetial search? For a 1 millio word dictioary, sequetial search does 500,000.5 comparisos o average (if the word is i the dictioary). For the same dictioary, biary search does 21 comparisos i the worst case. Note, however, that you must have a sorted list to use biary search. Iterestigly, the average case performace of biary search ca be improved usig iterpolatio search. The idea is that you do t start lookig i the middle of the list. You predict where the item should be usig iterpolatio. For example, Hak Aaro would be expected to be ear the start of the phoe book. It turs out that the worst-case performace of this algorithm is O(), but the average case is O(log log ). However, because of the computatio time to perform the iterpolatio, it is usually ot as fast as biary search! The best time to use this algorithm would be whe the list of items to search is larger tha ca fit i memory. Sice accessig the disk is expesive (compared to workig i memory), it is ideal to reduce the umber of times the disk eeds to be accessed. Powerpoit for Chapter 10 Practice problems (optioal): Carrao, Chapter 10: #2a,b,e,g,h, #3, #4, #5 #3a,b, e,g,h, #4, #7 Which of the followig fuctios are O( 2 )? (Remember that Big-O otatio provides a upper boud oly.) a. 15 1 d. log 10 b. 3 / 10 e. 4 / (( - 2) ( - 3)) c. 6 2 + 12 + 8 f. 2 / 2 For the typical algorithms that you use to perform calculatios by had, determie the ruig time i Big-O otatio to:
a. Add two N-digit itegers b. Multiply two N-digit itegers c. Divide two N-digit itegers