CIS 11 Data Structures ad Algorithms with Java Fall 017 Big-Oh Notatio Tuesday, September 5 (Make-up Friday, September 8) Learig Goals Review Big-Oh ad lear big/small omega/theta otatios Practice solvig recurreces ad provig asymptotic bouds Practice desigig algorithms ad justifyig rutime ad correctess What are Algorithms? What is a algorithm? A algorithm is ay well-defied computatioal procedure that takes some value, or set of values, as iput ad produces some value, or set of values, as output. Therefore, a algorithm is a sequece of computatioal steps that trasform the iput ito output. We ca also view a algorithm as a tool for solvig a well-specified computatioal problem. The statemet of the problem specifies the desired iput/output relatioship, ad the algorithm describes a specific computatioal procedure for achievig that relatioship. Best, Average, Worst Case Ru Time Aalysis Whe aalyzig algorithms, we are ofte iterested i aalyzig the best, average, ad worst cases of ruig time. Typically, best case performace is ot really of cocer due to triviality. Algorithms may be modified to make best case performace trivial for small iput by hardcodig. I these cases, the best case performace is effectively meaigless! Ofte, it is of more cocer to perform worst case aalysis, i.e. idetifyig the iputs that cause the algorithm to have the logest ruig time (or use the most space) ad idetifyig the ruig time ad usage bouds. Why is this useful? The worst case ruig time of a algorithm gives a upper boud o the ruig time for ay iput. Kowig this provides a guaratee that the algorithm ever takes ay loger. For some algorithms, the worst case may occur fairly ofte. Ofte, the average case is roughly as bad as the worst case. Fially, i some cases we may be iterested i the average case ruig time of a algorithm, where we would use probabilistic aalysis to examie particular algorithms. This does t surface too much, as what costitutes a average iput is ofte ot apparet. Ofte, we would the assume that all iputs of a particular size are equally likely (uiform distributio). This assumptio is ofte violated i practice, so we might modify a algorithm to be radomized, to eable a probabilistic aalysis ad expected ruig time. Such algorithms are outside the mai scope of this class. Big-Oh Defiitios Preseted below is a brief overview of asymptotic otatio that will be fudametal i this course. 1
Big-Oh Notatio Defiitio (1). f() O(g()) if there exist positive costats 0 ad c such that f(i) cg(i) for all i 0. f() Defiitio (). f() O(g()) if lim g() = 0 or a costat. Simplified: If f() is i O(g()), g() is a asymptotic upper boud for f(). Big-Omega Notatio Defiitio (1). f() Ω(g()) if there exist positive costats 0 ad c such that f(i) cg(i) for all i 0. f() Defiitio (). f() O(g()) if lim g() = or a costat. Simplified: If f() is Ω(g()), g() is a asymptotic lower boud for f(). Big-Theta Notatio Defiitio (1). f() Θ(g()) if ad oly if f() O(g()) ad f() Ω(g()). f() Defiitio (). f() O(g()) if lim g() = a costat. Simplified: If f() is Θ(g()), g() is a asymptotically tight boud for f(). The otatios refer to classes of fuctios. Whe you read f() = O(g()), this is equivalet to the statemet: f() O(g()). Specifically, f() is i the class of fuctios which are asymptotically bouded above by g(). Likewise, Big-Ω, Big-Θ, etc., all reflect classes of fuctios. I additio to Big-O, Ω, ad Θ, we also leared about tilde approximatio. Tilde Notatio f() Defiitio. f() g() if lim g() = 1. Simplified: Tilde approximatio of a fuctio ca be thought of as the highest order term with its coefficiet. Problems (Big-Oh) Problem 1 Problem 1 a Provide a ruig time aalysis of the followig loop. That is, fid both Big-Oh ad Big Ω: for ( it i = 0 ; i < ; i ++) for ( it j = i ; j <= ; j++) for ( it k = i ; k <= j ; k++) sum++;
Observe that for fixed values of i, j, the iermost loop rus max{1, j i + 1} times. For istace, whe i = j = 0, the iermost loop evaluates oce. Whe i = 0, j =, the iermost loop evaluates + 1 times. The middle loop rus a total umber of O ( ( i) ) times. Therefore, the etire block of code rus i O( 3 ). To fid a lower boud o the ruig time, we cosider smaller subsets of values for i, j ad lower-boud the ruig time for the algorithm o these subsets. (A lower boud there would also be a lower boud for the origial code s ruig time!) Cosider the values of i such that 0 i /4 ad values of j such that 3/4 j. For each of the /16 possible combiatios of these values of i ad j, the iermost loop rus at least / times. Therefore, the ruig time is at least Ω Alterate Solutio for Big-Oh. ( 16 ), or equivaletly, Ω( 3 ). Oe could solve this usig exact sums, but we will leverage some Big-Oh otatio. We kow that the iermost loop rus i at most (j i + 1) time for fixed i, j (see other solutio). Therefore the body of the middle loop rus at most c(j i + 1) times. Therefore, we ca express the ruig time of the code show as c(j i + 1) = O( 3 ) i=0 j=i Problem 1 b Provide a ruig time aalysis of the followig loop: for ( it i = ; i < ; i = i i ) for ( it j = 1 ; j < Math. s q r t ( i ) ; j = j+j ) System. out. p r i t l ( ) ; Let x ad y be the umbers of iteratios that the outer ad ier loops ru. Thus, we ca represet the ru time as a summatio:?? 1 (1) y=1 First, solve for the umber of iteratios of the outer loop (x) (i terms of ): x x lg lg () The, solve for the umber of iteratios of the ier loop (y) (i terms of i): Plug i the upper boud of x, y ito the recurrece: y i y lg i (3) lg lg i y=1 1 (4) Note that i is still o the top of the ier summatio, ow we wat to represet i i terms of x, which is the curret variable o the outer loop: i = x lg lg x y=1 1 (5) 3
Simplify the ier summatio: (lg x ) = lg lg lg Use the formula i=0 i = +1 1 to solve x lg 1 x lg (lg ) = T () = Θ( lg lg ) = Θ(lg ) lg 1 lg x = Θ( x ) (6) Problem You are give the followig algorithm for Bubble-Sort: Algorithm 1 Bubble Sort fuctio Bubble-Sort(A, ) for i 0 to do for j 0 to i do if A[j] > A[j + 1] the swap(a[j], A[j + 1]) ed if ed for ed for ed fuctio Give some sequece a 1, a,..., a i A, we say a iversio has occurred if a j < a i for some i < j. At each iteratio, Bubble-Sort checks the array A for a iversio ad performs a swap if it fids oe. How may swaps does Bubble-Sort perform i the worst-case ad i the average-case? It should be fairly obvious that the worst-case sceario for bubble-sort occurs whe A cotais elemets i reverse-sorted order. Let I deote the umber of iversios. I this situatio, the total umber of iversios is the exact umber of possible pairs of elemets, as each swap removes exactly oe iversio: ( ) I worst = I the average-case sceario, we determie the expected umber of iversios i a radom array of elemets. Specifically, we cosider a radom permutatio of distict elemets, a 1, a,..., a. Let X ij deote a idicator R.V. such that, { 1 if a i, a j iverted X ij = 0 otherwise 4
Thus, we have I average = E[I] = i,j : i<j i,j : i<j = i,j : i<j ( ) X ij = ( 1) = 4 E[X ij ] Pr[X ij = 1] I the above, we let Pr[X ij = 1] = 1 because i a radom permutatio (uiform probability), the probability of a i < a j is the same as a i > a j. Problem 3 Prove or disprove the followig statemet: lg(!) is Θ( lg ). We first show lg(!) is O( lg ). Pickig c = 1 ad 0 = 1, we have lg(!) = lg i lg i=1 This is clearly true for all > 0. Therefore, we are doe. We the show that lg(!) is Ω( lg ). Our strategy is to fid a easier to work with lower-boud for lg! that is larger tha some c lg. lg! = lg 1 + lg + + lg lg + lg( + 1) + + lg delete the first half of the terms lg replace remaiig terms by smallest oe Choosig c = 1 4 ad N = 4, it is clear that lg 4 lg with some algebraic maipulatio: Therefore, lg(!) is Ω( lg ). lg 4 lg lg 4 lg lg lg 5
Problem 4 Prove that a fuctio f() (which is evetually positive for all p ) is polyomially bouded iff log(f()) O(log ). I other words: where k is some costat. f() O( k ) log(f()) O(log ) ( ) Suppose f() O( k ) for some k. The c, 0 such that f() c k for all 0. Sice f() is evetually positive for all p, cosider all max( 0, p ). Take the log (a mootoically icreasig fuctio for positive values) of both sides, ad we get that: log(f()) log(c k ) log(c) + log( k ) log(c) + k log() O(log ) ( ) Suppose log(f()) O(log ). Note here that logarithm, o matter which base, is all i O(log ). WLOG, we ca assume here that the base is e, ad these are atural logs. The c, 0 such that l(f()) c l for all 0. Expoetiatig both sides (a mootoically icreasig fuctio), we get that: which is i O( k ) for k c. Problem 5 Prove or disprove the followig: 1. lg! is polyomially bouded.. lg lg! is polyomially bouded. e l(f()) e c l f() e l(c ) c You ca use the results from problem 3 ad 4 to solve this questio. 1. Let g() = lg!, ad let y = lg. The lg g() = lg y! by the result from problem 3. Substitutig y back i, Θ(y lg y) (y lg y) = ( lg )(lg lg ) / O(lg ), sice lg lg will evetually exceed ay costat c. (So c ( lg )(lg lg ) c lg.) Also if you calculate the limit ( lg )(lg lg ) lim = lim lg lg =, lg 6
it shows that ( lg )(lg lg ) = ω(lg ). So clearly ( lg )(lg lg ) / O(lg ). By the if ad oly if statemet from problem 4, sice lg g() / O(lg ), we ca also say that g() is ot polyomially bouded.. Let h() = lg lg!, ad let y = lg lg. Similar to part 1, Substitutig y back i, Or if you calculate the limit lg h() = lg y! Θ(y lg y), (y lg y) = ( lg lg )(lg lg lg ) = O( lg lg lg lg ) = O((lg lg ) ) = O( lg lg ) = O(lg ) ( lg lg )(lg lg lg ) lim = 0. lg This shows that ( lg lg )(log lg lg ) = o(lg ) ( lg lg )(lg lg lg ) O(lg ). By the if ad oly if statemet from problem 4, sice lg h() O(lg ), we ca also say that h() is polyomially bouded. 7