GC05: Algorithmic Complexity & Computability

Transcription

1 GC05: Algorithmic Complexity & Computability This part of the course deals with assessig the time-demad of algorithmic procedures with the aim, where possible, of fidig efficiet solutios to problems. We will ot be cosiderig issues related to demads o memory space; for those iterested, these are dealt with for example oe or other of the refereces below. Complexity: Computability: How cheaply ca it be doe? Ca it be doe i ay reasoable time? Ca it be doe at all? Backgroud readig The material i these books is supplemetary to the otes the books are ot essetial for this part of the GC05 course. ALGORITHMICS: The Spirit of Computig - David Harel (Addiso Wesley) A very readable itroductio to the subject, coverig most of the areas dealt with i these lectures ad also may further highly recommeded. INTRODUCTION TO THE DESIGN AND ANALYSIS OF ALGORITHMS - Aay Leviti (Pearso Iteratioal) Clear ad well writte ad about the right level. This course does't follow the book closely but uses a similar style of pseudocode ad some of its examples. ALGORITHMS: Theory ad Practice - Giles Brassard ad Paul Bratley (Pretice-Hall) Detailed mathematical treatmet, goes much further tha this course recommeded if you fid the material iterestig ad wat to lear more. COMPGC05: Part 2.1 1

2 1. INTRODUCTION What is a algorithm? A algorithm is a procedure composed of a sequece of welldefied steps, specified either i a atural laguage (a recipe ca be regarded as a algorithm), or i appropriate code or pseudocode. I these lectures algorithms will be preseted i a simplified pseudocode. A algorithm is able to take a legal iput eg for multiplicatio a pair of umbers is legal, but a pair of text files is ot carry out the specified sequece of steps, ad deliver a output. Algorithms are procedural solutios to problems. Problems to which algorithmic solutios may be sought fall ito four basic classes: Those that admit solutios that ru i reasoable time the class of tractable problems (eg sortig ad searchig). Those that probably do t have reasoable-time algorithmic solutios (eg the Travellig Salesma problem). Those that defiitely do t have such solutios (eg the Towers of Haoi). Those that ca t be solved algorithmically at all the class of o-computable problems (eg the haltig problem). The last three of these will be the subject of Sectio 2 of these lectures. The first part of the course, Sectio 1, will deal with methods for evaluatig the time-complexity of reasoable time algorithms for those tractable problems which do admit such solutios. COMPGC05: Part 2.1 2

3 A tractable problem may however also have a algorithmic solutio which does ot have reasoable time demads. Ofte such a algorithm comes directly from the defiitio of the problem a aïve algorithm but a more subtle approach will yield a more practically useful solutio. Here s a example of a problem which is tractable, but which has also has a aive algorithm that has a very fast growig timedemad that makes this algorithm useless for all but very small iput istaces. Example: evaluatig determiats A determiat is a umber that ca be associated with a square matrix, used for example i the calculatio of the iverse of a matrix. It has a recursive defiitio such that det(m), for a x ( rows ad colums) matrix, is a weighted sum of determiats of (-1)x(-1) sub-matrices. These i tur ca be expressed as the weighted sums of the determiats of (-1) determiats of (-2)x(-2) matrices, ad so o. " det(m) = (!1) j+1 a 1j det(m () [1,j]), >1 j=1 = a 11, =1 (M () [i,j] is the (-1) (-1) matrix formed by deletig the ith row ad jth colum of the matrix M.) This defiitio ca be used as the basis for a simple algorithm, referred to here as the recursive algorithm. Usig it for example for a 3x3 matrix is easy; the calculatio takes oly a few miutes. For a 4x4 or 5x5 it begis to get messy ad time-cosumig... COMPGC05: Part 2.1 3

4 ...but for a 10x10 matrix, forget it. Uless you are exceptioally patiet you would ot wat to do this by had. Ad eve usig a computer it takes a sigificat amout of time. What is't immediately apparet from the 'to calculate the quatity for a istace of size, calculate it for istaces of sizes -1' recursive defiitio is how much work this implies. The recursive algorithm takes a time i the order of!, writte O(!), where! = (-1) (-2) (later i the course we will show this). ('I O(...)' meas 'roughly like (...)' at a ituitive level later we will formalise the defiitio.)! is a extremely fast-growig fuctio, makig it ufeasible to use the recursive algorithm for all but very small matrices ( 5). However there is a alterative algorithm for evaluatig a determiat, based o Gaussia elimiatio, that oly takes time i O( 3 ). The differece betwee the time-demad of the two algorithms as the iput size grows is startlig: size of matrix recursive algorithm Gaussia elimiatio 5 x 5 20 secs 10 x miutes 0.01 secs 20 x 20 >10 millio years 100 x 100!! 5.5 secs COMPGC05: Part 2.1 4

5 Two ways to approach aalysis of algorithms: Empirical: repeatedly ru algorithm with differet iputs get some idea of behaviour o differet sizes of iput ca we be sure we have tested the algorithm o a sufficietly wide rage of iputs? this cosumes the very resource (time) we are tryig to coserve! Theoretical: aalysis of a paper versio of the algorithm ca deal with all cases (eve impractically large iput istaces) machie-idepedet The aim is to obtai some measure of how the time demad of a algorithm grows with the size of its iputs, ad to express the result i a simplified way, usig order otatio, so that the implicatios ca be more easily visualized. Time complexity fuctio Problem size log log x x10 30 > >10 100! 3x10 6 > > > COMPGC05: Part 2.1 5

6 Measurig size of a istace Formally, the size x of a iput istace x is the umber of bits eeded to ecode it, usig some easily-decoded format. e.g. for multiplyig 2 umbers x & y x=2, y=4 x y smaller umber padded with leadig zeros Size of iput = 2 x max ( 1+! " log 2 x# $, 1+! log y # " 2 $ ) [ We will use the fuctios ceilig(x) =! " x # $ floor(x) =! " x # $ = smallest iteger x, = largest iteger x ] But ormally a much more iformal defiitio is used which depeds o the cotext, eg problem sortig calculatig a determiat fidig a miimal spaig tree size of a iput istace umber of items to be sorted umber of rows ad colums i matrix umber of odes i the graph COMPGC05: Part 2.1 6

7 Measurig time take The objective is to make the time-cost aalysis machieidepedet. The differece betwee ruig the same algorithm o two differet machies is oly goig to be some costat factor (eg. this machie is twice as fast as that oe ) which is the same for all iput sizes. The kid of differece that really couts is the sort that itself icreases with size the differece betwee log ad 2, or betwee 3 ad!. A machie-idepedet measure of time is give by coutig elemetary operatios. These are simple operatios used as primitives by all the cadidate algorithms for example whe we say that the cost of a sortig algorithm grows like 2 we will usually be coutig the umber of comparisos doe as a fuctio of, the umber of thigs to be sorted. Other operatios that ca be used as elemetary time-couters are Boolea operatios (AND, OR, etc.), assigmets, ad mathematical operatios such as additio, subtractio, multiplicatio ad divisio. Elemetary operatios are cosidered to themselves be of egligible cost, ad are sometimes for simplicity referred to as beig of uit cost or takig uit time. Note: operatios which are primitive ad cosidered to be of trivial itrisic cost so that they ca be used as time-couters i some cotexts may ot be so lightly dismissed i others. (For example multiplyig 2 umbers ca almost always be thought of as a elemetary operatio, but there are some applicatios, such as cryptology, usig very large umbers (100s or 1000s of decimal digits), where the cost of multiplicatio is ot trivial! The multiplicatio itself eeds to be broke dow ito simpler operatios (sigle-bit oes) ad better algorithms (like Strasse s algorithm) looked for.) COMPGC05: Part 2.1 7

8 Forms of time-complexity aalysis Worst case This is the easiest form of aalysis ad provides a upper boud o the efficiecy of a algorithm (appropriate whe it is ecessary to esure a system will respod fast eough uder all possible circumstaces- eg. cotrollig a uclear power plat) Average case There may be situatios where we are prepared to put up with bad performace o a small proportio of iputs if the average performace is favourable. What does average performace mea? Either sum the times required for every istace of a particular size, divide by the umber of istaces, or evaluate performace with respect to a average istace. (For a sortig algorithm this might well be a radomly ordered file- but what is a average istace for a program processig Eglish-laguage text?) Average case aalysis is mathematically much more difficultmay algorithms exist for which o such aalysis has bee possible. Best case This kid of aalysis differs from the other two i that we cosider ot the algorithm but the problem itself. It should really be referred to as best worst case aalysis, because we aim to arrive at bouds o the performace of all possible algorithmic solutios, assumig their worst cases. Best case aalysis is based o a cosideratio of the logical demads of the problem at had what is the very miimum that ay algorithm to solve this problem would eed to do, i the worst case, for a iput of size? COMPGC05: Part 2.1 8

9 Example: Cosider the multiplicatio of two -bit umbers. Ay algorithm to solve this problem must at least look at each bit of each umber, i the worst case sice otherwise we would be assumig that the product could i geeral be idepedet of some of the 2 bits ad so we ca coclude that multiplicatio is bouded below by a liear fuctio of. Order Notatio The result of a time-complexity aalysis may be some log ad complicated fuctio which describes the way that time-demad grows with iput size. What we really wat to kow is how, roughly, these time-demad fuctios behave like? log? 3? The objective of usig order otatio is to simplify results of complexity aalysis so that the overall shape ad i particular, the behaviour as (asymptotic behaviour) of the timedemad fuctios are more clearly apparet. O-otatio O ca provide a upper boud to time-demad i either worst or average cases. Ituitively, f(x) is O(g(x)) meas that f(x) grows o faster tha g(x) as x gets larger. Formally, The positive-valued fuctio f(x) O(g(x)) if ad oly if there is a value x 0 ad a costat c>0 such that for all x! x 0, f(x) " c.g(x) (Note: the restrictio i the defiitio here for simplicity that f(x) be positive-valued is t likely to cause problems i algorithmic applicatios sice fuctios will represet work doe ad so will always retur positive values i practice.) COMPGC05: Part 2.1 9

10 Useful properties of O 1. O( k.f() ) = O( f() ), for ay costat k This is because multiplicatio by a costat just correspods to a re-adjustmet of the value of the arbitrary costat k i the defiitio of O. This meas that uder O-otatio, we ca forget costat factors (though these hidde costats might be importat i practice, the do t chage the order of the result). Note that as a cosequece of this, sice log a = log a b log b there is o effective differece betwee logarithmic bases uder O-otatio; covetioally we just use O(log ), forgettig the (irrelevat) base. COMPGC05: Part

11 2. O( f() + g() ) = O( max( f(),g() ) ) (for those iterested the proof is o p.56 of Leviti) max here is a shorthad way of sayig the part that grows the fastest as. This result eables us to simplify the result of a complexity aalysis, for example O( + ( )) = O(max(, )) = O( ) 3. O( f() ) U O( g() ) = O( f() + g() ) (ot so easy to prove!) = O( max(f(),g()) ), by 2. above. This last meas that where a algorithm cosists of a sequece of procedures, of differet time-complexities, the overall complexity is just that of the most time-demadig part. Examples of proofs usig O-otatio [Note: You ca assume i all such proofs that >0, as i this course will represet size of a iput.] For example, is it true that (i) 2 3 O( )? (ii) 3 2 O( )? The geeral way to proceed is as follows: Assume the assertio is true. Work from the defiitio of O, ad try to fid suitable values of c ad o If you ca fid ay pair of values (there s o uique pair) the assertio is, i fact, true. If there is some fudametal reaso why o pair c, 0 could be foud, the the origial hypothesis was wrog ad the assertio is false. COMPGC05: Part

12 ( i) Is 2 3 O( )? Assume it is. The 2! c 3 " 0, for all # 0 $ 2 (1! c) " 0, for all # 0 $ c # 1, for all # 0 $ # 1 c, for all # 0 Choosig (for example) c=2, 0 =1 is satisfactory ad so it s TRUE that 2 3 O( ). (ii) Is 3 2 O( )? Agai, assume it is. The 3! c 2 " 0, for all # 0 $ 2 (! c) " 0, for all # 0 $! c " 0, for all # 0 But c has to have a fixed value. There is o way to satisfy c, for all 0 for a fixed c. Hece the origial assumptio was FALSE, ad 3 2 O( ). Notes Whe aswerig the questio Is f() O(g())? it is ot sufficiet to draw a picture showig the curves f() ad g() -- that ca illustrate your argumet, but is t i itself a proof, as the questio is about what happes as, so ca t be resolved by lookig at ay fiite rage of. If you are asked to base a proof o 'the formal defiitio of O-otatio' do't base your argumet o the three properties listed o pp Argue from the defiitio of O-otatio, as above. COMPGC05: Part

13 Hierarchies of complexity Let be the size of a iput istace, i the usual iformal defiitio (eg degree of a polyomial, legth of a file to be sorted or searched, umber of odes i a graph). Complexity O(1) O(log ) O() O( log ) Costat time: all istructios are executed a fixed umber of times, regardless of the size of the iput. Example: takig the head of a list. Logarithmic: program gets oly slightly slower as grows (typically by usig some trasformatio that progressively cuts dow the size of the problem). Example: biary search. Liear: a costat amout of processig is doe o each iput elemet. Example: searchig a uordered list. Typical of divide ad coquer algorithms, where the problem is solved by breakig it up ito smaller subproblems, solvig them idepedetly, the combiig the solutios. Example: quicksort. O( k ) Polyomial: most ofte arises from the presece of k ested loops (examples: isertiosort (k=2); Gaussia elimiatio method for a calculatig a determiat (k=3). O(a ) Expoetial: very fast-growig (assumig a>1), essetially uusable for all but very small istaces. Example: Towers of Haoi (a=2). O (!) Factorial: eve worse! Example: recursive evaluatio of a determiat. Oly algorithms ruig i polyomial time (those which are i O( k ) for some k) are effectively usable; oly problems which admit such algorithms are effectively soluble (tractable). Thus fidig a determiat is soluble i reasoable time because it has a good algorithm ruig i O( 3 ) as well as a uusable O(!) oe, but the Towers of Haoi puzzle is t, because it ca be demostrated that there are o good algorithms possible i this case. (More about such itractable problems later.) COMPGC05: Part

14 2. ANALYSIS OF NONRECURSIVE ALGORITHMS There is ot a clear set of rules by which algorithms ca be aalysed. There are, however, a umber of techiques which appear agai ad agai. The best way to lear about these is really though examples. Algorithms cosist of sequeces of procedural steps which may themselves ivolve loops of fixed ( for... ) or idetermiate ( while, etc) legth, or of recursive fuctio calls (see later). We will start with the simplest cases: SEQUENTIAL OPERATIONS Step i O( f() ) Step (i+1) O( g() ) The combiatio of the ith ad (i+1)st steps takes a time i O( f() ) U O( g() ). Use O( f() ) U O( g() ) = O( max( f(),g() ) ) to justify eglectig all but the most time-costly step. COMPGC05: Part

15 Example: multiplicatio (i) Shift-ad-add multiplicatio ("log multiplicatio") x partial values bits i each umber logest is 2-1 bits result is 2 bits i worst case (1) Compute partial values, each requirig sigle-bit multiplicatios (2) Add the partial values (estimate as (-1) additios of pairs of (2-1)-bit umbers (upper boud)) Complexity (sigle-bit operatios) Step(1) 2 Step(2) (2-1)(-1) = Total O( 2 ) =5 example (19) (11) x5=25 sigle bit multiplicatios havig added top two rows (9 additios) havig added top two rows (9 additios) havig added top two rows (9 additios) havig added top two rows (9 additios) Total umber of sigle bit operatios = = 61 (= 3x5 2-3x5+1) As = = 0x256+1x128+1x64+0x32+1x16+0x8+0x4+0x2+1x1 = 209 COMPGC05: Part

16 (ii) A la russe method The idea is to start with two colums (where / i the first colum meas iteger divisio, ie droppig the remaider): a b a/2 2b a/4 4b i i i i 1 i Create a third colum, cotaiig a copy of the umber from the secod colum everywhere the umber i the first colum is odd. Add up this third colum to get the result. eg =209 There are O() etries i the colums, each ivolvig work O(1), sice each etry is made by either a right-shift (left colum) or by addig a zero (right colum). Addig the third colum is O( 2 ). So à la russe is also O( 2 ) overall but it s slightly faster tha shift-ad-add multiplicatio because it is still oly O() before the additio stage. Lower boud o the time-complexity of multiplicatio We argued earlier that every algorithm for multiplyig two -bit umbers will require at least 2 sigle-bit operatios, so has a best worst case i O(). There is thus scope for algorithms whose performace improves o that of the simple O( 2 ) oes above ad which have a worst-case performace somewhere i betwee O() ad O( 2 ), the best kow of which, Strasse s algorithm, has a worst-case performace i O( ). COMPGC05: Part

17 Sums of series Before movig o to look at algorithms with loops ad recursio, we eed to kow how to evaluate sums of series. Arithmetic series The otatio meas b i= a f (i) f(a)+f(a+1)+ +f(b). Note we ca use the otatio eve whe f is idepedet of i: b! c meas (c+c+ +c) = c(b-a+1) i=a!!! b-a+1 times The simplest ad most useful case is whe f(i) = i. I this case it is easy to derive a formula (a closed form which does ot have the summatio symbol) for f (i): b i= a 2 i= (-1) + i= (-1) =(+1)+ (+1)+ + (+1)+(+1) = (+1) copies i= 1 i= ( + 1) 2 COMPGC05: Part

18 It s sometimes the case that the sum to be evaluated does t start with i=1: i= i= j j 1 i - i i= 1 i= 1 ( j 1) = ( + 1) - j 2 2 Geometric series This is the other type of simple series summatio which is very useful i algorithmics. Let S() = a + a 2 + a a i = a a.s() = a 2 + a 3 + a a +1 = S() a + a +1 So i a(1 a ) a = for a 1 i= 1 1 a Note that the formula works oly for a 1 - if a=1 get sum 0/0, which gives a udefied value. The calculatio has to be doe differetly i this case: If a = 1: i a = i= 1 i= 1 1 = ( ) = terms i= 1 COMPGC05: Part

19 ALGORITHMS WITH LOOPS I the simplest cases where the loop is executed a fixed umber of times (a for loop) the complexity is just the cost of oe pass through the loop multiplied by the umber of iteratios. ALGORITHM Sum( A[0..-1] ) // Outputs the sum of the elemets i A[0..-1] sum < 0 for i < 0 to -1 do sum < sum + A[i] retur sum There is oly oe choice here for the elemetary operatio, additio. There are additios to 'sum' ad hece this takes time (measured by the umber of additios performed) i O(). Compare with this example from Leviti, p.61: ALGORITHM MaxElemet( A[0..-1] ) // Outputs the value of the largest elemet i A[0..-1] maxval < A[0] for i < 1 to -1 do if A[i] > maxval maxval < A[i] retur maxval I this case there are two operatios i the 'for' loop, compariso ad assigmet, that might be cadidates for the role of the elemetary operatio. However ote the assigmet is oly doe if the compariso returs true ad hece it is the compariso that gives the best measure of the worst case cost of the algorithm. As with the first example it's here easy to see the cost, i terms of the umber of comparisos, must be -1 ad hece this algorithms too is i O(). COMPGC05: Part

20 More formally-- Let C() be the cost of executig MaxElemet for a -elemet array. Coutig comparisos at uit cost!1 C() = " 1=!1 # O() i=1 * * * If there are several ested loops of this type the complexity is the cost of oe pass through the iermost loop multiplied by the total umber of iteratios. However for ay give loop the amout work doe may deped o the outer loop it is embedded i: for i < 1 to do for j < 1 to i do // somethig at uit cost The cost of the work here is i C() =!! 1=! i=1 j=1 i=1 (1 added to itself i times) =! i = (+1) (usig familiar formula for sum-of-i) 2 i=1 If you are asked to "simplify your aswer usig O-otatio" you are beig ivited to use the three rules o pp You ca use them iformally, you do't eed to quote the rules but you should bear them i mid ad make sure of your reasoig.! I this case sice O# " 2 (+1) $ & = O( max( 2 /2, /2 ) ) % = O( 2 /2) we just keep the leadig term, ad sice O(kf()) = O(f()) the 1/2 ca be dropped to give the work agai as O( 2 ). (I this case of questios askig you to "prove that f() is i the order of g()" the word "prove" implies you eed to use the formal defiitio of 'O' o p.9 ad oly this -- or aother equally formally structured mathematical argumet -- costitutes a full aswer.) COMPGC05: Part

21 A last loop example from Leviti (p.63): ALGORITHM UiqueElemets( A[0..-1] ) // Returs true if all elemets i A are distict, false otherwise for i < 0 to -2 do for j < i+1 to -1 do if A[i]=A[j] retur false retur true There is oly oe cadidate for elemetary operatio, the test 'A[i]=A[j]?' There are two worst case situatios: where the array cotais distict elemets (all passes through the ier loop are executed, with the coditioal evaluatig true every time); ad where oly the last two elemets A[-2], A[-1] are the same (as above but returs false o the very last test). I either of these situatios the cost of the work i terms of the umber of array elemet comparisos is!2!1 C() = "" 1= "(!1! (i +1) +1) = "(!1! i) i=0 j=i+1!2!2 i=0 = "(!1)!" i i=0!2!2 i=0 = (!1)" 1! " i i=0 (-1) is a costat w.r.t. the sum!2 i=1!2 i=0 the cotributio from i=0 is zero, so sum from i=1 for coveiece = (!1) 2! 1 (! 2)(!1) 2 use the sum-over-i formula with the upper limit adjusted = 2 (!1) " C() # O(2 ) ote this is the same as the umber of distict pairs should all array elemets be differet COMPGC05: Part