Proceedigs of the 10th WSEAS Iteratioal Coferece o APPLIED MATHEMATICS, Dallas, Texas, USA, November 1-3, 2006 316 A Geeralized Set Theoretic Approach for Time ad Space Complexity Aalysis of Algorithms ad Fuctios AHMED TAREK Easter Ketucky Uiversity Departmet of Computer Sciece 407, Wallace Buildig, 521 Lacaster Aveue, Richmod, Ketucky UNITED STATES OF AMERICA Abstract: There exists a variety of techiques for aalyzig the computatioal complexity of algorithms ad fuctios. This aalysis is critical i fidig out the upper ad the lower bouds o time ad space requiremets usig the big-oh ad the big-omega otatios. Besides these, there are other complexity criteria, such as, small omega ad small o complexities, which are also useful. Complexity aalysis is used i selectig a appropriate algorithm for solvig a give problem usig computer. Ufortuately, most of the existig techiques are complex, obsolete ad hard to use i practice. Besides, there is a tred to abuse otatioal complexities by cosiderig them as fuctios. However, otatioal complexities are sets of fuctios rather tha bare fuctios. I this paper, it has bee established that the otatioal complexities are sets of fuctios that iclude all algorithms ad fuctios o the give order satisfyig certai costraits. Moreover, startig from the scratch, we show how to determie the time ad space complexity fuctios for a give computatioal algorithm. We cosider applicatio of the proposed framework i determiig the otatioal complexities. The proposed framework may be exteded for fuctios ivolvig multiple variables. Also the space-time badwidth product has bee discussed i decidig the ecoomy of usig the computatioal algorithms. Key Words: Time complexity, Space complexity, Complexity fuctio, Complexity order, Asymptotic aalysis, Performace aalysis, Performace measuremet, Space-time badwidth product. 1 Itroductio Computatio time ad memory space requiremets are the major decidig factors i computerized implemetatio of computatioal algorithms. The performace of a algorithm or a fuctio depeds o iteral ad exteral factors. The iteral factors are related to the algorithm s efficiecy i terms of time ad memory space usage. There are a umber of exteral factors, which vary from platform to platform ad from software to software. Computatioal complexity of algorithms are depedet o iteral factors. We will cosider oly the time ad space complexity of algorithms ad fuctios. I geeral, time is more crucial tha space requiremet. We do ot determie the actual umber of CPU cycles or eve wat to cout every istructio executed i aalyzig algorithms. The big-o otatioal complexity is itroduced for this. Big-O complexity is idepedet of the hardware ad software used. Hece, it is also called the big-o estimate. It shows the growth of time ad space with the icreasig problem size without takig care of the costat coefficiets or smaller order terms. Sometimes, we ca prove that we caot compute somethig without a sufficiet amout of time or space. This does ot mea that we kow how to compute it i this lower boud. This provides us the Ω otatioal complexity. Similarly, ω ad small o complexities are used without tight lower ad upper bouds. To determie whether a fuctio lies at the set itersectio of big-o ad Ω complexities, the Θ otatioal complexity is used. All the otatioal complexities are idepedet of hardware ad software used for the implemetatio. For estimatig the space-time trade-off i implemetig a algorithm, we have itroduced the otio of cost fuctio, C(). The fuctio C() is sometimes called the space-time badwidth product, which is expressed as the product betwee g 1 () ad g 2 (), where the big-oh time complexity is O(g 1 ()) ad the space complexity is O(g 2 ()). Sectio 2 itroduces termiology ad otatio used throughout this paper. Sectio 3 explais the eed for asymptotic complexity i complexity aalysis. Differet otatioal complexities, their defiitios ad examples are also cosidered. The relatioship amog the otatioal complexities are discussed. Sec-
Proceedigs of the 10th WSEAS Iteratioal Coferece o APPLIED MATHEMATICS, Dallas, Texas, USA, November 1-3, 2006 317 tio 4 treats the set theoretic properties related to complexity aalysis. Sectio 5 discusses complexity criteria, ad itroduces the complexity class hierarchy usig graphs ad charts. Sectio 6 elaborates o guidelies i determiig the complexity fuctios. Sectio 7 cosiders a geeral framework i estimatig the time ad space complexities. The framework is exteded for multi-variable fuctios ad fractioal expressios, which may also be applied to the additio, ad multiplicatio of fuctios. Sectio 8 cosiders performace aalysis i space complexity. We have also cosidered the space-time badwidth product with examples. Sectio 9 discusses performace measuremet i time complexity. Sectio 10 is applicatio of space ad time complexity i combiatorial explosio ad AI. 2 Termiology ad Notatio I this paper, followig otatios are used to discuss the proposed framework ad applicatios. : Iput size. g(): Highest ordered term i modified complexity fuctio. f(): Complexity fuctio for a problem of size. f( 1, 2,..., m ): Complexity fuctio ivolvig m differet variables. Here, m = 1, 2,.... f(): Modified complexity fuctio obtaied by elimiatig the costat coefficiets from the complexity fuctio f(). f( 1, 2,..., m ): The modified complexity fuctio obtaied by removig the costat coefficiets from the origial fuctio f( 1, 2,..., m ). g( 1, 2,..., m ): The highest order term i the expressio for f( 1, 2,..., m ). O(g()): Big-oh complexity for a algorithm or a fuctio with a problem of size. O(g( 1, 2,..., m )): Big-oh complexity for a algorithm or a fuctio ivolvig m differet parameters. Here, m = 1, 2,.... Ω(g()): Big-omega complexity for a algorithm or a fuctio with a problem of size. Θ(g()): Cap-theta complexity for a algorithm or a fuctio with iput size. o(g()): Small o complexity for a algorithm or a fuctio with a problem of size. ω(g()): Small omega complexity for a algorithm or a fuctio with a problem of size. C(): Space-time badwidth product ivolvig the highest order terms i the time ad space complexity fuctios without coefficiets for a iput of size. 3 Asymptotic Complexity The eed for asymptotic complexity may best be explaied usig a example. Cosider the complexity fuctio, f()=3 2 + 14 + 27. The costat coefficiets i this fuctio may reflect implemetatio details. The lower order terms become isigificat with icreasig iput size as show i Table 1. Table 1: Effect of the highest order term with icreasig problem size. 3 2 14 + 17 1 3 31 10 300 157 100 30,000 1,417 1000 3,000,000 14,017 100000 300,000,000 140,017 As the iput size grows bigger ad bigger, the algorithmic performace coverges to the highest order term. Followig table verifies this fact. Table 2: As the iput size becomes larger ad larger, the umerator coverges to the highest order term, which is same as the deomiator, ad the ratio coverges to 1 asymptotically. 2 2 + 5 2 3 2 2 2 2 + 5 2 3 2 0 0-3 2 1 0.5 0 0 3 4.5 9 2 5 12.5 22 1.76 10 50 72 1.44 100 5000 5247 1.0494 1000 500,000 502,493 1.00495 10,000 50,000,000 50,025,997 1.00052 Hece, the complexity fuctio asymptotically approaches the highest order term as the problem size grows. Followig are the asymptotic defiitios of O ad Ω otatioal complexities. A complexity fuctio f() is O(g()) if there exist positive costats c ad 0 such that f() c g() for all 0. O(g()) is a set of fuctios, which ca be formally represeted as follows: O(g()) = {f() positive costats c ad 0 such that f() c g() 0.} (1)
Proceedigs of the 10th WSEAS Iteratioal Coferece o APPLIED MATHEMATICS, Dallas, Texas, USA, November 1-3, 2006 318 A more precise ad exact defiitio of the O- otatioal complexity follows [3]: Defiitio 1 If f ad g are fuctios o the size of a give problem, or the set of parameters 1, 2,..., m, m = 1, 2,... ivolved, the f() is O(g()), or f( 1, 2,..., m ) is O(g( 1, 2,..., m )) provided there exists costats c, ad 0 such that: f() c g(), wheever 0 or f( 1, 2,..., m ) c g( 1, 2,..., m ), wheever each of 1, 2,..., m 0. Here, 0 represets the threshold value for the big-oh otatioal aalysis to hold true. Similarly, Ω otatio is used to defie the asymptotic lower boud. A complexity fuctio f() is Ω (g()) if there exists positive costats c ad 0 such that 0 c g() f() for all 0. Ω (g()) is also a set of fuctios, for which, formal represetatio follows: Ω(g()) = {f() positive costats c ad 0 such that f() c g() 0.} (2) The Θ otatioal complexity represets a set of fuctios that lies at the itersectio of O(g()) ad Ω(g()). A complexity fuctio f() Θ(g()) provided that f() O(g()) ad also, f() Ω(g()). Formally, the Θ otatioal complexity may be defied as follows: Θ(g()) = {f() positive costats c 1, c 2, 0 such that c 1 g() f() c 2 g() 0.} (3) Usig set theoretic otatio: Θ(g()) = O(g()) Ω(g()). Example: Cosider the followig set of fuctios, which are o the order of O( 2 ). O( 2 ) = { 4lg() + 5, 5 + 7, 3lg() + 4, 8 2, 3 2 + 7, 9 2 + 6 } Next cosider the followig set of fuctios, for which, Ω ( 2 ) defies the lower boud. Ω( 2 ) = { 8 2, 3 2 + 7, 9 2 + 6 7 4, 32 + 7, 9 3 + 5 2 } Now Θ( 2 ) defies the set of fuctios that lies at the itersectio of O( 2 ) ad Ω( 2 ). Therefore, Θ( 2 ) = O( 2 ) Ω( 2 ) = {8 2, 3 2 + 7, 9 2 + 6}. We may defie o ad ω asymptotic complexities for those fuctios for which the upper ad lower bouds are ot tight. A fuctio f() is said to be o(g()), provided there exists positive costats c ad 0 such that for ay positive costat c ad for all iput sizes 0, the f() < c g() relatioship holds. Followig defies the set o(g()) formally usig the set theoretic otatios. o(g()) = {f() positive costats c ad 0 such that f() < c g(), 0 ad c.} (4) Sice the fuctio f() strictly satisfies f() < c g(), with the icreasig iput size, g() grows bigger ad bigger compared to f(), ad evetually, for large iput values, the asymptotic limit of the ratio betwee f() ad g() approaches 0. Stated mathematically, Lim ( )( f() g() ) = 0 (5) A fuctio f() is ω(g()), if there exists positive costats c ad 0 such that for positive costat c ad for all iput sizes 0, the relatioship f() > c g() holds true. Followig provides the formal defiitio of ω(g()). ω(g()) = {f() positive costats c ad 0 such that f() > c g(), 0 ad c.} (6) To be a member of the set ω(g(), f() has to be strictly greater tha the product c g(). Therefore, with icreasig values of, f() grows much bigger compared to the fuctio g(). Evetually, for very large values of, the ratio g() f() becomes egligible for the asymptotic case. Followig coditio holds true. Lim ( )( g() f() ) = 0 (7) Examples: The fuctio f() = 2 is O( 3 ), sice 2 1 3 + 1 2, ad the highest order fuctio becomes g() = 3. But there is o positive real costat c such that: 2 c 3. Therefore, 2 Ω( 3 ). Hece, 2 (O( 3 ) Ω( 3 )). Cosider aother fuctio f() = 3 + 4. Sice, 3 + 4 <= 7, hece for all 1, f() O(). Agai, f() = 3 + 4 3, for all 1. Hece, f() Ω(). Usig the defiitio of Θ(), f() Θ(). Now, 3 + 4 3 2 + 4 2 for all 1. Therefore, f() O( 2 ). Similarly, f() O( 3 ), f() O( 4 ), ad so. Therefore, O( 2 ), O( 3 ),... may always weake the asymptotic upper boud. Similarly, 3 + 4 3log() for all 1, ad f() Ω(log()). Also, 3 + 4 3 + 4 or 3+4 7 for all 1, ad f() Ω(1). Therefore, Ω(log()), Ω(1) may always weake the asymptotic lower boud. Give a positive costat c, 3 c for all but small itegers. Therefore, 3 Ω(). But it is impossible to fid a positive costat c such that for higher values of, 3 c. Hece, 3 O(). Give a positive c, log() c 7 for sufficietly
Proceedigs of the 10th WSEAS Iteratioal Coferece o APPLIED MATHEMATICS, Dallas, Texas, USA, November 1-3, 2006 319 large values of. Therefore, log() Ω( 7 ). However, log() O( 7 ). Therefore, log() (Ω( 7 ) O( 7 )). Next cosider f() = 3+16 for o() ad ω(). Now, 3 + 16 o(), ad also 3 + 16 ω(). For 3 + 16 to be o(), the relatioship 3 + 16 < c must hold for all positive costats c ad ay iput size 0. However, 3 + 16 < c fails for may positive costats c. Therefore, 3+16 o(). Similarly, 3 + 16 > c fails for may positive costats c. Hece, 3 + 16 ω(). However, 3 + 16 < c log() is always true beyod the cutoff iput size 0, ad for all real positive costats c. Therefore, f() = 3 + 16 o(log()). Similarly, 3+16 o( 2 ), ad so. Agai, 3+16 > c log() for all real positive costats c beyod a certai iput size 0. Hece, 3 + 16 ω(log()). Similarly, 3 + 16 ω(1). If we are posed with the problem of verifyig whether 1 log o(1), we may solve it usig the limitig approach. I this case, f() = 1 log, ad g() = 1. Thus, f() g() = 1 1 log = 1 log ad 1 log. As, log 0. By defiitio, (Lim )( f() g() ) eeds to be 0 for f() o(g()). Hece, 1 log o(1). 4 Set Theory i Complexity Aalysis Lemma 2 If positive costats c ad 0, such that 0, the a set of fuctios {f()} maps o to a set S(g()) of fuctios g() usig the set membership, where S {O, Ω, o, ω, Θ}, ad S deotes a otatioal complexity the otatio of which depeds o the relatio R i f() R c g(). Here, R is a relatio from the set {f()} to the set {c g()}, where R {, >,, <, =}, ad is a subset of the cartesia product betwee the sets {f()} ad {c g()}. Proof: We prove this lemma usig proof by cases. R is : f() c g(), ad f() Ω(g()). R is : Here, f() c g(), ad f() O(g()). R is >: I this case f() > c g(), ad f() ω(g()). R is <: I this case f() < c g(), ad f() o(g()). R is =: I this case f() = c g(), ad f() Θ(g()). I all five of the cases, f() S(g()), where the exact otio of S will deped o the relatio R betwee {f()} ad {c g()}. We already kow that O, Ω, o, ω ad Θ otatioal complexities are sets rather tha fuctios. However, it is a commo practice to abuse complexity otatios, such as writig f() = O(g()) istead of f() O(g()). Also, the expressios such as f() = h() + O(g()) are quite commo. But a fuctio caot be added to a set like this, ad i o way a fuctio could be equal to a set. Therefore, it is more appropriate to write: f() O(h() + g()). fuc() g() f() h() Figure 1: Notatioal complexities bear complimetary characteristics. From Figure 1, fuctio g() defies the upper boud for f() wheever 0. Therefore, f() O(g()). Stated i aother way, the fuctio f() defies the lower boud for the fuctio g(). Hece, g() Ω(f()). The figure also shows that the fuctio h() defies the lower boud of f(), ad f() Ω(h()). Therefore, the fuctio f() must defie the upper boud to the fuctio h() ad h() O(f()). Therefore, the otatioal complexity relatioship betwee a give fuctio ad its boudig fuctio is complimetary ad preserves reciprocity. Followig result is true. Theorem 3 If for a give fuctio f(), if f() O(g()), the g() Ω(f()), ad vice-versa. Proof: Sice f() O(g()), therefore, there exists positive costats c ad 0 such that for all 0, f() c g(). This meas, 1 c f() g(). 1 Deote the positive costat c by c 1. Therefore, g() c 1 f(). Hece, g() Ω(f()). 5 Complexity Criteria Followig hierarchical relatioship forms a basis for determiig the otatioal complexity order for fuctios ad expressios [3]. 1 < log b < < log b < 2 < 3 <... < k < 2 < c <!
Proceedigs of the 10th WSEAS Iteratioal Coferece o APPLIED MATHEMATICS, Dallas, Texas, USA, November 1-3, 2006 320 Here, k ad c are positive costats, ad b is a positive iteger defiig the base of the logarithmic fuctios. This hierarchical relatioship is kow as the complexity class. I terms of set theory: O(1) O(log b ) O() O(log b ) O( 2 ) O( 3 )... O( k ) O(2 ) O(c ) O(!) (8) The above equatio (8) holds true also for the o otatioal complexity class hierarchy. Similarly, the complexity class hierarchy for the Ω otatioal complexity becomes: Ω(1) Ω(log b ) Ω() Ω(log b ) Ω( 2 ) Ω( 3 )... Ω( k ) Ω(2 ) Ω(c ) Ω(!) (9) Equatio (9) is also applicable for the ω otatioal complexity class hierarchy. From these hierarchical relatioships, followig result follows. Θ(log b ) Θ() Θ(log b ) Θ( 2 ) Θ( 3 )... Θ( k ) Θ(2 ) Θ(c ) Θ(!) = {} Proceedig this way, it is possible to show that: Θ(c ) Θ(!) = {}, where { } deotes the empty set. Figure 2 illustrates complexity class hierarchy. Before 0, the hierarchy does ot hold. We discard iput values smaller tha 0. Beyod the iput size 0, the fuctios preserve the complexity class hierarchy. f() f() = 3 f() = 3 f() = 2 f() = log() Lemma 4 For Θ otatioal complexity, followig results hold true. Cosider iputs here f() = f() = log () Θ(1) Θ(log b ) Θ() Θ(log b ) Θ( 2 ) Θ( 3 )... Θ( k ) Θ(2 ) discard iputs here = 0 (cut-off iput size) Θ(c ) Θ(!) = {} Θ(log b ) Θ() Θ(log b ) Θ( 2 ) Θ( 3 )... Θ( k ) Θ(2 ) Here, { } deotes a empty set. Θ(c ) Θ(!) = {}. Θ(c ) Θ(!) = {} (10) Proof: Cosider a fuctio f 1 () with iput size. Suppose that f() Θ(1). I this case, f 1 () O(1) ad also, f 1 () Ω(1). Usig the defiitio of O-complexity, f 1 () O(log b ), f 1 () O(), ad so o. However, f 1 () Ω(1) ad f 1 () Ω(log b ), f 1 () Ω(), ad so o. These two relatioships hold true for ay such fuctio f(). Therefore, Θ(1) Θ(log b ) Θ() Θ(log b ) Θ( 2 ) Θ( 3 )... Θ( k ) Θ(2 ) Θ(c ) Θ(!) = {}. Next cosider aother fuctio f 2 () such that: f 2 () Θ(log b ). Therefore, f 2 () O(), f 2 () O(log b ), ad so o. But f 2 () Ω(), f 2 () Ω(log b ), ad so o. These two relatioships hold true for ay such fuctio f(). Hece, Figure 2: Graphical represetatio of fuctios i complexity class hierarchy. I literature, we fid three types of complexities. These are best case, worst case, ad average case complexities. Big-oh complexity defies a algorithm s upper-boud i computatio time ad memory space requiremets. The algorithm cosumes time or space i the order of big-oh i the worst possible case. Ω-otatioal complexity defies the order of the miimum time ad space required to execute the algorithm. This is possible i the best case. The average case complexity cosiders the effects of all possible cases icludig the best ad the worst cases as well. 6 Guidelies for Fuctios There are five commo guidelies i fidig out the complexity fuctio correspodig to a algorithm. We discuss each oe of them i sequece. 1. Loops: Loops play a major role i most codig structure. The maximum ruig time of a loop is the sigle ruig time of the statemets withi the loop icludig loop tests multiplied by the total umber of loop iteratios. For
Proceedigs of the 10th WSEAS Iteratioal Coferece o APPLIED MATHEMATICS, Dallas, Texas, USA, November 1-3, 2006 321 example, cosider the followig loop: for i = 1 to i step 1 do m = m + 2 Each executio of the loop takes a costat c amout of time. The loop executes for differet times. Therefore, the time complexity fuctio, f() = c. We eed oe memory uit for storig i, oe for ad oe for m. Therefore, altogether, we eed 3 memory uits or 6 bytes (fixed), ad f s () = 6. 2. Nested Loops: Loops withi the loop cotrol structure is quite commo i practice. For ested loops, we start at the iermost loop ad the aalyze iside out. Total ruig time is the product of the sizes of all the loops. As a example, cosider the followig: for i = 1 to i step 1 do for j = 1 to i step 1 do k = k + 1 There is a ier for loop ested withi the outer for loop. For each executio of the outer loop, the ier loop executes times. The outer loop executes times. Suppose that the assigmet statemet k = k + 1 takes a costat time c for its executio. Therefore, the time complexity fuctio f() = c = c 2. We eed 1 memory uit for i, oe for j, oe for, ad oe for k. Altogether we eed 4 memory uits or 8 bytes, ad f s () = 8. 3. Cosecutive Statemets: For cosecutive statemets, we fid out expressio for the total time i terms of the iput parameters for executig each idividual statemet, each loop ad each ested loop costructs. This gives us the time complexity fuctio. For example, cosider the followig statemets. p = p + 1; for i = 1 to i step 1 do m = m + 2 for i = 1 to i step 1 do for j = 1 to i step 1 do k = k + 1 I this example, the complexity fuctio f() = c 0 + c 1 + c 2 = c 0 + c 1 + c 2 2. Here, c 0 is the time required by the assigmet statemet: p = p + 1, c 1 is the time required by the statemet: m = m + 2, ad c 2 is the time cosumed by the statemet: k = k+1. However, we eed oly 1 memory uit for storig p, 1 for i, 1 for, 1 for m, 1 for j ad fially, 1 for storig k. Therefore, altogether we eed 6 iteger memory uits or 12 bytes. Here, f s () = 12. 4. If-the-else statemets: With If-the-else statemets, we are iterested i the worst-case time complexity fuctio. The worst-case total time is the time required by the test, plus either the the part or the else part time (whichever is the larger). As a example, cosider the followig code: if (x is equal to y) the retur false else { for (m = 0 to m < i step 1) do if (m is equal to y) the retur false ed if } ed if I this example, i the worst-case, both the if ad the else parts i the outer if-else structure will be executed. Let the time for the if test is c 0. Withi the else structure, the for loop will be executed differet times. If each test coditio of the for loop takes c 1 ad the if coditio check takes c 2 times, the the time complexity fuctio f() = c 0 + (c 1 +c 2 ). Here, we eed 2 memory uits or 4 bytes for storig x ad y, 2 bytes for keepig the retur address from the first if, 2 bytes for storig, 2 bytes for m, ad fially, 2 bytes for the retur address from the secod if statemet. Altogether, we eed (4+2+2+2+2) = 12 bytes of memory for storig this code segmet. I this case, f s () = 12. 5. Logarithmic complexity: A algorithm is of logarithmic complexity if it takes a costat time to cut dow the curret problem size by a costat fractio (usually by 1 2 ). A example is the biary search algorithm. Quite ofte we use the biary search for fidig a word iside a dictioary of pages. 7 A Framework for Aalysis Oce we have determied the time or the space complexity fuctio usig the guidelies described previ-
Proceedigs of the 10th WSEAS Iteratioal Coferece o APPLIED MATHEMATICS, Dallas, Texas, USA, November 1-3, 2006 322 ously, we may use the followig steps i determiig the big-oh otatioal complexity. Step 1: Expad the complexity fuctio f() completely for a iput of size. Step 2: Remove all costat coefficiets from the terms withi f(), ad obtai the modified complexity fuctio f(). If ecessary, expad ay ested fuctio withi the expressio for f(). Step 3: Fid out the highest ordered term i the expressio for f() ad express it as g(). If there are ested fuctios withi f(), the fial expressio for g() is obtaied by elimiatig the costat coefficiets ad after refiemet of the iitial expressio for g(). Step 4: The big-oh otatioal complexity is O(g()). Example: We apply the above steps to f() = a k k + a k 1 (k 1) +... + a 1 + a 0, here a k 0, k = 0, 1, 2,... ad a k, a k 1,..., a 0 are costat coefficiets. Applyig the secod procedural step ad removig all costat coefficiets from f(), the modified complexity fuctio, f() = k + (k 1) +... + 1, where k is a oegative iteger, ad k = 0, 1, 2,.... Usig step 3, g() = k, k = 0, 1, 2,.... Therefore, f() is O( k ). The proposed framework is geeral, ad may be applied to complexity fuctios ivolvig multiple variables. A applicatio of this i AI has bee cosidered i sectio 11. Quite ofte, complexity fuctios for algorithms are expressed as fractios. For fractioal expressios, the four steps described earlier eeds to be applied both to the umerator ad to the deomiator. Usig the procedural steps, we fid out the highest order term i the umerator as g 1 (), ad that i the deomiator as g 2 (). Fially, we calculate the ratio g 1() g 2 (), ad deote it by g(). The big-oh complexity for the fractioal expressio is O(g()). A applicatio of this appears at sectio 11. The steps may be applied i determiig the bigoh complexity for fuctios ivolvig fractioal expressios with multiple variables. 7.1 Additio ad Multiplicatio of fuctios Here, we exted our proposed framework i fidig out the big-oh complexity for the additio ad multiplicatio of fuctios. Suppose that f 1 () is O(g 1 ()) ad f 2 () is O(g 2 ()). It is ecessary to fid out the big-oh complexity for (f 1 + f 2 )() ad (f 1 f 2 )(). Now, f 1 () is O(g 1 ()). Therefore, f 1 () = K 1 g 1 () + Lower Order Terms. Similarly, f 2 () = K 2 g 2 () + Lower Order Terms. Therefore, (f 1 + f 2 )() = f 1 () + f 2 () = (K 1 g 1 () + K 2 g 2 () + Other Lower Ordered Terms from f 1 () ad f 2 (). Three cases are possible: (1) g 1 () > g 2 (): (f 1 + f 2 )() O(g 1 ()); (2) g 1 () < g 2 (): (f 1 + f 2 )() O(g 2 ()); (3) g 1 () = g 2 () = g(): (f 1 + f 2 )() = f 1 () + f 2 () = (K 1 + K 2 )(g()) + Other Lower Ordered Terms, ad (f 1 + f 2 )() is O(g()). Next cosider multiplicatio of fuctios. Now, (f 1 f 2 )() = f 1 ()f 2 () = (K 1 g 1 () + Lower Order Terms) (K 2 g 2 () + Lower Order Terms) = K 1 K 2 g 1 ()g 2 () + Other Lower Order Terms i the product. The modified complexity fuctio is obtaied by elimiatig the costat coefficiets from (f 1 f 2 )(). Sice, (f 1 f2 )() = g 1 ()g 2 () + remaiig lower order terms without coefficiets. Therefore, (f 1 f 2 )() is O(g 1 ()g 2 ()). This aalysis may be exteded to fuctios ivolvig multiple variables as well as to the multiplicatio of more tha two fuctios. 8 Space Complexity Aalysis The space complexity i implemetig a algorithm is the amout of memory that it eeds to ru the algorithm to completio. Complexity fuctio for the computatioal memory space requiremet is deoted by S(P ). Usig the set theory, S(P ) O(g()), where g() {1, log(),, 2, 3,..., 2,!}. Thus, g() is a member of the set cotaiig all possible complexity class fuctios. The memory space required to implemet a algorithm may be divided ito two parts. These are: 1. Fixed Space Requiremets, C: This computatioal memory requiremet is idepedet of the characteristics of iputs ad outputs. This space icludes istructio space, space for storig simple variables, fixed-size structured variable, costats, etc. This is also kow as the static storage requiremet. 2. Variable storage requiremet, S p (I): This memory space is required for the creatio of dyamic data etities. This requiremet depeds o the istace characteristic I. The umber, size, values of iputs, ad outputs associated with a particular istace, I, recursive stack space, space for storig the formal parameters, local variables, ad retur addresses. Therefore, the total computatioal memory space requiremet, S(P ) = C + S p (I). 8.1 Space Complexity of Algorithms We cosider two examples of determiig the space complexity fuctio i recursive ad iterative algorithms.
Proceedigs of the 10th WSEAS Iteratioal Coferece o APPLIED MATHEMATICS, Dallas, Texas, USA, November 1-3, 2006 323 Example 1: At first, cosider the followig iterative algorithm. Fuctio Sum adds elemets iside a give list of umbers, a[0: 1]. template < class T > T Sum (T a[ ], it ) { T tsum = 0 ; for i = 0 to ( 1) i step 1 do tsum + = a[i] ; retur tsum; } The memory space required for storig the poiter to the array a[ ], iteger, idex i, ad the parameter tsum, each is 2 bytes or 1 iteger memory uit log. Therefore, C = 42 bytes = 8 bytes. Also, S p (I) = 0. Hece, S(P ) = C + S p (I) = 8, ad S(P ) O(1). Similarly, S(P ) Ω(1). Suppose that the iitial assigmet statemet takes c 1 time. The for loop executes differet times. If each executio of the iterative loop takes c 2 uits of time, the the time complexity fuctio is, f() = c 1 + c 2. Therefore, f() O(). The space-time badwidth product for the fuctio is, C() = 1 =. Therefore, C() O(). Example 2: Followig is a recursive versio of the algorithm i Example 1 to add elemets iside a give list of umbers, a[0: 1]. template < class T > T RSum (T a[ ], it ) { if ( > 0) the retur RSum(a, 1) + a[ 1]; ed if retur 0; } For the give recursive program, the recursive calls are geerated i the followig sequece: Rsum(a, ), Rsum(a, 1), Rsum(a, 2),...,Rsum(a, 1), Rsum(a, 0). Therefore, the depth of recursio is ( + 1). Each call to Rsum takes up spaces for a (2 bytes), (2 bytes) ad a Retur Address (2 bytes log). A sigle call to Rsum eeds 2 3 = 6 bytes. Hece, S p (I) = 6 ( +1), with I =, with represetig the istace characteristics, ad C = 0. Now S(P ) = C + S p (I) = 6(+1). Fially, S(P ) O(). Also S(P ) Ω(). For ay > 0, the fuctio RSum executes differet times. If each executio of the fuctio takes c uits of time, the the time complexity fuctio, f() = c. Hece, f() O(). Therefore, the space-time badwidth product for the fuctio is, C() = 2, ad C() O( 2 ). 9 Performace Measuremet Laboratory-based techiques play a key role i decidig the performace of algorithms. For example, from theoretical aalysis, we kow that the selectio sort algorithm is O( 2 ), ad merge sort algorithm is O(log()). But we are further iterested i learig the effect of this order differece o computerbased implemetatio of the sortig algorithms. For selectio sort, the highest order term g s () i the complexity expressio is 2, ad that i the merge sort is, g m () = log(). Now, the ratio betwee g m () ad g s () is, r = g m() g s () = log() = log() 2. As iput size grows to a very large value, this ratio approaches 0. This is evidet from the mathematical fact that: Lim ( log() ) = 0. We verify this with performace measuremet. The algorithms are implemeted usig Microsoft s Visual C++.NET compiler uder Visual Studio.NET IDE. A PC with a Itel Petium 4, 2.80 GHz CPU ad 512 MB of RAM space has bee used. The result is listed below. With a i- Table 3: Performace measuremet data to compare merge sort with selectio sort (all timig values are i secods) T selectio T merge T merge T selectio 10 0 0 0 10000 52 0 0 20000 208 1 0.00481 30000 472 1 0.00212 40000 2 835 0.0024 50000 1313 3 0.002285 60000 1795 3 0.001671 put size, = 10, the time cosumed by both the algorithm is egligible (0). As the problem size grows, selectio sort cosumes more ad more time compared to the merge sort. At = 10000, the selectio sort requires 52 secods, whereas the merge sort still requires egligible time (0 secod). Therefore, the ratio becomes 0. With icreasig problem sizes, this ratio becomes smaller ad smaller with the smallest ratio beig 0.001671 at = 60000. This meas that for this particular iput, the time required by the merge sort is oly 0.1671% of the selectio sort. Hece, as Lim ( log() ) = 0 holds true.
Proceedigs of the 10th WSEAS Iteratioal Coferece o APPLIED MATHEMATICS, Dallas, Texas, USA, November 1-3, 2006 324 10 Applicatio I this sectio, we cosider a few well kow algorithms i Artificial Itelligece (AI). Example 1: AI Problems that cosume memory space i expoetial order are proe to combiatorial expositio. Techiques that rely o storig all possibilities i memory, or eve geeratig all possibilities, are out of the questio except for the smallest of these problems. As a example, followig table lists the total umber of visited odes by a set of AI Gamig Algorithms durig computatio, which are i the order of their computatioal memory requiremets. Algorithms listed above are all expoetial or Table 4: Memory space requiremets for some AI gamig algorithms.) Game Maxm. # of visited odes 8-Puzzle 9! 15-Puzzle 16! -city TSP! Checkers 10 20 Chess 10 40 factorial complexity i memory space requiremets. Example 2: Breadth-First-Search (BFS) is a frequetly used algorithm i AI. We aalyze the algorithm for time ad space complexity. Each ode usig BFS may be visited i a costat time. The time complexity fuctio f B is a fuctio of the brachig factor b, ad the solutio depth d (a multi-variable fuctio). I the the worst case, we will eed to geerate all the odes at level d. Now, f B (b, d) = 1 + b + b 2 +... + b d = (b(d+1) 1) (b 1). Next we apply our framework for the fractioal expressios. The highest order term i the umerator, g 1 (b, d) = b (d+1). Similarly, the highest order term i the deomiator, g 2 (b, d) = b. Hece, g(b, d) = b(d+1) b = b d ad f B (b, d) O(b d ). To geerate the solutio, we eed to store all of the visited odes. Hece, S B (b, d) = (b(d+1) 1) (b 1), ad S B (b, d) O(b d ). I the best possible case, we will fid the solutio at the first visited ode. Hece, f B (b, d) Ω(1), ad also, S B (b, d) Ω(1). The space-time badwidth product, C B (b, d) = (b d ) 2 = b ( 2d). Suppose that we wat to implemet the algorithm o a typical machie with a speed of 100 MHz. If it visits a ew state (ode) i every 100 istructios, ad if each istructio takes 1 machie cycle, the a total of 100 106 100 = 10 6 odes will be visited i each secod. Suppose, each ode occupies 4 bytes of memory. With a RAM size of 1 GB =109 byte, it s ode keepig capacity is, 109 4 =250 106. With a complex problem, oly after 250 106 10 6 = 250 secods or 4 miutes ad 10 secods of visitig (geeratig) odes, all memory space will be exhausted! This pheomeo is statespace explosio i AI. 11 Coclusio Complexity aalysis is sigificat i justifyig whether the algorithm will take up prohibitive amout of computatio time or computatioal memory space with cosiderable iput size. This is particularly true with expoetial algorithms. I this paper, a geeralized framework for the time ad space complexity aalysis of algorithms ad fuctios has bee proposed ad it s applicatios are discussed. Also, the sigificace of asymptotic complexity is established. Set theoretic properties relatig to complexity are cosidered. It has bee established that the otatioal complexities are sets of fuctios. AI algorithms are proe to both time ad the space complexity aalysis. For these algorithms, it is importat to use the otatio called Space-Time Badwidth Product, C. Performace is ot everythig. There ca be a tradeoff betwee ease of uderstadig, writig ad debuggig a code correspodig to a algorithm with the efficiet use of time ad space. However, it is still useful i comparig the performace of differet algorithms, eve if the optimal algorithm may ot be adopted. A problem may have more tha oe solutio each oe of which ca be expressed as a differet algorithm. Therefore, we eed to compare amog the performace of algorithms. After comparig differet, possible algorithms for solvig a problem, the user may select the oe that best fits his computatioal eeds. I future, the relatioship betwee the time ad the space complexity order of algorithms ad fuctios will be ivestigated. Also the proposed framework will be applied to critical AI problems for performace evaluatio. Refereces: [1] Douglas B. West, Itroductio to Graph Theory, Secod Editio, Pretice Hall, New Jersey 2001 [2] Richard E. Neapolita ad Kumarss Naimipour, Foudatios of Algorithms Usig C++ Pseudocode, Third Editio, Joes ad Bartlett Publishers, Massachusetts 2004 [3] A. Tarek, O Time Complexity Aalysis of Algorithms, Proceedigs of the 9th World Multicoferece o Systemics, Cyberetics ad Iformatics (WMSCI 2005), Orlado, Florida, U.S.A, July 10 to 13, 2005