Analysis of Algorithms

Transcription

1 Presetatio for use with the textbook, Algorithm Desig ad Applicatios, by M. T. Goodrich ad R. Tamassia, Wiley, 2015 Aalysis of Algorithms Iput 2015 Goodrich ad Tamassia Algorithm Aalysis of Algorithms Output 1

2 Scalability Scietists ofte have to deal with differeces i scale, from the microscopically small to the astroomically large. Computer scietists must also deal with scale, but they deal with it primarily i terms of data volume rather tha physical object size. Scalability refers to the ability of a system to gracefully accommodate growig sizes of iputs or amouts of workload Goodrich ad Tamassia Aalysis of Algorithms 2

3 Applicatio: Job Iterviews High techology compaies ted to ask uestios about algorithms ad data structures durig job iterviews. Algorithms uestios ca be short but ofte reuire critical thikig, creative isights, ad subject kowledge. All the Applicatios exercises i Chapter 1 of the Goodrich- Tamassia textbook are take from reports of actual job iterview uestios. xkcd Labyrith Puzzle. Used with permissio uder Creative Commos 2.5 Licese Goodrich ad Tamassia Aalysis of Algorithms 3

4 Algorithms ad Data Structures A algorithm is a step-by-step procedure for performig some task i a fiite amout of time. Typically, a algorithm takes iput data ad produces a output based upo it. Iput Algorithm Output A data structure is a systematic way of orgaizig ad accessig data Goodrich ad Tamassia Aalysis of Algorithms 4

5 Ruig Time Most algorithms trasform iput objects ito output objects. The ruig time of a algorithm typically grows with the iput size. Average case time is ofte difficult to determie. We focus primarily o the worst case ruig time. Easier to aalyze Crucial to applicatios such as games, fiace ad robotics Ruig Time best case average case worst case Iput Size 2015 Goodrich ad Tamassia Aalysis of Algorithms 5

6 Experimetal Studies Write a program implemetig the algorithm Ru the program with iputs of varyig size ad compositio, otig the time eeded: Plot the results Time (ms) Iput Size 2015 Goodrich ad Tamassia Aalysis of Algorithms 6

7 Limitatios of Experimets It is ecessary to implemet the algorithm, which may be difficult Results may ot be idicative of the ruig time o other iputs ot icluded i the experimet. I order to compare two algorithms, the same hardware ad software eviromets must be used 2015 Goodrich ad Tamassia Aalysis of Algorithms 7

8 Theoretical Aalysis Uses a high-level descriptio of the algorithm istead of a implemetatio Characterizes ruig time as a fuctio of the iput size, Takes ito accout all possible iputs Allows us to evaluate the speed of a algorithm idepedet of the hardware/ software eviromet 2015 Goodrich ad Tamassia Aalysis of Algorithms 8

9 Pseudocode High-level descriptio of a algorithm More structured tha Eglish prose Less detailed tha a program Preferred otatio for describig algorithms Hides program desig issues 2015 Goodrich ad Tamassia Aalysis of Algorithms 9

10 Pseudocode Details Cotrol flow if the [else ] while do repeat util for do Idetatio replaces braces Method declaratio Algorithm method (arg [, arg ]) Iput Output Method call method (arg [, arg ]) Retur value retur expressio Expressios: Assigmet = Euality testig 2 Superscripts ad other mathematical formattig allowed 2015 Goodrich ad Tamassia Aalysis of Algorithms 10

11 The Radom Access Machie (RAM) Model CPU A RAM cosists of A CPU A potetially ubouded bak of memory cells, each of which ca hold a arbitrary umber or character Memory cells are umbered ad accessig ay cell i memory takes uit time 2015 Goodrich ad Tamassia Aalysis of Algorithms Memory 11

12 Seve Importat Fuctios Seve fuctios that ofte appear i algorithm aalysis: Costat 1 Logarithmic log Liear N-Log-N log Quadratic 2 Cubic 3 Expoetial 2 I a log-log chart, the slope of the lie correspods to the growth rate T ( ) 1E+30 1E+28 1E+26 1E+24 1E+22 1E+20 1E+18 1E+16 1E+14 1E+12 1E+10 1E+8 1E+6 1E+4 1E+2 1E+0 Cubic Quadratic Liear 1E+0 1E+2 1E+4 1E+6 1E+8 1E Goodrich ad Tamassia Aalysis of Algorithms 12

13 Fuctios Graphed Usig Normal Scale Slide by Matt Stallma icluded with permissio. g() = 1 g() = lg g() = 2 g() = lg g() = 2 g() = g() = Goodrich ad Tamassia Aalysis of Algorithms 13

14 Primitive Operatios Basic computatios performed by a algorithm Idetifiable i pseudocode Largely idepedet from the programmig laguage Exact defiitio ot importat (we will see why later) Assumed to take a costat amout of time i the RAM model Examples: Evaluatig a expressio Assigig a value to a variable Idexig ito a array Callig a method Returig from a method 2015 Goodrich ad Tamassia Aalysis of Algorithms 14

15 Coutig Primitive Operatios Example: By ispectig the pseudocode, we ca determie the maximum umber of primitive operatios executed by a algorithm, as a fuctio of the iput size 2015 Goodrich ad Tamassia Aalysis of Algorithms 15

16 Estimatig Ruig Time Algorithm arraymax executes 7-2 primitive operatios i the worst case, 5 i the best case. Defie: a = Time take by the fastest primitive operatio b = Time take by the slowest primitive operatio Let T() be worst-case time of arraymax. The a(5) T() b(7-2) Hece, the ruig time T() is bouded by two liear fuctios 2015 Goodrich ad Tamassia Aalysis of Algorithms 16

17 Growth Rate of Ruig Time Chagig the hardware/ software eviromet Affects T() by a costat factor, but Does ot alter the growth rate of T() The liear growth rate of the ruig time T() is a itrisic property of algorithm arraymax 2015 Goodrich ad Tamassia Aalysis of Algorithms 17

18 Slide by Matt Stallma icluded with permissio. Why Growth Rate Matters if rutime is... time for + 1 time for 2 time for 4 c lg c lg ( + 1) c (lg + 1) c(lg + 2) c c ( + 1) 2c 4c c lg ~ c lg + c 2c lg + 2c 4c lg + 4c c 2 ~ c 2 + 2c 4c 2 16c 2 rutime uadruples whe problem size doubles c 3 ~ c 3 + 3c 2 8c 3 64c 3 c 2 c 2 +1 c 2 2 c Goodrich ad Tamassia Aalysis of Algorithms 18

19 Aalyzig Recursive Algorithms Use a fuctio, T(), to derive a recurrece relatio that characterizes the ruig time of the algorithm i terms of smaller values of Goodrich ad Tamassia Aalysis of Algorithms 19

20 Costat Factors The growth rate is miimally affected by costat factors or lower-order terms Examples is a liear fuctio is a uadratic fuctio T () 1E+26 1E+24 1E+22 1E+20 1E+18 1E+16 1E+14 1E+12 1E+10 1E+8 1E+6 1E+4 1E+2 1E+0 Quadratic Quadratic Liear Liear 1E+0 1E+2 1E+4 1E+6 1E+8 1E Goodrich ad Tamassia Aalysis of Algorithms 20

21 Big-Oh Notatio Give fuctios f() ad g(), we say that f() is O(g()) if there are positive costats c ad 0 such that 10,000 1, f() cg() for 0 Example: is O() c (c - 2) 10 10/(c - 2) Pick c = 3 ad 0 = , Goodrich ad Tamassia Aalysis of Algorithms 21

22 Big-Oh Example Example: the fuctio 2 is ot O() 2 c c The above ieuality caot be satisfied sice c must be a costat 1,000, ,000 10,000 1, ^ , Goodrich ad Tamassia Aalysis of Algorithms 22

23 More Big-Oh Examples is O() eed c > 0 ad 0 1 such that 7-2 c for 0 this is true for c = 7 ad 0 = is O( 3 ) eed c > 0 ad 0 1 such that c 3 for 0 this is true for c = 4 ad 0 = 21 3 log log + 5 is O(log ) eed c > 0 ad 0 1 such that 3 log + 5 c log for 0 this is true for c = 8 ad 0 = Goodrich ad Tamassia Aalysis of Algorithms 23

24 Big-Oh ad Growth Rate The big-oh otatio gives a upper boud o the growth rate of a fuctio The statemet f() is O(g()) meas that the growth rate of f() is o more tha the growth rate of g() We ca use the big-oh otatio to rak fuctios accordig to their growth rate f() is O(g()) g() is O(f()) g() grows more Yes No f() grows more No Yes Same growth Yes Yes 2015 Goodrich ad Tamassia Aalysis of Algorithms 24

25 Big-Oh Rules If is f() a polyomial of degree d, the f() is O( d ), i.e., 1. Drop lower-order terms 2. Drop costat factors Use the smallest possible class of fuctios Say 2 is O() istead of 2 is O( 2 ) Use the simplest expressio of the class Say is O() istead of is O(3) 2015 Goodrich ad Tamassia Aalysis of Algorithms 25

26 Asymptotic Algorithm Aalysis The asymptotic aalysis of a algorithm determies the ruig time i big-oh otatio To perform the asymptotic aalysis We fid the worst-case umber of primitive operatios executed as a fuctio of the iput size We express this fuctio with big-oh otatio Example: We say that algorithm arraymax rus i O() time Sice costat factors ad lower-order terms are evetually dropped ayhow, we ca disregard them whe coutig primitive operatios 2015 Goodrich ad Tamassia Aalysis of Algorithms 26

27 A Case Study i Algorithm Aalysis Give a array of itegers, fid the subarray, A[j:k] that maximizes the sum I additio to beig a iterview uestio for testig the thikig skills of job cadidates, this maximum subarray problem also has applicatios i patter aalysis i digitized images Goodrich ad Tamassia Aalysis of Algorithms 27

28 A First (Slow) Solutio Compute the maximum of every possible subarray summatio of the array A separately. The outer loop, for idex j, will iterate times, its ier loop, for idex k, will iterate at most times, ad the ier-most loop, for idex i, will iterate at most times. Thus, the ruig time of the MaxsubSlow algorithm is O( 3 ) Goodrich ad Tamassia Aalysis of Algorithms 28

29 A Improved Algorithm A more efficiet way to calculate these summatios is to cosider prefix sums If we are give all such prefix sums (ad assumig S 0 =0), we ca compute ay summatio s j,k i costat time as 2015 Goodrich ad Tamassia Aalysis of Algorithms 29

30 A Improved Algorithm, cot. Compute all the prefix sums The compute all the subarray sums 2015 Goodrich ad Tamassia Aalysis of Algorithms 30

31 Arithmetic Progressio The ruig time of MaxsubFaster is O( ) The sum of the first itegers is ( + 1) / 2 There is a simple visual proof of this fact Thus, algorithm MaxsubFaster rus i O( 2 ) time Goodrich ad Tamassia Aalysis of Algorithms 31

32 A Liear-Time Algorithm Istead of computig prefix sum S t = s 1,t, let us compute a maximum suffix sum, M t, which is the maximum of 0 ad the maximum s j,t for j = 1,, t. if M t > 0, the it is the summatio value for a maximum subarray that eds at t, ad if M t = 0, the we ca safely igore ay subarray that eds at t. if we kow all the M t values, for t = 1, 2,,, the the solutio to the maximum subarray problem would simply be the maximum of all these values Goodrich ad Tamassia Aalysis of Algorithms 32

33 A Liear-Time Algorithm, cot. for t 2, if we have a maximum subarray that eds at t, ad it has a positive sum, the it is either A[t : t] or it is made up of the maximum subarray that eds at t 1 plus A[t]. So we ca defie M 0 = 0 ad If this were ot the case, the we could make a subarray of eve larger sum by swappig out the oe we chose to ed at t 1 with the maximum oe that eds at t 1, which would cotradict the fact that we have the maximum subarray that eds at t. Also, if takig the value of maximum subarray that eds at t 1 ad addig A[t] makes this sum o loger be positive, the M t = 0, for there is o subarray that eds at t with a positive summatio Goodrich ad Tamassia Aalysis of Algorithms 33

34 A Liear-Time Algorithm, cot. The MaxsubFastest algorithm cosists of two loops, which each iterate exactly times ad take O(1) time i each iteratio. Thus, the total ruig time of the MaxsubFastest algorithm is O() Goodrich ad Tamassia Aalysis of Algorithms 34

35 Math you eed to Review Summatios Powers Logarithms Proof techiues Basic probability Properties of powers: a (b+c) = a b a c a bc = (a b ) c a b /a c = a (b-c) b = a log a b b c = a c*log a b Properties of logarithms: log b (xy) = log b x + log b y log b (x/y) = log b x - log b y log b xa = alog b x log b a = log x a/log x b 2015 Goodrich ad Tamassia Aalysis of Algorithms 35

36 Relatives of Big-Oh big-omega f() is Ω(g()) if there is a costat c > 0 ad a iteger costat 0 1 such that big-theta f() c g() for 0 f() is Θ(g()) if there are costats c > 0 ad c > 0 ad a iteger costat 0 1 such that c g() f() c g() for Goodrich ad Tamassia Aalysis of Algorithms 36

37 Ituitio for Asymptotic Notatio big-oh f() is O(g()) if f() is asymptotically less tha or eual to g() big-omega f() is Ω(g()) if f() is asymptotically greater tha or eual to g() big-theta f() is Θ(g()) if f() is asymptotically eual to g() 2015 Goodrich ad Tamassia Aalysis of Algorithms 37

38 Example Uses of the Relatives of Big-Oh 5 2 is Ω( 2 ) f() is Ω(g()) if there is a costat c > 0 ad a iteger costat 0 1 such that f() c g() for 0 let c = 5 ad 0 = is Ω() f() is Ω(g()) if there is a costat c > 0 ad a iteger costat 0 1 such that f() c g() for 0 let c = 1 ad 0 = is Θ( 2 ) f() is Θ(g()) if it is Ω( 2 ) ad O( 2 ). We have already see the former, for the latter recall that f() is O(g()) if there is a costat c > 0 ad a iteger costat 0 1 such that f() < c g() for 0 Let c = 5 ad 0 = Goodrich ad Tamassia Aalysis of Algorithms 38

39 Amortizatio The amortized ruig time of a operatio withi a series of operatios is the worst-case ruig time of the series of operatios divided by the umber of operatios. Example: A growable array, S. Whe eedig to grow: a. Allocate a ew array B of larger capacity. b. Copy A[i] to B[i], for i = 0,..., 1, where is size of A. c. Let A = B, that is, we use B as the array ow supportig A Goodrich ad Tamassia Aalysis of Algorithms 39

40 Growable Array Descriptio Let add(e) be the operatio that adds elemet e at the ed of the array Whe the array is full, we replace the array with a larger oe But how large should the ew array be? Icremetal strategy: icrease the size by a costat c Doublig strategy: double the size Algorithm add(e) if t = A.legth - 1 the B ew array of size for i 0 to -1 do B[i] A[i] A B + 1 A[-1] e 2015 Goodrich ad Tamassia Aalysis of Algorithms 40

41 Compariso of the Strategies We compare the icremetal strategy ad the doublig strategy by aalyzig the total time T() eeded to perform a series of add operatios We assume that we start with a empty list represeted by a growable array of size 1 We call amortized time of a add operatio the average time take by a add operatio over the series of operatios, i.e., T()/ 2015 Goodrich ad Tamassia Aalysis of Algorithms 41

42 Icremetal Strategy Aalysis Over add operatios, we replace the array k = /c times, where c is a costat The total time T() of a series of add operatios is proportioal to + c + 2c + 3c + 4c + + kc = + c( k) = + ck(k + 1)/2 Sice c is a costat, T() is O( + k 2 ), i.e., O( 2 ) Thus, the amortized time of a add operatio is O() 2015 Goodrich ad Tamassia Aalysis of Algorithms 42

43 Doublig Strategy Aalysis We replace the array k = log 2 times The total time T() of a series of push operatios is proportioal to k = + 2 k = 3-1 T() is O() The amortized time of a add operatio is O(1) geometric series Goodrich ad Tamassia Aalysis of Algorithms 43

44 Accoutig Method Proof for the Doublig Strategy We view the computer as a coi-operated appliace that reuires the paymet of 1 cyber-dollar for a costat amout of computig time. We shall charge each add operatio 3 cyber-dollars, that is, it will have a amortized O(1) amortized ruig time. We over-charge each add operatio ot causig a overflow 2 cyber-dollars. Thik of the 2 cyber-dollars profited i a isertio that does ot grow the array as beig stored at the elemet iserted. A overflow occurs whe the array A has 2 i elemets. Thus, doublig the size of the array will reuire 2 i cyber-dollars. These cyber-dollars are at the elemets stored i cells 2 i 1 through 2 i Goodrich ad Tamassia Aalysis of Algorithms 44