Algorithm Analysis CISC4080 CIS, Fordham Univ. Instructor: X. Zhang

Transcription

1 Algorithm Analysis CISC4080 CIS, Fordham Univ. Instructor: X. Zhang

2 Last class Review some algorithms learned in previous classes idea => pseudocode => implementation Correctness? Three sorting algorithms: insertion sort: gradually expand the sorted sub array/list bubble sort: bubble largest number to the right, also expand sorted sub array/list selection sort: find location of the largest and swap it with the last element, and then do the same for the sublist excluding last element 2

3 Outline Introduction to algorithm analysis using fibonacci seq calculation as example Running time Measurement Running time Analysis: based upon code or pseudocode count number of computer steps write recursive formula for running time of recursive algorithm math help: math. induction 3

4 Introduction to algorithm analysis Consider calculation of Fibonacci sequence, in particular, the n-th number in sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 4

5 Fibonacci Sequence 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, Formally, Problem: How to calculate n-th term, e.g., what is F 100, F 200? 5

6 A recursive algorithm Three questions: Is it correct? three questions to verify recursive solution How long does it take? How much memory is needed? Can we do better? (faster?) 6

7 In pursuit of better algorithms We want to solve problems using less resource: Space: how much memory is needed? Time: how fast can we get the result? how many CPU cycles? Usually, the larger input, the more memory it takes and longer it takes it takes longer to calculate 200-th number in Fibonacci sequence than the 10th number it takes longer to sort larger array, search a list for a particular value Efficient algorithms are critical for large input size/problem instance Calculate F100, Searching Web for certain keywords 7

8 Experimental approach Measure how much time elapses from algorithm starts to finishes use computer s built-in clock use command $man gettimeofday for more detail #include <sys/time.h> struct timeval t1; //t1, t2 are two variables of type timeval /* struct timeval { time_t tv_sec; /* seconds */ suseconds_t tv_usec; /* microseconds */ }; and gives the number of seconds and microseconds since the Epoch (see time(2)), Jan 1 midnight of 1970 */ gettimeofday(&t1,null); //set t1.tv_sec, t1.tv_usec 8

9 Experimental approach Implement, instrument and deploy #include <sys/time.h> struct timeval t1, t2; //t1, t2 are two variables of type timeval gettimeofday(&t1,null); result = Fib1(n); // or any other algorithm we want to measure gettimeofday (&t2, NULL); // calculate how many seconds have passed between t1 and t2 double timeinseconds = (t2.tv_sec-t1.tv_sec) + (t2.tv_usec-t2.tv_usec)/ ; 9

10 Example (Fib1: recursive) n T(n)ofFib1 F(n) 10 3e e Time (in seconds) 12 4e e e e e e e n Running time seems to grows exponentially as n increases How long does it take to Calculate F(100)?

11 Experimental approach Measure how much time elapses from algorithm starts to finishes results are realistic, specific and random specific to language, run time system (Java VM, OS), caching effect, other processes running cannot shed light on: larger input size(not always possible to test all input size), faster CPU, Measurement is important for a production system/ end product; but not informative for algorithm efficiency studies/comparison/prediction 11

12 Take-away One can perform model-fitting to find out T(n): running time of the algorithms given input size Cons of measurement approach: time consuming, maybe too late Does not explain why Draw recursive function call tree for fib1(5) Observation: wasteful repeated calculation 12

13 Can we do better? Idea: Store solutions to subproblems in array (key of Dynamic Programming) 13

14 Example (Fib2: iterative) 10 1e e e e e e e e e-06 Time (in seconds) Increase very slowly as n increases Model fitting to find out T(n)? n 14

15 Analytic approach Is it possible to find out how running time grows when input size grows, analytically? Does running time stay constant, increase linearly, logarithmically, quadratically, exponentially? Yes: analyze pseudocode/code, calculate total number of steps in terms of input size, and study its order of growth results are general: not specific to language, run time system, caching effect, other processes sharing computer shed light on effects of larger problem size, faster CPU, 15

16 Running time analysis Given an algorithm in pseudocode or actual program When the input size is n, how many total number of computer steps are executed? Size of input: size of an array, polynomial degree, # of elements in a matrix, vertices and edges in a graph, or # of bits in the binary representation of input Computer steps: arithmetic operations, data movement, control, decision making (if /then), comparison, Assume all computer steps take same amount of time Ignore: overhead of function calls (call stack frame allocation, passing parameters, and return values) 16

17 Running time fib2(n) Let T(n) represent total number of steps carried out in fib2(n) T(n)=1 // if n=0 return 0 +n // create an array of f[0 n] +2 // f[0]=0, f[1]=1 +(n-1) // for loop: repeated for n-1 times = 2n+2 T(n) is a linear function of n, or fib2(n) has linear running time 17

18 Alternatively Estimation based upon CPU: takes 1000us, takes 200n us each assignment takes 60us How long does it take for fib2(n) finish? T(n)= n+2*60+(n-1)*800=1000n+320 // in unit of us Again: T(n) is a linear function of n addition and assignment takes 800us Constants are not important: different on different computers System effects (caching, OS scheduling) makes it pointless to do such fine-grained analysis anyway! Algorithm analysis focuses on how running time grows as problem size grows (constant, linear, quadratic, exponential?) not the actual real world time 18

19 Case Studies: Fib1(n) Let T(n) be number of computer steps needed to compute fib1(n) T(0)=1: when n=0, first step is executed T(1)=2: when n=1, first two steps are executed For n >1, T(n)=T(n-1)+T(n-2)+3: first two steps are executed, fib1(n-1) is called (with T(n-1) steps), fib1(n-2) is called (T(n-2) steps), return values are added (1 step) 19

20 Running Time analysis Analyze running time of recursive algorithm first, write a recursive formula for its running time then, recursive formula => closed formula, asymptotic result How fast does T(n) grow? Can you see that T(n) > Fn? How big is T(n)? 20

21 Mathematical Induction F0=0, F1=1, Fn=Fn-1+Fn-2 We will show that Fn >= 2 0.5n, for n >=6 using strong mathematical induction technique Intuition of basic mathematical induction it s like Domino effect if one push 1st card, all cards fall because 1) 1st card is pushed down 2) every card is close to next card, so that when one card falls, next one falls 21

22 Mathematical Induction Sometimes, we needs the multiple previous cards to knock down next card Intuition of strong mathematical induction it s like Domino effect: if one push first two cards, all cards fall because the weights of two cards falling down knock down the next card Generalization: 2 => k 22

23 Fibonacci numbers F0=0, F1=1, Fn=Fn-1+Fn-2 show that for all integer n >=6 using strong mathematical induction basis step: show it s true when n=6, 7 inductive step: show if it s true for n=k-1, k, then it s true for k+1 given F, F k 2 k k 1 2 k F k+1 = F k 1 + F k 2 k k 2 2 k k 1 2 =2 2 k 1 2 =2 1+ k 1 2 =2 k

24 Fibonacci numbers F0=0, F1=1, Fn=Fn-1+Fn-2 F n 2 n 2 =2 0.5n Fn is lower bounded by In fact, there is a tighter lower bound n Recall T(n): number of computer steps to compute fib1(n), T(0)=1 T(1)=2 T(n)=T(n-1)+T(n-2)+3, n> n T (n) >F n n 24

25 Exponential running time Running time of Fib1: T(n)> n Running time of Fib1 is exponential in n calculate F 200, it takes at least computer steps On NEC Earth Simulator (fastest computer ) Executes 40 trillion (10 12 ) steps per second, 40 teraflots Assuming each step takes same amount of time as a floating point operation Time to calculate F 200: at least 2 92 seconds, i.e., 1.57x10 20 years Can we throw more computing power to the problem? Moore s law: computer speeds double about every 18 months (or 2 years according to newer version) 25

26 Exponential algorithms Moore s law (computer speeds double about every two years) can sustain for 4-5 more years 26

27 Exponential running time Running time of Fib1: T(n)> n = n Moore s law: computer speeds double about every 18 months (or 2 years according to newer version) If it takes fastest CPU of this year 6 minutes to calculate F50, fastest CPU in two years from today can calculate F52 in 6 minutes Algorithms with exponential running time are not efficient, not scalable 27

28 Summary: Running time analysis Given an algorithm in pseudocode or actual program When the input size is n, how many total number of computer steps are executed? Size of input: size of an array, polynomial degree, # of elements in a matrix, vertices and edges in a graph, or # of bits in the binary representation of input Computer steps: arithmetic operations, data movement, control, decision making (if, while), comparison, Ignore: each step take a constant amount of time Overhead of function calls (call stack frame allocation, passing parameters, and return values) Different execution time for different steps 28

29 Algorithm Analysis: Example What s the running time of MIN? Algorithm/Function.: MIN (a[1 n]) input: an array of numbers a[1 n] output: the minimum number among a[1 n] m = a[1] for i=2 to n: if a[i] < m: m = a[i] return m How do we measure the size of input for this algorithm? How many computer steps when the input s size is n? 29

30 Algorithm Analysis: bubble sort Algorithm/Function.: bubblesort (a[1 n]) input: a list of numbers a[1 n] output: a sorted version of this list for endp=n to 2: swapop=0; for i=1 to endp-1: if a[i] > a[i+1]: swap (a[i], a[i+1]); swapop++; if (swapop==0): return //no need to continue return How do you choose to measure the size of input? length of list a, i.e., n the longer the input list, the longer it takes to sort it Problem instance: a particular input to the algorithm e.g., a[1 6]={1, 4, 6, 2, 7, 3} e.g., a[1 6]={1, 4, 5, 6, 7, 9} 30

31 Algorithm Analysis: bubble sort bubblesort (a[1 n]) input: a list of numbers a[1 n] output: a sorted version of this list for endp=n to 2: swapop=0; for i=1 to endp-1: if a[i] > a[i+1]: swap (a[i], a[i+1]); swapop++; if (swapop==0): return //no need to continue return For following problem instances, how many comparisons, swap operations are performed? a[1 6]={1, 4, 5, 6, 7, 9} a[1 6]={9, 1, 4, 7, 6, 9} a[1 6]={9,7,6,5,4,1} 31

32 Algorithm Analysis: different cases Running time of an algorithm depends upon Size of the input, usually denoted as n often times, also upon the particular input, e.g. if the list is sorted already or not also upon random number chosen e.g., what s pivot value chosen (as in quicksort) Different cases: best case: some problem instances/inputs yield shortest possible running time (e.g.? ) worst case: some problem instances yield longest possible running time average case: if we assume all instances are equally likely (or any other prob. distribution), the expected running time 32

33 bubble sort: worst case bubblesort (a[1 n]) input: an array of numbers a[1 n] output: a sorted version of this array for endp=n to 2: for i=1 to endp-1: if a[i] > a[i+1]: swap (a[i], a[i+1]) return a a compute step endp=n: inner loop (for j=1 to endp-1) repeats for n-1 times endp=n-1: inner loop repeats for n-2 times endp=n-2: inner loop repeats for n-3 times endp=2: inner loop repeats for 1 times Total # of steps: T(n) = (n-1)+(n-2)+(n-3)

34 How big is T(n)? T(n) = (n-1)+(n-2)+(n-3)+ +1 Can you write big sigma notation for T(n)? Can you write simplified formula for T(n)? Can you prove the above using math. induction? 1) when n=2, left hand size =1, right hand size is 2) if the equation is true for n=k, then it s also true for n=k+1 34

35 Algorithm Analysis: Binary Search Algorithm/Function.: search (a[l R], value) input: a list of numbers a[l R] sorted in ascending order, a number value output: the index of value in list a (if value is in it), or -1 if not found if (L>R): return -1 m = (L+R)/2 if (a[m]==value): return m else: if (a[m]>value): return search (a[l m-1], value) else: return search (a[m+1 R], value) What s the size of input in this algorithm? length of list a[l R] 35

36 Algorithm Analysis: Binary Search Algorithm/Function.: search (a[l R], value) input: a list of numbers a[l R] sorted in ascending order, a number value output: the index of value in list a (if value is in it), or -1 if not found if (L>R): return -1 m = (L+R)/2 if (a[m]==value): return m else: if (a[m]>value): return search (a[l m-1], value) else: return search (a[m+1 R], value) Let T(n) be number of steps to search an list of size n best case (value is in middle point), T(n)=3 worst case (when value is not in list) provides an upper bound 36

37 Algorithm Analysis: Binary Search Algorithm/Function.: search (a[l R], value) input: a list of numbers a[l R] sorted in ascending order, a number value output: the index of value in list a (if value is in it), or -1 if not found if (L>R): return -1 m = (L+R)/2 if (a[m]==value): return m else: if (a[m]>value): return search (a[l m-1], value) else: return search (a[m+1 R], value) Let T(n) be number of steps to search an list of size n in worst case T(0)=1 //base case, when L>R T(n)=3+T(n/2) //general case, reduce problem size by half Next chapter: master theorem solving T(n)=log2n 37

38 summary Running time analysis identifying input size n counting, and recursive formula: number of steps in terms of n Chapter 2: Solving recursive formula Next: Asymptotic running time 38

39 Readings Chapter 0 of DPV 2.1, 2.2 of CLR 39