CS302 Topic: Algorithm Analysis #2. Thursday, Sept. 21, 2006

Transcription

1 CS302 Topic: Algorithm Analysis #2 Thursday, Sept. 21, 2006

2 Analysis of Algorithms The theoretical study of computer program performance and resource usage What s also important (besides performance/resource usage)? Modularity Correctness Maintainability Functionality Robustness User-friendliness Programmer time Simplicity Reliability

3 Why study algorithms and performance? Analysis helps us understand scalability Performance often draws the line between what is feasible and what is impossible Algorithmic mathematics provides a language for talking about program behavior The lessons of program performance generalize to other computing resources Speed is fun!

4 Comparing running times Suppose T 1 (n) = O(f(n)) and T 2 (n) = O(f(n)) Which of the following are true: 1. T 1 (n) + T 2 (n) = O(f(n)) 2. T 1 (n) - T 2 (n) = o(f(n)) 3. T 1 (n) / T 2 (n) = O(1) 4. T 1 (n) = O(T 2 (n)) TRUE FALSE Counterexample: T 1 (n) = 2n; T 1 (n) =n; f(n) = n FALSE Counterexample: T 1 (n) = n 2 ; T 1 (n) =n; f(n) = n 2 FALSE Counterexample: (same as #3 above)

5 We typically want tight bounds Example: for (i=0; i<n; i++) for (j=0; j<n; j++) k++; Runtime = O(n 2 ) So, does this mean that Runtime = O(n 3 ) is false? No! O(n 3 ) is true, but isn t as tight of a bound as we can define We prefer O(n 2 ), because it is a tighter bound

6 General rules for determining computational complexity of an alg. For loops: Running time is at most the running time of the statements inside the for loop (including tests) times the number of iterations Nested loops: Analyze from the inside out. Total running time is the running time of the interior statements times the product of the sizes of the loops Example: for (i=0; i<n; i++) for (j=0; j<n; j++) k++; Runtime? O(n 2 )

7 General rules, con t. Consecutive statements: Just add them up, keeping the maximum value Example: for (i=0; i<n; i++) a[i] = 0; for (i=0; i<n; i++) for (j=0; j<n; j++) a[i] += a[j] + i + j; Runtime? O(n 2 )

8 General rules, con t. If-Else statements: Running time is at most the running time of the test plus the larger of the running times of the two parts of the conditional Example: if (test == true) else { } a[0] = 0; for (i=0; i<n; i++) a[i] += a[i+1]; Runtime? O(n)

9 General rules, con t. Function calls Analyze first, using previous rules Recursive function calls Check to see if it s equivalent to a for loop, and if so, analyze accordingly Otherwise, set up recurrence and solve (e.g., using recursion tree) we ll talk about this shortly

10 Example 1: What is the runtime? O(1) O(n) sum = 0; for (i = 0; i < n; i++) O(1) sum++; T(n) = O(1) + O(n 1) T(n) = O(n)

11 Example 2: What is the runtime? O(1) O(n) sum = 0; for (i = 0; i < n; i++) O(n) for (j=0; j < n; j++) O(1) sum++; T(n) = O(1) + O(n n 1) T(n) = O(n 2 )

12 Example 3: What is the runtime? O(1) O(n) sum = 0; for (i = 0; i < n; i++) O(n 2 ) for (j=0; j < n * n; j++) O(1) sum++; T(n) = O(1) + O(n n 2 1) T(n) = O(n 3 )

13 Example 4: What is the runtime? O(1) O(?) sum = 0; for (i = 0; i < n; i++) n 1 i 1 i= 0 j= 0 for (j=0; j < i; j++) O(1) 1 sum++; for i=1, sum = 1 for i=2, sum = 1+ 1 for i=3, sum = for i=4, sum = n 1 i 1 n i 2 n( n 1) O( n ) i= 0 j= 0 i= 1 = = + = 2 2 ( ) (1) ( 1) ( ) Tn= O + On = On Easier way: we could also approximate runtime by looking at max values for each looping variable O(1) + O(n n 1) = O(n 2 )

14 Example 5: What is the runtime? O(1) O(n) sum = 0; for (i = 0; i < n; i++) O(n 2 ) for (j=0; j < i*i; j++) O(n 2 ) for (k=0; k<j; k++) O(1) sum++; T(n) = O(1) + O(n n 2 n 2 1) T(n) = O(n 5 )

15 Example 6: What is the runtime? MyFunc(int m, n) // Assume m and n are approx. equal if ((m = 0) or (n = 0)) then return m else return min{myfunc(m-1,n), MyFunc(m-1, n-1), MyFunc(m, n-1)} + 1

16 Example 6: What is the runtime? MyFunc(int m, n) // Assume m and n are approx. equal if ((m = 0) or (n = 0)) then return m else return min{myfunc(m-1,n), MyFunc(m-1, n-1), MyFunc(m, n-1)} + 1 T(m,n)

17 Example 6: What is the runtime? MyFunc(int m, n) // Assume m and n are approx. equal if ((m = 0) or (n = 0)) then return m else return min{myfunc(m-1,n), MyFunc(m-1, n-1), MyFunc(m, n-1)} + 1 c T(m-1,n) T(m-1,n-1) T(m,n-1)

18 Example 6: What is the runtime? MyFunc(int m, n) // Assume m and n are approx. equal if ((m = 0) or (n = 0)) then return m else return min{myfunc(m-1,n), MyFunc(m-1, n-1), MyFunc(m, n-1)} + 1 c T(m-2,n) T(m-2,n-1) c T(m-2,n-1) T(m-2,n-2) c c T(m-1,n-2) T(m-1,n-1)

19 Example 6: What is the runtime? MyFunc(int m, n) // Assume m and n are approx. equal if ((m = 0) or (n = 0)) then return m else return min{myfunc(m-1,n), MyFunc(m-1, n-1), MyFunc(m, n-1)} + 1 h = O(n) or O(m) T(m-2,n) T(m-2,n-1) c c c c T(m-2,n-1) T(m-2,n-2) T(m-1,n-2) c 3c 9c T(m-1,n-1) 3 i c =O(3 n ) = O(2 n )

20 The problem of sorting Input: sequence < a1, a2,..., an > of numbers < a 1, a 2,..., a n > Output: permutation such that a a a n Example: Input: Output:

21 Insertion sort pseudocode INSERTION-SORT(A,n) A[1.. n] for j 2 to n do key A[j] i j 1 while i > 0 and A[i] > key do A[i +1] A[i] i i 1 A[i +1] = key A: 1 i j n sorted key

22 Example of insertion sort done

23 Insertion sort analysis Worst case: Input reverse sorted n 2 ( ) ( ) ( ) Tn= Θ j=θ n j= 2 1 [Recall arithmetic series: ] k = 1 Average case: All permutations equally likely n k = n n+ =Θ n 2 2 ( 1) ( ) n 2 ( ) ( /2) ( ) Tn= Θ j =Θ n j= 2 Is insertion sort a fast sorting algorithm? Moderately so, for small n Not at all, for large n

24 Merge sort MERGE-SORT A[1..n] 1. If n = 1, we re done. 2. Recursively sort A[ 1.. n/2 ] and A[ n/ n ] 3. Merge the 2 sorted arrays Subroutine MERGE From the head of each array, output the smallest value and increment the pointer to that array; continue until reach end of both arrays

25 Merging two sorted arrays

26 Merging two sorted arrays

27 Merging two sorted arrays

28 Merging two sorted arrays

29 Merging two sorted arrays Etc Time = Θ(n) to merge a total of n elements (i.e., linear time)

30 Analyzing merge sort T(n) Θ(1) 2 T(n/2) Θ(n) MERGE-SORT A[1..n] 1.If n = 1, we re done. 2.Recursively sort A[ 1.. n/2 ] and A[ n/ n ] 3. Merge the 2 sorted arrays Sloppiness: Should be T[ n/2 ] + T[ n/2 ], but it turns out not to matter asymptotically Means constant time

31 Recurrence for merge sort Recurrence: an equation defined in terms of itself For merge sort: Tn ( ) Θ (1) if n = 1 = 2 Tn ( /2) + Θ ( n) if n> 1 We usually omit stating the base case when T(n) = Θ(1) for sufficiently small n, but only when it has no effect on the asymptotic solution to the recurrence

32 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 T(n)

33 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 T(n/2) T(n/2)

34 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 /2 /2 T(n/4) T(n/4) T(n/4) T(n/4)

35 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 /2 /2 Θ(1)

36 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 h = lg n /2 /2 Θ(1)

37 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 h = lg n /2 /2 Θ(1)

38 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 h = lg n /2 /2 Θ(1)

39 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 h = lg n /2 /2 Θ(1)

40 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 h = lg n /2 /2 Θ(1) #leaves = n Θ(n)

41 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 h = lg n /2 /2 Θ(1) #leaves = n Θ(n)

42 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 h = lg n /2 /2 Θ(1) #leaves = n Θ(n)

43 Solving recurrences: The recursion tree Solve T(n) = 2T(n/2) +, for constant c >0 h = lg n /2 /2 Θ(1) #leaves = n Θ(n) Total = Θ(n lg n)

44 Differences in asymptotic complexity Θ(n lg n) grows more slowly than Θ(n 2 ) Therefore, merge sort asymptotically beats insertion sort in the worst case In practice, merge sort beats insertion sort for n > 30 (or so)

45 To get a feel for differences in run time, watch this animation Compare Θ(n 2 ) algorithm with Θ(n log n) (and algorithms run on sequential versus parallel machines):

46 Example int magic (int A[], int first, int last) { if (first == last) return A[first]; int middle = (last+first) / 2; int result1 = magic (A[], first, middle); int result2 = magic (A[], middle+1, last); return (result1 < result2)? result1 : result2; } What is n for this problem? The size of A, which is last - first What is the T(N) recurrence for this problem? T(N) = 2T(n/2) + c

47 Options for solving recurrences Recursion tree Proof by induction (beyond scope of this class) Master method (beyond scope of this class) Solves recurrences of the form: Tn ( ) = atnb ( / ) + f ( n) Learn common solutions, such as: Recurrence: Solution: Tn ( ) = Tn ( /2) + c Tn ( ) = O(log n) Tn ( ) = Tn ( /2) + n Tn ( ) = On ( ) Tn ( ) = 2 Tn ( /2) + c Tn ( ) = On ( ) Tn ( ) = 2 Tn ( /2) + n Tn ( ) = On ( log n) Tn ( ) = Tn ( 1) + c Tn ( ) = On ( )

48 Typical running times of algorithms Θ(1): The running time of the program is constant (i.e., independent of the size of the input) Example: Find the 5 th element in an array Θ(log n): Typically achieved by dividing the problem into smaller segments and only looking at one input element in each segment Example: Binary search Θ(n): Typically achieved by examining each input element once Example: Find the minimum element

49 Typical running times of algorithms (con t.) Θ(n log n): Typically achieved by dividing the problem into subproblems, solving the subproblems independently, and then combining the results. Here, each element in subproblem must be examined. Example: Merge sort Θ(n 2 ): Typically achieved by examining all pairs of data elements Example: Insertion sort Θ(n 3 ): Often achieved by combining algorithms with a mixture of the previous running times Example: Matrix Multiplication

50 Efficient Exponentiation Obvious algorithm for computing x n uses n-1 multiplications Better way: Observation: N even x n = x n/2 x n/2 Runtime? N odd x n = x (n-1/2) x (n-1/2) x O(log n) long pow( long x, int n) { if (n == 0) return 1; if (n == 1) return x; if (iseven(n)) return pow(x*x, n/2); else return pow (x*x, n/2) * x; } What calculations are done for x 62? x 62 = (x 31 ) 2 x 31 = (x 15 ) 2 x x 15 = (x 7 ) 2 x x 7 = (x 3 ) 2 x x 3 = (x 2 )x = 9 multiplications

51 Considerations in choosing among competing algorithms 1. Which is fastest? Asymptotic analysis only provides a rough guide. Often, you have to implement the algorithms to determine which is faster 2. Which is simplest to implement? Often, if 2 algorithms have comparable running times, but one is far easier to implement and maintain, the simpler algorithm will be chosen 3. How often will the algorithm be executed? Programmer time may be more important than computer time 4. What is the size of the input? If input size is small, algorithm with a higher time complexity but smaller constants may be preferrable