Computational Complexity II: Asymptotic Notation and Classifica

Transcription

1 Computational Complexity II: Asymptotic Notation and Classification Algorithms Seminar on Theoretical Computer Science and Discrete Mathematics Aristotle University of Thessaloniki

2 Context 1 2 3

3 Computational Complexity Computational Complexity of an algorithm A:

4 Computational Complexity Computational Complexity of an algorithm A: time, space (memory)

5 Computational Complexity Computational Complexity of an algorithm A: time, space (memory) worst, average, best case

6 Computational Complexity Input snapshot size n:

7 Computational Complexity Input snapshot size n: #bits for the representation input data to the memory

8 Computational Complexity Input snapshot size n: #bits for the representation input data to the memory number of basic components which constitute the size and difficulty measure of snapshot (i.e., vetrices and edges of a graph)

9 Computational Complexity Computational Complexity of the problem P:

10 Computational Complexity Computational Complexity of the problem P: Complexity (worst case) best algorithm which solves the problem P.

11 Algorithm Design Valuation of computational complexity

12 Algorithm Design Valuation of computational complexity worst case

13 Algorithm Design Valuation of computational complexity worst case Running time guarantees for any input of size n.

14 Algorithm Design Valuation of computational complexity worst case Running time guarantees for any input of size n. - Generally captures efficiency in practice. - Draconian view, but hard to find effective alternative

15 Algorithm Design average case

16 Algorithm Design average case the performance of an algorithm averaged over random instances can sometimes provide considerable insight.

17 Algorithm Design best case

18 Algorithm Design best case The term best case performance is used to describe an algorithm s behavior under optimal conditions.

19 Algorithm Design best case The term best case performance is used to describe an algorithm s behavior under optimal conditions. Best solution depending on application requirements.

20 Algorithm Design best case The term best case performance is used to describe an algorithm s behavior under optimal conditions. Best solution depending on application requirements. Average performance and worst-case performance are the most used in algorithm analysis.

21 Asymptotic Valuation Running time of the algorithm A: Increasing function of T(n) that expresses in how much time is completed A when is applied in snapshot of size n.

22 Asymptotic Valuation Running time of the algorithm A: Increasing function of T(n) that expresses in how much time is completed A when is applied in snapshot of size n. We are interested in the size class T(n) Size class is intrinsic property of algorithm.

23 Asymptotic Valuation Running time of the algorithm A: Increasing function of T(n) that expresses in how much time is completed A when is applied in snapshot of size n. We are interested in the size class T(n) Size class is intrinsic property of algorithm. - binary search logarithmic time - dynamic programming linear time

24 Asymptotic Valuation Running time of the algorithm A: Increasing function of T(n) that expresses in how much time is completed A when is applied in snapshot of size n. We are interested in the size class T(n) Size class is intrinsic property of algorithm. - binary search logarithmic time - dynamic programming linear time ignores stables & focuses on runtime size class

25 Asymptotic Upper Bounds Big-Oh notation T(n) is O(Sf ˆn S) if there exist constants c A 0 and n 0 C 0 such that T(n) B c f(n) n C n 0. Figure: f(x) > O(g(x))

26 Big-Oh notation Example T(n) = pn 2 + qn + r, p,q,r A 0

27 Big-Oh notation Example T(n) = pn 2 + qn + r, p,q,r A 0 We claim that any such function is O(n 2 ).

28 Big-Oh notation Example T(n) = pn 2 + qn + r, p,q,r A 0 We claim that any such function is O(n 2 ). n C 1, qn B qn 2 and r B rn 2 T(n) = pn 2 + qn + r B pn 2 + qn 2 + rn 2 = (p+q+r) n 2 = c n 2

29 Big-Oh notation O( f(n) ) is a set of functions - T(n) > O( f(n) ) ( ) - T(n) = O( f(n) ) ( x, )

30 Big-Oh notation O( f(n) ) is a set of functions - T(n) > O( f(n) ) ( ) - T(n) = O( f(n) ) ( x, ) Nonnegative functions: When using Big-Oh notation, we assume that the functions involved are (asymptotically) nonnegative.

31 Asymptotic Lower Bounds Big-Omega notation T(n) is Ω( f(n)) if there exist constants c A 0 and n 0 C 0 such that T(n) C c f(n) n C n 0. Figure: Big-Omega notation

32 Asymptotic Lower Bounds Example T(n) = pn 2 + qn + r, p,q,r A 0

33 Asymptotic Lower Bounds Example T(n) = pn 2 + qn + r, p,q,r A 0 Let s claim that T(n) = Ω(n 2 ).

34 Asymptotic Lower Bounds Example T(n) = pn 2 + qn + r, p,q,r A 0 Let s claim that T(n) = Ω(n 2 ). n C 0, T(n) = pn 2 + qn + r C pn 2 = c n 2

35 Asymptotically Tight Bounds Big-Theta notation T(n) is Θ( f(n) ) if there exist constants c 1 A 0, c 2 A 0 and n 0 C 0 such that c 1 f(n) B T(n) B c 2 f(n) n C n 0. Figure: Big-Theta notation

36 Big-Theta notation Example T(n) = pn 2 + qn + r, p,q,r A 0

37 Big-Theta notation Example T(n) = pn 2 + qn + r, p,q,r A 0 We saw that T(n) is both O(n 2 ) and Ω(n 2 ). T(n) = Θ(n 2 )

38 Relational Properties Reflexivity: O, Ω, Θ

39 Relational Properties Reflexivity: O, Ω, Θ Transitivity: O, Ω, Θ

40 Relational Properties Reflexivity: O, Ω, Θ Transitivity: O, Ω, Θ Symmetry: f(n) = Θ( g(n) ) g(n) = Θ( f(n) )

41 Relational Properties Reflexivity: O, Ω, Θ Transitivity: O, Ω, Θ Symmetry: f(n) = Θ( g(n) ) g(n) = Θ( f(n) ) Transpose Symmetry (Duality): f(n) = O( g(n) ) g(n) = Ω( f(n) )

42 Master Theorem Master Theorem If T(n) = a T( n b ) + O(nd ) for constants a A 0, b A 1, d C 0, then T ˆn Oˆn d if d A log b a Oˆn d logn if d log b a Oˆn logba if log b a Nice Trick for computing quickly the computational complexity.

43

44 Efficient algorithm Definition 1 An algorithm is efficient if, when implemented, it runs quickly on real input instances.

45 Efficient algorithm Definition 1 An algorithm is efficient if, when implemented, it runs quickly on real input instances. the omission of where, and how well real input instances scale

46 Efficient algorithm Definition 2 An algorithm is efficient if it achieves qualitatively better worst case performance, at an analytical level, than brute-force search.

47 Efficient algorithm Definition 2 An algorithm is efficient if it achieves qualitatively better worst case performance, at an analytical level, than brute-force search. What do we mean by qualitatively better performance?

48 Efficient algorithm Definition 2 An algorithm is efficient if it achieves qualitatively better worst case performance, at an analytical level, than brute-force search. What do we mean by qualitatively better performance? Definition 3 An algorithm is efficient if it has a polynomial running time.

49 Poly-time algorithm Desirable scaling property: When the input size doubles, the algorithm should only slow down by some constant factor C.

50 Poly-time algorithm Desirable scaling property: When the input size doubles, the algorithm should only slow down by some constant factor C. Definition An algorithm is poly-time if the above scaling property holds.

51 Poly-time algorithm Desirable scaling property: When the input size doubles, the algorithm should only slow down by some constant factor C. Definition An algorithm is poly-time if the above scaling property holds. There exists constants c A 0 and d A 0 such that on every input of size n, its running time is bounded by cn d primitive computational steps.

52 Why we care for the asymptotic bound of an algorithm?

53 Classification algorithms by their design methodology Brute-force (or exhaustive search)

54 Classification algorithms by their design methodology Brute-force (or exhaustive search) Divide and conquer

55 Classification algorithms by their design methodology Brute-force (or exhaustive search) Divide and conquer Dynamic programming

56 Classification algorithms by their design methodology Brute-force (or exhaustive search) Divide and conquer Dynamic programming Randomized algorithms

57 Brute-force search Definition Brute-force search is a very general problem-solving technique that consists of systematically enumerating all possible candidates for the solution and checking whether each candidate satisfies the problem s statement.

58 Brute-force search Definition Brute-force search is a very general problem-solving technique that consists of systematically enumerating all possible candidates for the solution and checking whether each candidate satisfies the problem s statement. (-) can end up doing far more work to solve a given problem than might do a more clever or sophisticated algorithm

59 Brute-force search Definition Brute-force search is a very general problem-solving technique that consists of systematically enumerating all possible candidates for the solution and checking whether each candidate satisfies the problem s statement. (-) can end up doing far more work to solve a given problem than might do a more clever or sophisticated algorithm (+) is often easier to implement than a more sophisticated one, because of this simplicity, sometimes it can be more efficient

60 Bubble Sort

61 Bubble Sort Input: array a with n elements For i = 1 to n For j = i + 1 to n If ( a[i] A a[j] ) then swap their values End - If End - For End - For

62 Bubble Sort Computational Complexity: O(n 2 )

63 Bubble Sort Computational Complexity: O(n 2 ) WHY???

64 Bubble Sort Computational Complexity: O(n 2 ) WHY??? not a practical sorting algorithm when n is large

65 Divide and Conquer Definition A divide and conquer algorithm works by recursively breaking down a problem into two or more sub-problems of the same or related type, until these become simple enough to be solved directly. The solutions to the sub-problems are then combined to give a solution to the original problem.

66 3 steps of divide and conquer

67 Mergesort Example T(n) = number of comparisons to mergesort an input of size n.

68 Mergesort Example T(n) = number of comparisons to mergesort an input of size n. Divide array into two halves (divide O(1)).

69 Mergesort Example T(n) = number of comparisons to mergesort an input of size n. Divide array into two halves (divide O(1)). Recursively sort each half (sort 2T( n 2 )).

70 Mergesort Example T(n) = number of comparisons to mergesort an input of size n. Divide array into two halves (divide O(1)). Recursively sort each half (sort 2T( n 2 )). Merge two halves to make sorted whole (merge O(n)).

71 Mergesort T ˆn 0 if n = 1 2T ˆ n 2 Oˆn otherwise

72 Mergesort T ˆn 0 if n = 1 2T ˆ n 2 Oˆn otherwise Master Theorem,

73 Mergesort T ˆn 0 if n = 1 2T ˆ n 2 Oˆn otherwise Master Theorem, Solution: T(n) = O(nlog(n))

74 Dynamic programming Divide and Conquer Break up a problem into independent sub-problems, solve each sub-problem, and combine solution to sub-problems to form solution to original problem.

75 Dynamic programming Divide and Conquer Break up a problem into independent sub-problems, solve each sub-problem, and combine solution to sub-problems to form solution to original problem. Dynamic programming Break up a problem into a series of overlapping sub-problems, and build up solutions to larger and larger sub-problems.

76 Applications Dynamic programming applications:

77 Applications Dynamic programming applications: Bioinformatics

78 Applications Dynamic programming applications: Bioinformatics Control Theory

79 Applications Dynamic programming applications: Bioinformatics Control Theory Information Theory

80 Applications Dynamic programming applications: Bioinformatics Control Theory Information Theory Cocke-Kasami-Younger for parsing context-free grammars.

81 Knapsack Problem

82 Knapsack Given n objects and a knapsack.

83 Knapsack Given n objects and a knapsack. Item i weights w i A 0 and has value v i A 0.

84 Knapsack Given n objects and a knapsack. Item i weights w i A 0 and has value v i A 0. Knapsack has capacity of W.

85 Knapsack Given n objects and a knapsack. Item i weights w i A 0 and has value v i A 0. Knapsack has capacity of W. Goal: fill knapsack so as to maximize total value.

86 Knapsack problem OPT(i, w) = max profit subset of items 1,..., i with weight limit w

87 Knapsack problem OPT(i, w) = max profit subset of items 1,..., i with weight limit w Case 1: OPT does not select item i.

88 Knapsack problem OPT(i, w) = max profit subset of items 1,..., i with weight limit w Case 1: OPT does not select item i. OPT selects best of 1, 2,..., i 1 using weight limit w.

89 Knapsack problem OPT(i, w) = max profit subset of items 1,..., i with weight limit w Case 1: OPT does not select item i. OPT selects best of 1, 2,..., i 1 using weight limit w. Case 2: OPT selects item i.

90 Knapsack problem OPT(i, w) = max profit subset of items 1,..., i with weight limit w Case 1: OPT does not select item i. OPT selects best of 1, 2,..., i 1 using weight limit w. Case 2: OPT selects item i. New weight limit = w w i.

91 Knapsack problem OPT(i, w) = max profit subset of items 1,..., i with weight limit w Case 1: OPT does not select item i. OPT selects best of 1, 2,..., i 1 using weight limit w. Case 2: OPT selects item i. New weight limit = w w i. OPT selects best of 1, 2,..., i 1 using this new weight limit.

92 Knapsack problem Objective function: OPT ˆi, w 0 if i 0 OPT ˆi 1, w if w i A w max OPT ˆi 1, w, v i OPT ˆi 1, w w i otherwise

93 Knapsack Problem Knapsack problem: bottom-up KNAPSACK (n, W,w 1,...,w n,v 1,...,v n ) for w = 0 to W M [0, w] Ð 0. for i = 1 to n for w = 0 to W if ( w i A w ) M [i, w] Ð M [i 1, w]. else M [i, w] Ð max M [i 1, w], v i + M [i 1, w w i ] return M [n, W]

94 Knapsack problem: running time Theorem There exists an algorithm to solve the knapsack problem with n items and maximum weight W in Θ(nW) time and Θ(nW) space.

95 Knapsack problem: running time Theorem There exists an algorithm to solve the knapsack problem with n items and maximum weight W in Θ(nW) time and Θ(nW) space. not polynomial in input size! (pseudo-polynomial)

96 Knapsack problem: running time Theorem There exists an algorithm to solve the knapsack problem with n items and maximum weight W in Θ(nW) time and Θ(nW) space. not polynomial in input size! (pseudo-polynomial) NP - complete problem,

97 Randomized Algorithms Definition A randomized algorithm is an algorithm that employs a degree of randomness as part of its logic.

98 Randomized Algorithms Definition A randomized algorithm is an algorithm that employs a degree of randomness as part of its logic. Randomization: Allow fair coin flip in unit time.

99 Randomized Algorithms Why randomize?

100 Randomized Algorithms Why randomize? Can lead to simplest, fastest, or only known algorithm for a particular problem!,

101 Randomized Algorithms There are two large classes of such algorithms:

102 Randomized Algorithms There are two large classes of such algorithms: Las Vegas: A randomized algorithm that always outputs the correct answer, it is just that there is a small probability of taking long to execute.

103 Randomized Algorithms There are two large classes of such algorithms: Las Vegas: A randomized algorithm that always outputs the correct answer, it is just that there is a small probability of taking long to execute. Monte Carlo: Sometimes we want the algorithm to always complete quickly, but allow a small probability error.

104 Randomized Algorithms Any Las Vegas algorithm can be converted into a Monte Carlo algorithm by outputting an arbitrary, possibly incorrect answer if it fails to complete within a specified time.

105 Randomized Algorithms Any Las Vegas algorithm can be converted into a Monte Carlo algorithm by outputting an arbitrary, possibly incorrect answer if it fails to complete within a specified time. Monte Carlo algorithm cannot be converted into a Las Vegas (i.e., approximation of π)

106 Monte Carlo vs Las Vegas π 4 nˆ cycle 4 nˆsquare

107 Next Computational Complexity Complexity Classes (i.e., P, N P) Some nice computational problems, Some reductions ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y / ƒˆy Y /

108 References J.Kleinberg, E.Tardos. Algorithm Design. Boston, Mass.: Pearson/Addison-Wesley, cop I.Manwlìpouloc, A.Papadìpouloc, K.TsÐqlac. JewrÐa kai Algìrijmoi Grˆfwn, Aj na: Ekd. Nèwn Teqnologi n, TsÐqlac, K., GoÔnarhc, A., Manwlìpouloc, I., SqedÐash kai anˆlush algorðjmwn. [hlektr. bibl.] Aj na:sôndesmoc Ellhnik n Akadhmaðk n Bibliojhk n. Diajèsimo sto: Domèc dedomènwn, Mpozˆnhc Panagi thc D, EKDOSEIS A. TZIOLA & UIOI A.E., 2006

109 Thank you!!!