Complexity. Alexandra Silva.

Size: px

Start display at page:

Download "Complexity. Alexandra Silva."

Loren Miles
6 years ago
Views:

1 Complexity Alexandra Silva Institute for Computing and Information Sciences 22nd April 2014 Alexandra 22nd April 2014 Lesson 1 1 / 47

2 This is a course about algorithms and complexity We will analyze algorithms and assess efficiency It is an essential skill for advanced programmers Alexandra 22nd April 2014 Lesson 1 2 / 47

3 Basic course information Teachers Alexandra (lectures) Tim (tutorials) Robin and Wouter (practical assignments) There are 7 lectures plus 7 tutorials. No lecture on 17 June and no tutorial on 18 April (Good Friday) and 29 May (some other holiday). Final grade 0.6 T P T = test and P = practical assignment Mark of test must be at least 5. Alexandra 22nd April 2014 Lesson 1 3 / 47

4 Basic course information Lectures Weekly, 2 hours, on Tuesdays 15:45 Presence not compulsory... but active attitude expected, when present Tutorials Weekly, 2 hours, on Fridays 15:45. Homework not compulsory. Practical assignment Evaluation with several tests (easy, difficult). There will be a prize for the best assignment! Alexandra 22nd April 2014 Lesson 1 4 / 47

5 Basic course information Website complexity-2014/ Doubts, complains, suggestions,... My office: HG , check availability: Alexandra 22nd April 2014 Lesson 1 5 / 47

6 Basic course information How to pass this course... Practice, practice, practice... You don t learn it it by just staring at the slides - not a spectator sport! Test questions will be in line with exercises. Alexandra 22nd April 2014 Lesson 1 6 / 47

7 Basic course information Last but not least Alexandra 22nd April 2014 Lesson 1 7 / 47

8 Basic course information Last but not least You can fail for this course! (Statistics tells that some of you will) Alexandra 22nd April 2014 Lesson 1 7 / 47

9 Basic course information Last but not least You can fail for this course! (Statistics tells that some of you will) 3ec means 3 28 = 84 hours in total Let s say 20 hours for exam 64 hours for 8 weeks means: 8 hours per week! on average 4 hours for studying & making exercises Alexandra 22nd April 2014 Lesson 1 7 / 47

10 The importance of algorithms in Computer Science Alexandra 22nd April 2014 Lesson 1 8 / 47

11 Computer Science Alexandra 22nd April 2014 Lesson 1 9 / 47

12 Algorithms Algorithms = Human (high-level) way of teaching a computer what to do to solve a problem Alexandra 22nd April 2014 Lesson 1 10 / 47

13 One algorithm can change the world Alexandra 22nd April 2014 Lesson 1 11 / 47

14 One algorithm can change the world Why was the Google algorithm successful? Solved a problem search in an elegant, efficient, correct and scalable way. Alexandra 22nd April 2014 Lesson 1 11 / 47

15 Manao is traversing a valley inhabited by monsters. During his journey, he will encounter several monsters. The scariness of each monster is a number dread/scare. Manao is not going to fight the monsters (watje). Alexandra 22nd April 2014 Lesson 1 12 / 47

16 Instead, he will bribe some of them and make them join him. The monsters are not too greedy (price = 1 or 2). Alexandra 22nd April 2014 Lesson 1 13 / 47

17 Manao meets monster: bribe or not bribe? No bribe, not scary enough: monster will attack! Bribe: monster will join the party! So: if monster is stronger, Manao bribes. If monster is not strong: choice. Alexandra 22nd April 2014 Lesson 1 14 / 47

18 : example Dragon: bribe (always for the first monster). Hydra: Manao can either pass or bribe her. Killer Rabbit: 1 Bribed Hydra: total scariness = 13 > 10. Pass the rabbit! Total cost = = 2 coins. 2 Bribe the Rabbit. Total cost = = 3 coins. Alexandra 22nd April 2014 Lesson 1 15 / 47

19 : version 1 There are between 1 and 20 monsters. Their scariness level is between 1 and 2,000,000,000, inclusive. The bribe for each monster will be either 1 or 2. Alexandra 22nd April 2014 Lesson 1 16 / 47

20 : version 1 There are between 1 and 20 monsters. Their scariness level is between 1 and 2,000,000,000, inclusive. The bribe for each monster will be either 1 or 2. Goal Minimize the amount of money Manao needs to pay. Alexandra 22nd April 2014 Lesson 1 16 / 47

21 How do we solve it? Monster 0: no option, we bribe. Monster 1: two options. 1 Not bribe: we are scary enough, money and scariness. 2 bribe: scariness and money spent.. Monster p: we have passed the first p monsters and we have a total scariness level of currentscare. How do we decide what to do? Alexandra 22nd April 2014 Lesson 1 17 / 47

22 How do we solve it? Monster p: Bribe or scare? We will compute the minimum price at step p given how scary we are M(p, CurrentScare) by analyzing our two options: 1 bribe: it will cost us price p, we earn scare p. P 1 = price p + M(p + 1, CurrentScare + scare p ) 2 scare: currentscare > scare p, not pay, not scarier. P 2 = M(p + 1, CurrentScare) Choose best option: M(p, CurrentScare) = min(p 1, P 2 ). Alexandra 22nd April 2014 Lesson 1 18 / 47

23 How do we solve it? In the solution: We were using the minimum price of the next monster recursive algorithm. Alexandra 22nd April 2014 Lesson 1 19 / 47

24 How do we solve it? In the solution: We were using the minimum price of the next monster recursive algorithm. When do we stop: p = n all monsters done, price is 0. Alexandra 22nd April 2014 Lesson 1 19 / 47

25 How efficient is this algorithm? Let s measure how many options we have: At each step we double our options. For n monsters we have }{{} n times Alexandra 22nd April 2014 Lesson 1 20 / 47

26 How efficient is this algorithm? } 2 2 {{ 2 } = 2 n n times For 20 monsters we have Alexandra 22nd April 2014 Lesson 1 21 / 47

27 How efficient is this algorithm? } 2 2 {{ 2 } = 2 n n times For 20 monsters we have Good or bad? Alexandra 22nd April 2014 Lesson 1 21 / 47

28 : version 2 There can be now up to 50 monsters Alexandra 22nd April 2014 Lesson 1 22 / 47

29 : version 2 Is our previous algorithm ok? Alexandra 22nd April 2014 Lesson 1 23 / 47

30 : version 2 Is our previous algorithm ok? 2 50 = VERY BIG NUMBER Alexandra 22nd April 2014 Lesson 1 23 / 47

31 : version 2 Is our previous algorithm ok? 2 50 = VERY BIG NUMBER Actually, with 2 50 grains of rice you can cover the whole of the Netherlands with ca 2 meters of rice. Alexandra 22nd April 2014 Lesson 1 23 / 47

32 : version 2 Can we do better? Alexandra 22nd April 2014 Lesson 1 24 / 47

33 : version 2 Can we do better? Different perspective: what is the maximum price to pay for n monsters? Alexandra 22nd April 2014 Lesson 1 24 / 47

34 : version 2 Can we do better? Different perspective: what is the maximum price to pay for n monsters? 2 n Alexandra 22nd April 2014 Lesson 1 24 / 47

35 : version 2 Can we do better? Different perspective: what is the maximum price to pay for n monsters? 2 n Instead of minimizing, ask... For each price p from 0 to 2 n, can we cross without paying more than p? For n 50 there are at most 101 values for p and we can answer the problem fast (in polynomial time). Alexandra 22nd April 2014 Lesson 1 24 / 47

36 : version 2 Can we do better? Different perspective: what is the maximum price to pay for n monsters? 2 n Instead of minimizing, ask... For each price p from 0 to 2 n, can we cross without paying more than p? For n 50 there are at most 101 values for p and we can answer the problem fast (in polynomial time). solution = chocolate bar! Alexandra 22nd April 2014 Lesson 1 24 / 47

37 Topcoder appeared in a programming match of TopCoder. Both problems were solved in under 4 minutes! Alexandra 22nd April 2014 Lesson 1 25 / 47

38 Lessons to take home The parameters of the problem matter. Solutions often need to be adapted. A lot of improvement can be done before implementation. Alexandra 22nd April 2014 Lesson 1 26 / 47

39 Lessons to take home The parameters of the problem matter. Solutions often need to be adapted. A lot of improvement can be done before implementation. It is all about asking the right questions! Alexandra 22nd April 2014 Lesson 1 26 / 47

40 Lessons to take home The parameters of the problem matter. Solutions often need to be adapted. A lot of improvement can be done before implementation. It is all about asking the right questions! A lot of Computer Science is like solving puzzles! Alexandra 22nd April 2014 Lesson 1 26 / 47

41 Analysis of algorithms Study of different features that allow classifying and comparing algorithms Alexandra 22nd April 2014 Lesson 1 27 / 47

42 Analysis of algorithms Study of different features that allow classifying and comparing algorithms One important feature: the correction of an algorithm. Does the algorithm actually do what we meant it to do? (we will not be dealing with that in this course.) Alexandra 22nd April 2014 Lesson 1 27 / 47

43 Analysis of algorithms Then, it is important to take into account the resources needed to execute an algorithm: memory bandwidth hardware... and of particular interest in this course: execution/computation time The latter is what we will be using as comparison measure between algorithms. Alexandra 22nd April 2014 Lesson 1 28 / 47

44 Analysis of algorithms Then, it is important to take into account the resources needed to execute an algorithm: memory bandwidth hardware... and of particular interest in this course: execution/computation time The latter is what we will be using as comparison measure between algorithms. The strategy used in an algorithm to perform a certain task is very important! Alexandra 22nd April 2014 Lesson 1 28 / 47

45 In a nutshell... What we will be doing for the next couple of lectures. Analysis of execution times: Computational models Analyze the best, worst and average case of an algorithm Analysis of recursive algorithms (recurrence relations) (Analysis of average case; randomized algorithms) Analysis of strategies: linear, incremental divide and conquer, (greedy algorithms, dynamic programming) Sorting (and searching) algorithms Case studies: linear search, binary search, insertion sort, mergesort, quicksort, radix sort, heap sort. Alexandra 22nd April 2014 Lesson 1 29 / 47

46 For the sake of clarity: what is an algorithm An algorithm is a well-defined procedure which takes an input value and returns an output value Alternative definitions a sequence of computational steps transforming an input value into an output value a tool to solve a well defined computational problem, which defined the relation between input and output... Alexandra 22nd April 2014 Lesson 1 30 / 47

47 Why are algorithms important? Typical areas of application: Internet: routing, searching, etc Security and cryptography Operations research (have you wondered how TNT programs their deliveries?) Maps (TomTom and such) Alexandra 22nd April 2014 Lesson 1 31 / 47

48 Why are algorithms important? Typical areas of application: Internet: routing, searching, etc Security and cryptography Operations research (have you wondered how TNT programs their deliveries?) Maps (TomTom and such) Examples of hard problems how to find the shortest/fastest path from N (Nijmegen) to A (Amsterdam) in a given map? how to visit all the cities in the Netherlands in the shortest way possible? given a set of matrices (possibly not square), how to compute their multiplication? Alexandra 22nd April 2014 Lesson 1 31 / 47

49 Why are efficient algorithms important? For a given problem, the ultimate quest is to find the most efficient algorithm. Alexandra 22nd April 2014 Lesson 1 32 / 47

50 Why are efficient algorithms important? For a given problem, the ultimate quest is to find the most efficient algorithm. For harder problems, no efficient solution is known. These algorithms are NP (another course). Alexandra 22nd April 2014 Lesson 1 32 / 47

51 Why are efficient algorithms important? For a given problem, the ultimate quest is to find the most efficient algorithm. For harder problems, no efficient solution is known. These algorithms are NP (another course). A subclass of NP which is interesting is the class of NP-complete problems, because: The description of the problem is simple; they arise usually in common and important application areas; it is not know whether there is an efficient algorithm to solve them; if someone finds an efficient solution for one of these problems, then all of them are solvable efficiently! when (trying to) solves an NP-complete problem, solutions that come close to the ideal solution are enough, a long as computed in reasonable time. Alexandra 22nd April 2014 Lesson 1 32 / 47

52 Why is analysis of algorithms important? Why should we care to learn how to design and analyze algorithms? Even though many problems have been solved efficiently, the rapid change in the world (think of the amount of info) requires for constant improvement in the existing algorithms and design of new, more efficient algorithms; If we would have infinite resources (memory, processing speed,... ) then any solution would be fine; but in reality we have to optimize consumption and minimize the resource consumption (we have to be zuinig)! Alexandra 22nd April 2014 Lesson 1 33 / 47

53 Why is analysis of algorithms important? Why should we care to learn how to design and analyze algorithms? (c d) Algorithms for the same problem can have (crazily) different behaviors: good algorithm design can have more impact than the choice of hardware. In a nutshell: algorithms are important because the choices made at that level of devising a system can strongly affect its validity and scope of application. Alexandra 22nd April 2014 Lesson 1 34 / 47

54 Execution time analysis computational model To do this type of analysis we need to take to fix our setting: Which model should we use? What is the cost of the resources in that model Alexandra 22nd April 2014 Lesson 1 35 / 47

55 Execution time analysis computational model To do this type of analysis we need to take to fix our setting: Which model should we use? What is the cost of the resources in that model This is what we will use: Model: Random Access Machine (RAM). A generic computer, single processor, no concurrency (instructions executed sequentially) The algorithms are implemented as programs in such a simplified computer Alexandra 22nd April 2014 Lesson 1 35 / 47

56 Execution time analysis computational model We will not specify the model completely; we will be reasonable when it comes to using it... A computer does not have complicated functions as search or sort but more primitive/atomic ones such as: arithmetic operations: sum, multiplication,... basic data manipulation: store, access, copy,... basic control structures: if-then-else, loops,... For the purpose of this course, all these primitive operations execute in constant time. Alexandra 22nd April 2014 Lesson 1 36 / 47

57 Execution time analysis computational model Talking about being reasonable: Do you think exponentiation is executed in constant time? The RAM computational model does not take into account the memory hierarchy computers have (cache, virtual memory), however the analysis using this model is (almost) always a good representative. Alexandra 22nd April 2014 Lesson 1 37 / 47

58 Execution time analysis computational model Dimension of the input depends on the sort of problem being analyzed: sorting a list/vector: number of elements multiplying two integers: total number of bits used in the number representation shortest path in a graph: the input has two items (V,E) vertices and edges Execution time: number of primitive/atomic operations executed Operations are defined independently of a specific machine Do you think call of a sub-routine is an atomic operation? Alexandra 22nd April 2014 Lesson 1 38 / 47

59 In a nutshell blackboard... Alexandra 22nd April 2014 Lesson 1 39 / 47

60 In a nutshell blackboard... blue: 0.5n 2 + n black: 0.5n 2 red: 2 n green: lg(n) Alexandra 22nd April 2014 Lesson 1 39 / 47

61 Impact of complexity Alexandra 22nd April 2014 Lesson 1 40 / 47

62 Example: Looking up a word in a dictionary Alexandra 22nd April 2014 Lesson 1 41 / 47

63 Example: Looking up a word in a dictionary Alexandra 22nd April 2014 Lesson 1 42 / 47

64 How to compare execution functions We define classes of functions. Alexandra 22nd April 2014 Lesson 1 43 / 47

65 How to compare execution functions We define classes of functions. Intuitively: f O(g): f does not grow faster than g. e.g T (n) = 4n 2 2n + 2 Alexandra 22nd April 2014 Lesson 1 43 / 47

66 How to compare execution functions We define classes of functions. Intuitively: f O(g): f does not grow faster than g. e.g T (n) = 4n 2 2n + 2is in O(n 2 ). Alexandra 22nd April 2014 Lesson 1 43 / 47

67 How to compare execution functions We define classes of functions. Intuitively: f O(g): f does not grow faster than g. e.g T (n) = 4n 2 2n + 2is in O(n 2 ). More formally: means that there exists a point x 0 where function f is below cg. Alexandra 22nd April 2014 Lesson 1 43 / 47

68 How to compare execution functions Figure: Example: f O(g). Important: need to provide x 0 and c! (exercises) Alexandra 22nd April 2014 Lesson 1 44 / 47

69 Algorithms: Searching Alexandra 22nd April 2014 Lesson 1 45 / 47

70 Example I : Linear search in a vector int linsearch(int *v, int a, int b, int k) { int i; i=a; while ((i<=b) && (v[i]!=k)) i++; if (i>b) return -1; else return i; } Alexandra 22nd April 2014 Lesson 1 46 / 47

71 Example I : Linear search in a vector int linsearch(int *v, int a, int b, int k) { int i; i=a; c 1 while ((i<=b) && (v[i]!=k)) c 2 i++; c 3 if (i>b) c 4 return -1; c 5 else return i; c 5 } Cost Alexandra 22nd April 2014 Lesson 1 46 / 47

72 Example I : Linear search in a vector Cost int linsearch(int *v, int a, int b, int k) { int i; i=a; c 1 while ((i<=b) && (v[i]!=k)) c 2 i++; c 3 if (i>b) c 4 return -1; c 5 else return i; c 5 } m is the number of time i++ executes. repetitions 1 m+1 m Alexandra 22nd April 2014 Lesson 1 46 / 47

73 Example I : Linear search in a vector Cost repetitions int linsearch(int *v, int a, int b, int k) { int i; i=a; c 1 while ((i<=b) && (v[i]!=k)) c 2 i++; c 3 if (i>b) c 4 return -1; c 5 else return i; c 5 } 1 m+1 m m is the number of time i++ executes. Obs1: These value depends on the while loop condition being true the position i is in-between a and b, that is: 0 m b a + 1. Alexandra 22nd April 2014 Lesson 1 46 / 47

74 Total Execution Time T (n) = c 1 + c 2 (m + 1) + c 3 m + c 4 + c 5 Alexandra 22nd April 2014 Lesson 1 47 / 47

75 Total Execution Time T (n) = c 1 + c 2 (m + 1) + c 3 m + c 4 + c 5 For a given n (the size of the input to search, n = b a + 1), the value T (n) can vary: why? Alexandra 22nd April 2014 Lesson 1 47 / 47

76 Total Execution Time T (n) = c 1 + c 2 (m + 1) + c 3 m + c 4 + c 5 For a given n (the size of the input to search, n = b a + 1), the value T (n) can vary: why? Depends on when the element is found. Alexandra 22nd April 2014 Lesson 1 47 / 47

77 Total Execution Time T (n) = c 1 + c 2 (m + 1) + c 3 m + c 4 + c 5 For a given n (the size of the input to search, n = b a + 1), the value T (n) can vary: why? Depends on when the element is found. Best case scenario: The value k is in the first position (k = v[a]). Then: T (n) = (c 1 + c 2 + c 4 + c 5 ) T(n) is constant Worst case scenario: the value in not found. T (n) = c 1 +c 2 (n+1)+c 3 (n)+c 4 +c 5 = (c 2 +c 3 )n+(c 1 +c 2 +c 4 +c 5 ) T(n) is linear Alexandra 22nd April 2014 Lesson 1 47 / 47

Complexity. Alexandra Silva.

Complexity. Alexandra Silva. Complexity Alexandra Silva alexandra@cs.ru.nl http://www.cs.ru.nl/~alexandra Institute for Computing and Information Sciences 6th February 2013 Alexandra 6th February 2013 Lesson 1 1 / 39 Introduction