Data Structures and Algorithms

Transcription

1 Berner Fachhochschule - Technik und Informatik Data Structures and Algorithms Topic 1: Algorithm Analysis Philipp Locher FS 2018

2 Outline Course and Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Topic 1: Algorithm Analysis Page 2

3 Outline Course and Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Topic 1: Algorithm Analysis Page 3

4 Evaluation Project (50%) In groups of two Java implementation of selected topics Implementation and short presentation Written exam, 1.5h (50%) Last week in semester Open book E: > 50% and project, exam > 20% 4 ECTS Topic 1: Algorithm Analysis Page 4

5 Topic 1: Algorithm Analysis Page 5

6 Algorithm Design I Part I. Fundamental Tools 1. Algorithm Analysis 1.1 Methodologies for Analyzing Algorithms 1.2 Asymptotic Notation 1.3 A Quick Mathematical Review 1.4 Case Studies in Algorithm Analysis 1.5 Amortization 1.6 Experimentation 1.7 Exercises 2. Basic Data Structures 2.1 Stacks and Queues 2.2 Vectors, Lists, and Sequences 2.3 Trees 2.4 Priority Queues and Heaps 2.5 Dictionaries and Hash Tables 2.6 Java Example: Heap 2.7 Exercises 3. Search Trees and Skip Lists Topic 1: Algorithm Analysis Page 6

7 Algorithm Design II 3.1 Ordered Dictionaries and Binary Search Trees 3.2 AVL Trees 3.3 Bounded-Depth Search Trees 3.4 Splay Trees 3.5 Skip Lists 3.6 Java Example: AVL and Red-Black Trees 3.7 Exercises 4. Sorting, Sets, and Selection 4.1 Merge-Sort 4.2 The Set Abstract Data Type 4.3 Quick-Sort 4.4 A Lower Bound on Comparison-Based Sorting 4.5 Bucket-Sort and Radix-Sort 4.6 Comparison of Sorting Algorithms 4.7 Selection 4.8 Java Example: In-Place Quick-Sort 4.9 Exercises 5. Fundamental Techniques 5.1 The Greedy Method Topic 1: Algorithm Analysis Page 7

8 Algorithm Design III 5.2 Divide-and-Conquer 5.3 Dynamic Programming 5.4 Exercises Part II. Graph Algorithms 1. Graphs 1.1 The Graph Abstract Data Type 1.2 Data Structures for Graphs 1.3 Graph Traversal 1.4 Directed Graphs 1.5 Java Example: Depth-First Search 1.6 Exercises 2. Weighted Graphs 2.1 Single-Source Shortest Paths 2.2 All-Pairs Shortest Paths 2.3 Minimum Spanning Trees 2.4 Java Example: Dijkstra s Algorithm 2.5 Exercises 3. Network Flow and Matching Topic 1: Algorithm Analysis Page 8

9 Algorithm Design IV 3.1 Flows and Cuts 3.2 Maximum Flow 3.3 Maximum Bipartite Matching 3.4 Minimum-Cost Flow 3.5 Java Example: Minimum-Cost Flow 3.6 Exercises Part III. Internet Algorithmics 1. Text Processing 1.1 Strings and Pattern Matching Algorithms 1.2 Tries 1.3 Text Compression 1.4 Text Similarity Testing 1.5 Exercises 2. Number Theory and Cryptography 2.1 Fundamental Algorithms Involving Numbers 2.2 Cryptographic Computations 2.3 Information Security Algorithms and Protocols 2.4 The Fast Fourier Transform Topic 1: Algorithm Analysis Page 9

10 Algorithm Design V 2.5 Java Example: FFT 2.6 Exercises 3. Network Algorithms 3.1 Complexity Measures and Models 3.2 Fundamental Distributed Algorithms 3.3 Broadcast and Unicast Routing 3.4 Multicast Routing 3.5 Exercises Part IV. Additional Topics 1. Computational Geometry 1.1 Range Trees 1.2 Priority Search Trees 1.3 Quadtrees and k-d Trees 1.4 The Plane Sweep Technique 1.5 Convex Hulls 1.6 Java Example: Convex Hull 1.7 Exercises 2. NP-Completeness Topic 1: Algorithm Analysis Page 10

11 Algorithm Design VI 2.1 P and NP 2.2 NP-Completeness 2.3 Important NP-Complete Problems 2.4 Approximation Algorithms 2.5 Backtracking and Branch-and-Bound 2.6 Exercises 3. Algorithmic Frameworks 3.1 External-Memory Algorithms 3.2 Parallel Algorithms 3.3 Online Algorithms 3.4 Exercises Topic 1: Algorithm Analysis Page 11

12 Outline Course and Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Topic 1: Algorithm Analysis Page 12

13 Algorithms An algorithm is a step-by-step procedure for solving a problem in a finite amount of time Most algorithms transform input objects into output objects The running time (and the memory consumption) of an algorithm typically grows with the input size The average case running time is often difficult to determine We focus on the worst case running time Easier to analyse Crucial to applications such as games, finance, robotics Sometimes, it is also worth to study the best case running time Topic 1: Algorithm Analysis Page 13

14 Running Time: Example Running Time Best case Average case Worst case Input Size Topic 1: Algorithm Analysis Page 14

15 Problem Classes I There are feasible problems, which can be solved by an algorithm efficiently Examples: sorting a list, draw a line between two points, decide whether n is prime, etc. There are computable problems, which can be solved by an algorithm, but not efficiently Example: finding the shortest solution for large sliding puzzles There are undecidable problems, which can not be solved by an algorithm Example: for a given a description of a program and an input, decide whether the program finishes running or will run forever (halting problem) Topic 1: Algorithm Analysis Page 15

16 Problem Classes II All Problems Computable Problems Feasible Problems Topic 1: Algorithm Analysis Page 16

17 Experimental Studies There are two ways to analyse the running time (complexity) of an algorithm: Experimental studies Theoretical analysis Experiments usually involve the following major steps: Write a program implementing the algorithm Run the program with inputs of varying size n and composition Use a method like System.currentTimeMillis() to get an accurate measure of the actual running time t(n) Plot the results Topic 1: Algorithm Analysis Page 17

18 Experimental Studies: Example Running Time (ms) Input Size Topic 1: Algorithm Analysis Page 18

19 Experimental Setup I To setup and conduct an experiment, perform each of the following steps with care 1. Choosing the question Estimate the asymptotic running time in the average case Test which of two algorithm is faster If an algorithm depends on parameters, find the best values For approximation algorithms, evaluate the quality of the approximation 2. Deciding what to measure Running time (system dependent) Number of memory references Number of primitive (e.g. arithmetic) operations Number of calls of a specific function 3. Generating test data Enough samples for statistically significant results Topic 1: Algorithm Analysis Page 19

20 Experimental Setup II Samples of varying sizes Representative of the kind of data expected in practice 4. Coding the solution and performing the experiment Reproducible results Keep note of the details of the computational environment 5. Evaluating the test results statistically Topic 1: Algorithm Analysis Page 20

21 Limits of Experiments It is necessary to implement the algorithm, which may be difficult Results may not be indicative of the running time on other inputs not included in the experiment The running times may depend strongly on the chosen setting (hardware, software, programming language) In order to compare two algorithms, the same hardware and software environments must be used Topic 1: Algorithm Analysis Page 21

22 Theoretical Analysis Uses a high-level description of the algorithm instead of an implementation Describes the running time (complexity) as a function of the input size n: T (n) = number of primitive operations Allows to evaluate the speed of an algorithm independently of hardware software programming language actual implementation Takes into account all possible inputs Topic 1: Algorithm Analysis Page 22

23 Outline Course and Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Topic 1: Algorithm Analysis Page 23

24 Pseudo-Code Pseudo-code allows a high-level description of an algorithm More structured than English prose, but less detailed than a program Preferred notation for describing algorithms Hides program design issues and details of a particular language Topic 1: Algorithm Analysis Page 24

25 Pseudo-Code: Example Find the maximum element of an array: Algorithm arraymax(a, n) Input: array A of n integers Output: maximum element of A currentmax A[0] for i 1 to n 1 do if A[i] > currentmax then currentmax A[i] return currentmax Topic 1: Algorithm Analysis Page 25

26 Pseudo-Code: Example (simplified) Find the maximum element of an array: Algorithm arraymax(a, n) currentmax A[0] for i 1 to n 1 do if A[i] > currentmax then currentmax A[i] return currentmax Topic 1: Algorithm Analysis Page 26

27 Details of Pseudo-Code I The rules of using pseudo-code are very flexible, but following some guidelines is recommended Declaration Algorithm algorithmname(arg 1, arg 2,...) Input:... Output:... Control flow if... then... [else...] while... do... repeat... until... for... do... Indentation replaces braces {} resp. begin... end Topic 1: Algorithm Analysis Page 27

28 Details of Pseudo-Code II Expressions Assignments: Equality testing: = Superscript like n 2 and other mathematical notation is allowed Array access: A[i 1] Calling another algorithm algorithmname(arg 1, arg 2,...) object.algorithmname(arg 1, arg 2,...), which is equivalent to algorithmname(object, arg 1, arg 2,...) Return value return expression Topic 1: Algorithm Analysis Page 28

29 The Random Access Machine Model To study algorithms in pseudo-code analytically, we need a theoretical model of a computing machine A random access machine consists of a CPU a bank of memory cells, each of which can hold an arbitrary number (of any type and size) or character The size of the memory is unlimited Memory cells are numbered and accessing any memory cell takes 1 unit time (= random access) All primitive operations of the CPU take 1 unit time Topic 1: Algorithm Analysis Page 29

30 Primitive Operations The primitive operations of a random access machine are the once we allow in a pseudo-code algorithm Reading the value of a variable Assigning a value to a variable Evaluating an expression (arithmetic operations, comparisons) Indexing into an array Calling a method/function Returning from a method/function By inspecting the pseudo-code, we can count the number of primitive operations as a function of the input size T (n) = number of primitive operations for input of size n Largely independent from the programming language Topic 1: Algorithm Analysis Page 30

31 Primitive Operations: Example What is the running time of arraymax? Algorithm arraymax(a, n) currentmax A[0] for i 1 to n 1 do if A[i] > currentmax then currentmax A[i] return currentmax Topic 1: Algorithm Analysis Page 31

32 Primitive Operations: Example Algorithm arraymax(a, n) currentmax A[0] for i 1 to n 1 do if A[i] > currentmax then currentmax A[i] return currentmax # Operations (n 1) 3 (n 1) 2 Hence, the running time of arraymax is T (n) = 7n 2. Or did we forget something? Topic 1: Algorithm Analysis Page 32

33 Primitive Operations: Example (completed) Algorithm arraymax(a, n) currentmax A[0] for i 1 to n 1 do {i n 1} if A[i] > currentmax then currentmax A[i] {i i + 1} return currentmax # Operations n 4 (n 1) 3 (n 1) 3 (n 1) 2 Hence, the running time of arraymax is T (n) = 14n 5. Topic 1: Algorithm Analysis Page 33

34 Worst Case vs. Best Case For a given input size n, the running time of an algorithm may depend on other aspects of the input Sorting a list of size n (the list may already be sorted) Finding an element in a list of size n (the element may be at the beginning of the list) Usually, if an algorithm contains if-then-else statements, it can have different running times Worst case: maximal number of primitive operations Best case: minimal number of primitive operations Example: arraymax Worst case: T (n) = 14n 5 Best case: T (n) = 11n 2 Topic 1: Algorithm Analysis Page 34

35 Outline Course and Textbook Overview Analysis of Algorithm Pseudo-Code and Primitive Operations Growth Rate and Big-Oh Notation Topic 1: Algorithm Analysis Page 35

36 Math You Need to Review Logarithms/exponentials: a = b c log b a = c b log b a = a Properties of logarithms log b ac = log b a + log b c a log b c = log b a log b c log b a c = c log b a Properties of exponentials b a+c = b a b c b a c = (b a )/(b c ) b ac = (b a ) c = (b c ) a b (ac ) Base change: log b a = log c a log c b and bc = a c log a b Computer science: base b = 2, i.e. log a = log 2 a Topic 1: Algorithm Analysis Page 36

37 Estimating Actual Running Times In reality, the time to execute a primitive operation differs Let a be the time taken by the fastest primitive operation Let b be the time taken by the slowest primitive operation If t(n) denotes the actual running time of the algorithm on a concrete machine for a problem of size n, we get a T (n) t(n) b T (n) Changing the hardware/software environment Affects the constants a and b Therefore, affects t(n) only by a constant factor Topic 1: Algorithm Analysis Page 37

38 Asymptotic Growth Rates of Running Times The constants a and b do not change the asymptotic growth rate of t(n) Therefore, we can consider the asymptotic growth rate of T (n) as being the characteristic measure for evaluating an algorithm s running time We will use the terms number of primitive operations and running time interchangeably and denote them by T (n) The asymptotic growth rate of T (n) is an intrinsic property of the algorithm which is independent of its implementation Topic 1: Algorithm Analysis Page 38

39 Running Time Examples n 2n 2 10n log(n) 20 n 10 n log(n) Topic 1: Algorithm Analysis Page 39

40 Polynomial Running Times Feasible algorithms are those with polynomial (or better) running times: T (n) n d, d > 0 Linear: T (n) n Quadratic: T (n) n 2 Cubic: T (n) n 3 The asymptotic growth rate is not affected by constants, factors, or lower-order terms T (n) = 15n + 8 is linear T (n) = 100n is quadratic T (n) = 5n 3 + 5n 2 2n + 35 is cubic In a log-log chart, a polynomial running time turns out to be a straight line Topic 1: Algorithm Analysis Page 40

41 Log-Log Chart superpolynomial cubic Running Time T(n) quadratic linear sub- Polynomial Input Size n Topic 1: Algorithm Analysis Page 41

42 Constants and Factors Running Time T(n) n n 100n Input Size n Topic 1: Algorithm Analysis Page 42

43 Big-Oh Notation The so-called big-oh notation is useful to better compare and classify the asymptotic growth rates of different functions Given functions f (n) and g(n), we say that f (n) is O(g(n)), if there is a constant c > 0 and an integer n 0 1 such that Example 1: 2n + 10 is O(n) f (n) c g(n), for n n 0 2n + 10 c n n 10/(c 2), e.g. pick c = 3 and n 0 = 10 Example 2: n 2 is not O(n) n 2 c n n c, which can not be satisfied since c is a constant Topic 1: Algorithm Analysis Page 43

44 Big-Oh Example Running Time T(n) n +10 3n n Input Size n Topic 1: Algorithm Analysis Page 44

45 Big-Oh Example n n Running Time T(n) n n Input Size n Topic 1: Algorithm Analysis Page 45

46 Typical Running Times Constant: O(1) Logarithmic: O(log n) Linear: O(n) Quadratic: O(n 2 ) Cubic: O(n 3 ) Polynomial: O(n d ), d > 0 Exponential: O(b n ), b > 1 Others: O( n), O(n log n) Growth rate order: 1 log n n n n log n n 2 n 3 2 n 3 n Topic 1: Algorithm Analysis Page 46

47 Simplification Rules Constants, factors, and lower-order terms can be dropped 1. If f is a function and c > 0 a constant, then O(c f ) = O(f ) 2. If f and g are functions, then O(f + g) = max(o(f ), O(g)) 3. If f (n) = a d n d a 2 n 2 + a 1 n + a 0 is a polynomial of degree d, then f (n) is O(n d ) Example: f (n) = 5n 4 + 2n log n 3 is O(n 4 ) Indicate the smallest possible function, e.g., 2n + 3 is O(n) instead of 2n + 3 is O(2n) 4n 3 + 3n is O(n 3 ) instead of 4n 3 + 3n is O(n 3 + n) Topic 1: Algorithm Analysis Page 47

48 Growth Rate Comparison n log n n n n log n n 2 n 3 2 n ,096 65, ,024 32, , , ,384 2,097, ,048 65,536 16,777, , , , ,024 10,240 1,048, Atoms in the universe: ca Topic 1: Algorithm Analysis Page 48

49 The Importance of Asymptotics 1 operation/µs 256 ops./µs Running Time Maximum Problem Size n New Size 400n 2, , 000 9, 000, n 20n log n 4, , 666 7, 826, n log n 7+log n 2n , , n n n 2 n n second 1 minute 1 hour Topic 1: Algorithm Analysis Page 49

50 Summary The asymptotic analysis of an algorithm determines the running time (complexity) in big-oh notation To perform the asymptotic analysis Find the worst-case number of primitive operations T (n) Express this function with big-oh notation Example: arraymax We determine that algorithm arraymax executes at most T (n) = 14n 5 primitive operations We say that algorithm arraymax runs in O(n) time Since constant factors and lower-order terms are eventually dropped anyhow, we can disregard them when counting primitive operations Topic 1: Algorithm Analysis Page 50