IN101: Algorithmic techniques Vladimir-Alexandru Paun ENSTA ParisTech

Transcription

1 IN101: Algorithmic techniques Vladimir-Alexandru Paun ENSTA ParisTech License CC BY-NC-SA 2.0

2 Outline Previously on IN101 Python s anatomy Functions, modules, libraries Arrays and reference types Python Classes Complexity, search and sort Anatomy of an algorithm and practices Trees In this CM... Dynamic Programming

3 Algorithms

4 Writing an algorithm Simply writing a sequence of instructions is not sufficient to accomplish a certain task or respect its requirements 1 Devise the algorithm Creation of an algorithm - logical activity - can not be automated However e can use design strategies 2 Validate the algorithm Does it provide correct outputs for all possible (correct) inputs? Independent of programming language 3 Analyze the algorithm Determine the required amount of computing time and storage 4 Test the program Determine if the program written using the algorithm behaves property

5 C. Find an algorithm to solve it Simply writing a sequence of instructions is not sufficient to accomplish a certain task or respect its requirements Step 1: Obtain a description of the problem See the previous step and adapt it Step 2: Develop a high-level algorithm Step 3: Refine the algorithm by adding more detail Step 4: Review the algorithm

6 BinaryTreeNode Trees Outline Binary Trees BinaryTreeNode data : generic left : BinaryTreeNode right : BinaryTreeNode

7 BinaryTreeNode Trees Outline Binary Trees BinaryTreeNode data : generic left : BinaryTreeNode right : BinaryTreeNode root 3 5 X 9 X 1 X X 10 X X 4 X

8 BinaryTreeNode Trees Outline Binary Trees BinaryTreeNode data : generic left : BinaryTreeNode right : BinaryTreeNode class BinaryTreeNode : """ Node of a binary tree holding a value.""" def init (self, value, left, right) : """Create a binary tree node.""" self.value = value self.left = left self.right = right

9 Storage of a heap Trees Outline Binary Trees Use an array to hold the data. Store the root in position 1. We won t use index 0 for this implementation. For any node in position i its left child (if any) is in position 2i its right child (if any) is in position 2i + 1 its parent (if any) is in position i/2 (use integer division)

10 Storage of a heap Trees Outline Binary Trees

11 AVL Tree Trees Outline AVL Tree An AVL tree is a self-balancing binary search tree. Structural properties 1 Binary tree property (same as BST) 2 Order property (same as for BST) 3 Balance property: balance of every node is between -1 and 1 Need to keep track of height of every node and maintain balance as we perform operations.

12 Pros and Cons of AVL Trees Trees Outline AVL Tree Pro: All operations guaranteed O(log N) The height balancing adds no more than a constant factor to the speed of insertion allows Con: Space consumed by height field in each node Slower than ordinary BST on random data

13 Algorithmic Techniques Trees Outline AVL Tree Divide et impera Greedy Algorithms Dynamic Programming

14 Greedy Algorithms Greedy Algorithms A greedy algorithm for an optimization problem always makes the choice that looks best at the moment and adds it to the current sub-solution.

15 Greedy Algorithms Greedy Algorithms - how fast? Greedy algorithms don t always yield optimal solutions but, when they do, they re usually the simplest and most efficient algorithms available.

16 Greedy Algorithms Greedy Algorithms - Application Knapsack problem - a greedy algorithm for the fractional knapsack. We observe that greedy doesn t work for the 0-1 knapsack (which must be solved using DP).

17 Greedy Algorithms Greedy Algorithms - Application Knapsack problem - a greedy algorithm for the fractional knapsack. We observe that greedy doesn t work for the 0-1 knapsack (which must be solved using DP). The Knapsack Problem Maximize the value of items carried in a backpack. Items are defined by their weight and value.

18 Fractional Knapsack Greedy Algorithms Fractional Knapsack Problem : We can take a fraction of an item.

19 0-1 Knapsack Greedy Algorithms 0-1 Knapsack Problem : We can only take or leave an item. We can t take a fraction.

20 Greedy Algorithms Greedy solution for Fractional Knapsack Idea Sort items by decreasing cost per pound.

21 Greedy Algorithms Greedy solution for Fractional Knapsack Idea Sort items by decreasing cost per pound.

22 Greedy Algorithms Greedy solution for Fractional Knapsack Idea Sort items by decreasing cost per pound. If knapsack holds k = 5 pds, the solution is:

23 Greedy Algorithms Greedy solution for Fractional Knapsack Idea Sort items by decreasing cost per pound. If knapsack holds k = 5 pds, the solution is:

24 Greedy Algorithms Greedy solution for Fractional Knapsack Complexity General Algorithm O(nlogn) Given a set of item I: Let P be the problem of selecting items from I, with weight limit K, such that the resulting cost (value) is maximum.

25 Greedy Algorithms Greedy solution for Fractional Knapsack 1 Calculate v i = c i w i for i = 1, 2,..., n

26 Greedy Algorithms Greedy solution for Fractional Knapsack 1 Calculate v i = c i w i for i = 1, 2,..., n 2 Sort the items by decreasing v i. Let the sorted item sequence be 1, 2,..., i,..., n and the corresponding value and weight be v i and w i respectively

27 Greedy Algorithms Greedy solution for Fractional Knapsack 1 Calculate v i = c i w i for i = 1, 2,..., n 2 Sort the items by decreasing v i. Let the sorted item sequence be 1, 2,..., i,..., n and the corresponding value and weight be v i and w i respectively 3 Let k be the current weight limit (Initially, k = K ). In each iteration, we choose item i from the head of the unselected list. If k w i, we take item i, and k = k w i, then consider the next unselected item. If k < w i, we take a fraction f of item i, i.e., we only take f = k w i (< 1) of item i, which weights exactly k. Then the algorithm is finished.

28 Greedy algorithms remarks Greedy Algorithms Observe that the algorithm may take a fraction of an item, which can only be the last selected item. We claim that the total cost for this set of items is an optimal cost. The correctness proof is out of scope for IN101.

29 Dynamic Programming Dynamic Programming Dynamic programming is a very powerful, general tool for solving optimization problems on left-right-ordered items such as character strings.

30 Dynamic Programming Dynamic Programming Applications Areas Search Bioinformatics Control theory Operations research Some famous dynamic programming algorithms Unix diff for comparing two files. Knapsack Smith-Waterman for sequence alignment

31 Dynamic Programming Dynamic Programming A strategy for designing algorithms is dynamic programming A meta-technique, not an algorithm (like divide & conquer) The word programming is historical and predates computer programming Use when problem breaks down into recurring small subproblems

32 Dynamic Programming Dynamic Programming It is used when the solution can be recursively described in terms of solutions to sub-problems (optimal sub-structure). Recursively define the optimum Algorithm finds solutions to subproblems and stores them in memory for later use. More efficient than brute-force methods, which solve the same subproblems over and over again.

33 Dynamic Programming Dynamic Programming - Idea Basic idea: Optimal substructure: optimal solution to problem consists of optimal solutions to sub-problems Overlapping sub-problems: few sub-problems in total, many recurring instances of each Solve bottom-up, building a table of solved sub-problems that are used to solve larger ones Variations: Table could be 3-dimensional, triangular, a tree, etc.

34 Optimal substructure Dynamic Programming Dynamic programming works when a problem has optimal substructure: we can construct the optimum of a larger problem from the optima of a "small set" of smaller problems. Small: polynomial Not all problems have optimal substructure. Travelling Salesman Problem (TSP)?

35 Dynamic Programming Greedy vs. Exhaustive Search Greedy algorithms focus on making the best local choice at each decision point. In the absence of a correctness proof such greedy algorithms are very likely to fail. Dynamic programming gives us a way to design custom algorithms which systematically search all possibilities (thus guaranteeing correctness) while storing results to avoid recomputing (thus providing efficiency).

36 Recurrence Relations Dynamic Programming A recurrence relation is an equation which is defined in terms of itself. They are useful because many natural functions are easily expressed as recurrences: Polynomials: a n = a n 1 + 1, a 1 = 1 a n = n Exponentials: a n = 2a n 1, a 1 = 2 a n = 2n Weird: a n = na n 1, a 1 = 1 a n = n! Computer programs can easily evaluate the value of a given recurrence even without the existence of a nice closed form

37 Dynamic Programming Dynamic Programming Design technique, like divide-and-conquer. Example: Longest Common Subsequence (LCS) Given two sequences x[1..m] and y[1..n], find a longest subsequence common to them both. x: A B C B D A B y: B D C A B A? = LCS(x, y)

38 Dynamic Programming Dynamic Programming Design technique, like divide-and-conquer. Example: Longest Common Subsequence (LCS) Given two sequences x[1..m] and y[1..n], find a longest subsequence common to them both. x: A B C B D A B y: B D C A B A BCBA = LCS(x, y)

39 Dynamic Programming Brute-force LCS algorithm Check every subsequence of x[1..m] to see if it is also a subsequence of y[1..n]. Analysis Checking = O(n) time per subsequence. 2 m subsequences of x (each bit-vector of length m determines a distinct subsequence of x). Worst-case running time = O(n2 m )= exponential time.

40 Dynamic Programming Towards a better algorithm Simplification: 1 Look at the length of a longest-common subsequence. 2 Extend the algorithm to find the LCS itself. Notation: Denote the length of a sequence s by s. Strategy: Consider prefixes of x and y Define c[i, j] = LCS(x[1..i], y[1..j]) Then, c[m, n] = LCS(x, y) (because m, n are the size of x and y)

41 Recursive formulation Dynamic Programming Theorem c[i 1, j 1] + 1, if x[i] = y[j], c[i, j] = max{c[i 1, j], c[i, j 1]} otherwise Proof Case x[i] = y[j]: Let z[1..k] = LCS(x[1..i], y[1..j]), where c[i, j] = k. Then, z[k] = x[i], or else z could be extended. Thus, z[1..k 1] is CS of x[1..i 1] and y[1..j 1].

42 Proof (II) Dynamic Programming Claim: z[1..k 1] = LCS(x[1..i 1], y[1..j 1]). Suppose w is a longer CS of x[1..i 1] and y[1..j 1], that is, w > k 1. Then, cut and paste: w z[k] (w concatenated with z[k]) is a common subsequence of x[1..i] and y[1..j] with w z[k] > k. Contradiction, proving the claim. (c[i, j] = k) Thus, c[i 1, j 1] = k 1, which implies that c[i, j] = c[i 1, j 1] + 1. Other cases are similar.

43 Dynamic Programming Dynamic programming postulate #1 Optimal substructure An optimal solution to a problem (instance) contains optimal solutions to subproblems. If z = LCS(x, y), then any prefix of z is an LCS of a prefix of x and a prefix of y.

44 Dynamic Programming Recursive algorithm for LCS Figure: LCS in python Worst-case: x[i] y[j], in which case the algorithm evaluates two subproblems, each with only one parameter decremented.

45 Recursion tree Dynamic Programming m = 3, n = 4

46 Recursion tree Dynamic Programming m = 3, n = 4 Height = m + n work potentially exponential but we re solving subproblems already solved

47 Recursion tree Dynamic Programming m = 3, n = 4 Height = m + n work potentially exponential but we re solving subproblems already solved

48 Dynamic Programming Dynamic programming postulate #2 Overlapping subproblems A recursive solution contains a small number of distinct subproblems repeated many times. The number of distinct LCS subproblems for two strings of lengths m and n is only mn.

49 Memoization Dynamic Programming Memoization is another way to deal with overlapping subproblems in dynamic programming After computing the solution to a subproblem, store it in a table Subsequent calls just do a table lookup With memoization, we implement the algorithm recursively: If we encounter a sub-problem we have seen, we look up the answer If not, compute the solution and add it to the list of sub-problems we have seen. Must useful when the algorithm is easiest to implement recursively Especially if we do not need solutions to all sub-problems.

50 Memoization algorithm Dynamic Programming After computing a solution to a subproblem, store it in a table. Subsequent calls check the table to avoid redoing work. Figure: How can we transform the recursive calls for known values to a "tabel access"? Objective: Time = O(mn) = constant work per table entry. Space = O(mn).

51 Memoization algorithm Dynamic Programming IDEA: Compute the table bottom-up.

52 Memoization algorithm Dynamic Programming IDEA: Compute the table bottom-up. Time = O(mn). Reconstruct LCS by tracing backwards. Space = O(mn)

53 Conclusion Dynamic Programming Dynamic programming is a useful technique of solving certain kind of problems When the solution can be recursively described in terms of partial solutions, we can store these partial solutions and re-use them as necessary (memoization) Running time of dynamic programming algorithm vs. naive algorithm: 0-1 Knapsack problem: O(W*n) vs. O(2 n )