QB LECTURE #1: Algorithms and Dynamic Programming

Transcription

1 QB LECTURE #1: Algorithms and Dynamic Programming Adam Siepel Nov. 16, 2015

2 2 Plan for Today Introduction to algorithms Simple algorithms and running time Dynamic programming Soon: sequence alignment

3 3 Intro to Algorithms An algorithm is a sequence of instructions to solve a well-defined class of problems Usually defined in terms of inputs and outputs Two key questions: Does it work correctly? How fast? Elegance is prized, but not at the expense of correctness or speed

4 4 Examples of Algorithms Trivial: linear search, sum elements of list, search for largest element More subtle: binary search, quick sort, graph search Tricky: dynamic programming, branchand-bound search, minimum spanning tree Deep: nonlinear optimization, expectation maximization for MLE, fast Fourier transform, sorting by reversals

5 5 Running Time We study running time as a function of the size of the input, ignoring constants (depend on implementation) Can consider worst-case, best-case, average-case, amortized running time Worst-case most widely used: big-o notation Examples: linear-search = O(n), binary search = O(log n), sorting = O(n log n), sequence alignment = O(nm) Roughly speaking, algorithms are tractable if polynomial (e.g., O(n k )), otherwise intractable

6 Searching for GATTACA Where is GATTACA in the human genome? Strategy 1: Brute Force T G A T T A C A G A T T A C C G A T T A C A No match at offset 1

7 Searching for GATTACA Where is GATTACA in the human genome? Strategy 1: Brute Force T G A T T A C A G A T T A C C G A T T A C A Match at offset 2

8 Searching for GATTACA Where is GATTACA in the human genome? Strategy 1: Brute Force T G A T T A C A G A T T A C C G A T T A C A No match at offset 3

9 Searching for GATTACA Where is GATTACA in the human genome? Strategy 1: Brute Force T G A T T A C A G A T T A C C G A T T A C A No match at offset 9 <- Checking each possible position takes time

10 Brute Force Analysis Brute Force: At every possible offset in the genome: Do all of the characters of the query match? Analysis Simple, easy to understand Genome length = n [3B] Query length = m [7] Comparisons: (n-m+1) * m [21B] Overall runtime: O(nm) [How long would it take if we double the genome size, read length?] [How long would it take if we double both?]

11 Brute Force Reflections Why check every position? GATTACA can't possibly start at position 15 [WHY?] T G A T T A C A G A T T A C C G A T T A C A Improve runtime to O(n + m) [3B + 7] If we double both, it just takes twice as long Knuth-Morris-Pratt, 1977 Boyer-Moyer, 1977, 1991 For one-off scans, this is the best we can do (optimal performance) We have to read every character of the genome, and every character of the query For short queries, runtime is dominated by the length of the genome

12 Suffix Arrays: Searching the Phone Book What if we need to check many queries? We don't need to check every page of the phone book to find 'Schatz' Sorting alphabetically lets us immediately skip 96% (25/26) of the book without any loss in accuracy Sorting the genome: Suffix Array (Manber & Myers, 1991) Sort every suffix of the genome Split into n suffixes Sort suffixes alphabetically [Challenge Question: How else could we split the genome?]

13 Searching the Index Strategy 2: Binary search Compare to the middle, refine as higher or lower Searching for GATTACA Lo = 1; Hi = 15; Lo Hi # Sequence Pos 1 ACAGATTACC 6 2 ACC 13 3 AGATTACC 8 4 ATTACAGATTACC 3 5 ATTACC 10 6 C 15 7 CAGATTACC 7 8 CC 14 9 GATTACAGATTACC 2 10 GATTACC 9 11 TACAGATTACC 5 12 TACC TGATTACAGATTACC 1 14 TTACAGATTACC 4 15 TTACC 11

14 Searching the Index Strategy 2: Binary search Compare to the middle, refine as higher or lower Searching for GATTACA Lo = 1; Hi = 15; Mid = (1+15)/2 = 8 Middle = Suffix[8] = CC Lo Hi # Sequence Pos 1 ACAGATTACC 6 2 ACC 13 3 AGATTACC 8 4 ATTACAGATTACC 3 5 ATTACC 10 6 C 15 7 CAGATTACC 7 8 CC 14 9 GATTACAGATTACC 2 10 GATTACC 9 11 TACAGATTACC 5 12 TACC TGATTACAGATTACC 1 14 TTACAGATTACC 4 15 TTACC 11

15 Searching the Index Strategy 2: Binary search Compare to the middle, refine as higher or lower Searching for GATTACA Lo = 1; Hi = 15; Mid = (1+15)/2 = 8 Middle = Suffix[8] = CC => Higher: Lo = Mid + 1 Lo Hi # Sequence Pos 1 ACAGATTACC 6 2 ACC 13 3 AGATTACC 8 4 ATTACAGATTACC 3 5 ATTACC 10 6 C 15 7 CAGATTACC 7 8 CC 14 9 GATTACAGATTACC 2 10 GATTACC 9 11 TACAGATTACC 5 12 TACC TGATTACAGATTACC 1 14 TTACAGATTACC 4 15 TTACC 11

16 Searching the Index Strategy 2: Binary search Compare to the middle, refine as higher or lower Searching for GATTACA Lo = 1; Hi = 15; Mid = (1+15)/2 = 8 Middle = Suffix[8] = CC => Higher: Lo = Mid + 1 Lo = 9; Hi = 15; Lo Hi # Sequence Pos 1 ACAGATTACC 6 2 ACC 13 3 AGATTACC 8 4 ATTACAGATTACC 3 5 ATTACC 10 6 C 15 7 CAGATTACC 7 8 CC 14 9 GATTACAGATTACC 2 10 GATTACC 9 11 TACAGATTACC 5 12 TACC TGATTACAGATTACC 1 14 TTACAGATTACC 4 15 TTACC 11

17 Searching the Index Strategy 2: Binary search Compare to the middle, refine as higher or lower Searching for GATTACA Lo = 1; Hi = 15; Mid = (1+15)/2 = 8 Middle = Suffix[8] = CC => Higher: Lo = Mid + 1 Lo = 9; Hi = 15; Mid = (9+15)/2 = 12 Middle = Suffix[12] = TACC Lo Hi # Sequence Pos 1 ACAGATTACC 6 2 ACC 13 3 AGATTACC 8 4 ATTACAGATTACC 3 5 ATTACC 10 6 C 15 7 CAGATTACC 7 8 CC 14 9 GATTACAGATTACC 2 10 GATTACC 9 11 TACAGATTACC 5 12 TACC TGATTACAGATTACC 1 14 TTACAGATTACC 4 15 TTACC 11

18 Searching the Index Strategy 2: Binary search Compare to the middle, refine as higher or lower Searching for GATTACA Lo = 1; Hi = 15; Mid = (1+15)/2 = 8 Middle = Suffix[8] = CC => Higher: Lo = Mid + 1 Lo = 9; Hi = 15; Mid = (9+15)/2 = 12 Middle = Suffix[12] = TACC => Lower: Hi = Mid - 1 Lo = 9; Hi = 11; Lo Hi # Sequence Pos 1 ACAGATTACC 6 2 ACC 13 3 AGATTACC 8 4 ATTACAGATTACC 3 5 ATTACC 10 6 C 15 7 CAGATTACC 7 8 CC 14 9 GATTACAGATTACC 2 10 GATTACC 9 11 TACAGATTACC 5 12 TACC TGATTACAGATTACC 1 14 TTACAGATTACC 4 15 TTACC 11

19 Searching the Index Strategy 2: Binary search Compare to the middle, refine as higher or lower Searching for GATTACA Lo = 1; Hi = 15; Mid = (1+15)/2 = 8 Middle = Suffix[8] = CC => Higher: Lo = Mid + 1 Lo = 9; Hi = 15; Mid = (9+15)/2 = 12 Middle = Suffix[12] = TACC => Lower: Hi = Mid - 1 Lo = 9; Hi = 11; Mid = (9+11)/2 = 10 Middle = Suffix[10] = GATTACC Lo Hi # Sequence Pos 1 ACAGATTACC 6 2 ACC 13 3 AGATTACC 8 4 ATTACAGATTACC 3 5 ATTACC 10 6 C 15 7 CAGATTACC 7 8 CC 14 9 GATTACAGATTACC 2 10 GATTACC 9 11 TACAGATTACC 5 12 TACC TGATTACAGATTACC 1 14 TTACAGATTACC 4 15 TTACC 11

20 Searching the Index Strategy 2: Binary search Compare to the middle, refine as higher or lower Searching for GATTACA Lo = 1; Hi = 15; Mid = (1+15)/2 = 8 Middle = Suffix[8] = CC => Higher: Lo = Mid + 1 Lo = 9; Hi = 15; Mid = (9+15)/2 = 12 Middle = Suffix[12] = TACC => Lower: Hi = Mid - 1 Lo = 9; Hi = 11; Mid = (9+11)/2 = 10 Middle = Suffix[10] = GATTACC => Lower: Hi = Mid - 1 Lo = 9; Hi = 9; Lo Hi # Sequence Pos 1 ACAGATTACC 6 2 ACC 13 3 AGATTACC 8 4 ATTACAGATTACC 3 5 ATTACC 10 6 C 15 7 CAGATTACC 7 8 CC 14 9 GATTACAGATTACC 2 10 GATTACC 9 11 TACAGATTACC 5 12 TACC TGATTACAGATTACC 1 14 TTACAGATTACC 4 15 TTACC 11

21 Searching the Index Strategy 2: Binary search Compare to the middle, refine as higher or lower Searching for GATTACA Lo = 1; Hi = 15; Mid = (1+15)/2 = 8 Middle = Suffix[8] = CC => Higher: Lo = Mid + 1 Lo = 9; Hi = 15; Mid = (9+15)/2 = 12 Middle = Suffix[12] = TACC => Lower: Hi = Mid - 1 Lo = 9; Hi = 11; Mid = (9+11)/2 = 10 Middle = Suffix[10] = GATTACC => Lower: Hi = Mid - 1 Lo = 9; Hi = 9; Mid = (9+9)/2 = 9 Middle = Suffix[9] = GATTACA => Match at position 2! Lo Hi # Sequence Pos 1 ACAGATTACC 6 2 ACC 13 3 AGATTACC 8 4 ATTACAGATTACC 3 5 ATTACC 10 6 C 15 7 CAGATTACC 7 8 CC 14 9 GATTACAGATTACC 2 10 GATTACC 9 11 TACAGATTACC 5 12 TACC TGATTACAGATTACC 1 14 TTACAGATTACC 4 15 TTACC 11

22 Binary Search Analysis Binary Search Initialize search range to entire list mid = (hi+lo)/2; middle = suffix[mid] if query matches middle: done else if query < middle: pick low range else if query > middle: pick hi range Repeat until done or empty range [WHEN?] Analysis More complicated method How many times do we repeat? How many times can it cut the range in half? Find smallest x such that: n/(2 x ) 1; x = lg 2 (n) [32] Total Runtime: O(m lg n) More complicated, but much faster! Looking up a query loops 32 times instead of 3B [How long does it take to search 6B or 24B nucleotides?]

23 Sorting Quickly sort these numbers into ascending order: 14, 29, 6, 31, 39, 64, 78, 50, 13, 63, 61, 19 [How do you do it?] 6, 14, 29, 31, 39, 64, 78, 50, 13, 63, 61, 19 6, 13, 14, 29, 31, 39, 64, 78, 50, 63, 61, 19 6, 13, 14, 19, 29, 31, 39, 64, 78, 50, 63, 61 6, 13, 14, 19, 29, 31, 39, 64, 78, 50, 63, 61 6, 13, 14, 19, 29, 31, 39, 64, 78, 50, 63, 61 6, 13, 14, 19, 29, 31, 39, 50, 64, 78, 63, 61 6, 13, 14, 19, 29, 31, 39, 50, 61, 64, 78, 63 6, 13, 14, 19, 29, 31, 39, 50, 61, 63, 64, 78 6, 13, 14, 19, 29, 31, 39, 50, 61, 63, 64, 78 6, 13, 14, 19, 29, 31, 39, 50, 61, 63, 64, 78 6, 13, 14, 19, 29, 31, 39, 50, 61, 63, 64, 78 6, 13, 14, 19, 29, 31, 39, 50, 61, 63, 64, 78

24 Selection Sort Analysis Selection Sort (Input: list of n numbers) for pos = 1 to n // find the smallest element in [pos, n] smallest = pos for check = pos+1 to n if (list[check] < list[smallest]): smallest = check Analysis // move the smallest element to the front tmp = list[smallest] list[pos] = list[smallest] list[smallest] = tmp Outer loop: pos = 1 to n Inner loop: check = pos to n Running time: Outer * Inner = O(n 2 ) [Challenge Questions: Why is this slow? / Can we sort any faster?]

25 Divide and Conquer Selection sort is slow because it rescans the entire list for each element How can we split up the unsorted list into independent ranges? Hint 1: Binary search splits up the problem into 2 independent ranges (hi/lo) Hint 2: Assume we know the median value of a list n < = > 2 x n/2 < = > = < = > 4 x n/4 < = > = < = > = < = > = < = > 8 x n/8 16 x n/16 [How many times can we split a list in half?] 2 i x n/2 i

26 Algorithmic Complexity What is the runtime as a function of the input size?

27 27 Recursion Algorithms that call themselves are recursive Example: partialsum(n) Input: positive integer n Output: sum of integers from 1 to n if (n = 1) return 1; else return n + partialsum(n 1); Recursive algorithms can always be redefined as iterative algorithms (sometimes desirable)

28 Fibonacci Sequence def fib(n): if n == 0 or n == 1: return n else: f(5) F(6) f(4) return fib(n 1) + fib(n 2) f(4) f(3) f(3) f(2) f(3) f(2) f(2) f(1) f(2) f(1) f(1) f(0) f(2) f(1) f(1) f(0) f(1) f(0) f(1) f(0) f(1) f(0)

29 Fibonacci Sequence def fib(n): if n == 0 or n == 1: return n else: return fib(n 1) + fib(n 2) [How long would it take for F(7)?] [What is the running time?]

30 Bottom-up Fibonacci Sequence def fib(n): table = [0] * (n+1) table[0] = 0 table[1] = 1 for i in range(2,n+1): table[i] = table[i 2] + table[i 1] return table[n] [How long will it take for F(7)?] [What is the running time?]

31 31 Dynamic Programming Recursion with reuse of intermediate computations Widely used in computational biology: alignment, phylogenetics, HMMs Also graphical models, operations research, signal processing, timeseries analysis, etc.

32 Dynamic Programming General approach for solving (some) complex problems When applicable, the method takes far less time than naive methods. Polynomial time (O(n) or O(n 2 ) instead of exponential time (O(2 n ) or O(3 n )) Requirements: Overlapping subproblems Optimal substructure Applications: Fibonacci Longest Increasing Subsequence Sequence alignment, Dynamic Time Warp, Viterbi F(5) F(3) F(6) F(4) F(2) Not applicable: Traveling salesman problem, Clique finding, Subgraph isomorphism, F(1) F(0) The cheapest flight from airport A to airport B involves a single connection through airport C, but the cheapest flight from airport A to airport C involves a connection through some other airport D.

33 33 Introduction to Graphs 5 2 A graph consists of nodes and edges. The edges may be directed or undirected, and may be weighted or unweighted. A path from node u to node v is a sequence of connected edges leading from u to v The length of a path is its total number of edges. The weight of a path is the sum of the weights of all edges. Many optimization problems can be cast as problems of finding paths in graphs

34 34 Manhattan Tourist Problem Problem: move from source (59th St. & 8th Ave.) to sink (42nd & Lexington), maximizing number of tourist attractions City can be described as a graph with vertices for corners and edges for streets. This graph is a grid (lattice) The numbers of attractions are weights. The tourist can only move S&E, which directs the edges The problem is to find maximum weight from source to sink (not actual path)

35 35

36 36 Base Cases The best weight at the source is s0,0 = 0 What is the best weight along the northern boundary: s0,1, s0,2,...? Answer: s 0,j = s 0,j 1 + w (0,j 1) (0,j), 1 j m Similarly, for the western boundary: s i,0 = s i 1,0 + w (i 1,0) (i,0), 1 i n

37 37 Key Recurrence How do we express si,j in terms of a subproblem? Answer: s i,j = max { s i 1,j + w (i 1,j) (i,j) s i,j 1 + w (i,j 1) (i,j)

38 38 The Algorithm Running time: O(nm)

39 39 Key Points Identify sub-problem, find recurrence Recursive algorithm is not enough: no savings unless computations are reused Can be useful to convert to problem of finding path in graph

40 40 That s All Homework is coming (today or tomorrow). Will be due Nov 30.

41 Suffix Array Construction How can we store the suffix array? [How many characters are in all suffixes combined?] Pos Hopeless to explicitly store 4.5 billion billion characters Instead use implicit representation Keep 1 copy of the genome, and a list of sorted offsets Storing 3 billion offsets fits on a server (12GB) Searching the array is very fast, but it takes time to construct This time will be amortized over many, many searches Run it once "overnight" and save it away for all future queries TGATTACAGATTACC

42 QuickSort Analysis

43 QuickSort Analysis

44 QuickSort in Python list.sort() The goal of software engineering is to build libraries of correct reusable functions that implement higher level ideas Build complex software out of simple components Software tends to be 90% plumbing, 10% research You still need to know how they work Python requires an explicit representation of the strings

45 Longest Increasing Subsequence Given a sequence of N numbers A 1, A 2, A 3, A N, find the longest monotonically increasing subsequence 29, 6, 14, 31, 39, 78, 63, 50, 13, 64, 61, 19 Greedy approach (always extend the subsequence if you can): 29, 6, 14, 31, 39, 78, 63, 50, 13, 64, 61, 19 => 4 Brute force: Try all possible O(2 n ) subsequences 29, 6, 14, 31, 39, 78, 63, 50, 13, 64, 61, 19 => 1 29, 6, 14, 31, 39, 78, 63, 50, 13, 64, 61, 19 => invalid 29, 6, 14, 31, 39, 78, 63, 50, 13, 64, 61, 19 => invalid 29, 6, 14, 31, 39, 78, 63, 50, 13, 64, 61, 19 => 2

46 Longest Increasing Subsequence Idea: The solution for all N numbers depends on the solution for the first N-1 Look through the previous values to find the longest subsequence ending at X such that A X < A N Dynamic Programming: Def: L[j] is the longest increasing subsequence ending at position j Base case: L[0] = 0 Recurrence: LIS=max{L[i]} L[ j] = max{ L[i] }+1 i< j A[i ]<A[ j ] Index Value LIS Prev [What s the LIS of 0,8,4,12,2,10,6,14,1,9,5,13,3,11,7,15?]

47 Longest Increasing Subsequence // Initialize L[0] = 0; P[0] = i j n // Iteratively apply recurrence for i = 1 to N // find the best LIS to extend bestlis = 0; bestidx = -1 for j = 1 to i if ((A[j] <= A[i])) && (L[j] > bestlis)) bestlis = L[j]; bestidx = j L[i] = bestlis + 1; P[i] = bestidx A[1] A[2] A[3] A[i] A[j] L[1] L[2] L[3] L[i] L[i]+1 P[1] P[2] P[3] P[j] i A[n] // Scan the L array to find the overall LIS LIS = 0 for j = 1 to N if (L[j] > LIS) LIS = L[j] print The LIS is $LIS [What s the running time?]