DESIGN AND ANALYSIS OF ALGORITHMS (DAA 2017)

DESIGN AND ANALYSIS OF ALGORITHMS (DAA 2017) Veli Mäkinen Design and Analysis of Algorithms 2017 week 4 11.8.2017 1

Dynamic Programming Week 4 2 Design and Analysis of Algorithms 2017 week 4 11.8.2017

Longest Increasing Subsequence Example on typical steps for deriving a dynamic programming algorithm 3 Design and Analysis of Algorithms 2017 week 4 11.8.2017

LONGEST INCREASING SUBSEQUENCE (LIS) Find a longest subsequence of strictly increasing numbers: A=1, 5, 3, 6, 4, 7 Invent the definition of a table / recurrence to compute L[j] = length of the LIS of A[1..j] ending with A[j] Now, find a dependency among L[j] values such that there is a proper evaluation order to compute them (left to right in this case): L[j] = max 0 i<j {L[i]+1 : A[i]<A[j]}, for 1 j n, and L[0] = 0; A[0]=- (initialization) Show pseudocode to illustrate evaluation order and analyse running time. Prove correctess by induction. Show how to traceback an optimal solution. Design and Analysis of Algorithms 2017 week 4 11.8.2017 4

LIS EXAMPLE L[6]=? 1<4 3<4 L[7]=? A=1, 5, 3, 6, 4, 7 A=1, 5, 3, 6, 4, 7 max max +1 +1 L=1, 2, 2, 3, 3 L=1, 2, 2, 3, 3, 4 Traceback all optimal solutions: A=1, 5, 3, 6, 4, 7 L=1, 2, 2, 3, 3, 4 A=1, 5, 3, 6, 4, 7 L=1, 2, 2, 3, 3, 4 A=1, 5, 3, 6, 4, 7 L=1, 2, 2, 3, 3, 4 Design and Analysis of Algorithms 2017 week 4 11.8.2017

LEN, J, L = LIS(A[1..N]) L[0]=0; A[0]=- /* Initialization*/ For j=1 to n do /*evaluation order*/ L[j]=0 /* temporary max*/ For i=0 to j-1 do /* O(n 2 ) */ If A[i]<A[j] and L[i]+1>L[j] then L[j] = L[i]+1 /* update of temporary max */ lenmax = 0; indexmax=0 /* max L[i], argmax L[i] */ For j=1 to n do if L[j]>lenMax then (lenmax, indexmax) = (L[j],j) Return (lenmax, indexmax,l) Design and Analysis of Algorithms 2017 week 4 11.8.2017 6

A = TRACEBACKLIS(A[1..N], L[1..N],J) A = empty stack While j>0 do A.Push(A[j]) For i=0 to j-1 do Return A If A[i]<A[j] and L[i]+1=L[j] then j=i; (break) (Popping from A returns a longest increasing subsequence of A in proper order) Design and Analysis of Algorithms 2017 week 4 11.8.2017 7

LIS: OPTIMAL SUBPROBLEM PROPERTY Assume by induction that L[1..j-1] is correctly computed with the recurrence. If L[j] > max 0 i<j {L[i]+1 : A[i]<A[j]}, there would be a subsequence ending with some A[i], 0<i<j, of length L[j]-1, contradicting the optimality of L[i]. Also, any subsequence ending with A[i], A[i]<A[j], could be extended with A[j] to obtain a subsequence of length L[i]+1. Hence the recurrence is correct. (Check the base case.) This tight connection with the dynamic programming table values is called optimal subproblem property. In literature, full induction proofs for obvious cases are typically omitted by referring to this property holding. yet there are cases where the connection is more elaborate Design and Analysis of Algorithms 2017 week 4 11.8.2017 8

ALTERNATIVE: REDUCTION TO SHORTEST PATH n arcs c(i,j)=-1 n arcs 0 i j n+1 Let A[0]=- and A[n+1]=. DAG G=(V,E) with vertices V={0,1,,n+1} and arcs E defined as (i,j) E if A[i]<A[j]. Source s=0 and target t=n+1. Arc costs c(i,j)=-1, except for c(i,n+1)=0. Cost of shortest s-t path = - OPT LIS, where OPT LIS denotes the length of the longest increasing subsequence. Design and Analysis of Algorithms 2017 week 4 11.8.2017 9

ANOTHER REDUCTION TO SHORTEST PATH n arcs c(i,j)=j-i-1 n arcs 0 i j n+1 Maximizing subsequence length = minimizing number of elements deleted from A Let A[0]=- and A[n+1]=. DAG G=(V,E) with vertices V={0,1,,n+1} and arcs E defined as (i,j) E if A[i]<A[j]. Source s=0 and target t=n+1. Arc costs c(i,j)=j-i-1. Cost of shortest s-t path = n-opt LIS. Design and Analysis of Algorithms 2017 week 4 11.8.2017 10

Segmentation problem Another example of the power of dynamic programming 11 Design and Analysis of Algorithms 2017 week 4 11.8.2017

SEGMENTATION PROBLEM Clustering in 1D http://stackoverflow.com/questions/11513484/1d-number-arrayclustering Clustering in >=2D is NP-hard Clustering in ID is known as the segmentation problem and can be solved efficiently by dynamic programming Design and Analysis of Algorithms 2017 week 4 11.8.2017 12

SEGMENTATION PROBLEM: MOTIVATION A company owns real estates and the CEO requests a map with colour codes showing different property price ranges Design and Analysis of Algorithms 2017 week 4 11.8.2017 13

SEGMENTATION PROBLEM: MOTIVATION 30 000 50 000 75 000 150 000 200 000 400 000 500 000 750 000 1 000 000 3 000 000 Design and Analysis of Algorithms 2017 week 4 11.8.2017 14

SEGMENTATION: MOTIVATION 2 Compression boosting: Partition a string A[1..n] into segments that compress better by tailored compressor for each segment. E.g. ababbbacdaccddbbbaabaab Let c(i,j) give the compression result for segment A[i..j]. Then an optimal partitioning of A is one with the sum of its segment costs being minimum. Design and Analysis of Algorithms 2017 week 4 11.8.2017 15

SEGMENTATION: RECURRENCE Let us consider a table S[1..n,1..n] with S[j,k] storing the cost of an optimal segmentation of the prefix A[1..j] of the input array A[1..n] into k segments. We have S[j, k] = min i [1..j 1] S i, k 1 + c(i + 1, j). This recurrence considers all possible ending positions of segmentations with k-1 segments and adds the cost of adding one more segment. With the optimal subproblem property one can see this works correctly, assuming a suitable initialization for the boundary cases (exercise) and a suitable evaluation order is used (exercise). Design and Analysis of Algorithms 2017 week 4 11.8.2017 16

1D CLUSTERING: GENERIC DEFINITION Input: An array A[1..n] containing an increasing series of numbers, and parameter k. 1D clustering problem : Find cluster centers C=c 1, c 2,, c k that minimize total deviation, sumdev(a,c), when each A[i] is connected to its nearest center: sumdev(a,c)=σ a A min c C dev a, c. k clusters definition? Design and Analysis of Algorithms 2017 week 4 11.8.2017 17

1D CLUSTERING: L2 AND AVERAGE To fix a definition, let us define dev(x,y)=(y-x) 2 [L2 norm] Then an optimal center of a cluster is the average of the values, since that will minimize the total deviation from the center (omitting the proof here). Observation: Centers C form a segmentation (partitioning) of A into A=A[1..i 1 ][A[i 1 +1 i 2 ] A[i k-1 n], where 1 i 1 < i 2 < i k 1 <n, such that numbers in A[i k +1 i k +1 ] are closest to c k +1 among centers in C. c 1 k clusters segment= 1d cluster Design and Analysis of Algorithms 2017 week 4 11.8.2017 18

1D CLUSTERING: L2 AND AVERAGE Now, we seek for segmentation A=A[1..i 1 ][A[i 1 +1 i 2 ] A[i k- 1 n] minimizing σk k =1 c L2 (i k 1+1..ik ), where i 1 =0 and i k =n, and c c L2 (a..c)=σ b=a dev(a b, AVG(A[a.. c])), where c AVG(A[a..c])=σ b=a A b /(c a + 1). A[1..3]=0,4,5 0 4 5 Clustering with k=1 AVG(0,4,5)=3 c L2 (1..3)=(3-0) 2 +(3-4) 2 +(3-5) 2 =14 Design and Analysis of Algorithms 2017 week 4 11.8.2017 19

1D CLUSTERING: RECURRENCE devmin[j,k ]= minimum total deviation after segmenting A[1..j] into k segments devmin[j,k ]=min i<j devmin [i,k -1]+c L2 (i+1..j) k -1 clusters min C sumdev(a,c) = devmin[n,k] (optimal solution) O(n 3 k) running time with direct implementation Optimizations, extensions, correctness, initialization, traceback in study groups & exercises. + i i+1 j Design and Analysis of Algorithms 2017 week 4 11.8.2017 20

1D CLUSTERING: EVALUATION ORDER devmin[j,k ]=min i<j devmin [i,k -1]+c L2 (i+1..j) Evaluation order: k -1 k i min ( ) + cost of arc j Column-by-column (above), row-by-row, Design and Analysis of Algorithms 2017 week 4 11.8.2017 21

1D CLUSTERING: RECAP+INITIALIZATION A devmin[0..n,0..k] 0 k -1 k k k -1 min cost to segment A[1..i] to k -1 pieces 0 0... more segments than points: d[i,k ]=, for k >i i AVG dev() min i<j devmin[..] + cost of arc j cost to add segment A[i+1..j] n min cost to segment A[1..j] to k pieces optimum solution: traceback following arcs 0 segments only possible for empty sequence: d[i,0]=, except d[0,0]=0

Optimal chain of matrix multiplications Dynamic programming is not always a shortest path problem 23 Design and Analysis of Algorithms 2017 week 4 11.8.2017

OPTIMAL CHAIN OF MATRIX MULTIPLICATIONS m p p n x = n O(npm) time m m p q q n x m x p = n Design and Analysis of Algorithms 2017 week 4 11.8.2017 24

OPTIMAL CHAIN OF MATRIX MULTIPLICATIONS m p q q n x m x p = n npm+nqp multiplications n=10, m=15, p=7, q=4: 1330 multiplications n m x m p x p q = n q mqp+nqm multiplications n=10, m=15, p=7, q=4: 1470 multiplications Design and Analysis of Algorithms 2017 week 4 11.8.2017 25

OPTIMAL CHAIN OF MATRIX MULTIPLICATIONS Find an optimal order to multiply n matrices A 1, A 2,.., A n with dimensions l 1 l 2, l 2 l 3,, l n d, where d=l n + 1 M(i,j)= minimum cost to multiply A i, A i+1,.., A j M(i,j)=min k:i k j {M(i,k) +M(k+1,j) +l i l j + 1 l k + 1 }. A i A i+1 A k l k + 1 A k+1 A j l j + 1 l j + 1 l i x = l i l k + 1 Design and Analysis of Algorithms 2017 week 4 11.8.2017 26

OPTIMAL CHAIN: PSEUDOCODE M(i,j): /* this is slow.*/ If i=j return 0 value = For k=i to j-1 do value = min(value, M(i,k)+M(k+1,j)+l(1)*l(j+1)*l(k+1)) return value return M(1,n) m[i,j]= for all i,j MM(i,j): /* memoized variant, this is fast.*/ If i=j {m[i,i]=0; return 0} If m[i,j]= then For k=i to j-1 do m[i,j] = min(m[i,j], MM(i,k)+MM(k+1,j)+l(1)*l(j+1)*l(k+1)) return m[i,j] return MM(1,n) Design and Analysis of Algorithms 2017 week 4 11.8.2017 27

OPTIMAL CHAIN: ANALYSIS Each vertex of the dependency DAG created ones O(n 3 ) time Better to unroll the recursion by computing intervals of length 1,2,3,,n, in that order (see the book) M(i,k) M(i,j) M(k+1,j) Optimization over directed hyperedges Design and Analysis of Algorithms 2017 week 4 11.8.2017 28

OPTIMAL RECURSION TREE AND BALANCED PARANTHESES x x x A B x x C D E F ((AxB)x((CxD)x(ExF))) Design and Analysis of Algorithms 2017 week 4 11.8.2017 29

Range minimum query problem Another example of the power of dynamic programming 30 Design and Analysis of Algorithms 2017 week 4 11.8.2017

LCA VS RMQ v x r x w=lca(c,f) x A B p x x q min C D E F Height= 012121012323212323210 DFS= rvavbvrwpcpdpwqeqfqwr arg min LCA = lowest common ancestor RMQ= range minimum query Design and Analysis of Algorithms 2017 week 4 11.8.2017 31

RMQ: DYNAMIC PROGRAMMING Later in the course: O(n log n) time dynamic programming algorithm to preprocess an array for O(1) time range minimum queries. [Section 3: A Faster RMQ Algorithm] Bender & Farach-Colton. The LCA Problem Revisited. LATIN 2000. Same article also uses Cartesian trees, reductions, and so-called four-russians technique to improve preprocessing to optimal O(n) time (under RAM model of computation). Design and Analysis of Algorithms 2017 week 4 11.8.2017 32

DYNAMIC PROGRAMMING IN OTHER COURSES Viterbi (kind of clustering, segmentation) Biological Sequence Analysis, period III, 2018 Shortest paths, edit distance, sequence alignment Algorithms for Bioinformatics, period I, 2017 String Processing Methods, period II, 2018 Biological Sequence Analysis, period III, 2018 In many others as a common tool Design and Analysis of Algorithms 2017 week 4 11.8.2017 33