DESIGN AND ANALYSIS OF ALGORITHMS (DAA 2017)

Transcription

1 DESIGN AND ANALYSIS OF ALGORITHMS (DAA 2017) Veli Mäkinen Design and Analysis of Algorithms 2017 week

2 Dynamic Programming Week 4 2 Design and Analysis of Algorithms 2017 week

3 Longest Increasing Subsequence Example on typical steps for deriving a dynamic programming algorithm 3 Design and Analysis of Algorithms 2017 week

4 LONGEST INCREASING SUBSEQUENCE (LIS) Find a longest subsequence of strictly increasing numbers: A=1, 5, 3, 6, 4, 7 Invent the definition of a table / recurrence to compute L[j] = length of the LIS of A[1..j] ending with A[j] Now, find a dependency among L[j] values such that there is a proper evaluation order to compute them (left to right in this case): L[j] = max 0 i<j {L[i]+1 : A[i]<A[j]}, for 1 j n, and L[0] = 0; A[0]=- (initialization) Show pseudocode to illustrate evaluation order and analyse running time. Prove correctess by induction. Show how to traceback an optimal solution. Design and Analysis of Algorithms 2017 week

5 LIS EXAMPLE L[6]=? 1<4 3<4 L[7]=? A=1, 5, 3, 6, 4, 7 A=1, 5, 3, 6, 4, 7 max max L=1, 2, 2, 3, 3 L=1, 2, 2, 3, 3, 4 Traceback all optimal solutions: A=1, 5, 3, 6, 4, 7 L=1, 2, 2, 3, 3, 4 A=1, 5, 3, 6, 4, 7 L=1, 2, 2, 3, 3, 4 A=1, 5, 3, 6, 4, 7 L=1, 2, 2, 3, 3, 4 Design and Analysis of Algorithms 2017 week

6 LEN, J, L = LIS(A[1..N]) L[0]=0; A[0]=- /* Initialization*/ For j=1 to n do /*evaluation order*/ L[j]=0 /* temporary max*/ For i=0 to j-1 do /* O(n 2 ) */ If A[i]<A[j] and L[i]+1>L[j] then L[j] = L[i]+1 /* update of temporary max */ lenmax = 0; indexmax=0 /* max L[i], argmax L[i] */ For j=1 to n do if L[j]>lenMax then (lenmax, indexmax) = (L[j],j) Return (lenmax, indexmax,l) Design and Analysis of Algorithms 2017 week

7 A = TRACEBACKLIS(A[1..N], L[1..N],J) A = empty stack While j>0 do A.Push(A[j]) For i=0 to j-1 do Return A If A[i]<A[j] and L[i]+1=L[j] then j=i; (break) (Popping from A returns a longest increasing subsequence of A in proper order) Design and Analysis of Algorithms 2017 week

8 LIS: OPTIMAL SUBPROBLEM PROPERTY Assume by induction that L[1..j-1] is correctly computed with the recurrence. If L[j] > max 0 i<j {L[i]+1 : A[i]<A[j]}, there would be a subsequence ending with some A[i], 0<i<j, of length L[j]-1, contradicting the optimality of L[i]. Also, any subsequence ending with A[i], A[i]<A[j], could be extended with A[j] to obtain a subsequence of length L[i]+1. Hence the recurrence is correct. (Check the base case.) This tight connection with the dynamic programming table values is called optimal subproblem property. In literature, full induction proofs for obvious cases are typically omitted by referring to this property holding. yet there are cases where the connection is more elaborate Design and Analysis of Algorithms 2017 week

9 ALTERNATIVE: REDUCTION TO SHORTEST PATH n arcs c(i,j)=-1 n arcs 0 i j n+1 Let A[0]=- and A[n+1]=. DAG G=(V,E) with vertices V={0,1,,n+1} and arcs E defined as (i,j) E if A[i]<A[j]. Source s=0 and target t=n+1. Arc costs c(i,j)=-1, except for c(i,n+1)=0. Cost of shortest s-t path = - OPT LIS, where OPT LIS denotes the length of the longest increasing subsequence. Design and Analysis of Algorithms 2017 week

10 ANOTHER REDUCTION TO SHORTEST PATH n arcs c(i,j)=j-i-1 n arcs 0 i j n+1 Maximizing subsequence length = minimizing number of elements deleted from A Let A[0]=- and A[n+1]=. DAG G=(V,E) with vertices V={0,1,,n+1} and arcs E defined as (i,j) E if A[i]<A[j]. Source s=0 and target t=n+1. Arc costs c(i,j)=j-i-1. Cost of shortest s-t path = n-opt LIS. Design and Analysis of Algorithms 2017 week

11 Segmentation problem Another example of the power of dynamic programming 11 Design and Analysis of Algorithms 2017 week

12 SEGMENTATION PROBLEM Clustering in 1D Clustering in >=2D is NP-hard Clustering in ID is known as the segmentation problem and can be solved efficiently by dynamic programming Design and Analysis of Algorithms 2017 week

13 SEGMENTATION PROBLEM: MOTIVATION A company owns real estates and the CEO requests a map with colour codes showing different property price ranges Design and Analysis of Algorithms 2017 week

14 SEGMENTATION PROBLEM: MOTIVATION Design and Analysis of Algorithms 2017 week

15 SEGMENTATION: MOTIVATION 2 Compression boosting: Partition a string A[1..n] into segments that compress better by tailored compressor for each segment. E.g. ababbbacdaccddbbbaabaab Let c(i,j) give the compression result for segment A[i..j]. Then an optimal partitioning of A is one with the sum of its segment costs being minimum. Design and Analysis of Algorithms 2017 week

16 SEGMENTATION: RECURRENCE Let us consider a table S[1..n,1..n] with S[j,k] storing the cost of an optimal segmentation of the prefix A[1..j] of the input array A[1..n] into k segments. We have S[j, k] = min i [1..j 1] S i, k 1 + c(i + 1, j). This recurrence considers all possible ending positions of segmentations with k-1 segments and adds the cost of adding one more segment. With the optimal subproblem property one can see this works correctly, assuming a suitable initialization for the boundary cases (exercise) and a suitable evaluation order is used (exercise). Design and Analysis of Algorithms 2017 week

17 1D CLUSTERING: GENERIC DEFINITION Input: An array A[1..n] containing an increasing series of numbers, and parameter k. 1D clustering problem : Find cluster centers C=c 1, c 2,, c k that minimize total deviation, sumdev(a,c), when each A[i] is connected to its nearest center: sumdev(a,c)=σ a A min c C dev a, c. k clusters definition? Design and Analysis of Algorithms 2017 week

18 1D CLUSTERING: L2 AND AVERAGE To fix a definition, let us define dev(x,y)=(y-x) 2 [L2 norm] Then an optimal center of a cluster is the average of the values, since that will minimize the total deviation from the center (omitting the proof here). Observation: Centers C form a segmentation (partitioning) of A into A=A[1..i 1 ][A[i 1 +1 i 2 ] A[i k-1 n], where 1 i 1 < i 2 < i k 1 <n, such that numbers in A[i k +1 i k +1 ] are closest to c k +1 among centers in C. c 1 k clusters segment= 1d cluster Design and Analysis of Algorithms 2017 week

19 1D CLUSTERING: L2 AND AVERAGE Now, we seek for segmentation A=A[1..i 1 ][A[i 1 +1 i 2 ] A[i k- 1 n] minimizing σk k =1 c L2 (i k 1+1..ik ), where i 1 =0 and i k =n, and c c L2 (a..c)=σ b=a dev(a b, AVG(A[a.. c])), where c AVG(A[a..c])=σ b=a A b /(c a + 1). A[1..3]=0,4, Clustering with k=1 AVG(0,4,5)=3 c L2 (1..3)=(3-0) 2 +(3-4) 2 +(3-5) 2 =14 Design and Analysis of Algorithms 2017 week

20 1D CLUSTERING: RECURRENCE devmin[j,k ]= minimum total deviation after segmenting A[1..j] into k segments devmin[j,k ]=min i<j devmin [i,k -1]+c L2 (i+1..j) k -1 clusters min C sumdev(a,c) = devmin[n,k] (optimal solution) O(n 3 k) running time with direct implementation Optimizations, extensions, correctness, initialization, traceback in study groups & exercises. + i i+1 j Design and Analysis of Algorithms 2017 week

21 1D CLUSTERING: EVALUATION ORDER devmin[j,k ]=min i<j devmin [i,k -1]+c L2 (i+1..j) Evaluation order: k -1 k i min ( ) + cost of arc j Column-by-column (above), row-by-row, Design and Analysis of Algorithms 2017 week

22 1D CLUSTERING: RECAP+INITIALIZATION A devmin[0..n,0..k] 0 k -1 k k k -1 min cost to segment A[1..i] to k -1 pieces more segments than points: d[i,k ]=, for k >i i AVG dev() min i<j devmin[..] + cost of arc j cost to add segment A[i+1..j] n min cost to segment A[1..j] to k pieces optimum solution: traceback following arcs 0 segments only possible for empty sequence: d[i,0]=, except d[0,0]=0

23 Optimal chain of matrix multiplications Dynamic programming is not always a shortest path problem 23 Design and Analysis of Algorithms 2017 week

24 OPTIMAL CHAIN OF MATRIX MULTIPLICATIONS m p p n x = n O(npm) time m m p q q n x m x p = n Design and Analysis of Algorithms 2017 week

25 OPTIMAL CHAIN OF MATRIX MULTIPLICATIONS m p q q n x m x p = n npm+nqp multiplications n=10, m=15, p=7, q=4: 1330 multiplications n m x m p x p q = n q mqp+nqm multiplications n=10, m=15, p=7, q=4: 1470 multiplications Design and Analysis of Algorithms 2017 week

26 OPTIMAL CHAIN OF MATRIX MULTIPLICATIONS Find an optimal order to multiply n matrices A 1, A 2,.., A n with dimensions l 1 l 2, l 2 l 3,, l n d, where d=l n + 1 M(i,j)= minimum cost to multiply A i, A i+1,.., A j M(i,j)=min k:i k j {M(i,k) +M(k+1,j) +l i l j + 1 l k + 1 }. A i A i+1 A k l k + 1 A k+1 A j l j + 1 l j + 1 l i x = l i l k + 1 Design and Analysis of Algorithms 2017 week

27 OPTIMAL CHAIN: PSEUDOCODE M(i,j): /* this is slow.*/ If i=j return 0 value = For k=i to j-1 do value = min(value, M(i,k)+M(k+1,j)+l(1)*l(j+1)*l(k+1)) return value return M(1,n) m[i,j]= for all i,j MM(i,j): /* memoized variant, this is fast.*/ If i=j {m[i,i]=0; return 0} If m[i,j]= then For k=i to j-1 do m[i,j] = min(m[i,j], MM(i,k)+MM(k+1,j)+l(1)*l(j+1)*l(k+1)) return m[i,j] return MM(1,n) Design and Analysis of Algorithms 2017 week

28 OPTIMAL CHAIN: ANALYSIS Each vertex of the dependency DAG created ones O(n 3 ) time Better to unroll the recursion by computing intervals of length 1,2,3,,n, in that order (see the book) M(i,k) M(i,j) M(k+1,j) Optimization over directed hyperedges Design and Analysis of Algorithms 2017 week

29 OPTIMAL RECURSION TREE AND BALANCED PARANTHESES x x x A B x x C D E F ((AxB)x((CxD)x(ExF))) Design and Analysis of Algorithms 2017 week

30 Range minimum query problem Another example of the power of dynamic programming 30 Design and Analysis of Algorithms 2017 week

31 LCA VS RMQ v x r x w=lca(c,f) x A B p x x q min C D E F Height= DFS= rvavbvrwpcpdpwqeqfqwr arg min LCA = lowest common ancestor RMQ= range minimum query Design and Analysis of Algorithms 2017 week

32 RMQ: DYNAMIC PROGRAMMING Later in the course: O(n log n) time dynamic programming algorithm to preprocess an array for O(1) time range minimum queries. [Section 3: A Faster RMQ Algorithm] Bender & Farach-Colton. The LCA Problem Revisited. LATIN Same article also uses Cartesian trees, reductions, and so-called four-russians technique to improve preprocessing to optimal O(n) time (under RAM model of computation). Design and Analysis of Algorithms 2017 week

33 DYNAMIC PROGRAMMING IN OTHER COURSES Viterbi (kind of clustering, segmentation) Biological Sequence Analysis, period III, 2018 Shortest paths, edit distance, sequence alignment Algorithms for Bioinformatics, period I, 2017 String Processing Methods, period II, 2018 Biological Sequence Analysis, period III, 2018 In many others as a common tool Design and Analysis of Algorithms 2017 week