Exact vs approximate search. Text Algorithms (6EAP) Similarity. Problem. Edit distance (Levenshtein distance) 10/11/10. Similarity measures

Transcription

1 Exact vs approximate search Text Algorithms (6EAP) Similarity measures Jaak Vilo 21 fall In exact search we searched for a string or set of strings in a long text There are plenty of applicakons that require approximate search Approximate matching, i.e. find those regions in a long text that are similar to the query string E.g. to find substrings of S that have edit distacne < k to query string m. Jaak Vilo MTAT.3.19 Text Algorithms 1 Problem Given P and S find all approximate occurrences of P in S Similarity How can we measure the similarity of two strings? When are the two things almost the same? Edit distance (Levenshtein distance) Smallest nr of edit operakons to convert one string into the other I N D U S T R Y I N T E R E S T I N D U S T R Y I N T E R E S T Defini.on The edit distance D(A,B) between strings A and B is the minimal number of edit operakons to change A into B. Allowed edit operakons are delekon of a single le[er, inserkon of a le[er, or replacing one le[er with another. Let A= a 1 a 2... a m and B= b 1 b 2... b m. E1: Dele.on a i ε E2: Inser.on ε b i E3: Subs.tu.on a i b j (if a i b j ) Other possible variants: E4: Transposi.on a i a i+1 b j b j+1 and a i =b j+1 ja a i+1 =b j (e.g. lecture letcure) 1

2 How can we calculate this? How can we calculate this efficiently? α β a b D(αa, βb) = min 1. D(α, β) if a=b 2. D(α, β)+1 if a b 3. D(αa, β)+1 4. D(α, βb)+1 D(S,T) = d(i,j) = min Define: d(i,j) = D( S[1..i], T[1..j] ) min 1. D(S[1..n-1], T[1..m-1] ) + (S[n]=T[m])? : 1 2. D(S[1..n], T[1..m-1] ) D(S[1..n-1], T[1..m] ) d(i-1,j-1) + (S[n]=T[m])? : 1 2. d(i, j-1) d(i-1, j) +1 Recursion Recursion? F()=1 F(1)=1 F(n) = F(n- 1)+F(n- 2) 1,1,2,3,5,8, sub fib(int x) if (x<3) return 1; else return fib(x-1)+fib(x-2); i x d(i-1,j-1) d(i,j-1) j y d(i-1,j) d(i,j) Recursion? Algorithm Edit distance D(A,B) using Dynamic Programming (DP) i m j n n d(i-1,j-1) d(i-1,j) d(i,j-1) d(i,j) Input: A=a 1 a 2...a n, B=b 1 b 2...b m Output: Value d mn in matrix (d ij ), i m, j n. for i= to m do d i =i ; for j= to n do d j =j ; for j=1 to n do for i=1 to m do return d mn d ij = min( d i- 1, j- 1 + (if a i ==b j then else 1), d i- 1, j + 1, d i, j ) 2

3 Dynamic Programming i x m y n n d(i-1,j-1) d(i-1,j) d(i,j-1) j d(i,j) Edit distance is a metric It can be shown, that D(A,B) is a metric D(A,B), D(A,B)= iff A=B D(A,B) = D(B,A) D(A,C) D(A,B) + D(B,C) indust-r-y- in---terest Alignment Path of edit operakons OpKmal solukon can be calculated aterwards Quite typical in dynamic programming Memorize sets pred[i,j] depending from where the d ij was reached. 3

4 Three possible minimizing paths Add into pred[i,j] (i- 1,j- 1) if d ij = d i- 1,j- 1 + (if a i ==b j then else 1) (i- 1,j) if d ij = d i- 1,j + 1 (i,j- 1) if d ij = d i,j The path (in reverse order) ε c 6, b 5 b 5, c 4 c 4, a 3 a 3, a 2 b 2, b 1 a 1. MulKple paths possible All paths are correct There can be many (how many?) paths What are the other queskons in edit distance caclulakons? Space complexity Time complexity Other ways to look at the algorithm(s) ApplicaKons More complex nokons of similarity Space can be reduced CalculaKon of D(A,B) in space Θ(m) Input: A=a 1 a 2...a m, B=b 1 b 2...b n (choose m<=n) Output: d mn =D(A,B) for i= to m do C[i]=i for j=1 to n do C=C[]; C[]=j; for i=1 to m do d = min( C + (if a i ==b j then else 1), C[i- 1] + 1, C[i] + 1 ) C = C[i] // memorize new diagonal value C[i] = d write C[m] Time complexity is Θ(mn) since C[..m] is filled n Kmes 4

5 Shortest path in the graph h[p://en.wikipedia.org/wiki/shortest_path_problem Shortest path in the graph h[p://en.wikipedia.org/wiki/shortest_path_problem All nodes at distance 1 from source Edit distance = shortest path in ObservaKons? Shortest path is close to the diagonal If a short distance path exists Values along any diagonal can only increase (by at most 1) Diagonal Diagonal lemma Property of any diagonal: The values of matrix (d ij ) can on any specific diagonal either increase by 1 or stay the same Lemma: For each d ij, 1 i m, 1 j n holds: d ij =d i- 1,j- 1 or d ij = d i- 1,j (nokce that d ij and d i- 1,j- 1 are on the same diagonal) Proof: Since d ij is an integer, show: d ij d i- 1, j d ij d i- 1,j- 1 From the definikon of edit distance 1. holds since d ij d i- 1, j Diagonal nr. 2, d 2, d 13, d 24, d 35, d 46 Diagonal k, -m k n, s.t. diagonal k contains only d ij where j-i = k. InducKon on i+j : Basis is trivial when i= or j= (if we agree that d - 1, j- 1 =d j ) InducKon step: there are 3 possibilikes - On minimizakon the d ij is calculated from entry d i- 1, j- 1, or d ij d i- 1,j- 1 On minimizakon the d ij is calculated from entry d i- 1, j, or d ij =d i- 1,j +1 d i- 2,j d i- 1,j- 1 On minimizakon the d ij is calculated from entry d i,j- 1. Analogical to 2. Hence, d i- 1,j- 1 d ij 5

6 Transform the matrix into f kp For each diagonal only show the posikon (row index) where the value is increased by 1. Also, one can restrict the matrix (d ij ) to only this part where d ij d mn since only those d ij can be on the shortest path. We'll use the matrix (f kp ) that represents the diagonals of d ij f kp is a row index i from d ij, such that on diagonal k the value p reaches row i (d ij =p and j- i=k). IniKalizaKon: f,- 1 =- 1 and f kp =- when p k - 1 ; d mn = p, such that f n- m,p =m Calcula.ng matrix (f kp ) by columns Assume the column p- 1 has been calculated in (f kp ), and we want to calculate f kp. (the region of d ij =p) On diagonal k values p reach at least the row t= max ( f k,p- 1 +1, f k- 1,p- 1, f k+1,p- 1 +1) if the diagonal k reaches so far. If on row t+1 addikonally a i = b j on the same diagonal, then d ij cannot increase, and value p reaches row t+1. Repeat previous step unkl a i b j on diagonal k. f k,p same diagonal f k- 1,p- 1 - diagonal below f k+1,p diagonal above Algorithm A(): calculate f kp A(k,p) 1. t = max( f k,p- 1 +1, f k- 1,p- 1, f k+1,p- 1 +1) 2. while a t+1 == b t+1+k do t = t+1 3. f kp = if t>m or t+k >n then undefined else t f,2 t= max (3,2,3) = 2 A[3]=B[3], A[5]=B[5] => f(,2) = 5 f(1,2) t 6

7 Algoritm: Diagonal method by columns p= - 1 while f n- m,p m p=p+1 for k= - p to p do // f kp = A(k,p) t = max( f k,p- 1 +1, f k- 1,p- 1, f k+1,p- 1 +1) while a t+1 = b t+1+k do t = t+1 f kp = if t>m or t+k >n then undefined else t p can only ocur on diagonals - p k p. Method can be improved since k is oten such that f kp is undefined. We can decrease values of k: - m k n (diagonal numbers) Let m n and d ij on diagonal k. if - m k then k d ij m if 1 k n then k d ij k+m Hence, - m k m if p m and p- m k p if p m Some notes In applicakons small D(A,B) are most intereskng. Can modify the algorithm by providing maximum t Hence, O(tm) - the smaller the t, the faster the algorithm. Space can be reduced by keeping only previous column How to output the shortest path? RelaKvely simple, in Kme O(s), outputs a single path. Extensions to basic edit distance New operakons Variable costs TransposiKon (ab ba) E4: Transposi.on a i a i+1 b j b j+1, s.t. a i =b j+1 and a i+1 =b j (e.g.: lecture letcure ) d(i,j) = min 1. d(i-1,j-1) + (S[n]=T[m])? : 1 2. d(i, j-1) d(i-1, j) d(i-2,j-2) + ( if S[i-1,i] = T[j,j-1] then 1 else ) 7

8 Longest common subsequences The edit distance algorithm can be changed easily Space efficiency can also be achieved using 2 last columns, hence skll O(m) Diagonal method can be modified, since the diagonal lemma holds. Algorithms can be modified in a relakvely straigh orward manner Defini.on. String C=c 1 c 2...c r is a subsequence (alamjada) of A=a 1 a 2...a m if by removing from A null or more characters, one can get C. String C=c 1 c 2...c r is the longest common subsequence, LCS (pikim ühine alamjada) of A=a 1 a 2...a m and B=b 1 b 2...b n if C is the longest string that is both the subsequence of A and B. C=LCS(A,B) The length of LCS(A,B), C, can be used as the similarity measure for A and B. LCS(A,B) can be calculated similarly to edit distance LCS(A,B) = ( A + B - D'(A,B) )/2 Let D'(A,B) the edit distance where the only allowed opera.ons are inserkon and delekon (no replace). Theorem a) LCS(A,B) = ( A + B - D'(A,B) )/2 b) Lets have two sets D'(A,B) with opkmal nr of changes: 1. a i1 ε, a i2 ε,... a ip ε dele.ons from A and 2. ε b j1, ε b j2,... ε b jr inser.ons into B. Then LCS(A,B)=C can be constructed such that, C is A ater delekons of 1. and C is B ater delekon of all inserkons 2. (inserkons in 2. are reversely delekons from B). Proof b) According to construckon, C is uniquely defined C is a subsequence of A as well as B. If C was not the longest, then there would be C', C < C' s.t. C'=LCS(A,B). But then D'(A,B) A - C' + B - C' < A - C + B - C = D'(A,B), which is a contradickon. Hence, C=LCS(A,B). Proof a) According to b) LCS(A,B) = A - p and LCS(A,B) = B - r, or 2 LCS(A,B) = A + B - (p+r)= A + B - D'(A,B). Example. LCS(england,inglismaa)=ngla. D'(england,inglismaa)=8, ngla =4=(7+9-8)/2. Diagonal lemma holds, but the increase always occurs by two. Time complexity O(mn), with diag. method O (min(s,m) s) where s=d'(a,b), m= A, n= B. There exists other algorithms for LCS (e.g. Hunt- Szymanski) Unix command diff compares files row by row and searches the deviakons from the LCS of the two files. 8

9 Generalized edit distance Use more operakons E1...En, and to provide different costs to each. Defini.on. Let x, y Σ *. Then every x y is an edit operakon. Edit operakon replaces x by y. If A=uxv then ater the operakon, A=uyv We note by w(x y) the cost or weight of the operakon. Cost may depend on x and/or y. But we assume w(x y). Generalized edit distance If operakons can only be applied in parallel, i.e. the part already changed cannot be modified again, then we can use the dynamic programming. Otherwise it is an algorithmically unsolvable problem, since queskon - can A be transformed into B using operakons of G, is unsolvable. The diagonal method in general may not be applicable. But, since each diversion from diagonal, the cost slightly increases, one can stay within the narrow region around the diagonal. Applica.ons of generalized edit distance Examples Historic documents, names Human language and dialects TransliteraKon rules from one alphabet to another e.g. Tõugu => Tyugu (via Russian)... 9

10 näituseks näiteks Ahwrika - Aafrika weikese - väikese materjaali - materjali tuseks -> teks a -> aa, hw -> f w -> v, e -> ä aa -> a How? Apply Aho- Corasick to match for all possible edit operakons Use minimum over all possible such operakons and costs kavalam otsimine Dush, dušš, dushsh? Gorbatšov, Gorbatshov, Горбачов, Gorbachev režiim, rezhiim, riim ImplementaKon: Reina Käärik Possible problems/tasks Manually create sensible lists of operakons For English, Russian, etc Old language, Improve the speed of the algorithm (teskng) Train for automa.c extrac.on of edit opera.ons and respec.ve costs from examples of matching words 1