Weighted Finite-State Transducers in Computational Biology

Transcription

1 Weighted Finite-State Transducers in Computational Biology Mehryar Mohri Courant Institute of Mathematical Sciences Joint work with Corinna Cortes (Google Research).

2 This Tutorial Weighted finite-state transducers and software library Pairwise alignments for bioinformatics Learning weighted transducers page 2 2

3 General Definitions Weighted finite-state transducers: b:a/.2 b:a/.6 a:b/. a:b/.5 a:a/.4 2 b:a/.3 3/. Kernels: similarity measures between vectors, sequences or other structures. Efficient computation of inner products in highdimensional feature spaces. Extensive use in modern machine learning. Must satisfy a mathematical requirement (Mercer s condition). page 3 3

4 Example Problem: find the n best pairwise alignments between two very large sets of sequences (e.g., set >,). Example: S S2 a b a a b b b b b a a a b a b b a a a a a b b a a b b a b b b a a b a a b b b a a a b b b a b a b a b b b a b b a a b b a a ε a ε b b a a a a a b b b a a a page 4 4

5 Software Libraries FSM Library: Finite-State Machine Library. General software utilities for building, combining, optimizing, and searching weighted automata and transducers (MM, Pereira, and Riley, 2). OpenFst Library: Open-source Finite-state transducer Library (Allauzen et al., 27). page 5 5

6 Software Libraries GRM Library: Grammar Library. General software collection for constructing and modifying weighted automata and transducers representing grammars and statistical language models (Allauzen, MM, and Roark, 25). DCD Library: Decoder Library. General software collection for speech recognition decoding and related functions (MM and Riley, 23). page 6 6

7 FSM Library The FSM utilities construct, combine, minimize, and search weighted finite-states transducers. User Program Level: Programs that read from and write to files or pipelines, fsm(): fsmintersect in.fsm in2.fsm >out.fsm C(++) Library Level: Library archive of C(++) functions that implements the user program level, fsm(3): Fsm in = FSMLoad("in.fsm"); Fsm in2 = FSMLoad("in2.fsm"); Fsm out = FSMIntersect(fsm, fsm2); FSMDump("out.fsm", out); page 7 7

8 Definition Level: Specification of labels, of costs, and of types of FSM representations. page 8 8

9 This Lecture Weighted automata and transducers Rational operations Elementary unary operations Fundamental binary operations Optimization algorithms Search algorithms 9 page 9

10 Textual format FSM File Types automata/acceptor files, transducer files, symbols files. Binary format: compiled representation used by all FSM utilities. page

11 Compiling, Printing, and Drawing Compiling fsmcompile -s tropical -ia.syms <A.txt >A.fsm fsmcompile -s log -ia.syms -oa.syms -t <T.txt >T.fsm Printing fsmprint -ia.syms <A.fsm >A.txt fsmprint -ia.syms -oa.syms <T.fsm dot -Tps >T.ps Drawing fsmdraw -ia.syms <A.fsm dot -Tps >A.ps fsmdraw -ia.syms -oa.syms <T.fsm dot -Tps >T.ps page

12 Weight Sets: Semirings A semiring negation. (K,,,, ) is a ring that may lack sum: to compute the weight of a sequence (sum of the weights of the paths labeled with that sequence). product: to compute the weight of a path (product of the weights of constituent transitions). page 2 2

13 Semirings - Examples Semiring Set Boolean {, } Probability R + + Log R {, + } log + + Tropical R {, + } min + + with log defined by: x log y = log(e x + e y ). page 3 3

14 Automata/Acceptors Graphical Representation ( A.ps) red/.5 Acceptor file ( A.txt) red.5 green.3 2 blue 2 yellow green/.3 Symbols file ( A.syms) red green 2 blue 3 yellow 4 blue/ yellow/.6 2 /.8 page 4 4

15 Transducers Graphical Representation ( T.ps) red:yellow/.5 Transducer file ( T.txt) Symbols file ( T.syms) red green 2 blue 3 yellow 4 green:blue/.3 red yellow.5 green blue.3 2 blue green 2 yellow red blue:green/ yellow:red/.6 2 /.8 page 5 5

16 Paths - Definitions and Notation Path π input label output label p[π] i[π]:o[π] n[π] previous state or source state next state or destination state Sets of paths P (R,R 2 ) : paths from R Q to R 2 Q. P (R,x,R 2 ): paths in P (R,R 2 ) with input label x. P (R,x,y,R 2 ): paths in P (R,x,R 2 ) with output label y. page 6 6

17 General Definitions Alphabets: input Σ, output. States: Q, initial states I, final states F. Transitions: E Q (Σ {ɛ}) ( {ɛ}) K Q. Weight functions: initial: λ : I K. final: ρ : F K. page 7 7

18 Automata and Transducers - Definitions Automaton A =(Σ,Q,I,F,E,λ,ρ) x Σ, [[A]](x) = π P (I,x,F) Transducer T =(Σ,,Q,I,F,E,λ,ρ) x Σ,y, [[T ]](x, y) = λ(p[π]) w[π] ρ(n[π]) π P (I,x,y,F) λ(p[π]) w[π] ρ(n[π]) page 8 8

19 Weighted Automata b/.6 b/.2 a/. a/.5 a/.4 2 b/.3 3/. [[A]](x) = Sum of the weights of all successful paths labeled with x [[A]](abb) = page 9 9

20 Weighted Transducers b:a/.6 b:a/.2 a:b/. a:b/.5 a:a/.4 2 b:a/.3 3/. [[T ]](x, y) = Sum of the weights of all successful paths with input x and output y. [[T ]](abb, baa) = page 2 2

21 This Lecture Weighted automata and transducers Rational operations Elementary unary operations Fundamental binary operations Optimization algorithms Search algorithms 2 page 2

22 Rational Operations Sum [[T T 2 ]](x, y) =[[T ]](x, y) [[T 2 ]](x, y) Product [[T T 2 ]](x, y) = Closure [[T ]](x, y) = [[T ]] n (x, y) n= [[T ]](x,y ) [[T 2 ]](x 2,y 2 ). x=x x 2 y=y y 2 page 22 22

23 Conditions (on the closure operation): condition on T: e.g., weight of ε-cycles = (regulated transducers), or semiring condition: e.g., x = as with the tropical semiring (more generally locally closed semirings). Complexity and implementation: linear-time complexity: O(( E + Q )+( E 2 + Q 2 )) O( Q + E ) or lazy implementation. page 23 23

24 Sum - Illustration Program: fsmunion A.fsm B.fsm >C.fsm Graphical representation: red/.5 green/.4 / green/.3 blue/ yellow/.6 2 /.8 blue/.2 2 /.3 red/.5 6 eps/ eps/ green/.3 blue/ 2 /.8 yellow/.6 3 green/.4 4 / blue/.2 5 /.3 page 24 24

25 Product - Illustration Program: fsmconcat A.fsm B.fsm >C.fsm Graphical representation: red/.5 green/.4 / green/.3 blue/ yellow/.6 2 /.8 blue/.2 2 /.3 red/.5 green/.4 4 / green/.3 blue/ yellow/.6 2 eps/.8 3 blue/.2 5 /.3 page 25 25

26 Closure - Illustration Program: fsmclosure B.fsm >C.fsm Graphical representation: green/.4 / green/.4 eps/ / blue/.2 3 / eps/ blue/.2 2 /.3 eps/.3 2 /.3 page 26 26

27 This Lecture Weighted automata and transducers Rational operations Elementary unary operations Fundamental binary operations Optimization algorithms Search algorithms 27 page 27

28 Elementary Unary Operations Reversal Inversion [[ T ]](x, y) =[[T ]]( x, ỹ) [[T ]](x, y) =[[T ]](y, x) Projection [[A]](x) = y [[T ]](x, y) Linear-time complexity, lazy implementation (not for reversal). page 28 28

29 Reversal - Illustration Program: fsmreverse A.fsm >C.fsm Graphical representation: red/.5 green/.3 blue/ yellow/.6 2 green/.2 blue/2 3 / 4 /.3 red/.5 eps/ eps/ green/.2 blue/2 3 blue/ yellow/.6 2 green/.3 / page 29 29

30 Program: fsminvert A.fsm >C.fsm Graphical representation: red:bird/.5 Inversion - Illustration green:pig/.3 blue:cat/ yellow:dog/.6 2 /.8 bird:red/.5 pig:green/.3 cat:blue/ dog:yellow/.6 2 /.8 page 3 3

31 Projection - Illustration Program: fsmproject - T.fsm >A.fsm Graphical representation: red:bird/.5 green:pig/.3 blue:cat/ yellow:dog/.6 2 /.8 red/.5 green/.3 blue/ yellow/.6 2 /.8 page 3 3

32 This Lecture Weighted automata and transducers Rational operations Elementary unary operations Fundamental binary operations Optimization algorithms Search algorithms 32 page 32

33 Some Fundamental Binary Operations Composition ( (K,,,, ) commutative) [[T T 2 ]](x, y) = [[T ]](x, z) [[T 2 ]](z,y) z Intersection ( (K,,,, ) commutative) [[A A 2 ]](x) =[[A ]](x) [[A 2 ]](x) Difference ( A 2 unweighted and deterministic) [[A A 2 ]](x) =[[A A 2 ]](x) (Pereira and Riley, 997; MM et al. 996) page 33 33

34 Complexity and implementation: quadratic complexity: O(( E + Q )( E 2 + Q 2 )) path multiplicity in presence of ε-transitions: ε- filter; lazy implementation. page 34 34

35 Composition - Illustration Program: fsmcompose A.fsm B.fsm >C.fsm Graphical representation: c:a/.3 a:b/.6 a:b/. b:a/.2 2 a:a/.4 b:b/.5 3/.6 b:c/.3 a:b/.4 2/.7 c:b/.9 c:b/.7 (, 2) a:b/ a:c/.4 (, ) (, ) a:b/.8 (3, 2)/.3 page 35 35

36 Multiplicity and ε-transitions - Problem a:d!:e,,,2 (x:x) (!:!) b:! (!2:!2) b:e (!2:!) b:! (!2:!2) a:a b:! 2 c:! 3 d:d 4 c:!!:e 2, 2,2 (!:!) c:! (!2:!2) (!2:!2) a:d!:e 2 d:a 3!:e 3, 3,2 (!:!) d:a (x:x) 4,3 page 36 36

37 Solution - Filter F for Composition!:!!2:! x:x!:! x:x!2:!2!2:!2 x:x 2 Replace T T 2 with T F T 2. page 37 37

38 Intersection - Illustration Program: fsmintersect A.fsm B.fsm >C.fsm Graphical representation: red/.5 green/.4 green/.3 blue/ yellow/.6 2 /.8 / red/.2 blue/.6 yellow/.3 2 /.5 blue/.6 3 /.8 red/.7 green/.7 2 yellow/.9 4 /.3 page 38 38

39 Difference - Illustration Program: fsmdifference A.fsm B.fsm >C.fsm Graphical representation: red/.5 green green/.3 blue/ yellow/.6 2 /.8 red blue yellow 2 red/.5 red/.5 red/.5 green/.3 2 green/.3 3 blue/ yellow/.6 4 /.8 page 39 39

40 This Lecture Weighted automata and transducers Rational operations Elementary unary operations Fundamental binary operations Optimization algorithms Search algorithms 4 page 4

41 Optimization Algorithms Connection: removes non-accessible/noncoaccessible states. ε-removal: removes ε-transitions. Determinization: creates equivalent deterministic machine. Pushing: creates equivalent pushed/stochastic machine. Minimization: creates equivalent minimal deterministic machine. page 4 4

42 Conditions: there are specific semiring conditions for the use of these algorithms, e.g., not all weighted automata or transducers can be determinized using the determinization algorithm. page 42 42

43 Connection - Illustration Program: fsmconnect A.fsm >C.fsm Graphical representation: green/ /.2 red/.5 blue/ 2 /.8 green/.3 yellow/.6 red/ 5 red/.5 green/.3 blue/ yellow/.6 2 /.8 page 43 43

44 Connection - Algorithm Definition: Input: weighted transducer T. Output: equivalent weighted transducer T 2 with all states connected. Description: 3. Depth-first search of from. T I 4. Mark accessible and coaccessible states. 5. Keep marked states and corresponding transitions. Complexity: linear O( Q + E ). page 44 44

45 ε-removal - Illustration Program: fsmrmepsilon T.fsm >TP.fsm Graphical representation: a:b/.!:!/.2 b:a/.4!:!/.3!:!/.6 b:a/.9 b:b/.5 3!:!/ 2 a:a/.8 b:a/.7 4/ 4/ a:a/.6 b:a/ b:a/.7 a:b/. b:b/.7 b:a/.5 b:b/.5 b:a/2.3 a:a/.4 b:a/.7 4/ b:a/.4 b:a/.9 a:a/ b:a/.7 page 45 45

46 Definition: Input: weighted transducer. Output: equivalent WFST Description: Computation of ε-closures. with no ε-transition. Removal of εs. Complexity: T ɛ : O( Q 2 + Q E (T + T )). Acyclic ε-removal - Algorithm T 2 T General case (tropical semiring): O( Q E + Q 2 log Q ) (MM, 2) page 46 46

47 Computation of ε-closures Definition: for p in Q, C[p] = { (q, w) :q ɛ[p], d[p, q] =w }, where d[p, q] = w[π]. π P (p,ɛ,q) Problem formulation: all-pairs shortest-distance problem in T ɛ (T reduced to its ε-transitions). closed semirings: generalization of Floyd- Warshall algorithm. k-closed semirings: generic sparse shortestdistance algorithm. page 47 47

48 Determinization - Algorithm Definition: Input: weighted automaton or transducer (MM,997) Output: equivalent subsequential or deterministic machine T 2 : has a unique initial state and no two transitions leaving the same state share the same input label. Description: 3. Generalization of subset construction: weighted subsets {(q,w ),...,(q n,w n )}, where w i s are remainder weights. 4. Computation of the weight of resulting transitions. T page 48 48

49 Determinization - Conditions Semiring: weakly left divisible semirings. Definition: T is determinizable when the determinization algorithm applies to T. All unweighted automata are determinizable. All acyclic machines are determinizable. Not all weighted automata or transducers are determinizable. Characterization based on the twins property. Complexity: exponential, but lazy implementation. page 49 49

50 Determinization of Weighted Automata - Illustration Program: fsmdeterminize A.fsm >D.fsm Graphical representation: a/ b/4 2 b/ b/3 a/3 b/3 3/ b/ / b/5 a/ {(,2),(2,)}/2 b/ {(,)} b/ b/3 {(3,)}/ {(,),(2,3)}/ page 5 5

51 Determinization of Weighted Transducers - Illustration Program: fsmdeterminize T.fsm >D.fsm Graphical representation: a:a/. a:b/.2 a:c/.5 2 b:b/.3 b:eps/.4 3 c:eps/.7 a:eps/.5 a:b/.6 4/.7 a:a/.2 {(, b, )} a:b/. {(, eps, )} a:eps/. {(, c,.4), (2, a, )} b:a/.3 {(3, b, ), (4, eps,.)} /(eps,.8) a:b/.5 a:b/.6 c:c/. {(4, eps, )} /(eps,.7) page 5 5

52 Pushing - Algorithm (MM,997; 24) Definition: Input: weighted automaton or transducer Output: equivalent automaton or transducer such that the longest common prefix of all T T 2 outgoing paths be ε or such that the sum of the weights of all outgoing transitions be modulo the string or weight at the initial state. page 52 52

53 Description:. Single-source shortest distance computation: for each state q, d[q] = w[π]. π P (q,f) 2. Reweighting: for each transition e such that d[p[e]], w[e] (d[p[e]]) (w[e] d[n[e]]) page 53 53

54 Conditions (automata case): weakly divisible semiring, zero-sum free semiring or automaton. Complexity: automata case acyclic case: linear general case (tropical semiring): transducer case: O( Q + E (T + T )). O( Q log Q + E ). O(( P max + ) E ). page 54 54

55 Weight Pushing - Illustration Program: fsmpush -ic A.fsm >P.fsm Graphical representation: Tropical semiring: a/ b/ c/4 d/ e/ f/ e/ 3/ a/ b/ c/4 d/ e/ f/ e/ 3/ e/ 2 f/ e/ 2 f/ page 55 55

56 Log semiring: a/ b/ c/4 d/ e/ f/ e/ 3/ a/.3266 b/.3266 c/ d/.326 e/.332 f/.332 e/.332 3/-.639 e/ 2 f/ e/ f/.332 page 56 56

57 Label Pushing - Illustration Program: fsmpush -il T.fsm >P.fsm Graphical representation: a:a c:! b:d a:! 2 b:! 3 d:! c:d a:d 4 a:! 5 f:d 6 a:a c:d b:! a:d 2 b:! 3 d:d c:! a:d 4 a:! 5 f:! 6 page 57 57

58 Definition: Minimization - Algorithm (MM,997) Input: deterministic weighted automaton or transducer T. Output: equivalent deterministic automaton or transducer T 2 with the minimal number of states and transitions. Description: Canonical representation: use pushing or other algorithm to standardize input automata. Automata minimization: encode pairs (label, weight) as labels and use classical unweighted minimization algorithm. page 58 58

59 Complexity: Automata case acyclic case: linear, O( Q + E (T + T )). general case (tropical semiring): Transducer case O( E log Q ). acyclic case: O(S + Q + E ( P max + )). general case (tropical semiring): O(S + Q + E (log Q + P max )). page 59 59

60 Minimization - Illustration Program: fsmminimize D.fsm >M.fsm Graphical representation: b/ 2 c/2 d/ a/ b/ a/2 d/3 3 e/2 d/4 e/3 5 c/3 c/ 6 e/ 7/ 4 b/ 2 c/ d/ a/ b/ a/3 d/6 3 e/ d/6 e/ 5 c/ c/ 6 e/ 7/6 d/ a/ b/ b/ a/3 4 c/ 2 c/ e/ d/6 3 6 e/ 7/6 page 6 6

61 Definition: Description Equivalence - Algorithm Input: deterministic weighted automata A and B. Output: TRUE iff A and B equivalent. (MM,997): Canonical representation: use pushing or other algorithm to standardize input automata. Automata minimization: encode pairs (label, weight) as labels and use classical algorithm for testing the equivalence of unweighted automata. Complexity: (second stage is quasi-linear) O( E + E 2 + Q log Q + Q 2 log Q 2 ). page 6 6

62 Equivalence - Illustration Program: fsmequiv [-v] D.fsm M.fsm Graphical representation: blue/.7 2 /.3 red/.3 yellow/.9 3 /.4 red/ blue/ yellow/.3 2 /.3 page 62 62

63 This Lecture Weighted automata and transducers Rational operations Elementary unary operations Fundamental binary operations Optimization algorithms Search algorithms 63 page 63

64 Single-Source Shortest-Distance Algorithms - Illustration Program: fsmbestpath [-n N] A.fsm >C.fsm Graphical representation: red/.5 red/.5 red/.5 green/.3 2 green/.3 3 blue/ yellow/.6 4 /.8 green/.3 blue/ 2 /.8 page 64 64

65 Pruning - Illustration Program: fsmprune -c. A.fsm >C.fsm Graphical representation: red/.5 red/.5 red/.5 green/.3 2 green/.3 3 blue/ yellow/.6 4 /.8 green/.3 blue/ yellow/.6 2 /.8 page 65 65

66 Summary FSM Library: weighted finite-state transducers (semirings); elementary unary operations (e.g., reversal); rational operations (sum, product, closure); fundamental binary operations (e.g., composition); optimization algorithms (e.g., ε-removal, determinization, minimization); search algorithms (e.g., shortest-distance algorithms, n-best paths algorithms, pruning). page 66 66

67 References Cyril Allauzen and Mehryar Mohri. Efficient Algorithms for Testing the Twins Property. Journal of Automata, Languages and Combinatorics, 8(2):7-44, 23. Cyril Allauzen, Mehryar Mohri, and Brian Roark. The Design Principles and Algorithms of a Weighted Grammar Library. International Journal of Foundations of Computer Science, 6(3): 43-42, 25. Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut, and Mehryar Mohri. OpenFst: a general and efficient weighted finite-state transducer library. In Proceedings of the Ninth International Conference on Implementation and Application of Automata, (CIAA 27), volume 4783 of Lecture Notes in Computer Science, pages -23. Springer, Berlin, 27. Mehryar Mohri. Finite-State Transducers in Language and Speech Processing. Computational Linguistics, 23:2, 997. Mehryar Mohri. Weighted Grammar Tools: the GRM Library. In Robustness in Language and Speech Technology. pages Kluwer Academic Publishers, The Netherlands, 2. Mehryar Mohri. Statistical Natural Language Processing. In M. Lothaire, editor, Applied Combinatorics on Words. Cambridge University Press, 25. page 67 67

68 References Mehryar Mohri. Generic Epsilon-Removal and Input Epsilon-Normalization Algorithms for Weighted Transducers. International Journal of Foundations of Computer Science, 3(): 29-43, 22. Mehryar Mohri, Fernando C. N. Pereira, and Michael Riley. The Design Principles of a Weighted Finite-State Transducer Library. Theoretical Computer Science, 23:7-32, January 2. Mehryar Mohri, Fernando C. N. Pereira, and Michael Riley. Weighted Automata in Text and Speech Processing. In Proceedings of the 2th biennial European Conference on Artificial Intelligence (ECAI-96), Workshop on Extended finite state models of language. Budapest, Hungary, 996. John Wiley and Sons, Chichester. Mehryar Mohri and Michael Riley. A Weight Pushing Algorithm for Large Vocabulary Speech Recognition. In Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech'). Aalborg, Denmark, September 2. Fernando Pereira and Michael Riley. Finite State Language Processing, chapter Speech Recognition by Composition of Weighted Finite Automata. The MIT Press, 997. page 68 68