On the Computation of Document Frequency Statistics from Spoken Corpora using Factor Automata

Transcription

1 INTERSPEECH 3 On the Computaton of Document Frequency Statstcs from Spoken Corpora usng Factor Automata Doğan Can, Shrkanth S. Narayanan, Department of Computer Scence and Department of Electrcal Engneerng, Unversty of Southern Calforna, Los Angeles, CA dogancan@usc.edu, shr@sp.usc.edu Abstract Factor automaton s an effcent data structure for representng all factors (substrngs) of a set of strngs (e.g. a fnte-state automaton). Ths noton can be generalzed to weghted automata by assocatng a weght to each factor. In ths paper, we consder the problem of computng expected document frequency (DF), and TF-IDF statstcs for all substrngs seen n a collecton of word lattces by means of factor automata. We present an algorthm whch transforms an acyclc weghted automaton, e.g. an ASR lattce, to a weghted factor automaton where the path weght of each factor represents the total weght assocated by the nput automaton to the set of strngs ncludng that factor at least once. We show how ths automaton can be used to effcently construct other types of weghted factor automata representng DF and TF-IDF statstcs for all factors seen n a large speech corpus. Compared to the state-of-the-art n computng these statstcs from spoken documents, our approach ) generalzes the statstcs from sngle words to contguous substrngs, ) provdes sgnfcant gans n terms of average run-tme and storage requrements and ) constructs effcent nverted ndex structures for retreval of such statstcs. Experments on a Turksh data set corroborate our clams. Index Terms: Factor Automata, Weghted Fnte-State Transducers, Lattce Indexng, Spoken Document Retreval. Introducton Speech retreval (SR) s a key technology whch ntegrates automatc speech recognton (ASR) and nformaton retreval (IR) to provde large scale access to spoken content. Computng document smlarty s a fundamental need n SR just as n the case of text retreval. Inverse document frequency (IDF) [, ] s an mportant term specfcty measure used n almost every IR system n some form. In ts most basc form, t s defned as log of the fracton of documents that nclude a term. IDF computaton nvolves computng document frequency (DF), smply defned as the number of documents that nclude a term. Term frequency (TF) s another mportant measure, whch n ts most basc form s defned as the number of occurrences of a term n a document. The more general class of term weghtng schemes known as TF-IDF, whch nvolves multplyng an IDF measure by a TF measure, consttutes the bass for quantfyng document smlarty n almost every IR system. Whle t s farly straghtforward to compute TF and DF measures from textual documents, the same cannot be sad for spoken ones snce the computaton should be carred out over probablty dstrbutons Supported by the Intellgence Advanced Research Projects Actvty (IARPA) va Department of Defense U.S. Army Research Laboratory (DoD / ARL) contract number W9NF--C-. over strngs, e.g. ASR lattces. Approxmatng TF and DF measures by the statstcs of -best ASR output s a common strategy used n SR. However, as shown by [3], proper estmaton of these measures over ASR lattces provdes sgnfcant gans n performance over the -best baselne. Factor automaton (Secton.3) s an effcent data structure for representng all factors (substrngs) of a set of strngs (e.g. a fnte-state automaton). Ths noton can be generalzed to weghted fnte-state automata [, 5] by assocatng a weght to each factor (Secton 3). In ths paper, we consder the problem of computng TF, DF, and TF-IDF measures n the expected sense for all factors seen n a collecton of word sequence lattces by means of weghted factor automata. We frst ntroduce the concept of factor weght of a strng x, defned as the total weght assocated by a weghted automaton to the set of strngs whch nclude x as a substrng at least once (Secton 3). Then, we present a novel algorthm (Secton 3.) for transformng an acyclc probablstc automaton, e.g. an ASR lattce, nto a term probablty factor automaton whch s an deal ndex structure for spoken utterance retreval applcatons. We then show how ths automaton can be used to effcently construct other weghted factor automata representng DF (Secton 3.), IDF and TF-IDF (Secton 3.3) statstcs for all factors of a collecton of ASR lattces. Fnally we provde experments evaluatng the average run-tme and storage requrements of our approach and conclude wth a bref dscusson (Secton ).. Prelmnares Ths secton ntroduces the defntons and notaton related to weghted transducers and automata [6], as well as the defntons for TF, DF and TF-IDF measures as used n ths paper... Semrngs Defnton A semrng s a 5-tuple (K,,,, ) where (K,, ) s a commutatve monod, (K,, ) s a monod, dstrbutes over and s an annhlator for. Table : Common semrngs. SEMIRING SET Boolean {, } Real R + {+ } + Max-tmes R + {+ } max Log R {, + } log + + Tropcal R {, + } mn + + a log b = log(e a + e b ) Copyrght 3 ISCA August 3, Lyon, France

2 .. Weghted Fnte-State Transducers and Automata Defnton A weghted fnte-state transducer T over a semrng (K,,,, ) s an 8-tuple T = (Σ,, Q, I, F, E, λ, ρ) where: Σ, are respectvely the fnte nput and output alphabets; Q s a fnte set of states; I, F Q are respectvely the set of ntal and fnal states; E Q (Σ {ε}) ( {ε}) K Q s a fnte set of arcs; λ I K, ρ F K are respectvely the ntal and fnal weght functons. Gven an arc e E, we denote by [e] ts nput label, o[e] ts output label, w[e] ts weght, p[e] ts orgn state and n[e] ts destnaton state. A path π = e e k s an element of E wth consecutve arcs satsfyng n[e ] = p[e ], =,..., k. We extend n and p to paths by settng n[π] = n[e k ] and p[π] = p[e ]. The labelng and the weght functons can also be extended to paths by defnng [π] = [e ] [e k ], o[π] = o[e ] o[e k ] and w[π] = w[e ] w[e k ]. We denote by Π(q, q ) the set of paths from q to q and by Π(q, x, y, q ) the set of paths from q to q wth nput label x Σ and output label y. These defntons can be extended to subsets S, S Q, e.g. Π(S, x, y, S ) = Π(q, x, y, q ). q S,q S An acceptng path n a transducer T s a path n Π(I, F ). A strng x s accepted by T f there exsts an acceptng path π labeled wth x on the nput sde. T s unambguous f for any strng x Σ there s at most one acceptng path labeled wth x on the nput sde. T s determnstc f t has at most one ntal state and at any state no two outgong transtons share the same nput label. Let π and π be two paths of a transducer T wth the same nput and output labels: [π] = [π ] and o[π] = o[π ]. We say that π = e e n and π = e e n are dentcal f they have the same number of transtons (n = n ) wth the same labels: [e k ] = [e k] and o[e k ] = o[e k] for k =,..., n. A transducer T s sad to be normalzed, f any two paths π and π wth the same nput and output labels are dentcal. The weght assocated by a transducer T to any nput-output strng par (x, y) Σ s gven by T (x, y) = π Π(I,x,y,F ) λ(p[π]) w[π] ρ(n[π]) and T (x, y) s defned to be when Π(I, x, y, F ) =. We denote by s[a] the -sum of the weghts of all acceptng paths of A when t s defned and n K. s[a] can be vewed as the shortest-dstance from the ntal states to the fnal states. A weghted fnte-state automaton A can be defned as a weghted fnte-state transducer wth dentcal nput and output labels. The weght assocated by A to (x, x) s denoted by A (x). Smlarly, n the graphcal representaton of weghted automata, output labels are omtted. A weghted automaton A defned over the probablty semrng (R +, +,,, ) s sad to be probablstc f for any state q Q, π Π(q,q) w[π], the sum of the weghts of all cycles at q, s well-defned and n R + and x Σ A (x) =..3. Factor Automata Defnton 3 Gven two strngs x, y Σ, x s a factor (substrng) of y f y = uxv for some u, v Σ. More generally, x s a factor of a language L Σ f x s a factor of some strng y L. The factor automaton S(y) of a strng y s the mnmal determnstc fnte-state automaton recognzng exactly the set of factors of y. S(y) can be bult n lnear tme and ts sze s lnear n the sze of the nput strng y [7, 8]. We denote by S(A) the mnmal determnstc automaton recognzng the set of factors of a fnte-state automaton A, that s the set of factors of the strngs accepted by A. A recent work [9] showed that the sze of the factor automaton S(A) s lnear n the sze of the nput automaton A = Q + E and provded an algorthm for the constructon of S(A) n lnear tme when A s unweghted... Term Frequency, Document Frequency, TF-IDF We defne the term frequency T F (x, ) as the number of occurrences of a term x n document D, and the document frequency DF (x) as the fracton of documents ncludng the term x n a collecton of documents {D =,, n}: DF (x) = { x D}. () n TF-IDF can be defned n a number of ways by choosng a TF and an IDF defnton from a multtude of optons. Here we use: T F IDF (x, ) = T F (x, ) log DF (x). () 3. Factor Automata of Weghted Automata Factor automaton (FA) noton can be generalzed to weghted automata by assocatng a weght to each factor. When the nput automata are probablstc, there are several meanngful statstcs that can be used as a weght. The weghted ndex structure descrbed n [] uses expected term frequences derved from probablstc automata. In general, any weght satsfyng the semrng condtons can be used. For nstance, the tmed factor transducer structure descrbed n [5] uses (probablty, start tme, end tme) trplets derved from tme algned ASR lattces. The unweghted FA can be thought as a weghted automaton over the Boolean semrng where the path weghts represent factor occurrence,.e. each factor of the nput automaton s assgned weght, any other strng s assgned weght. The natural extenson of ths occurrence concept to probablstc automata s smply the probablty of occurrence (at least once). Gven a probablstc nput automaton, the probablty of occurrence of a factor s smply the sum of probabltes assgned to each acceptng path ncludng that factor. More formally, for a weghted fnte-state automaton A over the semrng K, we defne the factor weght A (x) of a strng x Σ as the -sum of the weghts of all acceptng paths ncludng x as a factor: A (x) = π Π(I,F ) u,v Σ [π]=uxv λ(p[π]) w[π] ρ(n[π]) (3) Over the Boolean semrng, factor weght s smply the logcal dsjuncton of Boolean path weghts and reduces to factor occurrence. Over the real (equvalently log) semrng, factor weght corresponds to the sum ( log -sum) of path weghts, whch s smply the probablty ( log probablty) of occurrence of a factor (at least once) when the nput automaton s probablstc. Next secton presents a recpe based on weghted transducer algorthms [6] for transformng an acyclc probablstc automaton nto a term probablty (TP) factor automaton where the path weght of each factor represents ts probablty of occurrence,.e. factor weght over the real semrng. TP factor automaton has an deal weght structure for spoken utterance retreval [, ] applcatons where the goal s to retreve all utterances 7

3 a/ b/ b/ a/ 3/ ε/.5 b/ a/ ε/ a/.5 3 / a/ /.5 b/ b/.5 /.5 a/.5 3/ (a) (b) (c) ε/ ε/.5 ε:α/ a:a/ ε:α3/ b:b/ ε:α/ a:a/ ε:α/ a:/ b:/ b:/.5 ε:/.5 3 ε:/.5 ε:/.5 ε:/ a:/ 5 ε:/ ε:/ 7 8 ε:/ ε:/ ε:/ / δ:β/ δ3:β3/ (d) 6/ δ:β/ δ:β/ Fgure : Constructon of S from A. (a) Input automaton (A) over the real semrng, (b) after preprocessng (B), (c) resultng factor automaton (S), (d) after factor generaton (T ), (e) after factor normalzaton and determnzaton over the max-tmes semrng (N). ε:/.5 ε:/.5 a:/.5 (e) 6 ε:/ a:/ 9 contanng a query term n a large spoken archve. We should note that, unlke TF whch can be expressed as a sum over the posteror probabltes of factor occurrences [], TP measure gven by (3) s a sum over the probabltes of acceptng paths ncludng a factor. Ths subtle dfference makes the TP computaton sgnfcantly harder snce we have to make sure that the acceptng paths ncludng a factor multple tmes are counted only once. The man dea s to construct a mappng from any factor to the acceptng paths ncludng t and elmnatng the path replcatons caused by factors occurrng multple tmes n a path before dong the probablty summaton. 3.. Term Probablty Factor Automaton Let A denote an acyclc automaton over the log (real) semrng. Fgure a gves such an automaton over the real semrng. We can derve the TP factor automaton S from A n four steps: 3... Preprocessng We assume the nput s an acyclc probablstc automaton over the log (real) semrng. When that s not the case, we can use the general weght-pushng algorthm [6] to convert nput weghts to posteror probabltes. We also requre the nput to satsfy three specfc condtons whch are crucal for the correctness of the algorthm: ) t should be unambguous, ) no two arcs comng nto a state should have the same label and ) t should be ε- free except for the arcs orgnatng from the ntal states. All of these condtons can be satsfed by reversng the nput automaton, applyng weghted epslon removal and determnzaton on the reverse machne, and then reversng the resultng automaton [6]. Fgure b llustrates the result of these operatons Generaton Let B = (Σ, Σ, Q, I, F, E, λ, ρ) denote the automaton obtaned after preprocessng A. We generate factors of B (Fgure d) smply by creatng a unque ntal state q I / Q, a unque fnal state q F / Q, and two new arcs for each nonntal state q Q I: (q I, ε, α q,, q) and (q, δ q, β q,, q F ). Here, δ q s a unque dsambguaton symbol. α q (or β q) s a unque non-termnal symbol representng the prefx transducer U q = ({ε}, Σ, Q, I, {q}, E, λ, ) (resp. suffx transducer V q = ({ε}, Σ, Q, {q}, F, E,, ρ)) derved from B by settng q as the sole fnal (resp. ntal) state and replacng the nput symbols wth ε,.e. E = {(p[e], [e], ε, w[e], n[e]) e E}. Techncally, the (root) transducer T obtaned after these changes and the set of fnte state transducers {U q q Q I} {V q q Q I} defne a recursve transton network (RTN) []. Due to the unque dsambguaton symbols {δ q q Q I} on the nput sde of the new arcs nto the unque fnal state q F and the fact that B s unambguous wth no two arcs comng nto a state havng the same label, T s a functonal transducer. Optonally, we can flter the generated factors by composng a fnte-state grammar G on the nput sde of T []. The default flter rejects empty factors,.e. any path π n T for whch the only non-ε nput label s the dsambguaton symbol at the end Normalzaton We obtan a normalzed transducer N from T by applyng weghted transducer determnzaton, replacng the dsambguaton symbols wth ε symbols, replacng the non-termnal symbols [] wth the correspondng prefx and suffx transducers and fnally applyng weghted epslon removal. N defnes a mappng from each factor x of B to the set of paths n B that nclude x as a factor. Weght of a path π n N equals B (o[π]), the weght assocated by B to the path defned by the output symbols o[π]. Furthermore, all paths n N wth the same nput and output labels are dentcal and all dentcal paths n N have the same weght by constructon snce B s unambguous and the path weghts n N are determned solely by the output labels. We merge the dentcal paths n N by vewng t as an acceptor,.e. encodng nput-output labels as a sngle label, and applyng weghted automata determnzaton over the tropcal (max-tmes) semrng. Snce tropcal semrng s dempotent, determnzaton smply elmnates the repeatng paths whch are generated when a factor occurs multple tmes on an acceptng path of B. Note that t s crucal to use an dempotent semrng at ths pont snce otherwse repeatng paths are merged by -summng dentcal path weghts whch n general produces a sum dfferent from the nput path weghts. Fgure e llustrates the normalzed transducer (wth the dashed arc) and the fnal result after encoded determnzaton (wthout the dashed arc) Optmzaton At ths step, we project N to ts nput labels (representng factors) and apply weghted ε-removal and mnmzaton [6] over 8

4 the log semrng to obtan the TP factor automaton S. Note that we can drectly apply mnmzaton snce the machne s already determnstc. Fgure c llustrates the result of optmzaton. 3.. Document Frequency Factor Automaton Usng TP factor automata, we can effcently compute and store expected document frequences for each factor of a collecton of probablstc automata, e.g. ASR lattces. For each nput automaton A of the collecton {A =,..., n}, we construct a TP factor automaton S over the log semrng. The document frequency factor automaton S DF of the entre collecton s constructed smply by takng the unon U of ndvdual TP factor automata, concatenatng a smple automaton N = ({ε}, {ε}, {, }, {}, {}, {(, ε, ε, log(n), )},, ) representng the sze of the collecton and U, and fnally applyng weghted ε-removal, determnzaton, and mnmzaton over the log semrng Term Frequency and TF-IDF Factor Automaton We can use the algorthm descrbed n [] to construct a term frequency factor automaton S TF for each nput automaton A n the collecton. These automata can be combned wth the DF factor automaton S DF of the entre collecton to construct the TF-IDF factor automata S TF,TF IDF n whch path weghts represent (TF, TF-IDF) pars. The desred relaton between these automata can be expressed as: S TF,TF IDF (x) = ( S TF (x), S TF (x) log S DF (x)) Ths operaton can be carred out wth the weghted ntersecton operaton over a specal semrng structure known as the expectaton (or entropy) semrng [, 3]. Expectaton semrng s defned as follows: E = ((R {, + }) (R {, + }),,, (, ), (, )) (x, y ) (x, y ) = (x + x, y + y ) (x, y ) (x, y ) = (x x, x y + x y ) Let log A denote the weghted automaton derved from A by replacng each weght w R + by log w and let Φ (A) and Φ (A) denote the weghted automata over the expectaton semrng derved from A by replacng each weght w by the par (w, ) and (, w) respectvely. The factor automata representng TF, DF and (TF, TF-IDF) statstcs satsfy the followng dentty n the expectaton semrng: S TF,TF IDF = Φ (S TF ) Φ ( log S DF ) Here the second term on the rght can be recognzed as the factor automaton representng IDF statstcs. Hence, TF-IDF computaton reduces to weghted automata ntersecton n the expectaton semrng. Consder the vector space model defned over factors,.e. each dmenson corresponds to a factor. Inner product computaton n ths vector space between a query factor automaton Φ (SQ TF ) representng term frequences over the expectaton semrng and each factor automaton {S TF,TF IDF =,..., n} can be carred out by ntersectng the two automata and then performng a sngle-source shortestdstance computaton [] over the entropy semrng: < Φ (S TF Q ), S TF,TF IDF >= s[φ (S TF Q ) S TF,TF IDF ] One applcaton for ths nner product s the computaton of cosne smlarty between two spoken documents, e.g. ASR lattces correspondng to a voced query and a spoken document. Table : Runtme Results: TP Factor Automata vs. Baselne Max length 3 6 all log (# factors) Baselne tme (s) TP FA tme (s) Table 3: Factor Automata Comparson FA Type UW TF TP DF TF-IDF Σ S (M) 6 7 On dsk (MB) Tme (mn) Experments and Dscusson We conducted experments on the tranng subset of the Turksh language pack provded by the IARPA Babel program whch ncludes 8 hours of conversatonal telephone speech. Lattces were generated wth a speaker dependent DNN ASR system that was traned on the same data set usng IBM s Attla toolkt. All lattces were pruned to a logarthmc beam wdth of 5. Estmatng document frequences of sngle-word factors n a collecton of lattces has prevously been addressed n [3]. Ther recpe for computng probablty of occurrence conssts of composng the nput lattce wth a smple fnte-state flter that rejects the paths ncludng the target word, computng the total probablty of the remanng paths and complementng. Computaton s carred out one factor at a tme,.e. factors are enumerated and each one s processed ndependently. We mplemented a generalzed verson of ths recpe whch can be used wth mult-word factors for our baselne results. Both ths baselne algorthm and the factor automata constructon algorthms n consderaton were mplemented usng the OpenFst Lbrary [5]. Table gves a runtme comparson between the baselne and the TP factor automata constructon algorthm. We randomly selected lattces from our data set (total sze: #states + #arcs = 8K, dsk sze:.mb) and compared the total runtme whle changng the maxmum factor length wth a fntestate length restrcton flter []. Runtme complexty of the baselne method s exponental n the maxmum factor length (or lnear n the number of factors) due to the enumeraton of factors. Proposed method takes advantage of weghted transducer algorthms to do the computaton jontly for all factors. Table 3 compares the total runtme and storage requrements for varous factor automata. For these experments, we used the entre tranng set whch ncludes 88K lattces (total sze: #states + #arcs = 33M, dsk sze: 8MB) and a maxmum factor length of 3. Frst column (UW) represents the unweghted factor automata obtaned by removng all weghts from the nput lattces. Storage requrements seem to be comparable for the types of factor automata n consderaton. The runtme for the constructon of DF factor automaton excludes the tme spent for the constructon of TP factor automata. Smlarly the runtme for the constructon of TF-IDF factor automata excludes the tme spent for the constructon of TF and DF factor automata. Just lke TF factor automaton constructon [], TP factor automaton constructon algorthm of the prevous secton s worst case exponental n the nput sze and may blow up for some nput. The runtme reported n Table 3 s domnated by the tme spent on a small fracton of nput lattces for whch the algorthm took several mnutes to complete. The dfference n the average runtme characterstcs calls for further study. 9

5 5. References [] S. Robertson, Understandng nverse document frequency: On theoretcal arguments for df, Journal of Documentaton, vol. 6, p.,. [] D. Harman, The hstory of df and ts nfluences on r and other felds, n Chartng a New Course: Natural Language Processng and Informaton Retreval, ser. The Kluwer Internatonal Seres on Informaton Retreval, J. Tat, Ed. Sprnger Netherlands, 5, vol. 6, pp [3] D. Karakos, M. Dredze, K. W. Church, A. Jansen, and S. Khudanpur, Estmatng document frequences n a speech corpus. n ASRU, D. Nahamoo and M. Pcheny, Eds. IEEE,, pp. 7. [] C. Allauzen, M. Mohr, and M. Saraclar, General ndexaton of weghted automata: Applcaton to spoken utterance retreval, n Proc. HLT-NAACL Workshop on Interdscplnary Approaches to Speech Indexng and Retreval, Boston, MA, USA,, pp. 33. [5] D. Can and M. Saraclar, Lattce ndexng for spoken term detecton, Audo, Speech, and Language Processng, IEEE Transactons on, vol. 9, no. 8, pp , nov.. [6] M. Mohr, Weghted automata algorthms, n Handbook of Weghted Automata, ser. Monographs n Theoretcal Computer Scence. An EATCS Seres, M. Droste, W. Kuch, and H. Vogler, Eds. Sprnger Berln Hedelberg, 9, pp [7] A. Blumer, J. Blumer, A. Ehrenfeucht, D. Haussler, M. T. Chen, and J. Seferas, The smallest automaton recognsng the subwords of a text, Theoretcal Computer Scence, vol., pp. 3 55, 985. [8] M. Crochemore, Transducers and repettons, Theoretcal Computer Scence, vol. 5, no., pp , 986. [9] M. Mohr, P. Moreno, and E. Wensten, General suffx automaton constructon algorthm and space bounds, Theoretcal Computer Scence, vol., no. 37, pp , 9. [] M. Saraclar and R. Sproat, Lattce-based search for spoken utterance retreval, n Proc. HLT-NAACL, Boston, MA, USA,, pp [] C. Allauzen and M. Rley, A pushdown transducer extenson for the openfst lbrary, n Proceedngs of the 7th nternatonal conference on Implementaton and Applcaton of Automata, ser. CIAA. Berln, Hedelberg: Sprnger-Verlag,, pp [] J. Esner, Expectaton semrngs: Flexble EM for fnte-state transducers, n Proceedngs of the ESSLLI Workshop on Fnte- State Methods n Natural Language Processng (FSMNLP), G. van Noord, Ed., Helsnk, Aug., extended abstract (5 pages). [3] C. Cortes, M. Mohr, A. Rastog, and M. Rley, On the computaton of the relatve entropy of probablstc automata, Int. J. Found. Comput. Sc., vol. 9, no., pp. 9, 8. [] M. Mohr, Semrng frameworks and algorthms for shortestdstance problems, Journal of Automata, Languages and Combnatorcs, vol. 7, no. 3, pp. 3 35,. [5] C. Allauzen, M. Rley, J. Schalkwyk, W. Skut, and M. Mohr, OpenFst: A general and effcent weghted fnte-state transducer lbrary, n Proceedngs of the Nnth Internatonal Conference on Implementaton and Applcaton of Automata, (CIAA 7), ser. Lecture Notes n Computer Scence, vol Sprnger, 7, pp. 3,