Forward-Decoding Kernel-Based Phone Sequence Recognition

Transcription

1 Fowad-Decoding Kenel-Based Phone Sequence Recognition Shantanu Chakabatty and Get Cauwenbeghs Cente fo Language and Speech Pocessing Depatment of Electical and Compute Engineeing Johns Hopkins Univesity, Baltimoe MD Abstact Fowad decoding kenel machines (FDKM) combine lage-magin classifies with hidden Makov models (HMM) fo maximum a posteioi (MAP) adaptive sequence estimation. State tansitions in the sequence ae conditioned on obseved data using a kenel-based pobability model tained with a ecusive scheme that deals effectively with noisy and patially labeled data. Taining ove vey lage datasets is accomplished using a spase pobabilistic suppot vecto machine (SVM) model based on quadatic entopy, and an on-line stochastic steepest descent algoithm. Fo speake-independent continuous phone ecognition, FDKM tained ove 177,080 samples of the TIMIT database achieves 80.% ecognition accuacy ove the full test set, without use of a pio phonetic language model. 1 Intoduction Sequence estimation is at the coe of many poblems in patten ecognition, most notably speech and language pocessing. Recognizing dynamic pattens in sequential data equies a set of tools vey diffeent fom classifies tained to ecognize static pattens in data assumed i.i.d. distibuted ove time. The speech ecognition community has pedominantly elied on hidden Makov models (HMMs) [1] to poduce state-of-the-at esults. HMMs ae geneative models that function by estimating pobability densities and theefoe equie a lage amount of data to estimate paametes eliably. If the aim is discimination between classes, then it might be sufficient to model discimination boundaies between classes which (in most affine cases) affod fewe paametes. Recuent neual netwoks have been used to extend the dynamic modeling powe of HMMs with the disciminant natue of neual netwoks [2], but leaning long tem dependencies emains a challenging poblem [3]. Typically, neual netwok taining algoithms ae pone to local optima, and while they wok well in many situations, the quality and consistency of the conveged solution cannot be waanted. Lage magin classifies, like suppot vecto machines, have been the subject of intensive eseach in the neual netwok and atificial intelligence communities [4]. They ae attactive because they genealize well even with elatively few data points in the taining set, and bounds on the genealization eo can be diectly obtained fom the taining data. Unde geneal conditions, the taining pocedue finds a unique solution (decision o egession suface) that povides an out-of-sample pefomance supeio to many techniques. Recently, suppot vecto machines (SVMs) [4] have been used fo phoneme (o phone) ecognition [5] and have shown encouaging esults. Howeve, use of a standad SVM

2 P(x 1) P(x 0) P(1 1) P(0 1) 1 0 P(1 0) (a) P(0 0) P(1 1,x) P(0 1,x) 1 0 P(1 0,x) (b) P(0 0,x) Figue 1: (a) Two state Makovian maximum-likehood (ML) model with static state tansition pobabilities and obsevation vectos xemitted fom the states. (b) Two state Makovian MAP model, whee tansition pobabilities between states ae modulated by the obsevation vecto x. classifie by itself implicitly assumes i.i.d. data, unlike the sequential natue of phones. To model inte-phonetic dependencies, maximum likelihood (ML) appoaches assume a phonetic language model that is independent of the utteance data [], as illustated in Figue 1 (a). In contast, the maximum a posteioi (MAP) appoach assumes tansitions between states that ae diectly modulated by the obseved data, as illustated in Figue 1 (b). The MAP appoach lends itself natually to hybid HMM/connectionist appoaches with pefomance compaable to state-of-the-at HMM systems [7]. FDKM [8] can be seen a hybid HMM/SVM MAP appoach to sequence estimation. It theeby augments the ability of lage magin classifies to infe sequential popeties of the data. FDKMs have shown supeio pefomance fo channel equalization in digital communication whee the eceived symbol sequence is contaminated by inte symbol intefeence [8]. In the pesent pape, FDKM is applied to speake-independent continuous phone ecognition. To handle the vast amount of data in the TIMIT copus, we pesent a spase pobabilistic model and efficient implementation of the associated FDKM taining pocedue. 2 FDKM fomulation The poblem of FDKM ecognition is fomulated in the famewok of MAP (maximum a posteioi) estimation, combining Makovian dynamics with kenel machines. A Makovian model is assumed with symbols belonging to classes, as illustated in Figue 1(a) fo. Tansitions between the classes ae modulated in pobability by obsevation (data) vectos ove time. 2.1 Decoding Fomulation The MAP fowad decode eceives the sequence and poduces an estimate of the pobability of the state vaiable ove all classes!, "$ % '&)(* % '!,+ -.0/, whee. denotes the set of paametes fo the leaning machine. Unlike hidden Makov models, the states diectly encode the symbols, and the obsevations modulate tansition pobabilities between states [7]. Estimates of the posteio pobability " ae obtained fom estimates of local tansition pobabilities using the fowad-decoding pocedue [7] " 798;: & 7 " 7 <= (1) whee & 7, '&)(>,?!@+A BC, EDF 9.0/ denotes the pobability of making a tansition fom class D at time BG to class! at time, given the cuent obsevation vecto. The fowad decoding (1) embeds sequential dependence of the data wheein the pobability estimate at time instant depends on all the pevious data. An on-line

3 ^ ^ ^ [ " estimate of the symbol is thus obtained: IHKJML GNPORQTS)NVU (2) The BCJR fowad-backwad algoithm [9] poduces in pinciple a bette estimate that accounts fo futue context, but equies a backwad pass though the data, which is impactical in many applications equiing eal time decoding. Accuate estimation of tansition pobabilities & 7 in (1) is cucial in decoding (2) to povide good pefomance. In [8] we used kenel logistic egession [10], with egulaized maximum coss-entopy, to model conditional pobabilities. A diffeent pobabilistic model that offes a spase epesentation is intoduced below. 2.2 Taining Fomulation Fo taining the MAP fowad decode, we assume access to a taining sequence with labels (class membeships). Fo instance, the TIMIT speech database comes labeled with phonemes. Continuous (soft) labels could be assigned athe than binay indicato labels, to signify uncetainty in the taining data ove the classes. Like pobabilities, label assignments ae nomalized: W 85:<X YZ X []\3 The objective of taining is to maximize the coss-entopy of the estimated pobabilities "$ given by (1) with espect to the labels X ove all classes! and taining data 254 1a2;4 G X ` 85: cbdfq "$ (3) 85: To povide capacity contol we intoduce a egulaize ef(*.0/ in the objective function [11]. The paamete space. can be patitioned into disjoint paamete vectos. 7 and g 7 fo each pai of classes!hd i\3h j such that & 7 depends only on. 7 and g 7. (The paamete g 7 coesponds to the bias tem in the standad SVM fomulation). The egulaize can then be chosen as the kml nom of each disjoint paamete vecto, and the objective function becomes 2;4 on ` 85: 85: X bpdzq " q 7985: 85: l whee the egulaization paamete n contols complexity vesus genealization as a biasvaiance tade-off [11]. The objective function (4) is simila to the pimal fomulation of a lage magin classifie [4]. Unlike the convex (quadatic) cost function of SVMs, the fomulation (4) does not have a unique solution and diect optimization could lead to poo local optima. Howeve, a lowe bound of the objective function can be fomulated so that maximizing this lowe bound educes to a set of convex optimization sub-poblems with an elegant dual fomulation in tems of suppot vectos and kenels. Applying the convex popety of the bdfq3(st/ function to the convex sum in the fowad estimation (1), we obtain diectly whee 254 ^ 7 u n 7 ` 8;: 1a2;4 8;: 798;: X cbdfqv& 7 3 (4) ^ 7 (5) 85: l with effective egulaization sequence n 7 wn " 7 x= (7) Disegading the inticate dependence of (7) on the esults of () which we defe to the following section, the fomulation () is equivalent to egession of conditional pobabilities & 7 fom labeled data and X, fo a given outgoing state D. ()

4 ^ 2.3 Kenel Logistic Pobability Regession Estimation of conditional pobabilities yoc(z!+ / fom taining data and labels X can be obtained using a egulaized fom of kenel logistic egession [10]. Fo each outgoing state D, one such pobabilistic model can be constucted fo the incoming state! conditional on : & 7 w{u (} 7 (z M/9/9~ {U (} 7 (z z// (8) 85: As with SVMs, dot poducts in the expession fo } 7 (z / in (8) convet into kenel expansions ove the taining data x by tansfoming the data to featue space [12] } 7 (z /. 7 ]g 7 7 x x ]g 7 (9) 5 zˆ Š 7fŒ (z x9 / g 7 whee Œ (sžpžt/ denotes any symmetic positive-definite kenel 1 that satisfies the Mece condition, such as a Gaussian adial basis function o a polynomial [11]. Optimization of the lowe-bound in (5) equies solving disjoint but simila suboptimization poblems (). The subscipt D is omitted in the emainde of this section fo claity. The (pimal) objective function of kenel logistic egession expesses egulaized coss-entopy () of the logistic model (8) in the fom [13, 14] Y +. + l n A X A mš xm}p A(* xz/ bpdzq(* V R =p V œ A mš q (10) The paametes 7 in (9) ae detemined by minimizing a dual fomulation of the objective function (10) obtained though the Legende tansfomation, which fo logistic egession takes the fom of an entopy-based potential function in the paametes [10] ^) 3 subject to constaints Ÿ ]n ( X x ~Pn /AbpdZQq( X 0q ~Zn / (11) \ (12) \ (13) n X 0 (14) Thee ae two disadvantages of using the logistic egession dual diectly: 1. The solution is non-spase and all the taining points contibute to the final solution. Fo tasks involving lage data sets like phone ecognition this tuns out to be pohibitive due to memoy and un-time constaints. 2. Even though the dual optimization poblem is convex, it is not quadatic and pecludes the use of standad quadatic pogamming (QP) techniques. One has to esot to Newton-Raphson o othe nonlinea optimization techniques which complicate convegence and equie tuning of additional system paametes. 1 M ; s 5 % «ª M q 9 ª M 5. The map ª need not be computed explicitly, as it only appeas in inne-poduct fom.

5 W Á ` W ¾ ` ` l 2.4 f!5! SVM fomulation The f!5! SVM pobabilistic model [15] povides a spase altenative to logistic egession. A quadatic ( Gini [1]) index eplaces entopy in the dual fomulation of logistic egession. The Gini index povides a lowe bound of the dual logistic functional, and its quadatic fom poduces spase solutions as with suppot vecto machines. The tightness of the bound povides an elegant tade-off between appoximation and spasity. Jensen s inequality (bpdzq; B ) fomulates the lowe bound fo the entopy tem in (11) in the fom of the multivaiate f!5! impuity index [1]: % l bpdzq; (15) whee \ FK±! and W ². Both foms of entopy W bpdzq; and f l each thei maxima at the same values 0³ I~ coesponding to a unifom distibution. As in the binay case, the bound can be tightened by scaling the f!5! index with a multiplicative facto [µ, of which the paticula value depends on. 2 The f!5! ^ SVM dual cost function is then given by ^ Ÿ ;n¹(m ( X 0q ~Pn / ]I/ (1) The convex quadatic cost function (1) with constaints in (11) can now be minimized diectly using standad quadatic pogamming techniques. The pimay advantage of the technique is that it yields spase solutions and yet appoximates the logistic egession solution vey well [15]. 2.5 Online f!5! SVM Taining Fo vey lage datasets such as TIMIT, using a QP appoach to tain f!5! SVM may still be pohibitive even though spasity dastically in the tained model educes the numbe of suppot vectos. An on-line estimation pocedue is pesented, that computes each coefficient ` in tun fom single pesentation of the data X. A line seach in the paamete ` and the bias g pefoms stochastic steepest descent of the dual objective function (1) of the fom whee ¼3 equation solved in at most ` nº X n X K( Ÿ `Z` 3²» g ÄÃ g 3 ` /; ]} (z¼½ z/½ n¹( Ÿ `Z` /; ¾ `FÀÁ Â (17) (18) denotes the positive pat of ¼. The nomalization facto n X K( Ÿ `Z`» /; =} A algoithmic iteations. 3 Recusive FDKM Taining ¾ ¾ ` À Á on¹( Ÿ `Z` is detemined by /; (19) The weights (7) in () ae ecusively estimated using an iteative pocedue eminiscent of (but diffeent fom) expectation maximization. The pocedue involves computing new estimates of the sequence " 7 Åw to tain () based on estimates of & 7 using pevious values of the paametes 7. The taining poceeds in a seies of epochs, each efining the

6 È Ç Ç È taining È 0 1 È 0 n-1 n n-2 n-1 n n-k n time K 1 Ç e-estimation taining 1 1 È Figue 2: Iteations involved in taining FDKM on a tellis based on the Makov model of Figue 1. Duing the initial epoch, paametes of the pobabilistic model, conditioned on the obseved label fo the outgoing state at time %), of the state at time ae tained fom obseved labels at time. Duing subsequent epochs, pobability estimates of the outgoing state at time ove inceasing fowad decoding depth É) «F Œ detemine weights assigned to data fo taining each of the pobabilistic models conditioned on the outgoing state. estimate of the sequence " 7 Bo by inceasing the size of the time window (decoding depth, É ) ove which it is obtained by the fowad algoithm (1). The taining steps ae illustated in Figue 2 and summaized as follows: 1. To bootstap the iteation fo the fist taining epoch (É Y ), obtain initial values fo " 7 u fom the labels of the outgoing state, " 7 Ê X 7 u. This coesponds to taking the labels X u as tue state pobabilities which coesponds to the standad pocedue of using fagmented data to estimate tansition pobabilities. 2. Tain logistic kenel machines, one fo each outgoing class D, to estimate the paametes in & 7,!hDu ËFph fom the taining data and labels X, weighted by the sequence " 7 <]. 3. Re-estimate " 7 Åu using the fowad algoithm (1) ove inceasing decoding depth É, by initializing " 7 < ÉÌ to X < ÉÌ. 4. Re-tain, incement decoding depth É, and e-estimate " 7 Bo, until the final decoding depth is eached (É) Œ ). The pefomance of FDKM taining depends on the final decoding depth Œ, although obseved vaiations in genealization pefomance fo lage values of Œ ae elatively small. A suitable value can be chosen a pioi to match the extent of tempoal dependency in the data. Fo phoneme classification in speech, the decoding depth can be chosen accoding to the length of a typical syllable. An efficient pocedue to implement the above algoithm is discussed in [15]. 4 Expeiments and Results The pefomance of FDKM was evaluated on the full TIMIT dataset [17], consisting of labeled continuous spoken utteances. The 0 phone classes pesented in TIMIT wee fist collapsed onto 39 classes accoding to standad folding techniques []. The taining set consisted of,300 sentences spoken by 3 speakes, esulting in 177,080 phone instances. The test set consisted of 192 sentences spoken by 24 speakes. The speech254 signal was fist pocessed by a pe-emphasis filte with tansfe function Íw\3 ÎAÏV. Subsequently, a 25 ms Hamming window was applied ove 10 ms shifts to extact a sequence of phonetic segments. Cepstal coefficients wee extacted fom the sequence, combined with thei fist and second ode time diffeences into a 39-dimensional vecto. Cepstal mean subtaction and speake nomalization wee subsequently applied. 2 Unlike the binay case (Ð ]Ñ ), the facto Ò fo geneal Ð cannot be chosen to match the two maxima at Ó Ô =Õ Ö Ð. n-1 n

7 Table 1: Pefomance Evaluation of FDKM (Œ «\ ) on TIMIT Machine Accuacy Insetion Substitution Deletion Eos FDKM, nc G\atØ 79.9%.3% 14.7% 5.4% 2.4% FDKM, nj «80.%.3% 14.3% 5.1% 25.7% FDKM, nc tø 79.% 5.3% 14.% 5.8% 25.7% Recognition ate (%) Taining Test Decoding depth k Figue 3: Recognition ate as a function of decoding depth ɹ «F Œ. Each phone utteance wee then subdivided into thee segments with elative popotions 4:3:4 [18]. The featues in the thee segments wee individually aveaged and concatenated to obtain a 117-dimensional featue vecto. Evaluation on the test was pefomed using thesholding of state pobabilities in the MAP fowad decoding (2) [19], with theshold \a Ø. The decoded phone sequence was then compaed with the tanscibed sequence using Levenshtein s distance to evaluate diffeent souces of eos. Multiple uns of identical phones in the decoded and tanscibed sequences wee collapsed to single phone instances to eflect tue insetion eos. Table 1 summaizes the esults of the expeiments with FDKM on TIMIT fo diffeent values of the egulaization constant n. The ecognition pefomance is compaable to the state of the at using HMMs and othe appoaches, in the uppe 70% and lowe 80% ange [2, 5, 20]. Figue 3 illustates the impovement in ecognition ate with inceasing decoding depth É. The optimum value É ÙÚ\ coesponds to inte-phonetic dependencies on a time scale of 100 ms. 5 Conclusion Expeiments with FDKM on the TIMIT copus have demonstated levels of speakeindependent continuous phone ecognition accuacy compaable to o bette than othe appoaches that use HMMs and thei vaious extensions. FDKM impoves decoding and genealization pefomance fo data with embedded sequential stuctue, poviding an elegant tadeoff between leaning tempoal vesus spatial dependencies. The ecusive estimation pocedue educes o masks the effect of noisy o missing labels X 7. Futhe impovements can be expected by tuning of hype-paametes and impoved epesentation of acoustic featues. Acknowledgement This wok was suppoted by a gant fom the Catalyst Foundation.

8 Refeences [1] L. Rabine and B-H Juang, Fundamentals of Speech Recognition, Englewood Cliffs, NJ: Pentice-Hall, [2] Robinson, A.J., An application of ecuent nets to phone pobability estimation, IEEE Tansactions on Neual Netwoks, vol. 5,No.2,Mach [3] Bengio, Y., Leaning long-tem dependencies with gadient descent is difficult, IEEE T. Neual Netwoks, vol. 5, pp , [4] Vapnik, V. The Natue of Statistical Leaning Theoy, New Yok: Spinge-Velag, [5] Clak, P. and Moeno, M.J. On the use of Suppot Vecto Machines fo Phonetic Classification, IEEE Conf. Poc., [] Lee, K.F. and Hon, H.W, Speake-Independent phone ecognition using hidden makov models, IEEE Tansactions on Acoustics, Speech and Signal Pocessing, vol. 37, pp , [7] Boulad, H. and Mogan, N., Connectionist Speech Recognition: A Hybid Appoach, Kluwe Academic, [8] Chakabatty, S. and Cauwenbeghs, G. Sequence Estimation and Channel Equalization using Fowad Decoding Kenel Machines, IEEE Int. Conf. Acoustics and Signal Poc. (ICASSP 2002), Olando FL, [9] Bahl, L.R., Cocke J., Jelinek F. and Raviv J. Optimal decoding of linea codes fo minimizing symbol eo ate, IEEE Tansactions on Infom. Theoy, vol. IT-20, pp , [10] Jaakkola, T. and Haussle, D. Pobabilistic kenel egession models, Poceedings of Seventh Intenational Wokshop on Atificial Intelligence and Statistics, [11] Giosi, F., Jones, M. and Poggio, T. Regulaization Theoy and Neual Netwoks Achitectues, Neual Computation, vol. 7, pp , [12] Schölkopf, B., Buges, C. and Smola, A., Eds., Advances in Kenel Methods-Suppot Vecto Leaning, MIT Pess, Cambidge, [13] Wahba, G. Suppot Vecto Machine, Repoducing Kenel Hilbet Spaces and Randomized GACV, Technical Repot 984, Depatment of Statistics, Univesity of Wisconsin, Madison WI. [14] Zhu, J and Hastie, T., Kenel Logistic Regession and Impot Vecto Machine, Adv. IEEE Neual Infomation Pocessing Systems (NIPS 2001), Cambidge, MA: MIT Pess, [15] Chakabatty, S. and Cauwenbeghs, G. Fowad Decoding Kenel Machines: A hybid HMM/SVM Appoach to Sequence Recognition, IEEE Int. Conf. of Patten Recognition: SVM wokshop. (ICPR 2002), Olando FL, [1] Beiman, L. Fiedman, J. H. et al. Classification and Regession Tees, Wadswoth and Books, Pacific Gove, CA, [17] Fishe, W., Doddington G. et al The DARPA Speech Recognition Reseach Database: Specifications and Status. Poceedings DARPA speech ecognition wokshop, pp , 198. [18] Fosle-Lussie, E. Geenbeg, S. Mogan, N., Incopoating contextual phonetics into automatic speech ecognition, Poc. XIVth Int. Cong. Phon. Sci., [19] Wald, A. Sequential Analysis, Wiley, New Yok, [20] Chengalvaayan, R. and Deng, Li., Speech Tajectoy Discimination Using the Minimum Classification Eo Taining, IEEE Tansactions on Speech and Audio Pocessing, vol., pp , Nov