Deep Neural Network Bottleneck Features For Generalized Variable Parameter HMMs

Transcription

1 Deep Neural Network Bottleneck Features For Generalzed Varable Parameter HMMs Xurong Xe 1,3, Rongfeng Su 1,3, Xunyng Lu 1, & Lan Wang 1,3 1 Shenzhen Insttutes of Advanced Technology, Chnese Academy of Scences Cambrdge Unversty Engneerng Dept, Trumpngton St., Cambrdge, CB 1PZ U.K. 3 The Chnese Unversty of Hong Kong, Hong Kong, Chna xr.xe@sat.ac.cn, rf.su@sat.ac.cn, xl07@cam.ac.uk, lan.wang@sat.ac.cn Abstract Recently deep neural networks (DNNs have become ncreasngly popular for acoustc modellng n automatc speech recognton (ASR systems. As the bottleneck features they produce are nherently dscrmnatve and contan rch hdden factors that nfluence the surface acoustc realzaton, the standard approach s to augment the conventonal acoustc features wth the bottleneck features n a tandem framework. In ths paper, an alternatve approach to ncorporate bottleneck features s nvestgated. The complex relatonshp between acoustc features and DNN bottleneck features s modelled usng generalzed varable parameter HMMs (GVP-HMMs. The optmal GVP-HMM structural confguraton and model parameters are automatcally learnt. Sgnfcant error rate reductons of 48% and 8% relatve were obtaned over the baselne mult-style H- MM and tandem HMM systems respectvely on Aurora. Index Terms: generalzed varable parameter HMM, deep neural network, bottleneck features, robust speech recognton 1. Introducton Recently deep neural networks (DNNs have become ncreasngly popular for acoustc modellng n automatc speech recognton (ASR systems [1,, 3, 4, 5, 6, 7, 8]. In order to ncorporate DNNs, or mult-layer perceptrons (MLPs n general, nto HMM based acoustc models, two approaches can be used. The frst uses a hybrd archtecture that estmates the H- MM state emsson probabltes usng DNNs [9]. The second approach uses an MLP or DNN as a feature extractor, traned to produce phoneme posteror probabltes. The resultng probablstc features [10], or bottleneck features [11] are used to tran standard GMM-HMMs n a tandem fashon. As these features capture addtonal nformaton complementary to standard front-ends, they are often combned n tandem systems. One mportant ssue assocated wth the tandem HMM approach s the approprate method used to combne the conventonal and bottleneck features. The precse nature of the relatonshp between the two s hghly complex. Compared wth the standard front-ends, bottleneck features provde a dfferent vew of the same speech sgnals. Certan correlaton can therefore exst between the two. At the same tme, complementary nformaton characterzng the underlyng hdden factors nfluencng Ths work s supported by Natonal Natural Scence Foundaton of Chna (NSFC , Natonal Fundamental Research Grant of Scence and Technology (973 Project: 013CB39305 Shenzhen Fundamental Research Program JC A, J- CYJ the surface acoustc realzaton are also mplctly learnt by bottleneck features. They are propagated nto HMMs as addtonal cues and constrants to mprove dscrmnaton. The standard approach augments the conventonal front-ends wth bottleneck features n a concatenated form. More advanced approaches that explctly approxmate the correlaton between them usng lnear, affne transformatons have also been proposed [1, 13]. In order to better capture the complex relatonshp between standard acoustc and bottleneck features, technques motvated by speech producton that can fully explot the hdden varablty n the bottleneck features may be used. Along ths lne, an alternatve method to ncorporate bottleneck features nto a tandem system s proposed n ths paper. DNN bottleneck features are used as nfluence factors to drectly ntroduce controllablty to the underlyng generatve acoustc models that are based on generalzed varable parameter HMMs (GVP- HMMs [14, 15, 16, 17, 18]. The contnuous trajectores of optmal HMM parameters aganst the tme-varyng hdden factors n the bottleneck features are modelled usng polynomal functons. Ther effects on the acoustc parameters are automatcally learnt by locally optmzed polynomal parameters and degrees. Usng the proposed GVP-HMM tandem approach, sgnfcant error rate reductons of 48% and 8% relatve were obtaned over the mult-style baselne HMM and tandem HMM systems respectvely on Aurora. The rest of ths paper s organzed as follows. Generalzed varable parameter HMMs and an assocated effcent complexty control technque are ntroduced n secton. Deep neural networks and bottleneck features are revewed n secton 3. A range of GVP-HMM systems usng varous modellng confguratons are descrbed n secton 4. In secton 5 varous GVP- HMM systems usng DNN bottleneck features are evaluated on Aurora. Secton 6 s the concluson and future research.. Generalzed Varable Parameter HMMs Generalzed varable parameter HMMs (GVP-HMMs [14, 15, 16, 17] explctly model the parameter trajectores of optmal Gaussan components, or more compact ted lnear transformatons, that vary wth respect to some nfluence factors. In ths paper, trajectores of Gaussan means and varances are used..1. Model Defnton For a D dmensonal observaton o t emtted from Gaussan mxture component m, assumng P th order polynomals modellng a total of N regresson varables are used, the form of

2 GVP-HMMs consdered n ths paper s gven by ( o (t p o (t ; µ (m (v t, Σ (m (v t. (1 vt s a (P N + 1 dmensonal Vandermonde vector [19], [ vt = 1, f t,1,..., f t,p,..., f ] t,p. ( and ts N dmensonal pth order subvector s defned as f t,p = [v p t,1,..., vp t,j,..., vp t,n ], where v t,j s the jth element of an N dmensonal factor vector Gaussan parameters are condtoned on at frame t, for example, the DNN bottleneck features, ft = [v t,1,..., v t,j,..., v t,n ]. (3 µ (m ( and Σ (m ( are the P th order mean and covarance trajectory polynomals of component m respectvely. When dagonal covarances are used, the trajectores of the th dmenson of the mean and varance parameters are computed as µ (m (v t = v t c (µ(m σ (m, (v t = ˇσ (m v t c (σ(m, (4 where c ( s a (P N +1 dmensonal polynomal coeffcent vector and ˇσ (m, s the conventonal HMM varance estmate. As a natural form of generatve model nspred by speech producton, a range of factors nfluencng the acoustc realzaton of speech have been nvestgated n prevous research usng GVP-HMMs, or ther precursors based on more restrcted forms of parameter trajectores, such as multple regresson HMMs (MR-HMM [0] and varable parameter HMMs (VP-HMM [1, ]. These acoustc factors nclude prosodc features [0], envronment nose condton represented by the sgnal-to-nose rato (SNR [14, 15, 16, 17, 18, 1, ], and more recently artculatory features for speech synthess [3]. GVP-HMMs share the same nstantaneous adaptaton power and good controllablty as MR-HMMs and VP-HMMs. For any varablty ndcated by the factor vector, e.g. the bottleneck features, or SNR level, present or unseen n the tranng data, GVP-HMMs can nstantly produce the matchng HMM model parameters by-desgn wthout requrng any mult-pass decodng and adaptaton process... Parameter Estmaton for GVP-HMMs For the form of GVP-HMMs of equaton (1 the assocated ML auxlary functon s gven by [14, 15, 4], Q GVP (θ, θ = ( γ m(t log p o (t ; µ (m (v t, Σ (m (v t (5 m,t where γ m(t s the posteror probablty of frame o t beng emtted from component m at a tme nstance t. Combnng the above wth equatons (1 and (4, the correspondng parts of the above auxlary functon assocated wth the polynomal coeffcent vectors of the Gaussan mean and varance trajectores respectvely can be re-arranged nto convex quadratc forms, Q (µ(m, (θ, θ = 1 c(µ(m Q (σ(m, +k (µ(m (θ, θ = 1 c(σ(m +k (σ(m U (µ(m c (µ(m c (µ(m + const U (σ(m c (σ(m c (σ(m + const (6 where the constant terms ndependent of the coeffcent vectors c ( can be gnored. Settng the above gradents aganst the respectve polynomal coeffcent vectors to zero, the followng ML solutons of the coeffcent vectors can then be derved ĉ (µ(m ĉ (σ(m = U (µ(m, = U (σ(m, and the suffcent statstcs are U (µ(m k (µ(m = t = t U (σ(m, = t k (σ(m, = t 1 k (µ(m 1 k (σ(m γ m(tσ (m 1, (v tvt v t γ m(tσ (m 1, γ m(tˇσ (m, vt v t (v to (t, (7 v t ( γ m(t o (t µ (m (v t v t (8.3. Model Complexty Control for GVP-HMMs An mportant ssue assocated wth GVP-HMMs s the approprate polynomal degree to use. The use of hgher degree polynomals can result n severe over-fttng and oscllaton [5]. In addton, the precse form of ndvdual parameter trajectores should be n lne wth the nature of the dstnct effects mposed on them by the nfluencng factors. In order to more flexbly capture these complex, potentally locally varyng effects and mprove robustness, the optmal polynomal degrees of Gaussan mean and varance trajectores can be automatcally determned at local level usng complexty control technques [18]. In Bayesan learnng, when no pror knowledge over model structures {M} s avalable, the optmal model structure or complexty, s determned by maxmzng the evdence, p(o W, M = p(o θ, W, Mp(θ Mdθ (9 where θ s a parameterzaton of M, O = {o 1,..., o T } s a tranng data set of T frames and W the reference transcrpton. For standard HMMs and GVP-HMMs, t s computatonally ntractable to drectly compute the evdence n equaton (9. To handle ths problem, an effcent approxmaton usng the BIC style frst order asymptotc expanson [6] of a lower lower bound [18, 7, 8, 9] of the evdence ntegral can be used. The optmal model complexty s determned by ˆM = arg max M { Q (M (ˆθ, θ ρ k log T }. (10 where the ML auxlary functons assocated wth Gaussan mean and varance trajectory parameters gven n equaton (6 evaluated at the optmal model parameters ˆθ usng the statstcs gven n equatons (7 and (8. k denotes the number of free parameters n M and ρ s a tunable penalty term [30]. When determnng the optmal order for a partcular polynomal assocated wth the th dmenson of the mth Gaussan component n the system, µ (m (, for example, the above s- tatstcs n equaton (8 are accumulated for the hghest order P max beng consdered. The correspondng statstcs for any other order 0 P (µ(m < P max can be derved by takng the assocated submatrces or subvectors from the full matrx statstcs accumulated for P max. Usng these statstcs and the ML

3 solutons n equaton (7, the ML auxlary functon assocated wth µ (m ( n equaton (6, can be effcently evaluated at the optmum for each canddate polynomal degree. The number of free parameters (polynomal coeffcents n the BIC metrc of equaton (10 s k = P (µ(m +1. The number of frame samples for the current Gaussan s computed as the component level occupancy counts T (m = t,m γm(t. The same approach can also be used to determne the optmal degree of Gaussan varance polynomals by evaluatng the respectve auxlary functons wth ther respectve suffcent statstcs to compute the metrc n equaton ( DNN Bottleneck Features Bottleneck features are normally generated from a narrow hdden layer of an MLP that s traned to predct phonemes or phoneme states. Compared wth the sze of other layers, ths hdden layer has a sgnfcantly smaller number of hdden unts [11]. Ths narrow layer ntroduces a constrcton n the network whle retanng the nformaton useful to classfcaton n the resultng low dmensonal features extracted va a non-lnear and dscrmnatve transformaton. In ths paper the bottleneck features used for tandem HMM systems are extracted from deep neural network (DNN multlayer perceptrons (MLP [1,, 3, 4]. DNNs are MLPs wth many hdden layers. The nputs are formed from a stacked set of adjacent frames of the acoustc feature for each tme nstance. Wthn each hdden layer, the nput to each unt s computed as a lnearly weghted sum of the outputs from the prevous layer. Each hdden node transforms ts nput wth a sgmod actvaton to acheve non-lnearty. An softmax output actvaton functon s used at the output layer to compute the posteror probablty of phonemes or phoneme state targets. In all the experments of ths paper, a pretraned DNN consstng of sx hdden layers s used. The frst fve layers have a total of 51 hdden nodes whle the last bottleneck layer has 6 unts. The network s traned on nputs formed by splcng 11 frames of 39 dmensonal MFCC features together. The layer-by-layer RBM based pre-tranng mplemented n the Kald toolkt [31] was used. Followng DNN tranng 6 dmensonal bottleneck features are extracted and decorrelated usng PCA. For the baselne tandem HMM systems, they are appended to standard MFCC features to form the tandem feature vector. Pror to recognton, tandem GMM-HMMs are then traned based on the new concatenated tandem features. For GVP-HMM systems, these are used as the nput factor vectors at each frame to estmate contnuous trajectores of Gaussan mean and varance parameters. An extended verson of the HTK toolkt [3] was used to tran varous GVP-HMM systems. 4. Usng DNN Bottleneck Features In GVP-HMMs and Tandem GVP-HMMs In order to adjust the trade-off between modellng resoluton, robustness and computatonal effcency, a range of GVP-HMM confguratons may be consdered to ncorporate DNN bottleneck features. Descrpton of these GVP-HMM varant systems confguratons and the number of parameters used for the s- tandard Aurora task are shown n table dmensonal standard MFCC features ncludng the frst and second order dfferentals were used. All the baselne GVP-HMMs wth no complexty control used nd degree polynomals for all parameter trajectores, as suggested n [1, ]. The penalty term n the complexty control metrc of equaton (10 was fxed as ρ = 1 n all experments. For all parameter polynomals the range of canddate degree to consder s [0, 5]. Baselne HMM and tandem HMM systems: In the frst 3 lnes of table 1, the number of parameters for the multstyle [33] traned baselne HMM system and two tandem H- MM systems are shown. The second tandem HMM system, tandem, used 18 Gaussans per state thus has a model complexty comparable to the other complexty controlled GVP- HMM systems n the table. The Gaussan parameters of these baselne HMM or tandem HMM systems were traned on standard MFCC or tandem features whle no parameter trajectory modellng was used. Parm Poly Com Model Type System mean var Ctrl #Parm HMM GVP-HMM mcond tandem tandem mean mv mean 79K K 396k.15M 7K.7M 36K.M tandem 98K GVP-HMM.3M 406K Table 1: Descrpton of the baselne mult-style HMM, tandem HMM systems, GVP-HMM and tandem GVP-HMM systems on Aurora n terms of model confguratons and the number of parameters. Followng the settng of prevous works [1, 17, 18], all systems used 6 Gaussans per state except the tandem baselne system used 18 Gaussans per state. GVP-HMM systems: In the second secton of table 1, a total of four GVP-HMM modellng confguratons, denoted as mean and mv respectvely, whch use trajectory modellng for Gaussan component means usng the DNN bottleneck features as the factor nput n equatons from (1 to (3, wth the further optons of usng varance trajectores condtoned on the SNR varable, and wth or wthout applyng the model selecton technque presented n secton.3, are shown from the 4th to 7th lne n table 1. As expected, usng the standard GVP- HMMs wth no complexty control on the 6 dmensonal bottleneck features results n a massve ncrease n model parameters. Determnng the optmal degrees for parameter trajectory polynomals usng the model selecton method of secton.3 sgnfcantly reduced the model complexty by over to 80%. Tandem GVP-HMM systems: In the last secton of table 1, four comparable tandem varants of the above four GVP-HMM systems are shown. In these tandem GVP-HMM systems, the DNN bottleneck features are not only used as the nput factor vectors to estmate the contnuous trajectores of Gaussan parameters n the acoustc feature subspace, but also used as normal features to tran the standard mean and varance parameters n the bottleneck feature subspace. For example, the fnal mean vector of component m at tme nstance t s thus computed as µ (m t = [µ (m GVP (vbn t, µ (m BN ]. (11 where the µ (m GVP (vbn t s the mean subvector trajectory takng a Vandermonde vector nput vt BN constructed usng the 6 dmensonal DNN bottleneck features, as descrbed n secton.1.

4 µ (m BN s the remanng statc mean subvector estmated usng the bottleneck features. These tandem GVP-HMMs are expected to draw strength from both the conventonal tandem and GVP- HMM based approaches to fully explot the complementary nformaton n the DNN bottleneck features. 5. Experments and Results In ths secton, the performance of varous GVP-HMM systems usng DNN bottleneck features are evaluated the Aurora task. The Aurora database contans dfferent nosy condtons. Durng the experments, 40 utterances from each of four dfferent SNR condtons (-5dB, 5dB, 15dB, 5dB of nose envronments of subway, babble, car and exhbton were used to tran all the systems, whle 1000 utterances selected from each nose envronment at 0dB, 5dB, 10dB, 15dB and 0dB SNR respectvely were used for word error rate (WER evaluaton. Nose Com Type System Ctrl 0dB 5dB 10dB 15dB 0dB Ave subway babble car exhbton mcond mcond mcond mcond Table : WER performance of GVP-HMM systems usng DNN bottleneck features on Aurora test set A of four nose types. All systems used the same namng conventons as n table 1. The WER performance of the mult-style HMM baselne, mcond, and varous GVP-HMM systems shown from the 4th to 7th lne of table 1 are gven n table. The followng trends can be found n the table. Frst, the use of DNN bottleneck features gave sgnfcant WER reductons for all GVP-HMM modellng confguratons across varous nose types over the moncon HMM baselne. Second, as expected, usng the model selecton technque of secton.3, n addton to the model sze compresson shown prevously n table 1, an average WER reducton of.41% absolute (9% relatve was obtaned over varous standard GVP-HMM systems wth no complexty control. Thrd, combned wth model complexty control, the use of varance trajectory polynomals gave further mprovements over usng mean trajectory modellng only. Usng the best GVP- HMM systems hghlghted n bold n table, an average WER reducton of 3.64% absolute (40% relatve over the mult-style MFCC feature traned baselne mcond HMM system was obtaned. However, all of these four GVP-HMM systems were outperformed by the baselne tandem HMM system shown n the 1st lne of each nose specfc secton n table 3. The WER performance of the two baselne mult-style Nose Com Type System Ctrl 0dB 5dB 10dB 15dB 0dB Ave subway babble car exhbton tandem tandem tandem tandem tandem tandem tandem tandem Table 3: WER performance of tandem GVP-HMM systems usng DNN bottleneck features on Aurora test set A. All systems used the same namng conventons as n table 1. traned tandem HMM systems, tandem and tandem, and varous tandem GVP-HMM systems shown from the 8th to 11th lne n the bottom secton of table 1 are gven n table 3. Consstent wth the trends found n table, every complexty controlled tandem GVP-HMM system n table 3 outperformed ts comparable baselne usng no complexty control. The use of varance trajectory modellng also gave further small reductons n WER. Usng the best complexty controlled tandem GVP- HMM mv system hghlghted n bold n table 3, an average WER reducton of 4.38% absolute (48% relatve, and 0.4% absolute (8% relatve over the mult-style baselne mcond system of table, and the baselne tandem HMM system of table 3 respectvely were obtaned. Smlar consstent mprovements were also obtaned over the more complex baselne tandem system wth a comparable number of parameters as shown n table 1, and a thrd baselne tandem HMM system usng the bottleneck features extracted from a DNN traned on concatenated MFCC and SNR features. 6. Concluson An alternatve approach to ncorporate bottleneck features nto a tandem system usng generalzed varable parameter HMMs s nvestgated n ths paper. The complementary nformaton characterzng the hdden factors nfluencng the surface acoustc realzaton mplctly learnt by bottleneck features are exploted to mprove controllablty and robustness. The proposed technque sgnfcantly reduced the error rate by 48% and 8% relatve over the baselne mult-style HMM and tandem HMM systems respectvely on Aurora. Future research wll focus on usng bottleneck features to model the trajectores of more effcent feature space transforms [17].

5 7. References [1] F. Sde, G. L, and D. Yu (011. Conversatonal speech transcrpton usng context-dependent deep neural networks, n Proc. ISCA INTERSPEECH011, pp , Florence, Italy. [] D. Yu and M. L. Seltzer (011. Improved Bottleneck Features Usng Pretraned Deep Neural Networks, n Proc. ISCA INTER- SPEECH011, pp , Florence, Italy. [3] G. Dahl, D. Yu, L. Deng, and A. Acero (01. Context- Dependent Pre-Traned Deep Neural Networks for Large- Vocabulary Speech Recognton, n Proc. IEEE Transactons on Audo, Speech, and Language Processng, vol. 0, no. 1, pp. 30C4, jan 01. [4] G. E. Hnton, L. Deng, D. Yu, G. E. Dahl, A.-R. Mohamed, N. Jatly, A. Senor, V. Vanhoucke, P. Nguyen, T. N. Sanath, and B. Knsbury (01. Deep neural networks for acoustc modelng n speech recognton, IEEE Sgnal Processng Magazne, pp. -17, nov 01. [5] M. Seltzer, D. Yu and Y. Wang (013. An Investgaton Of Deep Neural Networks For Nose Robust Speech Recognton, n Proc. IEEE ICASSP013, pp , Vancouver, BC, Canada. [6] S. Thomas, M. L. Seltzer, K. Church, and H. Hermansky (013. Deep neural network features and sem-supervsed tranng for low resource speech recognton, n Proc. IEEE ICASSP013, pp , Vancouver, BC, Canada. [7] P. Bell, P. Swetojansk and S. Renals (013. Mult-level adaptve networks n tandem and hybrd ASR systems, n Proc. IEEE ICASSP013, pp , Vancouver, BC, Canada. [8] K. M. Knll, M. J. F. Gales, S. P. Rath, P. C. Woodland, C. Zhang and S.-X. Zhang (013. Investgaton of multlngual deep neural networks for spoken term detecton, n Proc. IEEE ASRU013, pp , Olomouc, Czech Republc. [9] H. A. Bourlard and N. Morgan (1993. Connectonst Speech Recognton: A Hybrd Approach, Kluwer Academc Publshers, Norwell, MA, USA, [10] H. Hermansky, D. Ells and S. Sharma (000. Tandem connectonst feature extracton for conventonal HMM systems, n Proc. IEEE ICASSP000, vol. 3, pp , Istanbul, Turkey. [11] F. Grezl, M. Karafat, S. Kontar and J. Cernocky (007. Probablstc and bottle-neck features for LVCSR of meetngs, n Proc. IEEE ICASSP007, Vol. 4, pp , Honolulu, Hawa, USA. [1] J. Zheng, O. Cetn, M. Y. Hwang, X. Le, A. Stolcke and N. Morgan (007. Combnng dscrmnatve feature, transform, and model tranng for large vocabulary speech recognton, n Proc. IEEE ICASSP007, Vol. 4, pp , Honolulu, Hawa, USA. [13] T. Ng, B. Zhang, S. Matsoukas and L. Nguyen (011. Regon dependent transform on MLP features for speech recognton, n Proc. ISCA INTERSPEECH011, pp. 1-4, Florence, Italy. [14] N. Cheng, X. Lu and L. Wang (011. Generalzed varable parameter HMMs for nose robust speech recognton, n Proc. IS- CA INTERSPEECH011, pp , Florence, Italy. [15] N. Cheng, X. Lu and L. Wang (011. A flexble framework for HMM based nose robust speech recognton usng generalzed parametrc space polynomal regresson, Scence Chna, Informaton Scences, 54(, pp , 011. [16] Y. L, X. Lu and L. Wang (01. Structured modelng based on generalzed varable parameter HMMs and speaker adaptaton, n Proc. IEEE ISCSLP01, pp , Hong Kong, Chna. [17] Y. L, X. Lu and L. Wang (013. Feature space generalzed varable parameter HMMs for nose robust recognton, n Proc. ISCA INTERSPEECH013, pp , Lyon, France. [18] R. Su, X. Lu, L. Wang (013. Automatc model complexty control for generalzed varable parameter HMMs, n Proc. IEEE ASRU013, pp , Olomouc, Czech Republc. [19] A. Bjorck and V. Pereyra (1970. Soluton of Vandermonde Systems of Equatons, Mathematcs of Computaton (Amercan Mathematcal Socety 4(11: pp [0] K. Fujnaga, M. Naka, H. Shmodara and S. Sagayama (001. Multple-Regresson Hdden Markov Model, n Proc. IEEE I- CASSP001, Vol. 1, pp , Salt Lake Cty, Utah, USA. [1] X. Cu and Y. Gong (007. A study of varable-parameter Gaussan mxture hdden Markov modelng for nosy speech recognton, IEEE Transactons on Audo, Speech and Language Processng, 15(4: , 007. [] D. Yu, L. Deng, Y. Gong and A. Acero (009. A novel framework and tranng algorthm for varable-parameter hdden Markov models, IEEE Transactons on Audo, Speech and Language Processng, Vol 17(7, pp , 009. [3] Z. Lng, K. Rchmond and J. Yamagsh (013. Artculatory control of HMM-based parametrc speech synthess usng feature s- pace swtched multple regresson, IEEE Transactons on Audo Speech and Language Processng, vol.1, no.1, pp , Jan do: /TASL [4] A. P. Dempster, N. M. Lard and D. B. Rubn (1977. Maxmum lkelhood from ncomplete data va the EM algorthm, Journal of the Royal Statstcal Socety, 39(1:1-39,1977. [5] C. Runge (1901. Über emprsche Funktonen und de Interpolaton zwschen äqudstanten Ordnaten, Zetschrft für Mathematk und Physk, 46:4-43. [6] G. Schwartz (1978. Estmatng the Dmenson of a Model, The Annals of Statstcs, pp , Vol. 6, No., February [7] X. Lu and M. J. F. Gales (003. Automatc model complexty control usng margnalzed dscrmnatve growth functons, n Proc. IEEE ASRU003, pp. 37-4, St. Thomas, U.S. Vrgn Islands. [8] X. Lu and M. J. F. Gales (004. Model complexty control and compresson usng dscrmnatve growth functons, n Proc. IEEE ICASSP004, Vol. 1, pp , Montreal, Quebec, Canada. [9] X. Lu and M. J. F. Gales (007. Automatc model complexty control usng margnalzed dscrmnatve growth functons, n Proc. IEEE Transactons on Audo, Speech and Language Processng, vol.15, no.4, pp , May 007. [30] W. Chou and W. Rechl (1999. Decson tree state tyng based on penalzed Bayesan nformaton crteron, n Proc. IEEE I- CASSP1999, Vol. 1, , Phoenx, Arzona, USA. [31] The Kald speech recognton toolkt. [3] S. Young et al., The HTK Book Verson 3.4.1, 009. [33] R. Lppmann, E. Martn and D. Paul (1987. Mult-style tranng for robust solated-word speech recognton, n Proc. IEEE ICASSP1987, pp , Dallas, Texas, USA.