Feature Mining for GMM-Based Speech Quality Measurement

Transcription

1 Feature Mnng for GMM-Based Speech Qualty Measurement Tago H. Falk and Wa-Yp Chan Department of Electrcal and Computer Engneerng Queen s Unversty, Kngston, Ontaro, Canada Emal: {falkt, chan}@ee.queensu.ca Abstract We propose a novel approach to objectve speech qualty measurement usng feature mnng and Gaussan mxture models (GMMs). A large pool of perceptual dstorton features s extracted from the speech sgnal and data mnng technques are used to sft out the most relevant feature varables from the pool. We examne usng multvarate adaptve regresson splnes (MARS), classfcaton and regresson trees (CART), a hybrd CART-MARS scheme, and the sequental forward selecton (SFS) algorthm for data mnng. For our speech databases, the SFS algorthm provdes best performance wth a fve-feature, threecomponent GMM. A reducton of 21.7% n root-mean-squared mean opnon score estmaton error s obtaned n comparson wth ITU-T P.862 PESQ. I. INTRODUCTION The telecommuncatons ndustry s gong through a phase of rapd development. New servces and technologes emerge contnuously. Customers now have the freedom of selectng from as array of telecommuncatons servces at affordable cost. Faced wth offerng voce servces over ncreasngly heterogenous network connectons, the evaluaton of speech qualty s becomng crtcally mportant for the servce provder, servng as an nstrument for montorng and mprovement of qualty of servce and network capacty. Tradtonally, the most relable way to measure the qualty of a speech sgnal was through the use of subjectve speech qualty assessment tests such as MOS (mean opnon score) tests [1]. These tests are hghly unsutable for onlne qualty measurement and are also very expensve and tme consumng. The research descrbed below s motvated by the fact that objectve methods have replaced subjectve testng, allowng computer programs to automate speech qualty measurement n real tme, makng them sutable for feld applcatons. The Internatonal Telecommuncatons Unon ITU-T P.862 standard, also known as Perceptual Evaluaton of Speech Qualty (PESQ) [2] s the latest objectve qualty measurement standard algorthm. Nevertheless, the algorthm stll falls short of the accuracy that can be obtaned from subjectve lstenng tests wth szable lstener panels. In [3] and [4], an approach s ntroduced that uses data mnng technques to mprove the accuracy of audtory-model based qualty measurement; sgnfcant performance mprovement over PESQ was reported. Here we extend the work of [3] and [4] by proposng a novel, smple yet robust, method of speech qualty estmaton based on Gaussan mxture models (GMMs). TABLE I SUBJECTIVE RATING SCALE FOR THE MEAN OPINION SCORE - MOS Ratng Speech Qualty Level of Dstorton Excellent Imperceptble 4 Good Just perceptble but not annoyng 3 Far Perceptble and slghtly annoyng 2 Poor Annoyng but not objectonable 1 Unsatsfactory Very annoyng and objectonable A large pool of feature measurements s created from the dstorton surface between the orgnal speech sgnal and the degraded speech sgnal. Frst, we use four statstcal data mnng methods, multvarate adaptve regresson splnes (MARS) [], classfcaton and regresson trees (CART) [6], a hybrd CART-MARS technque, and the sequental forward selecton (SFS) algorthm [7] to sft out good features. We then model the jont densty of these features (x) wth the subjectve MOS (y) as a Gaussan mxture. We use ths model to derve the mnmum mean squared error (MMSE) estmate, E[y x], of the subjectve MOS value. Smulatons show that our approach outperforms PESQ by as much as 21.7% n root-mean-squared MOS estmaton error. II. OBJECTIVE SPEECH QUALITY ESTIMATION Tradtonally, the only way to measure the perceved qualty of a speech sgnal was through the use of subjectve testng,.e, a group of qualfed lsteners are asked to rate the speech they had just heard accordng to the scale gven n Table I. The average of the lstener scores s the subjectve MOS. Ths s the most relable method of speech qualty assessment but t s very expensve and tme consumng, makng t unsutable for frequent or rapd applcatons. Hence, models have been developed to dentfy audble dstortons through an objectve process based on human percepton. Ths objectve method can be mplemented by computer programs and can be used n real tme measurement of speech qualty. Several perceptual models for comparng the orgnal and the coded speech sgnal have been proposed [8] [11]. The qualty of classcal codng algorthms s estmated by waveform matchng usng sgnal-to-nose rato (SNR) and the segmental SNR. These are easy to mplement, have straghtforward nterpretatons and can estmate the qualty of speech n waveform-preservng systems. But newer generaton speech

2 coders do not preserve the waveform so these measures are of lttle relevance. Researchers have employed models of human audtory percepton n ther estmaton of perceved speech qualty. It s known that the perpheral audtory system of human preprocesses nformaton and compact feature extracton s done n hgher-level bran functons. The human decson s based on ths compacted data. An adequate model should emulate ths bologcal preprocessng and hgher-level functons, and delver ratngs that have hgh correlatons wth the subjectve results. The preprocessng part s relatvely well understood but the hgher-level bran functons are dffcult to model. A. GMMs for Speech Qualty Estmaton Gaussan mxture models have been used extensvely n speech processng, especally n speech recognton. The reasons behnd ths wdespread use are not concdental: (1) unvarate Gaussan denstes have a smple and concse representaton, dependng unquely on two parameters, mean and varance, and (2) the Gaussan mxture dstrbuton s unversally studed and ts behavors are wdely-known [12]. At a cost of extra parameters, GMMs mprove on Gaussans by allowng asymmetry and multmodalty. In prncple, GMM can approxmate any probablty densty functon to an arbtrary accuracy. Let u be an K-dmensonal vector, a Gaussan mxture densty s a weghted sum of M component denstes p(u µ, Σ, α) = α.b (u) (1) where α 0, = 1,..., M are the mxture weghts, wth M α = 1, and b (u), = 1,..., M are the K-varate Gaussan denstes wth mean vector µ and covarance matrx Σ. GMMs can assume several dfferent forms, dependng on the type of covarance matrces. The two most wdely used are full and dagonal covarance matrces. If K s the dmenson of the feature vector and M the number of Gaussan components, then the number of parameters that have to be estmated durng tranng s gven by M 2 (K2 + 3K + 2) for full matrces and M(2K + 1) for dagonal matrces. The effect of usng M full covarance matrces can be equvalently obtaned by usng a larger set of dagonal covarance Gaussans [13]. The GMM for speech qualty estmaton s bult on perceptual feature varables. The varables are obtaned from mnng a large pool of canddate feature varables. These canddate features are obtaned by classfyng perceptual dstortons nto a varety of contexts, as proposed n [3]. Frst, the clean and degraded sgnals are splt nto 7 frequency bands. The spectral power dstorton between the clean and degraded speech sgnals s then found. Tme segmentaton labels the speech frames as actve or nactve. Actve frames are further classfed nto voced or unvoced. The total dstorton of each frame s gven severty classfcatons of low, medum, or hgh by smple thresholdng. Dstorton samples n tme-frequency bns are thus labelled accordng to ts frequency band, tme-segmentaton type, and severty level. Addtonal contexts are created where each subband s further labelled wth the rank order obtaned by rankng the 7 dstortons n a frame n the order of decreasng magntude. Weghted mean and root-mean dstortons, probablty of each frame type and the lowest-frequency band and hghestfrequency band energy of the clean speech frames are also used to form a pool of 209 canddate features. Statstcal data mnng s used to sft out the most relevant varables from the pool of varables. The top- most mportant feature varables as ranked by MARS, CART, a CART-MARS hybrd confguraton, or the SFS algorthm wll be used here. We model the jont densty of the top- most mportant feature varables (x) wth the subjectve MOS (y) as a Gaussan mxture gven by (1) wth u = [y, x] T. We then predct the value of the subjectve MOS, y, gven the observed values of the -dmensonal feature vector, x. The MMSE estmate of y gven x, namely E[y x], s [14] E[y x] = h (x)[µ y + Σyx (Σ xx ) 1 (x µ x j )] (2) where h (x) denotes the probablty that the th Gaussan component of the margnal predctor densty p(x) generated the vector x and s gven by h (x) = α Σ xx k=1 1/2 e α k Σ xx k 1/2 e ) ( 2 1 (x µx )T (Σ xx ) 1 (x µ x ) ( 1 2 (x µx k )T (Σ xx k ) 1 (x µ x k ) ). (3) The covarance matrx of the th GMM component s ( Σ yy Σ = Σ yx ). Σ xy Σ xx If the covarance matrces are restrcted to be dagonal, the least squares estmate smplfes to E[y x] = h (x)µ y. (4) Ths restrcton has to be used wth care, as t can result n large estmaton errors when there exsts a sgnfcant amount of correlaton between the predctor and the response varables,.e. Σ yx are far from zero. III. EXPERIMENTAL RESULTS We compare our algorthm to PESQ usng MOS labelled speech databases. The performance of the algorthms s assessed by the correlaton between subjectve MOS w and objectve MOS y, usng Pearson s formula N R = (w w)(y ȳ) N (w w) () 2 N (y ȳ) 2 where w s the average of w, and ȳ s the average of y. MOS measurement accuracy s assessed usng the root-mean-square MOS error (RMSE) RMSE = N (w y ) 2. (6) N

3 Subjectve MOS score TABLE II PERFORMANCE COMPARISON FOR CART SELECTED VARIABLES - FULL GMM GMM TABLE III PERFORMANCE COMPARISON FOR MARS SELECTED VARIABLES - FULL Objectve MOS score GMM GMM Fg. 1. Subjectve MOS versus Objectve MOS for CART selected features usng fve dagonal Gaussan components. In [3], RMSE s shown to be the sum of unexplaned varance n the regresson model and the bas error between subjectve MOS and objectve MOS. The calculaton of R does not take nto consderaton ths bas error; therefore, unless the estmates are unbased or all suffer from the same bas, RMSE s a more realstc measure of estmator performance. The speech databases nclude seven multlngual databases n ITU-T P-seres Supplement 23, two wreless databases and a mxed wrelne-wreless database. We combne these ten databases nto a global database and then use 10-fold cross valdaton to measure performance. The global database s randomly dvded nto 10 data sets of almost equal sze. Tranng and testng s performed n 10 trals, where, n each tral, one of the data sets serves as a test set and the remanng 9 are combned to serve as a tranng set. Each data set serves as a test set only once. The ten resultng R s and RMSE s are averaged to obtan the cross-valdaton R and RMSE. The parameters of the GMM are estmated va the EM algorthm [1]. The algorthm teratons produce a sequence of models wth monotoncally nondecreasng (log-)lkelhood values. Though the EM algorthm converges to a maxmum lkelhood t has a few drawbacks: t s a greedy algorthm and may converge to a local maxmum and not the global maxmum. GMMs produced by the EM algorthm are, consequently, senstve to ntalzaton. We use the k-means algorthm [16] to ntalze the GMM parameters. Intal tests show that by usng dagonal covarance matrces only a modest mprovement over PESQ s acheved. We attrbute ths to the fact that the features selected by CART and/or MARS have sgnfcant correlaton amongst them and the use of a small number of dagonal Gaussan components does not compensate for ths. When lookng at the graph of the subjectve MOS versus objectve MOS for one of the cross-valdaton trals, we see the penalty of usng dagonal matrces (vde Fg. 1). The promnent vertcal algnment of ponts suggests poor predcton performance. The algnment dsappears and better predcton performance s obtaned when full covarance matrx s used, as we show below. Wth full covarance matrces the number of parameters that need to be estmated scales quadratcally wth the feature space dmenson. When dealng wth lmted data, as n our case, severe problems arse due to sngulartes and local maxma n the log-lkelhood functon. Many regularzaton schemes have been proposed to mprove the smoothness and generalzaton propertes of the estmated densty functon. Here we avert ll-condtonng by addng a small dagonal matrx, namely ɛi n n, to each covarance matrx n each M-step teraton of the EM algorthm. Typcally, the optmal value for ɛ s not known a pror. A smple procedure used here s to vary ɛ over a range of values and choose the value that leads to the best performance on the valdaton data set. We vared ɛ from to 1 and the value that led to best performance was ɛ = The performance results for the feature varables selected by CART and MARS are shown n Tables II and III, respectvely. GMM- stands for a Gaussan mxture model wth components and % ndcates percentage ncrease n R or percentage reducton of RMSE relatve to PESQ. The result for PESQ s based on usng a 3rd order regresson polynomal traned on the global database. On our data, ths method outperformed the PESQ-LQ mappng proposed n [17]. The most salent feature varables are lsted n Table IV for all four data mnng technques. The varables are defned n the Appendx. Wth the correlaton between features properly modelled, an average mprovement of 4.9% and 1.74% n R and RMSE, respectvely, s acheved for CART selected features. Further mprovement can be seen for MARS selected features; an average mprovement of 6.23% and 17.7% n R and RMSE s acheved. As can be seen, only voced frames are captured by the features selected usng CART. In whspered speech, all normally voced phonemes are not vocalzed,.e. they become unvoced. Such stuaton, though rare, would cause problems for these features. It can be nferred that the estmator s not senstve to degradaton n the unvoced regons and the non-

4 TABLE IV FEATURE VARIABLES TABLE VI SUBSET OF THE VALIDATION SET Rank MARS CART CART-MARS SFS 1 I P VUV V WM I P VUV V O 1 2 V B V O 2 REF 1 I P VUV 3 V B 2 V O 1 V WM 2 REF 1 4 V B 2 2 V RM V B V B 2 U P VUV V O 0 U P V O TABLE V PERFORMANCE COMPARISON FOR CART-MARS SELECTED VARIABLES - FULL Full GMM Full GMM speech regons. The CART-MARS hybrd scheme uses CART to pre-screen features from the feature canddate pool. The features selected by CART are then used as a smaller feature canddate pool for MARS to sort through. By dong ths, features can be drawn from voced and unvoced frames. An average mprovement of 6.27% and 18.6% n R and RMSE s acheved. The performance results for the feature varables selected by ths hybrd scheme are shown n Table V. Careful analyss of the features selected by the three aforementoned data mnng schemes led us to beleve that further mprovement s possble. A certan trend was noted wthn the feature varables as shown n Table VI. The table s composed of the subjectve MOS, fve MARS-selected features (x1 to x), and objectve MOS ( MˆOS) estmated usng three Gaussan components for sx dstnct test speech sgnals. Note that all sx test sgnals have the same subjectve MOS,.e., they have been rated as havng, on average, the same objectonable level of dstorton. It can be nferred that these sx speech sgnals belong to the same dstorton class and ther feature values should not vary consderably. Table VI shows that the features selected by MARS, on the contrary, vary consderably and ths varaton s reflected on the estmates. For test vector 2 we obtan an estmate of 3.98,.e. an error of 2.8%, but for test vector 6 we obtan an estmate of 2.308, an error of 34%! We conjecture that n order to obtan further mprovement better features would have to be used, preferably features that do not vary consderably wthn the same dstorton class. To ths end, we use the SFS algorthm. The algorthm starts wth the varable that s most correlated wth the target varable, and at each step adds a new varable that, together wth the prevous ones, most accurately predcts the target. A partal F-test s ncorporated n the algorthm such that the varables chosen have small varances wthn each dstorton class. An mprovement of 8% and 21.7% n R and RMSE s provded by the SFS algorthm. The performance results of ths scheme are shown n Table VII. Table IV shows that the features selected by the SFS algorthm are gleaned from the # MOS ˆ MOS x1 x2 x3 x4 x TABLE VII PERFORMANCE COMPARISON FOR SFS SELECTED VARIABLES - FULL Full GMM Full GMM top three features selected by MARS, CART, and the CART- MARS hybrd scheme. Fgure 2 depcts a scatter plot of the subjectve MOS versus objectve MOS for CART-MARS selected features, usng a GMM wth three Gaussan components and full covarance matrces. The data shown are for one of the cross-valdaton trals. We see that the ponts are no longer algned wth the vertcal axs as n the case of dagonal covarance matrces. A further method for measurng model performance s to plot the dstrbuton of absolute resdual errors between objectve and subjectve MOS [18]. Fgure 3 plots the dstrbuton of errors for SFS selected features for one of the trals. As can be seen, almost 78% of the GMM estmates are wthn 0.0 unt of the subjectve MOS. Subjectve MOS score Objectve MOS score Fg. 2. Subjectve MOS versus Objectve MOS for CART-MARS selected features usng three full Gaussan components.

5 Fg. 3. % of absolute errors n gven range Absolute error magntude Resdual error dstrbuton for SFS selected features IV. CONCLUSION AND FURTHER INVESTIGATION A novel objectve speech qualty estmaton algorthm s proposed based on Gaussan mxture modelng. We have nvestgated the usefulness of features selected by CART, MARS, a CART-MARS hybrd scheme, and the SFS algorthm. The SFS algorthm provded the best performance. The CART-MARS hybrd scheme mproved on CART by ncludng features from unvoced frames. When usng dagonal Gaussan components we observed that our approach provded only modest mprovement over PESQ. Ths was attrbuted to the fact that the fve most salent feature varables selected by the data mnng technques were correlated and the use of only fve dagonal components was not suffcent to compensate for ths. Wth full Gaussan components we have obtaned an mprovement over PESQ of 8% and 21.7% n R and RMSE, respectvely. Whle our results show that feature mnng n conjuncton wth GMM modellng can produce smple estmators that outperform PESQ, the robustness of the estmators s also an mportant ssue. Our use of cross-valdaton to measure performance offers some robustness. We are currently pursung other avenues ncludng the choce of feature varables. Ongong research examnes feature selecton that drectly optmzes GMM estmaton performance. Currently, CART/MARS selected features could be suboptmal for GMM estmaton. Deeper nsghts would shed lght on why the approach works well or not so well, and whether there s a gap relatve to best possble performance. V. APPENDIX Here we descrbe the feature varables shown n Table IV. The seven subbands are ordered from 0 to 6 and the three dstorton severty classes from 0 to 2. I P VUV: Rato of the number of nactve frames to the total number of actve speech frames; U P VUV: Rato of the number of unvoced frames to the total number of actve speech frames; U P: Percentage of unvoced frames n the speech fles; REF 1: Hgh-frequency spectral energy of reference sgnal; V B : Dstorton for subband of voced frames, wthout dstorton severty classfcaton; V B j: Dstorton for severty class j of subband of voced frames; V O : Dstorton for ordered subband of voced frames, wthout dstorton severty classfcaton; V WM : Weghted mean dstorton for severty class of voced speech frames; V WM: Weghted mean dstorton of voced speech frames; V RM: Root-mean dstorton of voced speech frames. REFERENCES [1] ITU-T Rec. P.830, Subjectve performance assessment of telephoneband and wdeband dgtal codecs, Internatonal Telecommuncaton Unon, Geneva, Swtzerland, Feb [2] ITU-T Rec. P.862, Perceptual evaluaton of speech qualty (PESQ): An objectve method for end-to-end speech qualty assessment of narrowband telephone networks and speech codecs, Internatonal Telecommuncaton Unon, Geneva, Swtzerland, Feb [3] W. Zha and W.-Y. Chan, A data mnng approach to objectve speech qualty measurement, n Proc. of the Internatonal Conference on Acoustcs, Speech, and Sgnal Processng 2004, vol. 1, May 2004, pp [4], Voce qualty assessment usng classfcaton trees, n Proc. of the 37 th Aslomar Conference on Sgnals, Systems and Computers, vol. 1, November 2003, pp [] J. H. Fredman, Multvarate adaptve regresson splnes, The Annals of Statstcs, vol. 19, no. 1, pp , March [6] L. Breman, J. Fredman, R. Olshen, and C. Stone, Classfcaton and Regresson Trees. Monterey, CA: Wadsworth & Brooks, [7] A. Jan and D. Jongker, Feature selecton: evaluaton, applcaton, and small sample performance, n IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 19, no. 2, February 1997, pp [8] S. Wang, A. Sekey, and A. Gersho, An objectve measure for predctng subjectve qualty of speech coders, IEEE Journal on Selected Areas n Communcatons, vol. 10, no., pp , June [9] R. Kubchek, E. Quncy, and K. Kser, Speech qualty assessment usng expert pattern recognton technques, n Proceedngs of the IEEE Conference on Communcatons, Computers and Sgnal Processng, June 1989, pp [10] S. Voran, Objectve estmaton of perceved speech qualty - part I: Development of the measurng normalzng block technque, IEEE Transactons on Speech and Audo Processng, vol. 7, no. 4, pp , July [11], Objectve estmaton of perceved speech qualty - part II: Evaluaton of the measurng normalzng block technque, IEEE Transactons on Speech and Audo Processng, vol. 7, no. 4, pp , July [12] X. Zhuang, Y. Huang, K. Palanappan, and Y. Zhao, Gaussan mxture densty modelng, decomposton and applcatons, IEEE Transactons on Image Processng, vol., no. 9, pp , September [13] D. Reynolds and R. Rose, Robust text-ndependent speaker dentfcaton usng Gaussan mxture speaker models, IEEE Transactons on Speech and Audo Processng, vol. 3, no. 1, pp , January 199. [14] Z. Ghahraman and M. I. Jordan, Supervsed learnng from ncomplete data va an EM approach, n Advances n Neural Informaton Processng Systems, vol. 6. Morgan Kaufmann Publshers, Inc. [1] A. Dempster, N. Lar, and D. Rubn, Maxmum lkelhood from ncomplete data va the EM algorthm, J. Royal Statstcal Socety, vol. 39, pp. 1 38, [16] A. Gersho and R. Gray, Vector Quantzaton and Sgnal Compresson. Kluwer Academc Publshers, [17] A. W. Rx, A new PESQ scale to assst comparson between P.862 PESQ score and subjectve MOS, ITU-T SG12 COM12-D86, May [18] A. Rx, J. Beerends, M. Holler, and A. Hekstra, PESQ - the new ITU standard for end-to-end speech qualty assessment, n 109 th AES Conventon, no. pre-prnt 260, September, pp