Performance Study of Objective Voice Quality Measures in VoIP

Transcription

1 Performance Study of Objective Voice Quality Measures in VoIP Lijing Ding, Ayman Radwan2, Mohamed Samy El-Hennawey3 and Rafik A. Goubrant Department of Systems and Computer Engineering, Carleton University 1125 Colonel By Drive, Ottawa, ON, KJS 5B6, CANADA [Iding, sce. carleton. ca 2Department ofelectrical and Computer Engineering, Queen's University 99 University Avenue, Kingston, ON, K7L 3N6, CANADA ayman. ece. queensu. ca 3Enterprise Multimedia Systems, Nortel 25 Sidney Street, Belleville, ON, K8N 5B7, CANADA hennawey@nortel.com Abstract With the advent of Voice over Internet Protocol (VoIP) service, assessing its voice quality is an area of intense research interest. Due to time-consuming and expensive natures of widely accepted subjective Mean Opinion Score (MOS) test, objective methods are often used as an alternative. This paper presents applicability and accuracy analysis of several leading objective methods in quality testing. Particularly, the paper focuses on packet loss, which is one of major impairments in VoIP. A speech database covering typical packet loss conditions is designed. The subjective MOS test is conducted and the results are evaluated with objective MOS. The database is also used to verify a non-intrusive VoIP speech quality assessor the authors developed [1]. The results show that Perceptual Evaluation of Speech Quality (PESQ) based algorithms are generally acceptable for quantifying the effects of packet loss. In addition, we also find that their performance is limited for codec G. 711 without Packet Loss Concealment (PLC). 1. Introduction Nowadays Internet has evolved into a universal communication network that carries all kinds of traffic. Voice over Internet Protocol (VoIP) has emerged as an important application and it is expected to replace the current public switched telephone networks in the next few years [2]. However, Internet is originally designed for non-real time data communications, VoIP speech quality is not guaranteed. The advance of technology and liberalization of telecommunication market have lead to a large number of voice services being offered with different levels of quality and price. Assessing VoIP speech quality is an area of intense research interest [3], [4], [5]; it is also an imperative task for network designs and optimizations. Subjective test is the most reliable approach for assessing voice quality. The widely accepted metric is Mean Opinion Score (MOS) test defined by ITU-T Rec. P.8 [6]. However, subjective MOS test is timeconsuming and expensive. Recently, many objective methods have been developed to produce a good MOS estimate, such as Perceptual Analysis/Measurement System (PAMS) [7], Perceptual Evaluation of Speech Quality (PESQ, ITU-T Rec. P.862) [8], Mean Opinion Score - Listening Quality Objective (MOS-LQO, ITU- T Rec. P.862.1) [9] and ITU-T Rec. P.563 [1]. They are machine-executable programs and require little human involvement. Objective methods are often used as a simple and quick replacement to the subjective MOS test in many cases. As they are developed from subjective MOS databases, inevitably, they have some performance limitations. Therefore, it is important to recognize how these current objective methods perform under various applications. In [1], the authors developed a VoIP speech quality assessor. One of its components is a packet loss effect model. Tens of thousands of speech signals with a wide range of degradation conditions were analyzed to design the model. MOS-LQO was used as a tool to measure the voice quality, because conducting /7/$ IEEE 197

2 subjective MOS test in such a large scale was beyond any reasonable allocated time and budget. Naturally, a next step is to investigate the applicability and accuracy of MOS-LQO under various packet loss conditions in VoIP, by using the subjective MOS. In this paper, a speech database covering relatively small but key VoIP conditions is designed. The subjective MOS test is conducted in elaborate lab settings. Also, voice quality in MOS-LQO is measured and compared to the subjective one. The purposes of the research are two-fold. First, the subjective MOS database is used to calibrate the developed VoIP speech quality assessor and to verify some findings suggested by using the MOS-LQO. Second, the performance limitations of the objective methods are pointed out. For comparison purposes, the voice quality in PESQ and P.563 is also measured. The results show that, MOS-LQO and PESQ have good accuracy and are generally applicable for voice quality testing under packet loss. However, they have some limitations when Packet Loss Concealment (PLC) is not deployed. Also, the results show that performance of P.563 is moderate and it should be used only when necessary. Although the objective methods are developed from a wide range of subjective databases, they suffer from some performance limitations. On the one hand, even for the impairments they are designed to measure, their performance may be better for some impairment categories than for the others. On the other hard, the deployment of new technologies, such as advanced coding scheme, packet-based transmission, noise reduction algorithms, poises new challenges to them. The current objective methods may not fully capture those effects and their interactions, and they need further validation A developed VoIP speech quality assessor Non-intrusive voice quality testing relies on the receive-end signal only. Our developed parametric, inservice non-intrusive VoIP quality assessment model aims to be deployed at the receive-end media gateway or IP terminal. It is a useful tool for identifying root causes of voice quality degradation, and for voice quality assessment purposes. The overall structure of the assessor is illustrated in Figure Background overview Packet loss detection 2.1. Voice quality testing Voice quality is inherently subjective, as it is determined by the listener's perception. In the widely accepted subjective MOS test, listeners express their opinions on the quality of the speech materials in terms of five categories: excellent, good, fair, poor and bad, with a corresponding integer score: 5, 4, 3, 2, and 1, respectively. The average rating is known as MOS [6]. Although the subjective MOS test is the most reliable, its drawbacks are apparent. The test is quite expensive. Moreover, it provides little technical information on the causes of speech quality degradation. In short, it is impractical for automated, frequent testing purposes. Objective methods aim to yield a subjective MOS estimate. They can be intrusive or non-intrusive, based on whether a reference signal is needed or not during the test. For the former, the well-known algorithms are PAMS and PESQ. MOS-LQO is a single mapping version of PESQ and is standardized as ITU-T Rec. P It is claimed [9] to slightly outperform the original PESQ. P.563 is an example for the nonintrusive methods. Normally, the non-intrusive ones are not as accurate as the intrusive methods, but they are very useful in scenarios where the reference signal is inaccessible, such as on-line live network performance monitoring. Temporal clipping detection Noise Echo Figure 1. Structure of the VoIP speech quality assessor The assessor combines the merits of analyzing both voice payload and packet header. It adopts a three-step strategy [1]. First, the impairments, including packet loss, temporal clipping, echo and noise, are detected. In order to detect the occurrences of packet loss, Internet protocol analysis approach is used. The detections of temporal clipping [11], echo [12] and noise are solely based on processing voice payload. Then, for each impairment, its effect is quantified by using the corresponding parameters detected from the first step. Finally, an overall listening-only model and a conversational model are built [13] by integrating individual models and the ITU-T E-model [14]. 198

3 The packet loss effect part utilizes a set of parameters extracted from the packet loss detection model, mainly through Real-time Transport Protocol (RTP) analysis. Those parameters include codec type, packet size, loss rate, loss pattern and concealment information. Moreover, for lost packets, a polynomial interpolation scheme is used to estimate their features from surrounding good packets. The features are then used to classify the lost packets into three types, namely, voiced, unvoiced and silence, improving the accuracy of packet loss effect modeling. In order to train and test the developed non-intrusive assessor, real MOS for all the degraded speeches is needed. Conducting the subjective MOS test on such a large scale is infeasible. Instead, MOS-LQO is used as the real MOS estimate. As mentioned, this paper presents subjective MOS test design and result analysis for packet loss, for the model verification and performance limitation analysis purposes. 3. Experiment design 3.1. Speech database The developed speech database contains a wide range of typical packet loss conditions, as summarized in Table 1. Several prevalent codecs in both wired and wireless VoIP, including G.7 11, G.729 and Adaptive Multi-Rate (AMR), are selected. G.711 is simply a waveform based codec. G.729 and AMR operate on frames and are based on the code excited linear prediction principles [15]. For G.711, built-in PLC refers to the algorithm given in ITU-T Rec. G.711 Appendix I, and no PLC stands for using silence to replace lost packet. A total of 18 packet loss conditions are included. Table 1. Parameters used in packet loss Item Parameter Codec G.711, G.729, AMR(12.2 kb/s) Packet size 1, 2, 3 ms for G.711 and G.729; 2 ms for AMR Loss rate 1%, 2%, 4%, 7%, 1% Loss pattern random, bursty PLC built-in and no PLC for G.711; built-in PLC for G.729 and AMR Tandeming G.729 x G.729, AMR x AMR The database also contains 16 reference conditions, including unfiltered, Modified Intermediate Reference System (MIRS) send filtered, Modulated Noise Reference Unit (MNRU), waveform and code excited linear prediction codecs under a clean channel. These reference conditions serve as an anchor, and allow meaningful comparison with other databases [16]. Raw speeches are recorded in high-quality anechoic chamber. Four native Canadian French talkers (two males and two females) in Ottawa area are used for the above conditions. Raw speeches are MIRS send filtered (expect one unfiltered condition in the reference), level adjusted to -26 dbov and saved in 8 Hz, 16-bit format. Each speech sample contains two different sentences separated by 65 ms silence (background noise). In total, 496 speech samples are dedicated to these conditions Subjective MOS test The subjective MOS test was conducted by the Speech and Audio Processing Lab, University of Sherbrooke in 26. The test follows the requirements defined in ITU-T Rec. P. 8 and P.83. Sixty native French listeners (38 males and 22 females) participate in the test. The ambient noise level in the listening room is measured daily at the head position of each listener (in the absence of a listener). It is consistently situated at around dba. The headphones used are Beyerdynamic DT Objective measurements MOS-LQO, PESQ and P.563 are used to measure the voice quality. The former two are implemented by using a hardware toolbox called Digital Speech Level Analyzer (DSLA) [17]. P.563 measurement is based on the ITU-T source C code [1]. 4. Results analysis As a starting point, for the reference conditions, the MNRU curve is depicted in Figure 2 (a). It shows that MOS ratings increase monotonically with increases in dbq. Diminishing returns for improvements above 25 dbq are exactly as expected. The ratings for the clean codec are presented in Figure 2 (b). The direct (no filtering) and MIRS send filtering cases both use linear quantization. G.711 introduces non-linear quantization steps, and the remaining cases introduce compression with various codecs as indicated. The ratings are expected: the direct one has the highest MOS, followed by the MIRS send filtered, G.711 and various codecs. One exception is that the rating for AMR 7.4 kb/s seems to be a little bit higher than normal. Generally speaking, the results are consistent with expectation, given the range of quality in the study as a whole. 199

4 5 4.5 U) O 3.5 a) > 3 a) 2.5 U) (a) 5 4.5~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ t < 3.5 -J 4 _ t O ta t++vaw AX ~~~~ , 2_ + 7' Subjective MOS Figure 3. The subjective MOS vs. MOS-LQO for the database (b) Figure 2. The reference conditions (a) MNRU curve (b) clean codecs A MOS cloud for the database is illustrated in Figure 3, where a dot representing a subjective MOS and its corresponding objective MOS. Both subjective and objective MOS values are observed within range from the low-end 1. to the high-end It shows that MOS-LQO performs very well as the dots are closely and equally distributed around the diagonal line. The performance of these objective algorithms is characterized by the Pearson correlation coefficient p and Root Mean Square Error (RMSE) a. The former measures the linear relationship between the subjective and objective MOS. The latter characterizes how the objective one is close to the subjective MOS. The analysis is based on the database as a whole, the reference, random loss and bursty loss conditions. The statistics for p and a are summarized in Table 2. It is seen that p is as high as.89 for PESQ and MOS-LQO; for P.563, only a moderate.56 is seen. Also, a for PESQ and MOS-LQO is smaller than that of P.563. As expected, the intrusive methods are more effective than the non-intrusive one, because the former have access Table 2. Correlation and RMSE of the objective methods Random Bursty All Reference loss loss MOS- p LQO a PESQ p a P.563 p a to the reference. P.563 is not a reliable approach, and should be used with cautions. Similar performance in terms of p and a is observed when comparing MOS-LQO to PESQ. The margin is too small to test the conclusion that MOS-LQO is superior to PESQ. Perhaps this needs wide range MOS databases. The rest of the analysis mainly focuses on the MOS-LQO, as it is used in our assessor. Then, we look into the performance over different codecs. The results are illustrated in Figures 4 (a) and (b) for random and bursty loss, respectively. In order to make the figures clear, per condition MOS, that is, the averaged MOS from the four samples are used. As seen from the figures, a very good linear relationship is observed, suggesting that MOS-LQO correlates well with both random and bursty loss conditions. By further examining distribution of the dots around the diagonal line, for random loss, it is biased upward for G.711 without PLC, which means that MOS-LQO tends to be optimistic on voice quality in this case. For bursty loss, the distribution is quite reasonable although it is a bit biased downward. Finally, the above finding regarding to G.711 without PLC leads us to investigate its performance 2

5 a cx cs C\o D _ 4X , a 3.5 cn 2 3 c'i ' 2.5 L * G.711 w PLC G.711 w/o PLC * G.729 AMR Subjective MOS (a) * G.711 w PLC x G.711 w/o PLC * G.729 AMR Subjective MOS * a (b) Figure 4. Packet loss conditions (a) random loss (b) bursty loss 3.5,,) 3 a) 2.5 a) ") I- G.71 1 Random vs. bursty loss without PLC * Random IM Burst E E E E E E U) C\J CO U) C\j CO C\J E E ) Figure 5. The subjective impacts of bursty packet loss for G.711 without PL( from another angle. Under the E-model [14] and our assessor using MOS-LQO [1], bursty loss would result in lower voice quality relative to random loss when U,) E Co Cor other conditions are the same, as the former causes longer clipping and the effect is expected to be more annoying. However, the subjective test shows contrary results for G.711 without PLC, as illustrated in Figure 5. This can be explained as follows, when PLC is not used, the effect of reduced frequency of clicks may compensate more that of the loss of multiple packets in the bursty cases. In summary, MOS-LQO and PESQ are generally applicable in evaluating the effects of packet loss in VoIP, and they have similar performance. They are limited in handling G.711 without PLC. As a nonintrusive measure, P.563 only shows moderate performance in packet loss. 5. Conclusions This paper examines the applicability and accuracy of several leading objective voice quality measures in packet loss, which is a major impairment to voice quality in VoIP. The subjective MOS test is designed and conducted. The database verifies their applicability and also reveals some performance limitations of the PESQ-based algorithms. Voice quality assessment is a challenging task. Future work will be focusing on design of subjective MOS database covering more scenarios. Also, efforts will be undertaken to improve generalizability and accuracy of the PESQ-based algorithms. Acknowledgment This research was supported by Nortel. The authors would like to thank Dr. Leigh Thorpe of Nortel for test condition definition, experiment design and her invaluable comments on the interpretation of testing results. We are also indebted to Dr. Roch Lefebvre, Mr. Martin Brousseau and Mr. Cedric Demers of University of Sherbrooke for conducting the subjective MOS test. References [1] M.S. El-Hennawey, R.A. Goubran, A. Radwan and L. Ding, "Method and apparatus for non-intrusive single-ended voice quality assessment in VoIP," Europe patent publication no. W A1, April 6, 26. [2] K. Salah and M. Almashari, "An Analytical Tool to Assess Readiness of Existing Networks for Deploying IP Telephony," In Proc. 11th IEEE Symposium on Computers and Communications, June 26, pp [3] L. Carvalho, E. Mota, R. Aguiar, A.F. Lima, and J.N. de Souza, "An E-model implementation for speech quality 21

6 evaluation in VoIP systems," In Proc. 1th IEEE Symposium on Computers and Communications, June 25, pp [4] A.P. Markopoulou, F.A. Tobagi and M.J. Karam, "Assessing the quality of voice communications over Internet backbones," IEEE/ACM Trans. Networking, vol. 11, no. 5, pp , October 23. [5] A. Tarraf and M. Meyers, "Neural network-based voice quality measurement technique," In Proc. IEEE Symposium on Computers and Communications, July 1999, pp [6] ITU-T Rec. P.8, Methods for subjective determination of transmission quality, August [7] A.W. Rix and M.P. Hollier, "The perceptual analysis measurement system for robust end-to-end speech quality assessment," In Proc. IEEE ICASSP, June 2, vol. 3, pp [8] ITU-T Rec. P.862, Perceptual Evaluation of Speech Quality (PESQ): An objective method for end-to-end speech quality assessment of narrowband telephone networks and speech codecs, February 21. [9] ITU-T Rec. P.862.1, Mapping function for transforming P.862 raw result scores to MOS-LQO, November 23. [1] ITU-T Rec. P.563, Single-ended method for objective speech quality assessment in narrow-band telephony applications, May 24. [11] L. Ding, A. Radwan, M.S. El-Hennawey and R.A. Goubran, "Measurement of the effects of temporal clipping on speech quality," IEEE Trans. Instrum. Meas., vol. 55, no. 4, pp , August 26. [12] L. Ding, M.S. El-Hennawey and R.A. Goubran, "Nonintrusive measurement of echo path parameters in VoIP environments," IEEE Trans. Instrum. Meas., vol. 55, no. 6, pp , December 26. [13] M.S. El-Hennawey, R.A. Goubran, A. Radwan and L. Ding, "Apparatus, method, and computer program for nonintrusive single-ended voice quality assessment in VoIP," U.S. patent application no. 6/81575, June 26. [14] ITU-T Rec. G.17, The E-Model, a computational model for use in transmission planning, March 25. [15] A.S. Spanias, "Speech coding: a tutorial review," Proc. IEEE, vol. 82, no. 1, pp , October [16] L.A. Thorpe and W. Yang, "Performance of current perceptual objective speech quality measures," In Proc. IEEE Workshop on Speech Coding, June 1999, pp [17] Digital Speech Level Analyzer, User Guide, Revision 4., Malden Electronics Ltd., Surrey, UK,