Quality of Service and Quality of T-Labs Berlin

Transcription

1 Quality of Service and Quality of T-Labs Berlin Sebastian Möller, Alexander Raake and Marcel Wältermann Quality and Usability Lab Deutsche Telekom Laboratories TU Berlin {sebastian.moeller, alexander raake, marcel.waeltermann}@telekom.de

2 Outline Quality of Service (QoS) and Quality of Experience (QoE) Definitions Quality taxonomy Subjective evaluation methods Quality prediction 1

3 QoS and QoE Definitions. System developer s point-of-view: Performance: The ability of a unit to provide the function it has been designed for. (Möller, 2005) Quality of Service (QoS): The collective effect of service performance which determines the degree of satisfaction of the user of the service. (ITU-T Rec. E.800, 1994) Includes service support, service operability, serveability, and service security User s point-of-view: Quality: Result of appraisal of the perceived composition of the service with respect to its desired composition. (ITU-T Rec. P.851, 2003, following Jekosch, 2000, 2005) Quality of Experience (QoE): The overall acceptability of an application or service, as perceived subjectively by the end user. (ITU-T Rec. P.10, 2007) Includes the complete end-to-end system effects May be influenced by user expectations and context 2

4 QoS and QoE Definitions. Definitions discussed by participants of the Dagstuhl Seminar "From Quality of Service to Quality of Experience (May 2009) Quality of Experience (QoE): Degree of delight of the user of a service. In the context of communication services, it is influenced by content, network, device, application, user expectations and goals, and context of use. Service: An event in which an entity takes the responsibility that something desirable happens on the behalf of another entity. Acceptability: Characteristic of a service describing how readily a person will use the service. Acceptability is the outcome of a decision which is partially based on the Quality of Experience. 3

5 QoS and QoE Taxonomy. Quality of Service (QoS) Influencing factors Interaction performance Static factors Cognitive workload User Dynamic factors Perceptual effort User Physical response effort Context Environmental factors Form appropriatness Input performance Service factors Output modality appropriatness Input modality appropriatness System Agent factors Contextual appropriateness Dialog management performance Interpretation performance System Functional factors Output quality Cooperativity Input quality Quality of Experience (QoE) Aesthetics Appeal System Personality Joy of use Interaction quality Usability Ease of use Learnability Effectiveness Efficiency Intuitivity Usefulness Utility Hedonic Acceptability Pragmatic (Möller et al., 2009) 4

6 Outline Quality of Service (QoS) and Quality of Experience (QoE) Subjective evaluation methods Dimension-based approach Evaluation of conversational quality Quality prediction 5

7 Dimension-based Assessment of Speech Quality Auditory analysis. Auditory tests Multidimensional Scaling Semantic Differential Quality Dimensions: Discontinuity Noisiness Coloration (Wältermann et al., 2006; 2008) 6

8 Dimension-based Assessment of Speech Quality Modelling principle. tot Subjective Quality and Feature Judgements Transmission System dis noi col Knowledge Model Estimated Quality Index Estimated Quality Dimensions (Heute et al., 2005; Wältermann et al. 2009) 7

9 Subjective Evaluation Methods Evaluation of conversational quality from 2 to 3 subjects. persons welcome request/ proposal objection/ proposal necessary information summary open question interactive task discussion of open question goodbye 8

10 3CTs conversation scenarios. Conversation tests. Two conversation test series conducted, yielding a total of 2*8*10 = 160 (recorded) 3-person conversations. Conversation analysis ongoing. Basis for further research. (Raake and Schlegel, 2008) Speech quality judgments 3D Audio versus diotic. In conversation, reduced spatial quality advantage when compared with listening only (e.g. Baldis, 2001; Raake et al., 2007) Listening test in preparation including memory & performance. 9

11 Outline Quality of Service (QoS) and Quality of Experience (QoE) Subjective evaluation methods Quality prediction Signal-based: P.OLQA Parametric: E-model IPT 10

12 Quality Prediction Models Signal-based speech quality prediction: P.OLQA. Listener Room Acoustics Signal Processing Transmission System Codec... Noise Linear Dist. Packet loss Model MOS Estimated MOS Measurement Tool (Wältermann et al., 2008) 11

13 P.OLQA Candidate Model. Signal-based Core & Single Dimension Estimators. Transmission System Pre- Processing Internal Represent. d MOS Comparison Integration Transform. Pre- Processing Internal Represent. (Côté 2008; Wältermann et al., 2008b,c) Discontinuity Indicator Noisiness Indicator Coloration Indicator Loudness Indicator bi dis bi noi bi col I bw bi lou 12

14 4 4 Quality Prediction Models Parametric speech quality prediction: E-model. Linear distortion, delay IP IP WAN WAN Backgr. noise, acoustic coupling Codec Jitter buffer, AD Packet loss Talker echo, listener echo Circuit noise Backgr. noise, acoustic coupling 13

15 4 4 Quality Prediction Models Parametric speech quality prediction: E-model. SLR, RLR, Ta IP IP WAN WAN Ps, Ds, STMR Ie, qdu Bpl Ppl TELR, T, WEPL, Tr Nc, Nfor Pr, Dr, LSTR Impairments SNR simultaneous delayed nonlin./timevar. Overall quality R = Ro - Is - Id - Ie,eff Estimated user judgment MOS = f (R ) 14

16 Quality Prediction Models Parametric speech quality prediction: E-model. Extension to wideband and beyond Ro,max = 129 AMR-WB (23.05) AMR-WB (6.6) R NB R max = R NB/WB (Raake, 2006; Möller et al, 2006; Appendix II, ITU-T Rec. G.107, 2006) 15

17 Quality of IPT Overview perceptual bitstream model (Raake & Garcia, 2008; Garcia & Raake, 2009) 16

18 Quality of IPT ideo quality model first algorithm ideo oip (E-model) Qv = Qo + Fres + Fcod + Ftra + Fdis R = Ro Is Id Ie, eff Qo + Fres + Fdis = max( Qv subjective ) Fcod Ftra = = a 1 ( b exp( a 2 2 Fcod bitrate) + a ) Ppl Ppl + b 1 3 Ie, eff = Ie+ (95 Ie) Ppl Ppl+ Bpl b 1 = c 1 + c 2 / bitrate Comparison with test results: r > 0.95,RMSE < 7 (Raake et al., 2008) 17

19 Quality of IPT ideo quality under packet loss ideo H.264, SD, 4 MBit/s, 1 fps PLCs: 1 slice/2 Macroblock lines vs. skipping oip G.729, 2 frames/packet, native PLC Quality (test) 2-state Markov loss Paremeter: Average number of packets lost in a row μ 10 = 1/q 18

20 Thankyouforyourattention. isit for more information.