From pure technical viewing quality to more realistic audiovisual QoE testing Werner Robitza 1, Marie-Neige Garcia 2, Alexander Raake 2,3 1 Telekom Innovation Laboratories, Deutsche Telekom AG 2 Assessment of IP-Based Applications, TU Berlin 3 Department of Audiovisual Technology, TU Ilmenau ETSI Workshop on Telecommunication Quality beyond 2015 Vienna, October 21, 2015
(PARAMETRIC VIDEO) QUALITY MODELS That was bad! Source signal Transmission system Subjective quality rating (MOS) Bitstream parameters Model Estimated quality index ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 2
A TYPICAL TEST DESIGN FULL TEST MATRIX Source samples, ca. 10 30 seconds SRC01 SRC02 SRC03... Different test conditions Cond. 1 Cond. 2 Cond. 3 Cond. 4... Full test design 1 2 3 1 2 3 1 2 3 1 2 3 Cond. 1 Cond. 2 Cond. 3 Cond. 4... ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 3
classic test METHODS Content that is not entertaining 10 second clips The same clip again and again? ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 10/ 19/2015 4
Quality of Experience Real perceived technical quality? models with higher ecological validity Medium-term (audio)visual quality with other technical influence factors (e.g. stalling) Short-term technical (audio)visual quality 10-second quality scores Low-level quality features Human visual system models Auditory models Image quality models
HTTP ADAPTIVE STREAMING Copyright Blender Foundation ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 10/ 19/2015 6
A Thought Example Short video clips (10 seconds) Tests done to train Model X Model X Let s use Model X to monitor an adaptive streaming service MOS from Model X every 10 seconds: 4.7 2 4.7 2 4.7 2 4.7 2 4.7 2 4.7 4.7 Average all scores: The final MOS is 3.575! Invalid assumption ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 10/ 19/2015 7
ITU-T P.NATS Parametric non-intrusive assessment of TCP-based multimedia streaming quality, considering adaptive streaming Ongoing ITU-T Study Group 12, Question 14 work item ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 10/ 19/2015 8 https://upload.wikimedia.org/wikipedia/commons/0/0c/peanutjar.jpg
P.NATS Building Blocks Audio Quality Module Integration Module Audiovisual Integration A/V Quality Integral Quality Video Quality Module Stalling Impact Stalling Quality Model submission: November 2015, winning model selection: March 2016 ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 9
Test Approach FOR P.NATS TESTS PC and mobile testing Source videos 1 5 min length Adaptivity and stalling events ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 10
FROM SHORT TO LONG SEQUENCES 10 seconds are not enough to create realistic adaptivity conditions and stalling effects ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 11
a New immersive DesigN Content is not repeated People can relate to the content (exciting, funny, nice to look at, ) Content from movies or online portals, not test content Long duration content (minutes, not seconds) M. H. Pinson, M. Sullivan, and A. Catellier, A new method for immersive audiovisual subjective testing, in VPQM, 2014. W. Robitza, M.N.-Garcia, A. Raake, At Home in the Lab: Assessing Audiovisual Quality of HTTP-based Adaptive Streaming with an Immersive Test Paradigm, QoMEX Seventh International Workshop on Quality of Multimedia Experience, May 2015. ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 12
Goodbye, Source Repetition Keep users entertained sources are not repeated! SRC01 SRC02 SRC03 SRC04 SRC05 SRC06 SRC66... Cond. 1 Cond. 2 Cond. 3 Cond. 4 Cond. 22... 1 2 3 4 5 6 Cond. 1 Cond. 2... 66... Cond. 22 ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 13
Positive Implications For users in tests: More entertaining Potential to be more engaged For users of models: More realistic usage context Still technical quality Still accurate Potentially broader scope but results should be more valid for that scope ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 14
Are users satisfied? I had no problems concentrating during the test Strongly agree Agree Disagree Strongly disagree ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 15
Conclusion Immersive tests a way to create more realistic scenarios and models Stronger dependency on sources, high quality and realistic sources needed Results still mean technical quality based on user experiencing Chances for standardization to create models for the next generations of services ETSI Workshop on Telecommunication Quality beyond 2015 October 21, 2015 16
THANK YOU!