EURESCOM P905 (AQUAVIT) Audio and audiovisual quality for mobile services Harald Klaus T-Systems Rapporteur of Q9/12 Harald.Klaus@T-Systems.com 1
Outline!Personal Profile!Framework and Objectives of AQUAVIT!Test beds of AQUAVIT!Test methods developed in AQUAVIT!Results of Investigations!Conclusions 2
Personal Profile Harald Klaus!Studied electrotechnical engineering at the Technical University of Berlin!Research assistant for speech recognition applications and speech quality assessment!head of quality assessment of tele-services at T-Systems Nova Berkom in Berlin!Rapporteur in ITU-T Study Group 12 for instrumental speech quality assessment methods!rapporteur in ETSI Technical Committee STQ 3
The EURESCOM Project P905 (AQUAVIT) 4
Framework of EURESCOM P905 AQUAVIT = Assessment of Quality for Audio-Visual Signals over IP and UMTS!Start: January 1999!End: December 2000!3 Partners... 5
Objectives of AQUAVIT!Development of test beds for IP and UMTS!Development of subjective and objective test methods!quality assessment of typical audio-video transmission scenarios for IP and UMTS!Correlation between subjective and objective quality evaluations 6
Testbeds Coding of signals for UMTS on send side Encoders MPEG-4 Elementary Stream Elementary Stream Compression Layer SL-packetized stream Access Unit Access Unit Access Unit AU AU AU AU AU Sync. Layer Transmission errors 7
Testbeds (2) Decoding of signals for UMTS on receive side Application Executive Presenter T Demux Media Stream Media Stream BIFSDecoder Decoder MediaT Stream Root Scene Object Media Object Visual Renderer Audio Renderer 8
Testbeds (3) IP Testbed! Running on standard PCs in realtime! Adjustable set of parameters for!coding/decoding!bit rate!bit and Frame Errors!Error distribution! Windows user interface 9
Subjective test methods Speech quality evaluation Sequence I just one vote for the whole sequence MOS a) ordinary way of voting Time SI Part a SI Part b SI Part c SI Part d SI Part e SI Part f Six different votes for the sequence MOS Time b) simulation of continuos voting 10
Subjective test methods Speech quality evaluation 11
Subjective test methods Audio quality evaluation 12
Subjective test methods Video & A/V quality evaluation 13
Subjective test methods Acceptability tests Questionnaire Conjoint analysis Focus groups 14
Objective test methods!tosqa (Telecommunication objective speech quality assessment)!peaq (Perceptual evaluation of audio quality)!a new video quality assessment model developed by British Telecom 15
Results of investigations Correlation of subjective and objective speech quality (TOSQA) 5 5 5 4 4 4 TMOS 3 TMOS 3 TMOS 3 2 2 2 1 1 2 3 4 5 MOS 1 1 2 3 4 5 MOS 1 1 2 3 4 5 MOS Experiment 1 narrow band & wideband speech ρ = 96.6% Experiment 2a Only narrow band speech ρ = 95.4% Experiment 2b Only wideband speech ρ = 98.2% 16
Results of investigations Subjective test results - audio quality Low quality conditions (average of all sequences) 100 90 80 70 60 50 40 30 20 10 0 Mushra Peaq Re fe renc e Anch or 7.0 A nch or 3.5 AAC 3 2 UMTS 2 AAC 3 2 UMTS 1 AAC 32 IP3 AAC 32 IP2 AAC 3 2 IP1 CELP 24 AAC 24 AAC 32 17
Results of investigations Subjective test results - audio quality High quality conditions (average of all sequences) 120 100 80 60 40 Mushra Peaq 20 0 Reference H Anchor 7.0 H Anchor 3.5 H AAC 96 UMTS 2 AAC 96 UMTS 1 AAC 9 6 IP3 AAC 96 IP2 AAC 96 IP1 AAC 48 A AC 64 A A C 96 18
Results of investigations Acceptability test results!quality and cost are the equally important main factors to the population questioned.!benefit of mobility does not compensate for any deficit in quality. 19
Conclusions!AQUAVIT developed a bundle of improved subjective quality methods for assessment of!speech quality (narrowband and wideband)!audio quality (intermediate and high quality)!video and A/V quality (intermediate and high q.)!all tests use the same set of material!results of all tests can be directly used for comparisons 20
New test services of T-System T Nova are based on AQUAVIT methods and results User-centred Quality Engineering. Quality assessment at T-Systems Nova. Consulting # Test concepts and test design # Quality Monitoring and Reporting # Process Optimization Speech, audioand A/V quality measures # According to international recommendations and standards # Optimized, efficient test procedures # Tests with representative user groups # Intrusive and non-intrusive measurement methods # Measurements with Head-and-Torso simulators (HATS) ===!" ==Syst ems= Our Labs # High quality according to ITU requirements # for auditoryand instrumental quality assessment # for recordings using HATS # for speech and audio production T-Systems Nova Berkom Harald Klaus Head of Quality Assessment 21