An E2E Quality Measurement Framework

Transcription

1 An E2E Quality Measurement Framework David Hands BT Group CTO Perceptual Engineering Research Group ETSI Workshop on Effects of transmission performance on multimedia quality of service June Prague, Czech Republic

2 Overview Objective Quality Measurement Metrics Brief overview and applications Computational, parametric and perceptual methods Framework rationale Why is a QMF necessary? Description of QMF Application of metrics within QMF

3 Objective Quality Measurement Metrics Planning models (E-model, SG12 proposals for video and multimedia) Parametric models Use information from data stream and packet header Designed to provide monitoring (but really limited to aggregating over flows as absence of content information makes them unsuited to individual stream analysis) Perceptual models Bitstream Hybrid Pixel-level

4 Framework Rationale Various models exist with different applications For network dimensioning planning models For basic performance monitoring parametric models For specific stream / service analysis perceptual models Framework defines point-of-use of different models, extension to include input/output parameter and signalling requirements

5 Application of Metrics within QMF: Planning Models Inputs: Pre-transmission network dimensioning applications Network performance characteristics (throughput, latency, plr and loss pattern) Service definition 1 (e.g. IPTV, internet service, speech service) Service definition 2 (codec, error concealment, FEC, buffer size) Content descriptor (some indicator of video content, possibly a flag differentiator to indicate low / medium / difficult content) Outputs: Index / rating of network performance Estimated MOS (but not recommended as it is known to be poorly correlated with subjective scores)

6 Application of Metrics within QMF: Parametric Models In-service performance monitoring applications Inputs: Network performance characteristics (throughput, latency, packet loss events) Packet header (e.g. RTP, MPEG-2 TS) information Service definition (codec, error concealment, FEC, buffer size) Outputs: Estimated MOS (but unknown performance in predicting subjective scores) Aggregated measurement not stream specific monitoring (due to inability to account for content)

7 Application of Metrics within QMF: Perceptual Models Designed to predict subjective quality Various approaches defined (Bitstream, Hybrid, FR/RR/NR pixel-level) Hybrid Model Inputs: Bitstream parameters Decoded picture parameters Outputs: Predicted MOS Stream specific monitoring

8 Quality Measurement Framework

9 Transmission Planning Planning Models - Real traffic flows not used - General overall performance rating desired - But valuable to have indicator of worst performance

10 In-Service Monitoring Parametric Models - Monitoring actual traffic flow - Computationally [very] light and real-time - Need to detect problems - Feedback signalling necessary - Beware! False alarms - Requirement: shown to outperform basic network performance characteristics

11 Client-End Monitoring Perceptual Models (remote monitoring) - Monitoring stream quality - Computationally light and real-time - Signalling requirement - Beware! False alarms - Requirement: prediction accuracy paramount

12 Headend Measurement Perceptual Models (headend measurement) - Measures encoded stream quality - Computationally light and real-time (NAR) - Requirement: prediction accuracy paramount

13 Conclusions QMF introduced and application of different measurement and monitoring tools defined Proposal: For edge measurement use perceptual models In fact use the same perceptual model!! For in-service network monitoring Perceptual models not appropriate (not only encryption, but computational requirements too) Need to prove these are superior to traditional network performance characteristics Beware false alarms, tools must be accurate (essential irrespective of measurement approach))