ITU-T P.NATS Parametric Non-intrusive Assessment of TCP-based multimedia Streaming quality

Transcription

1 ITU-T P.NATS Parametric Non-intrusive Assessment of TCP-based multimedia Streaming quality Development conducted in ITU-T SG 12 / Q.14 Alexander August, 2012Raake, TU Ilmenau, Germany / T-Labs, Deutsche Telekom, Berlin Jörgen Gustafsson, Ericsson, Luleå, Sweden Marie-Neige Garcia, Werner Robitza, Bernhard Feiten, T-Labs, Deutsche Telekom, Berlin

2 Quality Models Stalling That was bad! Source A/V signal Transmission system Subjective quality rating Probe Model Estimated quality index 2

3 ITU-T Standards related to P.NATS \Scenario Timing Existing / upcoming Phase 1 Network planning G.1071 H.264 HD audio-visual G.OM_HEVC Extension of G.1071 to H.265 UDP TCP Service monitoring Network planning Service monitoring P.1201.x same as G.1071 P.1201.x_HEVC Extension of P.1201to H.265 G.1071-PD (non-adaptive streaming) G.OM_HEVC inserted into P.NATS P.NATS (adaptive streaming) same as G.1071 fixed, mobile P.1201.x-PD (non-adaptive streaming) G.OM_HEVC inserted into P.NATS Collaborative approach Re-use of G.OM_HEVC databases Phase 2 4K (Ultra-HD) 3

4 P.NATS Phase 1 Overview Services: HTTP Adaptive Streaming Video-encoding-resolutions: 360p, 480p, 720p, 1080p Covered degradations: Encoding changes (bitrate/qp, framerate, scaling) due to stream-switching Stalling Initial loading Input: model mode-dependent, from side-info to full bitstream Output: audiovisual quality estimate for up to 5 min video + diagnostic information 8 participants 4

5 P.NATS Ph1. Building Blocks ITU-T T13-SG TD-GEN-0671!R1!MSW-E-P.NATS Terms of Reference (ToR). Output: audio visual quality per session ( 5 min) à O.46 Diagnostic information Audio, video, and audiovisual coding quality, per second/segment/session (O.21/22/34/35) Stalling quality per session (O.23) Model inputs (I.11/13/14) 5

6 P.NATS Modes Inputs of audio and video coding modules Mode 0 Mode 1 Mode 2 Stalling and initial loading info Audio & video codecs, bitrates and resolutions Frame sizes Frame types (I vs. Non-I 2% bitstream parsing Level of information Mode 3 Full bitstream parsing 6

7 P.NATS Ph.1 Sources, test design, and rating example Perceived audiovisual quality? Blender Open Movie Project Blender Open Movie Project ITS NTIA ITS NTIA 7

8 P.NATS Ph.1 Subjective Tests Test method Audiovisual ACR, 5-point scale No SRC repetition within a test, but 50% - 67% SRCs re-used between labs Use of anchors, at least for tests with same SRCs duration Degradation types: quality switches, stalling, initial loading, mix Same proportion of degradation types in all tests Training 5 existing databasesfromp.1201-pd (progressive download) 2 buy-in databases (progressive download and adaptive streaming) 16 new training databases 9 LCD: 4x1min-SRCs, 3x3min-SRCs, 2x5min-SRCs 8 mobile: 4x1min-SRCs, 3x3min-SRCs, 1x5min-SRCs Validation 16 validation databases 8

9 P.NATS Work Plan Training databases submitted and checked: 30 Sept (done) Model submission: 20 Nov (done) Validation databases submitted and checked: 20 May 2016 (on-going, subjective tests have started) ITU-T SG12 meeting: 7 16 June 2016 Winning model selection: Summer 2016 Consent P.NATS Phase 1: Oct (tbd. at June 2016 SG12 meeting) 9

10 P.NATS Phase 2 Phase 1 Phase 2 Training & Validation Characterization Audiovisual, Pa, Pv, Pq T in [1, 2, 3, 4, 5] min Video only, Pv T = 10 sec Audiovisual, Pa, Pv, Pq T in [1, 3, 5] min Re-use some short sequences extracted from P.NATS Phase 1 10

11 P.NATS Phase 2: Joint project VQEG AVHD? Current plan: Collaboration with VQEG AVHD Ongoing discussions about character of joint project work Multitude of models considered in one project for first time! Signal-based: FR, RR, NR Bitstream-based: Modes 0, 1, 2, 3 Hybrid Technology UHD-1, HD HFR Video codecs: H.264, HEVC, VP9 11

12 P.NATS Phase 2: Joint project VQEG AVHD? Stream I.01 Media parameter extraction I.11 I.13 Pa: Audio cod. qualityimpact Pv: Video cod. qualityimpact Pq: Integration O.21 O.22B O.22 O.34 O.35 Buffer parameter extraction I.14 Input-level I.15: Pixel info I.16: Reference AVHD: Pv and Pq in one block O.46 O.23 MOS 12

13 P.NATS Phase 2: Joint project VQEG AVHD? P.NATS Phase 1 AVHD / P.NATS Phase 2 Audiovisual, Pa, Pv, Pq T in [1, 2, 3, 4, 5] min Short-term Video Modules: Training & Validation Video only, Pv T = 10 sec AVHD: Validation P.NATS Ph.2: Characterization Audiovisual, Pv in Pq context T in [1, 3, 5] min Re-use some conditions representative of P.NATS Phase 1 à P.NATS Ph. 2 / VQEG AVHD: To be followed! 13

14 Thank You!