looking at the relationship between video bitrates and end-user quality assessment: subjective tests approach

Transcription

1 looking at the relationship between video bitrates and end-user quality assessment: subjective tests approach Orange Labs Ricardo Pastrana-Vidal, Jean-Charles Gicquel June 2008, Effect of transmission performance on Multimedia Quality Service - Prague, Czec Republic

2 contents section 1 section 2 section 3 section 4 problem statement subjective quality evaluation bit rate vs end-user quality assessment conclusions

3 Problem statement compression transmission reconstruction? Target bit rate Effective bit rate decoding GOP Sharpness codec packet loss delay network congestion post-processing Quality Relation? univocal? content dependent? how can we answer them?

4 quality assessment : how to measure it Subjective quality evaluation well recognized directly from humans Mean Opinion Score QAV(1) QAV(n) 0 excellent good fair poor bad MOS = E{QAV}

5 Subjective tests controlled environment luminance peak white background ITU-R BT.814 distances Individual, about 60cm reconstruction computer CRT multimedia test room

6 Subjective tests : methods SAMVIQ : Subjective Assessment Methodology for Video Quality multi-stimulus réf A B C D H Original explicit Impaired Algo 1 Impaired Algo 2 Original hidden Impaired Algo 3 Impaired Algo N Implicit comparison low to high Quality scores stability standardized at EBU and ITU-R [BT.1788] participants screening

7 Subjective tests : methods DSCQS (Double Stimulus Continuous Quality Scale) recommendation UIT-R BT conceived for SD TV 10s 2s 10s 2s 10s 2s 10s 6s test sequence ref test sequence ref vote

8 bit rate vs end-user quality assessment

9 Subjective tests : multimedia codec solutions : proprietary 35 to 991 kbps CIF format 6 contents 25 ips 21 participants bit rates codecs formats? 6 contents tennis, basketball, motorcycle news, film, jazz concert SAMVIQ method Quality

10 quality assessment results: multimedia bit rate vs quality : No univocal relation Same bitrate MOS content dependent Intra content relation Identification of limits?

11 quality assessment results: multimedia multimedia context MPEG4 part2 100 references 0.4 to 2 Mbps 30 participants reference : equivalent quality 40 different codec same conclusions MOS bit rate (Mbps) same quality bitrate kayak news foot A foot S basket

12 quality assessment: tv TV context MPEG2 et MPEG4 AVC Bitrates: 1.7 to 3.4 Mbps DSCQS (Double Stimulus Continuous Quality Scale) : recommendation UIT-R BT Real contents: football, film, ping-pong, cycling, mobile&calendar Format : SD 720x novice participants 6H distance, Sony PVM-20M4E

13 quality assessment results : tv No univocal Inter content differences : 2 subjective categories different impact depending on content MOS SDTV contexte bit rate (Mbps) mobile&calendar mpeg4 avc mobile&calendar mpeg2 cycling mpeg2 cycling mpeg4 avc ping-pong mpeg2 ping-pong mpeg4 avc foot mpeg2 foot mpeg4 avc film mpeg2 film mpeg4 avc

14 conclusions bit rate vs. perceived video quality: not directly related content dependent relation existent: intra content bit rate vs Quality : identification of limits or tradeoff same conclusion independently of codec solutions & image formats transmission: variations of effective bit rate not studied

15 thank you Orange, the Orange mark and any other Orange product or service names referred to in this material are trade marks of Orange Personal Communications Services Limited. Orange Personal Communications Services Limited. France Telecom Group restricted.