Lec 14 QoE Quality of Experience

Transcription

1 CS/EE 5590 / ENG 401 Special Topics (17804, 17815, 17803) Lec 14 QoE Quality of Experience Zhu Li Course Web: Z. Li, Multimedia Communciation, 2016 Spring p.1

2 About the Project Sign Up Outline QoE QoS and QoE Def and Standards Body Subjective QoE evaluation MOS Objective QoE metrics Perceptual QoE metrics Summary Z. Li, Multimedia Communciation, 2016 Spring p.2

3 Project Sign Up Objective: For PhD, MS Thesis Option students, align to your research goal For course MS/BS students, deep dive on an useful technology that can help your job hunting Z. Li, Multimedia Communciation, 2016 Spring p.3

4 Video Streaming/Networking: Project Sign Up Z. Li, Multimedia Communciation, 2016 Spring p.4

5 Part II: Video Streaming and Networking Tentative Topics: QoE Metrics: Referenced, Light Reference, and Reference-less QoE metrics Video over Multiple Access Networks : Resource Pricing Solution, DP+Lagrangian Framework MPEG Systems: File Format (MP4Box), Streaming Solution (DASH.js), MMT Media Transport: RTP/RTSP, HTTP/WebSocket, WebRTC, and QUIC Congestion Measure and Modeling in Media Networking P2P Systems Content Identification and Info Centric Networking Z. Li, Multimedia Communciation, 2016 Spring p.5

6 QoS QoS Quality of Service A network centric metric Measuring the delay, loss, throughput,..etc Does not directly translate into user experiences Typically characterized by the packet arrival and departure curves Buffer size: b(a,d, t), delay, d(a,d, t) Z. Li, Multimedia Communciation, 2016 Spring p.6

7 QoE Quality of Experience A user centric metric, how a piece of audio/visual signal delivered by the network looks/feels Usually a function of QoS, content, and viewing conditions Fig credit: Touradj Ebrahimi, EPFL, ACM MM 2009 Z. Li, Multimedia Communciation, 2016 Spring p.7

8 ITU STRUCTURE 3 Sectors: Standardization (ITU-T): promotes enabling technical, policy and regulatory frameworks to boost ICT development Radiocommunication (ITU-R): coordinates the shared global use of radio spectrum and geostationary satellite orbit Development (ITU-D): works to improve telecommunication infrastructure in the developing world Z. Li, Multimedia Communciation, 2016 Spring p.8

9 ITU-T Structure and organization WTSA TSAG Study Group x Study Group y Study Groups Working Party 1/x Working Party 2/x Working Party 3/x Working Party 1/y Working Parties Question 1/1 Question 1/2 Question 1/3 Question 1/1 Z. Li, Multimedia Communciation, 2016 Spring p.9

10 Study Group 9 Overview Lead Study Group on integrated broadband cable and television networks Responsible for studies relating to: use of telecommunication systems for contribution, primary distribution and secondary distribution of television, sound programmes and related data services including interactive services. use of cable and hybrid networks, primarily designed for television and sound programme delivery to the home, as integrated broadband networks to also carry voice or other time-critical services, video on demand, interactive services, etc. Z. Li, Multimedia Communciation, 2016 Spring p.10

11 SG9 QoE metrics work Rec. # Name Qu Title Timing J.249 J.redref Q2/9 Perceptual video quality measurement techniques for digital cable television in the presence of a reduced reference J.340 J.ra-psnr Q2/9 Reference Algorithm for Computing Peak Signal to Noise Ratio (PSNR) of a Video Sequence with Constant Spatial Shifts and a Constant Delay J.341 J.vqhdtv-fr Q2/9 Objective perceptual multimedia video quality measurement of HDTV for digital cable television in the presence of a full reference 2010 J.bitvqm J.bitvqm Q12/9 Hybrid perceptual bitstream video quality assessment 2013 J.av-dist J.av-dist Q12/9 Methods for subjectively assessing audiovisual quality of internet video and distribution quality television, including separate assessment of video quality and audio quality 2013 J.3D-dispreq J.3D-dispreq Q2/9 Display requirements for 3D video quality assessment 2013 Z. Li, Multimedia Communciation, 2016 Spring p.11

12 Performance, QoS and QoE Study Group 12 Overview Responsible for Recommendations on performance, quality of service (QoS) and quality of experience (QoE) for the full spectrum of terminals, networks and services ranging from speech over fixed circuit-based networks to multimedia applications over networks that are mobile and packet based. Included in this scope are the operational aspects of performance, QoS and QoE. A special focus is given to interoperability to ensure end-to-end users' satisfaction. SG 12 is the Lead SG on QoS and Performance Z. Li, Multimedia Communciation, 2016 Spring p.12

13 SG12 Visual Quality Assessment Rec. # Name Qu Title Timing P.1201 P.NAMS Q14/12 Parametric non-intrusive assessment of audiovisual media streaming quality P.1202 P.NBAMS Q14/12 Parametric non-intrusive bitstream assessment of video media streaming quality G.1080 G.IPTV- QoE Q13/12 Quality of experience requirements for IPTV services 2008 G.1050 Q13/12 Network model for evaluating multimedia transmission performance over the Internet Protocol G.OMVAS G.OMVAS Q13/12 Opinion model for video and audio streaming applications P.1401 P.STAT Q9/12 Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models Z. Li, Multimedia Communciation, 2016 Spring p.13

14 Video Quality Experts Group Founded 1997 ITU-T SG 12, SG 9, and ITU-R 11E (now 6C) experts Web ( ); Primary mission: Advance the field of video quality assessment by investigating new and advanced subjective and objective measurement techniques VQEG does not develop or publish standards Conducts tests and reports results to ITU and other standards organizations Tests are conducted using specifically defined procedures (i.e., carefully developed test plans). First VQEG meeting (Turin, Italy 1997) Z. Li, Multimedia Communciation, 2016 Spring p.14

15 VQEG Projects Completed FRTV I & II (5 ITU Recommendations) Multimedia I (7 ITU Recommendations) RRNR (3 ITU Recommendations) HDTV I (2 ITU Recommendations) Active: 3DTV (3 ITU Recommendations in progress) Joint Effort Group JEG-Hybrid Hybrid Perceptual/Bitstream (3 ITU Recommendations in progress) Multimedia II MM2 (1 ITU Recommendation in progress) Quality for Recognition Tasks QART (Public Safety, Surveillance Applications) (1 ITU Recommendation) Ramping up High Dynamic Range Video HDR HDTV Phase II HDTV2 Monitoring of Audio Visual Quality by Key Indicators MOAVI Real-Time Interactive Communications Evaluation RICE Z. Li, Multimedia Communciation, 2016 Spring p.15

16 VQEG/Standardization Process ITU-T & ITU-R SG9, SG12, SG16, WP6C Standards & Reports Other Standards Orgs ATIS, IEEE, ETSI, MPEG Industry & Academia VQEG Results Z. Li, Multimedia Communciation, 2016 Spring p.16

17 Useful Links VQEG: Z. Li, Multimedia Communciation, 2016 Spring p.17

18 About the Project Sign Up Outline QoE QoS and QoE Def and Standards Body Subjective QoE evaluation MOS Objective QoE metrics Perceptual QoE metrics Summary Z. Li, Multimedia Communciation, 2016 Spring p.18

19 QoE Subjective Evaluation MOS Mean Opinion Score, an user study based quality evaluation A subjective tests aiming at producing MOS is a delicate mixture of ingredients and choices: Test/lab environment Test material Test methodology Analysis of the data credit: Touradj Ebrahimi, EPFL, ACM MM 2009 Z. Li Multimedia Communciation, 2016 Spring p.19

20 Test/lab environment Type of Monitors/Speakers and other test equipments Lighting /Acoustic conditions Laboratory architecture, background, Viewing distance /Hearing position Z. Li Multimedia Communciation, 2016 Spring p.20

21 Test material Meaningful content for the envisaged scenario/application Typical content Worst case content p01 p06 p10 bike cafe woman Z. Li Multimedia Communciation, 2016 Spring p.21

22 Test methodology (I) Single Stimulus (SS) Categorical numerical grading scale: Rate from 1 to 11 Categorical adjectival grading scale: Non-categorical adjectival or numerical grading scale 5 Excellent 4 Good 5 Imperceptible 4 Perceptible but not annoying 100 Excellent 3 Fair 3 Slightly annoying 2 Poor 1 Bad 2 Annoying 1 Very annoying 0 Bad Z. Li Multimedia Communciation, 2016 Spring p.22

23 Test methodology (II) Double Stimulus Impairment Scale (DSIS) Categorical impairment grading scale: 5 Imperceptible 4 Perceptible but not annoying 3 Slightly annoying 2 Annoying 1 Very annoying Z. Li Multimedia Communciation, 2016 Spring p.23

24 Test methodology (III) Double Stimulus Continuous Quality Scale (DSCQS) Non-categorical adjectival or numerical grading scale: Sample 1 Sample Excellent 100 Excellent 0 Bad 0 Bad Sample 21 Z. Li Multimedia Communciation, 2016 Spring p.24

25 Stimulus Comparison (SC) Test methodology (IV) Categorical adjectival comparison scale: Non-categorical judgement: same or different much worse worse slightly worse the same slightly better better much better Much worse Much better Z. Li Multimedia Communciation, 2016 Spring p.25

26 Test methodology (V) Single Stimulus Continuous Quality Evaluation (SSCQE) (Very annoying) (Imperceptible) Z. Li Multimedia Communciation, 2016 Spring p.26

27 Test methodology (VI) Simultaneous Double Stimulus for Continuous Evaluation (SDSCE) (Much better) (Reference) (Test sequence) (Much worse) Z. Li Multimedia Communciation, 2016 Spring p.27

28 Improve MOS data quality Analysis of the MOS data Scores distributions across subjects (testing people) is assumed to be close to normal distribution Outlier detection and removal Mean Opinion Scores (MOS) and 95% confidence intervals (CI j ) t MOS CI obs j j N i 1 N m t(1 / 2, N) MOS A MOS 2 2 A B N N ij B j N m ij = score by subject i for test condition j. N = number of subjects after outliers removal. t(1-α/2,n) = t-value corresponding to a two-tailed t- Student distribution with N-1 Degrees of Freedom (DoF) and a desired significance level α (α=0.05 in our case, 95% confidence). σ j = standard deviation of the scores distribution across subjects for test condition j. Z. Li Multimedia Communciation, 2016 Spring p.28

29 What is behind a MOS? JPEG Image Quality Assessment Study: Z. Li Multimedia Communciation, 2016 Spring p.29

30 Relationship between estimated mean values Hypothesis test to find out whether the difference between two MOS values are statistically significant Two-sided t-test: H H 0 a T-statistic: : MOS A MOS B : MOS MOS A B t obs MOS A MOS 2 A N N 2 B B Decision rule to reject H 0 : t obs t( / 2, N) OR t t(1 / 2, N) obs Z. Li Multimedia Communciation, 2016 Spring p.30

31 MOS hypothesis test JPEG :2:0 6 JPEG :4:4 JPEG JPEG XR MS JPEG XR PS 0.25 bpp 0.50 bpp 0.75 bpp JPEG :2:0 JPEG :4:4 JPEG 2 1 JPEG XR MS JPEG XR PS JPEG :2:0 JPEG :4:4 JPEG JPEG XR MS 1.00 bpp JPEG XR PS JPEG :2:0 JPEG :4:4 JPEG JPEG XR MS 1.25 bpp JPEG XR PS JPEG :2:0 JPEG :4:4 JPEG JPEG XR MS 1.50 bpp JPEG XR PS 0 Number of times H 0 is rejected Z. Li Multimedia Communciation, 2016 Spring p.31

32 About the Project Sign Up Outline QoE QoS and QoE Def and Standards Body Subjective QoE evaluation MOS Objective QoE metrics Perceptual QoE metrics Summary Z. Li, Multimedia Communciation, 2016 Spring p.32

33 Objective QoE metrics Subjective tests are time consuming, expensive, and difficult to design Objective algorithms, i.e. metrics, estimating subjective MOS with high level of correlation are desired Full reference metrics Input/Reference signal signal processing Output/Processed signal FR METRIC No reference metrics Input/Reference signal signal processing Output/Processed signal Reduced reference metrics Input/Reference signal signal processing Output/Processed signal Features extraction RR METRIC

34 PSNR - Peak Signal to Noise Ratio PSNR def: PSNR 10log 10 B (2 1) MSE 2 where: MSE 1 MN M N y1 x1 [Im a (x,y) Im b (x, y)] 2 M, N = image dimensions Im a, Im b = pictures to compare B= bit depth Widely used because of its simplicity and ease in formalizing optimization problems! For image and video data (Y component), a correlation of circa 80% reported when compared to subjective MOS evaluation

35 PSNR for color images/video Multiple channel info, several options to compute metric Weighted PSNR WPSNR = w 1 PSNR 1 + w 2 PSNR 2 + w 3 PSNR 3 Weigthed MSE: WPSNR_MSE 10log 10 (w MSE 1 1 (2 1) w MSE 2 B 2 2 w 3 MSE 3 ) Weighted Pixel Value PSNR: WPSNR_PIX 10log 10 1 MN M N w 1Ima1(x,y) w2 Ima2(x,y) w3 Ima 3(x,y) w1imb 1(x,y) w2 Imb2(x,y) w3 Imb3 (x,y) y1 x1 ( 2 B 1) 2 2

36 PSNR in RGB vs YCbCr PSNR (db) on R component: PSNR for color images/video PSNR (db) on G component: on B component: PSNR(dB) bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) on Y component: on Cb component: on Cr component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)

37 Standard IQA Model: Error Visibility Reference signal Distorted signal Preprocessing Channel Decomposition... Error Normalization... Error Pooling Quality/ Distortion Measure Motivation Simulate relevant early HVS components E l k 1/ e l, k Key features Channel decomposition linear frequency/orientation transforms Frequency weighting contrast sensitivity function Masking intra/inter channel interaction No separation of objects and illuminance! Z. Li Multimedia Communciation, 2016 Spring p.37

38 Structural Similarity New Paradigm Philosophy Purpose of human vision: extract structural information HVS is highly adapted for this purpose Estimate structural information change Classical philosophy Bottom-up Predict Error Visibility New philosophy Top-down Predict Structural Distortion How to define structural information? How to separate structural/nonstructural information? Z. Li Multimedia Communciation, 2016 Spring p.38

39 Structural Similarity Measurement The SSIM system: Image is a product of illuminance and object reflectance Try to separate the object structural info from the illuminance Full reference solution, compare image block x with y, have 3 components: o Luminance, contrast, and structure comparison [1] (, ) [2] [3] (, ) (, ) Z. Li Multimedia Communciation, 2016 Spring p.39

40 Luminance Comparison l(x,y) Basic Operations Operate on image regions (can be block, or circular) For each channel, compute the region mean and variance for block x and its reference y: = 1 = 1 Luminance comparison function: L=dynamic range, 2 B, e.g, 256 for B=8 K 1, small const << 1., = ( ) Z. Li Multimedia Communciation, 2016 Spring p.40

41 Contrast Comparison c(x,y) Compare the illuminance dynamic range and behavior of two blocks Based on the variance of the channel = 1 1 / = 1 1 / Contrast function:, = Z. Li Multimedia Communciation, 2016 Spring p.41

42 Structural Comparison s(x,y) Luminance subtraction and variance normalized comparison, supposedly removed illuminance factor, compare objects:, = + + = 1 1 ( )( ) Z. Li Multimedia Communciation, 2016 Spring p.42

43 SSIM Index Structural Similarity Measure (SSIM) General factorized form of power a, b, c:, = [,,, ] Typically used: a=b=c=1, =, let Then: = =, = (2 + )(2 + ) ( + + )( + + ) Z. Li Multimedia Communciation, 2016 Spring p.43

44 Structural Similarity (SSIM) Index in Image Space i k j x x i + x j + x k = 0 x - x O luminance change contrast change structural change x i = x j = x k ), ( ), ( ), ( ), ( y x y x y x y x s c l SSIM ), ( C C l y x y x y x ), ( C C c y x y x y x 3 3 ), ( C C s y x xy y x [Wang & Bovik, IEEE Signal Processing Letters, 02] [Wang et al., IEEE Trans. Image Processing, 04] Z. Li Multimedia Communciation, 2016 Spring p.44

45 JPEG2000 compressed image original image SSIM index map absolute error map

46 Gaussian noise corrupted image original image SSIM index map absolute error map

47 JPEG compressed image original image SSIM index map absolute error map

48 Demo Images MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.989 MSE=215, MSSIM=0.671 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723

49 Validation with MOS Scores SSIM is a better predictor than PSNR Dataset JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur Error # of images PSNR SSIM JPEG images JPEG2000 images Fitting with Logistic Function JPEG images JPEG2000 images Fitting with Logistic Function MOS 50 MOS PSNR MOS(PSNR) MSSIM (Gaussian window, K1 = 0.01, K2 = 0.03) MOS(MSSIM) Z. Li Multimedia Communciation, 2016 Spring p.49

50 Summary QoE is an important component in the multimedia communication system Subjective QoE study: User study generate MOS scores Objective Metrics: compare communicated content as pieces of signals Perceptive Metrics: try to model HVS and have a better approximation of MOS Next Class: Reduced Reference, Non-Reference QoE metrics Z. Li Multimedia Communciation, 2016 Spring p.50