Maximum Differentiation Competition: Direct Comparison of Discriminability Models

Transcription

1 Maximum Differentiation Competition: Direct Comparison of Discriminability Models Zhou Wang & Eero P. Simoncelli Howard Hughes Medical Institute, Center for Neural Science, and Courant Institute for Mathematical Sciences New York University

2 reference Image Quality Assessment distorted Which model best accounts for perceived image quality?

3 reference Image Quality Assessment distorted Which model best accounts for perceived image quality?

4 reference Image Quality Assessment MSE distorted SSIM Which model best accounts for perceived image quality?

5 MSE: Mean Squared Error Example Models E(X, Y) = 1 N (x i y i ) 2 i SSIM: Structural Similarity [Wang, et. al. 04] local cross-correlation measure: s(x, y) = (2µ xµ y + C 1 )(2σ xy + C 2 ) (µ 2 x + µ 2 y + C 1 )(σ 2 x + σ 2 y + C 2 ) pooling S(X, Y) = i w(x i, y i )s(x i, y i ) i w(x i, y i ) where w(x, y) = log 2 (1 + σ 2 x/c) + log 2 (1 + σ 2 y/c)

6 Conventional Method Procedure 1. Choose set of reference and distorted images 2. Perform subjective tests 3. Compare model prediction with subjective responses Difficulties Subjective experiments expensive Curse of dimensionality : impossible to cover image space

7 Conventional Method: MSE vs. SSIM Mean Subject Rating Mean Subject Rating -log(mse) SSIM Distortion: JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur Error # images: LIVE image database, UT Austin MSE SSIM

8 Conventional Method: MSE vs. SSIM Mean Subject Rating Mean Subject Rating -log(mse) SSIM Distortion: JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur Error # images: MSE SSIM

9 Proposed Method: MAximum Differentiation (MAD) Competition

10 Proposed Method: MAximum Differentiation (MAD) Competition Let two models compete

11 Proposed Method: MAximum Differentiation (MAD) Competition Let two models compete... by synthesizing optimal stimuli

12 Proposed Method: MAximum Differentiation (MAD) Competition Let two models compete... by synthesizing optimal stimuli... that maximally differentiate the models

13 Geometric Description in Image Space

14 Geometric Description in Image Space all images with same MSE

15 Geometric Description in Image Space reference image all images with same SSIM

16 Geometric Description in Image Space worst MSE reference image

17 Geometric Description in Image Space worst MSE reference image best MSE

18 Geometric Description in Image Space best SSIM

19 Geometric Description in Image Space worst SSIM

20 Geometric Description in Image Space reference image

21 MAD Competition: MSE vs. SSIM reference add noise

22 MAD Competition: MSE vs. SSIM reference best SSIM worst SSIM

23 MAD Competition: MSE vs. SSIM reference best MSE worst MSE

24 MAD Competition: MSE vs. SSIM best SSIM reference best MSE worst MSE worst SSIM

25 2AFC Experiment distortion level (MSE) initial image best SSIM worst SSIM Subjects: 5 (4 naïve, 1 author) Images: 10 reference, viewed at 16 pixels/degree Trials: 20 per distortion-level per subject

26 2AFC Experiment distortion level (MSE) initial image best SSIM worst SSIM Subjects: 5 (4 naïve, 1 author) Images: 10 reference, viewed at 16 pixels/degree Trials: 20 per distortion-level per subject

27 Psychometric Functions % correct best/worst SSIM best/worst MSE initial distortion level (MSE)

28 Psychometric Functions % correct best/worst SSIM all 5 subjects chose top best/worst MSE initial distortion level (MSE) 1 chose top twice 2 chose bottom twice 2 gave 1-1 tie

29 Summary MAximum Differentiation (MAD) Competition Let two models compete... by synthesizing optimal stimuli... that maximally differentiate the models Advantages Optimized images maximize opportunity for model failure Efficient (minimal # of 2-alternative comparisons) Images reveal model weaknesses => potential improvements To Do Full experiment, with more reference images Application to other discriminable quantities Physiology