Natural Scene Sta,s,cs of Color and Range. Che- Chun Su, Lawrence K. Cormack, and Alan C. Bovik

Transcription

1 Natural Scene Sta,s,cs of Color and Range Che- Chun Su, Lawrence K. Cormack, and Alan C. Bovik

2 Mo,va,on Color and range/depth play important roles in natural scenes and human vision systems. Percep,on of color and range/depth are related 1,2, but their joint sta,s,cs are unknown. Sta,s,cal rela,onships between color and range/depth could be used to help understand how humans perceive 3D improve image/video algorithms 1 Jordan et al., Color as a source of information in the stereo correspondence process, Vision Research, Dec Jordan and Bovik, Using chromatic information in dense stereo correspondence, Pattern Recognition, Apr

3 Contribu,ons/Findings LIVE (Laboratory for Image and Video Engineering) Color+3D database Sta,s,cal rela,onships and models between luminance/chrominance and range/disparity in natural scenes Improvement on computa,onal stereo algorithms Posit that human vision systems could use these rela,onships in 3D percep,on. 3

4 Data Acquisi,on - LIVE Color+3D Database RIEGL- VZ400 laser scanner mounted with Nikon D700 digital camera 1.5 m 600 m Ver,cal: 100 o (- 40 o 100 o ) Horizontal: 360 o Angular step- width: o Time: 2 min for 60 o wide with 0.04 o step- width 4

5 Data Acquisi,on - LIVE Color+3D Database RIEGL- VZ400 laser scanner mounted with Nikon D700 digital camera 5

6 Data Acquisi,on - LIVE Color+3D Database 12 sets of co- registered color images with ground- truth range maps High- defini,on resolu,on of 1280x720, 60o- by o field of view Color Image Range Map 6

7 Analysis Transform RGB color images into a more perceptually relevant CIELAB color space. 7

8 Analysis Convert ground- truth range maps into disparity maps. 8

9 Analysis 9

10 Analysis Marginal sta,s,cs and distribu,on L*, a*, b* responses Disparity responses Condi,onal sta,s,cs and distribu,on L*, a*, b* responses condi,oned on disparity responses 10

11 Condi,onal Distribu,on of Luminance Response on Disparity Response L* channel frequency = 5.87 (cycle/degree), orienta,on = horizontal (0 degree) Blue line: true distribu,on, red- dofed line: fifed Weibull distribu,on 11

12 Plot of Weibull Parameters Fihng the Condi,onal Distribu,on 12

13 Condi,onal Distribu,on of Chrominance Response on Disparity Response a* channel frequency = 5.87 (cycle/degree), orienta,on = horizontal (0 degree) Blue line: true distribu,on, red- dofed line: fifed Weibull distribu,on 13

14 Plot of Weibull Parameters Fihng the Condi,onal Distribu,on 14

15 Condi,onal Distribu,on of Chrominance Response on Disparity Response b* channel frequency = 5.87 (cycle/degree), orienta,on = horizontal (0 degree) Blue line: true distribu,on, red- dofed line: fifed Weibull distribu,on 15

16 Plot of Weibull Parameters Fihng the Condi,onal Distribu,on 16

17 Applica,on and Demonstra,on Improvement on computa,onal stereo algorithms Applica,on to Bayesian stereo algorithm Co- occurrence of luminance and range edges in natural scenes 17

18 Applica,on to Bayesian Stereo Algorithm Given the lei and right images, compute the disparity map. Formula,on Canonical model left image (L*) right image (L*)! smoothness energy!. photometric energy " simulated annealing disparity map 18

19 Applica,on to Bayesian Stereo Algorithm Given the lei and right images, compute the disparity map. Formula,on Previous NSS (natural scene sta,s,cs) model 1 conditional distribution of disparity given L* left image (L*) right image (L*)! NSS energy!. photometric energy " simulated annealing disparity map 1 Liu et al., Statistical modeling of 3D natural scenes with application to Bayesian stereopsis, Image Processing, IEEE Transaction on, to be published. 19

20 Applica,on to Bayesian Stereo Algorithm Given the lei and right images, compute the disparity map. Formula,on Gabor- based NSS model by this work conditional distribution of L*,a*,b* given disparity marginal distribution of disparity left image (L*) right image (L*)! NSS energy_1 NSS energy_2!. photometric energy " simulated annealing disparity map 20

21 Ground- truth: Tsukuba 21

22 Visual Comparison Canonical Previous NSS Gabor- based NSS 22

23 Ground- truth: Venus 23

24 Visual Comparison Canonical Previous NSS Gabor- based NSS 24

25 Numerical Comparison "Tsukuba" Canonical Previous NSS Gabor- based NSS % all non- occlusion textured Metric (bad- pixel- rate) 25

26 Numerical Comparison "Venus" Canonical Previous NSS Gabor- based NSS 9.79 % all non- occlusion textured Metric (bad- pixel- rate) 26

27 Visual Comparison between with/without Chrominance Informa,on Only Luminance Only Chrominance Both 27

28 Visual Comparison between with/without Chrominance Informa,on Only Luminance Only Chrominance Both 28

29 Numerical Comparison between with/ without Chrominance Informa,on "Tsukuba" only a*b* only L* both L* and a*b* % all non- occlusion textured Metric (bad- pixel- rate) 29

30 Numerical Comparison between with/ without Chrominance Informa,on "Venus" only a*b* only L* both L* and a*b* % all non- occlusion textured Metric (bad- pixel- rate) 30

31 Co- occurrence of Luminance and Range Edges Given the prior informa,on that there is a luminance edge, the probability of finding a range edge increases. Sampling distribu,on of uncondi,onal and condi,onal probability of range edges 31

32 Conclusion We built the LIVE 3D+Color database with high- defini,on resolu,on color images and co- registered range maps. The sta,s,cal models between luminance/chrominance and range/disparity in natural scenes are developed, and we demonstrated that they are helpful in understanding human 3D percep,on. useful in applica,on to Bayesian stereo algorithms. 32

33 Acknowledgement Thanks to LIVE members for contribu,ng to this work. Applica,on to Bayesian stereo algorithm Yang Liu Co- occurrence of luminance/chrominance and range edges Anish Mifal Michele Saad This work is funded by Na,onal Science Founda,on (NSF) grant #IIS