Reliability Based Cross Trilateral Filtering for Depth Map Refinement

Transcription

1 Reliability Based Cross Trilateral Filtering for Depth Map Refinement Takuya Matsuo, Norishige Fukushima, and Yutaka Ishibashi Graduate School of Engineering, Nagoya Institute of Technology Nagoya , Japan

2 Contents Background Free Viewpoint Image Synthesis Depth Image Based Rendering (DIBR) Stereo Matching Proposed Method Experimental Results Depth Estimation View Synthesis Conclusion

3 Free Viewpoint Image Synthesis Original Image Depth Map Accurate Depth Map Depth Image Based Rendering Free Viewpoint Image High Quality Free Viewpoint Image Getting Depth Map from Stereo Matching

4 Z Z Z Depth Image Based Rendering (1)(4) (2) Warping (5) Blending Filling Inpainting to target the L-R small depth images view-point holes (3) Reverse warping for view holes

5 Stereo Matching (1/3) Advanced approach Optimization - e.g. Semi-Global Block Matching(SGBM) Graph Cuts Problems Indefinite discontinuity Long runtime, because complex algorithm Solution Post processing with filter - Edge accuracy goes up - Short runtime Accuracy of edges is important for Free Viewpoint Image synthesis

6 Stereo Matching (2/3) Advanced approaches Matching cost computation Cost aggregation Disparity computation / optimization Disparity refinement Easy filter e.g. Median filter Gaussian filter

7 Stereo Matching (3/3) Our proposed method Matching cost computation Cost aggregation Disparity computation / optimization Disparity refinement Easy stereo matching e.g. Block matching Some complex filter

8 Proposed Method (1/6) Ideal Correct edge of an original image Smooth flat surfaces Low noises Bilateral filter extended for cross trilateral filter - The inputs are original image and depth map - Go up accurate and keep edge with reliability

9 Proposed Method (2/6) Cross Trilateral Filter The inputs are original image and depth map Three kinds of kernel weight are used - Correcting edge and smoothing flat surface D p = s N s N w p, s c p, s r p, s D s w p, s c p, s r p, s Space weight: w p, s = e 1 2 p s 2 σs Color weight: c p, s = e 1 2 Reliability: r p, s = e Space weight (Deference of distance) - Color weight (Deference of intensity) - Reliability (Deference of disparity) Ip Is 1 σc Dp Ds 1 σr (σ s, σ c, σ r : const. )

10 Proposed Method (3/6) Problem It is potential that noise has large weight, when attention pixel has large noise Solution Define a classification method to decide reliability dynamically - Reduce propagation effect of noisy pixels

11 Proposed Method (4/6) We have two original images and one depth map Left original image Right original image Left depth map

12 Proposed Method (4/6) Left original image Left depth map If pixel and pixel point same object, intensity and disparity are similar

13 Proposed Method (4/6) Left original image Right original image Disparity and view distance is known Left depth map Correspondence pixels in left and right view are obvious

14 Proposed Method (4/6) Left original image and depth map Right original image If disparity is correct, correspondence pixels are similar intensity

15 Proposed Method (5/6) Classification Approach Conditions l: Left view, r: Right view, α, β, γ: Thresholds The reliability r(p, s) is defined as r p, s = e 1 2 D p l D s l I p l I s l I l r s I s+ds l D p D s 1 σ r (meet conditions) 0 (else) Noise of depth maps is not imparted to the reliability as much as possible Depth maps are smoothed while keep object edge α β γ

16 Proposed Method (6/6) Kernel of filter Reliability

17 Experimental Results Depth estimation Competitive these methods about error rate - Block Matching (BM) + Proposed (C-Tri) - BM - Semi-Global Block Matching - BM + Median Filter - BM + Bilateral Filter Using datasets (height width, depth) : tsukuba ( , 16) : venus ( , 32) : teddy ( , 64) : cones ( , 64)

18 Experimental Results Free Viewpoint Image Synthesis Free viewpoint image synthesis with Depth Image Based Rendering Depth maps to be used are acquired by proposed method, BM, and SGBM Synthesis image and original image are compared and evaluated - Peak Signal-to-Noise Ratio (PSNR) - Structural SIMilarity (SSIM) Using data set is teddy : teddy ( )

19 Depth Estimation (1/6) Error Rate BM C-Tri Bi Med SGBM tsukuba venus teddy cones (%) Large improvement is confirmed in tsukuba When number of gradation is high, improvement is low Adding another evaluation item and reevaluate

20 Depth Estimation (1/6) Error Rate BM C-Tri Bi Med SGBM tsukuba venus teddy cones (%) Large improvement is confirmed in tsukuba When number of gradation is high, improvement is low Adding another evaluation item and reevaluate

21 Depth Estimation (1/6) Error Rate BM C-Tri Bi Med SGBM tsukuba venus teddy cones (%) Large improvement is confirmed in tsukuba When number of gradation is high, improvement is low Adding another evaluation item and reevaluate

22 Depth Estimation (1/6) Error Rate BM C-Tri Bi Med SGBM tsukuba venus teddy cones (%) Large improvement is confirmed in tsukuba When number of gradation is high, improvement is low Adding another evaluation item and reevaluate

23 Depth Estimation (2/6) New estimation item is Relative Improvement Rate (RIR) RIR = E BM E C Tri E BM E X : Error rate of method X Additional experiment - Convert the depth ranges which are 16 or 32, and use narrow baseline in teddy data set

24 Depth Estimation (3/6) Relative Improvement Rate C-Tri BM RIR tsukuba venus teddy teddy teddy cones (%)

25 Depth Estimation (3/6) Relative Improvement Rate C-Tri BM RIR tsukuba venus teddy teddy teddy cones (%)

26 Depth Estimation (3/6) Relative Improvement Rate C-Tri BM RIR tsukuba venus teddy teddy teddy cones (%)

27 Depth Estimation (4/6) Before: BM After: Proposed method (BM + C-Tri)

28 Depth Estimation (5/6) BM Proposed

29 Depth Estimation (6/6) Running time until to acquire depth map - Experimental environment: Intel Core i GHz - Kernel size of filter: (13 13) Running Time BM C-Tri Sum SGBM tsukuba venus teddy ms

30 View Synthesis (1/2) PSNR, SSIM Method PSNR (db) SSIM C-Tri BM SGBM BM+C-Tri BM SGBM

31 View Synthesis (2/2) BM Proposed

32 Conclusion Conclusion Depth estimation The accuracy of proposed method is about the same as the SGBM In point of edge keeping, proposed method is more effective than optimized method like the SGBM The BM with proposed method is faster than SGBM in any data sets Free viewpoint image synthesis Proposed method is more effective than optimized method like the SGBM

33 Conclusion Future Works Extend this filter to be independent of the number of gradation of depth map and to improve accurate