Input. Output. Problem Definition. Rectified stereo image pair All correspondences lie in same scan lines

Transcription

1 Problem Definition 3 Input Rectified stereo image pair All correspondences lie in same scan lines Output Disparity map of the reference view Foreground: large disparity Background: small disparity

2 Matching Cost Volume 2 C x, y, d denotes the matching cost of pixel (x,y) at different disparity level d

3 WTA (Winner Takes All) 3 C x, y, d denotes the matching cost of pixel (x,y) at different disparity level d Select d with lowest cost as final disparity

4 Some Milestone Approaches 4 Graph Cut (Energy Minimization via Graph Cuts) Boykov et al., ICCV 1999 ASW (Cost Aggregation by adaptive support weight) Yoon and Kweon, CVPR 2005 SGM (Semi-Global Matching) Hirschmuller, CVPR 2005, PAMI 2008 PatchMatch Stereo (Cost aggregation using slanted support windows) Bleyer et al., BMVC 2011

5 Graph Cut 5 Frame the problem as an energy minimization on a multilabeled MRF Solve the MRF by Graph Cut Unary Cost Photo consistency of each label Pairwise Cost Penalize disparity difference between neighboring pixels

6 Graph Cut 6 Frame the problem as an energy minimization on a multilabeled MRF Solve the MRF by Graph Cut

7 Traditional Local Methods 7 C x, y, d Cost aggregation Bilateral filter each C(x, y, i) C A x, y, d

8 ASW (Adaptive Support Weights) 8 Given an initial matching cost volume, Refine the volume by aggregating cost locally and adaptively

9 ASW (Adaptive Support Weights) 9 Given an initial matching cost volume, Refine the volume by aggregating cost locally and adaptively

10 SGM (Semi-Global Matching) 10 Instead of aggregating cost at a local window, SGM Aggregate cost in paths

11 SGM (Semi-Global Matching) 11 Instead of aggregating cost at a local window, SGM Aggregate cost in paths

12 PatchMatch Stereo 12 Parametrize each pixel as a disparity plane Aggregate cost in the slanted window induced by the plane Too many (infinite) possible states, solve by PatchMatch

13 PatchMatch Stereo 13 Parametrize each pixel as a disparity plane Aggregate cost in the slanted window induced by the plane Too many (infinite) possible states, solve by PatchMatch

14 MeshStereo: A Global Stereo Model with Mesh Alignment Regularization for View Interpolation Chi Zhang, Zhiwei Li, Yanhua Cheng, Rui Cai, Hongyang Chao, Yong Rui Presented by Chi Zhang Dec. 15 th, 2015

15 Motivation 2 Goal Output high-quality mesh for view interpolation Motivation Depth estimation and mesh generation are separated in traditional approach, which is not optimal in terms of rendering We aim at unifying such separation, and develop an integrated stereo approach for view interpolation

16 Movitation 3 Traditional Pipeline I L, I R -> Point Clouds (Disparity Maps) Point Clouds -> Mesh Mesh -> New View Angles Ours I L, I R -> Mesh Mesh -> New View Angles

17 Formulation 17 Mesh Representation Delauney triangulated SLIC Segmentation Assign a depth value to each vertex Lifting the 2D triangulation to 3D naturally generate a mesh Technical Difficulty How to split vertices into multiple copies at depth discontinuities Solution Assign a splitting probability to each vertex

18 Formulation 18 Parameterization A Splitting probability for each 2D vertex denoted by α A depth value for each triangle s barycenter and a normal for each triangle denoted by N,D

19 Formulation 19 Objective Function Objective function is a two-layered MRF glued by an Alignment energy term Lower Layer MRF Gluer Upper Layer MRF

20 Formulation 20 The Lower Layer The lower layer MRF is on the dual grid of the 2D triangulation

21 Formulation 21 The Lower Layer Favorites photo-consistent triangles

22 Formulation 22 The Lower Layer Encourages normal smoothness. Encouraged Discouraged

23 Formulation 23 The Upper Layer The upper layer MRF is on the original grid of the 2D triangulation

24 Formulation 24 The Upper Layer Favorite non-split vertices on homogeneous regions

25 Formulation 25 The Upper Layer Similar visual complexity Encourages similar splitting properties when adjacent vertices share similar visual complexity Non-similar visual complexity

26 Formulation 26 The Upper Layer Encourages similar splitting properties when adjacent vertices share similar visual complexity

27 Formulation 27 The Gluer Enforce strong alignment or split a 2D vertex to multiple copies in 3D according to corresponding splitting probability

28 Optimization 28 Objective Function Optimization Iterative Gradient Descent in the blue part and the orange part Fix N, D, minimize the orange part w.r.t. α in closed form Fix α, minimize the blue part by PatchMatch with iterative relaxation (detail at next page)

29 Optimization 29 The orange part sub-energy Optimization Fix α, minimize E LOWER by PatchMatch with iterative relaxation When θ goes to, minimizing E RELAXED is equivalent to minizing E LOWER

30 Optimization 30 The orange part sub-energy Optimization Fix α, minimize E LOWER by PatchMatch with iterative relaxation Minimize by PatchMatch Minimize in closed form

31 Results 31 Stereo Results on Herodion Dataset

32 Results 32 Ranking on Midd3 benchmark

33 Results 33 Examples of generated meshes

34 Results 34 Some synthesized views

35 Results 35 Some synthesized views

36 Conclusion 18 We proposed an integrated stereo model for view interpolation Take I L, I R as inputs, produce a mesh directly It achieves state-of-the-arts performance on both stereo quality and rendering

37 Thank you! The End