Geometric and Semantic 3D Reconstruction: Part 4A: Volumetric Semantic 3D Reconstruction. CVPR 2017 Tutorial Christian Häne UC Berkeley

Transcription

1 Geometric and Semantic 3D Reconstruction: Part 4A: Volumetric Semantic 3D Reconstruction CVPR 2017 Tutorial Christian Häne UC Berkeley

2 Dense Multi-View Reconstruction Goal: 3D Model from Images (Depth Maps)

3 A Standard Pipeline Input Images Structure-from-Motion Sparse Reconstruction Dense Matching Depth Maps

4 A Standard Pipeline Depth Maps Depth Map Fusion Dense 3D Model

5 Challenges Noise in Depth Maps Inconsistencies Between Views Incomplete Data Mistakes in Depth Maps

6 Domain / Representation Volume Outside / Inside Mesh Triangles Representing the Surface Surface Element (Surfel). Dense Set of Small Disks

7 Truncated Signed Distance Field (TSDF) [Curless & Levoy, 1996, Levoy et al. 2000] Weighted Average over Multiple Viewpoints

8 Marching Cubes [Lorensen & Cline 1987] Conversion from Volume to Mesh Extract Mesh as Iso-Surface (Zero-Crossing) [Levoy et al. 2000]

9 Real-Time on GPU [Newcombe et al. 2011] Input: Depth Cameras (Kinect)

10 Regularization Noise and Outliers Energy minimization Data Term Smoothness Term

11 Discrete Domain / Graph Cut [Lempitsky & Boykov, 2007] Label Voxels as Inside/Outside (1 or 0) Energy minimization via Graph Cut Cut Edges -> Smoothness Cost [Lempitsky & Boykov, 2007] Metrication Artifacts

12 Continuous Domain / Variational Segment Continuous Domain Inside/Outside Variational Optimization Total Variation as Smoothness Penalizes Surface Area u(x) = 0 u(x) = 1 [Zach, 2007]

13 Direct Reconstruction Input Images Structure-from-Motion Sparse Reconstruction Photoconsistency, Energy Minimization Dense 3D Model

14 Per Voxel Photoconsistency No explicit Computation of Depth Maps Photo Consistency Evaluated per Voxel Silhouettes / Visual Hull (Object only) [Kolev et al. 2009]

15 Adding Surface Normals [Kolev et al. 2009] Surface Normals for High Frequency Details Estimated Normal Field Guides Reconstruction

16 Formulation over Rays [Liu & Cooper, 2010, 2014] First Transition to Occupied Space Along Ray Color Consistent over all Rays Discrete Graph Based Formulation Alternating Minimization, Belief Propagation

17 Challenging Cases Some Examples Using Depth Map Fusion

18 Adding Semantics Input Images Semantic Classifier Class Likelihoods Sparse Reconstruction Dense Matching Depth Maps

19 Dense Semantic 3D Reconstruction Class Likelihoods Depth Maps Joint Fusion, Convex Optimization Dense Semantic 3D Model

20 Adding Semantics to Geometry [Sengupta, Greveson, Shahrokni, Torr, 2013] Geometry not Improved

21 Dense Semantic 3D Reconstruction [Häne et al. 2013, 2016] Dense 3D Model Dense Semantic 3D Model Dense semantic 3D model takes class-specific surface orientation into account! likely unlikely For example: direction of ground: horizontal more likely than vertical

22 Multi-Label Formulation Discrete Domain Continuous Domain Smoothness: Transitions Along Edges Linear Program Belief Propagation Graph Cuts Smoothness: (anisotropic) boundary length Convex Program Discretized (for Iterative Optimization)

23 Anisotropy / Wulff Shape Wulff Shape [Wulff 1901, Esedoglu and Osher 2004] Convex shape defined by

24 Semantic Reconstruction Formulation [Häne et al. 2013, 2016] Data Term: Described as pervoxel unary potentials Regularization Term: Class-specific, direction dependent, surface area penalization Learned from training data

25 Data Term Sum of Weight in Each Voxel Sky -> Free Space Weight Along Whole Ray

26 Entering Unary Weights Weights for all voxels and all views entered

27 Entering Unary Weights Weights for all voxels and all views entered Model Recovered based on weights

28 Smoothness Term Isotropic + Anisotropic (Wulff Shape) Maximum Likelihood Estimation Grid Search Training Data Two Wulff Shape Parameterizations

29 Energy Evolution [Häne et al. 2013, 2016] First Order Primal-Dual Algorithm [Chambolle & Pock, 2010]

30 Weakly Observed Structures [Häne et al. 2013, 2016] Buildings Standing on the Ground

31 Weakly Observed Structures [Häne et al. 2013, 2016] Building and Vegetation Separated

32 Unobserved Surfaces Labels can be Separated

33 Unobserved Surfaces Labels can be Separated

34 Extension To Octrees [Blaha, Vogel, et al. 2016] Exploiting Sparsity of Surface

35 Using More Classes [Cherabier, Häne, Oswald, Pollefeys, 2016] Exploiting Sparsity of Labels / Transitions Only Relevant Labels Active In Block

36 Using Only Sparse Points [Kundu et al. 2014] Discrete Formulation Message Passing

37 Traditional, Unary Potential Weights for all voxels and all views entered

38 Traditional, Unary Potential Weights for all voxels and all views entered Model Recovered based on weights

39 Issues with Unary Potentials [Savinov, Ladicky, Häne, Pollefeys, 2015, 2016] Data given as Information along rays Approximation with unary weights -> artifacts Artifact Closed Archway Inflated Roofline

40 Ray Potential Costs along rays for placing surface First transition to occupied semantic class

41 Ray Potentials Costs along rays for placing surface Optimization contains per-ray variables

42 Ray Potentials [Savinov et al., 2015, 2016] Data given as Information along rays Keeping information along rays in formulation Correct Archway Correct Roofline

43 Formulation [Savinov, Häne, Ladicky, Pollefeys, 2016] Local View: Describes Visible Surface in Camera Global View: Describes Voxel Labeling Data Term: Described as ray potential Regularization Term: Class-specific, direction dependent, surface area penalization Constraints: Make Local and Global View Consistent

44 Formulation [Savinov, Häne, Ladicky, Pollefeys, 2016] Convex Relaxation Weak Solution: One Non-Convex Constraint Change of Visibility Along Ray <-> Cost Assumed Majorize-Minimize Optimization

45 Results [Savinov, Häne, Ladicky, Pollefeys, 2016] Thin Structures

46 Results [Savinov, Häne, Ladicky, Pollefeys, 2016] Semantic 3D Reconstruction Image Unary Potential Ray Potential

47 Object Shape Priors Real-World Objects Reflective Transparent Specular Exploit Object-Class Specific Similarity Example Input Image Reconstruction without Prior

48 Shape Model Bao, Chandraker, Lin, Savarese, 2013

49 Results [Bao, Chandraker, Lin, Savarese, 2013] Image PMVS PSR with Prior Ground Truth Only Object Reconstructed

50 Normal-Based 3D Object Shape Priors [Häne, Savinov, Pollefeys, 2014] Reconstruction Volume aligned with object Surface normals locally similar between instances

51 Training the Prior [Häne, Savinov, Pollefeys, 2014] Per-voxel surface normal direction distribution Regularization with discretized Wulff shapes

52 Reconstruction Formulation [Häne, Savinov, Pollefeys, 2014] Reconstruction volume aligned Data Term: Described as per-voxel unary potentials Extract per-voxel Wulff shapes Training Data: Mesh models

53 Results Cars [Häne, Savinov, Pollefeys, 2014]

54 Results Cars [Häne, Savinov, Pollefeys, 2014]

55 Alignment Transform [Maninchedda, Häne, et al. 2016]

56 Semantic 3D Reconstruction of Heads [Maninchedda, Häne et al. 2016]

57 Conclusion Semantics Helps to Improve Geometry Consistent Semantic Segmentation Richer Representation Multi-Label Shape Priors Future Directions: Real-Time Applications End to End Learning