Deep 3D Machine Learning for Reconstruction and Repair of 3D Surfaces

Transcription

1 Deep 3D Machine Learning for Reconstruction and Repair of 3D Surfaces TalkID This session will give the audience a quick overview of recent developments in the field of 3D surface analysis with deep learning techniques and an insight into our approach for 3D surface repair.

2 Pascal Laube PhD Student at the Institute for Optical Systems at the HTWG Konstanz government-funded by Main focus: Machine Learning for Surface Reconstruction Defect Detection and Repair (Inpainting) Medical Imaging

3 Representation: The 2D case Output Grid in euclidean space Neural Network (in this case CNN)

4 Representation: In 3D? Point Cloud? Mesh?? Neural Network Any manifold (NURBS, impl. surf., )

5 Representations: Voxels [Vishakh Hegde et al., NIPS (2016)]

6 Representations: Voxels [Zhirong Wu et al., CVPR (2015)]

7 Representations: Multi-View [Hang Su et al., ICCV (2015)]

8 Representations: Multi-View [Liuhao Ge et al., CVPR (2016)]

9 Representations: Graph Signal Processing Graph Laplacian or Laplace Beltrami Operator as f = div( f) Laplacian Eigenfunctions generalize to Fourier bases. [M. Bronstein et al., Sig. Proc. Mag (2017)] Convolution in the spectral domain is defined but filters are base dependent.

10 Representations: Graph Signal Processing Train filters in geodesic polar coordinates. Pool rotation angles [J. Masci et al., ICCV (2015)] Many other methods using different kernels (heat diffusion, gauss )

11 Data Sets 127,915 CAD Models 662 Object Categories Different Subsets 51,300 Models 270 Object Categories in Model Subsets Many smaller specialized Data Sets

12 Problem: Defect on Surface with Detail- and Base-Geometry Fraunhofer IPT

13 Problem: Defect on Surface with Detail- and Base-Geometry Werkzeugbau Siegfried Hofmann GmbH

14 Problem: Defect on Surface with Detail- and Base-Geometry (3) High resolution meshes with > 1m vertices Base Geometry and Relief

15 Our Approach B-Spline Approx. or Base Geometry Approx. by Geometric Primitive or Multiresolut. Surfaces Detail Geometry Heightmap Surface with Defect Seperation Novelty Detection using 1 Base Geo. Detail Geo. 2 3 Autoencoders Multiresolution Neural Nets for Inpainting

16 2 Novelty Detection using Autoencoders Defect unknown Healthy state unknown What do we know? Textures have to be ergodic: Statistical properties are constant for single sample and whole collection

17 2 Novelty Detection using Autoencoders Autoencoder should be unable to sufficiently reconstruct Defects Train Autoencoder on Ergodic Set of Textures

18 Loss government-funded by 2 Novelty Detection using Autoencoders Parallelizable to multiple GPUs Samples

19 3 Multiresolution Neural Nets for Inpainting: Texture Synthesis Activation Network Synth. Network [L. Gatys et al., NIPS (2015)]

20 3 Multiresolution Neural Nets for Inpainting: Style Transfer [L. Gatys et al., arxiv.org (2015)]

21 3 Multiresolution Neural Nets for Inpainting: Example Defect Closeup 2048x2048

22 3 Multiresolution Neural Nets for Inpainting: Patches Inpainting a Region with arbitrary size? Inpaint Patch by Patch Local Style Global Style 2048x2048

23 3 Multiresolution Neural Nets for Inpainting: Results 1. Start 2. Inpaint Patches: Large Parent Weight 3. Inpaint Patches: Apply Detail Large Child Weight Small Parent Weight

24 3 Multiresolution Neural Nets for Inpainting: Results Result Result Closeup

25 3 Multiresolution Neural Nets for Inpainting: Results Heightmap Parallelizable to multiple GPUs

26 3 Multiresolution Neural Nets for Inpainting: Results Surface

27 Outlook Neural Nets in high dimensional irregular domains Michael M. Bronstein et al., Geometric deep learning: going beyond Euclidean data (2017) [M. Bronstein et al., Sig. Proc. Mag (2017)] Michaël Defferrard, Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (2016) [J. Masci et al., ICCV (2015)]

28 Thank You