Analysis and Synthesis of 3D Shape Families via Deep Learned Generative Models of Surfaces

Transcription

1 Analysis and Synthesis of 3D Shape Families via Deep Learned Generative Models of Surfaces Haibin Huang, Evangelos Kalogerakis, Benjamin Marlin University of Massachusetts Amherst

2 Given an input 3D shape family

3 Goal 1/4: part segmentation & labeling wing horizontal stabilizer vertical stabilizer fuselage Applications: assembly based modeling, manufacturing

4 Goal 2/4: correspondences stabilizer top tip fuselage front tip left wing tip right wing tip Applications: texture transfer, exploration, morphing

5 Goal 3/4: learn shape descriptors & fine grained classification Commercial jets Propeller airplanes Military jets Applications: text based shape retrieval, database organization

6 Goal 4/4: shape synthesis Applications: virtual worlds with lots of (infinite?) content variations, training data for vision algorithms

7 Challenges Shape families can have large structural & geometric variability. All tasks depend on each other. Generate surface geometry.

8 Related work: modeling structural & geometric variability Box part templates [Kim et al. 13] Non rigid alignment / distance learning [Huang et al. 13] Probabilistic model for shape synthesis [Kalogerakis et al. 12]

9 Related Work: deep learning Deep representations for volumetric shapes [Wu et al. 15]

10 Joint Analysis and Synthesis Probabilistic Model for analysis: Learns template geometry and deformations Estimates fuzzy point correspondences Segments and labels parts Deep Probabilistic Model for synthesis: Generates point sampled surfaces Learns class specific descriptors Combine both models and jointly learn them from input collection

11 Probabilistic model Given the input shapes, estimate assignments to random variables representing template geometry Learned templates Input collection

12 Probabilistic model Given the input shapes, estimate assignments to random variables representing template geometry, template deformations for each shape Input collection Non rigid deformation [video]

13 Probabilistic model Given the input shapes, estimate assignments to random variables representing template geometry, template deformations for each shape, surface part labels and point labels for each shape Input collection Segmented & labeled shapes

14 Probabilistic model Given the input shapes, estimate assignments to random variables representing template geometry, template deformations for each shape, surface part labels and point labels for each shape, and descriptors for each shape. Input collection Descriptors

15 Probabilistic model Given the input shapes, estimate assignments to random variables representing template geometry, template deformations for each shape, surface part labels and point labels for each shape, and descriptors for each shape. Maximize: P crf (Y, D, S, U X) P bsm (D, H)

16 Probabilistic model Given the input shapes, estimate assignments to random variables representing template geometry, template deformations for each shape, surface part labels and point labels for each shape, and descriptors for each shape. Maximize: P crf (Y, D, S, U X) P bsm (D, H) Probabilistic model for analysis: how likely are the shape & template attributes given each input surface geometry?

17 Probabilistic model Given the input shapes, estimate assignments to random variables representing template geometry, template deformations for each shape, surface part labels and point labels for each shape, and descriptors for each shape. Maximize: P crf (Y, D, S, U X) P bsm (D, H) Generative probabilistic model: How likely is the output deformed geometry? Ensures consistency within the input collection

18 Probabilistic model Given the input shapes, estimate assignments to random variables representing template geometry, template deformations for each shape, surface part labels and point labels for each shape, and descriptors for each shape. Maximize: P crf (Y, D, S, U X) P bsm (D, H) New Collection Generative probabilistic model: How likely is the output deformed geometry? Ensures consistency within the input collection

19 Input shape representation Each input surface is uniformly sampled:

20 Surface random variables For each surface point p of shape t with position X t,p : Part label: S t,p {fuselage, left wing, right wing } {S t,3,u t,3 } {S t,4,u t,4 } X 4 X 3 {S t,1,u t,1 } X 1 {S t,2,u t,2 } X 2 Point label: U t,p {left wing tip, front fuselage tip } Assignments are probabilistic ( fuzzy correspondences).

21 Y 3 Y 1 Y 2 D t,1 D t,2 D t,3 Template Random variables For each template point k: Template point position: Y k {point position in 3D space} Deformed template point position for each input shape t: D t,k {point position in 3D space} Template geometry will be estimated.

22 Factors All these random variables are not independent of each other! The probabilistic model is defined through factors over variable assignments. The higher the factor values, the more compatible the assignments are.

23 Unary deformation factor Encourage template points deform towards corresponding, closeby surface points. D t,1 D t,2 X t,1 X t,2 Learned template surface D t,3 Deformed template surface X t,3 Input shape surface t φ 1 (D t,k, U t,p =k, Χ t,p )=exp{-.5[d t,k X t,p ] T Σ 1-1 [D t,k X t,p ]} Learned covariance matrix

24 Deformation smoothness factor Encourages smoothness in the deformations of the template parts. Y 2 Y 1 D t,2 D t,1 X t,1 Y 3 Learned template surface D t,3 Deformed template surface X t,2 X t,3 Input shape surface t φ 2 (D t,k,d t,k,y k,y k )=exp{-.5[(d t,k -D t,k ) - (Y k -Y k )] T Σ 2-1 [(D t,k -D t,k ) - (Y k -Y k )]} Learned covariance matrix

25 Correspondence factor Encourage correspondences between template and surface points with similar geometry SDF=0.9 Y 1 D t,1 X t,1 SDF=0.9 SDF=0.5 Y 2 D t,2 X t,2 SDF=0.4 SDF=0.1 Y 3 Learned template surface D t,3 Deformed template surface X t,3 SDF=0.2 Input shape surface t φ 3 (U t,p = k)=exp{-.5[f k - f t,p ] T Σ 3-1 [f k - f t,p ]} Learned covariance matrix

26 Segmentation factor Encourage consistent part labels between corresponding template and surface points Y 1 S t,1 Y 2 S t,2 Y 3 Learned template surface S t,3 Input shape surface t

27 Segmentation smoothness factor Encourage surface points within convex patches and similar geometry to have similar part label. S t,1 S t,2 [Asafi et al. 2013] S t,3 Input shape surface t

28 H 1 (3) H 2 (3) Generative model H 2 (2) H 1 (2) H 3 (2) H (1) H (1) 2 H (1) 1 3 H (1) 4 D t,1 D t,2 D t,3 D t,4 D t,5 D t,6 wing point tailplane point positions positions

29 Generative model D t,1 D t,2 D t,3 D t,4 D t,5 D t,6 wing point tailplane point positions positions

30 Generative model H 1 (1) H 2 (1) H 3 (1) H 4 (1) D t,1 D t,2 D t,3 D t,4 D t,5 D t,6 wing point tailplane point positions positions delta shaped wing straight wing

31 Generative model H 1 (1) H 2 (1) H 3 (1) H 4 (1) D t,1 D t,2 D t,3 D t,4 D t,5 D t,6 wing point tailplane point positions positions swept tailplanes straight tailplanes

32 Generative model H 1 (2) H 2 (2) H 3 (2) H (1) H (1) 2 H (1) 1 3 H (1) 4 correlations between part configurations D t,1 D t,2 D t,3 D t,4 D t,5 D t,6 wing point tailplane point positions positions

33 H 1 (3) H 2 (3) Generative model {0.9, 0.1 } {0.7, 0.2 } {0.1, 0.0 } {0.1, 0.2. } vs H 2 (2) H 1 (2) H 3 (2) shape descriptor H (1) H (1) 2 H (1) 1 3 H (1) 4 D t,1 D t,2 D t,3 D t,4 D t,5 D t,6 wing point tailplane point positions positions

34 G 1 (3) G 2 (3) Generative model G 2 (2) G 1 (2) G 3 (2) G (1) G (1) 2 G (1) 1 3 G (1) 4 E t,1 E t,2 E t,3 E t,4 E t,5 E t,6 wing point existence tailplane point existence

35 Inference / parameter learning Parameter learning: Piecewise training for covariance matrices of the analysis part (12 parameters) Contrastive divergence for the generative model parameters (millions) Mean field inference: Efficient compared to other forms of inference Convergence properties Approximate Requires initialization (template parts from co segmented shapes we use [Kim et al., 2013] to initialize the segmentations)

36 Deformation examples Template surface Input surface

37 Deformation examples Template surface Input surface [video]

38 Deformation examples Template surface Input surface

39 Deformation examples Template surface Input surface [video]

40 Sampled shapes

41 Sampled shapes

42 Sampled shapes

43 Assembly based modeling + deformation (Embedded deformation for shape manipulation, Sumner et al., 2007)

44 Correspondence quality BHCP dataset 18.2% more correct predictions within.05 distance than Kim et al. s 2013 method. % Correspondences Euclidean Distance

45 Segmentation quality

46 MTurk User study: plausibility of synthesized shapes

47 Summary Joint segmentation, correspondence, non rigid deformation, descriptor learning, shape synthesis Analysis by synthesis Deep learning for modeling shape surfaces

48 Future Work Full shape synthesis from scratch without assembly Reconstruction, link descriptor to other forms of user input More robust and automatic initialization schemes What is the best input shape representation?

49 Thank you! Acknowledgements: Qi xing Huang, Vladimir Kim, Siddhartha Chaudhuri Szymon Rusinkiewicz, anonymous reviewers Our project web page (code, data, supp. material):