Semantic 3D Reconstruction of Heads Supplementary Material

Transcription

1 Semantic 3D Reconstruction of Heads Supplementary Material Fabio Maninchedda1, Christian Ha ne2,?, Bastien Jacquet3,?, Amae l Delaunoy?, Marc Pollefeys1,4 1 ETH Zurich 2 UC Berkeley 3 Kitware SAS 4 Microsoft Fig. 1: From left to right: Input image; Input labels and depth; Depth map fusion (TV-Flux fusion from [9]); Statistical model of [7] fitted into our raw input data; Our semantic reconstruction; Our result skin class only; Our model textured.? Work done while authors were at the Department of Computer Science, ETH Zu rich

2 2 F. Maninchedda, C. Häne, B. Jacquet, A. Delaunoy, M. Pollefeys 1 Additional Renderings and Visualizations Fig. 2: From left to right: Close-up of input image; Close-up of baseline using VisualSFM [8] + PMVS [2] + PSR [5]; Close-up of depth map fusion (TV-Flux fusion from [9]); Close-up of statistical face model of [7] fitted into our raw input data (depth maps + semantic labels); Close-up of our result (all classes merged). In this section we show results on two additional datasets and more comparisons as indicated in Sec. 5 of the main paper. For completeness in Fig. 1 we show all of our results, including those shown in the main paper. To point out methods that make use of prior information for the skin class we have used skin colored models. To facilitate a fair comparison of the reconstruction quality we have rendered the output of the different methods and our result in gray in Fig. 3. The surfaces of the models computed with the TV-Flux fusion approach [9] look a bit rough on some models. This is because there is a trade-off between smoothness and the amount of reconstructed geometry. If the smoothness is too strong the models shrink considerably due to the well known shrinking bias of methods penalizing the surface area. This is especially true for models where we have only captured images of the front of the head. We believe that Fig. 3 and Fig. 2 clearly show that the shape prior formulation that our method uses is effective at removing small artifacts present in the baselines that do not use any prior information while preserving details that are explained strongly by the data. The rather low quality of the reconstruction that we achieve using patch-based multi-view stereo (PMVS) [2] with subsequent Poission surface reconstruction (PSR) [5] is due to

3 Semantic Reconstruction of Heads Supplementary Material 3 the fact that the input data is captured in an uncontrolled environment without any special camera setup. The results that we achieve are very similar in quailty to the results presented in [6]. This method is very similar to PMVS and data which is used for their results on faces, is closer to the kind of data that we used than what is more commonly used for getting high quality results using PMVS and PSR. Fig. 3: From left to right: Input image; Baseline using VisualSFM [8] + PMVS [2] + PSR [5]; Baseline depth map fusion (TV-Flux fusion from [9]); Statistical face model of [7] fitted into our raw input data (depth maps + semantic labels); Our result (all classes merged).

4 4 F. Maninchedda, C. Häne, B. Jacquet, A. Delaunoy, M. Pollefeys Fig. 4: Some example 3D prints (all classes merged and skin class only). Due to the volumetric nature of our reconstructions they can naturally be 3D printed. In Fig. 4 we show some example printouts. 2 Object Shape Prior Comparison As indicated in Sec. 3.3 of the main paper we show in Fig. 5 that the proposed Wulff shape approximation is effective at removing artifacts caused by the discretization of the directions. This is due to the fact that we are directly fitting the surrogate Wulff shapes into the original training data and do not need to discretize the directions. This allows us to achieve a much smoother reconstruction of the unobserved surface between the hair and the skin on the top of the head and removes some of the edginess around the jaw bone where often the data is not very strong. At this point we also want to make two remarks about the used formulations: Despite the fact that some normal directions are discretized in this shape prior formulation it is fundamentally different form a purely discrete graphbased formulation. In a discrete graph-based formulation, in order to represent arbitrary directions, connections between not directly neighboring nodes need to be included. In the continuously inspired shape prior formulation that we are using, half spaces with arbitrary direction can be included in the Wulff shape even tough only directly neighboring voxels are linked with constraints. This is thanks to the fact that in the continuously inspired formulation the connectivity is introduced into the formulation as a discretization of the gradient operator which is independent of the surface direction. When using the original shape prior formulation of [3] the surface between the hair and the skin which is not observed gets reconstructed as facets which only have a discrete set of surface normals. This is due to the representation

5 Semantic Reconstruction of Heads Supplementary Material 5 of the Wulff shapes as polyhedra. Already in [1] it is pointed out for the 2D case that polygonal Wulff shapes lead to piecewise linear boundaries. 1 cluster 2 clusters 3 clusters General Wulff shape Fig. 5: (Left) Slice through the proposed volumetric shape prior (see main paper for more details). (Middle) Reconstruction that uses only general Wulff shapes as proposed in [3]. (Right) Proposed approach. 3 Data Term Comparison As indicated in Sec. 3 of the main paper, using the data term from [4] is not a viable choice in the presence of thin layers of semantic classes. For this experiment we collapsed the beard, eyebrow and hair label to a single hair label. Fig. 6 shows that using the data term originally proposed in [4] does not lead to satisfactory results in the presence of thin layers of semantic classes such as the facial hair on top of the skin. Fig. 6: Comparison between our proposed data term and the one from [4]: Left to right, data term from [4] with all labels and just skin label, our proposed data term with all labels and just skin label. 4 Additional Alignment Experiment To illustrate the importance of the optimization with respect to the alignment we also conducted an experiment where we omitted this part of the optimization. Therefore, the alignment is purely based on the initial landmark based registration. A comparison between with and without optimization with respect

6 6 F. Maninchedda, C. Häne, B. Jacquet, A. Delaunoy, M. Pollefeys to the alignment is shown in Fig. 7. It is clearly visible that the optimization with respect to the alignement has a siginificant impact on the reconstruction quality. Fig. 7: From left to right: Semantic model without alignment; Skin class only without alignment; Aligned semantic model; Skin class only for aligned model. References 1. Esedoglu, S., Osher, S.J.: Decomposition of images by the anisotropic rudin-osherfatemi model. Communications on pure and applied mathematics (2004) 2. Furukawa, Y., Ponce, J.: Accurate, dense, and robust multi-view stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2010) 3. Häne, C., Savinov, N., Pollefeys, M.: Class specific 3d object shape priors using surface normals. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014) 4. Häne, C., Zach, C., Cohen, A., Angst, R., Pollefeys, M.: Joint 3d scene reconstruction and class segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013) 5. Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Eurographics symposium on Geometry processing (SGP) (2006) 6. Lhuillier, M., Quan, L.: A quasi-dense approach to surface reconstruction from uncalibrated images. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2005) 7. Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3d face model for pose and illumination invariant face recognition (2009) 8. Wu, C.: Visualsfm: A visual structure from motion system. (2011) 9. Zach, C.: Fast and high quality fusion of depth maps. In: International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT) (2008)