3D Object Model Acquisition from Silhouettes

Transcription

1 4th International Symposium on Computing and Multimedia Studies 1 3D Object Model Acquisition from Silhouettes Masaaki Iiyama Koh Kakusho Michihiko Minoh Academic Center for Computing and Media Studies Kyoto University Yoshida Honmachi Sakyo Kyoto JAPAN iiyama@media.kyoto-u.ac.jp Keywords: Shape from Silhouettes Articulate Object Photometric Stereo EM Algorithm Abstract This paper proposes an approach for acquiring a 3D object model from silhouettes. Photometry geometry and motion of the object consist the model. Aquisition of 3D object models requires restrictions as to an object s shape and it also requires prepared object s shape. We show three approaches for removing the restriction and preparation. 1 Introduction This paper proposes an approach for acquiring a 3D object model from silhouettes. The 3D object model is a set of object properties and the properties are necessary for reproducing the object s appearance. The object s appearance is given by various factors: Objects vary their appearance varying their pose at all times. Lighting environments also vary the object s appearance. Object s surface reflects incident lights and the reflection provides the object s appearance. Reflectance properties and surface normals provide the reflection. Viewpoints also vary the object s appearance. When the viewpoint is given object s shape provides the appearance. The object s appearance is given by three properties; photometry modeled by reflection properties geometry modeled by object s shape and motion modeled by a sequential object s pose. A 3D object model with these three properties provides the object s appearance. Reproducing the object s appearance makes an important role in various applications. The applications include teleconference virtual museums and rapid prototyping. Traditional teleconference systems observe a scene by cameras and transmit the camera images as they are. Requirement of the teleconference is realistic image but the camera images do not have enough reality. Replacement of such traditional images by new images which have enough reality is desired task. Synthesizing arbitrary views from camera images is one of the solutions; it provides views which traditional teleconference systems could not provide. The arbitrary views will overcome the lack of reality. The 3D object model which can synthesize the arbitrary views is required. Virtual museum is a collection of digitalized specimen. The collection includes digitalized various objects in the world. Ordinary virtual museum collects static objects such as painting sculpture and artifact. Requirement of virtual museum is enabling something new which traditional non-virtual museum could not enable. Collecting moving objects such as insects as digitalized specimen answers the requirement. Traditional non-virtual museums collect moving objects but they can not provide how the objects had moved. On the contrary the virtual museum can provide the object s appearance with arbitrary pose when the motions of them are captured as 3D object model. To realize such virtual museum we have to reconstruct the 3D model which includes object s motion. Rapid prototyping is a technology for the speedy fabrication of sample parts for demonstration evaluation or testing. Measuring a shape of a mock-up is one of the important processes. Measured shape data is converted into CAD data and is used for product development. Rapid prototyping requires accuracy for the 3D model. In order to create accurate 3D model smooth surfaces and concave surfaces of objects should be modeled. To realize rapid prototyping with the 3D model we have to reconstruct accurate 3D shape of sample parts which includes smooth and concave surfaces.

2 2 4th International Symposium on Computing and Multimedia Studies These applications require 3D models of various objects. Versatile approach is required for acquiring the 3D models of various objects. The use of silhouettes for acquiring 3D object models satisfies the requirement because object s silhouette can be robustly extracted. Acquisition of 3D object models from silhouettes requires restriction as to an object s shape and it also requires prepared object s shape: Many works had acquired object s reflectance properties by using pre-measured object s shape. Many works had acquired object s motion by using a rough shape of object. Shape from silhouettes has a restriction on object s surfaces; it can not measure any concave surface. In this paper we show approaches for removing the restriction and preparation. In section 2 we show an approach by which the reflectance properties and the shape are acquired at the same time. The approach does not require any pre-measured shape. An approach which acquires the object s motion without using prepared object s shape is described in section 3. A method for measuring the concave and smooth surface is described in section 4. 2 Reflection Properties Acquisition Previously proposed methods had acquired reflection properties by using an object s shape. Our method acquires both the shape and the reflection properties. Our method acquires the shape from silhouettes using the volume intersection method. The shape is acquired as a set of voxels. It also acquires the reflection properties of each voxel. Previously proposed method which had acquired both the shape and the reflection properties required heavy computational cost. In our method the calculation is done independently at each voxel; it reduces the computational cost. Reconstructed reflection property represents both diffuse reflection and specular reflection at each surface voxel. Based on the Torrance-Sparrow reflection model we propose and improved reflection model which is suitable for the voxel-independent reconstruction. Reconstruction process consists of three steps; first the surface voxels are extracted. Then surface normal at each surface voxel is calculated. Finally its reflection property is estimated. Figure 1: Volume intersection method 2.1 Surface Normal Estimation The object shape is reconstructed as a visual hull. A set of voxel consist the visual hull. Surface voxel is a voxel some of whose neighbor voxels are not included in the visual hull. The surface voxel has its surface normal and its reflection properties. When we project the surface voxel into the camera images it is projected on the border of at least one of the silhouettes. We call a pixel on the border an edge pixel. An edge pixel has its 2D surface

3 4th International Symposium on Computing and Multimedia Studies 3 normal and the normal can be extracted only from the silhouette; so the normal can be extracted voxelindependently. The surface normal of the surface voxel is parallel to the 2D surface normal and it is orthogonal to a view line on the edge pixel; it is extracted voxel-independently. 2.2 Reflection Properties Estimation To estimate the reflection properties we use simplified Trannce-Sparrow reflection model. The model has two parameters; diffuse reflection coefficient and specular reflection coefficient. They are estimated from images which observe the surface voxel. We use the surface normal and cameras position to determine which images observe the surface voxel. 2.3 Experimental Results A polyvinyl chloride blue ball whose size is 22.5cm in diameter was used for the experiments. Each camera has pixels and observes [cm] region. We set each voxel size 0.5cm. The total number of voxels is = The shape and color property were reconstructed from images taken by eight cameras. The reconstruction results are shown in Figure2. The shape of the ball is reconstructed by using the visual hull method (Figure2 (ii)). Figure2 (iii) and (iv) are synthesized views from two cameras viewpoints. Figure2 (v) is a synthesized view which synthesizes only the diffuse color from the same camera s viewpoint as Figure2 (iii). Synthesized views from viewpoints where the cameras are not arranged are shown in Figure2 (vi) (vii) and (viii). (i) input image (ii) reconstucted volume (iii) a camera view (iv) the other camera view (v) diffuse color(camera(ii)) (vi) virtual viewpoint 1 (vii) virtual viewpoint 2 (viii) virtual viewpoint 3 Figure 2: Reconstruction results. Comparison between Figure2 (iii) and (v) shows that our method reconstructs not only a diffused color but also a specular color. As can be seen from the high-light on an input image in Figure2 (i) the position of the high-light is changed when the viewpoint is changed; this result also confirmed that our method reconstructs a specular color. 3 Motion Acquisition A method to acquire the motion of an articulated object is described. Several methods to estimate an articulate motion had been proposed. These methods require a shape model of each body parts. The

4 4 4th International Symposium on Computing and Multimedia Studies requirement is not suitable for the 3D object model acquisition because the applications described in section 1 require 3D models of various objects and preparing the shape model of each objects takes a lot of costs. Our method measures both the shape of body parts and the articulate motion at the same time. A whole shape of an articulate object is acquired with the volume intersection method. Making a correspondence between regions acquired in different times provides us the motion and the shape of the body parts. Unnecessary voxels in the visual hull makes the correspondence difficult. We propose a multidimesional voxel feature which is not affected by the unnecessary voxels and make the correspondence by using the feature. A whole shape of an articulate object is acquired as a visual hull and a set of voxels consist the visual hull. Successive visual hulls are used to acquire the shape and the motion. All the voxel in a body part are always under the same rigid motion. We extract such voxels from the whole shape and the voxels keeps the stablability. It is difficult to know the areas where no unnecessary voxels exist however. Our solution for this difficulty is the use of multi-dimensional distance. A multi-dimensional distance contains distances along several directions. The distances along some directions will receive effects from the unnecessary voxels. The distances along other directions receive no effect from it. The use of part of multi-dimensional distance instead of using whole multi-dimensional distance overcomes the effect from unnecessary voxels. 3.1 Experimental results A cow model shown in Figure3(b) is used for the experiment. The cow model consists five parts; a body and four legs. Ten frames of walking motion data observed by 20 cameras are used as input image sequences. The walking motion consists four different rotation: each leg rotates backwards and forwards. Figure3(a) shows sequential visual hull reconstructed from the input. Each visual hull consists approximately voxels. Figure3(c) shows the acquired part with our method. Figure3(d) illustrates the same result as Figure3(c) and zooms in a joint between the body and the right-front leg. Figure3(c) shows that the shape of five parts of the cow model are acquired by using our method. 4 Shape Acquisition The volume intersection method has an advantage over other methods. The advantage is that the method can acquire the shape of texture-less object. The volume intersection method requires object s silhouettes and does not require point correspondence which other methods require. Extracting silhouettes of texture-less is easier task than obtaining point correspondence. The volume intersection method has also a disadvantage. The disadvantage is the difficulty of acquiring smooth and concave surfaces. The visual hull which is acquired by the volume intersection method is convex hull circumscribing the object. Acquiring concave surface with the volume intersection method is impossible. The volume intersection method requires many cameras in order to acquire a smooth surface even if the surface is not a concave surface. We employ photometric stereo in order to acquire the smooth and concave surfaces. Photometric stereo estimates surface normals as a needle map. The needle map contains surface normals of concave surfaces. The needle map acquired by photometric stereo does not directly express a shape of the object however. Acquiring the shape requires reconstruction of a distance map from the needle map. Maximizing the following consistency gives the distance map. The consistency is that the needle map is consistent with the surface normal derived from the distance map. We call the consistency needle map

5 4th International Symposium on Computing and Multimedia Studies 5 (a) View Volume V t0 V t 9 (b) 3D cow model at t 0 (c) Segmented result (d) Segmented result(zoom) (e) Motion after the segmentation (f) Motion after the segmentation Figure 3: Simulation results. consistency. Depth edges make it difficult to calculate the consistency. A depth edge is an area on the distance map: On the area a depth from camera to the surface varies discontinuously. The discontinuity disables calculation of surface normal; it means that existence of depth edges disables a calculation of the needle map consistency. We propose an approach which uses silhouettes taken from different viewpoints. The silhouettes reduce the bad effects of depth edge. An incorrect depth image is not consistent with the silhouettes which are taken from other view points. Based on this fact our method minimizes two types of energy to reconstruct the depth image: one energy is based on a consistency between depth image and needle map and the other is based on a consistency between depth image and silhouette. Depth Edge Body Region Pole Region Figure 4: Depth edge

6 6 4th International Symposium on Computing and Multimedia Studies Our method uses multiple camera. Let the number of cameras to be C and let a silhouette on camera c (c = 1... C) to be S c and let the number of pixels on S c to be M c and let each pixel to be m c i (i = 1... M c ) and let n c i to be a normal vector of a surface observed by mc i. Let us explain a pixel m c i and a distance map. A pixel mc i included in a silhouette S c occupies the square region on the image. We describe it ([x c i xc i + 1) [yc i yc i + 1)) and call a point (xc i yc i ) representative point of m c i. A distance of a representative point of mc i is defined as a distance between focal point of the camera c and a point on the surface projected on the representative point of m c i. We denote the distance as Z(x c i yc i ). A distance map is a 2D matrix which consists the distances of each representative points. 4.1 Needle Map Consistency Let Z(x c i yc i ) to be a depth of a pixel mc i and let a surface normal which is observed by the pixel mc i to be n c i = (pc i qc i 1)T. n i c X i c Z( x i yi ) m i c ( x i +1 y i +1) ( x i yi +1) ( x i +1 y i ) ( x i yi ) Figure 5: Needle map vs surface normal Let us suppose that the surface observed by m c i is a plane containing surface normal nc i. Depths of three points (x c i + 1 yc i )(xc i yc i + 1) and (xc i + 1 yc i + 1) are given by nc i and Z(xc i yc i ). When we express the depths Z 1 Z ±0 Z 1 they can be written ±0 1 1 ( ) Z 1 (x c i + 1 yi c ) =Z(x c i yi c ) 1 + pc i (1) ±0 f c ( ) Z ±0 (x c i yi c + 1) =Z(x c i yi c ) 1 + qc i (2) 1 f c ( ) Z 1 (x c i + 1 yi c + 1) =Z(x c i yi c ) 1 + pc i + qc i (3) 1 f c f c where f c is a focal length of the camera c. These depths show that we have four ways of acquiring the depth of (x c i yc i ); Z(xc i yc i ) a depth from (x c i 1 yc i ) with Equation1 a depth from (xc i yc i 1) with Equation2 and a depth from (xc i 1 yc i 1) with Equation3. These ways give the needle map consistency. When the four depths are close together they provide a high consistency. It gives the following energy expressing the needle map consistency. Lower energy shows the higher consistency. E N = m c i Sc ( Z(xc i yi c ) Z 1 (x c i yi c ) ±0 2 + Z(xc i yi c ) Z ±0 (x c i yi c ) Z(xc i yi c ) Z 1 (x c i yi c ) 1 2 ) (4)

7 4th International Symposium on Computing and Multimedia Studies Silhouette Consistency Visual Hull Line Some pixels included in a silhouette are adjacent to pixels which are not included in the silhouette. We call such pixels edge pixels. Pixels containing at least one of 8-neighbor pixels which are not included in a silhouette are extracted by our method. Suppose a view-line which starts from the focal point of a camera and passes through a representative point of the edge pixel. Projecting the view-line to the other camera gives a 2D line on this camera s image. The 2D line intersects a silhouette on this camera s image. That is some parts of the 2D line are included in the silhouette. In other words some parts of the view-line are projected into the silhouette. A part of the view-line which is projected into all the silhouettes is called a visual hull line. Ignoring a sampling error we consider that the visual hull lines consist the surface of the visual hull. The silhouettes of the object completely correspond with silhouettes of the object s visual hull. The visual hull line gives the following constraints. The object never intersects any visual hull lines. The visual hull line is tangent to the object at more than one point. These constraints are called visual hull line constraints. 4.3 Experimental Results A orange toy is used for the experiment. We put the toy into the center of our multicamera system[1] and acquired input images with 8 cameras and 24 lights. Each camera has pixels and 4 cameras are arranged on the front side of the toy and the other 4 camera are on the back side. Twelve lights irruminate the toy from the front side of the toy and the other lights irruminate the toy from the back side. Figure6 shows reconstructed whole shape by integrating all cameras depth maps. As Figure6(a) shows Visual Hull which uses only the silhouette consistency does not reconstructs smooth surface. Figure6(b) is a result by using needle map consistency and not using silhouette consistency. Figure6(b) shows that lack of silhouette consistency produces unnatural shape. On the contrary using the silhouette and needle map consistency does not produce such unnnatural shape as Figure6(c) shows. These results show the effectiveness of our method for objects on which depth edges exist. (a) visual hull (b) only needle map constraint (c) proposed method Figure 6: Recovered shape

8 8 4th International Symposium on Computing and Multimedia Studies 5 Conclusion This paper proposed approaches for acquiring a 3D object model. When we try to acquire the 3D object model we meet the problem; the restriction and preparation of object s shape. The proposed approaches solved the problem. Voxel-independent approach reduced the computational cost for acquiring the shape and the reflection properties the cost was the problem other methods had. The use of multi-dimensional voxel feature solved the problem of the unnecessary voxels and acquired the body parts and the motion of articulate objects without preparing any shape model. The silhouette constraint solved the problem of the depth edge and the use of the constraint enabled us to acquire the concave and smooth surface. References [1] MINOH Michihiko IIYAMA Masaaki KAMEDA Yoshinari. 4pi measurement system: A complete volume reconstruction system for freely-moving objects. In IEEE Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI2003) pages p