3D Models from Contours: Further Identification of Unexposed Areas

Transcription

1 3D Models from Contours: Further Identification of Unexposed Areas Jiang Yu Zheng and Fumio Kishino ATR Communication Systems Research Laboratory 2-2 Hikaridai, Seika, Soraku, Kyoto , Japan Abstract This paper explores shape from contour method for acquiring a 30 model by using a continuous sequence of images taken as an object rotates. We analyze the areas that are unexposed to contours ana' detect them for further investigation. We describe a general approach to use the shape from contours. Our direct goal is to establish a 30 model of human face for the growing needs of visual communications. We have obtained some good results. 1. Introduction In recent years, developments in model based coding, stereoscopic display, and graphics generated virtual space have been an increasing demand for three-dimensional modeling of the human face as well as other objects. The application areas include visual communication, education, (and entertainment. We are attempting to generate views in some 3D virtual space for teleconferencing and video phone where a complete 3D head in needed. Although laser range finders and other structured light-projecting systems have been developed for acquiring precise 3D shape, They still have the following drawbacks: (1) they receive bad data from areas where the surface reflectance is low (such as black hair), and, (2) people do not feel comfortable when a laser continually scans across their face. We therefore explore a vision method. Shape from rotation is a natural way to observe an object. We look at an object while rotating it by hand, or, alternatively, we move our view point around the object. Rotational movement and its axis are particularly easy to obtain either from control or input images. There are different kinds of information in the image sequence of a rotating object --- information such as contour, flow, stereo disparity, and shading. Among them, the occluding contour in a 2D image directly reflects the 3D shape. At a contour point, the surface normal is perpendicular to both the line of sight and the contour itself. Giblin et al. [l] described a scheme to recover shape from continuous contours, but they did not consider the visibility of concave parts. Cipolla et al. [2] generalized the relative motion between camera and object to arbitrary translation and rotation, and applied algorithm to simple objects, but they were unable to detect the concave parts. Here we use occluding contour to recover shape because it has a good detectability in front of a homogeneous background. We will further deal with area that is invisible when viewing only silhouettes. We found that the unexposed area, which might be concave or planer surface, is numerically related to the nonsmoothness of the contour distribution (or to the discontinuity in the image velocity of a contour). We detect these areas by filtering contour distribution according to the model resolution required. The detection of unknown areas allows us to plan further acquisition of information inside these areas --- that is, by looking for such other infomation as fixed features or selfoccluding edges, and etc.. We also think that it as an important step towards automatic exploration of a model by an active vision approach. 2. Recovery of a 3D Model from Contours 2.1 Previous Work For simplicity, we consider the orthorgonal projection, but the results needed for perspective projection can be deduced in the same way. We have a face rotating around an axis that is stationary relative to a camera. Assume that the camera is set so that the y axis of the image frame is parallel to the axis of rotation. The surface of object can thereby be decomposed into connected boundaries on rotational planes (see Fig. 1). The rotation axis is at the position 0, the rotation angle of the face is 8, and the point on the boundary viewed at the angle is denoted as P(X,Z). In the image frame, the projection of this point is a contour point p(6) and its image distance from the projection of the axis x0 is o(6). The point P should be on the line of sight L(6): X cos0 + Zsin0 = o(8) (1). Imagefame /XO Fig. 1. Recovering shape from occluding contours. The shape on the slice is the envelope of lines of sight through the contour points p(6) changing in the image sequence, and the normal of the boundary (X',Z) is perpendicular to the line of sight. We have (X', Z') (cos& sin0)=o (2) for a smooth boundary where the first derivative exists. Taking the first derivative of Eq. (l), we obtain X/92 $3.00 Q 1992 IEEE 349

2 X'cos0 +Zsin0 - Xsin0 +Zcos0 = U'@). (3) Solving for X and Z, the coordinztes of PGZ) can be given as P(X, Z): X = w cos0- w'sin0 (4.1) 2 = w sin6 + w' cos0 (4.2) The depth of P(XZ) from the image fmme is w'(e) and the radius of curvature of the boundary is w+w" [l]. In practise, w can be located in the image to the precision of a pixel, whereas d(e) is less precise because it is determined from digitized values. The method is therefore good at estimating the distance of 3D points from the axis (the first terms in Eqs. (4.1) and (4.2)) but contains errors in the angle with respect to the axis (determined by the second terms in these equations). 2.2 Shape Unexposed to Contour A good vision algoriihm should be able to locate where the unknown part is in its output so that the area can be further explored by other methods. In the shape from contour method, a nonconvex shape on the moving plane, as well as other occluded parts, are unknown. Let us look at the behavior of line of sight at an occluded boundary on a rotational plane (Fig. 2). Lie of sight\ Fig.2. Detecting the boundary uneyposed to contour. A concave or linear segment of the boundary is between two convex segments and is occluded by them on both sides. As a face rotates, the line of sight shifts as a tangent line on a convex boundary. Along a particular viewing direction 80, the line of sight will touch two convex tangent points at the same time and then jump from the first mountain to the second. At the angle 80, the trace of the image point p(8), or the change is not smooth, although it is still continuous. Figure 3 shows three objects set put on a turntable or modelling: a sphere, a cylinder, and a box. A horizontal epipolar plane at the center of the sphere is shown in Fig. 4. We can intuitively see the cusp points (which are due to the existence of occluded areas between the sphere and cylinder) on the traces of the left and right contours. Denoting the left and right derivatives of as 0'- and a'+, and denoting the tangent points sharing the same line of sight at the separated convex segments as Pl(X1,Zl) and Pz(X2,Z2), we have X1 cos sin0 = w(8) (5.1) x2 cos sin0 = w(8) (5.2). Taking the left derivative about 8 at the point P1 and the right derivative about 8 at the point P2, we obtain -Xi sin6 + Z1 cos0 = U'- (e) L'- (e): (6.1) L'+(8): -X2 sin cos6 = 0'+(8) (6.2) Solving for X1, X2,Z1, and 22, we have Pi(X1, U): X1 =w cos0-0'- sin0 (7.1) Zl =w sin0 + 0'- cos0 (7.2) P2(X2,22): X2 =w cos0 - U'+ sin0 (8.1) 22 =a sin0 + a'+ cos0 (8.2) D(AX, U) X2 -XI = -(a'+ - d-1 sin0 (9.1) zz - zi = (o'+ - UP-) case. (9.2) The vector jumping from Pl(X1, Zl) to P2(X2,Z2) thus has the length a'+ - CO'-, and we can compute two margin points that indicate the unexposed segment. For a smooth trace where w'+ = w'- = a', the jump is zero and the formulas for computing the points on the boundary are the same as Eqs. (4.1) and (4.2). If we find discontinuity in the first derivative o'(e) and, correspondingly, a linear segment connecting two tangent points, we can assert there is a concave, linear, or occluded convex boundary between these points. 2.3 Detecting the Unexposed Areas Suppose that the resolution required for a 3D model is expressed as the size of the smallest patch on the model. This resolution's component on the rotational plane is denoted as the length 6. According to this given 6, we identify areas unexposed to contours by locating the cusp points of o(0). Because the length over an unexposed boundary is equal to the difference between the left and right first derivatives of de), we have the criterion o'+ - 0'- > 6. (10) As 8 increases, w(e) comes in continuously by tracking a contour point. We fed the sequence w(e) to a connected pair of differential operators, which has the same effect as a second-order differential filter denoted F. The first convolution with the filter yields the step-type signals at the cusp points and then the second convolution yields the peaks in the output at cusp points. It is interesting to notice that cusp points on the trace of the right silhouette are always leftward, and the cusp points on the left trace are rightward. We can therefore find nonsmooth points of w(0) from the right silhouette by extracting the local maxima in the filtered sequence F(o(8)). To ignore the concave or linear segments that are shorter than the required length 6, we set the threshold of F(o(8)) at the 6 before finding the local maximum. At the same time, we set a window moving over the w(e) to compute precise first derivative for Eqs. (4) at smooth place. The window size is adaptive near the extracted cusp points so that the cusp points are excluded from the estimation of 61'- (e) and a'+@) for the Eqs. (7-9). We simply use the least square approximation to fit a line to the data in the window so that we can obtain its tangent as the derivatives. This processing for the shape and unknown area

3 identification can be carried out almost instantaneously as images come in continuously; with the only delay as the size of the filter F(o(8)). 3. General Approach of Using Contours 3.1 Relations between Contours and 3D Points There are several kinds of relationship between contours in the image and 3D points on the observed object, and the distribution of estimated 3D points is related to the surface curvature. Let us first look at the case where the contour in the image is not parallel to the rotational plane. As a line of sight shifts across a smooth convex surface on the rotational plane, there is a one-to-one correspondence between a contour point and a surface point. Differentiating Eqs (4.1) and (4.2). we obtain AX = - (ww") sin0 A8 (11.1) AZ = (WO") cos0 A8 (11.2). If an interval of A8 is given, the density of the estimated 3D points on the boundary is 1 1 curvature =- denrity=~~=(o+o")a8 A8 ' (12) According to Giblin's [l] results, this density is proportional to the curvature of the boundary. This is desired for the data reduction in representing shape of the 3D model. In the extreme case of infinite curvature, which is a convex cusp point on the surface, more than one line of sight goes through the cusp point and the recovered 3D points should be at the same position. This forms a manyto-one correspondence between contours and a surface point, and this kind of correspondence may allow us to refine the estimation by using redundant dab. Moreover, in the case of unexposed boundary, there is a one-to-many correspondence between a contour point through which a line of sight touches two convex boundaries, and points that are occluded. The shape invisible to contours should be investigated by using other visual cues. If the contour is, on the other hand, parallel to the rotational plane, a line of sight through that contour may touch many 3D points, and a 3D point will be projected to contours in more than one image. This produces a manyto-many correspondence. A plane on the rotational plane is one such case. This kind of contour can be detected at the discontinuity in each contour o(8,y) along the y axis. 3.2 Implementation We choose to implement this method in an algorithm for a fixed camera and an object that rotates around an axis. This geometry needs little space, is easy to set up, and allows the use of a static background that reduces the cost in image processing. Moreover, it needs almost no calibration of camera to get the rotational axis. Our algorithm starts tracking continuous contours from image to image and generates a matrix W(8,y) that describes the distribution of o(8,y) at each height. Figure 5(a) shows the contour distribution of the objects shown in Fig. 3: the period here is 21c and the value of o is displayed in grey level. To find the vertical valleys in this image, the dynamically yielded W(8,y) is filtered horizontally by the operator F(o(8)). After thresholding its output at k6, where k is a weighted constant, local maxima are detecled and marked as cusp points. At the same time, we locate horizontal edge points in W(B,y), where discontinuities in the vertical direction appear in the contour, by convoluting with a normal vertical differential operator. The resulting positions are marked for exclusion from shape estimation by Eqs. (4) and (7-9). For the distribution W(6,y) in Fig. 5(a), vertically distributed nonsmooth points and horizontal edges are shown in Fig. 5(b). Surface points are estimated as fallows: Small adaptive windows shift over W(8,y) at the places without cusp points or discontinuous points. The derivatives are approximated inside the windows, and the 3D positions are computed either by Eqs. (4) or by Eqs. (7)-(9), with the unexposed area detected. Figure qa) shows two views of the estimated shape of the objects shown in Fig. 3. The obtained 3D points are connected by triangular patches. In figure 6@), the unexposed areas are shown as gray patches. The occluded part may still have some convex and concave segments. Due to the differences in surface orientation, the convex segments extended from both sides can be calculated from the self-occluding edges by using Eqs. (4.1) and (4.2) (see Fig. 2). But if the concave part is shallow, it is difficult to locate self-occluding contours in a long period of rotation. We need some verification of the results. Fixed points due to the discontinuity in surface albedo or surface orientation can help to determine the shape in an unexposed area [3]. 4, Experiments Experiments were performed using plater models and real people. The cvnera was fixed far away from subjects and a lens with long focal length was used to approximate orthorgond projection. A model was put on a turntable that could be controlled by computer step by step. The results generated from a total of 360 images are shown in Figs In a more difficult experiment, people sat in a swivel chair and about 200 images (256*256) were taken by a video camera as the chair rotated. We do not have control of the chair so the changeable rotation speed and unknown axis position had to be obtained from image sequence. We tracked two marks attached to the subjects' shoulders and succeeded in estimating the information about their positions. The texture of the face was also collected in a dynamic projection image [4]. When the images were taken, this texture was dynamically sampled from a vertical pixel line at the center of both left and right silhouettes. The memorized texture was then mapped onto the recovered 3D model. Figures 11 show one example of models made from real people. Our algorithm has stably modelled many people by using image sequences about 7 seconds long. 35 I

4 352

5 5. Conclusion We made 3D models from a sequence of rotational images. We analyzed the behavior of contour and the unexposed area in the image sequence, and we gave a general approach to use contour. Identification of the areas that an algorithm is unable to model will allow other vision modules to further plan active investigation. We think this ability is very important for an algorithm in qualitative understanding of the visual world. The established facial model will be used in graphics-generated vi mal space for visual communications.- References [l] P. Giblin, R. Weiss, "Reconstruction of surface from profiles", 1st ICCV, pp , [2] R. Cipolla, and A. Black, "The dynamic analysis of apparent contours", 3th ICCV, pp [3] J.Y. Zheng. F.Kishino, "Verifying and combining different visual cues into a complete 3D model", CVPR92, June, [4] J. Y. Zheng, and S. Tsuji, "From Anothorscopic Perception to dynamic vision", 1990 IEEE Conf. Robotics and Automation, V01.2, pp Fig. 10. Area unexposedto the contours on the plaster figure. Fig. 9. Wire frame model and shaded model of the recovered figure. Fig. 11. An example of 3D face models obtained from a real person with only a swivel chair and a homogeneous background behind the subjects. Several views of the recovered model are shown. It contains about 80 x 100 3D points. 3.53