Systems and Computers in Japan, Vol. 33, No. 9, 2002 Translated from Denshi Joho Tsushin Gakkai Ronbunshi, Vol. J84-D-II, No. 5, May 2001, pp. 778 788 Topology-Adaptive Modeling of Objects Using Surface Evolutions Based on 3D Mathematical Morphology Kenji Hara, 1 Hongbin Zha, 2 and Tsutomu Hasegawa 3 1 Mechanics & Electronics Research Institute, Fukuoka Industrial Technology Center, Kitakyushu, 807-0831 Japan 2 Center for Information Science, Peking University, Beijing, 100871 China 3 Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka, 812-8581 Japan SUMMARY 1. Introduction Level set methods were proposed mainly by mathematicians for constructing a model of a 3D object of arbitrary topology. However, those methods are computationally inefficient due to repeated distance transformations and increased dimensions. In the paper, we propose a new method of modeling fast objects of arbitrary topology by using a surface evolution approach based on mathematical morphology. Given sensor data covering the whole object surface, the method begins with an initial approximation of the object by evolving a closed surface into a model topologically equivalent to the real object. The refined approximation is then performed using energy minimization. The method has been applied in several experiments using range data, and the results are reported in the paper. 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(9): 58 67, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.1152 Key words: topology-adaptive object modeling; mathematical morphology; level set method; surface evolution; pseudo curvature flow. Recently, many methods have been proposed for automatic modeling of 3D free-form objects from multiview range images [1]. The resulting models, however, are usually still large data sets comprising great amounts of unorganized 3D points. To build a model base from which it is easy to extract the significant features such as surface normals or orientations, we have to transform the data further into surface models describing explicitly how those vertices are connected. The framework of optimization based on deformable surfaces was successfully used in the past for such object modeling [2 4]. Deformable surfaces have the ability to generate smooth closed surfaces on the basis of the principle of energy minimization. However, deformable surfaces have a serious drawback in modeling multiobject scenes or objects of arbitrary topology such as those with holes. One method for overcoming this problem is to apply shape evolution with blending [5]. However, while this method can allow the addition of a hole to the model, it cannot handle multiobject scenes adequately. On the other hand, topology-adaptive modeling method using a level set approach has been presented [6]. This level set modeling method can model multiobject scenes or objects of arbitrary topology by embedding an evolving surface (curve) as the zero level set of a function with a higher dimension [7, 8]. 58 2002 Wiley Periodicals, Inc.
However, this modeling method is computationally inefficient since it requires repeated distance transformations and increased dimensions. In the paper, we propose a new fast method of topology-adaptive modeling by a surface evolution approach mainly based on 3D mathematical morphology. Mathematical morphology usually treats binary or gray scale images, and operates on them by morphological transformations using structuring elements [9]. This paper regards mathematical morphology as the dynamic model governing the 3D surface evolution, and then evolves surfaces by repeated morphology operations. It is well known that mathematical-morphology transformation based on a single dilation of a shape with a convex structuring element induces a differential deformation [10]. Thus, by introducing the curvature-dependent structuring elements, we can attain approximately but fast the surface deformations introduced in the level set approach, that is, curvature-dependent flow. First, by dilating every data point with a ball structuring element, we build a connected region, and then define it as the stopping region in which any surface point attains zero speed. Moreover, we set a volume including the whole 3D data, and then define the set of voxels lying on its boundary as the initial closed surface. In this paper, the surfaces are mainly represented by sets of the voxels interconnecting compactly. Next, we perform erosion transformation of the volume with a ball structuring element, and then extract the boundary of the transformed volume. This shrinking and splitting process is repeated until no surface point moves in the stopping region. Here, to keep the smoothness of the surface during its evolution, we use a structuring element with curvature-dependent size. After that, we extract the external boundary of the stopping region on the basis of the resulting surface, and then perform erosion along this boundary with the structuring element used in building the stopping region. By this updating of the stopping region, we shrink the surface further. We build a surface model based on mesh representation by viewing the surface as a set of hexahedra and then extracting those external squares. After the above rough estimation, by solving the optimal positions of those vertices on the basis of the principle of energy minimization, the resulting model is refined. The method has been applied in several experiments using synthetic and real images, and the results are reported in the paper. F. Here F can be a function of variables such as curvatures and normal directions [8]. The main idea of the level set method is to embed this moving surface Γ(t) as the zero level set of a higher-dimensional function u = φ(x, y, z, t). Let φ(x, t = 0), where x = (x, y, z), be defined by where d is the distance from x to Γ(t = 0), and the plus (minus) sign is chosen if the point x is outside (inside) Γ(t = 0). The stipulation that the zero level set of the function φ must always match the moving surface means that By the chain rule, we get The above equation is called the level set equation. One of the most desirable aspects of the level set formulation is that it allows the tracking of surfaces even as they undergo topological changes during evolution, as shown in Fig. 1. Malladi and colleagues proposed a method of extracting multiple contours from 2D images using the level set scheme [6]. They utilized a curve evolving at a curvaturedependent speed where ε is a positive constant and κ is the curvature. The problem here is how to define a speed function from the image data that serve as a halting criterion for this (1) (2) (3) (4) 2. Level Set Methods Given an initial 3D closed surface Γ(t = 0), we now derive a level-set formulation of the motion equations of the surface Γ(t) moving along its normal directions with speed Fig. 1. Curve evolution in a level set method. 59
speed function. Thus, by multiplying the above speed function by the filter we obtain the new image-based speed where G σ I denotes the image convolved with a Gaussian smoothing filter whose characteristic width is σ. The term G σ I is essentially zero except where the image gradient changes rapidly, in which case the value becomes large. The filter k I (x, y) is close to unity away from boundaries, and drops to zero near sharp changes in the image gradient, namely, in the stopping region. Thus, k I (x, y) serves to prevent the moving curve from passing out of the regions within which the original curves have to be constrained. To extend the above 2D modeling approach stated earlier to 3D modeling, one defines the level set equation as u = φ(x, y, z, t). However, this level set 3D modeling method has some difficulties, as follows. To determine the speed function F in the original method, searching for the closest surface points over the entire computational domain requires O(N 5 ) operations in 3D space, if we assume that there are N discretizing cells in each coordinate direction, plus approximately O(N 2 ) points on the surfaces. As an alternative, one can also perform searching only in a neighborhood of the zero level set; this is known as a narrow-band algorithm [12]. In this case, the operation count in 3D space drops to O(kN 2 ), where k is the number of cells in the narrow band. One of the other fast level set schemes is to apply an adaptive mesh in which each voxel has a size dependent on the curvature [13]. So far as we know, the fastest level set method is the fast marching method, which is a numerical technique for solving the eikonal equation (7). The computational cost is O(n 2 log n), where n is the number of points in each coordinate direction. However, this method is not computationally efficient yet. Thus, in this paper, we present a new method of modeling objects of arbitrary topology by using a morphology-based surface evolution. (5) (6) 3.1. Mathematical morphology and surface evolution Kimia and colleagues have shown that surface evolutions generated by 3D mathematical morphology operations are equivalent to those governed by a partial differential equation [10, 11]. For a ball structuring element, B, the set nb, where n is a positive integer, is defined as where 8 represents the dilation operator of 3D mathematical morphology. Here, if B is convex, nb is the same shape as B and n times as large as B. A single dilation with a structuring element nb is equivalent to n repeated dilations with B, which uses the distribution law in the dilation operation. Equation (8) shows that for a single operation with a convex structuring element, this operation can be decomposed as a sequence of n operations with the same structuring element but of size equal to 1/n of the original. In the limit as n, this morphology-based deformation can be viewed as a differential equation. On the other hand, the morphological transformation from a region S of 3D space to S can be regarded as deformation from its boundary S of S to S. In this paper, these boundaries are approximately described by sets of connected voxels. In object modeling, however, these sets of the points must be transformed to surface models by postprocessing. Now, let a shape S evolve by dilation (erosion) with a convex structuring element of size λ. Here, we assume that a point p on the surface S moves a distance Γ(p, t, λ) along the outward normal. Then, since Γ(p, t, 0) = 0, the speed F in the outward normal direction is represented as (7) (8) 3. Morphology-Based Surface Evolution To apply the surface evolution approach to topologyadaptive object modeling, we require that the surface retain smoothness and avoid self-intersection during its evolution. Though the level set approach satisfies these conditions, this method is computationally inefficient due to the distance transformations and the increased dimensions. In the section, we give a brief overview of the fast surface evolution approach in fitting closed surfaces to 3D data. It has been shown that dilation (erosion) operations can be regarded as surface evolutions governed by a partial differential equation expressed by Eq. (9). It is well known that the amount β(p, t) of differential deformation is the maximal projection of B onto the normal of S at p [10]. For (9) 60
example, for a unit ball structuring element B, we have the evolution equation (9) in the form This is the equation of a surface evolving at a constant speed along the normal. Differential equation (9) has been derived on the assumption that the surface S is smooth at point p; otherwise, even if p is a discontinuity, we can obtain the solution of Eq. (9) by using the level set method. It is known that this solution is equal to the above surface deformation by dilations (or erosions) [11]. 3.2. Pseudo curvature flow (10) Fig. 3. Amount of surface convexity. (12) The surface driven by Eq. (10) easily develops singularities during its evolution [Fig. 2(a)], and it cannot necessarily be applied to object modeling. In this subsection, we define the surface evolving while preserving smoothness and developing no self-intersections as a solution satisfying the equation in which Eq. (4) is extended to three dimensions. Differing from the level set approach [6, 8], this evolving surface can be traced at small computational cost without increased dimensions and repeated search for the closest points. In the paper, we let the closed surface smoothly contract and split by repeated erosions with a variable-size ball structuring element, and then apply this surface evolution approach to object modeling. The radius λ of the ball is defined as (11) where max (, 0) is a representation introduced to prevent λ from being negative. λ 0 and ε are positive integral and positive real constants, respectively. K = K(p, t) is the convexity of the surface, introduced instead of noise-sensitive 3D curvatures, and therefore, the second term of Eq. (11) can be regarded as a smoothing term. K is defined as the volume rate of the region outside the surface in the neighborhood of p, and thus it is defined as where α = α(p, t) and β = β(p, t) are the numbers of inside and outside cells, respectively, within a distance ρ from p (Fig. 3). In the above surface evolution process, the surface smoothness is preserved as the mean-curvature flow defined in Eq. (4) [Fig. 2(b)], and therefore, we call it pseudo curvature flow. When dilations are used, Eq. (11) is replaced by λ = max (λ 0 ε K, 0). Here, we discuss pseudo curvature flow from the viewpoint of computational expense. There are approximately O(n 2 ) voxels in the evolving surface, assuming that there are n voxels in each coordinate direction. First, to determine the size λ of the structuring element for these voxels, the approximate value K of the mean curvature must be calculated. However, it only takes values equal to the weighted sum of neighboring voxels. Next, we must calculate the region to be eroded for each voxel in the surface. We prepare a set of several ball structuring elements of different sizes as (13) where B is the ball structuring element of unit size and λ 0 is the positive integer in Eq. (11). After that, we take the structuring element from G which has the size of the integral part of λ obtained by Eq. (11). If the integral part of λ is larger than 2λ 0, the structuring element of the size of 2λ 0 is selected. Consequently, it is obvious that pseudo curvature flow results in a time complexity of O(n 2 ). 4. Topology-Adaptive Modeling by Surface Evolution Fig. 2. Surface evolution using size-variable structuring element. One of the difficulties arising in modeling 3D objects from range data by the surface evolution approach is determining how to halt the evolving surface at the object surface. In our strategy, we define the region in which any surface point halts moving, and therefore, a rough but topologically correct model can be obtained by surface 61
evolution based on pseudo curvature flow; we then refine this model by energy minimization. In this section, we give an overview of this topology-adaptive object modeling method based on both surface evolution and the conventional deformable surface. 4.1. Topology-adaptive modeling algorithm The algorithm we developed consists of a two-phase procedure. First, by dilating a given set of 3D image data [Fig. 4(a)] with the ball structuring element of size δ, we build a connected region [Fig. 4(b)]. This region is called the stopping region, in which any surface point halts moving. Next, we set a single closed surface surrounding the whole body of data as the initial shape, and then repeat the processes of shrinking and splitting by the pseudo curvature flow until every surface point stops moving near the border of the stopping region [Fig. 4(c)]. After that, we separate the surface of the stopping region into the boundary surface lying inside the object (internal boundary surface) and the other surface (external boundary surface) on the basis of the resulting surface. We also erode the stopping region along the external boundary surface with a ball structuring element of size δ, which updates the stopping region [Fig. 4(d)]. Within this stopping region, we shrink the surface again until it does not evolve further [Fig. 4(e)]. We build a surface model based on a mesh representation by viewing the surface as a set of hexahedra and then extracting those external squares. The above step for initial estimation corresponds to the first stage. At the second stage, the surface model resulting from the above step is refined for the purpose of improving its accuracy. The optimal positions of those vertices are calculated based on the energy minimization principle [Fig. 4(f)]. Further details are described below. 4.2. Initial estimation Given a set of 3D image data, our goal here is to estimate a topologically correct model by surface evolution based on pseudo curvature flow. We expand every data point into a ball with radius δ by eroding it with a ball structuring element δb of size δ. If δ is sufficiently large, the expanded data points are connected with each other. However, inappropriate selection of the δ values causes a failure in modeling due to the topologically wrong stopping regions. The stopping regions resulting from too large δ and small δ are illustrated in Figs. 5(c) and 5(d), respectively. It is seen that both are topologically wrong. We determine the δ value as follows. We first build a stopping region with comparatively large δ, and then extract the voxels of its boundary. After that, we repeat the process consisting of decreasing δ and extracting the border surface of the stopping region until the stopping region is regarded as being topologically equivalent to the object surface, and thus obtain the estimated value of δ [Fig. 5(b)]. Here, as a criterion for the topology judgment, we introduce the detection rate (14) where N r is the number of data points having model points within a distance δ, and N is the total number of data points. Fig. 4. Outline of our algorithm. 62
Fig. 6. Surface shrinking-and-splitting. Fig. 5. Influence of variation of δ. Here, if q is smaller than a threshold th, δ is lowered, since it is inferred that the current model is wrong in its topology. From the above step, a set of discrete data is transformed into the stopping region X, which is between the external border surface and the internal one, and X is represented as (15) where L is a set of data points. Next, we set a single closed surface surrounding the whole body of data as the initial shape. Then, we shrink and split this surface by repeated erosions with the variable-size ball structuring element λb until every surface point stops moving (Fig. 6). Here, the radius λ = λ(p, t) of the ball structuring element centered on a point p of the evolving surface at time t is defined by modifying Eq. (11) as with this corrected stopping region, we can obtain a surface stopping near the surface of the object. However, the resulting model is still a set of voxels, and thus, it is not a surface model explicitly describing how the vertices are connected. To transform the set of voxels into a polygonal mesh, we regard the surface as a set of hexahedra, and then extract their external squares. In addition, we can decrease the number of patches by dividing the 3D space into larger sections. 4.3. Model refinement This stage is employed mainly to enhance the accuracy of modeling by removing local irregularities resulting from the earlier evolution. The current surface model provides a reasonable approximation to the object surface in the topology. Moreover, it has a comparatively smooth shape owing to the smoothing effect of pseudo curvature flow. Therefore, we can apply conventional deformable surface techniques to this model [2 4]. For the set v of position vectors of the vertices, the energy function to be minimized can be expressed by (16) (17) where λ 0 and ε are the same positive constants as those used in Eq. (11). K = K(p, t) is the amount of convexity of the surface defined in Eq. (12). Equation (16) states that p is not moved if λ 0 + ε K is negative or p lies in X, and thus, the surface stops evolving so that the object is covered by the surface [Fig. 6(d)]. After that, we extract the external boundary surface of the stopping region X by checking whether each voxel of the boundary surface of the stopping region lies outside this surface or not. Moreover, we erode X along this external boundary surface with the structuring element δb used earlier. By restarting surface evolution E int (v) corresponds to a smoothness constraint based on springs imaginarily placed between neighboring vertices. E est (v) is a data closeness term based on springs placed between the vertices and their nearest data points [3]. Equation (17) can be minimized fast using a greedy algorithm [14]. As a result, we can obtain the optimum mesh fitted to 3D data. 5. Experimental Results In this section, we present the results of some experiments using synthetic range data and real data acquired by a laser range finder. 63
Figure 7(a) shows 3D data synthesized from CAD data for a hexagon nut. This experiment is made mainly to show the capability of our method for modeling sparse data and its superiority to the conventional level set modeling method. At the beginning, we specify a cubic closed surface covering the data, and select δ as 8 using the parameter estimation technique in Section 4.2. Next, we shrink and split the surface until no surface point moves as shown in Fig. 7(b). After that, we update the stopping region, and obtain the model for this region shown in Fig. 7(c). We obtain the final model by optimizing the positions of vertices in the model refinement stage in Fig. 7(d). Since the object has a simple shape, we reduce the number of patches to about 1/4 in extracting the mesh from the object model shown in Fig. 7(c) by dividing the 3D space into double sections. To compare our method with the existing topologyadaptive modeling method, intermediate results obtained using the fast level set approach called the narrow-band algorithm [12] are shown in Figs. 8(a) and 8(b). These models correspond to Figs. 7(b) and 7(c), respectively. As is evident here, there is not much difference between them. Table 1 summarizes the surface evolution process in the first row, indicating the values in Figs. 7 and 8. The subsequent rows correspond to the widths of stopping region (δ), numbers of vertices (N), and used CPU times (T) in seconds on a Pentium II 300-MHz personal computer. Comparing the two CPU times corresponding to Figs. 7(c) and 8(b) in Table 1, our method is approximately Fig. 8. Experiment by a level set modeling method. 10 times faster than level set modeling method using the narrow-band technique. Figure 9 illustrates the process of surface evolution in the initial estimation stage by cross-sectional views. Figures 9(b) and 9(c) illustrate the intermediate and final results, respectively, obtained by surface shrinking and splitting for the initial stopping region. Figures 10(a) to 10(c) show cross sections of the evolving surface when no stopping region is set. The smoothness of the surface is gradually increased in each step due to the curvature-dependent speed. In the second experiment, the object to be modeled is a teacup with a hole and rim shown in Fig. 11(a). This object is a typical difficult-to-model example. If we connect the data points in the neighborhood one by one, a data point on the inside of this cup may be connected with one on the outside. While the deformable surfaces can avoid this problem, the problem of how to restore the hole formed by the handle arises. We obtained images of a teacup suspended by a wire from several viewpoints. Figure 11(b) shows the 3D data of this object. The intermediate and final results are illustrated in Figs. 12(a) and 12(b). The number of vertices is 24,352 and the CPU time is 165 seconds on a Pentium III 600-MHz personal computer. The final experiment aims to show the capability of our method in modeling multiobject scenes completely without a priori knowledge about the topology. The objects to be modeled are three statues, including one with a hole, shown in Fig. 13(a). Figure 13(b) shows the 3D data. Table 1. Result summary of the experiment in Section 5.1 Fig. 7. Experiment for an object with a hole. 64
Fig. 9. Process of topological changes. Fig. 13. Multiple objects. Fig. 10. Process of surface evolving with smoothness. Figures 14(a) to 14(d) show the intermediate results. Here, the single closed surface shrinks as shown in Figs. 14(a) to Fig. 14(c), and then splits into three surfaces while developing a hole as shown in Fig. 14(d). The results at the first stage and the final stage are shown in Figs. 14(e) and 14(f), respectively. The number of vertices is 21,144 and the CPU time is 2276 seconds on the same computer as in the first experiment. Fig. 11. Object with a hole and a rim. Fig. 12. Experiment for an object with a hole and a rim. Fig. 14. Experiment for multiple objects. 65
6. Conclusions In the paper, we have proposed a method of modeling a 3D object of arbitrary topology by using variable-size ball marching. Our algorithm has the advantage of modeling objects fast and easily. It can also be implemented without any knowledge about correspondences between data points and sensor viewpoints, which constitutes the main difference between our method and the statistical level set method [15]. In the future, we will investigate more deeply the problem of designing the criterion of topological consistency between the model and 3D data in building the stopping region. In addition, we must analyze the proposed marching method with greater mathematical rigor. REFERENCES 1. Zha H, Morooka K, Hasegawa T. Active modeling of 3D objects: Planning on the next best pose (NBP) for acquiring range images. Proc Int Conf on Recent Advances in 3-D Digital Imaging and Modeling, p 68 75, 1997. 2. Delingette H, Hebert M, Ikeuchi K. Shape representation and image segmentation using deformable surfaces. Proc Conf on Computer Vision and Pattern Recognition, p 467 472, 1991. 3. McInerney T, Terzopoulos D. A finite element model for 3D shape reconstruction and nonrigid motion tracking. Proc Int Conf on Computer Vision, p 518 523, 1993. 4. Hara K, Zha H, Hasegawa T. Regularization based 3-D object modeling from multiple range images. Proc Int Conf on Pattern Recognition 1998;2:964 968. 5. DeCarlo D, Metaxas D. Shape evolution with structural and topological changes using blending. IEEE Trans Pattern Anal Machine Intell 1998;20:1186 1205. 6. Malladi R, Sethian JA, Vemuri BC. Shape modeling with front propagation: A level set approach. IEEE Trans Pattern Anal Machine Intell 1995;17:158 175. 7. Sethian JA. Level set methods and fast marching methods. Cambridge University Press; 1999. 8. Osher S, Sethian JA. Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton Jacobi formulation. J Comput Phys 1988;79:12 49. 9. Serra J. Image analysis and mathematical morphology. Academic Press; 1982. 10. Arehart AB, Vincent L, Kimia BB. Mathematical morphology: The Hamilton Jacobi connection. Proc Int Conf on Computer Vision, p 215 219, 1993. 11. Kimia BB, Tannenbaum AR, Zucker SW. Shapes, shocks, and deformations, I: The components of shape and the reaction-diffusion space. Int J Comput Vision 1995;15:189 224. 12. Chopp DL. Computing minimal surfaces via level set curvature flow. J Comput Phys 1993;106:77 91. 13. Milne B. Adaptive level set methods interfaces. J Comput Phys, Ph.D. thesis, Department of Mathematics, University of California, Berkeley, 1995. 14. Williams DJ, Shah M. A fast algorithm for active contours. Proc Int Conf on Computer Vision, p 592 595, 1990. 15. Whitaker RT. A level-set approach to 3D reconstruction from range data. Int J Comput Vision 1998;29:203 231. AUTHORS Kenji Hara received his M.S. degree in mathematics and sciences from Kyoto University in 1984 and Ph.D. degree from Kyushu University in 1999. He joined Takeda Chemical Industries Co., Ltd. in 1989, where he engaged in software development as a system engineer. From 1991 to 2000, he was associated with the Mechanics & Electronics Research Institute at Fukuoka Industrial Technology Center. He is currently an adjunctive researcher at the Institute of Industrial Science, University of Tokyo. His current research interests include physics-based vision, 3D object recognition, and automatic geometric modeling. 66
AUTHORS (continued) (from left to right) Hongbin Zha received his B.E. degree in electrical engineering from Hefei University of Technology, China, in 1983, and M.S. and Ph.D. degrees in electrical engineering from Kyushu University in 1987 and 1990. From 1990 to 1991, he was a research associate in the Department of Control Engineering and Science at Kyushu Institute of Technology. From 1991 to 2000, he was with the Department of Computer Science and Communication Engineering, and then the Department of Intelligent Systems, Graduate School of Information Science and Electrical Engineering, Kyushu University, as an associate professor. He is currently a professor at the Center for Information Science, Peking University, China. His research interests include robotics, computer vision, automatic geometric modeling, and sensor-based interaction between robots or robots and environments. Tsutomu Hasegawa received his B.E. degree in electronic engineering and Ph.D. degree from Tokyo Institute of Technology in 1973 and 1987. He was associated with the Electrotechnical Laboratory of the Japanese Government from 1973 to 1992, where he performed research in robotics. From 1981 to 1982, he was a visiting researcher at the Laboratoire d Automatique et d Analyses des Systemes, Toulouse, France. He joined Kyushu University in 1992, and is currently a professor in the Deparment of Intelligent Systems, Graduate School of Information Science and Electrical Engineering. His research interests are in manipulator control, geometric modeling and reasoning, motion planning, and man machine interaction. 67