Segmentation of 3-D Range Images Using Pyramidal Data Structures

Transcription

1 Segmentation of 3-D Range Images Using Pyramidal Data Structures Bikash Sabata, Farshid Arman, and J.K. Aggarwal Computer and Vision Research Center, Department of Electrical and Computer Engineering, University of Texas at Austin, Austin, Texas 7872 U.S.A. Abstract Given a 3-D range image of a scene containing multiple arbitrarily shaped objects, we segment the scene into homogeneous surface patches. A new modular framework for the segment& tion task is proposed. In the first module, over-segmentation is achieved wing zeroth and first order local surface properties. The segmentation is then relined in the second module using high order surface representations dictated by the high level vision tasks. The procedure has been applied successfully to many range images obtained from various institutions, five of which are presented here. Introduction In most higher level vision tasks, such as object recognition, motion estimation, navigation, etc., it is important to pariztzon the input data and to extract a set of primitives from the partitions. The primitives may then be used to complete the desired task. In this paper, we address the problem of partitioning data which represent the 3-D coordinates of each point in the scene, obtained using range imaging devices. Specifically, the problem is stated as the following: Given a 3-D dense range image of a scene containing multiple arbitrarily shaped objects, segment the scene into homogeneous surface patches. In general, a segmentation procedure partitions a given image into homogeneous regions. The segmentation should depend on the input data type and on the final representation of the homogeneous regions. These qualifications suggest a framework for segmentation which has two parts: one driven by the local properties of the input data and a second driven by the final region representation (figure ). In this paper, we adopt such a modular framework. The first module of the segmentation procedure groups the image pixels in terms of the local properties derived from the input data. This stage acts as a preliminary segmentation that has to be refined by the second module based on the final vision task being addressed. The low level segmentation procedure uses a pyramid data structure and an iterative algorithm to result in an over-segmented output. The over-segmented results of the low level segmentation module must be merged into homogeneous regions. The merging criterion should depend on the final application of the segmentation results. We assume that the high level task uses a surface-based representation; therefore, the merging has to depend on surface descriptions. The description of the surface depends on the representation; hence, the output of the merging module is a function of the surface representations. This module uses the output of the low level segmentation, the raw input data, and the surface representation of the higher tasks to arrive at a final partition of the scene. The output is a partitioning of the scene into disjoint surface patches. The first module of the algorithm uses four local properties: the surface normal at every point and its three projections onto the zyplane, the yr-plane, and the zr-plane. These projections are equivalent to viewing the scene from three orthogonal directions. The four properties used exploit the zeroth and the first order surface properties. Locally computed higher order surface properties, such as curvatures, are extremely sensitive to noise. To distinguish surfaces that differ in higher orders, the higher order properties are used indirectly in the merging module. Much work has been done in the past on the segmentation of range images. Besl and Jain [4) use curvature labeling to obtain a coarse segmentation which is then refined using bivariant polynomials in a region growing process. Hoffman and Jain [2] cluster the input data into many regions. The regions are then merged based on area, shape, and the result of many statistical measures. Fan, Medioni, and Nevatia [9] combine the extrema and the $er0 crossings of four directional curvature values over several scales. The detected discontinuities are then closed to yield a complete segmentation. Most other work in past segment restrictive object shapes. For example Boulanger and Rioux [SI segment spheres, ellipsoids, and other simple quadric surfaces; Flynn and Jain [lo] segment spheres, cylinders, and planes. The common drawback to most algorithms is their requirement for many empirically determined thresholds. These thresholds depend mostly on the quality and type of the input data; hence, such algorithms can only be used for a small class of input data. For each class of input data, the thresholds have to be re-adjusted. For the algorithm to be independent of its input data, it is important that the thresholds be derived from the data itself. Pymmidal clustering algorithms, used in the first module, have the useful property that they do not depend on arbitrarily or empirically determined, thresholds. The paper is organized as follows. Section 2 describes the outline of the segmentation procedure, and section 3 presents the details of the algorithm. Section 4 illustrates and discusses examples using real range images obtained from various institutions. The last section gives concluding remarks. 2 Overview This section presents an overview of our segmentation scheme. The following sections present the details. Figure shows the overall organization of the segmentation scheme. The procedure is divided into two modules. The first module is the low level segmentation module where the local properties are extracted from the given input data and clustered into homogeneous regions. This module gives an initial oversegmented output. These over-segmented regions are merged in the second module using the surface representations used by higher vision tasks. 2. Low Level Segmentation The low level segmentation module arrives at a preliminary segmentation of the input data using local information. The module is divided into two stages: the preprocessing stage and the pyramidal clustering stage (see Figure ). Preprocessing: The initial clustering is performed using four properties calculated by the preprocessing stage for each point in the range image. The four properties are the surface normal vector and its three projections onto the zyplane, the yr-plane, and the zr-plane. The pr- CH2934-8/90/0000/0662$0.OO IEEE 662

2 Input Data. Ob& normal at pm6 compooeotrsafuturc 2. clusraing for each of four Propeaies 3.addresults e 2. pick best neighbor for cumat qim 3.ifoL,thcnmcrge 4. repeat 2,3 for all regions 5. go to next higher description and repeat fb. 8.pPsnt.tiOII jections are equivalent to the views generated by viewing the scene from three orthogonal directions. Prior to the calculation of these properties, smoothing is performed to reduce noise. Pyramidal Clustering: This stage acts as a preliminary segmentation refined by the next stage based on the final vision task. The pyramidal algorithm is an iterative procedure that clusters pixels with similar properties into groups in a hierarchical manner. Each of the four images generated by the preprocessing stage is used by pyramidal algorithms, independently, to result in four initial segmentations of the input data. The four segmentation outputs are then "added," resulting in a maximally partitioned image. This is the input to the next stage. 2.2 High Level Merging The resulting segmentation from the first stage represents a grouping of the local properties. The second stage of the procedure merges these regions based on a certain homogeneity criterion. The criterion is derived from the final representations of the surfaces as dictated by the high level tasks, This stage can be modified according to the application that uses the segmentation results. Here we use bivariate polynomials of up to fifth degree to represent surfaces. Two adjacent surface patches are merged if parameters of one of the patches, when used to extrapolate over the neighboring patch, results only in a small error. Section 3.3 presents this procedure in detail (see Figure ). 3 Segmentation Procedure In this section we present the details of each of the modules. 3. Preprocessing In the preprocessing stage, the input data is smoothed and used to compute the local properties. The local properties used are the surface normals and its three projections onto the zy-plane, the yz-plane, and the 2%-plane. Besl [3] describes a procedure that uses separable operators to compute the normals for graph surfaces. We extend that procedure for the case of any general surface. Usually it is a good approximation to consider the images obtained from a range scanner as a uniform grid graph surface, but the assumption is invalid for scenes spanning a large area. The uniform grid assumption becomes more inaccurate as the distance from the center of the image increases. The position vectors of the points in the scene are given as a function of the (u,v) coordinates of the parametric (image coordinates) space. If r?, and 2" are the partial derivatives of 2 with respect to U and v, then: z(u U) zu(u, U) R = ( y(u:v) ) ; Ru = ( YU(U,V) ) ;R" = (;:i:::"l). () 4% v) 4 % U) " u,v The derivatives can be computed using a separable convolution operator. To each of the coordinate functions, z(u,u), y(u,u), and z(u,v), a local least squares polynomial function fit is evaluated. This polynomial function is then differentiated to compute the derivatives. Given a function f (U, U), the partial derivatives are evaluated as [3]: fu=du*s*f and f,,=d"*s*f (2) where S is the smoothing operator' and D, and D, are given by: where 20 = -[ 7 IT and d; = -[ IT. Therefore, using equations (2), each of the three components of the partial derivatives zu and d, are computed. Consequently, the normals are computed from the partial derivatives using The surface normal and its three projections form three images, which are used in the next stage. 3.2 Pyramid Segmentation Following the preprocessing module, each of the four generated images are partitioned, independently, by pyramidal algorithms into homogeneous regions. The pyramidal algorithms are divided into three stages. The first process is initialization, where the nodes of the pyramidal data structure are initialized. The next stage is the node linking stage, an iterative stage where the grouping is accomplished. The final stage is tree generation, where the nodes are assigned unique region labels [7,,2]. As the four input images represent different types of data (normal vectors and scalars), the pyramidal algorithm is modified appropriately to account for the variation. The pyramid is a layered arrangement of square arrays. Each layer, referred to as a level, is one fourth the size of the lower layer. The basic element of each of these levels in a pyramid is referred to as a node. The bottom-most level of an h level pyramid (called the base level) is of size 2h x 2h. The nodes in level I are linked to some nodes in level I + and to some in level I -. The nodes in level I + linked to the node n in level I are referred to as the fathers of node n and the nodes in level I - linked to node n are referred to as the sons of node n. For the following discussion, we introduce the following notations: Any arbitrary node in level I is represented by [i,j,fl, abbreviated for convenience to n = [i,j,fl for nodes in level I and to f = [i',j',l+ for nodes in level I +. Also, the nodes in level I - are represented by s = [i', j', I -. Let the set of nodes in level I denoted by XI, then XI = {[i,j,fl} where 0 5 i,j 5 2h-' and 0 5 I < h. Initialization Assuming that the original image is 2h x 2h, the base level is initialized by assigning the pixel values of the image to the 'A binomial filter [3] is implemented to smooth the input data, however any other filter may be used. (3) (4) 663

3 corresponding nodes. The other levels of the h level pyramid, level to level h-, are initialized by taking the averages of a 2(span) x 2(span) area in level I- to generate a node in level I, where the integer span 2 is the span factor [ll], and 5 5 h- is the level being initialized. The span factor determines the number of nodes designated as valid sons. The assumption that the image size is 2h x 2h does not lessen the generality of the typ& of images because any image of size W x H can be padded with NULLS to be made 2h x 2h. Here h is the smallest integer such that 2h 2 maz( W, H). Designating p[i,j, to be the property of node n on the Ith level, initialization is described as: where tu,, = w(i,jf,i,j,l) is the membership function for the valid sons s, (referenced to as i,j,l-l]) of the node n. The nodes [i,j,4 [ are, in turn, used in determining the values of the four father nodes located at (assuming span = 2) [ l(i-)/2j, (i-)/2j7l+i, [ l(i-l)/2j, l(j+l)/2j,l+ll, [ L(i+)/2J, lg-)/2j,j+~l, and [ l(i+l)pj, l(j+)/2j,+ (LzJ designates the integer part of 2). Representing the set of valid fathers for node [i, j,4 as the set {father[i, j,l]}, we have: wnj = w(i, j, i,jf, I + ) = 8 [i, j, I + E {father [i, j, 4) 0 otherwise Node Linking Node linking is an iterative process in which each node chooses its best father [7,3] based on a closeness measurement. The closeness, in property value, between a node in level I and a node in level I + is evaluated using where. is any dot product induced norm. If the input data (Le., the image input in the base level) is in the form of real numbers, then the norm is the standard Euclidean distance norm, and if the data is in the form of vectors (for the case of the surface normals), the norm is v ( = (v. C)*. A weight u,,j between a node n and its candidate father f IS then assigned as follows: For each level l above the base level, the weights u and the node values p[i, j,q are updated in an iteration until Jz reaches a fixed point. Starting from the level above the base level, the node values for each level are computed from the values of the nodes in the level below. For each level the iteration stabilizes and Ja for that level reaches the fixed point before progressing to the next higher level. This process is continued over all the levels until the topmost level is reached. The entire algorithm is independent of the type of.input data. For each different data type, only the definition of the norm in equation (6) changes. Tree Generation The last step of the Pyrddal clastering algorithm is the tree geheration step:.this step use8 the results of the linking step and assigns a region label to each node. Starting from a level H Sh-, a distinct label is assigned to all nodes with distinct property values. Then the nodes on level H - are assigned the I&-.ls d their chosen fathers (i.e.; the faaher with the greatest weight). This process is repeated for all the levels below, each son being assigned the label of its chosen father. At the end of this step, the nodes of the base level are wigned one of the labels of the nodes on the chosen level H, the founding fathers. Note that as one increases the value of H, the total number of possible labels decreaqs, resulting in a fewer number of segments in the image at the base level. Since we ensure that the image is always over-segmented the performance of the segmentation procedure is not dependent on the value of H. The segmentation results from each one of the four images are added to form one oversegmented image. 3.3 Merging The over-segmented results of the low level segmentation stage must be merged into homogeneous regions. The merging module uses the output of the low level segmentation module, the input raw data, and the surface representation of the higher tasks to arrive at a final segmented description. By using the raw input data to guide the final merging prc+ cess the errors and distortions introduced by the low level preprocessing are corrected. Merging Criterion In this paper, we assume that the surfaces are represented as bivariate polynomials of different orders. These representations have many useful properties while being very simple. Besl [3] discusses in detail the relative merits and demerits of different representations. An m*h order bivariate polynomial surface is represented as: PE%+, Once the weights between each node and its fathers are determined, we recalculate the property value of each node, on level I, as follows: U:,.p[i,j,l-l] where u.~ is the weight between the node n and its son s. For tlie singular casea where the &j is zero in equation (7), the weight between that node and the father f becomes one and the other 3 fathers get the weight of zero. In the process of iteration, as nodes and their fathers on the level above become closer or farther in value, all weights approach either one or zero. Using the weights, a descent function, Jz, on the iterates can be defined for each level: (9) where a is the parameter vector, m is the order of the surface, and (z, y) are the coordinates of the points. To merge two adjacent surface patches, the closeness in terms of a bivariate polynomial fit has to be evaluated. The closeness is evaluated by eztrapolating the surface that has been fit to a region in the least square sense, over the neighbors, and then computing the mean squared error of fit. If the error of fit to the clcaest neighbor is less than a threshold E, then the two regions can be said to be close enough to be merged into one region. Using local approximating functions, such as splines, is not suitable here because they have been designed for interpolation and not for extrapolation. The coefficients depend solely on local data points; therefore, only the variations at the region boundaries would be reflected on the extrapolating surface. Merging Algorithm The output of the low level segmentation pr+ cedure gives a preliminary partition of the data. Suppose that the set of regions is represented by {&}. For every region R, there is a set of neighbors {Ni}. The other input to the final merging module is the set of surface representations, {S,,,}, to be used by the high level processes. The 664

4 I mrh surface representation is used by the points belonging to the it' region to obtain a surface patch S,(R,). The parameters of the patch are determined such that the surface patch fits the data in the least squares sense. To measure the closeness between two regions, we define an error measure did,(&, R,) as the mean squared error obtained by eztrapolating the surface S,(&) over the points belonging to the region Rj. Now, two regions, & and R,, will merge if they satisfy. Rj E Nj} 3. dist, I 2. dist, &,& 5 dist,(&,rk) VRk E {Nj} &, Rj < E The test is carried out for each surface representation S,. the merging algorithm consists of Therefore. Select the first surface representation i.e. m =. 2. Compute the surface description S,,,(&) for each region Rj. 3. Find the closest neighbor Rj of 4, i.e., Rj E {Ni} and dist,(&,&) 5 dist,(&,rk) VRr E {Ni}. 4. Merge the two regions if they satisfy the merging criterion. 5. Repeat the above three steps until no further merging takes place. 6. Repeat steps 2-5 for each of the surface representations S, using as input the result of the previous steps. Figure 2: The ongznal range image is on the IeJt and the..dls 0,: t!.s nght. At the end of the merging process, we obtain surface patches which are homogeneous in the sense that every point in the patch is represented by the same set of parameters. Unique labels are assigned to each separate surface patch. These labels can be used by the high level processes to accomplish different tasks. 4 Examples This section presents some of the results obtained. As mentioned earlier, the input data are 3-D (2, U, z) coordinates measured for each point on the scene. This data is the input to the segmentation procedure, the parameters of which are fixed. These parameters are as follows: First, the top most level of the pyramid, H, (section 3.2) is equal to 6. This fixes the total number of clusters in each pyramidal clustering as 6; the final number of segments however, is much higher after the four initial partitions are added (for example, fig. 2 was initially segmented into 208 clusters). Since the goal of the clustering module is to over-segment the input image, H could be smaller in exchange for a higher computational expense in the second module. The final results will remain the same. Second parameter is the fixed point measure for Jz, which is set to (Le., J?") -Jp) I). Third, the surface fitting tolerance level (section 3.3) of the second module, has been set to for all the images. The last two factors are the results of finite precision used in the computers; i.e., they signify the tolerance required in representing the floating point zero. The performance of the algorithm is independent of these factors. Jz converges to the fixed point asymptotically [5]; therefore, the weight changes, Uik, are small after a few iterations and the chosen fathers (section 3.2) remain constant. The last factor is the closeness of fit threshold E of section 3.3. This is set to Physically, the value of E sets a tolerance for the mean distance of each point from the extrapolating surface. This suggests that E is related to the resolution of the measuring system, that is the smallest detectable depth change on the surface; in our case, it is The value of E is not changed for different surface representations. While the same value is used in all examples, E may be estimated directly using each image (see [3]). The first example (Fig. 2), obtained by a looa Technical Arts laser scanner, is a complex range image of several objects. Note the correct segmentations of the different surface types in the same image. The second example is a range image of a model space shuttle obtained from SRI (Fig. 3). Notice that the fuselage has correctly been segmented into three regions, representing the nose, the body, and the exhaust outlet. In addition, the side of the body and the side of the closer wing have been segmented into regions where there have been a change in shape, as defined by the bivariate polynomial representation scheme. The entire back plane has been merged into one region. Figure 3: The original range image is on ihe left and the resulls on the Figure 4 shows the third example obtained from USC. The effect of the smoothing window overlapping over the shape discontinuities is apparent in this image. Notice the transition region from the seat of the chair to the back. The rest of the chair has been segmented correctly. Figure 4: The original range image is on the left and the results on the Next, consider figure 5 obtained using a looa Technical Arts Scanner. The coke can has been segmented correctly into two regions. The screw and the polyhedron have also been segmented correctly. IIowever, the middle region of the oil bottle remains over-segmented. Notice that the coke can in this image to the coke can in figure 2, were not segmented similarly. An explanation for these phenomena is that the extent of the surfaces sampled differ in the two images; i.e, the actual area span of the cylindrical sections is different. A single bivariate polynomial can approximate only a section of a quadric surface; therefore, when a large section of a cylinder is present in the scene, more than one bivariate polynomial is necessary to describe it. A single higher order polynomial may be used for a better approximation, but the sensitivity to noise and quantization errors increase with the order. The over-segmentation of

5 the coke can in figure 2 is an artifact of using bivariate polynomials. With a different surface representation, such as quadrics or conics, the entire surface would be merged into a single region. By using pyramidal algorithms for the low level stage the algorithm is guaranteed to converge [5]. Finally, the pyramidal algorithms may be implemented in parallel [,4]. Acknowledgments We would like to thank several institutions for providing some of the range images: data shown in figure 4 was provided to us by the University of Southern California, figure 3 was provided to us by SRI via the University of Utah, Michigan State University provided the data shown in figure 6. We would also like to thank Dr. Lisa Judge for proof reading the paper. This research was supported in part by ARO under grant number DAAL03-87-K Figure 5: The original range image is on the CeJf and the results on the right Figure 6 shows the final example obtained from Michigan State University. The scene consists of two polyhedrals, lying on top of one another. A third object, consisting of a cylindrical region, is also present. Figure 6: The original range image is on the left and ihe results on the 5 Summary and Conclusions We introduce a modular framework to segment dense range images. Using the framework, a robust segmentation procedure is developed and successfully tested on several range images, six of which are presented here. The success of the procedure over a wide variety of scenes demonstrates the validity of the framework. The scenes contain simple polyhedral blocks to complex collections of different types of curved objects. The framework consists of two independent modules, the first performing low level segmentation and the second carrying out the subsequent merging of regions. The segmentation is accomplished by an iterative pyramidal clustering scheme using zeroth and first order local surface properties at each point of the scene. The merging task uses a surface-bawd representation. The advantages of this procedure are numerous. First, by achieving such modularity, the low level segmentation process is independent of the surface types and descriptions. Similarly, the high level process is independent of the local properties derived from the input data as well as on the method used to achieve the over-segmentation. Second, no restrictions have been placed on the type or size of the objects in the scene. Third, the procedure s dependency on empirically determined thresholds is minimal. This is not true for most existing segmentation schemes. References [l] N. Ahuja and S. Swamy, Multiprocessor Pyramid Architectures for Bottom-Up Image Analysis, Multiresolution Image Processing and Analysis, chapter 3, pp 38-58, Springer-Verlag, New York, 984. [2] F. Arman, B. Sabata, and J. K. Aggarwal, Hierarchical Segmentation of 3-D Range Images, in Proc. of 989 IEEE International Conf. on Sys. Man and Cyb, Cambridge, Ma. (Nov. 5-7), pp 56-6, 989. P. J. Besl, Surface in Range Image Understanding. Springer-Verlag: New York, NY, 989. P. J. Besl and R. C. Jain, Segmentation Through Variableorder Surface Fitting, IEEE tranaactiom on Pattern Analysis and Machine Intelligence, vol. 0, no. 2, (Mar), pp 67-92, 988. P. J. Besl and R. C. Jain, Three-dimensional Object Recognition, ACM Computing Surueys, vol. 7, no., pp 75-45, March 985. P. Boulanger, and M. Rionx, Segmentation of Planar and Quadric Surfaces, Intelligent Robots and Computer Vision: Sizth in Series, D. P. Casasent and E. L. Hall, Eds., Proc. SPIE 848, Cambridge, Massachusetts, (Nov 2-6), pp , 987. P. J. Burt, T. H. Hong, and A. Rosenfeld Segmentation and Estimation of Image Region Properties Through Cooperative Hierarchical Computation, IEEE transactions on sus, man, and cyb, SMC-, No. 2, pp , December 98. T. J. Fan, Describing and Recognizing 3-D Objects Using Surface Properties. Springer-Verlag, New York, New York, 990. T. J. Fan, G. Medioni, and R. Nevatia, Segmented Descriptions of 3-D surfaces, IEEE international Journal of Robotics and Automation, pp , 987. P. J. Flynn and A. K. Jain, Surface Classification: Hypothesis Testing and Planar Estimation. proceedings of Computer Vision and Pattern Recognition, pp , 988. W.J.Grosky and R. Jain, A Pyramid-Based Approach to Segmentation Applied to Region Matching, IEEE transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-8, No. 5, pp , September 986. [2] R. Hoffman and A.K. Jain, Segmentation and Classification of Range Images, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-9, No.5, September 987. [3] T. H. Hong, K. A. Narayanan, S. Peleg, and A. Rosenfeld, Image Smoothing and Segmentation by Multiresolution Pixel Linking: Further Experiments and Extensions, IEEE transactions on systems, man, and cybernetics, SMC-2, No. 5, pp 6-622, September 982. [4] C. Lengauer, B. Sabata, and F. Arman, A Mechanically Derived Systolic Implementation of Pyramid Initialization, work3hop on Hardware Specification, Verification and Synthesis: Mathematical Aspects, Ithaca, NY, July, 989. [5] B. Sabata, F. Arman, and J.K. Aggarwal, Convergence of Fuzzy Pyr* mid Algorithms, Submitted for publication 666