Segmentation of Architecture Shape Information from 3D Point Cloud

Transcription

1 Segmentation of Architecture Shape Information from 3D Point Cloud Xiaojuan NING Dept. Comp. Sci. & Eng., Xi an Univ. of Tech., Xi an, China. LIAMA-NLPR, Institute of Automation, CAS, Beijing, China. Xiaopeng ZHANG LIAMA-NLPR, Institute of Automation, CAS, Beijing, China. Yinghui WANG Dept. Comp. Sci. & Eng., Xi an Univ. of Tech., Xi an, Shannxi, China. Marc JAEGER Project DigiPlante, CIRAD, AMAP, Montpellier, France. Abstract Object Segmentation is an important step in object reconstruction from point cloud data of complex urban scenes and in applications to virtual environment. This paper focuses on strategies to extract objects in 3D urban scenes for further object recognition and object reconstruction. Segmentation strategies are proposed according to object shape features. Rough segmentation is first adopted for classification of objects, and further detailed segmentation is implemented for object components. Normal directions are adopted to segment each planar region, so that architectures and the ground can be extracted from other objects. Architectural components are further extracted through an analysis of planar residuals, and the residuals are used to choose seed point for region growing. And meanwhile, the size of segmental regions is used to determine whether or not it includes sparse noisy points. Experimental results on the scene scan data demonstrate that the proposed approach is effective in object segmentation, so that more details and more concise models can be obtained corresponding to real outdoor objects. CR Categories: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling Keywords: Architectures, Segmentation, Point Cloud, Urban Scenes, Residual, Planarity 1 Introduction Object reconstruction from 3D point cloud data becomes a hot research topic for practical applications, such as tree reconstruction and architecture reconstruction. Architecture reconstruction has become widely cared due to its various applications in architectural design, virtual tourism, urban planning, and accident surveillance, etc. However, This is still a challenge because of the shape complexity and the immensity of the data acquired with laser scanner. The terrestrial laser scanning (TLS) provides us a primary data source for more abundant facade information of architectures. A large complex scene often contains different objects, like architectures, ground, trees, bicycles, and cars. Noisy points are often included in the scan data, because complex shape objects normally reflect scan rays irregularly. Therefore, the technique on how to segment or extract the meaningful part becomes a crucial step. In fly-snow2001@163.com xpzhang@nlpr.ia.ac.cn wyh@xaut.edu.cn jaeger@cirad.fr (a) Data 1 (b) Data 2 (c) Data 3 (d) Data 4 Figure 1: Scan data for segmentation general, the work-flow for building reconstruction from point cloud data should include three steps: building segmentation, component extraction through shape information (like planar surface), and model generation. Therefore, we have to separate different objects away from each other, and object reconstruction will be performed next step. In this paper we focus on the segmentation strategies for different objects from a complex data of urban scenes. Four scan data used for segmentation in our paper are displayed in Figure 1. Segmentation is the process of aggregating points with similar attributes or portioning a point cloud data into smaller, coherent and connected regions. The segmental subsets should be meaningful in the sense, so that they correspond to integral objects in the scene (e.g. trees, ground) or object components thereof. In this paper, an efficient method is proposed to separate a building away from other objects considering the planarity and normal consistency for buildings and the ground. Further for detailed segmentation, like architectural components (walls and windows ), different strategies are proposed to tackle different objects. The main advantage of this work is to use the residual of an area to be planar rather than using the measure of confidence rate, so that this work is Residual Based Segmentation (RBS). The overall structure of the paper is organized as follows. Section 2 illustrates related work and representative comparison. Section 3 provides an overview of our work. In section 4, calculated are differential properties, like normal vectors, for point-based plane fitting and for residual of the local planar shape on each point. Rough segmentation is provided in section 5 to extract different objects in the scenes, and further detailed segmentation is constructed for building components. Experimental results and conclusions are discussed in section 6 and 7 respectively.

2 Point Cloud Data of a Large Scene Normal Estimation Differential Property Residual from Plane Fitting Rough Segmentation Seed Point Selection Region Growing Planar Region Segmentation Seed Region Growing Building Part Extraction Object Segmentation Detail segmentation Wall, Windows Roof Tree, Ground, Building Figure 2: An overview of this approach. 2 Related Work Many methods are presented for segmenting the point cloud data in recent years. A comparative analysis of segmentation methods was presented in [Trucco and Fisher 1995] and [Hoover et al. 1996]. 2.1 Classification of related techniques We adopt the idea in [Liu and Xiong 2008] that 3D point cloud segmentation methods are roughly categorized into three types: edgebased, region-based and hybrid segmentation methods. Edge Detection Segmentation. Edge-based methods attempt to detect the discontinuities in the surface that form close boundaries of different segments. Typical examples on the edge-based are three typical methods for edge detection proposed in [Bhanu et al. 1986]. [Fan et al. 1987] implemented the boundary detection using surface curvature. A scan line grouping technique was presented in [Jiang et al. 1996] for fast segmentation of range images[sappa and Devy 2001] for extracting close contours from a binary edge map. A drawback of these methods is that they are sensitive to noisy points. Region-based Method. Region-based approaches can be classified as two categories: bottom-up and top-down. Top-down methods start by assigning all the points to one group, and they are fitted to a surface. As long as a chosen figure of merit for fitting is higher than a threshold, region subdivision is continued [Chen and Chen 2008]. For bottom-up approaches, a number of seed regions or a seed point are chosen first, and then they will grow by adding neighbor points based on some compatibility thresholds. Some of the representative work are reported in [Besl and Jain 1988], [Chang and Park 2001], [Köster and Spann 2000], and [Wang et al. 2003].However, choosing seed regions as well as controlling the growing process would be difficult and time consuming, and the segmental results could be sensitive to the chosen compatibility thresholds. Hybrid Segmentation Methods. Hybrid methods have been developed using both edge-based information and region-based information. [Yokoya and Levine 1989] presented a region segmentation approach based on HK-sign maps and with jump and roof edge maps to obtain a final segmentation. In the approach proposed in [Ghosal and Mehrotra 1993], the initial segmentation is obtained by region-based technique, and it is refined based on the detected edge maps to produce the final segmentation. 2.2 An analysis on typical techniques [Besl and Jain 1988] and [Jiang et al. 1996] are typical approaches for object segmentation. In the work of [Besl and Jain 1988], points are first segmented into 8 surface types based on the curvature of each point, and a region growing process is implemented to refine. This method is sensitive to noisy data, takes many more parameters and is time-consuming even on range images. As for method of [Jiang et al. 1996], scan lines of the range image are splitted into curves, and they are then clustered to represent surfaces. Compared to [Besl and Jain 1988], it is advantageous not only in segmentation quality, but in the running speed also. But it is not suitable to point cloud data since it is based on scan lines. [Dorninger and Nothegger 2007] and [Chen and Chen 2008] are necessarily referred for the segmentation of architectures. [Dorninger and Nothegger 2007] improved the time complexity from O(n 2 ) to O(m) by using a sequential implementation of the clustering algorithm. The method segmented the original points by hierarchical clustering. Only coarse contour can be obtained however, without detail component information. The proposed pipeline in [Chen and Chen 2008] is a very new advance in architecture modeling. The segmentation of this work is based on confidence rate of the local area to be planar, and it is realized through clustering based on planarity. The confidence rate for each point is a critical parameter in segmentation, and region growing method is also used in segmentation. This approach is called as Confidence Based Segmentation (CBS) in this paper. However it may have over-segmentation and under-segmentation, and it is can not be used to other objects, like plants in the scene. We have implemented this method, and a comparison of this approach with our work will be discussed and implemented in subsection Contributions of the proposed approach The major contributions of our work are in two aspects: (1) Two stages as rough and detail segmentation process are proposed to segment objects and architectural components. Rough segmentation is used to extract main objects in the scene based on the consensus of normal vector in the same plane. Detailed segmentation is used as a refined process to extract finer information for object components, which is important for object reconstruction. (2) Our segmentation algorithm is robust to noisy data, fast and spends less memory space. It is efficient not only for the planar surface but other non-planar objects.

3 The residual, a new measure that the local shape around a point is close to a planar area, will be the main factor for segmentation. 3 Overview The main idea of our approach is that architectures and the ground exhibit planarity property, and they can be extracted by a comparison of normal vectors. As to trees with disordered normal vectors, the clustering in [Ning et al. 2009] based on distance and normal differences is adopted. The whole process of this approach is described as follows with a flowchart in Figure 2. (a) Original Point Cloud (b) Normal Distribution (1) Differential Properties. Normal vector of each point in the data is estimated by plane fitting. Then the residual of a point to its locally estimated plane is calculated as the distance of the point to the plane, which represents whether the local shape is flat or not. (2) Rough Segmentation. The point with minimum residual will be chosen as a seed point for region growing. The segmental results will be refined by restricting the size of each cluster, so that noise is filtered consequently. Based on a hypothesis that points from a plant and those from a building are not connected, a region growing method is defined and implemented to segment points into objects. (3) Detailed Segmentation. By making use of the planarity of modern architectures, a detailed segmentation process is proposed to cluster architectural points in different regions. Architectural components, like windows, doors, and walls, are finally segmented. (4) Segmental Results. After (2) and (3), a large scene is segmented into independent clusters, and each cluster corresponds to one object in the large scene. For architectural components, each cluster may represent one planar region or one approximate planar region, and thus the different surface region for the building is obtained. 4 Computation of Basic Shape Information Two quantities are used as the basic shape information. One is the normal vector n(p) on each sample point p. The other is an error ε(p) of the local shape around point p, which is called as residual. Planar regions in the point cloud data may be represented by the same normal or approximated equally normal. The normal vector estimation is obtained by least-squarely fitting a plane to the set of points within its neighborhood, details see in [Ning et al. 2009]. Here the result of normal vector estimation of Data 3 is shown in Figure 3. In addition, in our approach, the neighboring points of a given point are selected by searching for its k points which have minimum Euclidean distance from the given point. And k-d tree algorithm, a binary tree of k-dimensional keys, is adopted to organize the data which can accelerate the algorithm. We have adopted the ANN library (Approximate Nearest Neighbor)which supports data structures and algorithms for both extract and approximate nearest neighbor searching in arbitrarily high dimensions. (c) Plant Crowns Figure 3: Normal Vector Estimation on Data 3 where (x0, y0, z0 ) is the coordinate of centroid P, and εi is the vertical distance from the plane for point pi. The distribution of the residual can reflect which part is flatter. Two examples of residual are given in Figure 4, Figure 4(a) shows the residual of Data 1 and Figure 4(b) shows the residual of Data 3. The corresponding relation of the residual and the color variation represents different planarity information. In Figure 4, the color denotes the variation of residual which vary from blue to red (the saturated blue means the minimum residual and the saturated red means the maximum residual). Therefore, the smaller the residuals are, the flatter the planes fitted by the points and their neighbors are. 5 Object Segmentation Object segmentation is performed in two levels: rough segmentation and detailed segmentation. 5.1 Rough Segmentation In this section, we adopted rough segmentation for different data which often display each characteristic feature. Building and ground often show planarity, other objects such as trees, cars are always different from the planar regions. Thus we can first extract the planar part in the data, and then for the remaining non-planar parts a fusion method is adopted. Seed Points Selection As aforementioned, a fitting plane will be acquired for every point p and its k neighboring points. However, not all the k points will be located on the fitting plane. Thus we can obtain a residual set as εi. For each point there is a corresponding According to normal obtained,we can derive a plane for point p and its neighboring points pi (i = 1, 2 k). Generally, the equation of the plane can be defined as equation (1). n pi d = nx xi + ny yi + nz zi d = 0, (1) where n is normalized, pi = (xi, yi, zi )T and d is the distance from the origin. On the other hand the plane must be through the centroid p and not all the points will be on the plane. There must be some points which has a distance from the plane, thus the plane can be expressed as equation (2). nx (xi x0 ) + ny (yi y0 ) + nz (zi z0 ) d = εi, (2) (a) Residuals for Data 1 (b) Residuals for Data 3 Figure 4: Residual Computations

4 ε which indicated that the neighborhood of p can be safely approximated by a plane or not. The smaller ε i is, the flatter the plane will be. The residual set ε i is ordered as ε 1 < ε 2 < < ε i. Initial Seed Point. The initial seed point is determined by the point with smallest ε, that is to say we will begin the segmentation process from the flattest surface. Succeeding Seed Points. Every point has an index (initially is 0) which indicates whether it is labeled or not. After the region growing of the initial seed point, the remaining unlabeled points are ordered by the ε and the unlabeled point with minimum ε will be chosen. Meanwhile, the neighboring points of the seed whose residual is less than ε th will be put into the seed points array. Rough Segmentation. Since points on a plane may share the same normal orientation, we can extract the planar regions based on the computed normal of each point p, separate the ground away from the large scene, and the surfaces in the building including windows and doors are segmented. Our method is based on the enforcement of the constraint illustrated below: Algorithm 1 Rough Segmentation from Point Cloud Data 1: Initialization: Point List P L = p 1, p 2,, p n; 2: Region List R L = φ; Seed List S L = φ, 3: R L for each point are computed and sorted; 4: While PointList P L φ do 5: Point with minimum r is set as seed point; 6: i.e. P arg minr and P S L,P R L; 7: for i = 0 : S L.size() do 8: Find k neighborhood points for S L[i] 9: n s S L[i].normal; 10: for all k neighborhood points do 11: 12: if n s n p θ T then R L R L p and PL P L \ p; 13: if p.r < r th then p S L; 14: end if 15: end if 16: end for 17: end for 18: end while 19: if R L.size< S t then 20: put the points in R L to another region Z c; 21: end if 22: for every point p in Z c do 23: Find its k neighbor points and their cluster number,and the frequent cluster number is as the number of p; 24: end for 25: Return the final segmentation results. Planarity. Most of the buildings can exhibit planarity, and the points in a cluster should have a smooth surface. i.e. the angle between two normal vectors does not vary too much. To achieve this we define a threshold θ T denoted the angle threshold, which is used to restrict the normal of points in a cluster or a segment. Region Growing. Assigning the points to the region that current seed point belongs to if they are within θ T. If no more seed point can be determined, the point data have been segmented completely. Remove Noise. An analysis on the size of the cluster or segment will be implemented. The size of cluster which is smaller than S t will be reconsidered by the frequent index of its neighborhood points. By this step, the noise can be removed from the segmental results. Our algorithm is described as follows and the detail steps with pseudo-code are also displayed thereinafter. (1) The seed point is first chosen according to the residual explained in section 4, and the minimum of the residual is considered as the (a) Rough Segmentation (c) Segmented Ground (b) Building Part (d) Detailed Segmentation Figure 5: Segmentation Process of Data 2. initial seed points. Our algorithm begins from the initial seed point with initial index labeled 0 which means the first cluster. (2) Find the neighboring points for the current seed point. If the angle between the normal vector of seed points and its neighbor points is smaller than a given threshold, then the point will be assigned to the same cluster with current point. Afterwards, the size of the cluster is estimated and the one less than S t will be disposed. (3) If all the points have been segmented then go to (4). Otherwise select next point with minimum residual as the current seed and if there is no any seed points unlabeled then go to (4), else to (2). (4) The segmental results are obtained according to different index of the cluster. (5) For those clusters or segments whose size is less than S t, evaluate their clusters index by the index of their neighboring points. (6) A new segmentation result is returned. All the points can be divided into different regions after rough segmentation. The scenes are segmented into several regions in which different region belong to different objects (e.g. ground, building, plants) as shown in Figure 5. Figure 5(a) displays the results after rough segmentation. Figure 5(b) illustrates the final result of the building part and Figure 5(c) represents the ground. 5.2 Detailed Segmentation After rough segmentation, several different objects are extracted from the complex scenes such as ground, building, etc in Figure 6 and Figure 7. However, there is another problem that different parts(e.g.windows, doors) of the building can not be extracted. Hence a further detailed segmentation strategy is proposed, and different data there will be different strategies to dispose which can be determined by users manually. The pseudo-code of detailed segmentation algorithm is illustrated in the following, and also the descriptions are given below.

5 Algorithm 2 Detailed Segmentation for Buildings 1: compute the residuals for each point, denoted as r. 2: while PointList is not NULL do 3: Set minr as seed point 4: Find k neighborhood points for seed point 5: for all k points do 6: if n s n p >= θ T then R R + p 7: end if 8: end for 9:end while (a) Rough Results (b) Noise Removal (c) Tree Part (1) For the building part, in order to extract the windows, doors, a normal-based method similar to the method proposed in section 5.1 is implemented. The difference is the selection of the succeeding seed point, since the windows are on the wall it is difficult to distinguish them. The process is illustrated in algorithm 2 in detail. (2) If the rough segmentation results are not satisfying especially for the complex scenes(like Data 3) in which the trees can not be segmented from the ground, then a weighted method [Ning et al. 2009] is adopted to make further improvement. Here the detailed segmentation result is shown in Figure 5(d). The two walls should belong to two different planes and the windows on each wall are extracted. Also we can see that the air-condition on the facade wall which colored red can be segmented. 6 Experimental Results In this section, our segmentation method is experimentally evaluated. Algorithms are programmed with VC++ and OpenGL for display and rendering. All experiments were carried out on a PC with Intel Pentium D processor and 1G RAM. Most of the data in the experiments are obtained by Faro scanner LS 880 HE40. The detail information of data used in our paper are as follows. (1)Data 1 contains points and the scene consists of a building, ground, several small trees and a car; (2) Data 2 includes points in which there are trees, bicycles, ground and building; (3) Data 3 has points with building and ground; (4) Data 4 contains points in which there are several trees in line before the large building. 6.1 Normal Vector Estimation The normal vector is implemented by an estimation method in section 4. In order to validate the effectiveness of the results, distribution of normal vectors for urban scenes is shown in Figure 3. It can be seen that the regularity of normal vector distribution can be used for effective segmentation. Figure 3(a) is raw point clouds of Data 2, where the normal vector are viewed from different angles in Figure 3(b), and normal vector for the plants crown are shown in Figure 3(c). Normal of plant crowns and a small bicycle are distributed in disorder, but it is more regular in the building facades. (a) Rough Segmentation (b) Detailed Segmentation Figure 6: Segmentation on Data 3 (d) Building Part (e) Detail Results (f) Windows Figure 7: Segmentation on Data Segmentation Results Our algorithm is implemented and the segmentation results are shown in Figure 6 and Figure 7. In Figure 6(a), the ground and part of the building are represented with green color, which means that they are classified as one cluster since they connect each other by the stairs. Then different objects are represented with different colors, and the detailed segmentation result is shown in Figure 6(b). Segmentation results for Data 2 are displayed in Figure 5. Figure 7 shows results of segmentation on Data 4, where green, blue, yellow, etc represent different objects. Objects here can be classified into building part, trees and the roof. Figure 7(a) is after rough segmentation, and the noisy points are pruned in Figure 7(b). Building part and other objects are respectively displayed in Figure 7(c) and Figure 7(d). Figure 7(e) also demonstrates the detail results in which the windows are extracted in Figure 7(f). The following abbreviations are adopted in Table 1: TPN (Total Points Number), PBR (Points of Building Regions), POO (Points of Other Objects), and NP (Noisy Points ). Each line in Table 1 shows the number of points after segmentation. It can be seen that our method can remove noisy points automatically by segmentation. In addition, for the results the θ T is evaluated as pi/12 which can control the co-planar condition. S t is evaluated as S t = 50. k for neighboring points is set as k = 30, and r th is determined by the ninetieth residual after sorting ascendingly. 6.3 Comparisons Comparisons of segmentation results on Data 1, Data 4 have been implemented with our method and the approach of [Chen and Chen 2008] in Figure 8. It can be seen that both techniques are available to segment the roof and the ground away in Figure 8(c). But for more detail information, like windows, our method works properly in Figure 8(a). With CBS method, points sampled from components, like opened windows, might be considered as noisy points and could be deleted. In conclusion, RBS method has two advantages over CBS method: (1) more information of the building can be achieved with RBS, such as windows, wall and roof, even air conditions can be separated properly. Two parallel planes with far distance may be clustered as one by CBS however. (2) noisy points can be eliminated accurately with our method, but some useful points may be removed with CBS.

6 Table 1: Data Sets and Results Analysis Data Name TPN. PBR. POO. NP. Data Data Data Data (a) Data 1 with RBS (c) Data 1 with CBS (b) Data 4 with RBS (d) Data 4 with CBS Figure 8: Comparison of Segmentation for Data 1 and Data 4. The effects at Figure 8(a) and Figure 8(b) are with Residual Based Segmentation (RBS), and those at Figure 8(c) and Figure 8(d) are with Confidence Based Segmentation (CBS) of [Chen and Chen 2008]. 7 Conclusions A segmentation algorithm is presented for separating point cloud data in a complex scene into different groups corresponding to different objects or different components. Rough segmentation makes it possible to extract points from each object, like architecture, ground, cars and etc, through the relation of neighbor normal directions and local planarity. Detailed segmentation is to extract different components of objects. Two parameters (S t, θ th ) are constructed to restrict the segmentation process. Residuals are used to determine if a point is in a planar area, so this approach is called as Residual Based Segmentation (RBS). Experimental results has displayed the effectiveness of our proposed approach. Acknowledgements Special thanks to Beijing Zhonghan Instrument Co.Ltd for providing scan data of urban scenes. This work is supported in part by National Natural Science Foundation of China under No , , , and ; in part by the National High Technology Development 863 Program of China under No.2008AA01Z301; in part by Shaanxi Province Technological Project under No.2007F51 and 2008K4-11; in part by the key project of Xi an innovation-supporting plan under No.XY080030; and in part by French ANR Bio-energy EMERGE project. References BESL, P. J., AND JAIN, R. C Segmentation through variable-order surface fitting. IEEE Transaction on Pattern Analysis and Machine Intelligence 10, 2, BHANU, B., LEE, S., HO, C., AND HENDERSON, T Range data processing: Representation of surfaces by edges. In Proc.Int. Pattern Recognition Conf., IAPR-IEEE. CHANG, I. S., AND PARK, R. H Segmentation based on fusion of range and intensity images using robust trimmed methods. Pattern Reconition 34, CHEN, J., AND CHEN, B Architectural modeling from sparsely scanned range data. Int. J. Comput. Vision 78, 2-3, DORNINGER, P., AND NOTHEGGER, C d segmentation of unstructured point clouds for building modeling. In Stilla U et al (Eds) PIA07. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences. FAN, T., MEDIONI, G., AND NEVATIA, R Segmented description of 3-d surfaces. IEEE Transactions on Robotics and Automation 3, 6, GHOSAL, S., AND MEHROTRA, R Segmentation of range images: an orthogonal moment-based integrated approach. IEEE Transactions on Robotics and Automation, 9, 4, HOOVER, A., JEAN-BAPTISTE, G., JIANG, X., FLYNN, P. J., BUNKE, H., GOLDGOF, D. B., BOWYER, K., EGGERT, D. W., FITZGIBBON, A., AND FISHER, R. B An experimental comparison of range image segmentation algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 18, 7, JIANG, X. Y., BUNKE, H., AND MEIER, U Fast range image segmentation using high-level segmentation primitives. In WACV 96: Proceedings of the 3rd IEEE Workshop on Applications of Computer Vision (WACV 96), IEEE Computer Society, Washington, DC, USA, 83. KÖSTER, K., AND SPANN, M Mir: An approach to robust clustering-application to range image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22, 5, LIU, Y., AND XIONG, Y Automatic segmentation of unorganized noisy point clouds based on the gaussian map. Comput. Aided Des. 40, 5, NING, X., ZHANG, X., AND WANG, Y Tree segmentation from scanned scenes data. In Proceedings of PMA09., IEEE Press. RABBANI, T., VAN DEN, H. F. A., AND G, V Segmentation of point clouds using smoothness constraint. In ISPRS 2006 : Proceedings of the ISPRS commission V symposium Vol. 35, part 6 : image engineering and vision metrology, SAPPA, A., AND DEVY, M Fast range image segmentation by an edge detection strategy. In 3D Digital Imaging and Modeling, International Conference on, IEEE Computer Society, Los Alamitos, CA, USA, vol. 0, TRUCCO, E., AND FISHER, R. B Experiments in curvaturebased segmentation of range data. IEEE Transactions On Pattern Analysis And Machine Intelligence 17, 2, WANG, G., HOUKES, Z., JI, G., ZHENG, B., AND LI, X An estimation-based approach for range image segmentation: On the reliability of primitive extraction. Pattern Recognition 36, 1 (January), YOKOYA, N., AND LEVINE, M. D Range image segmentation based on differential geometry: A hybrid approach. IEEE Trans. Pattern Anal. Mach. Intell. 11, 6,