BUILDING MODEL RECONSTRUCTION FROM DATA INTEGRATION Ruijin Ma Department Of Civil Engineering Technology SUNY-Alfred Alfred, NY 14802 mar@alfredstate.edu ABSTRACT Building model reconstruction has been attracting a great attention from researchers in the geomatics society all over the world. Buildings provide essential information to urban planning, tourism, telecommunication, homeland security, and many others. Much building reconstruction research has been conducted on aerial photographs and LIDAR point data. Some studies are commenced by researchers in last few years to explore the integration of LIDAR and aerial photographs for building model reconstruction. In this paper, a methodology of 3D building model reconstruction will be examined based on the integration of aerial photographs and LIDAR data. The methodology is comprised of two elements. The first one is to reconstruct 3D building models from LIDAR data. Rough building models are the outcome of this step. The second element is to refine the rough models with information derived from aerial photographs. The research will focus on the model refinement process. It will also address the issue of co-registration of LIDAR data and aerial photographs. INTRODUCTION Building models provide essential information to many applications such as urban planning, tourism, and many others. Much building reconstruction research has been conducted on aerial photographs and LIDAR point data. Some researchers have performed experiments trying to take advantages of both LIDAR and aerial photographs by integrating these forms of data. Data integration has been attracting a great attention from researchers. Some promising results have been reported from different studies on data integration. The two technologies, LIDAR and photogrammetry, are treated by researchers as complementary to each other (Baltsavias, 1999). The integration of both technologies is believed to lead to more accurate and complete products (Baltsavias, 1999). Some research works tried to integrate LIDAR data, imagery data and GIS data from different sources for building reconstruction. How to fuse or integrate the data is an important and active research topic. Haala and Brenner (1999) reported their works on building and tree extraction in urban areas using both LIDAR data and imagery data. They used multi-spectral imagery data and LIDAR data to classify buildings, trees and grasscovered areas; then they used the LIDAR data and building ground plan data to reconstruct building models. The ground plan data was used to provide the basic information of a building, especially the boundary information. Stamos and Allen (2000) reconstructed building models using LIDAR data and images obtained from ground platforms. LIDAR data was segmented to identify planar facets. Linear features were extracted from both range data and images. These linear features were used to co-register images with LIDAR data. 3D building models were derived from LIDAR data; and imagery data was projected to building models to contruct a geometric and photogrammetric 3D scene. Because the LIDAR data they used were very dense and highly accurate, fine linear features can be extracted directly from the LIDAR data. Mcintosh and Krupnik (2002) presented their research work on generating accurate surface models. 3D lines from stereo images were used to improve laser-derived DSM. Vosselman and Suveg (2001) used ground plan data and LIDAR data to reconstruct building models. They decomposed a building ground plan into polygon segments. Each segment indicates a planar facet of a building roof; and each segment was used to extract LIDAR points from which the parameters of the planar surface corresponding to the segment can be derived. Derived planar surfaces were analyzed and tied together using CSG operators such as the union operator and the intersection operator. The decomposition is a tricky work because sometimes weird decomposition can be produced. Csathó et al. (1999) proposed a theoretical framework of data fusion for aerial images, LIDAR data, and other multi-sensor images in order to obtain more information for object recognition, especially building reconstruction. They proposed a data fusion scheme that can be performed at different levels, namely the data level, the feature level and the object level. Surfaces would be constructed from LIDAR data using a perceptual organization methodology.
Edges extracted from stereo aerial images would be used as discontinuity lines to match LIDAR surfaces. They proposed that objects could also be analyzed and integrated, which is a kind of fusion at the object level. Csathó and Schenk (2002) performed an experiment on integration of LIDAR, aerial images, and hyper-spectral images for object recognition. Despite the achievements accomplished by researchers, data integration has not been well explored. Further research should be performed to investigate how to integrate data, features and objects at different levels. In the study presented here, a new method will be explored to integrate information from LIDAR and aerial photographs at a feature level for building model reconstruction. The LIDAR data used in the study has a point density of approximately 1 point per square meter, with a vertical accuracy of approximate 15 centimeters and a horizontal accuracy of approximate 0.5 meter. The aerial photographs have a ground resolution of approximately 0.3 meter with a scale of approximate 1 to 20000. BUILDING MODEL RECONSTRUCTION FROM LIDAR Building model reconstruction is to derive CAD building models, which are of vector format. In this study, building models are assumed as polyhedral models, which are composed of planar surfaces. In addition, the boundary of a building is assumed to be rectangular that consecutive boundary line segments are perpendicular to each other. Building roofs and vertical walls are the primitives of building models. They were detected and their parameters were calculated from LIDAR DSM. Building regions were detected first from LIDAR DSM and their outlines were used in building model reconstruction. Details of building detection related to this study can be found in (Ma, 2005). To reconstruct building models, the roofs of a building should be detected first. Surface normal data was used in this study for roof detection. However, the normal of a surface is sensitive to variations in height data, especially when the LIDAR data has a high density. In order to smooth out the divergence in normal data, the mean-shift algorithm was employed to filter the normal data. This filtering algorithm is a mode-seeking algorithm that seeks cluster centers in a feature space. Such a cluster center is called a mode in the feature space. Details of the algorithm can be found in (Fukunaga and Hostetler, 1975; Cheng, 1995; Comaniciu and Meer, 2002). Figure 1 demonstrates the difference between the used normal data before and after the filtering using the mean-shift algorithm. Figure 1. Comparison of normal data before filtering (left) and after filtering (right). After the normal data was filtered, a classification was performed to detect building roofs. Small segments from the classification result were merged to their adjacent large segments, which share the longest boundary with them. Figure 2 shows the classification result and the extracted roofs in vector format. The boundary of a roof was used to retrieve LIDAR points to calculate the parameters of the roof using an orthogonal regression model. Similar to leastsquare constraint, the orthogonal regression minimizes the orthogonal distance from data points to the derived planar surface.
Figure 2. Roof segments from normal data classification (left) and extracted roofs in vector format (right) To reconstruct a 3D building model, the topology of its roofs and vertical walls need to be built after 3D roof surfaces were calculated. This produces an adjacency graph stores the inter-relationship of those 3D surfaces. With such an adjacency graph, adjacent surfaces can be easily retrieved to derive building corners. At the same time, the belonging-to relationship between a 3D surface, which can be a roof or a vertical wall, and its 3D corners can be built. After all building corners were reconstructed, 3D surfaces are built from bounding corners. Consequently, a 3D building model can be reconstructed. Figure 3 presents an example of reconstructing 3D building corners and an example of 3D building model. Figure 3. Reconstructed 3D building corners (the red stars in the left image) and the corresponding 3D building model (right). MODEL REFINEMENT FROM AERIAL PHOTOGRAPHS Compared with imagery data, LIDAR data has poor context information. LIDAR data cannot directly capture sharp linear features. Building models reconstructed from LIDAR data will thus have low geometrical accuracy, especially their bounding boundaries. There is a great potential to improve a building s geometry through data integration with imagery data such as aerial photographs. Although LIDAR data cannot directly capture sharp linear features, some linear features with high accuracy can be derived by intersecting two planar surfaces derived from LIDAR data because the planar surfaces derived from LIDAR points have high accuracy due to a great data redundancy. Instead of being refined, these features can be used as control features to co-register LIDAR data and aerial photographs.
Co-Registration of LIDAR Data and Aerial Photographs The co-registration of LIDAR and photograph is to derive the exterior orientation parameters of photographs in the LIDAR data coordinate system. Exterior orientation can be performed using direct methods and indirect methods. The direct method uses GPS and INS to calculate exterior orientation parameters directly. To improve the internal consistency between LIDAR and photograph, an indirect method was employed to perform exterior orientation for aerial photographs. Control linear features were extracted from LIDAR data and aerial photographs to build a mathematic model for exterior orientation. The condition used for co-registration from linear features is the co-planarity condition. This condition states that the 3D line derived from LIDAR data and its corresponding 2D line extracted from an aerial photograph lie on the same 3D plane which is formed with the exposure center of the photograph. Figure 4 illustrates the co-planarity condition. v O (X 0, Y 0, Z 0, ω, φ, κ) a (x 1, y 1 ) b (x 2, y 2 ) A (X 1, Y 1, Z 1 ) B (X 2, Y 2, Z 2 ) Figure 4. Co-planarity condition of 2D and 3D line segments In Figure 4, line segment ab is the 2D line, line segment AB is the 3D line, and O is the exposure center. The mathematical model of the co-planarity can be expressed in equation 1. v OA = 0 v OB = 0 v = Oa Ob (1) Each pair of control feature provides two equations, OA v = 0 and OB v = 0. To get the solutions for 6 exterior orientation parameters, three pairs of linear control features are the minimum. For better accuracy, more pairs of control features are needed for redundant checks. In addition, the least-square method can be used to improve the solution accuracy. Building Model Refinement The refinement was performed in image spaces. After exterior orientation, ground building models derived from LIDAR data can be projected to image spaces using the perspective projection. These projected building models can provide guidance to locate and match image edges derived from aerial photograph. Figure 5 shows the projected models onto a stereo pair of aerial photographs.
Figure 5. Projected building models onto stereo pair. Projected model lines were refined by matching their corresponding image edges. To achieve this objective, the Canny edge detector was employed to detect edge pixels from aerial photographs. A buffer from a building s boundary was constructed to retrieve a sub-image for process so that the whole image is not necessary to be processed. The detected image edges were then matched with their corresponding projected edges from LIDAR models. During the matching process, a projected model line provides two important evidences to help find its corresponding image edge. The evidences are the location and the direction of the projected model lines. After projected building lines were refined using image information, new corners can be generated by intersecting the refined lines in the image space. In this way, the coordinates of 2D original corners can be updated from the refined lines. Figure 6 illustrates the refinement procedure in image spaces. Figure 6. Model refinement in image space.
In this study, it is required that conjugate corners on stereo images are updated together in order to avoid situations that a corner is updated in one image while its conjugate corner in the other image is not updated correspondingly. After the refinement in stereo image spaces was completed, 3D ground coordinates were calculated from conjugate corner points using the co-linearity condition. While the co-linearity equations are the basic ones applied in space intersection, there is other information that can be applied to derive reliable and high accuracy 3D ground coordinates. The condition is that the ground 3D points should lie in roofs, which are planar surfaces. The planar parameters of roofs derived from LIDAR are accurate because plenty of LIDAR points were used to derive the parameters. Great point redundancy ensures that the derived roof parameters are accurate. Thus, a constrained least-square regression was utilized to perform space intersections using co-linearity conditions. This also ensures that 3D points bounding a roof surface lie on the same planar surface. CONCLUSIONS A new method of building reconstruction from the integration of LIDAR and imagery data was explored in this study. It refines building models in a consistent approach; and it utilizes stereo imagery information and roof constraints so that it can produce reliable and consistent building models. The experiment results show that data integration can take advantage of multi-source data and thus provide more reliable and accurate products. This can be verified by the improved building model accuracy for LIDAR derived building models using imagery information. In this study, a building model is assumed to have a rectangular boundary and planar roofs. Further studies will explore methods of reconstructing curved building roofs and non-rectangular boundary building models. Due to the limitation of LIDAR point data, small features cannot be detected or reconstructed in this experiment. Methods of reconstructing small structures from imagery data will also be studied. ACKNOWLEDGEMENT The author would like to express his sincere gratitude to Mr. Will Meyer from Harris County Flood Control District, and Mr. Elle Lewis Anderson from Brown & Gay Engineers Inc. for helping get the experimental data. The author also appreciates the supports from the GIS and Mapping Lab, especially Dr. Ron Li, and Dr. Raul Ramirez from the Center for Mapping at The Ohio State University. REFERENCES Baltsavias, E. P. (1999). A comparision between photogrammetry and laser scanning, ISPRS Journal of Photogrammetry & Remote Sensing, Vol.54 pp83-94 Cheng, Y. (1995). Mean shift, mode seeking, and clustering, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol.17 (8), pp790-799 Comaniciu, D. and P. Meer (2002). Mean shift: a robust approach toward feature space analysis, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol.24 (5), pp603-619 Csathó, B. and T. Schenk (2002). Multisensor fusion to aid automatic image understanding of urban scenes, http://dfc.jrc.it/doc/csatho%20020624.pdf, visited August 2003 Csathó, B., T. Schenk, D.C. Lee, and S. Filin (1999). Inclusion of multispectral data into object recognition, International Archive of Photogrammetry and Remote Sensing, Vol. 32, Part 7-4-3 W6, Valladolid, Spain, 3-4 June, 1999 Fukunaga, K. and L.D. Hostetler (1975). The estimation of the gradient of a density function, with applications in pattern recognition, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol.IT21 (1), pp32-40 Haala, N. and C. Brenner, 1999. Extraction of buildings and trees in urban environments, ISPRS Journal of Photogrammetry & Remote Sensing, Vol.54 pp130 137 Ma, R. (2005). DEM generation and building detection from LIDAR data, PE&RS, Vol. 71, No. 7, pp847-854 Mcintosh, K. and A. Krupnik (2002). Integration of laser-derived DSMs and matched image edges for generating an accurate surface model, ISPRS Journal of photogrammetry & remote sensing, Vol. 56 pp167-176
Stamos, I. and P. K. Allen (2000). 3-D Model Construction Using Range and Image Data, http://www.cs.columbia.edu/~allen/papers/cvpr2000.pdf, visited June 2002 Vosselman, G. and I. Suveg (2001). Map based building reconstruction from laser data and images, in Baltsavias et al. (edit), Automatic Extraction of Man-made Objects from Aerial and Space Images (III)