Optics and Lasers in Engineering

Transcription

1 Optics and Lasers in Engineering 51 (2013) Contents lists available at SciVerse ScienceDirect Optics and Lasers in Engineering journal homepage: Integration of LiDAR data and optical multi-view images for 3D reconstruction of building roofs Liang Cheng a, Lihua Tong a, Yanming Chen a, Wen Zhang a, Jie Shan b,c, Yongxue Liu a, Manchun Li a,n a Department of Geographical Information Science, Nanjing University, Nanjing , China b School of Remote Sensing and Information Engineering, Wuhan University, Wuhan , China c School of Civil Engineering, Purdue University, USA article info Article history: Received 1 May 2012 Received in revised form 26 September 2012 Accepted 20 October 2012 Available online 22 November 2012 Keywords: LiDAR data Multi-view imagery Building roof 3D reconstruction abstract An approach by integrating airborne LiDAR data and optical multi-view aerial imagery is presented for automatic reconstruction of 3D building roof models. It includes two main steps: roof point segmentation and 3D roof model reconstruction. A coarse-to-fine LiDAR data segmentation is proposed to separate LiDAR points of a building into a set of roof planar segments, which includes initial segmentation by using point normal estimate, and segmentation refinement by using a new Shrink- Expand technique. A point-based integration mechanism by incorporating the segmented roof points and 2D lines extracted from optical multi-view aerial images is then proposed for 3D step line determination, with which 3D roof models are reconstructed. The experimental results indicate that the proposed approach can provide high-quality 3D roof models with diverse roof structure complexities. & 2012 Elsevier Ltd. All rights reserved. 1. Introduction The combination of Light Detection And Ranging (LiDAR) data and optical images has been increasingly focused for many engineering applications, such as 3D building reconstruction. Optical stereo image technology has been used for three-dimensional compilation of urban objects for many years. Although delivering accurate results, its major disadvantage is that the automation level of this optical stereo-based process [1]. It should be noted that building that has only planar roofs is a quite common assumption. Airborne LiDAR technology is able to provide data of directly measured three-dimensional points, which has been an important way for the derivation of 3D building models. However, compared with optical aerial stereo images with a spatial resolution of a centimeter level, airborne LiDAR technology currently only has the ability to collect 3D points with a spatial resolution of 1- m level [2]. Therefore, in contrast with optical products, it is hard to reach the same level in geometric accuracy and details for the building roof models derived from LiDAR data, due to these models are restricted by the ground resolution of LiDAR data and its process algorithms. The goal of this study is to synthetically utilize the complementary data characteristics of LiDAR data and optical aerial images for the derivation of more reliable and accurate 3D building roof models with a high automation level. Among most of the current related studies, the objectives of data integration involve building boundary refinement, building extraction n Corresponding author. address: limanchun_nju@126.com (M. Li). improvement, texture modeling, and model validation. However, few approaches focus on improving reliability and geometric accuracy of the reconstructed 3D roof structures. Using LiDAR data alone is still the main solution for the most of studies in 3D roof reconstruction. Their common idea is to develop methods for LiDAR data analysis to segment LiDAR roof points and then to reconstruct 3D roof planes from these segmented points. Optical multi-view aerial images with available complementary information (e.g., very high spatial resolution, highly rich texture information, multi-view observations) are still not used enough to assist this 3D roof reconstruction processes. Once used appropriately, information extracted from optical multiview aerial images can help to increase the accuracy and reliability of roof point segmentation. Furthermore, the 3D reconstruction quality of building roofs, especially step-style roof structures, can be effectively promoted, which is significantly influenced by irregular point spacing and the performance of the commonly used boundary regularization algorithms in only LiDAR-based approach. Therefore, for automatic reconstruction of 3D roof models with reliable structures and accurate geometric position, a new approach by integrating airborne LiDAR data and optical multi-view aerial imagery is proposed, which consists of two main steps: roof point segmentation and 3D roof model reconstruction. 2. Related work The focus of combining LiDAR data and imagery for 3D building reconstruction has been raised in recent years. To provide a better representation of surface discontinuities, Mcintosh and Krupnik [3] /$ - see front matter & 2012 Elsevier Ltd. All rights reserved.

2 494 L. Cheng et al. / Optics and Lasers in Engineering 51 (2013) produced the digital surface model from LiDAR data and then merged edges detected and matched in this aerial images and digital surface model. In the approach proposed by Ma [4], roof planes of a building were derived based on a clustering algorithm, and the refinement of building boundaries was performed by using aerial stereo imagery. Sohn and Dowman [5] collected rectilinear lines around building boundaries, and then obtained a full description of building boundaries by merging polygons. Cheng et al. [2] proposed an approach by integrating aerial imagery and LiDAR data to reconstruct 3D building models. In this approach, an algorithm for determination of principal orientations of a building was introduced and 3D boundary segments were then determined by incorporating LiDAR data and the 2D segments extracted from images, a strategy including automatic recovery of lost boundaries was finally used for 3D building model reconstruction. The focus of this study is to improve the quality of building boundaries, not building roofs. This study has also given a detailed review on techniques of integrating LiDAR data and images for 3D building reconstruction. Some other related approaches were introduced by [6 15]. As discussed in previous section, according to the data integration objective, these reported studies can be classified into four main types: the data integration to refine building boundary especially by using information derived from very high resolution images, the data integration to improve results of building segmentation and extraction by using information from both two datasets, the data integration to implement photo-realistic 3D modeling by using image for texture mapping, and the data integration for modeling checking by cross-validation of two datasets. However, few approaches focus on how to advance the quality of 3D roof models, especially in the reliability and geometric accuracy of detailed 3D roof structures, by integrating LiDAR data and images. Using LiDAR data alone has almost attracted most of work in 3D roof reconstruction. Dorninger and Pfeifer [16] determined planar regions by hierarchical clustering and performed the determination of regularized building outlines from point clouds, and then reconstructed a building model based on the assumption that the model shall be the best approximation of the given point cloud. Sohn et al. [17] used a Binary Space Partitioning (BSP) tree for reconstructing polyhedral building models from airborne LiDAR data, in which the core is to globally reconstruct geometric topology between adjacent linear features by adopting a BSP tree. In the solution framework proposed by Sampath and Shan [18], the surface normals of all planar points were clustered with the fuzzy k-means method and a potentialbased approach was then used to optimize this clustering process; an adjacency matrix was afterward formed to express the relationship of the segmented roof planes for building roof reconstruction and an extended boundary regularization approach was finally developed to achieve topologically consistent and geometrically correct building models. Kim and Shan [19] presented a level set-based approach for building roof modeling from airborne laser scanning data, in which roof segmentation was performed by minimizing an energy function formulated as multiphase level set and 3D roof models were reconstructed based on roof structure points derived by intersecting adjacent roof segments or line segments of building boundary. Some other approaches associated with the 3D building reconstruction using LiDAR data were also presented [20 22]. 3. Integration of LiDAR data and optical images for roof point segmentation Focusing LiDAR points belonging to one building, the study of this section is to segment these LiDAR points into a set of segments which uniquely represent roof planar surfaces. A coarse-to-fine segmentation is performed, including initial roof segmentation and roof segmentation refinement. The initial segmentation part provides coarse but reliable segmentation results from LiDAR data analysis by using point normal estimate. The refinement part integrates LiDAR data and aerial images to obtain more reliable and accurate point clusters by using a new Shrink-Expand technique. The basic idea behind the sequence of coarse to fine comes from the specific data characteristic of these two datasets. In the Fig. 1. Initial roof segmentation. (a) Triangle network construction, (b) The detected edges by adjacent triangle analysis, (c) Point normal estimate based on local triangle clusters, (d) The detected edges after point normal analysis, (e) Wall line removal, (f) Short line removal (marked by circles), and (g) The initial roof planar segments.

3 L. Cheng et al. / Optics and Lasers in Engineering 51 (2013) first step, LiDAR data, which has the characteristics with directly measured height values, are used for average normal analysis of points to provide a coarse but relatively reliable segmentation results. In this step, only LiDAR data is used without aerial images, because the reliability of segmentation may be reduced and complexity of computation may be enlarged, if the aerial images with very high spatial resolution and highly complex texture information are used inappropriately. With support of the initially segmented results, both LiDAR data and aerial images are used for providing more accurate segmentation, by deeply excavating the rich information of aerial images Initial roof point segmentation from LiDAR data A Delaunay triangle network is generated from the LiDAR points belong to one building, shown in Fig. 1(a). If angle value between two adjacent triangles is larger than the threshold (451 in this study), the connected edge between two triangles is labeled, which may be a ridge line or a step line. Fig. 1(b) shows the labeled edges by checking all triangle pairs in this triangle network. Due to the inherent data noise, this process may always lead to overdetection results, which means some lines in this triangle network are wrongly considered as the ridge lines or step lines. To reduce the influences of LiDAR data noises, point normal estimate based on local triangle clusters is introduced for removing wrongly detected edges. For Point P i in Fig. 1(c), its neighboring triangles (T 1,T 2,T 3,T 4,T 5 ), which called its triangle cluster, can be found by analyzing their topological relationship. Based on all vertices of this triangle cluster, a local fitting RANSAC plane is estimated. The principle of this RANSAC algorithm is to find the best plane among these 3D vertices. Three points are randomly selected for calculating the parameters of the corresponding plane. Then all points of the vertices belonging to the calculated plane are detected, according to a given threshold. These procedures are then repeated N times. The obtained result in each iteration is compared with the last saved one. If the new result covers more points, then the saved result is replaced by the new one; otherwise, the iteration is stopped. The normal value of P i is then assigned by the computed normal value of this RANSAC plane. For the labeled edges in Fig. 1(b), two adjacent triangles associated to it are detected, and then two corresponding points P i, P j of this labeled edge are found. The angle difference between these two point normals is computed. If this angle difference is smaller than the threshold (201 in this study), this edge is unlabeled as a regular line (not a ridge line or step line). Fig. 1(d) shows the remained edges by the point normal analysis. Further processes include wall line removal, short line removal and suspended line removal. In Fig. 1(e), these wall lines are detected by analyzing two parameters. One is the height difference of two endpoints belonging to this line. Another is the angle value between this line and the horizontal plane. After removing wall lines, short lines (marked by circles in Fig. 1(f)), and suspended lines, the point clusters surrounding by remained edges represent the roof planar segments. Each roof planar segment is labeled in Fig. 1(g), respectively Segmentation refinement based on LiDAR data and optical images A Shrink-Expand technique is proposed for refining the above initial roof segmentation. The above section can lead to initial but relatively reliable results for roof patch segmentation, however, due to the inherent LiDAR data noises and the complexity of building roof structures, some problems may still exist during the initial segmentation process. Especially, near the edge areas of roof planes, a few wrong detections and/or missed detections could happen. For Instance, some points located in trees that are very close to buildings may be wrongly segmented into roof planar segments. In addition, some points belonging to a small roof structure of a building may be not kept as roof points during segmentation process. Therefore, to overcome these problems, a Shrink-Expand technique is proposed. The Shrink-Expand technique includes two steps: Shrink and Expand, in which the core is a suspicious point removal and reevaluation strategy. The basic idea of Shrink is suspicious point removal, which is to remove the suspicious points, even including a few correct points, near edge areas of roofs. The idea of Expand is candidate point re-evaluation in reliability, which is to find and accept the missed but correct points as many as possible based on the Shrink results. Therefore, the principle of Shrink-Expand process is to advance the reliability meanwhile prevent information loss. This process is applied to each roof plane one by one Shrink Step A: grid-based point index establishment. A grid with a size of three to five times average point spacing is built for covering the segmented roof patches. Fig. 2(a) illustrates one roof plane covered by this grid. This grid is also used to improve the index speed of searching adjacent points. Step B: edge area detection. For each roof planar segment, its edge areas are detected. Based on the relationship between points and this index grid, for all points in a grid, there are four possibilities: points labeled as non-building (non-building points are used here), points labeled by one roof cluster, points labeled by more than one roof clusters, and points labeled by non-building and roof clusters. If the member points of a grid are identified as the late two types, this grid is taken as an edge grid, shown as black grids in Fig. 2(a). Step C: points in edge grids removal. A buffer is established based on the edge grids (Fig. 2(b)). For one roof plane, all points belonging to its edge grids or their buffer grids are removed. Fig. 2(c) illustrates the roof planar segments after point removal Expand Step A: baseline selection. Based on the remained points in Fig. 2(c), a Delaunay triangular network is generated, shown in Fig. 2(d). A baseline, which is a boundary in this triangle network, can be found by analyzing the relationship between lines and triangles. If a line has both left and right triangles, it is not a baseline. Otherwise, it is a baseline (Fig. 2(e)). Step B: candidate point determination. For the subsequent Expand process, candidate points can be selected based on the selected baselines. Before that, the region for searching the candidate points needs to be determined. In Fig. 2(f), one baseline is selected and its two endpoints are shown as bright points. Based on the aforementioned grid-based point index, we can know which grids contain these two endpoints. Furthermore, their eight neighboring grids are taken as the searching region. The frame in Fig. 2(f) shows the region for searching candidate points based on this baseline by merging 16 grids. The candidate points are those points in the black frame and beyond the triangle network. Among all candidate points, the point which has the largest angle with the baseline is considered as the highest priority, labeled as Point d in Fig. 2(f). Step C: dynamic triangle propagation. Afterward, a new triangle is created with this point and this baseline. This point

4 496 L. Cheng et al. / Optics and Lasers in Engineering 51 (2013) Fig. 2. The Shrink-Expand process for roof segmentation refinement. (a) Grid-based point index establishment and edge grids of each roof patch detection, (b) Buffer zone of each grid generation, (c) Shrink process by removing suspicious points in buffer zone, (d) Triangle network construction based on the remained points, (e) Baseline selection (cyan lines), (f) Expand process by dynamic triangle propagation from a baseline to a candidate point, (g) The refined roof planar segments by Shrink-Expand process, (h) A comparison to illustrate the refined results (points in the box of left image are removed in the refinement results). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) can be checked whether it is reliable or not by a criterion. This criterion is introduced later in details. If the point cannot reach the criterion, delete it and find a new candidate point in the searching region to create a new triangle; otherwise, update the triangular network by adding the new triangle. The two edges of this accepted triangle are considered as new baselines. Step D: repeat Step B and Step C, triangle propagation stops until no new baselines can be found. The criterion for judging the reliability of a candidate point follows a sequence of three aspects: (a) slope, (b) spectral information, and (c) texture information. For a candidate point, slope changes between it and its neighbors are considered at first. If the change is not consistent with its adjacent regions, this candidate point cannot be accepted as a new member of the original triangle network. Similarly, spectral information and texture information are then considered orderly. If a candidate point can pass the following three checks, it is accepted as a reliable point and is added into the triangle network. Slope: Fig. 2(f) illustrates a new triangle created from a baseline (between Point a and b) with a candidate Point d. Another Point c lies at the left side of this baseline. We calculate the average normal of Point c using aforementioned method. The angle between normal vector of Point c and this new triangle is computed. If this angle value is smaller than the threshold (701), transitional zone from the original triangles to this new triangle is considered to be not smooth and the candidate point cannot be accepted. Otherwise, Point d passes the slope check. Spectral information: Much wrong segmentation happened due to trees are very close to buildings (Fig. 2(h)). Spectral information is used to deal with this problem. For Point d in Fig. 2(f), it is easy to find the corresponding place in an aerial image after the registration of LiDAR data and aerial images. Since the orientation parameters of the aerial images are known, LiDAR points would be directly projected onto the aerial images by means of the collinearity equation. The aerial imagery covers three bands (red, green, and blue). Two parameters are considered: the ratio of blue band to red band and the sum of three band reflectance. If the ratio is larger than the threshold (1.3 in this study) and the sum value is smaller than the threshold (210 in this study), the candidate point d is considered as non-building point and cannot be accepted. Otherwise, Point d passes the spectral check. Texture information: Entropy is used as a measurement to represent texture information. For Point d, its entropy value is compared with the value of its neighboring region that are

5 L. Cheng et al. / Optics and Lasers in Engineering 51 (2013) represented by Point c in Fig. 2(f). By taking Point d as a center, a rectangle window (with a size of 1.5 m 1.5 m, average point spacing, in this study) is created in the corresponding aerial image. Eq. 1 is used to convert a color image to a gray image. Eq. 2 is used to calculate the entropy of this window, which is taken as the entropy value of Point d. Similarly, the entropy value of Point c is computed. The difference entropy value between Point d and c is calculated. If the difference value is larger than the threshold value (3.5 in this study), this candidate point is rejected. Otherwise, Point d passes the entropy check. Gray ¼ Rn0:299þGn0:587þBn0:114 ð1þ H ¼ XM P ij X N i ¼ 1 j ¼ 1 P ij log 2 P ij refers to the probability of various gray values in the whole image. Fig. 2(g) illustrates the segmentation refinement results by the Shrink-Expand technique. Fig. 2(h) shows a detailed comparison to illustrate the refined results, in which the wrongly detected points (points located in trees in left image) are removed in the refinement results. 4. Integration of segmented roof points and optical images for 3D roof model reconstruction 3D roof models are reconstructed in this section by using the segmented roof points and 3D roof lines derived from optical images. 3D roof lines include 3D boundaries, 3D roof ridge lines, and 3D roof step lines. Among them, 3D boundaries are determined using the approach proposed by Cheng et al. [2]. The brief introduction is as follows. The principal orientations of a building are estimated. A dynamic selection strategy based on LiDAR point density analysis and K-means clustering is then used to identify ð2þ boundary segments from non-boundary segments. 3D boundary segments are finally determined by incorporating LiDAR data and the 2D segments extracted from multi-view imagery. 3D roof ridge lines can usually be derived by intersecting two adjacent planes, since all of roof planes are detected by the above segmentation process for each building. The novelty of this section is on 3D step line determination. A step line is produced where two roof planes are adjacent to each other with a large height discontinuity but without an intersected line. The step lines are commonly extracted by regularizing their boundaries [13]. The main limitation of this method is that it is hard to derive highly accurate step lines with good details only based on LiDAR data, due to the nature of discrete and irregular data acquisition and its relatively low spatial resolution. By addressed this problem, this section integrates the segmented roof points and aerial images to determine 3D step line with accurate geometric position and good details, thus benefiting to improve the quality of 3D roof reconstruction D roof step line extraction from optical images A building image is created at first using the method proposed by Cheng et al. [2], for reducing the complexity of line extraction in an aerial image with high resolution. Based on the LiDAR points belonging to a building, a bounding rectangle (BR) and a corresponding raster image of this building can be created. The background of Fig. 3(a) is the building image by cutting and filtering an aerial image using the BR and the raster image, respectively. Afterward, the Edison detector [23] is used as the edge detector and line segments are then extracted using Hough Transformation, shown in Fig. 3(b). The extracted line segments are automatically separated into two sets: step line segments and non-step line segments, by analyzing the relationship between the extracted line segments and the segmented roof points (Fig. 3(c)). Two rectangle boxes with a certain width are generated along two Fig. 3. 2D step line extraction. (a) A workspace image and 3D roof planar segments, (b) Line segment extraction from the image in (a), (c) Overlay of the extracted segments and the segmented points, (d) The left and right polygons belonging to a line segment, (e) The selected roof lines (red) and the boundaries (black). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

6 498 L. Cheng et al. / Optics and Lasers in Engineering 51 (2013) orthogonal directions of a line segment which is invoked by the method proposed by Sohn and Dowman [5]. Fig. 3(d) shows two kinds of rectangular boxes created for each segment. If points are found in both boxes, the line segment is taken as a candidate step lines, because the line segment surrounded by LiDAR points should locate on the roofs. Furthermore, the segment is taken as a step line segment, only when points belonging to left box are attributed with one roof cluster and points belonging to right box are attribute with another roof cluster, as illustrated in Fig. 3(d). Fig. 3(e) illustrates the extracted 2D step lines (gray) and the boundaries (black) determined by Cheng et al. [2] D step line determination from 2D lines and 3D points A point-based integration mechanism by combining 2D lines derived from optical images and 3D points derived from LiDAR data is proposed for 3D step line determination. It includes a sequence of transfers 2D line - 2D point - 3D point - 3D line. In first transfer 2D line to 2D point, 2D points are used to represent the extracted 2D lines (Fig. 4(b)) in an aerial image by simply taking two endpoints of this line. The task of the second transfer 2D point to 3D point is to determine a 3D point by a 2D point. The task of the third transfer 3D point to 3D line is to obtain a 3D line from 3D points. In the second transfer, a single point-based photogrammetric technique is introduced to determine a 3D point by a single 2D point. In Fig. 4(c), l is an extracted 2D line in an aerial image. Point p 0 and Point p 1 are the two endpoints of this line. For Point p 0,the nearest LiDAR point can be found by overlaying it with the LiDAR points projected into the aerial image by collinearity equation. The elevation value Z of this nearest LiDAR point is taken as the elevation value of Point p 0. The task of this second transfer is to determine the 3D point coordinate P 0 (X, Y, Z) in object space by using a single 2D point coordinate p 0 (x, y) in image space. Image a ray going through theexposurecenters of a camera and the Point p 0 in an image toward a plane with height value Z in object space. The 3D Point P 0 can be obtained by the ray intersected with the plane. By collinearity equation of a ray (Eq. 3), with the known three parameters ox, y, Z4, the unknown two parameters ox, Y4 can be calculated. That is to say, Point P 0 with a 3D coordinate (X,Y,Z) is determined. x x 0 ¼ f m 11ðX X L Þþm 12 ðy Y L Þþm 13 ðz Z L Þ m 31 ðx X L Þþm 32 ðy Y L Þþm 33 ðz Z L Þ y y 0 ¼ f m 21ðX X L Þþm 22 ðy Y L Þþm 23 ðz Z L Þ m 31 ðx X L Þþm 32 ðy Y L Þþm 33 ðz Z L Þ This equation map the 3D real world objects (X,Y,Z) to 2D image observations (x,y). Parameters of inner orientation are (x,y,f). The three angle parameters of the exterior orientation are implicitly defined in the coefficients m ij and the location of the exterior orientation are given by the terms (X L,Y L,Z L ). In the third transfer 3D point to 3D line, multi-view aerial images (Fig. 4(a)) are used to provide multiple sets of 2D lines corresponding to a real line, shown in Fig. 4(d). Two endpoints of an extracted 2D line in an aerial image can be transferred to be two 3D points in object space. Therefore, multi-view aerial images can provide many 3D points for one real step line in object space. All these 3D points are involved to construct a new 3D line by using a RANSAC-based line fitting, shown in Fig. 4(e). In those endpoints, there may exist inliers and outliers. RANSAC is here selected to construct the new segment, which can lead to the more robust results in comparison with a least squares method. The contribution of multi-view aerial imagery is to increase the reliability and accuracy of 3D line determination. Fig. 4(f) illustrates the final determined 3D step lines (gray) surrounding by 3D boundaries (black). ð3þ Fig. 4. A point-based integration mechanism for 3D step line determination. (a) A building covered by multi-view aerial images (four views), (b) 2D line to 2D point by selecting the endpoints from the extracted 2D lines, (c) 2D point to 3D point by the single point-based photogrammetric technique (S is Exposure Center, O is projection center, l is an image line, p0 and p1 are image points, L is an object line, P0 and P1 are object points), (d) Four sets of 2D lines extracted from four aerial images shown by different line styles, (e) 3D point to 3D line by RANSAC-based line fitting, and (f) The final determined 3D step lines (red) surrounding by 3D boundaries (black). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

7 L. Cheng et al. / Optics and Lasers in Engineering 51 (2013) D roof model reconstruction from 3D lines and 3D points 3D roof models are reconstructed by the segmented roof points, the determined 3D step lines, 3D ridge lines, and 3D boundaries. The main problem of this 3D reconstruction process is that the topology relationships of these line segments are lost. The SMS (Split-Merge-Shape) method proposed by Rau and Chen [24] is modified to automatically recover the topology relationships between these 3D segments. A polygon is surrounded by the boundaries (black) in Fig. 4(f). Each roof line segment (gray lines in Fig. 4(f)) is extended to split the polygon into two parts, shown in Fig. 5(a). Many small polygons surrounded by both the extended lines and boundaries are created. These small polygons are compared with their neighboring polygons. If the points in two adjacent polygons are attributed with same label, these two polygons are merged. Fig. 5(b) illustrates the merged polygons, which represent the roof structures of this building. For each polygon, RANSAC algorithm is used to fit a plane based on LiDAR points belong to it. Fig. 5(c) illustrates the reconstructed 3D roof models and their corresponding LiDAR points. 5. Evaluation An experimental region with an area of 2000m 2000 m, containing many buildings with different roof structures, different sizes, different orientations, and different roof texture conditions, are selected to validate the effect and applicability of this approach. Fig. 6(a) shows LiDAR data, which have average point spacing of 1.0 m, with a horizontal accuracy of 0.3 m and a vertical accuracy of 0.15 m. Optical aerial images with known orientation parameters are collected at the same region, illustrated in Fig. 6(b). Each of them is 3056 pixels by 2032 pixels, with a spatial resolution of 0.05 m. 3D building models constructed manually are taken as reference data to compare with the 3D models reconstructed by this approach. Autodesk 3DMax 2010, a powerful software for 3D modeling, is used for manual operation to construct 3D building models. In addition, trueorthophoto is available for checking quality of the reconstructed 3D models. The quality of the reconstructed 3D rooftop models is evaluated in two aspects: qualitative analysis (visual appearance) and quantitative analysis (correctness, completeness, and geometric accuracy) Visual appearance Fig. 7(a) illustrates the reconstructed 3D building roof models in the experimental area. Six models with different roof styles in Fig. 7(a) are shown in detail in Fig. 7(b). From Fig. 7, we can find that the proposed approach can deal with buildings with various complex roof structures. By visually comparing the models reconstructed by the proposed approach with reference data (including the models derived by manual operation, true-orthophoto, and aerial images), most of the reconstructed 3D models have a high coincidence with the reference data, which means the proposed approach can lead to reliable 3D building models with Fig. 5. 3D roof model reconstruction. (a) The splits by extending the roof lines in Fig. 4(f), (b) The recovered roof lines (gray) by merge processes, (c) The reconstructed 3D roof models and the corresponding LiDAR points. Fig. 6. LiDAR and optical images in the experimental area. (a) LiDAR data (different elevation values shown by different gray), and (b) LiDAR data and optical aerial images shown by rectangles.

8 500 L. Cheng et al. / Optics and Lasers in Engineering 51 (2013) Fig. 7. 3D building roof model reconstruction. (a) The reconstructed 3D building roof models in the experimental area, and (b) Six models with different roof styles in (a). complex roof structures. Region 1 and 2 are selected in the experimental area for the subsequent quantitative analysis Correctness and completeness The focus of this analysis is to estimate the correctness and completeness of the reconstructed roof planes in these 3D models. The correctness and completeness of 3D roof planes are calculated according to Eq. (4). Completeness ¼ TP TPþFN Correctness ¼ TP ð4þ TPþFP where TP is the number of true roof planes, FP is the number of wrong roof planes, and FN is the number of mis-detected roof planes. The roof planes derived by manually modeling are taken as reference data. By overlaying the reconstructed roof planes and reference data, the overlapped area of them is calculated. If the ratio of this overlapped area to area of the reconstructed roof plane is larger than 80%, the reconstructed roof plane is taken as true one. Otherwise, it is considered as a wrong one. If a roof plane is not be detected by the automatic process, we call it a mis-detected roof plane. Table 1 lists the correctness and completeness of the reconstructed 3D roof planes in Region 1 and 2. In Region 1, the correctness and completeness of the reconstructed 3D roof planes are 91% and 89%, respectively. In Region 2, the correctness and completeness of the reconstructed 3D roof planes are 96% and 96%, respectively. From this statistics data, compared with the results in Region 1, the proposed approach provides better results in Region 2, due to the buildings in this area have relatively simple roof structures. Furthermore, to find the problems in this proposed approach, we compare the 3D roof models derived by the proposed approach and manual modeling operation. Fig. 8 illustrates their differences: (a) some detailed planes may be reconstructed in wrong way by automatic operation, shown by A labels, (b) some artifacts, especially some small roof structures, may be fictionally created by the automatic operation, shown by B labels, (c) some tiny structures may be lost by the automatic operation, shown by C labels Geometric accuracy Another concern is the geometric accuracy estimation of these 3D roof planes. It is difficult to estimate geometric accuracy in vertical direction, because the manually constructed 3D models without enough height accuracy cannot be taken as reference data for vertical accuracy assessment. So the focus of geometric accuracy is on the horizontal accuracy estimation. The trueorthophoto is taken as reference data. Based on the analysis of

9 L. Cheng et al. / Optics and Lasers in Engineering 51 (2013) correctness and completeness, a number of correctly reconstructed roof planes are selected. For a roof plane, each vertex is compared with its corresponding point in reference data in horizontal directions, by overlaying the reconstructed 3D roof planes and true-orthophoto. For one vertex, we measure the distance from it to its reference point (the corresponding point in true-orthophoto). In Region 1 and 2, 25 and 20 vertices are selected, respectively. According to the statistics values of the selected vertices, mean error, RMSE, and maximum error are calculated for checking the accuracy of roof planes. Table 2 lists the geometric accuracy of the reconstructed 3D roof planes. Illustration of error distribution for each selected vertex is shown in Fig. 9. The short bold lines refer to errors of the vertices, which are shown as vectors given the length and orientation. In Region 1, mean error, RMSE, and Maximum error MAE are 0.30 m, 0.39 m, and 0.74 m, respectively; in Region 2, mean error, RMSE, and Maximum error MAE are 0.26 m, 0.31 m, and 0.63 m, respectively. All statistics values are much smaller than the average point spacing of LiDAR data used in the experiment, which benefited from the integration of multi-view aerial images. From Table 2 and Fig. 9, it is clear that this approach can provide 3D building roof models with high geometric accuracy. LiDAR data and optical aerial images for advancing the reliability of segmentation and avoiding information loss, and (b) a pointbased integration mechanism to combine 2D lines extracted from optical multi-view aerial images and 3D points segmented from LiDAR data for determining 3D step lines with high geometric accuracy and good details, which benefits to improving quality of the final 3D models. Further studies may be performed in integrating airborne LiDAR and ground-based data for creating 3D building models with detailed fac-ade information. In addition, the proposed approach still requires further improvement in order to automatically reconstruct very complex structures of buildings (e.g., curve structures). Table 2 The geometric accuracy of the reconstructed 3D roof planes. Region Mean error (m) RMSE (m) Maximum error (m) Region Region Conclusion A new approach by integrating LiDAR data and optical multiview aerial images is proposed in this study, to automatically obtain 3D roof models with reliable structures and accurate geometric position. The main contributions of the proposed approach are as follows: (a) a new Shrink-Expand technique to synthetically utilize the complementary data characteristics of Table 1 The correctness and completeness of the reconstructed 3D roof planes. Region True Wrong Mis-detected Correctness (%) Completeness (%) Region Region Fig. 9. Geometric accuracy of the reconstructed 3D roof planes in Region 1 (left side) and Region 2 (right side); the short lines (bold) refer to errors of roof plane corners, which are shown as vectors given the length and orientation; each error line has been enlarged 20 times. Fig. 8. Comparison of 3D roof planes derived by the proposed approach (left) and manual operation (right): A, the wrong details; B, some artefacts; C, the missed tiny planes.

10 502 L. Cheng et al. / Optics and Lasers in Engineering 51 (2013) Acknowledgment This work is supported by the National Natural Science Foundation of China (Grant No ), the National 973 Project of China (Grant No. 2012CB719904), the National Key Technology R&D Program of China (Grant No. 2012BAH28B02). Sincere thanks are given for the comments and contributions of anonymous reviewers and members of the Editorial team. References [1] Suveg I, Vosselman G. Reconstruction of 3D building models from aerial images and maps. ISPRS J Photogrammetry Remote Sensing 2004;58(3 4): [2] Cheng L, Gong J, Li M, Liu Y. 3D building model reconstruction from multiview aerial imagery and LiDAR data. Photogrammertric Eng Remote Sensing 2011;77(2): [3] Mcintosh K, Krupnik A. Integration of laser-derived DSMs and matched image edges for generating an accurate surface model. ISPRS J Photogrammetry Remote Sensing 2002;56: [4] Ma R. Building model reconstruction from LiDAR data and aerial photographs. Ph.D. dissertation. Columbus, Ohio: The Ohio State University; p. [5] Sohn G, Dowman I. A model-based approach for reconstructing terrain surface from airborne LiDAR data. The Photogrammetric Record 2008;23(122): [6] Stamos I, Allen PK. Geometry and texture recovery of scenes of large scale. J Comput Vision Image Understanding 2002;88(2): [7] Rottensteiner F, Briese C. Automatic generation of building models from LiDAR data and the integration of aerial images. International Arch Photogrammetry, Remote Sensing Spat Inf Sci 2003:34 Part 3/W13. [8] Seo S. Model-based automatic building extraction from LiDAR and aerial imagery. Ph.D. dissertation. Columbus: The Ohio State University Ohio; p. [9] Frueh C, Zakhor A. An automated method for large-scale, ground-based city model acquisition. Int J Comput Vision 2004;60(1):5 24. [10] Brenner C. Building reconstruction from images and laser scanning. Int J Appl Earth Obs Geoinf 2005;6(3 4): [11] Zhang Y, Zhang Z, Zhang J, Wu J. 3D building modeling with digital map, LiDAR data and video image sequences. The Photogrammetric Record 2005;20(111): [12] Hu J. Integrating complementary information for photorealistic representation of large-scale environments. Ph.D. dissertation, University of Southern California; California: Los Angeles; p. [13] Sampath A, Shan J. Building boundary tracing and regularization from airborne LiDAR point clouds. Photogrammetric Eng Remote Sensing 2007;73(7): [14] Sohn G, Dowman I. Data fusion of high-resolution satellite imagery and LiDAR data for automatic building extraction, ISPR. J Photogrammetry Remote Sensing 2007;62(1): [15] Haala N, Brenner C. Virtual city models from laser altimeter and 2D map data. Photogrammetric Eng Remote Sensing 1999;65(79): [16] Dorninger P, Pfeifer N. A comprehensive automated 3D approach for building extraction, reconstruction, and regularization from airborne laser scanning point clouds. Sensors 2008;8: [17] Sohn G, Huang X, Tao V. Using a Binary Space Partitioning Tree for Reconstructing Polyhedral Building Models from Airborne LiDAR Data. Photogramm Eng Remote Sensing 2008;74(11): [18] Sampath A, Shan J. Segmentation and reconstruction of polyhedral building roofs from aerial LiDAR point clouds. IEEE Trans Geosci Remote Sensing 2010;48(3): [19] Kim K, Shan J. Building roof modeling from airborne laser scanning data based on level set approach. ISPRS J Photogrammetry Remote Sensing 2011;66: [20] Vosselman G, Kessels P. The utilisation of airborn laser scanning for mapping. Int J Appl Earth Obs Geoinf 2005;6(3 4): [21] Vestri C. Using range data in automatic modeling of buildings. Image Vision Comput 2006;24(7): [22] Elberink S, Vosselman G. Quality analysis on 3D building models reconstructed from airborne laser scanning data. ISPRS J Photogrammetry Remote Sensing 2011;66(2): [23] Meer P, Georgescu B. Edge detection with embedded confidence. IEEE Trans Pattern Analy Mach Intell 2001;23(12): [24] Rau JY, Chen LC. Robust reconstruction of building models from threedimensional line segments. Photogramm Eng Remote Sensing 2003;69(2):