Robust Model-Based Detection of Gable Roofs in Very-High-Resolution Aerial Images

Transcription

1 Robust Model-Based Detection of Gable Roofs in Very-High-Resolution Aerial Images Lykele Hazelhoff 1,2 and Peter H.N. de With 2 1 CycloMedia Technology B.V., The Netherlands lhazelhoff@cyclomedia.com 2 Eindhoven University of Technology, The Netherlands p.h.n.de.with@tue.nl Abstract. This paper describes an improved version of our system for robust detection of buildings with a gable roof in varying rural areas from very-high-resolution aerial images. The algorithm follows a custom-made design, extracting key features close to modeling, such as roof ridges and gutters, in order to allow a large freedom in roof appearances. It starts by detecting straight line-segments as roof-ridge hypotheses, and for each of them, the likely roof-gutter positions are estimated. Supervised classification is employed to select the optimal gutter pair and to reject unlikely detections. Afterwards, overlapping detections are merged. Experiments on a large dataset containing 220 images, covering different rural regions with significant variation in both building appearance and surroundings, show that the system is able to detect over 87% of the present buildings, thereby handling common distortions, such as occlusions by trees. Keywords: Building detection, Object detection, Remote sensing. 1 Introduction Very-high-resolution aerial images are captured from The Netherlands at a yearly basis, providing a recent overview of the country infrastructure. Updating of civil community databases based on aerial images is time consuming when performed manually, leading to a demand for automated interpretation of these images. This is of particular interest in rural areas, since these cover a widespread area together with a low population density, increasing the cost per citizen. Since buildings are dominant features in those images, accurate extraction of their locations is important. However, accurate and large-scale detection of buildings in aerial images is a complicated problem, since buildings vary considerably in appearance, may feature complex compositions and occlusions by trees occur frequently. Besides this, large variations also exist in the source data due to varying capturing conditions, illumination circumstances and sensor differences. This causes large variations in both visual appearance and statistical properties. Therefore, development of a building-detection solution that is able to handle these issues is a rather complicated problem. A. Berciano et al. (Eds.): CAIP 2011, LNCS 6854, pp , c Springer-Verlag Berlin Heidelberg 2011

2 Robust Model-Based Detection of Gable Roofs 599 In literature, localization of buildings in remote sensing data is researched for decades, based on many different types of source data, including satellite and aerial images, lidar data, etc. As the information content of these sources varies from 2D grayscale images to 3D structural information, the proposals vary from building localization to automatic 3D city reconstruction. However, our country-covering datasets only contain color aerial images, disabling the use of any 3D information. Relevant proposals in literature for detection of buildings in single color images are e.g. based on image segmentation [1]. There, the input image is segmented using a range of parameters, resulting in multi-layer segmented images. This is analyzed with a tree structure by means of rooftop constraints. Potential buildings are evaluated relying on shadow information and a fixed height. Other strategies include searching of closed loops [2]. In this work, after mean-shift segmentation, edge pixels that form closed loops are converted into polygons. These polygons are used to deduce the building shape. Nosrati et al. [3] apply dynamic programming to line intersections for searching of closed loops. Both rely on the visibility of the outer edges, which possibly disables detection of buildings in low-contrast situations. A combination of multiple information sources is followed by Jin et al. [4], where three different detectors, focusing at structural, contextual and spectral information, are applied for localization of buildings in high-resolution satellite images. They report a significant increase in detection performance by combining these detectors. Benedek et al. [5] apply a probabilistic approach for building extraction, where building footprints are represented by combinations of rectangular segments. The optimal building configuration is retrieved based on a global optimization process. Although many proposals report accurate results, the test data is often limited in both numbers and in-set variety. In contrast, we focus on robustness to the above-described variations, and we have described a novel, specific algorithm for localization of buildings with a gable roof in rural areas [6]. Whereas we aim at applying the algorithm at a large-scale database, we have designed the algorithm for robustness against both variations in building appearance, variations in the source data and commonly occurring distortions, such as overhanging trees. These constraints have guided us to design a more generic system, aiming at an high overall score, instead of a very high score in a specific situation, which is a different approach compared to many proposals in literature. In this paper, we describe an improved version of the original algorithm, which still operates under the above-mentioned conditions. Next to this, we will also present new and more accurate results obtained with this system, where we used an extended version of our test set, now containing more images and captured in three different years. 2 Algorithm Description 2.1 Preprocessing Figure 1 portrays the schematic overview of our algorithm, based on [6]. Prior to image analysis, the image is segmented using a region-segmentation procedure similar to [7]. Vegetated and large, uniform farmland areas are then

3 600 L. Hazelhoff and P.H.N. de With Fig. 1. Schematic overview of our building detection system discarded using vegetation detection and a minimum size constraint of 500 m 2. Pixels (represented by RGB values) are marked as vegetation when they satisfy the following empirically determined rule: G min (1.175 B,0.975 R). (1)

4 2.2 Roof-Ridge Hypothesis Generation Robust Model-Based Detection of Gable Roofs 601 Within the accepted regions of interest, hypotheses of occurring roof ridges are generated by detection of straight line segments. For this, a Canny edge detector is applied on a single-channel version O of the input RGB image, where the employed color transformation is chosen such that clear transitions are expected along roof ridges. Since roofs are usually either red or gray, and often contain a shadowed roof side, the following transformation is selected: O =0.5 (R + B). (2) Connected edge pixels forming straight lines are extracted based on a technique described in [8], resulting in a set of line pieces. To enable by-passing of lineinterrupting objects, like chimneys, individual line pieces having a position and orientation such that they jointly form a straight line are combined when they are located near the same region segment. 2.3 Roof Gutter Position Estimation For all hypotheses of occurring roof ridges, the set of likely roof-gutter positions is estimated for both sides of the ridge. This results in a set of hypotheses of the roof configuration, given by each combination of gutter positions at both ridge sides, as displayed in Box 3 Fig. 1. To identify the likely gutter positions, two rectangular regions located parallel to the ridge are analyzed, where the region size is inferred from training data. For ease of description, we assume that the ridge is oriented vertically. Within the regions, vertical Sobel filters are applied to both the red and blue color channels, since roofs are usually red or gray. Per color channel and for each column, the resulting edge energy is sorted by magnitude, and divided into 4 groups, each containing the sum of the represented 25% of the pixels. The resulting signal matrix ES containing 8 features per column i is represented with a Gaussian model. This model is trained from the first 10 columns, i.e. the columns closest to the roof ridge. The distance towards the center of the ellipsoid is calculated, which is proportional to: D (ES (i)) e 1 2([ES(i) µ ES ] T C 1 ES [ES(i) µ ES ]), (3) where µ ES and C ES denote the mean and covariance matrix extracted from the 10 training samples. Column i with D (ES (i)) satifying (1) local maximality, (2) higher than twice the running average, and (3) larger than 66% of the running maximum, is identified as roof-gutter position. 2.4 Roof Analysis Each roof-configuration hypothesis is validated, where at first infeasible candidates are rejected, and second, machine learning is employed to classify each remaining configuration between roof half and non roof-half. This results in a

5 602 L. Hazelhoff and P.H.N. de With likelihood value for each configuration, used to select the optimal roof configuration for each ridge hypothesis and to discard low-valued detections. The first step aims at rejecting very unlikely configurations by checking a number of loose constraints on size, aspect ratio, shadow profile and vegetation coverage. This step exploits physical properties of gable roofs, and is described in detail [6]. All remaining configurations are subject to classification based on supervised classification. For each of both roof halves, the following features are extracted. 1. Roof-Gutter Edge Orientation Histogram: a gradient orientation histogram is extracted along the gutters, indicating the deviation in orientation w.r.t. the ridge orientation. This histogram contains 6 bins representing 15 each. 2. Segmentation Features: The five largest clusters covering each roof half are expressed as a percentage of the total amount of roof-half pixels. 3. Color Similarity Histogram: patch-based similarity analysis is employed to investigate roof similarities. For each patch, the most similar patch is searched (with a minimum distance constraint), and after sorting, the values for which 20%, 40%, etc. of the pixels are lower, are extracted. Next to these features, also the ridge length, aspect ratio and ratio of roof-half widths are extracted. The total feature vector consists of 51 features, containing the 3 global features, 2 16 features for each roof half, appended by their sum. Note that during the training phase, we have also included the samples with reversed roof halves. Classification is performed using a Support Vector Machine (SVM) with radial basis function kernel, which outputs the distance towards the decision bound, as a kind of likelihood information. For each ridge hypothesis, the configuration with the highest value is selected as optimal configuration, while configurations with a low SVM output value are discarded. 2.5 Detection Merging Each physical roof may contain several detections, where at most one detection corresponds to the actual roof ridge, as shown in Box 5 in Fig. 1. Therefore, overlapping detections are fused using supervised classification, where we have found that multiple classifiers rejecting detections in specific situations outperform a single, large classifier. As a result, we have installed multiple linear SVM classifiers, each of them set up to analyze a specific situation. These situations include differences in red profile, compliance with the expected shadow profile, detections with complete overlap, etc. For each situation, we extract relevant features, and append this by features representing the differences in likelihood, length and angle between the ridges. The continuous output of each classifier is subject to a threshold such that at most 0.5% of the positive training samples are rejected. This threshold setting gives a trade-off between detections and false alarms. Note that each classifier fuses a small fraction of the overlapping detections. In some cases, the optimal fusion process does not satisfy our generic principles, so that not all overlapping detections are fused. Even then, the amount of detections is reduced significantly.

6 3 Experiments and Results 3.1 Dataset Description and Test Setup Robust Model-Based Detection of Gable Roofs 603 The algorithm is tested on 220 ortho-rectified aerial images, captured in 2008, 2009 and 2010 during spring and fall time. All images are normalized such that each pixel corresponds to 0.1 m on the ground plane. The set contains 1, 523 buildings, including houses, garages, barns and small sheds, which are all marked manually for evaluation purposes. The set is randomly divided in two equal parts, denoted as SetA and SetB, where for each test, the other set is used for training. The performance of our system is analyzed in two ways. First, we have evaluated the detection accuracy of the system, i.e. how accurate our algorithm indicates the presence of a gable roof. Second, we have analyzed the accuracy of the estimated building dimensions, i.e. how well the algorithm indicates the building size. These two aspects are separated since we concentrate on detection of buildings, where we consider the size estimate as a byproduct. The detection performance is assessed using the following metrics: Recall: R = TP TP+FN 100%, Precision: P = TP TP+FP 100%. We count a detection as a True Positive (detected building) when it overlaps the roof ridge for at least 51%, all other detections are treated as False Positives (falsely detected buildings). Since we aim at localizing the somewhat larger buildings in rural areas, we have neglected buildings smaller than 5 3 meters based on a minimum roof-ridge length and building width. These minimum sizes are chosen such that typical side-buildings, like e.g. garages, are still detected. 3.2 Results The recall-precision curves for both sets are shown in Fig. 2. As follows from this graph, our system is capable of detecting over 87% of the target buildings, where for a detection rate of 80%, around 77% of the detections are correct. For this specific working point, we have analyzed the performance in more detail. Buildings are missed due to large occlusions by trees, or shadows from trees on the roof. We have already reported in [6] that about 15% of the buildings are covered by trees to a certain extent, where we add that about 2% of the buildings have at least one roof half covered completely by trees (as in e.g. Fig. 3(h)). In numerous cases, we still find a detection on the outer building border, but these are counted as false detections. Other causes of misdetections are e.g. low-contrast roof ridges, which especially occur at completely black roofs. Such special cases may be handled by additional detectors, as e.g. applied in [4]. False detections are mainly located at the outer building borders of missed buildings and on gable-roof-like objects, including road segments. For both categories, the SVM output is relative low, but not smaller than some detected buildings. Fig. 3 displays some examples of correct detections, false detections and missed buildings; more examples can be found on our website 1.

7 604 L. Hazelhoff and P.H.N. de With % of correct detections % of buildings detected Set A Set B Fig. 2. Recall-precision curve for our system, shown for both seta and setb (a) (b) (c) (d) (e) (f) (g) (h) Fig. 3. Examples of correct detections (a)-(d), false detection (e)-(f) and missed building (f)-(h). The estimated ridge positions are drawn green (TP), yellow (FP) or red (FN); corresponding gutter positions are drawn dotted white. We have also assessed the modeling capabilities of our algorithm by comparing the estimated length and width against the ground-truth dimensions. The found roof ridge is often smaller than the real roof length, due to chimneys and roofs where the ridge ends in a triangular shape, as in e.g. Fig. 3(d). The main cause for inaccurately found building widths are low-contrast gutters. Quantitatively, 80.2% of the detected ridges deviate less than 10% from the ground-truth length, 83.7% of the estimated widths deviate less than 10% and 61.8% of the detections have an area deviation smaller than 10%. We consider this as a reason- 1 See

8 Robust Model-Based Detection of Gable Roofs 605 ably accurate result as our system is primarily targeting at localizing buildings with a diverse test set, where numerous buildings are overlapped by trees. 4 Conclusions and Future Work We have presented an improved version of our algorithm for detection of gable roofs in very-high-resolution aerial images, focusing on rural areas. Our system aims at detecting buildings with a high robustness in a generic way, and is therefore designed with special attention to the large existing variations in both building appearance and source data. This deviates from the common practice to pursue high scores in a specific number of cases. A specific novel aspect of our algorithm is the example-based detection-merging step, based on multiple linear SVM classifiers. Tests on a diverse dataset, containing numerous geographical locations, have shown that our system is able to localize over 87% of the buildings larger than 5 3 m, where for a recall of 80%, a precision of around 77% is obtained. The reported system is able to detect around 8% more buildings than our previously described system. Considering that our test set contains large variations in both source data and target objects, and no height information is incorporated in the detection process, we consider this as an accurate result. Improvement of the modeling capabilities is part of our future work. References 1. Izadi, M., Saeed, P.: Automatic building detection in aerial images using a hierarchical feature based image segmentation. In: 20th International Conference on Pattern Recognition, pp (2010) 2. Ok, A.O.: Automated description of 2-d building boundaries from a single color aerial ortho-image. In: Proceedings of ISPRS, High Res. Earth Imag. for Geospat. Inf., pp (2009) 3. Nosrati, M.S., Saeedi, P.: A novel approach for polygonal rooftop detection in satellite/aerial imageries. In: Int. Conf. on Image Processing (ICIP), pp (2009) 4. Jin, X., Davis, C.H.: Automated building extraction from high-resolution satellite imagery in urban areas using structural, contextual, and spectral information. EURASIP Journal on Applied Signal Processing, (2005) 5. Benedek, C., Descombes, X., Zerubia, J.: Building detection in a single remotely sensed image with a point process of rectangles. In: 20th International Conference on Pattern Recognition (2010) 6. Hazelhoff, L., De With, P.: Localizations of buildings with a gable roof in very-highresolution aerial images. In: Proceedings of IS&T SIE Electronic Imaging, Visual Information Processing and Communication II (2011) 7. Comanicu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, (2002) 8. Guru, D.S., Shekar, B.H., Nagabhushan, P.: A simple and robust line detection algorithm based on small eigenvalue analysis. Pattern Recognition Letters 25, 1 13 (2004)