A Rule-Based System for House Reconstruction from Aerial Images

Transcription

1 A Rule-Based System for House Reconstruction from Aerial Images Wolfram Willuhn and Frank Ade Communication Technology Lab, Swiss Federal Institute of Technology Zürich CH-8092 Zürich, Switzerland {Wolfram.Willuhn, Abstract We present a system for the reconstruction of houses from aerial images. Most of the methods proposed sofar use rather specific models. These models do not hold when dealing with european-style houses. Our system works with rules instead of models merely based on shape. This enables us to include additional domain specific knowledge, allowing a larger variability of the objects. However, this also necessitates a more complex structure of the knowledge base and a more sophisticated reasoning control. We explain the practical application of such a system for the reconstruction of a house from an aerial image Introduction The acquisition of 3D descriptions of houses is currently an issue of high importance to users of geoinformation. When using aerial imagery, which has proven to be a valuable source for this task, the reconstruction of a building means to find or hypothesize all parts of the roof because from all building parts they are the best visible in the images. The intended use of these algorithms requires that the precision of the obtained 3D description should be approximately the same as for manual measurements. For our current test set, the pixel size is about 7.5 cm x 7.5 cm on the ground and the achieved accuracy using standard photogrammetric techniques is 10 cm horizontally and vertically. This implies a high level of detail. Also the reliability of the extracted information should be comparable to that achieved when done by a human operator, which means that all buildings that appear in the image have to be found and no parts of their roofs should be missing. Models that were used for house extraction have been rather specific. Either they are based on explicit shape models that can be adapted by tuning a small number of parameters as in Sester [1] or on a single, horizontal roof surface bounded by straight segments forming right angles 1 This work is supported by ETH project (e.g. [2]). One exception is Bignone and Henricsson [3] who use a more generic approach to extracting house roofs. Achieving a good robustness and a high level of detail at the same time is very difficult. Predefined parameterizable geometric shape models can not be employed due to the great variability of the objects that are to be extracted. One way to come closer to this goal is to try to draw upon as many different types of data about the same scene as possible. For our data set, which was produced by the Institute of Geodesy and Photogrammetry at ETH Zürich, multiple color images taken from different points of view, a Digital Surface Model (DSM), and a manually measured CAD model for evaluating the results are available. Aerial images are generally very large. Our images have a size of 1800 x 1800 pixels showing 12 stand-alone houses and a few roads. Parts containing one house have been cut manually out from the large images. Baltsavias [4] and Haala [5] demonstrate how the detection of blobs in DSMs allows to establish hypotheses of house regions based on the idea that houses show up as rises in the ground in the DSM. Using as much as possible of the available data, our definition of a house is based on a set of rules that tend to hold for many instances of the category building. Some of these rules also deal with higher-level, structural knowledge. Methods of Artificial Intelligence have long been used to improve computer vision systems. In the last two decades, several systems have been implemented to extract three-dimensional objects from images, like the systems by Brooks [6], by Hanson and Riseman [7], or the MESSIE system [8, 9], or to extract man-made objects from aerial images, e.g. the SIGMA system [10] or the work of McKeown [11]. The following section describes the general architecture of the system: the knowledge representation, the levels of rules used, and the reasoning control. In Section 3., we demonstrate the application of this framework to the reconstruction of a house roof from an aerial image set.

2 2. Overview of the system The system s general architecture is a blackboard. This choice allows us to flexibly control the reasoning process depending on the data to be processed. Engelmore and Morgan [12] provide a good overview about different implementations of the blackboard architecture which has been widely used in Artificial Intelligence as a framework for reasoning. The short-term knowledge is stored in a semantic network, while rules encode the long-term knowledge forming the procedural part of the knowledge sources in the blackboard Knowledge Representation The semantic network, parts of which are shown in Figure 1, holds all the information that is accessible to the Attributes PlaneParameters, IsHorizontal, IsVertical Location in 3D, IsHorizontal, IsVertical Location in 3D Objects Volumes Surface Segments Point BoundsVolume BoundsSurface BoundsSegment Relations Connectivity Coplanarity Figure 1. Data structure for knowledge in the three-dimensional real world space knowledge sources and is structured into objects, attributes, and relations. The objects serve as anchors for the attributes and relations. Photometric and chromatic attributes describe the appearance of objects in the image. For contours, small flanking regions along both sides are analyzed and median color and interquartile ranges are computed (for a detailed description see [13] and [14]). Geometric attributes are measurable object properties, like the location, size, or shape of objects in the two-dimensional image space and the three-dimensional real world space. Likewise, spatial relations are those that are computed based on geometric attributes. Spatial relations are used at the beginning to structure the data and find larger coherent components. Contour relations are described in [13] and the segment coplanarity relation in [15]. Additional relations link objects in real world space with objects in image space corresponding to their projections into the image Blackboard Rules All long term knowledge about images, image features, three-dimensional objects in general, and houses in particular is coded into rules. Each knowledge source consists of a rule as its core and some meta-information declaring which of the available parts of the semantic network it needs for execution and which it might change. This information is used to control the execution of rules. The rules perform their actions based on multiple evidences. We regard interaction among rules as the best way to detect surprising coexistences of facts and, thus, to remove irrelevant features, to structure the data, and to solve higher-level problems. The rules should mutually reinforce their decisions rather than draw conclusions merely from individual facts. We categorize the rules into different levels and distinguish between those performing bottom-up or top-down functions. Bottom-up rules establish relations among features and larger structures, providing the basis for the reasoning at higher levels. Problems at higher level occur because low-level vision routines often fail to extract all relevant features and, on the other hand, sometimes find spurious ones. Top-down rules use a description of such a problem and try to solve it by explicitly searching for or inferring missing features at a lower level. Feature level. Contours in the image are extracted by edge detection and edgel aggregation [16], further augmented by geometric, photometric [13], and chromatic attributes [14], and linked by spatial relations. The stereo matching [15] generates segments in 3D, however it produces usually more than one possible match for each contour. Since this procedure is based only on individual segments and their respective contours, the result of the stereo matching can be checked and improved by comparing spatial relations among contours and among segments. Structure level. Regions and surfaces are structures that are made up of contours and segments, respectively. Regularities in the location and spatial relations of points, segments, and surfaces are used to find surfaces and volumes. This reflects our model of the objects that we want to find and describe. The roofs of the houses of the type we work with are neither flat nor can they generally be described by a rectilinear boundary. However, roofs tend to be made up of planar surfaces that intersect. According to this geometric model, segments that lie on the same plane are first grouped into planar surface hypotheses [15]. For each surface, a region is constructed by analyzing the photometric and chromatic attributes of the flanking regions of the contours, thus implementing

3 our region shape and color model. Contours which do not conform to this model are removed from the region and their related segments from the corresponding surface. Other rules establish intersections among planar surfaces. Each intersection should be bounded on either side by at least one other surface, forming a corner, and each surface should be enclosed entirely by intersections with other surfaces. If this is not the case, top-down rules try to solve this problem by finding two segments - one on either of the intersecting surfaces - that can be used to hypothesize a new planar surface. Furthermore, segments that have been missed by the stereo matching can be computed by adding contours to regions and constraining the segments s location to the plane of the surface. Conceptual level. Rules that are on the highest level contain domain specific knowledge about houses and house roofs. These rules are necessary for several reasons: The rules at the two lower levels apply to any type of object that complies with the assumption that it is made up of planar surfaces. Therefore there is no guarantee that the object that we reconstruct is really a house. However, one of the implications of the requirements stated at the beginning was that the algorithms has to be capable of self-evaluation. The system by itself should be able to determine the degree of confidence that this object is really a house and whether it should be shown to the human operator in order to check its correctness. There are some peculiarities in the construction of houses that are due to practical reasons or architectural considerations that should be taken into account. This knowledge can help us to solve problems that arise from failures in low-level routines and to choose among alternatives. Especially when trying to fill in missing features, this will result in better robustness. Finally, we can reduce the number of possibilities that we have to check. We can quickly rule out very unlikely constellations. A faster recognition can be achieved. Two conceptual features shall illustrate what we intend to implement in the future: the roof ridge and the roof gable. The roof ridge is a high, horizontal segment from where both intersecting surfaces will slope down. One can clearly see the two roof ridges of the house in our test image (see Figure 2). In a large percentage of houses, the roof ridge is terminated on both ends by the roof gable. This is a vertical wall that is delimited by an upside-down V. It is also common in some parts of Europe that the gable is divided up into a smaller triangular surface that is not vertical and the actual vertical wall (as it is to be seen in the lower part of the image). Symmetry is another element that is employed frequently in architecture and which could be used as an additional hint. We will show in the application section that it is possible to reconstruct the house roof already with the two lower levels. These rules work indepedently and can therefore also act as a fallback mechanism if the house s shape does not conform to any of the conceptual house models implemented. In this case, the system will still try to recognize an object and then might ask the human operator to check and if necessary correct the description Blackboard Control One important part of a blackboard system is the reasoning control. As there are no specific models for what we look for, and because of the fact that low-level vision procedures often produce incomplete results, no simple topdown or bottom-up strategy suffices to control the activation of the blackboard s knowledge sources. Instead, the control should be rather flexible. The execution of rules should be dependent on the data which includes both simple features and higher-level structures. Since problems during the creation of these structures are also documented explicitly in the data, not only bottom-up rules but also top-down rules can be triggered by changes in the data. These changes generate events and each knowledge source declares the types of data that it uses and thus implicitly also lists the events that it is interested in. This gives us the opportunity to treat both strategies - top-down and bottom-up - the same way. For each state of the blackboard there is a set of feasible knowledge source executions constructing the set of active knowledge sources. For each such configuration, we prefer those rules that let us anticipate the largest amount of changes in the knowledge base when being executed. A first heuristic rates all active rules based on the knowledge which parts of the semantic network are affected by each knowledge source, thus defining what are desirable actions. However, certain precedences among the rules exist. Some rules compute data that others depend upon (e.g. attribute values), forming a group of necessary actions. The knowledge source to be executed next can now be determined based on which one is the most desirable and which others are necessary for this. No single rule, no single run through all rules is able to solve a task like this. It is rather the interaction of rules that can achieve the desired results. Several iterations through many rules are generally necessary, especially when checking the results of stereo matching or removing surfaces that have no connection with others. In order to solve higher-level problems, usually new low-level features that have been searched for or inferred are inserted into the knowledge base and have to be checked, relations to existing features to be established, and new structures to be built until one of those problems is solved.

4 3. Application For this project we used a test set of images which is special in that it involves some difficulties that arise when trying to reconstruct typical european-style houses. The scene shows several stand-alone houses each of which has a different roof shape. Figure 2 depicts one of them with a After several iterations through rules that structure data, pose problems, and try to solve problems, we reach a state where no more changes in the data occur. Figure 4 shows the result as the intersections among the surfaces of the roof and the vertical walls. These surfaces are based on Figure 2. Original image (Courtesy of IGP at ETH Zürich) complex roof. Figure 3 presents the results obtained from the segment Figure 4. Final result of the object extraction 40 segments, 32 of which are not vertical and thus related to contours in the image. 24 of those 32 segments were present in the initial data, the other 8 have been inferred based on contours in the image and the location of the planes. All 5 planar roof surface hypotheses and 1 vertical wall were contained in the input data. A CAD description was obtained manually by photogrammetrists and is used for comparison (see Figure 5). Figure 3. Initial 3D information from stereo matching [15] stereo matching [15]. This information along with contours, their attributes and relations and the results of the coplanar grouping forms the input data for the knowledgebased analysis and contains 301 contours, 189 segments, and 34 planar surface hypotheses. The accuracy achieved in the stereo matching process is approximately 15 cm horizontally and 25 cm vertically. Figure 5. Manually extracted CAD description (Courtesy of IGP at ETH Zürich) All major roof surfaces have been extracted correctly. On the other hand, we can state two differences between the automatically and the manually generated house roof descriptions. First, the automatically extracted description explicitly represents the roof as a set of bordering planar surfaces, whereas the CAD description obtained by a human operator consists of connected lines. Second, the latter additionally includes smaller details like dormer windows.

5 4. Conclusions In this paper we introduced a system that aims at the semi-automatic reconstruction of houses from aerial images. The underlying architecture is a blackboard with a semantic network as knowledge base. The objects of this semantic network are augmented by photometric, chromatic, and geometric attributes and linked by spatial relations. Because no predefined parameterizable model solely based on shape can be used due to the high variability of the objects that we want to detect, all knowledge about how to build larger structures from lower-level features is coded into rules. Problems at higher levels are made visible in the knowledge base and invoke again lower-level rules that explicitly search for or infer missing features. The blackboard control works event-driven and supports both bottom-up and top-down reasoning. In the application section, we demonstrated an implementation of rules at the feature and structure levels. The rules at these levels try to reconstruct an object that - conforming to our model - is composed of intersecting planar surfaces bounded by straight segments. 5. Future work In the future, we want to implement the rules that encode conceptual knowledge. One set of rules will try to detect the most frequently used roof elements and shapes and thus affect the grouping of existing and the inference of missing features. Other rules will exploit symmetries as a commonly used architectural element. This will lead to the following enhancements: more robustness by introducing more domain specific knowledge, faster recognition by constraining the search process, and self-evaluation of the results. When performing an automatic building detection based on DSMs, we could also extract useful knowledge for the building reconstruction such as an estimated building height or the approximate location of the roof ridge. References [1] M. Sester and W. Förstner. Object location based on uncertain models. In H. Burkhardt, K. Böhme, and B. Neumann, editors, Mustererkennung 1989, pages DAGM, Springer Verlag, [2] R. Collins, A. Hanson, E. Riseman, and H. Schultz. Automatic extraction of buildings and terrain from aerial images. In A. Grün, O. Kübler, and P. Agouris, editors, Automatic Extraction of Man-Made Objects from Aerial and Space Images, pages Birkhäuser Verlag, [3] F. Bignone, O. Henricsson, P. Fua, and M. Stricker. Automatic extraction of generic house roofs from high resolution aerial imagery. In B. Buxton and R. Cipolla, editors, European Conference on Computer Vision, pages Springer Verlag, April [4] E. Baltsavias, S. Mason, and D. Stallmann. Use of DTMs/ DSMs and orthoimages to support building extraction. In A. Grün, O. Kübler, and P. Agouris, editors, Automatic Extraction of Man-Made Objects from Aerial and Space Images, pages Birkhäuser Verlag, [5] N. Haala and M. Hahn. Data fusion for the detection and reconstruction of buildings. In A. Grün, O. Kübler, and P. Agouris, editors, Automatic Extraction of Man-Made Objects from Aerial and Space Images, pages Birkhäuser Verlag, [6] R. Brooks. Model-based three-dimensional interpretation of two-dimensional images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 5(2): , March [7] A. Hanson and E. Riseman. The VISIONS image-understanding system. In C. Brown, editor, Advances in Computer Vision, volume 1. Lawrence Erlbaum Associates, Publishers, [8] V. Clément, G. Giraudon, S. Houzelle, and F. Sandakly. Interpretation of remotely sensed images in a context of multisensor fusion using a multispecialist architecture. IEEE Transactions on Geoscience and Remote Sensing, 31(4): , July [9] F. Sandakly and G. Giraudon. Multispecialist system for 3D scene analysis. In European Conference on Artificial Intelligence, [10] T. Matsuyama and V. Hwang. SIGMA: A Knowledge- Based Aerial Image Understanding System. Plenum Press, [11] D. McKeown, W. Harvey, and J. Dermott. Rule-based interpretation of aerial imagery. IEEE Transactions on Pattern Analysis and Machine Intelligence, 7(5): , September [12] R. Engelmore and T. Morgan, editors. Blackboard Systems. Addison-Wesley Publishing Company, [13] O. Henricsson. Inferring homogeneous regions from rich image attributes. In A. Grün, O. Kübler, and P. Agouris, editors, Automatic Extraction of Man-Made Objects from Aerial and Space Images, pages Birkhäuser Verlag, [14] O. Henricsson and M. Stricker. Exploiting photometric and chromatic attributes in a perceptual organization framework. In Asian Conference on Computer Vision, pages , [15] F. Bignone. Segment stereo matching and coplanar grouping. Technical Report BIWI-TR-165, ETH Zürich, Communication Technology Lab, Computer Vision Group, [16] O. Henricsson and F. Heitger. The role of key-points in finding contours. In J. O. Eklundh, editor, European Conference on Computer Vision, pages Springer Verlag, 1994.