Building Facade Detection, Segmentation, and Parameter Estimation for Mobile Robot Localization and Guidance

Transcription

1 Bilding Facade etection, Segmentation, and Parameter Estimation for Mobile Robot Localization and Gidance Jeffrey A. elmerico SUNY at Bffalo Philip aid Army Research Laboratory Adelphi, Maryland Jason J. Corso SUNY at Bffalo Abstract Bilding facade detection is an important problem in compter ision, with applications in mobile robotics and semantic scene nderstanding. In particlar, mobile platform localization and gidance in rban enironments can be enabled with an accrate segmentation of the arios bilding facades in a scene. Toward that end, we present a system for segmenting and labeling an inpt image that for each pixel, seeks to answer the qestion Is this pixel part of a bilding facade, and if so, which one? The proposed method determines a set of candidate planes by sampling and clstering points from the image with Random Sample Consenss (RANSAC), sing local normal estimates deried from Principal Component Analysis (PCA) to inform the planar model. The corresponding disparity map and a discriminatie classification proide prior information for a two-layer Marko Random Field model. This MRF problem is soled ia Graph Cts to obtain a labeling of bilding facade pixels at the mid-leel, and a segmentation of those pixels into particlar planes at the high-leel. The reslts indicate a strong improement in the accracy of the binary bilding detection problem oer the discriminatie classifier alone, and the planar srface estimates proide a good approximation to the grond trth planes. I. INTROUCTION Accrate scene labeling can enable applications that rely on the semantic information in an image to make high leel decisions. Or goal of labeling bilding facades is motiated by the problem of mobile robot localization in GPS-denied areas. This problem arises in rban areas, so the approach crrently being deeloped by or grop depends on detection of bildings within the field of iew of the cameras on a mobile platform. Within this problem, accrate detection and labeling is critical for the high leel localization and gidance tasks. We restrict or approach to identifying only planar bilding facades, and reqire image inpt from a stereo sorce. Since most bildings hae planar facades, and many mobile robotic platforms are eqipped with stereo cameras, neither of these assmptions is particlarly restrictie. In this paper, we propose a method for bilding facade labeling in stereo images that frther segments the indiidal facades and estimates the parameters of their 3 models. Or approach proceeds in three main steps: discriminatie modeling, candidate plane detection throgh PCA and RANSAC, and energy minimization of MRF potentials. Or The athors are gratefl for the financial spport proided in part by NSF CAREER IIS , ARPA W911NF , and ARO Yong Inestigator W911NF Sample Points w/ Valid isparity Compte Local Normals w/ PCA Generate Candidate Planes w/ RANSAC Compte Labeling w/ MRF Fig. 1. Workflow of or candidate plane selection and labeling method. We iteratiely rn RANSAC on a set of sampled points from the image, remoing the inliers (green) from the set, to generate a set of candidate planes. Or planar model incorporates the fit of the PCA local normal estimate into the error term. The set of candidate planes proides the label set for the high-leel MRF model. contribtion is the se of plane fitting techniqes from stereo imagery to the problem of bilding facade segmentation in the context of mobile platform localization. A diagram of the workflow for candidate plane detection and high-leel labeling is proided in Fig. 1. Or work leerages stereo information from the beginning. Or discriminatie model is generated from an extension of the Boosting on Mltileel Aggregates (BMA) method [1] that incldes stereo featres [2]. Boosting on Mltileel Aggregates ses hierarchical aggregate regions coarsened from the image based on pixel affinities, as well as a ariety of high-leel featres that can be compted from them, to learn a model within an AdaBoost [3] two- or mlti-class discriminatie modeling framework. The mltileel aggregates

2 exploit the propensity of these coarsened regions to adhere to object bondaries, which in addition to the expanded featre set, offer less pollted statistics than patch-based featres, which may iolate those bondaries. Since many mobile robot platforms are eqipped with stereo cameras, and can ths compte a disparity map for their field of iew, or approach of sing statistical featres of the disparity map is a natral extension of the BMA approach gien or intended platform. Since bildings tend to hae planar srfaces on their exteriors, we se the stereo featres to exploit the property that planes can be represented as linear fnctions in disparity space and ths hae constant spatial gradients [4]. We se the discriminatie classification probability as a prior for inference of facade labeling. In order to associate each bilding pixel with a particlar facade, we mst hae a set of candidate planes from which to infer the best fit. We generate these planes by sampling the image and performing Principal Component Analysis (PCA) on each local neighborhood to approximate the local srface normal at the sampled points. We then clster those points by iteratiely sing Random Sample Consenss (RANSAC) [5] to find sbsets which fit the same plane model and hae similar local normal orientations. From these clsters of points, we are able to estimate the parameters of the primary planes in the image. We then incorporate both of these sorces of information into a Bayesian inference framework sing a two-layer Marko Random Field (MRF). We represent the mid-leel MRF as an Ising model, a layer of binary hidden ariables representing the answer to the qestion Is this pixel part of a bilding facade? This layer ses the discriminatie classification probability as a prior, and effectiely smooths the discriminatie classification into coherent regions. The high-leel representation is a Potts model, where each hidden ariable represents the labeling of the associated pixel with one of the candidate planes, or with no plane if it is not part of a bilding. For each pixel, we consider its image coordinates and disparity ale, and ealate the fitness of each candidate plane to that pixel, and incorporate it into the energy of labeling that pixel as a part of that plane. A more in-depth discssion of these methods can be fond in Section II-B. We se the Graph Cts energy minimization method [6] to compte minimm energy labelings for both leels of or MRF model. In principle or approach is modlar, in that for each of the three phases (modeling, candidate plane detection, and labeling), a different method that prodces the same type of otpt (probability map, candidate plane set, facade segmentation, respectiely) cold be sbstitted. Howeer, the specific techniqes we hae deeloped hae been motiated by the featres of this specific problem. A. Related Work Bilding facade detection and segmentation hae been and contine to be well-stdied problems. Many recent papers in the literatre hae focsed on segmentation of bilding facades for se in 3 model reconstrction, especially in the context architectral modeling or geo-spatial mapping applications sch as Google Earth. Korah and Rasmssen se textre and other a priori knowledge to segment bilding facades, among other facade-related tasks [7]. Wendel et al. se intensity profiles to find repetitie strctres in coherent regions of the image in order to segment and separate different facades [8]. Hernández and Marcotegi employ horizontal and ertical color gradients, again leeraging repetitie strctres, to segment indiidal facades from blocks of contigos bildings in an rban enironment [9]. Seeral other methods tilize anishing points for planar srface detection. aid identifies anishing points in a monoclar image by groping line segments with RANSAC and then determining plane spport points by the intersection of the segments which point toward orthogonal anishing point ltimately clstering them to extract the planes of the facade [10]. Baer et al. implement a system for bilding facade detection sing anishing point analysis in conjnction with 3 point clods obtained by corresponding a sweep of images with known orientations [11]. Lee et al. se a line clstering-based approach, which incorporates aerial imagery, anishing points, and other projectie geometry ces to extract bilding facade textres from grond-leel images, again toward 3 architectral models reconstrction [12]. Or work draws on the contribtions of Wang et al., whose facade detection method sing PCA and RANSAC with LiAR data inspired or approach with stereo images [13]. Perhaps the approach most similar in spirit to ors is that of Gallp et al. [14], who also se an iteratie method for generating candidate plane models sing RANSAC, and also sole the labeling problem sing graph cts [6]. Howeer, their approach relies on mltiiew stereo data and leerages photoconsistency constraints in their MRF model, whereas we perform segmentation with only single stereo images. In addition, on a fndamental leel their method inoles finding many planes that fit locally, and stitching them together, whereas we aim to extract or planar models from the global data set, withot an explicit restriction on locality. We present qantitatie reslts on the accracy of or planar modeling as well. Althogh many of these reslts are directed toward 3 model reconstrction, some other work sing nrelated techniqes has been focsed toward or intended application of ision-based naigation, namely [10], [15], [16], [17]. Additionally, or work is focsed on retrieal of the estimated plane parameters, as implemented in the planar srface model of [4], and not on 3 model reconstrction. A. BMA+isparity Classifier II. METHOS Based on the work in [2], we model bilding facade featres sing the Boosting on Mltileel Aggregates (BMA) [1] method, with the extension to stereo featres. In principle, any classifier cold be sed for this step, so long as it cold prodce a probability map for binary classification in

3 following modified plane parameters n = (a, b, c ), where: sch that n a = ab d, b = bb d, c = cfb d 1 (5) = a + b + c = (, ) (6) This new set of plane parameters relates the image coordinates and their corresponding disparity ales by incorporating the constant bt nknown camera parameters. Fig. 2. Seeral examples of probability maps to be sed as priors for or MRF. For each example, they are (L to R) the original image, by standard BMA, and by BMA+isparity. identifying bilding pixels. We choose the BMA+isparity method becase of its sperior performance to standard AdaBoost, as well as to BMA withot the addition of stereo featres (See Fig. 2). B. Plane Parameters Throghot this discssion, we assme that we hae stereo images which are rectified, bt since we do not aim for fll 3 reconstrction, for the prposes of the following deriation the camera s calibration parameters are left as nknown constants. We can determine the srface normal parameters p to a constant that describes the camera parameters, and since that constant will be the same across all candidate planes, we can se the compted srface normals to differentiate between planes. Known camera parameters wold enable recoery of the srface normal parameters in world coordinates. A plane in 3 space can be represented by the eqation: ax + by + cz = d (1) and for non-zero depth, z, this can be rewritten as: a x z + by z + c = d (2) z We can map this expression to image coordinates by the identities = f x z and = f y z, where f is the focal length of the camera. We can also incorporate the relationship of the stereo disparity ale at camera coordinate (, ) to the depth, z, sing the identity (, ) = fb z, where is the disparity and B is the baseline of the stereo camera. Or plane eqation becomes: which redces to: ( ab d a f + b f ) + + c = d (, ) fb ( ) bb + d (3) ( ) cfb = (, ) (4) d Althogh n = (a, b, c) T is the srface normal in world coordinates, for or prposes we can seek to determine the C. Candidate Plane etection We perform the second phase of or approach by iteratiely sing RANSAC to extract a set of points which fit a plane model in addition to haing a local normal estimate which is consistent with the model. The extracted plane models become the set of candidate planes for or high-leel MRF labeling. 1) Local Normal Estimation: Based on or assmption of rectilinear bilding facades, we can se Principal Component Analysis to determine a local normal to a point in disparity space as in [18]. We first constrct the coariance matrix of the neighborhood arond the point in qestion. To do this, we consider any points, in a 5 5 window centered on or point p = (,, (, )), that hae a alid disparity ale. Here, and represent row and colmn indices, respectiely. Note that stereo cameras that compte the disparity map with onboard processing in real-time often do not prodce dense disparity maps. The pixels that are not matched in the left and right images or are at a distance beyond the sable range of the camera will be labeled with the maximm ale for that disparity image, representing that the camera failed to compte the disparity at that pixel. Conseqently, the neighborhood we se for PCA may be sparse. Other neighborhood sizes cold be sed, bt we fond that a 5 5 window proided good estimates while remaining local. We compte the centroid, p = 1 N N i=1 p i, of the points {p i } i=1...n in the neighborhood with alid disparity, and calclate the 3 3 coariance matrix with: W = 1 N (p i p) (p i p) (7) N i=1 where is the oter prodct. We then compte the eigenales of W, and the eigenectors corresponding to the largest two eigenales indicate the directions of the primary directions on the local planar estimate. The eigenector corresponding to the smallest eigenale ths indicates the direction of the local srface normal, n (,). 2) RANSAC Plane Fitting: We take a sample, S, of points from the image, which all hae alid disparity ales, and compte the local planar srface normal estimates by the aforementioned method. We then seek to fit a model to some sbset of S of the form: α + β + ɛ( (, )) + θ = 0 (8)

4 where ñ = 1 ɛ (α, β, θ) is the srface normal from Eq. (6). Since RANSAC finds the largest consenss set, P in, that it can among S, we will fit the most well-spported plane first [5]. We then remoe the inliers, S = S \ P in, and repeat this process iteratiely, finding progressiely less wellspported planes, ntil a fixed percentage of the original S has been clstered into one of the extracted planes. In or experiments, we sed a sample of 2000 points from the image, and conclded the plane extraction once 80% of the points had been clstered, or when RANSAC failed to find a consenss set among the remaining points. We also se a RANSAC noise standard deiation of σ η = 5, representing the amont of Gassian noise that is assmed on the positions of the inlier points. Althogh we se RANSAC to fit a standard plane model, we se a modified error term in order to incorporate the information in the local normal estimates. Here, since or local normal estimate reqired the se of a three dimensional coordinate system (,, (, )), and prodces a normal of that form, we mst se a slightly different normal formlation of n m = (α, β, ɛ). The standard measre of error for a plane model is the distance of a point from the plane: E m = α + β + ɛ( (, )) + θ, assming n m = (α, β, ɛ) is a nit ector. We compte another measre of error, E norm, the dot prodct of the model plane normal n m and the local normal estimate n (,), which is the cosine of the dihedral angle between the two planes defined by those normals. If we take its magnitde, this metric aries from 0 to 1, with 1 representing normals which are perfectly aligned, and 0 representing a dihedral angle of 90. Since the range of E depends on the properties of the image (resoltion, disparity range), we combine these two metrics as follows: E = E m (2 E norm ) = E m (2 n m, n (,) ) (9) sch that the dihedral angle scales the error term from E m to 2E m, depending on the consistency of the model and local normals.. MRF Model We model or problem in an energy minimization framework as a pair of copled Marko Random Fields. Or midleel representation seeks to infer the correct configration of labels for the qestion Is this pixel part of a bilding facade? Based on this labeling, the high-leel representation seeks to associate those pixels which hae been positiely assigned as bilding facade pixels to a particlar candidate plane. Figre 3 shows a graphical representation of this MRF model. Or motiation for this design stems from the fact that these are related bt distinct qestions, and they are informed by different approaches to modeling bildings. The mid-leel MRF represents an appearance-based model, while the high-leel MRF represents a generatie model for the planar facades. 1) Mid-leel Representation: We want or energy fnction for the mid-leel model to captre the confidence (probability) of or discriminatie classification, and we want there to be a penalty wheneer a pixel with a high confidence Fig. 3. Or two-layer MRF model. is mislabeled, bt a smaller penalty for pixels with lower confidence in their a priori classification. We will se an Ising model to represent or mid-leel MRF, where or labels x s for s λ, or image lattice, come from the set { 1, 1}. We define a new ariable b s to represent a mapping of the X s { 1, 1} label to the set {0, 1} by the transformation b s = Xs+1 2. For a particlar configration of labels l, we define or mid-leel energy fnction as: E(l) = s λ [(1 b s )p(s) + b s (1 p(s))] γ s t x s x t (10) where p(s) is the discriminatie classification probability at s and γ is a constant weighting the nary and binary terms. The b s qantity in the nary term essentially switches between a penalty of p(s) if the label at s is set to 1, and a penalty of 1 p(s) if the label at s is set to 1. Ths for p(s) = 1, labeling x s = 1 will incr an energy penalty of 1, bt labeling x s = 1 will incr no penalty. Similarly for p(s) = 0, labeling x s = 1 will incr no penalty, bt labeling it 1 will incr a penalty of 1. A probability of 0.5 will incr an eqal penalty with either labeling. Or smoothness term is from the standard Ising model. In or experiments, we sed a γ ale of 10. 2) High-leel Representation: In designing or energy fnction for the high-leel MRF, we want to penalize points which are labeled as being on a plane, bt which do not fit the corresponding plane eqation well. Or label set for labels y s, s λ, is {0,..., m}, with m eqal to the nmber of candidate planes identified in the plane detection step. It corresponds to the set of candidate planes indexed from 1 to m, as well as the label 0, which corresponds to not on a plane. We define a set of eqations E p (s) for p {0,..., m} sch that E p (s) = a p + b p + c p (s) (11) where the srface normal n p = (a p, b p, c p) corresponds to the plane with label p, and (s) is the disparity ale at s. We normalize this energy fnction by diiding by the maximm disparity ale, in order to scale the maximm energy penalty down to be on the order of 1. For consistency

5 in or notation, we define E 0 (s) to be the energy penalty for a label of 0 at s, corresponding to the not on a plane classification. We set E 0 (s) = b s, sch that a labeling of 1 in the mid-leel representation reslts in b s = 0, so there is no penalty for labeling s as not on a plane. Similarly, when x s = 1, b s = 1, so there is a penalty of 1 to label any of the non-planar pixels as a plane. To constrct or oerall energy fnction for the high-leel MRF, we incorporate the exponential of the set of planar energy fnctions E p with a delta fnction, so the energy cost is only for the plane corresponding to the label y s. Since we cannot compte E p withot a alid disparity ale, we se an indicator ariable χ {0, 1} to switch to a constant energy penalty for all planes and the no-plane option, in order to rely strictly on the smoothness term for that pixel s label. For the smoothness term, we se a Potts model, weighted like the mid-leel representation, with a constant γ = 1. Ths the high-leel energy fnction we are seeking to minimize is: E(l) = m δ ys=p exp (χ E p (s)) + γ δ ys=y t (12) s λ p=0 s t E. Energy Minimization To perform the energy minimization, we se the graph cts expansion algorithm, specifically the implementation presented in [6]. We perform the minimization in two stages. Althogh the two labeling problems are copled, the reslts, at least in terms of their appearance, were more accrate when performed in this way. We first minimize the energy of the mid-leel MRF to obtain an approximation to the optimal labeling of planar srface pixels. This step ses prior knowledge from the discriminatie classification. Next, we se the mid-leel labeling as well as the detected candidate planes as a prior for the high-leel MRF, and we se graph cts again to compte an approximation to that optimal labeling. III. EXPERIMENTAL RESULTS We hae performed two experimental stdies sing or method on a new benchmark dataset 1. We are not aware of another pblicly aailable, hman-annotated, stereo bilding dataset. Or fll data set consists of 142 grayscale images from the left camera of a stereo imager 2, each with a corresponding 16-bit disparity map. All images hae resoltion and hman-annotated grond trth. We sed 100 randomly selected images for training or discriminatie classifier. Of the remaining 42 images, 21 were sed for testing; 21 images were exclded from the test set becase they are negatie examples or the facades were too distant for the stereo camera to generate sefl disparity ales. A. Single Plane Segmentation Or testing data set contains 13 images which featre only one plane in the scene. In order to alidate or methods, we 1 Aailable at jcorso/r/gbs 2 Tyzx eepsea V2 camera with 14 cm baseline and 62 horizontal field of iew. TABLE I SINGLE PLANE MOELING ACCURACY Image # Mid F-score High F-score Angle ( ) Ag: first tested the accracy of or approach for both labeling and parameter estimation on this restricted data set. For each image, we hand-labeled a grond trth segmentation, and sed only that region of the image to determine a grond trth plane with RANSAC. Next, we applied or method for inferring the segmentation labels and plane parameters and compared the reslts with or grond trth by compting the F-score for each mid- and high-leel labeling, and the dihedral angle between the grond trth and primary estimated plane from the high-leel labeling. In the case of the mid-leel labeling, the f-score represents the accracy of labeling bilding pixels from backgrond, and for the highleel it represents the accracy of labeling the correct plane within the foregrond. It shold be noted that in most cases, seeral candidate planes were generated and then labeled by the MRF, bt for or planar comparisons, we mst select only one to compare with the grond-trth. This portion of or method is not yet atomated, and we manally selected the best candidate plane from the set of applied labels. In most cases, this choice coincided with the label with the largest segmentation area, and was also generally the most ertical of the candidate planes. Bt in a few cases, notably image sets 12 and 13, the choice was somewhat ambigos based on area or the magnitde of the component of the normal. Ftre work will be deoted to refining this portion of or method for disambigating labels in the absence of grond-trth. Or reslts are smmarized in Fig. 4 and Table I. Or reslts on the single-plane images indicate that or proposed method is robst and highly accrate. The aerage f-scores for both the mid-leel and high-leel labeling tasks are both aboe 80%, and the aerage dihedral angle is Additionally, all of the planes were accrate to within 10 of their grond trth orientations. It shold be noted that althogh this image set was restricted to those with single facades, all of the images are of natral scenes which inclde occlsions and differences in illmination, the bildings were captred at different angles, and the bildings themseles hae different sizes, textres, and architectral featres.

6 TABLE II MULTIPLE PLANE MOELING ACCURACY Image # Mid F-score High F-score Angle ( ) Ag: Fig. 4. Segmentations and planar facade estimates on single-facade images. For each example, they are (L to R) the original image, grond trth segmentation, high-leel MRF labeling. B. Mltiple Plane Segmentation We now proceed to analysis of more sophisticated scenes that inclde mltiple facades at different orientations. We selected the 8 images from or testing set that contained at least 2 facades in the scene. Similarly to the single-plane experiments, we hand-labeled a grond trth segmentation, and for each grond-trth facade, sed only that region of the image to determine a grond trth plane with RANSAC. When the MRF segmentation prodced more than one plane label oer a grond-trth plane region, we again manally chose the best of those candidates for or comparison of anglar accracy to the grond trth. Once again, the best candidate plane often coincided with the largest label area and most ertical plane, bt not consistently. For example, in image set 1 of Fig. 5, the segmentation of the left facade is diided among white, light gray, and dark gray labels, bt we we compte the dihedral angle only for the dark gray candidate plane becase it was the most accrate to the grond-trth for that region. Or reslts are smmarized in Fig. 5 and Table II. As with the single-plane experiments, the accracy of labeling the primary facades in the image is generally ery good, and in most cases does not reqire any manal disambigation for the anglar accracy. Howeer, for some of the minor facades in the image (either small, distant, or obliqely angled) the error in either the labeling or the plane estimation, or both, was large. In particlar, image sets 4 and 6 in Fig. 5 show the difficlty in achieing good estimates for distant planes, as they hae large errors in the plane estimation, and both plane estimation and labeling, respectiely. It shold be noted, howeer, that the plane estimation errors in these cases are primarily in the

7 rotation abot the -axis, leading to plane estimates which are reasonably accrate in their rotation abot the -axis, bt are far from being ertical. Since we are working nder the assmption that most bildings hae pright facades, it may be possible in the ftre to apply a constraint to correct these estimates. C. Analysis Aeraged oer all 21 images in or testing set, the midleel labeling achieed an accracy of , and the highleel labeling achieed an accracy of The aerage dihedral angle among all of the 31 grond-trth planes in the 21 images was The accracy of or methods on natral scenes is correlated with the qality of the disparity maps, as well as the location of the facades within the sable range of the stereo camera. For example, the camera sed to captre or dataset can only resole featres p to 45 cm at a distance of 15 m. Ths, een moderately distant facades are likely to be significantly more prone to large errors in their estimates; they will be both small in the frame and less likely to find an accrate consenss set in RANSAC de to the ncertainty in their disparity ales. Similarly, for a facade with many inalid disparity ales, it may not be sampled adeqately, and the points it does hae may erroneosly be inclded as part of an inlier set that does not actally lie on the facade. Limitations of this natre will depend on the specific stereo imager sed, bt in general, a more accrate and dense disparity map will enable the discoery of a more accrate candidate plane set. One of the drawbacks of or method is that for alidation, we manally disambigate inconsistent labels for or accracy measres. For localization and naigation prposes, this may not be an isse, as a mobile platform may hae hae odometry information that can track the planes in the scene and help distingish between these labels. In the absence of grond-trth information, the mobile platform cold also consider the power set of candidate plane labels to find if one set corresponds well with its other semantic information abot the srronding area. Howeer, we intend to inestigate this area frther in order to deelop a more atomated way of determining the best plane estimate for each facade in the absence of grond-trth information. Incorporating other semantic information sch as anishing points may help to improe the qality of both the candidate planes and their segmentations. IV. CONCLUSION We present a system for bilding facade segmentation from stereo images with parameter estimation of the identified planar srfaces. Or reslts show high accracy in both detection of bildings ( 90%) and their facades ( 85%), and in estimation of their plane parameters ( 10 ). The performance we hae demonstrated indicates a promising step toward mobile robot localization and gidance ia semantic scene nderstanding. Or intended ftre work in that direction incldes refinement and improement of or methods and eental deployment on a mobile platform as part of a semantic gidance system. A first step will be to perform greater performance anaysis throgh cross alidation with or fll data set and the application of or method to frther data sets that we intend to gather. We also plan to incorporate more semantic information (e.g. throgh anishing point analysis or detection of other objects in the scene) into or framework to improe accracy. REFERENCES [1] J. J. Corso, iscriminatie modeling by boosting on mltileel aggregates, in Proceedings of IEEE Conference on Compter Vision and Pattern Recognition, [2] J. A. elmerico, J. J. Corso, and P. aid, Boosting with Stereo Featres for Bilding Facade etection on Mobile Platforms, in e- Proceedings of Western New York Image Processing Workshop, [3] Y. Frend and R. Schapire, A ecision-theoretic Generalization of On-Line Learning and an Application to Boosting, Jornal of Compter and System Sciences, ol. 55, no. 1, pp , [4] J. J. Corso,. Brschka, and G. Hager, irect plane tracking in stereo images for mobile naigation, in IEEE International Conference on Robotics and Atomation, [5] M. A. Fischler and R. C. Bolles, Random sample consenss: a paradigm for model fitting with applications to image analysis and atomated cartography, Commn. ACM, ol. 24, no. 6, pp , [6] Y. Boyko, O. Veksler, and R. Zabih, Fast approximate energy minimization ia graph cts, Pattern Analysis and Machine Intelligence, IEEE Transactions on, ol. 23, no. 11, pp , [7] T. Korah and C. Rasmssen, Analysis of bilding textres for reconstrcting partially occlded facades, Compter Vision ECCV 2008, pp , [8] A. Wendel, M. onoser, and H. Bischof, Unsperised Facade Segmentation sing Repetitie Patterns, Pattern Recognition, pp , [9] J. Hernández and B. Marcotegi, Morphological segmentation of bilding façade images, in Image Processing (ICIP), th IEEE International Conference on. IEEE, 2010, pp [10] P. aid, etecting Planar Srfaces in Otdoor Urban Enironments, ARMY Research Lab, Adelphi, M. Comptational and Information Sciences irectorate, Tech. Rep., [11] J. Baer, K. Karner, K. Schindler, A. Klas, and C. Zach, Segmentation of bilding models from dense 3 point-clods, in Proc. 27th Workshop of the Astrian Association for Pattern Recognition. Citeseer, 2003, pp [12] S. Lee, S. Jng, and R. Neatia, Atomatic integration of facade textres into 3 bilding models with a projectie geometry based line clstering, in Compter Graphics Form, ol. 21, no. 3. Wiley Online Library, 2002, pp [13] R. Wang, J. Bach, and F. Ferrie, Window detection from mobile LiAR data, in Applications of Compter Vision (WACV), 2011 IEEE Workshop on. IEEE, 2011, pp [14]. Gallp, J. Frahm, and M. Pollefeys, Piecewise planar and nonplanar stereo for rban scene reconstrction, in Compter Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010, pp [15] J. Kosecka and W. Zhang, Extraction, matching, and pose recoery based on dominant rectanglar strctres, Compter Vision and Image Understanding, ol. 100, no. 3, pp , [16] W. Zhang and J. Kosecka, Image Based Localization in Urban Enironments, in Proceedings of the Third International Symposim on 3 ata Processing, Visalization, and Transmission. IEEE Compter Society, 2006, pp [17]. Robertson and R. Cipolla, An image-based system for rban naigation, in Proc. of British Machine Vision Conf., ol. 1. Citeseer, 2004, pp [18] H. Hoppe, T. erose, T. champ, J. Mconald, and W. Stetzle, Srface reconstrction from norganized points. Compter Graphics(ACM), ol. 26, no. 2, pp , 1992.

8 Fig. 5. Segmentations and planar facade estimates on mlti-facade images. For each example, they are (L to R) the original image, grond trth segmentation, high-leel MRF labeling, and 3 plane projection. In the plane projection plots, the perspectie of the original image is looking down on the 3 olme from the positie -axis. The grond-trth planes are in ble, and the estimated planes are in green (iew in color).