Automatic 3D Building Reconstruction

Transcription

1 Automatc 3D Buldng Reconstructon Ildko Suveg, George Vosselman Delft Unversty of Technology, The Netherlands Secton of Photogrammetry and Remote Sensng {I.Suveg, ABSTRACT We present a knowledge-based system for automatc 3D buldng reconstructon from aeral mages. Our approach reles on combnng pars of stereo mages wth 2D GIS map and doman knowledge. Snce most buldngs can be descrbed as aggregaton of smple buldng types, the doman knowledge s represented n a buldng lbrary contanng buldng prmtves (flat, gable, and hp roof buldng). The approach of modelng buldngs usng a set of basc buldng models suggests the usage of Constructve Sold Geometry representaton for buldng descrpton. The buldng reconstructon process s formulated as a hypothess generaton and verfcaton scheme. It starts wth the parttonng of a buldng n smple buldng parts based on the ground plan defned n the map. For each buldng partton dfferent buldng hypotheses are generated correspondng to the buldng prmtves defned n the buldng lbrary. The evaluaton of the generated buldng models s based on the formulaton of the mutual nformaton between the model and the mages. The CSG tree representng a buldng s gven by the best ft of the buldng models correspondng to the buldng parttons. We used ths method to reconstruct buldngs n suburban and urban scenes. The method worked well even n dffcult condtons (nose, shadow). Keywords: 3D reconstructon, mnmum descrpton length, model fttng, evaluaton, mutual nformaton 1. INTRODUCTION 3D reconstructon of buldngs has been an actve research topc n computer vson n recent years. 3D reconstructon of buldngs from aeral mages s becomng of ncreasng practcal mportance. Tradtonal applcatons are those of cartography and photo nterpretaton. Newer applcatons nclude urban plannng, constructon, envronment, communcaton, transportaton, energy and property management, toursm, and vrtual tours of ctes. Photogrammetrc methods are well establshed but show neffcences due to the extensve amount of data. Manual 3D processng of aeral mages s very tme consumng and requres hghly qualfed personnel and expensve nstruments. Therefore, speedng up ths process by automatc or semautomatc procedures has become a necessty. The current state of automaton n the reconstructon of buldngs from aeral mages s stll surprsngly low. A lot of algorthms and systems have been proposed towards ths problem. However, a versatle soluton to the automatc reconstructon has not been found yet, wth only partal solutons and lmted success n constraned envronments beng the state of art. The dffculty n obtanng a general soluton to ths problem can be attrbuted to the complexty of the reconstructon tself, as t nvolves processng at dfferent levels: low level processng (feature extracton), mddle level processng (representaton and descrpton of buldng models) and hgh level processng (matchng and reasonng). The success of a reconstructon system depends upon the succeedng at all these levels and n combnng these levels of processng. Most approaches have focused on the reconstructon of specfc buldng models: rectlnear shapes [13, 15], flat roofs [8, 9] or parametrc models [4]. But buldngs show a much wder varety n ther shape. Other approaches employ a generc roof model that assumes planar roof surfaces [3, 12, 16]. These 3D roof planes are generated by groupng the coplanar 3D lnes or corners computed from the mages. Nevertheless, the feature extractors can fragment or mss boundary lnes, due to low contrast, occlusons, and bad perspectve. To overcome these problems, the mage data has to be combned wth other data sources, for example fusng mages wth scanned [11] or dgtal map [7]. These approaches represent the newest trend n 3D buldng reconstructon. Three-Dmensonal Image Capture and Applcatons V, Bran D. Corner, Roy P. Pargas, Joseph H. Nurre, Edtors, Proceedngs of SPIE Vol (2002) 2002 SPIE X/02/$

2 Our strategy for 3D reconstructon of buldngs combnes pars of stereo mages wth large-scale Geographc Informaton System (GIS) maps and doman knowledge as addtonal nformaton sources. The 2D GIS map contans the outlne of footprnts of the buldngs. The knowledge about the problem doman s represented by a buldng lbrary contanng prmtve buldng models. Although, buldngs reveal a hgh varablty n shape, even complex buldngs can be generated by combnng smple buldng models wth flat, gable or hp roof. Ths paper s organzed as follows: Secton 2 presents an overvew of the steps nvolved n the proposed method for 3D reconstructon of buldngs. The parttonng of a buldng and rankng of the resultant parttonng schemes are dscussed n secton 3. The next secton descrbes the generaton of buldng hypotheses. Secton 5 presents an evaluaton functon based on mutual nformaton for determnng the best buldng hypothess. The refned reconstructon of a complete buldng s descrbed n secton 5. Secton 6 shows some expermental results. The conclusons and future work are dscussed n the fnal secton. 2. METHOD OVERVIEW The buldng reconstructon process s formulated as a mult-level hypothess generaton and verfcaton scheme and t s mplemented as a search tree. The tree s generated ncrementally by the search method. To cope wth the complexty of aeral mages specfc knowledge about buldngs have to be ntegrated n the reconstructon process. Snce most buldngs can be descrbed as an aggregaton of smple buldng types, the knowledge about the problem doman can be represented n a buldng lbrary contanng the smple buldng models (flat roof, gable roof, and hp roof buldng). The approach of modelng buldngs usng a set of basc buldng models (prmtves) suggests the usage of Constructve Sold Geometry (CSG) representaton for buldng descrpton. In ths way, a complex buldng can be seen as a CSG tree, where the leaf nodes contan prmtve buldng models and the nternal nodes contan boolean operatons such as unon, ntersecton, dfference. The reconstructon process starts wth the parttonng of a buldng n smple buldng parts. All possble parttonng schemes of a buldng are represented on the frst level of the search tree. The second level of the tree contans the parttons correspondng to each parttonng scheme. Next, the tree s expanded wth a level correspondng to dfferent buldng hypotheses generated for each buldng partton. Correspondng to each buldng prmtve defned n the buldng lbrary, buldng hypotheses are generated. The estmaton of the parameters of the buldng model s performed usng a fttng algorthm. The buldng hypotheses are verfed by back projectng them nto the mages and then matchng wth the nformaton extracted from the mages. The matchng defnes a score functon that s used to gude the search n the tree. The CSG tree representng a buldng s gven by the best ft of the buldng models correspondng to the buldng parttons. In the fnal verfcaton step the complete CSG tree s ft to the mage data. To mprove the results, constrants, whch descrbe geometrc relatonshps between buldng prmtves, are ncorporated n the fttng algorthm. 3. PARTITIONING OF A BUILDING The frst step of the actual reconstructon process s the parttonng of a buldng n smple buldng parts, whch mght correspond to the buldng models defned n the buldng lbrary. Frst, the parttonng s done usng only the ground plan of the buldng defned n the GIS map. If the ground plan of the buldng s not a rectangle, then t s dvded n rectangles, called parttons. The lne segments created by parttonng are labeled as nternal lnes, whle the orgnal house segments are labeled as external lnes. A parttonng scheme can be defned as a subdvson of a buldng nto dsjont parttons. These parttonng schemes are generated by sequental mergng of the parttons. Two parttons can be merged f they share a common edge. A buldng can have multple parttonng schemes (fg. 2), more or less lkely to occur n realty. Each of these parttonng schemes wll start up a branch n the search tree. The number of possble parttonng schemes ncreases wth the complexty of a buldng. Due to the exponental complexty of the tree search, searchng all the paths would be very tme consumng. Therefore, we consder only the parttonng schemes that have some support n the mages. The goal s to defne a measure of ths support and to rank the parttonng schemes based on ths measure. Optmzng the order, n whch the parttonng schemes are consdered n the search tree, can reduce the number of parttonng schemes that have to be verfed and, thus, the reconstructon tme. In ths way the most lkely parttonng schemes are evaluated frst and then the search s stopped when an optmum buldng model s found, thus mnmzng the number of unnecessary verfcaton of parttonng schemes. 60 Proc. SPIE Vol. 4661

3 a) Fgure 1. a) Condtonal probablty densty P(l l p ) b) Probablty densty functon of the mage lne length P(l ) c) Mutual nformaton I(l ; l p ) For rankng parttonng schemes, a metrc based on mutual nformaton has been developed. Ths metrc employs mage nformaton, namely mage lnes. The mage lnes are extracted usng the Foerstner operator [5]. The metrc measures the support that the mage lnes gve to a parttonng scheme. Ths support should be hgh for a good parttonng scheme, and t should be negatve f there s no mage nformaton to support a parttonng scheme. Therefore, the best parttonng scheme can be defned as the one wth the hghest mutual nformaton. The formulaton of mutual nformaton s largely credted to Shannon (1948). Snce then, there have been many uses of mutual nformaton especally n communcaton theory. In computer vson mutual nformaton was used for relatonal matchng [21] and for mage regstraton [22]. The mutual nformaton between two descrptons M = {m 1, m 2, } and D = {d 1, d 2, } s defned as the dfference between the self-nformaton and the condtonal nformaton [23]: I(m ; d j ) = I( d j ) - I( m d j ) (1) where I( d j ) = -log P( d j ) and I(m d j ) = -log P(m d j ) Thus the mutual nformaton can be wrtten as: P(d j m ) I(m ;d j) = log (2) P(d j) The metrc used for rankng of parttonng schemes s based on the formulaton of mutual nformaton between the length of the lnes of parttonng schemes and the length of the lnes extracted from mages. To calculate the mutual nformaton between an mage lne length l and a partton lne length l p, we have to know the condtonal probablty densty of the mage lne length gven the partton lne length P(l l p ). Ths expresson can be gven by: l lp 1,f 0 l lp l 2 p(l p l p) = l (3) 1 l p 0, else Ths functon s shown n fgure 1a for the buldng from fgure 2. The length of the mage lne can be at most length of the projected partton lne l p. Snce the probabltes of the lnes length n the parttonng schemes are known, the probablty densty functon of the mage lne length can be calculated by: p(l ) = p(l l )P(l ) (4) p p lp Fgure 1b shows the probablty densty functon of the mage lne length, wthout the knowledge of the projected partton lne length. Conform (1) the mutual nformaton between an mage lne length and a partton lne length can be calculated as: p(l l p) I(l p;l ) = log (5) p(l ) Proc. SPIE Vol

4 a) b) Rank = 4.21 c) Rank = 3.35 d) Rank = e) Rank = f) Rank = Fgure 2. a) Lnes extracted from mage b) f) Possble parttonng schemes and t s shown n fgure 1c. The rank of a parttonng scheme s gven by the sum of the mutual nformaton between ts lnes and mage lnes: I(P;ID) = I(l l ) (6) lp p Clearly, the external house lnes do not provde any nformaton about a parttonng scheme, therefore ther support can be consdered null. On the other hand, the nternal house lnes are more mportant than the external house lnes and thus wll have a large mpact on the rank measure. Ther support can be computed usng (5). Another crteron that has to be consdered at the rankng of the parttonng schemes s the smplcty, expressed by Mnmum Descrpton Length (MDL) crteron. Ths crteron provdes a means of gvng hgher prorty to the parttonng schemes wth a smaller number of parttons. The measure of the smplcty then becomes the number of bts necessary to descrbe the parttonng scheme. For an optmal descrpton ths s gven by: log 2 n where n s the number of parttons composng the partton scheme. Hence, the rank of a parttonng scheme s gven by: Rank(PS) = I(P;ID) log n (7) If the rank of a partton scheme s hgh enough then the partton can be classfed as standalone partton and t s not used n further partton mergng operatons. Thus the number of parttonng schemes s reduced. Usng ths rankng method for orderng the possble parttonng schemes, we can avod a blnd search of the tree. An example s gven n fgure 2. There are fve possble parttonng schemes for that buldng. These parttonng schemes together wth ther ranks are presented n fgure 2 b-f. The hghest rank was assgned to the frst parttonng scheme. Other examples are presented n fgure 3. Here only the best ranked parttonng schemes are shown. Fgure 3. Best ranked parttonng schemes selected from the lst of all possble parttonng schemes 4. GENERATION OF BUILDING HYPOTHESES After the parttonng process the next step s the generaton of buldng hypotheses. For each partton of the parttonng schemes dfferent buldng hypotheses are generated correspondng to the buldng models defned n the buldng lbrary. To avod multple generaton of buldng hypotheses for a partton ncluded n more parttonng schemes a dynamc programmng method s used. The generated buldng hypotheses are represented on the last level of the search tree. The basc buldng models n the buldng lbrary are descrbed by parametrc models havng pose and shape parameters. For nstance to descrbe a flat roof buldng 6 parameters are necessary: wdth, length, heght, x, y coordnates of the buldng reference pont and the orentaton n the xy-plane. For a gable roof an extra parameter, the heght of the rdge has to be consdered. The parameters of the model are estmated n a two-step method. Frst an approxmate estmaton s done based on the nformaton from the map and 3D nformaton extracted from mages. Afterwards the values of the parameters are mproved by fttng the model to the mages. 62 Proc. SPIE Vol. 4661

5 5.1 Reconstructon of 3D corners and 3D lnes For the approxmate estmaton of the parameters of a buldng model 3D nformaton are extracted from mages. Ths 3D nformaton conssts of 3D corners and 3D lnes obtaned by matchng 2D features extracted from dfferent mages. To cope wth the combnatoral complexty of the matchng, many constrants have been ncorporated n the matchng process. The constrants used for corner ponts matchng are: Eppolar geometry: the eppolar constrant s appled to restrct the search for correspondences along one lne, the eppolar lne. Heght: The 3D pont obtaned by trangulaton from two 2D corner ponts must have a heght between the mnmum and maxmum heght of the buldng. Ths problem s dentcal to the determnaton of the dsparty search range along the eppolar lne. Ground plan of a buldng: The 3D pont must le nsde or suffcently close to the ground plan of the buldng defned n the map. The problem of matchng lne segments s slghtly more complex than that of matchng ponts, because the lne segment extracton algorthm often produces dfferent results n the two mages. Hence, two lne segments generally do not correspond globally and only contan a subset of homologous ponts. The constrants used for lne segments matchng are: Eppolar geometry: The lne segment n one vew must le at least partally wthn a bean defned by the eppolar geometry and the buldng heght constrants. Consder fgure 4. By the eppolar constrants for ponts, the match for A must le on the eppolar lne ep1, defned by A. Smlarly, a match for B must le on ep2. By knowng the heght range of buldng, the search space can be reduced to the segments on the eppolar lnes defned by extreme heght values. Hence, the search space for matchng lne segments s lmted to a four-corner polygon. Each lne segment at least partally nsde of ths polygon can be matched wth the ntal lne segment. Unqueness: Ths constrant mposes that one lne segment from the frst mage has, at most, one correspondng lne segment n the second mage (a symmetrc constrant apples to a lne segment of the second mage). Order: Ths constrant assumes the preservaton of the order of correspondng lnes from the two mages. 5.2 Estmaton of buldng model parameters In the ntal approxmaton the x, y coordnates and the orentaton of a buldng prmtve are gven by the ground plan of the buldng. The wdth and length parameters are the wdth and the length of the rectangle correspondng to the ground plan of the buldng part. The heght of the buldng prmtve s computed takng nto account the heghts of the reconstructed 3D corners of the buldng part. For a gable roof the heght of the rdge s consdered as the heght of the reconstructed 3D top lne f the top lnes were detected n both mages and the 3D lne could be reconstructed. Otherwse, the approxmate poston of the projected rdge n the mages can be deducted taken nto account the symmetry of a gable roof. Then the 3D rdge can be reconstructed by matchng these two approxmate lne segments. At ths stage the estmaton s nfluenced by uncertantes of the knowledge sources. The uncertantes are due to the accuracy of the GIS map, the roof extensons, and estmated heght [17]. In order to handle these uncertantes, a more precse estmaton of the parameters s obtaned usng a fttng algorthm. Ths algorthm fts the edges of the projected wre frame of the model to gradents of the pxels from both mages smultaneously [20]. Ths algorthm s smlar to the one descrbed by Lowe [10]. The fttng method s descrbed as an teratve least-square algorthm. It estmates the changes of the parameter values that have to be appled n order to mnmze the square sum of the perpendcular dstances of the mage pxels to the nearest wre frame edge. An observaton equaton s set up for each mage pxel wthn some range of a wre frame edge. The lnearzed observaton equaton for a pxel j can be wrtten as: K u j E{ u j} = p (8) p = 1 B B1 A1 A2 B2 Fgure 4. Eppolar constrant for matchng lne segments Proc. SPIE Vol

6 where u j s the observed perpendcular dstance of the pxel to the nearest edge of the wre frame, p are the model parameters, K s the number of parameters, and p are the changes of the parameters that have to be estmated. In order to ensure that the pxels wth hgher gradents domnate the parameter estmaton, the squared gradent of the pxel can be used as a weght to ts observaton equaton. 5. EVALUATION OF BUILDING MODELS The 3D reconstructon of a buldng can be seen as a tree search. The search space for the best ft buldng model can be represented as a tree wth the nodes of the tree representng the dfferent buldng prmtve hypotheses. The root node of the tree represents the ntal state, where only the ground plan of the buldng s known. The frst level of the tree contans all the possble parttonng schemes of the buldng ordered by ther ranks. The second level contans the parttons correspondng to each parttonng scheme. The last level of the search tree contans the dfferent buldng hypotheses generated for each buldng partton. Two problems have to be consdered at the search of the tree: Defnton of an evaluaton functon to gude the search for the best soluton Defnton of stops crtera, whch speed up the search by reducng the search space. In ths secton, we descrbe the defnton of an evaluaton functon. From the possble hypotheses of a matchng between an object model and an mage, one wants to select the hypothess that maxmzes some approprate metrc. Therefore, an evaluaton functon s defned to measure for the qualty of the match. Usually, an evaluaton functon s based on error models that descrbe how an mage feature may dffer from what the object model has predcted. Two man categores of approaches for defnng evaluaton functons can be dstngushed. Ad hoc evaluaton functons were used by Ayache [1], Beverdge [2], Grmson [6]. Wth ths approach, components of the evaluaton functon are combned usng trade-off parameters that are determned emprcally. Other class of evaluaton functons s based on statstcal theory. Match qualty measures are often defned usng Bayesan probablty theory ([14], [19]). Our evaluaton functon belongs to ths latest category, usng a mutual nformaton based measure. The evaluaton functon defnes the score of matchng between the hypotheszed buldng model and the mages. Matchng can be seen as a communcaton problem, where the model descrpton M = {m 1, m 2, } s transmtted through a communcaton channel nto the mage D = {d 1, d 2, }. The mage data wll be smlar to the model data but sometmes t s corrupted due to occlusons, nose, etc. The smlarty between the two descrptons can be measured by the mutual nformaton I(M; D). A lot of work has been done on computng matchng scores. Generally, one can dstngush between feature based and ntensty based approaches. The feature-based methods requre segmentaton of the mages before the matchng process. However, usually the segmentaton needs selecton of a threshold. In addton, the extracted features are nfluenced by nose, bad contrast and occlusons n the mage. To overcome these problems we do the matchng between the model and the mages and the evaluaton of the matchng at the lowest level of abstracton, namely at pxel level. The attrbutes dealt wth at ths level are gradents. Another advantage of our evaluaton functon s the smplcty. The dstrbuton of the gradents at random mage ponts P(grad ) and the condtonal dstrbuton of the gradents along the projected roof edges P(grad pont m ) can be determned by tranng (see secton 5.3). These dstrbutons are shown n fgure 5a, b. Thus, the mutual nformaton between an mage pxel and the correspondng model pont s gven by: P(grad po nt m) I (pont m;pont ) = log (9) P(grad ) and t s shown n fgure 5c. Our evaluaton functon gves a postve response where ponts match wth hgh confdence, a negatve response where there s a clear msmatch and zero response n the ponts where there s nether evdence for match nor evdence aganst a match. The mutual nformaton for a model lne s found by takng the sum of the ponts of the lne: I(lne ;lne ) = I(pont ;pont ) (10) m m pontm lnem The total nformaton for a buldng model n both mages s gven by the sum over all ponts on all projected model lnes n all mages: 64 Proc. SPIE Vol. 4661

7 I(M;D) P(grad pont ) P(grad ) 2 m = log (11) k= 1 lnem pontm 5.1 Mnmum Descrpton Length Prncple The goal s to select the model M from a lst of models M = { M 1, M 2, }, whch best fts the mage data, knowng the transformatons. If all the models have had the same complexty, ths could have been acheved by choosng the model wth the hghest mutual nformaton. But the mutual nformaton between a model and the mage data ncreases wth the complexty of a model. Therefore, the mutual nformaton between dfferent buldng models and mage data are not drectly comparable and cannot be used as an evaluaton functon. The problem can be solved usng the MDL prncple. Ths prncple selects the model M wth the shortest complete descrpton of the data, thus the model whch mnmzes L(D M )+L(M ). If the code used for the descrpton s optmal, the length of the descrpton s equvalent to ts nformaton content. L( M ) = I( M ) and L( D M ) = I( D M ) (12) Thus, the mnmum descrpton length prncple mnmzes: I( D M ) + L( M ) (13) By usng the defnton of the mutual nformaton: I( D; M ) = I( D) - I( D M ) (14) the formula (13) can be expressed as: L( M ) + I( D ) - I( D; M ) => mn (15) Snce I( D ) s constant, the expresson I(D; M ) - L(M ) has to be maxmzed. Therefore, t follows that the best model s gven by: M opt :max(i(m ;D) L(M )) (16) and the expresson Score(M )=I(M ;D) L(M ) can be used as an evaluaton functon for the matchng between buldng model and mage data. 5.2 Relaton MAP and MDL It can be shown that the MAP (maxmum a posteror) and the MDL prncple lead to the same soluton [18]. The maxmum a posteror strategy selects the model M that maxmzes the condtonal probablty of the model gven the data D, P(M D). By usng Bayes formula: P(D M )P(M ) P(M D) = (17) P(D) snce P(D) s constant, MAP states that P(D M )P(M ) has to be maxmzed. The mnmum descrpton length prncple mnmzes (13) I(D M ) + I(M ) Ths can be wrtten as: P(grad) grad P(grad model) grad I(grad;model) grad a) b) c) Fgure 5. a) Gradent dstrbuton P(grad ) b) Condtonal probablty densty of the gradent P(grad pont m ) c) Mutual nformaton I(pont m ; pont ) Proc. SPIE Vol

8 I(D M ) + I(M ) = log P(D M ) log P(M ) (18) = log P(D M )P(M ) => mn Hence, MDL maxmzes P(D M )P(M ) lke MAP. The frst term of (17), P(M D) can be compute, whle the second term P(M ) cannot be specfed easly. The smplest specfcaton of the pror probabltes s that they are all the same,.e. P(M ) s a constant. Ths leads to the maxmum lkelhood strategy of choosng M that maxmzes P(M D). In case that no nformaton about the a pror probabltes s avalable, nstead of MAP crteron MDL crteron can be used, snce MDL does not requre the defnton of a pror probabltes. 5.3 Densty estmaton In order to calculate the score functon for matchng between a buldng model and the mage data as defned n (16), we need to know the a pror probabltes P(grad ) and the condtonal probabltes P(grad pont m ). In estmatng the denstes we don t know the shape of the densty functon whch descrbes the data. Therefore we have to use non-parametrc technques. The easest and the fastest method s hstogrammng. In ths method, the frequency of sample ponts n each bn are counted and then normalzed so that the hstogram frequences all add to one. One alternatve to the hstogrammng could be Parzen wndow densty estmaton [22]. The probabltes of the gradent at random mage ponts can be obtaned drectly from the mages. The gradent dstrbuton s determned as the hstogram of the gradent values n the regons of the mages where there s a buldng. The delneaton of these regons n the mages was descrbed n [17]. The obtaned a pror probablty P(grad ) s shown n fgure 5a. The condtonal probablty densty functon of the gradent along the projected roof edges can be determned from tranng matches by analyzng the probabltes of gradents n these tranng matches. Some mage lnes correspondng to model lnes are selected manually. Next, the hstogram of the gradent values along these lnes s computed. The obtaned condtonal probablty densty functon P(grad pont m ) s shown n fgure 5b. Knowng these two dstrbutons, the mutual nformaton can be computed conform (9) and ths s shown n fgure 5c. 6. CSG TREE FITTING In the fnal verfcaton step the complete CSG tree wll be ft to the mage data. In the prevous stages of our method, the buldng models correspondng to dfferent buldng parttons were treated as solated objects. The results can be further refned f contextual nformaton s utlzed. The fact that the buldng models contaned n the CSG tree form a complex buldng can be seen as contextual nformaton. Between the buldng models of a complex buldng many geometrc relatonshps can be dentfed, whch consttute very valuable nformaton. Therefore, the global fttng algorthm performs a smultaneous adjustment of the buldng models contaned n the CSG tree takng nto account the geometrc relatonshps between them. The geometrc relatonshps between buldng models can be represented by constrants. In the parameter estmaton process, these constrants mean that the parameters of dfferent buldng models are correlated wth each other. The usage of the constrants reduces the degree of freedom of some parameters, therefore the precson of the parameter estmaton s ncreased. Consder the buldng from fgure 9a, composed from a man part and a small attached shed. The estmaton of the parameters of the small buldng extenson can produce problems. By mposng the constrant that the shed s connected to the man buldng, ts parameters can be estmated more precsely, snce a) b) c) Fgure 6. Constrants a) Connecton constrant b) Corner constrant c) Extenson constrant much nformaton can be derved from the man buldng. In our buldng reconstructon system the followng types of constrants are used: Parameter constrant: establshes a relaton between two parameters of two buldng models. For example, two buldng models have the same orentaton. Connecton constrant. One edge of a buldng model les on one of edges of the other buldng model (fgure 6a). Corner constrant. Two buldng models share a common corner (fgure 6b). 66 Proc. SPIE Vol. 4661

9 Extenson constrant. Two buldng models share a common edge (fgure 6c). Constrants can be mplemented as ether hard constrants or soft constrants. A hard constrant mposes a relaton n any condton, whle a soft constrant allows small relaxaton n the specfcaton of the constrant. Our goal s to ntegrate the constrants n the fttng algorthm (8) used to estmate the parameters of the buldng models. We have chosen to mplement the constrants as soft constrants. A soft constrant can be mplemented n a least-square adjustment as weghted observaton wth a standard devaton. The weght specfes the strength of the constrant n the adjustment. At each teraton of the parameter estmaton, the constrants are lnearzed n the neghborhood of the current estmate and then ncluded together wth the observatons equatons correspondng to the mage pxels n the estmaton of the parameters of the buldng. 7. RESULTS The test data conssts of hgh resoluton aeral mages wth the scale of 1:3000. The nteror orentaton parameters of the camera and the exteror orentaton parameters of the mages are known. A 2D GIS map contanng the ground planes of the buldngs s gven. In our current mplementaton, three hypotheses are generated correspondng to a flat roof buldng prmtve and two gable roof prmtves wth dfferent orentatons. Therefore we can reconstruct only flat roof buldngs, gable roof buldngs or buldngs formed by combnng these two buldng types. However, the buldng lbrary can be easly extended wth other prmtve buldng models. Also, we assume that the buldngs have only 90 o corners. Ths s actually a lmtaton of the models defned n the buldng lbrary, snce the current models requre rectangular base. The frst experment was to generate buldng hypotheses for smple buldngs composed by only one buldng prmtve and to select the rght buldng Frst buldng hypotheses derved from outlnes of buldng footprnts from the map are generated correspondng to the buldng models from the buldng lbrary. Next, the buldng hypotheses are ft to the mage data. The scores computed for matchng the hypotheses aganst the mages are used to choose the best model. The resultant buldng models projected back nto one of the mages are presented n fgure 7. The evaluaton functon based on mutual nformaton relably selects the correct model. The buldng models are reconstructed correctly even though some of the roof edges have very bad contrast. Next, we tested our approach on complex buldngs. Frst, the parttonng of the buldng nto buldng prmtves based on the ground plan was performed. Then, for each Score flat = Score flat = Score gable = Score gable = Fgure 7. Reconstructon of smple buldngs a) b) Fgure 8. Refned reconstructon of a buldng. a) Reconstructon wthout constrants Score = b) Reconstructon wth constrants Score = resultant buldng prmtve, hypotheses were generated. Evaluatng the partton schemes we found that the partton schemes presented n fgure 8, 9 are the best ones. Fgure 8 shows the mprovement ganed by ntegratng geometrc constrants between buldng models n the fttng. When fttng wthout constrants the two buldng models are not connected to each other, there s slght dfference n ther orentaton (fgure 8a). Ths problem s solved by ntroducng the connecton constrant between the two buldng models. The evaluaton score ncreased from at the fttng wthout constrants to at constraned fttng. Fgure 10 shows the results obtaned by applyng our approach on an entre scene. Artfcal vertcal walls were added to the automatcally extracted 3D models of the roofs. The results from the proposed approach are encouragng. The method worked well even n dffcult condtons, where feature based approaches would have faled. The constrants ncorporated n the prmtve buldng models Proc. SPIE Vol

10 contrbute to the robustness of the method. Hence the reconstructon has a hgh redundancy and can succeed even n dffcult condtons. 8. CONCLUSIONS a) b) c) d) Fgure 9. Reconstructon of complex buldngs. a, c) Parttonng schemes wth the hghest scores projected back nto mages. b, d) Vrml models correspondng to these parttonng schemes A knowledge-based approach for automatc 3D reconstructon of buldngs from aeral mages was presented. The 3D reconstructon of buldngs was descrbed as a search tree. The generaton of the search tree contanng the multple consstent buldng prmtve hypotheses was descrbed. To search the tree two metrcs based on mutual nformaton were defned. A metrc orders the buldng parttonng schemes such that the most lkely s presented frst for verfcaton to the search tree. Another metrc allows comparson and evaluaton of dfferent buldng hypotheses. Ths metrc s unque n that t compares 3D buldng models drectly to raw mages. No preprocessng or edge detecton s requred. The metrc has been rgorously derved from nformaton theory. The results showed the usefulness of the mutual nformaton as evaluaton metrc. Ths technque works well n domans where edge based methods have dffcultes. Future work wll be drected towards the defnton of the expected amount on mutual nformaton needed for a relable matchng. Ths s gong to be used as stop crtera for the tree search n order to speed up the search. The reconstructon can be further speed up by ncorporatng more knowledge for parttonng the buldngs. References 1. N.Ayache, O.D.Faugeras, HYPER: A Fgure 10. 3D model of a reconstructed scene New Approach for the Recognton and Postonng of Two-Dmensonal Objects, IEEE PAMI, 8, 1, pp , Jan J. Beverdge, R.Wess, E.Rseman, Optmsaton of 2D Model Matchng, In Proc. Image Understandng Workshop, pp , F. Bgnone, O. Henrcsson, P. Fua, M. Strcker, Automatc Extracton of Generc House Roofs from Hgh Resoluton Aeral Imagery, Computer Vson - ECCV 96, Sprnger Verlag, vol.1, pp , A. Fscher, T.H. Kolbe, F. Lang, A.B. Cremers, W. Förstner, L. Plümer, V. Stenhage, Extractng Buldngs from Aeral Images Usng Herarchcal Aggregaton n 2D and 3D, Computer Vson and Image Understandng, Vol. 72, no. 2, Nov W. Foerstner, A Framework for Low Level Feature Extracton, Computer Vson - ECCV94, vol.2, Sprnger Verlag, Berln, 1994, pp W.E.L. Grmson, Object Recognton by Computer: The Role of Geometrc Constrants, MIT Press, Proc. SPIE Vol. 4661

11 7. N. Haala, M. Hahn, Data fuson for the detecton and reconstructon of buldngs, Automatc Extracton of Man- Made Objects from Aeral Images, pp , Brkhauser Verlag, Basel, C. Jaynes, M. Marengon, A. Hanson, E.Rseman, H.Schultz, Knowledge-Drected Reconstructon from Multple Aeral Images, Proc. ARPA Image Understandng Workshop, New Orleans, LA, Vol. II, pp , C. Ln, A. Huertas, and R. Nevata, Detecton of Buldngs Usng Perceptual Groupng and Shadows, Proc. CVPR, IEEE Computer Socety Press, Los Alamtos, CA, pp , D.G. Lowe, Fttng Parameterzed Three-dmensonal Models to Images, IEEE PAMI, 13, 5, pp , May H.Matre, I. Bloch, H. Mossnac, C. Gounaud, Cooperatve Use of Aeral Images and Maps for the Interpretaton of Urban Scenes, Automatc Extracton of Man-Made Objects from Aeral Images, pp , Brkhauser Verlag, Basel, T.Moons, D.Frere, J.Vandekerckhove, L.Gool, Automatc Modelng and 3D Reconstructon of Urban House Roofs from Hgh Resoluton Aeral Imagery, ECCV98, pp , S. Noronha, R. Nevata, Detecton and Descrpton of Buldngs from Multple Images, Proc. CVPR, pp , A.R. Pope, D.G. Lowe, Modelng Postonal Uncertanty n Object Recognton, Unversty of Brtsh Columba, Computer Scence, TR-94-32, November M. Roux, D.M. McKeown, Feature Matchng for Buldng Extracton from Multple Vews, Proceedngs of the ARPA IUW, Menterey, CA, pp , C. Schmd, A. Zsserman, Automatc Lne Matchng Across Vews, Proc. CVPR, pp , I. Suveg, G. Vosselman, 3D reconstructon of Buldng Models, Int. Archves of Photogrammetry and Remote Sensng, vol XXXIII, part B2, pp , I. Suveg, G. Vosselman, 3D Buldng Reconstructon by Map Based Generaton and Evaluaton of Hypotheses, BMVC I. Wess, Statstcal Object Recognton, Ph.D. dssertaton, Cambrdge MIT, G. Vosselman, H. Veldhus, Mappng by Draggng and Fttng of Wre-Frame Models, Photogrammetrc Engneerng & Remote Sensng, vol. 65, no.7, July, pp , G. Vosselman, Relatonal Matchng, Lecture Notes n Computer Scence 628, Sprnger-Verlag, P. Vola, W. Wells, Algnment by Maxmzaton of Mutual Informaton, Proceedngs of 5th Internatonal Conference on Computer Vson, pp , T. M. Cover, J. A. Thomas, Elements of Informaton Theory, John Wley & Sons, Proc. SPIE Vol