KNOWLEDGE-BASED TOPOLOGICAL RECONSTRUCTION FOR BUILDING FAÇADE SURFACE PATCHES

Transcription

1 KNOWLEDGE-BASED TOPOLOGICAL RECONSTRUCTION FOR BUILDING FAÇADE SURFACE PATCHES Yxang Tan a,b,, Qng Zhu b, Markus Gerke a,george Vosselman a a Internatonal Insttute for Geo-Informaton Scence and Earth Observaton (ITC), Hengelosestraat 99, P.O.Box 6, 7500AA, Enschede, the Netherlands - (ytan, gerke, vosselman)@tc.nl b State Key Lab of Informaton Engneerng n Surveyng Mappng and Remote Sensng, Wuhan Unversty, 129 Luo Yu Road, Wuhan, Hube, , P.R. Chna zhuq66@263.net KEY WORDS: topologcal, façade, reconstructon, vdeo mage sequence ABSTRACT: 3D cty models constructed from ground based data are becomng an nterestng and challengng problem as they present the realstc facades, whch contan more detals than the models constructed from aeral data. Such knd of nformaton s nterestng for qute a lot of applcatons. Ths paper presents a new method for connectng buldng façade surface patches that generated from vdeo mage sequence, whch ntegrates buldng structure knowledge nto reconstructon. Therefore reasonable and correct topologcal relatonshp can be bult up between them even when some surface patches are not observed or wrongly detected. The results show our method correctly set up topologcal relatonshp between generated surface patches and gettng reasonable structure model about area wth occlusons. 1. INTRODUCTION Nowadays, reconstructng 3D cty models has become an nterestng and challengng problem n computer vson and photogrammetry. Ths s because of the ncreasng demand for 3D models from several socety felds such as urban plannng, archtectural desgn, emergency response and vrtual toursm. Commonly, ground based object extracton has manly reled on manual operatons wth the support of some commercal 3D modelng packages such as 3D Studo Max, and fnally the 3D model s textured by manually selectng certan parts from mages. Due to the huge number of urban objects n a cty and varety of shapes, manual reconstructon of a cty s rather tmeconsumng and expensve procedure (Brenner, 2005). Automated methods for relable and accurate 3D reconstructon of manufactured objects are essental to many users and provders of 3D cty data. Many processes have been reported n the areas of sem-automated and fully automated reconstructon (Werner and Zsserman, 2002; Dck et al., 2004; Mayer, 2008; Remondno et al., 2008). However, they are not always success n practcal applcatons. In general, there are two remote sensng technques for 3D reconstructon of real objects and scenes that are commonly used. One s based on actve range data (e.g., laser scannng), and the other s based on camera or vdeo mages. Actve range based modelng methods drectly capture the 3D geometrc nformaton of an object. They provde a hghly detaled and accurate representaton of shapes. On the other hand, they can be cumbersome and expensve. Image-based 3D modelng generally requres some user s nteracton n the dfferent steps of the modelng ppelne. Because recoverng relable three dmensonal nformaton from two dmensonal mages s stll a problem (Remondno et al., 2008). However, economc and flexble data acquston procedures together wth the automatc structure from moton approach are attractve advantages of usng vdeo mage sequence as the source data for object reconstructon. In recent years, there has been ntensve research actvty on the reconstructon of 3D objects and scenes from mage sequences or mage sets, especally n the feld of computer vson (Mayer and Reznk, 2007; Cornels et al., 2008; Pollefeys et al., 2008). (Cornels et al., 2008) presented a system for real-tme cty modelng that employs a smple model for geometry. More common way s usng textured polygonal meshes to present the objects. As a majorty of buldngs n exstence nowadays satsfy the assumpton that they can geometrcally be modelled as an ensemble of planar polygonal surface patches, usng polyhedral models seem to be a relatvely smple and effcent way to present buldng structures (Werner and Zsserman, 2002). Such representatons that buldng models wth detaled roof structures and planar facades are for example suffcent for smulatons or vsualzatons at small or medum scale. However, from vdeo mage sequence that captured from monocular camera even other ground based mages, there are always some edges or surface patches cannot be observed. And occlusons of buldngs or buldngs parts by themselves or other objects n front of them cause falure n complete surface patch generaton. Therefore, only the actually observed/detected features are connected f usng a data-drven method only. However ths may not ft the actual stuaton. At the other hand, the enormous varatons n the structure and shape of the buldng facades prevent to use too tght constrants to recover the structure. In ths paper the knowledge of buldng structure s ntegrated nto reconstructon step. The ntegraton of knowledge does not complcate the processng rather t smplfes the reconstructon process. The term knowledge s wdely used n many mage Correspondng author

2 analyss methods and t may descrbe any knd of nformaton (Baltsavas, 2004). In ths paper, knowledge s rules or constrants that are retreved from a general buldng façade structure. Topologcal propertes are not metrcal, but concern such thngs as connectvty and dmensonal contnuty. Such character makes them useful to be consdered n buldng modelng (Amer and Frtsch, 2000; Heuel and Förstner, 2001). So, our buldng models contan both geometrc and topologcal nformaton and our emphass s on quckly recoverng the shape of buldng. The paper s organzed as follows: Secton two descrbes the prevous work of ths research. In secton three, the method to set up topologcal relatonshp between buldng façade surface patches are ntroduced. Secton four presents and dscusses some buldng reconstructon results. Conclusons are presented n secton fve. 2. PREVIOUS WORK The method for extractng pont and edge features from vdeo mage sequence s presented n (Tan et al., 2008). A bref ntroducton s gven here for t s the basc of our work and the feature ponts and edges are used n the followng steps. Startng from 3D ponts tracked from vdeo mage sequence, pont accuracy s analyzed frst to obtan relable matched ponts. Only edges near the relable matched ponts are consdered as edge canddates and all the edge canddates are used to estmate a 3D edge. Based on the estmated varance factor, only good 3D edge estmaton are accepted, whch ensure the accurate poston of matched 3D edges. In order to ntroduce more constrants for reconstructon and fll the gaps n 3D pont clouds, 3D edges are also used as prmtves for reconstructon. An automatc approach to generate surface patch s presented n (Tan et al., 2009). Ths step has strong relevance wth buldng topologcal reconstructon. Our method to get the surface patches can be dvded nto four steps. Frst, plane hypotheses are formulated from cues based on pont-cloud segmentaton and on some of the 3D edges derved beforehand. The hypotheses then are verfed by ncorporatng unused 3D edges. Afterwards, plane parameters are obtaned by all the edges and ponts n the plane. The one wth the least resdual RMS s chosen as the best ft. The fnal step s to defne the surface patches outlne. In our approach, walls and roofs are man structures of buldngs. Extrusons, lke wndows and doors, whch are attached to the façade planes, are not consdered n the surface patch generaton step. Fgure 1 shows the surface patch generaton result. Fgure 1. Surface generaton result, 2D vew (above), 3D sde vew (down), relable ponts (dot), extracted edges (fnte lne), surface patches (polygon) The above rules and processng steps can reasonably group extracted sparse 3D ponts and edges. Although the reconstructed surface patches present the buldng façade geometry, three aspects need to be mentoned. Frst, f there s no pont or edge extracted from a surface patch, the pure geometrc reconstructon fals to determne t. Secondly, small surface patches appear between surface patches whch should be adjacent. Thrd, the outlnes stll need to be modfed. Fgure 1 also llustrates these aspects. 3. TOPOLOGICAL RECONSTRUCTION Ths secton presents the method for connectng buldng façade surface patches beng generated from a vdeo mage sequence, ntegratng buldng structure knowledge nto reconstructon. From the structural nformaton of buldng we specfy our assumpton about buldng models and dvde knowledge nto two dfferent knds, the oblgatory one and the preferental one. Based on the oblgatory knowledge, buldng types are restrctng to some extend as ths method only ams at ordnary buldngs not specfc archtectures. The knowledge appled for ths purpose s: All the buldng facades are planner. Walls are vertcal. Roof ntersect wth walls As we known, there are many dfferent knds of buldngs n the world. Some of them wll be neglected n ths approach f they are not satsfyng above rules. Durng fndng the adjacent surface patch and further makng model hypothess based on two adjacent surface patches, the preferental knowledge provdes essental gudance when there s no other evdence: Buldng ground plane mostly has rectangular angles. Structural regularty, often as result of economcal, manufacturng, functonal, or aesthetc consderatons, results n smple models can descrbe basc buldng models. Repeatng structures are also an essental component n archtecture desgn, such as wndows of one buldng are smlar. Our method for topologcal reconstructon can be dvded nto three steps. Frst, surface patch neghbourhood relatonshp are set up. The local model hypotheses then are made based on adjacent surface patches. Afterwards, all local models are

3 connected to form a whole buldng model. The detals are presented n the followng. 3.1 Searchng the neghbourhood surface patches There are two knds of neghbourhood relaton between surface patches. One s the two surface patches can form a volume, such as surface 1 and 2 n fgure 2. The other one s one surface patch s attached to another, e.g., surface 2 s attached to surface 3 n fgure 2. These two types can be separated by the locaton of ntersectng lne and surface patches. n1 s v1 n2 v1 Fgure 3. Coherent adjacent surfaces 3 Fgure 2. Surface patches neghbourhood relaton Obvously, f two surface patches contan the same edge, they must have some relaton. Due to the mperfect surface patch generaton result, the boundares of extracted surface patches are not always consstent wth the actual case. Some surface patches need to be extended or subscrbed nto small parts. Most of tme, the adjacent surface patches are searched along boundary edges that formed by two boundary ponts. If two surface patches are adjacent, as shown n fgure 2, the projectons of surface patches to ther ntersectng lne must overlap and the ntersectng lne s perpendcular to the normal vectors of surface patches. So for two ntersectng surface patches, the overlap for ther projectons on ntersectng lne s compute frst. The angle between object boundary edge and other surface patches s one parameter used to evaluate the found surface patch. As there are usually some parallel surface patches n buldng façade, the dstance between object boundary edge and ntersectng lne s another evaluator that needs to be consdered. So the one that wth smallest dstance and s almost perpendcular to the object boundary edge s consdered as the adjacent surface from that boundary edge. 3.2 Constructng local model Ths step only consders surface patches wth the frst knd of neghbourhood relaton Surface coherence If two surface patches form a volume, as we defned as the local model, one mportant thng need to be consdered s surface coherence that means the normal vectors of them should be consstent to pont nteror or exteror to the volume. As shown n fgure 3, two adjacent surface patches are coherent along ther common edge f: [ v, n ] [ s, v, n ] 0 s, < (1) Where [ a b, c] 1 2, stand for the trple vector product. Fgure 4. Non coherent adjacent surfaces Fgure 4 shows some counter examples. If two nearby surface patches are not coherent, one of them needs to be modfed and a new surface patch hypothess can be made based on them to form a sutable local model. Fgure 5. New surface patch hypothess Fgure 5 shows one example how to make a new surface patch hypothess. We extend one surface patch versa the common edge as the preferental knowledge shows nearby thngs are smlar compare to far away ones. The new surface patch and the unmodfed surface patch must coherent and ther normal vectors should pont to exteror of the volume Surface patch verfcaton These new surface patch hypotheses need to be verfed by ntensty smlarty over the mage sequence as we ddn t fnd evdence from extracted ponts and edges n those areas. If a surface patch s vsble n the mages, we can get a hgh crosscorrelaton result for the ponts on t between any two mages. The hypothess s accepted or rejected accordng to the average cross-correlaton value for ponts n vald mages. In more detal, gven the planeπ, there s a homographc represented by 3 3 matrx H between the frst and th frame, so that correspondng ponts are mapped as x Hx 0 = (2) Where x 0 and x are mage ponts represented by homogeneous 3-vectors.

4 The homography matrx s obtaned from 3 4 camera projecton matrces for each frame. For example, f the projecton matrces for the frst and th frame are P 0 = [ I 0] and P = [ A a], and a plane defned by T T T π X = 0 wthπ = ( v,1), then the homography nduced by the plane s: surface patch can not be observed from mage sequence, we also accept the hypothess. H T = A av (3) In our process, we consder the frst frame that the surface patch s vsble as the reference frame and compute the crosscorrelaton between the reference frame and frames wthn vsblty frame range. The homography for r T r x = H r x r s: H = H H (4) Where presents th frame, r presents reference frame The ponts of nterest are endponts of 2d edge extracton result wthn the projected mage area. If the number of ponts s too few, some ponts are chosen regularty n that mage area. The smlarty score for the average cross-correlaton value for ponts n the vald mages s: sm = Cor 2 ( x, H x) r framevald POI If the smlarty score s hgher than the threshold (0.8 for experence from mages), the hypothess s accepted. Otherwse, the hypothess s rejected and plane sweepng method s used to fnd a more relable surface patch. Based on the common edge, the optmal angle s computed by searchng for the maxmum of π π functon sm over a range [, ] 6 6 wth o 1 each tme. (5) Fgure 6. Example for basc local models 3.3 Connectng dfferent local models For each surface patch, a collecton of pont numbers followng the boundary sequence s recorded. The topologcal relatonshp wth other surface patches can be shown by edges from other surface patches that connected wth each boundary edge. For surface patches that are attached to other surface patches, they also have two dfferent types. If the surface patch s located nsde the local model, t s consdered as the ntruson that s attached to one surface. If the surface patch s located outsde the local model, t must connect two dfferent local models. So the whole buldng model s reconstructed durng connectng local models one by one and presented as a collecton of ponts, pont numbers n sequence and topologcal connectons. 4. EXPERIMENT Fgure 7 shows the reconstructon result for the buldng façade n fgure 1. The vdeo was captured by a hand-held SONY camera. The mages have a resoluton of pxels and a frame rate of 30 frames per second. There are 189 frames n total n ths case Basc local model As the vdeo s captured from ground, only part of roof structure can be observed. Compare wth buldng façade, there may be less features extracted from roof, due to the roof part s smpler than walls t connects wth. Therefore, n our approach roof surface patches are consdered before wall surface patches. Followng the sequence of topologcal connecton, f a roof contans the same edge wth a wall s dealt frst. Then we wll search other walls connectng wth t f there are un-dealt boundary edges after local model generaton. The steps for two adjacent walls are more or less smlar. For those roofs that can not be observed from ground, we assume they are horzontal. After two coherent adjacent surface patches are found, not only ther topologcal relaton can be bult, but also a local model s constructed based on them. Durng ths step, surface patches are searched to support the hypothess. The searchng step s smlar to the above step that searchng the neghbourhood surface patches. But here are two boundary edges should be consdered. If three surface patches that form a local model are all found, the local model s defned by them. Otherwse, we chose smple block types, based on buldng structure defned before, as shown n fgure 6 to ft them. Durng ths fttng step, f the new

5 objects when there are observatons contradct the preferental knowledge. In fact, the method descrbed should be more or less ndependent from the orgnal mage source. However, ths paper presents part of our work named buldng structure recoverng from vdeo mage sequence. So our tests are based on vdeo mage sequence. The results show our method correctly set up topologcal relatonshp between generated surface patches and gettng reasonable structure model about area wth occlusons. ACKNOWLEDGEMENTS The work descrbed n ths paper s supported by Natonal Natural Scence Foundaton of Chna ( and ), and Natonal Hgh Technology Research and Development Program of Chna (2006AA12Z224). REFERENCE Amer, B. and Frtsch, D., Automatc 3D buldng reconstructon usng plane roof structures, ASPRS Congress, Washngton, DC. Baltsavas, E.P., Object extracton and revson by mage analyss usng exstng geodata and knowledge: current status and steps towards operatonal systems. ISPRS Journal of Photogrammetry and Remote Sensng, 58(3-4): Fgure 7. Reconstructon result of buldng façade n fgure 1, 3D vew (top), projecton on frst frame (mddle), projecton on last frame (down) The buldng has three man parts that can be observed from mages. One part only has one floor. Another part has two floors and one bg extruson s attached to t. The result shows the structure of ths buldng s successfully recovered as we seen from the mages. Some unobserved parts are generated accordng to assumptons, whch may be dfferent from realty. However, f there are mages vable for these parts, the models can easly update to the real case. As n ths approach, the man attenton s to set up topologcal relatonshp between generated surface patches and quckly recover the buldng shape. The locaton of each surface patch has small change when ntersectng t wth other surface patches. So the accuracy for surface patches need to be mproved n the further work. 5. CONCLUSIONS The major advantage of the strategy s the easy and natural way n whch varous types of knowledge are ntegrated nto the data-drven processng. The ntegraton of knowledge does not complcate the processng rather t smplfes the reconstructon process. And such model restrctons make the reconstructon much more robust. Meanwhle, our method does not restrct ourselves to basc structures. We can also dentfy complex Brenner, C., Buldng reconstructon from mages and laser scannng. Internatonal Journal of Appled Earth Observaton and Geonformaton, 6(3-4): Cornels, N., Lebe, B., Cornels, K. and Van Gool, L., D Urban Scene Modelng Integratng Recognton and Reconstructon. Internatonal Journal of Computer Vson, 78(2): Dck, A., Torr, P. and Cpolla, R., Modellng and Interpretaton of archtecture from several mages Internatonal Journal of computer vson, 60(2): Heuel, S. and Förstner, W., Topologcal and geometrcal models for buldng extracton from multple mages. In: E.P. Baltsavas, A. Grün and L. VanGool (Edtors), Automatc Extracton of Man-Made Objects from Aeral and Satellte Images III. A.A.Balkema Publshers, Ascona, Swtzerland, pp Mayer, H., Object extracton n photogrammetrc computer vson. ISPRS Journal of Photogrammetry and Remote Sensng, 63(2): Mayer, H. and Reznk, S., Buldng facade nterpretaton from uncalbrated wde-baselne mage sequences. ISPRS Journal of Photogrammetry and Remote Sensng, 61(6):

6 Pollefeys, M. et al., Detaled Real-Tme Urban 3D Reconstructon from Vdeo. Internatonal Journal of Computer Vson, 78(2): Remondno, F., El-Hakm, S.F., Gruen, A. and L, Z., Turnng mages nto 3-D models. Sgnal Processng Magazne, IEEE, 25(4): Tan, Y., Gerke, M., Vosselman, G. and Zhu, Q., Automatc Edge Matchng Across an Image Sequence Based on Relable Ponts. In: J. Chen, J. Jang and W. Förstnner (Edtors), the Internatonal Archves of the Photogrammetry, Remote Sensng and Spatal Informaton Scence Bejng, pp Tan, Y., Gerke, M., Vosselman, G. and Zhu, Q., Automatc Surface Patch Generaton from a Vdeo Image Sequence, 3D Geo-Informaton Scences: requrements, acquston, modelng, analyss, vsualzaton. Sprnger, Berln, pp Werner, T. and Zsserman, A., New technques for automated archtectural reconstructon from photographs, Seventh European Conference on Computer Vson, pp