AN AUTOMATIC MOSAICKING METHOD FOR BUILDING FACADE TEXTURE MAPPING

Transcription

1 AN AUTOMATIC MOSAICKING METOD FOR BUILDING FACADE TETURE MAPPING Zhzhong Kang a, *, Lqang Zhang b, Ss Zlatanova c a Faculty o Aerospace Engneerng, Delt Unversty o Technology, 69 S Delt, The Netherlands - Z.Kang@tudelt.nl b Research Center or Remote Sensng and GIS, School o Geography, Bejng Normal Unversty, Bejng, Chna - zhanglq@bnu.edu.cn c OTB Research Insttute or ousng, Urban and Moblty Studes, Delt Unversty o Technology, 68 B Delt, The Netherlands - S.Zlatanova@otb.tudelt.nl KEY WORDS: Terrestral magng, Dgtal camera, Strp model, Mosackng, Buldng acade, Texture mappng, Image rectcaton, Least-square adjustment ABSTRACT: Ths paper presents an automatc mosackng method or generatng buldng acade texture rom a sequence o color close-range dgtal camera mages. The constrant o straght lne bundle and the constrant o known orentaton o parallel lnes n object space are used or determnng the dgtal camera s parameters. Smlar to the strp block adjustment used n aeral trangulaton, a strp model o the least-square adjustment s used to determne the relatve spatal relatonshp o the mage sequence. Then a method smlar to orthophoto generaton s employed to generate the entre buldng acade texture. Our results demonstrated that ths method s proved to be more applcable. Two renng strateges are presented to optmze the mosac result. One s renng the mosac regon where correspondng ponts are dcult to be matched but there are plenty o horzontal lnes. The constrant o correspondngly horzontal lnes s ntroduced to mplement ths process. The other s renng the mosac regon by densyng the correspondng ponts accordng to the relatvely spatal relatonshp o the mage sequence.. INTRODUCTION Three-dmensonal (3D spatal models o urban envronments are useul n varety o today s applcatons rangng rom urban plannng, vrtual toursm, hertage protecton, to tranng and smulaton or urban dsaster and emergency scenaros. Dgtal be acqured by terrestral magng technques. Snce the lack o the ndex o mage sequences, the processes o mage searchng, preprocessng, and correspondng are stll mplemented manually. As a result, the extracton o buldng acade texture s a tme-consumng and labor-ntensve task. An alternatve, ast and low-cost opton s urgently needed. mages acqured by remote sensng technques rom terrestral, arborne, spaceborne platorms are currently one o the most mportant data sources to create and update such models. Arborne and satellte magery has been used or the extracton o texture normaton o the top surace o the 3D cartographc Frueh et al. 4 presented an approach to texture mappng a 3D cty model obtaned rom aeral and ground-based laser scans wth oblque aeral magery. In ther method, the mages are rst automatcally regstered by matchng D mage lnes objects (e.g., terran and buldngs. For the producton o wth projectons o 3D lnes rom the cty model. Then, or each ly-through or walk-through and 3D vews to assst envronmental plannng and dsaster management, detaled texture normaton on buldng acades s also requred and can *Correspondng author. Tel: ; ax: trangle n the model, the optmal mage s selected by takng nto account occluson, mage resoluton, surace normal orentaton, and coherence wth neghborng trangles. Fnally, the utlzed texture patches rom all mages are combned nto one texture atlas or compact representaton and ecent renderng. In (Antone et al. 3D models are created usng

2 lnes extracted rom merged camera mages. (Stamos et al., Thrun et al. use laser scanners mounted on a moble robot or scannng buldngs, but the tme requred or data acquston o a entre cty s prohbtvely large; also, the relablty o autonomous moble systems n outdoor envronments s stll relatvely poor. In (Früh, C. et al. a method that s capable o rapdly acqurng 3D model and texture data o an entre cty at the ground level by usng ast D laser scanners and dgtal cameras. Ths approach has the advantage that data can be acqured contnuously, rather than n a stop-and-go ashon, and s thereore much aster than exstng methods. The data acquston system s mounted on a truck movng at a normal speed on publc roads, collectng data to be processed olne. Relatve poston changes can be computed wth centmeter accuracy by matchng successve laser scans aganst each other; however, small errors and occasonal msmatches can accumulate to a sgncantly large level n global poston estmaton o the acquston vehcle, and hence the 3D reconstructed models, ater only a ew hundred meters. In (Takase et al. 3 a system or automatc generaton o 3D cty models has been developed, usng laser proler data, D dgtal map, and aeral mage, whch are processed by the sotware newly developed. MapCube, the 3D cty models generated by the system, has already covered all major ctes o Japan at the end o. Applcatons or vrtual realty (VR that deal wth accurate and photo-realstc 3D cty models are becomng ndspensable n many elds. The acquston o buldng acades textures rom vehcle-based dgtal magng sensors has advantages n resoluton and convenence over oblque arborne mages and rapdness. Ths essental advantage makes land-based moble mappng systems very useul n collectng 3D cty models and other photorealstc walkthrough or drve-through applcatons. Usually or large and tall buldngs, several mages, nstead o only one mage, are requred to cover the acade texture o the whole buldng. A mosackng process thereore s requred n order to obtan the whole acade texture. At present, exstng commercal mage mosackng sotware packages are manly developed to process ether satellte and aeral mages such as ERSI s eyamosac and Supresot s Imageute, or dgtal camera mages to create the vrtual realty panoramc mages such as Panaroma Factory, MGI PhotoVsta, and Cannon PhotoSttch. All o them show satsactory perormance when processng mages wth small geometrc dstorton. owever, mages acqured by land-based moble mappng sensors have large oblquty due to the atttude and drecton o the dgtal magng sensors whch leads to consderable geometrc dstorton. Thereore, alternatve mage mosackng methods that could handle such geometrc dstorton are needed. Because buldng acades are usually planar, there are two groups o lnes parallel to -axs and Y-axs n object space whch are possble to be automatcally extracted rom magery. Thereore the automatc rectcaton and mosackng o such mages are easble. Ths study attempts to develop a novel automatc mosackng algorthm or buldng acade texture mappng usng a monocular close-range dgtal mage sequence. The process ollows our steps,.e. mage rectcaton, subsecton o mage sequence, mage mosac and renng o mosac result. In the sectons that ollow, our approach or automatc mosackng o such an mage sequence wth consderable geometrc dstorton wll be descrbed, tested, and dscussed. IMAGE RECTIFICATION Snce the oblque angle o close-ranged DC mages s large, acade texture n the mage has obvous geometrc dstorton. Thereore, the process o rectcaton o raw mage s necessary to acqure the texture smlar to real acade. The key o mage rectcaton s how to compute orentaton parameters (except the coordnates o projectve center usng mage normaton. The algorthm presented n ths paper s based on vanshng pont geometry and the geometrc constrans o known orentaton o parallel object lnes. Snce buldng acades are planar, there are two groups o lnes parallel to -axs and Y-axs n object space respectvely on each o them. As a result, only ocus and three angular orentaton parametersϕ, ω, k can be calculated accordng to theores o vanshng pont geometry. Facade textures are accordngly rected usng, ϕ, ω, k and the prncple pont whch s xed at the center o mage n ths case. The process o rectcaton s llustrated as Fg.. P s the orgnal mage and P s the rected mage o P on the plumb plane.

3 p: x y system. Because the ntal value o orentaton parameters s decomposed rom vanshng pont coordnates, the accuracy s not always hgh owng to the qualty o mages and the error o lne extracton. Thereore, t has a small convergent radus that S p : x y the model o adjustment system employed the constrant o straght lnes bundle. The model o adjustment system s not robust accordngly. Fg. The process o rectcaton j d S V V Fg. The constran o Straght lnes bundle Straght lnes are extracted n the mage. Lnes parallel to -axs and Y-axs n object space respectvely are use to calculate the vanshng ponts. Vanshng pont V ( x V, y V s the ntersecton pont o a set o straght lnes whch are parallel wth each other n object space. Accordngly, each straght lne, e.g. lne j, belongng to ths set should pass through the vanshng pont as shown n Fg.. Thereore Eqs. s evolved. ( xv x ( yv y d = ( y j y ( x j x ( sv sv Where, d s the dstance o pont j to lne V ; S v s the dstance between pont and vanshng pont v. Then ntal values o DC angular parameters are decomposed accordng to vanshng pont geometry (Caprle, B. et al. 99. Accordng to the vanshng pont geometry the relatonshp o vanshng pont and DC orentaton parameters s deduced as Eqs.. x = x y = y xy = x yy = y xz = x yz = y + cot ϕ sec ω cos κ tan ω sn κ cot ϕ secω sn κ tan ω cos κ + sec ω cscω sn κ tan ω sn κ + sec ω cscω cos κ tan ω cos κ tan ϕ sec ω cos κ tan ω sn κ + tan ϕ sec ω sn κ tan ω cos κ Thereater, vanshng pont coordnate ( x V, y V n Eqs. s replaced wth nteror and exteror orentaton (angular parameters accordng to Eqs.. A model o adjustment system o observatons and parameters employed the constrant o straght lnes bundle s then deduced to calbrate DC angular parameters. The nteror orentaton parameters are xed n the adjustment ( In ths artcle, the constrant o known orentaton o parallel lnes n object space s ntroduced to control the adjustment system. n m n m j Fg.3 the constrant o known orentaton o parallel lnes n object space There are two group lnes parallel to -axs and Y-axs n object space respectvely. The projectons o these two group lnes n rected mage P should be parallel to -axs and Y-axs n object space respectvely n deal nstance. Namely Eqs.3 and 4 are deduced. In drecton: b b x b y b x + b y b j + j 3 3 = (3 c x + c y c3 c x j + c y j c3 In Y drecton: Where: axm + a ym a3 axn + a yn a3 = (4 c x + c y c c x + c y c ( x, m m y,( x j, lne parallel to -axs ( x m lne parallel to Y-axs 3 n y j : the coordnates o two end ponts o, y m,( x n, y n : the coordnates o two end ponts o : ocal length a b c a b c a3 b3 : rotaton matrx c3 n 3 j

4 Eqs.3 and 4 can be control condton or the calbraton o DC angular parameters. Thereore t s deduced that a model o adjustment system combned wth constrants o straght lnes bundle and known orentaton o parallel lnes n object space. The rgdly geometrc constrant s mposed on the adjustment system, thereore the model o adjustment system deduced here has the characterstcs o large convergent radus and hgh stablty. other (Fg.6 are represented n the hstogram by peak areas. Moreover, the locaton o range varance n mage s ndcated by the locaton o correspondng peak area n the hstogram because the hstogram s drawn ollowng the movng drecton o vehcle. Accordng to the locaton ndcated, mages ncludng cross-road are excluded rom mage sequence and mage segment correspondng to each buldng s extracted rom mage sequence. 3. IMAGE SEQUENCE SUBSECTION To recover buldng acade texture rom mage sequence ecently, the mage sequence should be automatcally subsected nto segments correspondng to each buldng. Accordngly, the hstogram o projectve derence o correspondng ponts s used or automatc mage sequence subsecton. a Orgnal a Orgnal b Ater lterng b Ater lterng (Cross-road (Connected buldngs wth large range varance Fg.5 the hstogram o projectve derence o correspondng ponts A A P S Y a a p d A a a A, A Z b x S b y b z p' P Fg.4 The projectve geometry o correspondng ponts a. Cross-road b. Connected buldngs wth large range varance Fg.6 Segments wth large range varance As Fg.4, the orgn o the coordnate system chosen n ths paper s the projectve center S o the rst mage. The plane Y s parallel to the acade. The deal mage plane P o the rst mage (the dstance between pont S and P s ocus s consdered as publc projectve plane. As ar as correspondng pont on the acade a and a be concerned, the projectve derence between ther projecton pont A and A should be very small or even deal zero. owever, projecton pont A and A o correspondng pont a and a not on the acade have large projectve derence (Fg.4. Accordngly -D hstogram along x-axs o mage coordnate system s drawn to show the dstrbuton o projectve derence (Fg.5. Cross-road and acades havng large range varance wth each 4. TE AUTOMATIC MOSAIC OF FACADE TETURE Usually or bg buldngs, acade texture s covered by several mages nstead o only one mage. Thereore to acqure the whole acade texture, mosac process s requred. In ths paper, a strp method or mage mosac s presented. A model o least square adjustment s deduced to determne the relatvely spatal relatonshp o the mage sequence. Ths model s smlar to the one o the strp block adjustment usng n aeral trangulaton and based on the ormula o the projecton o mage pont on the projectve plane. In ths model the DC angular parameters

5 determned ater the process o rectcaton are consdered as weghted observaton. In ths paper, the acade s consdered as a plane and parallel to the publc projectve plane P (Fg.4. Thereore, the relatvely spatal relatonshp o the mage sequence wll be recovered when the correspondng rays o each two mages ntersect on the publc projectve plane va adjustng the exteror orentaton parameters (EOPs o each mage. Accordng to the relatvely spatal relatonshp o the mage sequence determned above, a method smlar ortho-mage generaton s employed to recover the whole acade texture. Frstly, the ormula, descrbng the projectve relatonshp as Fg.4, s deduced as below based on the projectve ormula o mage pont on the projectve plane. ax+ a y a3 ( + Z S + S = cx+ c y c3 ax + a y a3 ( + Z S + S cx + c y c3 b x+ b y b3 ( + Z S + YS = cx+ c y c3 bx + b y b3 ( + Z S + YS cx + c y c3 Where, : Focal length ( S, Y S, Z S : the coordnate o projectve center S a b c a b c a b c : the rotaton matrx o mage p (x, y : the coordnate o mage pont a ( =,. To calculate the coordnate derence between correspondngly projectve ponts, Eqs.5 s transorm to Eqs.6. ax+ a y a3 = ( + Z S + S + cx+ c y c3 ax + a y a3 ( + Z S S cx + c y c3 b x+ b y b3 Y = ( + Z S + YS + cx+ c y c3 bx + b y b3 ( + Z S YS cx + c y c3 Accordng to the theory o optc projecton, the correspondng rays o each mage can ntersect each other on the publc projectve plane va adjustng the EOPs o each mage. Namely, the least square adjustment system deduced on (6 (5 bass o Eqs.6 s employed to calculate the EOPs ollowng the prncple that all o coordnate derence between correspondngly projectve ponts s least square. Eqs. 7 s the ormula descrbng the dsplacement o projectng ponts n the easel plane caused by small angle. d = dbx + dbz + ( + dϕ + Y dω Ydκ y + Y Y Y dy = db + dbz + dϕ + ( + dω where: Then, the observaton equaton s: v v Y = d d ( = dy dy ( Y Y dκ Y d = dbx + dbz + ( + dϕ + dω Y d Y Y dy = dby + dbz + d ϕ Y + ( + dω + d ax + a y a3 = ( + Z S + S cx + c y c3 bx + b y b3 Y = ( + Z S + YS cx + c y c3 Fnally, the relatvely spatal relatonshp o the mage sequence s determned by the least square adjustment system deduced above and a method smlar to ortho-mage generaton s employed to generate the whole acade texture, as Fg. 7. p 4 p 5 p p p 6 p 3 Fg.7 The generaton o mosac mage 5. REFINING OF MOSAIC RESULT As presented above, the mosac result o vehcle-based magery s optmzed employed the strp method. owever, the observaton values n the least square adjustment system deduced above are the coordnates o correspondng ponts. Thereore the mosac results n those areas havng ew correspondng ponts matched are not perect, as Fg 8.a and Fg P (7 (8 κ κ

6 .a llustrated. In ths secton, two renng strategy are presented. One s renng the mosac regon where correspondng ponts are dcult to be matched but there are plenty o horzontal lnes. The constrant o correspondngly horzontal lnes s ntroduced to mplement ths process. The other s renng the mosac regon by densyng the correspondng ponts accordng to the relatvely spatal relatonshp o the mage sequence. we can see, t s much better ater mposng the constrant o correspondngly horzontal lnes. Fg.9 The determnaton o correspondng lnes 5. The constrant o correspondngly horzontal lnes A L dy A ΔY L dy L ΔY A Fg. The ncluded range o correspondng lnes 5. Densyng the correspondng ponts a. Beore renng b. Ater renng Fg. 8 The rened result wth constrant o correspondngly horzontal lnes As Fg.a llustrated, the reason or stagger along the mosac edge s lackng correspondng ponts n ths area. Thereore the best way to rene the result s to densy the correspondng ponts. As presented n secton 3, the projectve derence between correspondng ponts n the mosac mage becomes very small ater the least square adjustment. As a result, t s possble to densy the correspondng ponts n the mosac mage. As Fg. 9 llustrated, although ew correspondng ponts n ths area, there are plenty o horzontal lnes. Thereore, the queston s how to match correspondng lnes. Frstly, those lnes are projected nto the mosac mage. Included range o correspondng lne s then determned or every lne n the let mage. As Fg. llustrated, so-called ncluded range s a horzontal strp whch center s lne L and wdth s twce as dy. dy s the Y component o the projectve derence o the correspondng ponts near lne L. Snce lnes n Fg. are horzontal, the center A o lne L s employed to determne correspondng lnes. Thereore, those lnes n the rght mage are correspondng lnes ther center ponts (A and A are wthn the ncluded range o L. As we can see, the projectve derence ΔY and ΔY between lne L and L, L are equal to the Y coordnate derence between center A and A, A. Thereore, the renng strategy or the mosac result s to make ΔY be the least square by the least square adjustment. The mosac result s llustrated as Fg. 8b ater renng process. As a. Beore renng b. Ater renng Fg. The renng result by densyng correspondng ponts The regon need to be densy should be determned rstly. For ths purpose, a grd s overlad on the mage. The sze o grd s depends on the sze o mage. The mage sze n Fg. s pxels 5 pxels and a grd s overlad. Thereore, the grd cell sze s 5 pxels 75 pxels. Those grd cells overlad n the regon o sky are thrown o accordng to roo

7 boundary tracked (Blue curve n the mage because t s useless to densy correspondng ponts n the regon o sky. Those grd cells wth ew correspondng ponts ncluded should be the regons to densy correspondng ponts. Then, correspondng ponts are matched n these regons. The projectve derence, between the exstng correspondng ponts near or n these regons, can be employed to orecast the poston o correspondng ponts. More correspondng pont are matched accordng to the poston orecasted, as Fg. b llustrated. Fnally, those correspondng ponts are added to the least square adjustment system. The mosac result rened by the strategy densyng the correspondng ponts s llustrated as Fg. b. From the result we can see that the stagger exstng n Fg. a s elmnated n b. Fg.3 A segment o street acade Both o mage sequence subsecton and strp method are based on the rectcaton o raw mage, thereore, an algorthm based on vanshng pont geometry and geometrc constrans o parallel object lnes presented n Secton s used to automatc recty the raw mage. The average tme consumng or the rectcaton o a sngle mage s 3 seconds. As a 8m long buldng acade as be concerned, the number o raw mages s 36 and total tme o rectcaton becomes 8 mnutes. 6. EPERIMENTAL RESULTS Experment was mplemented on automatc mage sequence subsecton usng the method presented n Secton 3. As shown a Cross-road A b Connected buldng c Cross-road B Fg.4 The hstograms o projectve derence o correspondng ponts Experments were mplemented on sde-shootng mage sequences (Fg. taken by dgtal camera KODAK PROFESSIONAL DCS Pro SLR/n along the pedestran street. The dstance between photographc center and buldng s about m. The mage sze s 3 pxels 45 pxels. To x the nteror parameter durng the process o takng photos, the ocal length s set to nntude and the uncton o automatc ocalzaton s turned o. Actually, t s not necessary to process the magery wth such large mage sze, thereore, the raw mages were compressed and mage sze s reduced to 5 pxels pxels. n Fg.3 a 46 meters long segment o street acade conssts o two cross-roads and our connected buldngs. There are 58 mages taken to cover ths segment. The hstograms o projectve derence o correspondng ponts (Fg.4 were generated to ndcate the poston o large range varance rom those mages. In Fg.4, the postons o peak values exactly ndcate the postons n the raw mages (Fg.5 o two cross-roads and the buldng (hghlghted n a red rectangle n Fg.3 whch has large range derence wth others. Fg. Sde-shootng mage sequence a Cross-road A b Connected buldng c Cross-road B Fg.5 Large range varance n raw mages Accordng to the poston detected, two mage segments were subsected rom 58 mages and automatcally corresponded to the buldngs o nterest because mages were taken along the

8 street n sequence. Ater the subsecton o mage sequence, every buldng has a correspondng mage segment to recover acade texture. As presented n Secton 4, mage mosac s requred to generate the whole acade texture. For comparson, acade texture s mosacked rom close-ranged dgtal camera mages respectvely by the exstng mage mosac sotware and the strp method presented n Secton 4. The results o mage mosac sotware Supresot ImageSute and Cannon Photostch are shown n Fg.6. Obvously, the exstng sotware tools o mage mosac are not applcable or processng mages wth large oblquty o the posture o dgtal camera. As Fg. 7 llustrated, the mosac result was acqured usng the strp method and rened by the two strateges presented n Secton 5. Obvously t s satsyng and realzng the seamless mosac. As the 8m long buldng acade as be concerned, there are 6 unknown parameters n the adjustment system o strp method and the tme consumng s only seconds. The whole process s ully automatc and ast. consderable geometrc dstorton due to the large oblque angle. The expermental results ndcate that the proposed automatc mosackng algorthm s eectve and relable or buldng acade texture mappng. Moreover, the perormed tests have revealed several advantages. Because the acquston way o mage sequence s vehcle-based and moreover the recovery o buldng acade texture whch s always low ecency to exstng methods realzes automaton n our method, the whole process rom data acquston to the generaton o buldng acade texture s mplemented n a rapd and hgh ecent way. Terrestral close-ranged mages have hgh resoluton and moreover are taken along street, thereore a walk-through 3D scene closng to the realty can be reconstruct. The approach presented n ths paper s applcable or streets wth hgh densty o buldngs. Future work wll concentrate on the 3D model reconstructon or more complex acade and mprovement o avalablty. REFERENCES Reerences rom Journals: Caprle, B. and V. Torre, 99. Usng vanshng ponts or camera calbraton, Internatonal Journal o Computer Vson, 4(: 7 4. a. ImageSute b.photosttch Fg.6 Mosac results by exstng sotware 7. CONCLUSIONS Reerences rom Other Lterature: Antone, M.E.; Teller, S. Automatc recovery o relatve camera rotatons or urban scenes. Proc. IEEE Con. on Computer Vson and Pattern Recognton, lton ead Island,, p.8-9. Ths paper presented an automatc mosackng algorthm that can generate the entre acade texture o large buldngs rom the close-range color mages wth the Frueh, C., R. Sammon, and A. Zakhor, 4. Automated texture mappng o 3D cty models wth oblque aeral magery, Proceedngs o the nd Internatonal Symposum on 3D Data Fg.7 Whole acade texture

9 Processng, Vsualzaton and Transmsson (3DPVT September, Thessalonk, Greece, pp Früh, C. and A. Zakhor,. Fast 3D model generaton n urban envronments, Proceedngs o IEEE Conerence on Multsensor Fuson and Integraton or Intellgent Systems, pp Stamos, I.; Allen, P.E. 3-D model constructon usng range and mage data. Proceedngs IEEE Con. on Computer Vson and Pattern Recognton, lton ead Island,, p Takase, Y., N. Sho, A. Sone and K. Shmya, 3. Automatc generaton o 3D cty models and related applcatons, Internatonal Archves o the Photogrammetry, Remote Sensng and Spatal Inormaton Scences, 4-8 February, Tarasp, Swtzerland, 34(5/W Thrun, S.; Burgard, W.; Fox, D.: A real-tme algorthm or moble robot mappng wth applcatons to mult-robot and 3D mappng, Proc. o Internatonal Conerence on Robotcs and Automaton, San Francsco,, p..3-8, vol.. 4 Acknowledgements Ths research was supported by the Natonal gh Technology Research and Development Program o Chna (863 Program wth the seral number 6AAZ.