AN INTEGRATED APPROACH FOR THE ORIENTATION OF TERRESTRIAL OUTDOOR SCENES

Transcription

1 AN INTEGRATED APPROACH FOR THE ORIENTATION OF TERRESTRIAL OUTDOOR SCENES Norbert Haala, Jan Böhm, Darko Klnec Insttute for Photogrammetry (fp), Unversty of Stuttgart, Germany Commsson II, WG II/1 KEY WORDS: Navgaton, Orentaton, Augmented Realty, Generalsaton, Vsualsaton ABSTRACT: Wthn the paper a low-cost system for the provson of geoerenced terrestral mages n urban envronments s presented. Based on an mage wth approxmate exteror orentaton from a low-cost GPS and a dgtal compass and a 3D CAD model of a vsble model as provded from an 3D vrtual cty model, the exact locaton of the buldng n the mage s detected automatcally and used for a ned orentaton of the mage. The work presented n ths paper s part of a project amng on the development of a moble devce, whch enables access to locaton based servces n a complex urban envronment. The ntutve access to object related nformaton s realsed by so-called telepontng. For that purposes a spatal model of the user s envronment s mapped to the orented mage on order to allow the access to object related nformaton by pontng to the respectve mage sectons. The camera used for mage collecton s ntegrated to a moble devce, whch has to be capable for data transmsson n order to provde the requred locaton based servces. Snce the provson of locaton based servces currently s one of the most promsng markets for the applcaton of spatal data as they are collected by photogrammetrc technques, these applcatons wll also be dscussed brefly wthn the paper. 1. INTRODUCTION Applcatons amng on the provson of locaton-dependent access to spatal data are currently emergng to the consumer market. In order to provde nformaton n the framework of a locaton aware envronment, the actual poston and orentaton of the user has to be avalable. Hence, to reach the goal of an area-coverng provson of locaton aware applcatons lke personal navgaton to any user anywhere, these systems have to be based on small, yet powerful, moble devces, whch ntegrate features lke the accurate localzaton of the user as well as a locaton-dependent processng and transmsson of spatal data va wreless networks. Up to now photogrammetry has manly benefted from the dynamc development trggered by these applcaton scenaros due to the resultng demand for the collecton and update of geographc data bases. Addtonally, the emergng spread of locaton based servces also stmulated the development of new products currently not provded by standard 2D GIS envronments. One of the most obvous examples are current developments n car navgaton systems, amng on the supersedng of 2D map-lke representatons by a 3D vsualzaton of the envronment. Thereby a more realstc and ntutve presentaton of the path to be followed by the user can be acheved. Snce ths type of vsualzaton of course presumes the avalablty of a complete 3D model of the envronment, ths wll result n an ncreasng demand for area coverng 3D data collecton manly n urban envronments. In addton to a realstc vsualzaton of the 3D envronment the ntutve provson of supplementary nformaton related to the vsble objects s another key feature of locaton based servces. In order to enable a very realstc presentaton of object related nformaton as well as a ntutve realzaton of spatal queres, so-called augmented realty (AR) technques can be appled. By ths technques object related nformaton as t s represented by computer graphcs s overlad to the user s current feld of vew based on optcal see-through head-mounted dsplays. One of the most valuable applcatons of AR, whch can also be expected to become of consderable nterest for the consumer market, s ts capacty to provde stuaton awareness n buldup areas. Ths results from the fact that urban envronments are usually complcated, dynamc, and nherently threedmensonal. Wthn an urban envronment, AR can for example be appled for the presentaton of name labels or addtonal alphanumerc data appearng to be attached to a sde of a buldng. Thereby a user s supported f he navgates through an unknown buld-up area. As an example (Höllerer et al. 1999) present the names of buldngs to the user dependng on hs actual feld of vew by see-through data glasses. By pontng to the respectve buldngs, addtonal nformaton s made accessble va an ntegrated wreless access to the nternet. Ths feature s also realzed wthn the project NEXUS (Frtsch et al 2000), whch s the bass of the work presented wthn ths paper. For smplfcaton of the overall system wthn ths project the head-mounted dsplay s replaced by an mage of the user s envronment. Ths mage can for example be captured by a camera ntegrated nto a small hand-held dsplay. If the exteror orentaton of the captured mage s avalable, the augmented world data,.e. the spatal model of the user s envronment enrched by addtonal objects, can be drectly mapped to the correspondng sectons of the mage. After ths step access to object related nformaton can be realzed by pontng to respectve regons of nterest drectly on the mage dsplay (telepontng). The requred mappng between vrtual objects and the real world as t s depcted by the mage must guarantee a correct overlay of the generated computer graphcs to the correspondng objects n the perceved envronment. For ths reason, poston and orentaton of the camera or the head mounted dsplay, respectvely, has to be determned at an suffcent accuracy. Wthn the paper the automatc algnment between terrestral mages and an avalable spatal model n an urban envronment wll be descrbed. The man component of ths spatal model s provded by a 3D representaton of the vsble buldngs. The

2 algnment s based on the drect measurement of exteror orentaton by low cost components. For nement of ths coarse measurement, the vsble slhouettes of the depcted buldngs are localzed automatcally n the mage based on a Generalzed Hough Transform (GHT). After ths matchng check ponts can be generated automatcally based on the 3D coordnates of the vsble buldng prmtves and subsequently used for the mprovement of the exteror orentaton by a spatal resecton. Even though our work s currently motvated by the provson of locaton dependent servces, the approach can also be appled f mage pose has to be automatcally determned accurately n urban envronments for a data collecton lke the automatc provson of mage texture for the facades of the buldng models. 2. MODEL TO IMAGE MAPPING In order to enable spatal queres by pontng to object related regons of nterest drectly on the mage dsplay, the augmented world data s drectly mapped to the captured mage based on the exteror orentaton of the current vewpont. Ths augmented world date consst of a spatal model of the user s envronment enrched by addtonal objects and object related nformaton. 2.1 Spatal Model In an urban envronment, the man component of the requred spatal model, whch has to represent the perceved envronment, s provded by a 3D cty model. Ths 3D cty model essentally contans the 3D wre-frames of the vsble buldngs. In addton to other applcatons n the context of vsualzatons, 3D cty models are also requred for network plannng or cty clmate and envronmental research. Also trggered by the requrements of these applcatons, recent years have shown great effort n the development of tools for the accurate and effcent reconstructon of 3D cty models. For ths reason a rapdly growng number of data bases s becomng avalable. Even though the qualty of these 3D cty models s an mportant factor for the accuracy of the model to mage mappng, the descrpton of avalable approaches for the requred 3D buldng reconstructon s beyond the scope of ths paper. Fgure 1: 3D cty model of Stuttgart test area For our test area n the cty of Stuttgart two dfferent 3D cty model datasets are avalable. The frst dataset was derved automatcally at Stuttgart Unversty based on a combnaton of laser DSM and groundplan nformaton (Haala & Brenner 1999). The second data set has been collected manually by photogrammetrc stereo measurement of mages at scale 1: (Wolf 1999). Ths data has been provded by the Cty Surveyng Offce of Stuttgart. For both datasets the outlne of each buldng s defned by the publc Automated Real Estate Map (ALK), whch provdes accuraces n the centmetre level. 2.2 Requred Mappng Accuraces In general, the mappng accuracy, whch s requred for the overlay of supplementary nformaton to the real envronment, s tghtly coupled to the type of data to be ntegrated. The accuracy requrements are farly low, f annotatons lke the name of a buldng and ts known functon are overlad to the correspondng object n the real world, or f the translaton of road sgns are projected to the respectve sgnposts. The same order of accuracy s also suffcent for the presentaton of routng nformaton, for example f the path that has to be followed to reach a partcular destnaton s overlad to the vsble street. Although all ths type of nformaton s of course coupled to the user s current poston, a relatvely coarse mappng s suffcent snce the generated graphcs have to be algned only roughly to relatvely large scale features. In contrast, f the system shows hghly localzed nformaton lke the wre frame versons of buldngs, a user cannot afford to see them far apart from the real edges of the observed objects. If the graphcs are not exactly algned n ths type of applcaton, the result wll be annoyng or possbly even msleadng. Smlar problems wll occur f the presentaton of otherwse hdden features such as the locaton of power lnes, water supples or other nfrastructure and utlty nformaton s aspred. If the reconstructed vrtual envronment s drectly overlad to the observers vew, even small errors n the model n the order of tens of centmeters can lead to sgnfcant errors, undermnng the effectveness of an AR system. For ths reason all of ths data can only be accurately regstered to the correspondng observed object prmtves, f both a detaled model of the envronment and an accurate trackng system s avalable. 2.3 Image Geoerencng n Urban Envronments For arborne magery, the exteror orentaton usually s determned ndrectly based on a spatal resecton or a bundle block adjustment, respectvely. For ths purpose commercal software tools are avalable, whch enable an effcent and almost autonomous processng. In contrast to that, the requred procedures for hghly automated te and control pont measurement do not exst for terrestral mages of natural outdoor scenes. Hence, drect geoerencng,.e. the drect measurement of camera poston and orentaton at the tme of mage capture by a sutable sensor system usually s the perable soluton for the processng of the outdoor scenes to be processed by our system. A commercal system, whch ntegrates drectly geocoded mage sequences together wth supplementary nformaton nto electronc cty maps s descrbed by (Sood & Fahrenhorst 1999). Ther system manly ams on a coarse nspecton of an area of nterest based on the collected mages, whch are lnked to a dgtal 2D map. Thus, the qualty of ther collected exteror orentaton provded by GPS measurement s not suffcent for a task lke telepontng. A system for the collecton of geoerenced terrestral mages n urban areas at hgh accuraces usng ntegrated DGPS/INS measurements s presented by (Bosse et al. 2000). In ther applcaton the collected mages are used for the subsequent measurement of

3 buldng geometry. Theore the accuracy demands to be met by ther system hardware are consderable hgh and thus would also meet the specfcatons for precse object to mage mappng. Alternatvely to the applcaton of very precse and expensve sensors for drect geoerencng, the accuracy requrements of the measured camera poston and orentaton can be reduced f as n our case - a 3D model of the buldngs at the ste s already avalable. In that case the terrestral magery can be algned to the reconstructed buldngs by an automatc matchng of correspondng prmtves between object and mage space. NEXUS staton and nformaton between platform and staton wll be exchanged by wreless communcaton. 2.4 Exemplary Applcaton The platform we used for data collecton wthn our current system s depcted n Fgure 2. It conssts of a standard resoluton color vdeo camera wth extreme wde-angle lens, a GPS recever, an electronc compass and a tlt sensor. By combnng mage data and orentaton data we obtan an mage wth an approxmate exteror orentaton. The camera s a consumer style Sony DFW-500 vdeo camera connected to the system va IEEE 1394 also known as FreWre. The camera was calbrated on a test-feld usng a ten parameter camera model. The GPS recever s a Garmn LP-25, whch can be operated both n normal and dfferental mode. We used the ALF servce (Accurate Postonng by Low Frequency) of Deutsche Telekom for dfferental mode, thus obtanng a correcton sgnal every three seconds. All the devces are connected to a laptop. Whle the camera and compass/tlt sensor are hand held, the GPS s attached to a backpack. Fgure 3: Exemplary applcaton of the NEXUS prototype 2.5 Drect Geoerencng Based on Low-Cost Components Whle the theoretcal accuracy of dfferental GPS as t s used n our prototype s very hgh, there are many practcal lmtatons. Ths s especally true when usng GPS n bult-up areas. Shadowng from hgh buldngs causes poor satellte confguratons, sometmes the sgnal s lost completely. Addtonally, sgnal lectons from buldngs nearby cause so called multpath effects, whch are further reducng the accuracy of GPS measurement. Our experence shows that the system as t s depcted n Fgure 2 allows for a determnaton of the exteror orentaton of the camera to a precson of 7-10 m n planar coordnates. In our system, the vertcal component of the GPS measurement was dscarded and substtuted by heght values from a dgtal elevaton map due to the hgher accuracy of that data source. The orentaton accuracy provded by the dgtal compass and the tlt sensor resulted n an error of approxmately 1 2. Fgure 2: Prototype of the moble photogrammetry devce. An exemplary applcaton based on our current prototype s depcted n Fgure 3. Smultaneously to the capture of the mage, the poston and orentaton of the camera s determned. Afterwards the vewng frustrum s calculated for the captured mage and projected to an ortho mage or a map. Based on ths nformaton, the vsble buldng s selected from the avalable data base and correspondng object related nformaton as t s for example provded by a webste s presented by the graphcal user nterface. These webstes then gve access to servces lke tcket sales f for example a theatre s vsble. For demonstraton of the telepontng functonalty, the system s currently realzed wthn a standard GIS software package. In the fnal system the NEXUS platform wll provde both the management of the postonng components and the provson of the spatal models. A small moble devce (PDA) wll be utlzed as personal Fgure 4: Slhouette of the buldng as projected to the mage based on the exteror orentaton from GPS and dgtal compass. An example of the accuracy whch can be acheved for model to mage mappng based on the drectly measured parameters of exteror orentaton s gven n Fgure 4. For the gven orentaton and calbraton of the camera, a rendered vew of the

4 depcted buldng computed. The outlne of ths rendered vew,.e. the vsble slhouette of the buldng then has been overlad to the mage as a red polygon. As t s clearly vsble n Fgure 4, the accuracy of drect geoencng based on the appled lowcost components only allows for an approxmate model to mage mappng. Fgure 5: 3D model used for generaton of buldng slhouette. In order to addtonally show the amount of detal, whch s represented by the avalable buldng model, Fgure 5 gves a 3D vew of the buldng already depcted n Fgure 4 from a slghtly dfferent vewpont. Even though the accuracy of drect geoeerencng by our lowcost system s suffcent for some applcatons, ths coarse mappng as t s depcted n Fgure 4 s not suffcent f hghly localzed nformaton has to be presented to the user. Thus, n order to enable a precse access object related nformaton by pontng to correspondng mage regons, the model to mage mappng has to be ned. For the nement of the drectly measured exteror orentaton a method for automated appearance-based detecton of buldngs n terrestral mages s appled. The problem s stated as follows: From an mage wth a gven approxmated exteror orentaton and a threedmensonal CAD model of the buldng, detect the exact locaton of the buldng n the mage and use ths nformaton for a ned pose estmaton of the camera. 3. IMAGE BASED LOCALIZATION OF BUILDING SHAPES When the task s to detect a three-dmensonal shape n an mage, two general strateges for object representaton are avalable. One s the mappng of the nherent three dmensonal representaton of the object, whch leads to a 3D to 2D matchng problem, the other s the applcatons of a twodmensonal representaton, whch leads to a 2D to 2D matchng problem. Whle the former s the more general and theoretcally more appealng approach, there are several practcal problems, whch often prevent ts use. One of the problems s the relablty of feature extracton, the other the exponental complexty of the matchng task. For the later approach n order to have a two-dmensonal representaton of a three-dmensonal shape, one has to decompose the shape nto several vews and store a two-dmensonal representaton for each vew. Ths approach s erred to as an aspect-graph. For our system we do not have to buld the whole aspect graph. As an approxmated exteror orentaton of the magng devce s avalable, a sngle vew of the shape can be created on-the-fly for each mage n correspondence to the respectve orentaton data. Addtonally, when desgnng a object recognton system one has to choose the type of features used for recognton. The decson on the feature type s often guded by the avalable model data. In our case, the buldngs are modelled as polyhedrons, no n-plane facade detal or texture nformaton s avalable. Ths strong dscrepancy n feature detal n-between model and mage data, prevented us from usng edge or corner detecton. Snce there s no texture nformaton avalable, mage correlaton was also not an opton. To acheve a robust detecton we chose to detect the overall shape of the buldng n the mage rather than extractng sngle features. As already demonstrated n Fgure 4, the overall shape of the buldng as t s represented by ts slhouette can be calculated based on the avalable drectly measured exteror orentaton and the 3D model of the buldng. Now, ths representaton has to be detected and exactly localzed wthn the correspondng mage. For ths purpose a Generalzed Hough Transform (GHT) s appled. 3.1 Generalzed Hough Transform The well known Hough transform s a technque whch can be used to solate features of a partcular shape wthn an mage. Snce the desred features have to be specfed n some parametrc form, the classcal Hough transform s most commonly used for the detecton of regular curves such as lnes, crcles, ellpses, etc. (Hough 1962). The GHT (Ballard & Brown 1982) s a generalzaton of ths concept for the employment n applcatons, where a smple analytc descrpton of a feature s not possble. In ths case, nstead of usng a parametrc equaton of the curve, a look-up table s appled to defne the relatonshp between the boundary postons and orentatons and the Hough parameters. Y X, Y P β ω Fgure 6: Representaton of prototype shape as recorded n the framework of the Generalzed Hough Transform. Durng the offlne phase, whch has to be computed once for each 2D prototype shape, a look-up table, the so-called R-table s generated. In our applcaton ths object shape s defned by the slhouette of the buldng as t s calculated from the approxmate orentaton. Frst, an arbtrary erence pont x, y s defned wthn the feature. Usually the centrod of the shape s selected for that purpose. As t s depcted n Fgure 6 the shape of the feature can then be defned wth respect to ths pont of erence by the dstance r and angle β of normal lnes drawn from the boundary. In order to generate the X

5 R-table, frst for every pont of the prototype shape the edge orentaton ω s calculated. The R-table then conssts of dstance and drecton pars r, β, whch are ndexed by the orentaton ω of all ponts along the boundary of the gven shape. The Hough transform space s now defned n terms of the possble locatons of the shape n the mage,.e. the possble range of the centrod s poston x, y. In the onlne phase, when the actual localzaton of the shape n the collected mage s performed, gradents are computed by an arbtrary edge operator wthn the search mage. After ths step for each edge pxel at poston x, y the orentaton ω s avalable. Now for each edge pxel p( x, y ) the correspondng values for r and β can be selected from the generated R-table based on the avalable orentaton ω. These values can now be used to calculate the poston x, y of the prototype shape n the mage by x = x + rcos β y = y + r sn β Durng processng now each edge pxel votes for a certan poston x, y of the gven shape. Each vote s represented by an update of the accumulator array at ths calculated poston. The poston n the mage recevng the most votes at the end s selected as the poston of the shape n the mage. If the orentaton of the object s allowed to vary, as s the case n our applcaton, a separate R-table has to be computed for each dscrete rotaton angle. The same s true for scalng, whch has also to be enabled n our envronment. Thus the formaton of the R-tables n ths case s qute complex and computatonally expensve. search space s drastcally reduced. Addtonally, the expected accuracy of the shape s locaton can be appled for an further reducton of the search space. Fgure 8: Buldng model projected to the mage after nement of orentaton. 3.2 Implementaton and Results Fgure 7: Detected slhouette of the buldng For our mplementaton the HALCON mage processng envronment was used, whch provdes a shape detecton mechansm based on the GHT as t s dscussed above (Ulrch et al 2001). In order to compensate for the computatonal costs of large R-tables, ths operator ncludes several modfcatons to the orgnal GHT. For example t uses a herarchcal strategy generatng mage pyramds to reduce the sze of the tables. By transferrng approxmaton values to the next pyramd level the Fgure 9: Buldng localzaton based on Generalzed Hough Transform. Top mage shows the 3D model projected by measured orentaton, bottom depcts the detected slhouette. Fgure 7 shows the slhouette of a buldng as automatcally detected by the GHT wthn the captured mage. Based on the

6 estmated parameters for shft, rotaton and scale wthn ths process, the approxmate mage coordnates of the vsble object model already shown n Fgure 4 can now be mproved. After the nement of the shape s poston n the mage, correspondng ponts n object and mage space are avalable. These control ponts can now be appled n order to mprove the orgnal exteror orentaton by a spatal resecton. In prncple, the complete process, extracton of buldng slhouette, mprovement of mage coordnates by GHT and spatal resecton then has to be teratvely repeated n order to avod errors resultng from the smplfcaton of the orgnal 3D to 2D matchng to a 2D to 2D problem. Nevertheless, for our applcaton the dfferences between the projected wre-frame and the mage were manly caused by errors of the avalable 3D buldng model. Thus ths teraton was not appled. In addton to measurement errors durng the collecton of the 3D cty model, these errors manly result from generalsaton effects. Fgure 8 shows the orgnal 3D buldng model projected to the mage after orentaton nement. A second example of the algorthm for a dfferent buldng s gven n Fgure 9. In ths example the orgnal model s overlad to the collected mage based on the drectly measured exteror orentaton (top), the result of the GHT s represented by the overlay of the buldng s slhouette (bottom). Telegeoprocessng, pp. pp Haala, N. and Brenner, C. [1999] Vrtual Cty Models from Laser Altmeter and 2D Map Data. Photogrammetrc Engneerng and Remote Sensng 65[7], Höllerer, T., Fener, S., Terauch, T., Rashd, G. & Hallaway, D.: [1999], Computers & Graphcs 23, pp Hough, P. V. C. [1962] Method and means for recognzng complex patterns. Sood, R-A. & Fahrenhorst, C. [1999], Geocoded Image Sequences n Photogrammetrc Week 99, pp Ulrch, M., Steger, C., Baumgartner, A. & Ebner, H. [2001], Real-Tme Object Recognton n Dgtal Images for Industral Applcatons n 5th Conference on Optcal 3D Measurement Technques, Venna, pp Wolf, M. [1999], Photogrammetrc Data Capture and Calculaton for 3D Cty Models n Photogrammetrc Week '99, pp CONCLUSION Wthn the paper the use of orented mages as an nterface to provde locaton based servces has been dscussed. Snce the spatal model of the envronment s mapped to the correspondng mage, an ntutve access to addtonal object related nformaton s feasble for a user by pontng to the respectve mage sectons. Even though mappng accuraces of a few meters can be suffcent for some applcaton, the drectly measured poston and orentaton of the captured magery by standard low-cost hardware has to be mproved f hghly localzed nformaton has to be presented. For that purpose the successful mplementaton of a fully automated process for the detecton of buldngs n terrestral mages has been demonstrated. Based on the GHT the shape of the depcted buldng can be detected no matter whether t s shfted, rotated or optonally even scaled n relaton to the mage. Snce the orentaton s only known approxmately, these degrees of freedom are requred for our applcaton. Addtonally the GHT allows for a certan tolerance n shape devaton. Ths s also necessary, snce the CAD model of the buldng provdes only a coarse generalzaton of ts actual shape as t s appearng n the mage. In addton to the realsaton of spatal queres by telepontng as t has been dscussed wthn ths paper, ths accurately orented terrestral mage can also be appled for mage based data collecton n an urban envronment lke a geometrc nement of the collected 3D buldng model or an automatc texture mappng for the facades of the depcted buldngs. 5. REFERENCES Ballard, D.H. and Brown, C.M.: [1982], Computer Vson, Prentce-Hall Inc., Englewood Clffs, NJ. Bosse, M., De_Couto, D., and Teller, S. [2000] Eyes of Argus: Geoerenced Imagery n Urban Envronments. GPS World Aprl, Frtsch, D., Klnec, D. & Volz, S. [2000], NEXUS - Postonng and Data Management Concepts for Locaton Aware Applcatons n Proc. of the 2nd Internatonal Symposum on