Image-based Localization in Urban Environments

Transcription

1 Image-based Localzaton n Urban Envronments by Phlp Davd ARL-MR-0738 March 2010 Approved for publc release; dstrbuton unlmted.

2 NOTICES Dsclamers The fndngs n ths report are not to be construed as an offcal Department of the Army poston unless so desgnated by other authorzed documents. Ctaton of manufacturer s or trade names does not consttute an offcal endorsement or approval of the use thereof. Destroy ths report when t s no longer needed. Do not return t to the orgnator.

3 Army Research Laboratory Adelph, MD ARL-MR-0738 March 2010 Image-based Localzaton n Urban Envronments Phlp Davd Computatonal and Informaton Scences Drectorate, ARL Approved for publc release; dstrbuton unlmted.

4 REPORT DOCUMENTATION PAGE Form Approved OMB No Publc reportng burden for ths collecton of nformaton s estmated to average 1 hour per response, ncludng the tme for revewng nstructons, searchng exstng data sources, gatherng and mantanng the data needed, and completng and revewng the collecton nformaton. Send comments regardng ths burden estmate or any other aspect of ths collecton of nformaton, ncludng suggestons for reducng the burden, to Department of Defense, Washngton Headquarters Servces, Drectorate for Informaton Operatons and Reports ( ), 1215 Jefferson Davs Hghway, Sute 1204, Arlngton, VA Respondents should be aware that notwthstandng any other provson of law, no person shall be subject to any penalty for falng to comply wth a collecton of nformaton f t does not dsplay a currently vald OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY) March REPORT TYPE DRI 4. TITLE AND SUBTITLE Image-based Localzaton n Urban Envronments 3. DATES COVERED (From - To) 5a. CONTRACT NUMBER 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) Phlp Davd 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) U.S. Army Research Laboratory ATTN: RDRL-CII-A 2800 Powder Mll Road Adelph, MD PERFORMING ORGANIZATION REPORT NUMBER ARL-MR SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S) 11. SPONSOR/MONITOR'S REPORT NUMBER(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for publc release; dstrbuton unlmted. 13. SUPPLEMENTARY NOTES 14. ABSTRACT Ths report descrbes an effcent algorthm to accurately determne the poston and orentaton of a camera n an outdoor urban envronment usng camera magery acqured from a sngle locaton on the ground. The requrement to operate usng magery from a sngle locaton allows a system usng our algorthms to generate nstant poston estmates and ensures that the approach may be appled to both moble and mmoble ground sensors. Localzaton s accomplshed by regsterng vsble ground mages to urban terran models that are easly generated offlne from aeral magery. Provded there are a suffcent number of buldngs n vew of the sensor, our approach provdes accurate poston and orentaton estmates, wth poston estmates that are more accurate than those typcally produced by a global postonng system (GPS). 15. SUBJECT TERMS localzaton, aeral to ground mage regstraton, omndrectonal camera, vanshng ponts 16. SECURITY CLASSIFICATION OF: a. REPORT Unclassfed b. ABSTRACT Unclassfed c. THIS PAGE Unclassfed 17. LIMITATION OF ABSTRACT UU 18. NUMBER OF PAGES 28 19a. NAME OF RESPONSIBLE PERSON Phlp Davd 19b. TELEPHONE NUMBER (Include area code) (301) Standard Form 298 (Rev. 8/98) Prescrbed by ANSI Std. Z39.18

5 Contents Lst of Fgures Acknowledgments v v 1. Objectve 1 2. Approach Sensors for Ar-to-Ground Regstraton Image Vanshng Ponts Estmatng the 3-space Orentaton of Image Lnes Poston Estmaton Results Conclusons References Transtons 17 Lst of Symbols, Abbrevatons, and Acronyms 18 Dstrbuton Lst 19

6 Lst of Fgures Fgure 1. Vsual Learnng Systems, Inc., LIDAR Analyst system: (a) a LIDAR dgtal elevaton model (DEM) (.e., raw LIDAR data), (b) buldng footprnts extracted from the LIDAR DEM, (c) 3-D buldng models generated from DEM, (d) a close-up of a LIDAR DEM of a complex buldng, and (e) a 3-D model generated from DEM of the complex buldng....3 Fgure 2. Aeral mage of ARL (left) and the buldng footprnt model manually generated from that mage (rght)...4 Fgure 3. Pont Grey Research Ladybug2 sphercal vson camera system....5 Fgure 4. Images from the Ladybug camera taken nsde the ARL courtyard. The top mage s from the vertcal camera and the lower fve mages are from cameras n the horzontal rng. The pncushon dstorton effect apparent n these mages s a result of warpng of the orgnal mages (not shown) to correct the optcal dstortons that were present n the orgnal mages....5 Fgure 5. The mage of parallel 3-space world lnes ntersectng at a common vanshng pont n the mage. Vanshng ponts (shown as colored dots n ths fgure) may le nsde or outsde the mage. The blue lnes ntersect at a pont hgh above the mage....6 Fgure 6. Image processng for straght lne segment detecton: (a) orgnal mage, (b) Canny edges, (c) edge contours, and (d) straght lne segments ft to contours....8 Fgure 7. Results of usng the vanshng pont algorthm to classfy the 3-space orentaton of the lne segments detected n the mage of fgure 6a. All lnes of the same color have the same 3-space orentaton Fgure 8. Lne segments are detected n the ARL courtyard mages from fgure 4 and are color coded (ndependently n each mage) based on the estmated orentaton of the assocated 3-space lne Fgure 9. Computaton of the LFO vector for the locaton n the buldng footprnt terran model marked wth the blue dot. Thrty-sx footprnt orentatons are assgned from 36 equally spaced vewng drectons (left). For each of these vewng drectons, the ground plane orentaton s computed as the average over all 3-space lne orentatons that are ntersected by a ray wthn 5 of the gven vewng drecton. One such vewng drecton s shown n the rght mage Fgure 10. True and estmated camera poston for ARL courtyard experment 1. The localzaton error s approxmately 0.5 m Fgure 11. True and estmated camera poston for ARL courtyard experment 2. The localzaton error s approxmately 0.5 m v

7 Acknowledgments I would lke to thank Mr. Nck Fung of the U.S. Army Research Laboratory (RDRL-CII-A) for provdng the panoramc magery and assstng wth experments. v

8 INTENTIONALLY LEFT BLANK. v

9 1. Objectve Imagng sensors are currently beng deployed n large numbers on vehcle systems and wll lkely be deployed n the future on ndvdual Solders as well. These sensors are ntended to serve many purposes, ncludng target detecton and trackng, detecton and locaton of hostle fre, navgatonal ad, and terran model acquston. Informaton regardng events observed on the battlefeld s most useful when these events can be accurately localzed wth respect to some larger coordnate system. Localzaton allows multple, non-collocated systems to exchange and fuse nformaton, and coordnate ther actons. The frst step to localzng observed events s to locate the poston and orentaton * of the observng sensor. There are many ways to determne the poston of a sensor n an envronment. These nclude usng a global postonng system (GPS), compass, TV or cell phone networks, and landmark recognton from optcal or range data. None of these methods wll solve the localzaton problem all of the tme: GPS works best n unobscured outdoor envronments, but does not provde orentaton and s not accurate enough for some applcatons (such as autonomous navgaton of unmanned ground vehcles). Cell phone networks can provde ndoor and outdoor localzaton, but are accurate to only 100 m. Landmark recognton can gve accurate poston and orentaton, but may be unrelable and computatonally ntense, and requre laborous offlne terran modelng. Some combnaton of these technques wll be needed to robustly solve the battlefeld sensor localzaton problem. In the short term, a system s envsoned that performs localzaton usng a combnaton of GPS and landmark recognton. GPS, when avalable, wll provde the rough postonng and the landmark recognton subsystem wll refne that poston and fll n the gaps durng GPS outages. The research descrbed here s focused on localzaton usng vsual landmark recognton n urban envronments. The goal of our research s to develop and demonstrate effcent and accurate algorthms to determne the poston and orentaton of a camera n an outdoor urban envronment usng camera magery acqured from a sngle locaton on the ground. The requrement to operate usng magery from a sngle locaton allows a system usng our algorthms to generate nstant poston estmates and ensures that the approach may be appled to both moble and mmoble ground sensors. Localzaton s accomplshed by regsterng vsble ground mages to urban terran models that are easly generated offlne from aeral magery. Provded there are a suffcent number of buldngs n vew of the sensor, our approach provdes accurate poston and orentaton estmates, wth poston estmates that are more accurate than those typcally produced by GPS. * Hereafter, for concseness, the terms poston and locaton wll often be used synonymously wth the phrase poston and orentaton. 1

10 2. Approach The poston of a camera can be determned from objects observed n the scene by recognzng landmarks (.e., mmoble objects lke buldngs, statues, natural features, etc.) and retrevng ther prerecorded postons from an exstng database or terran model. Approaches to landmark recognton may be broadly classfed as ether appearance-based or model-based. Appearancebased approaches represent objects as collectons of two-dmensonal (2-D) mages. Modelbased approaches represent objects n terms of some hgher level structures (e.g., 2-D or threedmensonal [3-D] geometrc models). In both approaches, a search s performed to match a new mage to the model. Although research n vsual landmark recognton has been ongong for over 30 years, all exstng technques seem to suffer from at least one of the followng problems (2, 3): (1) The sensor must have prevously observed the scene from smlar vantage ponts for t to be recognzed; (2) excessve computatons are needed to match mage data to a terran model, thus lmtng real-tme operaton; (3) accurate localzaton nformaton cannot be obtaned from recognzed landmarks; and (4) background clutter causes recognton and localzaton errors. Our approach to landmark recognton addresses all of these ssues. We regster ground-based magery to a terran model that s easly created from a sngle aeral mage. Our terran model conssts of a 2-D map of buldng footprnts. Thus, the sensor does not have to vst an area of the terran prevously to localze tself n that terran. The locaton of the camera n the urban terran s determned by estmatng, from a sngle mage, the footprnts of vsble buldng facades and then regsterng ths local footprnt to the terran model. Both the local footprnt estmaton and the regstraton steps are fast. Local buldng footprnt estmaton s performed usng mage vanshng ponts to compute the 3-space orentatons on the ground plane of lne segments detected n an omndrectonal camera mage. In other words, nformaton derved from vanshng ponts s used to dentfy mage lne segments that correspond to vertcal buldng facades and then used agan to project these lne segments onto the ground plane. Gven the ground plane projectons, a vector descrbng the footprnt orentatons at equal angles over a 360 feld of regard s computed. The local footprnt orentaton (LFO) vector s then matched to the 2-D terran model to determne the camera s poston and orentaton. Each of these steps s descrbed n more detal n sectons 2.1 through Sensors for Ar-to-Ground Regstraton As mentoned prevously, our terran model conssts of a 2-D map of buldng footprnts. The footprnt of a buldng conssts of the projecton onto a horzontal plane of all large vertcal facades of that buldng. The buldng footprnts for an urban envronment can be created n a number of ways. Numerous approaches for automated buldng footprnt detecton exst usng hgh-resoluton aeral monoscopc vsble (0, 0), stereoscopc vsble (0 0), and Lght Detecton and Rangng (LIDAR) (0 0) data. Approaches usng LIDAR are currently more robust than 2

11 those usng only optcal magery. In fact, there s a commercal product named LIDAR Analyst (0) that automatcally extracts buldng footprnts and 3-D computer-aded desgn (CAD) models from hgh-resoluton LIDAR data. Fgure 1 shows some of the outputs of ths product found on the company s Web ste. As shown n ths fgure, t s possble to automatcally generate accurate buldng footprnt models usng exstng systems. However, because the focus of ths research s on sensor localzaton and not terran modelng, we dd not use any of these exstng methods to generate our models, but nstead generated our buldng footprnt models by hand from sngle aeral mages. Ths was a cheap and fast way for us to test our method. A graphcal user nterface was wrtten that allows the user to easly and quckly draw lnes on top of an mage and save these lnes as a terran model. Fgure 2 shows an example of a publcally avalable (from the U.S. Geologcal Survey) aeral mage of part of the U.S. Army Research Laboratory (ARL) and the buldng footprnt model that we manually generated from that mage. (a) (b) (c) (d) (e) Fgure 1. Vsual Learnng Systems, Inc., LIDAR Analyst system: (a) a LIDAR dgtal elevaton model (DEM) (.e., raw LIDAR data), (b) buldng footprnts extracted from the LIDAR DEM, (c) 3-D buldng models generated from DEM, (d) a close-up of a LIDAR DEM of a complex buldng, and (e) a 3-D model generated from DEM of the complex buldng. 3

12 Fgure 2. Aeral mage of ARL (left) and the buldng footprnt model manually generated from that mage (rght). The system to be localzed must possess a sensor that wll observe the surroundng envronment. Because LFO vectors descrbe buldng footprnts over the full 360 range of vewng, any sensor for ths purpose that does not have a 360 feld of vew would need to be panned to cover the full range. A number of dfferent sensors may be used, ncludng a monocular vsble camera, a stereo vsble camera par, or a LIDAR camera. As stereo and LIDAR cameras generate range drectly, they should allow for faster and more accurate calculaton of the LFO vectors. However, these two sensors are usually only accurate at short ranges. The monocular vsble camera has the ablty to observe over much longer ranges, but requres sgnfcantly more processng to generate LFO vectors. In our research, we use the Pont Grey Research, Inc., Ladybug 2 sphercal camera shown n fgure 3. Ths dgtal vdeo camera system conssts of sx 0.8 megapxel color charge-coupled devce (CCD) mage sensors, each wth a 2.5-mm focal length lens, ntegrated nto a sngle enclosure. Fve of the cameras are postoned n a horzontal rng and one s postoned vertcally. Ths enables the camera to collect magery from more than 75% of the full vewng sphere. An ntegrated 12-bt analog-to-dgtal converter along wth an IEEE-1394b (FreWre) nterface s used to stream full 12 megapxel mages at 15 FPS to the host system. Ths camera system s small (110 mm x 100 mm x 141 mm, L W H) and lght enough (1190 g) to be mounted on a portable robot. Fgure 4 shows a set of sx mages that were smultaneously acqured from the Ladybug camera as t sat on the ground n an outdoor courtyard at ARL. 4

13 Fgure 3. Pont Grey Research Ladybug2 sphercal vson camera system. Fgure 4. Images from the Ladybug camera taken nsde the ARL courtyard. The top mage s from the vertcal camera and the lower fve mages are from cameras n the horzontal rng. The pncushon dstorton effect apparent n these mages s a result of warpng of the orgnal mages (not shown) to correct the optcal dstortons that were present n the orgnal mages. 2.2 Image Vanshng Ponts Optcal mage formaton s usually modeled usng perspectve projecton. In an dealzed perspectve camera, a pont n 3-space s mapped onto an mage at the locaton where a ray connectng the center of projecton (the lens center) and the 3-space pont ntersects the mage. Optcal aberratons such as focus and lens dstorton cause devatons from ths model, but these effects can be accounted for va standard camera calbraton technques. The planar perspectve mage x of a 3-space pont X can be modeled as 5

14 x = K[ I 0]X, (1) where x and X are the homogeneous representatons of the mage and 3-space ponts, respectvely; I s the 3x3 dentty matrx; 0 s the length three column zero vector; and K s the camera calbraton matrx. α x K = 0 s α 0 y 0 0 x0 y. (2) 1 Equaton 1 makes the assumpton that the 3-space coordnates of X are gven n the camera frame of reference. The parameters of the camera calbraton matrx are α and α, the focal lengths n the horzontal and vertcal drectons, respectvely; s, the axs skew; and x 0 and y 0, the poston of the optcal axs on the mage plane (1). Under perspectve projecton, an nfntely long lne n the world can have a fnte extent n the mage; the mage of the pont at nfnty on ths lne s called the lne s vanshng pont. Parallel lnes n the world that are not parallel to the mage plane wll be maged as convergng lnes that ntersect at a sngle fnte vanshng pont. When mage ponts and lnes are represented usng homogeneous coordnates, the mage of any set of parallel world lnes, whether or not they are parallel to the mage plane, wll ntersect at a common vanshng pont. Ths pont may be a pont at nfnty, but these ponts are treated dentcally to fnte vanshng ponts. Fgure 5 shows the mage of an urban envronment wth some parallel world lnes and ther vanshng ponts dentfed. x y Fgure 5. The mage of parallel 3-space world lnes ntersectng at a common vanshng pont n the mage. Vanshng ponts (shown as colored dots n ths fgure) may le nsde or outsde the mage. The blue lnes ntersect at a pont hgh above the mage. Homogeneous coordnates are used throughout ths report to represent mage and 3-space ponts. 6

15 The 3-space drecton of a lne relatve to the camera reference frame may be determned from the lne s vanshng pont as follows. The lne through the 3-space pont A wth drecton T T 3 D = (d, 0) ( d R, d 0 ) may be represented as the set of ponts wth homogeneous coordnates X( λ ) = A + λd. From equaton 1, the mage of pont X ( λ) s x T T ( λ ) K[ I 0] A + λk[ I 0] ( d, 0) = a + λkd =, (3) where a s the mage of the pont A. The vanshng pont v of the lne s then ( λ) = lm ( a + λkd) Kd. (4) v = lm x = λ Thus, gven the vanshng v pont of a 3-space lne and the camera calbraton matrx K, the 3-space drecton of the lne s λ 3 and the projecton onto a ground plane wth unt normal n R s d = -1 d = K v (5) K ( v ( v n) n). (6) Estmatng the 3-space Orentaton of Image Lnes The frst step n our approach to estmatng the 3-space orentatons of mage lne segments s to detect the vanshng ponts n the mage. Vanshng ponts are detected as follows. The Canny edge detector (14) wth hysteress thresholdng s frst appled to generate a bnary mage of the edge ponts (fgure 6b). Straght lne segments are then extracted from ths edge mage by lnkng edges nto contours (fgure 6c) and then splttng the contours nto straght segments (15). The fnal lne segments are those whose sum of squared dstances to the contour ponts are mnmzed and whose length s at least 5 pxels long (fgure 6d). 7

16 (a) (b) (c) (d) Fgure 6. Image processng for straght lne segment detecton: (a) orgnal mage, (b) Canny edges, (c) edge contours, and (d) straght lne segments ft to contours. 8

17 L s dentfed by ts two endponts: {( ) ( )} Each lne segment L = x, y, x, y. For effcency n computng the mage vanshng ponts, for each lne segment L we compute the normalzed 1 2 homogeneous representaton of the concdent nfnte lne, l ; the endponts, e and e ; and the mdpont, m. These are calculated accordng to e e 1 2 m = = 1 1 ( x, y, 1) = T 2 2 ( x, y, 1), T, ( x + x ) 2, ( y + y ) 2, 1) T, (7) l 1 2 = e e, l = l l 2 2 ( 1) + l ( 2). The Random Sample Consensus (RANSAC) algorthm (16) s then appled several tmes to the above data; each tral s used to locate the sngle vanshng pont wth the most support. Before each new tral, the data supportng the vanshng pont found n the prevous tral are removed. Ths process s repeated untl V max = 4 vanshng ponts are found, or untl the sze of the largest consensus set s less than S mn = 20. On each tral of RANSAC, T=50, random samples of lne pars are examned. The lne par l and l j seeds a potental vanshng pont v j when the lne segments L and L j are each at least H seed = 15 pxels long and when ther angle s no longer than Θ seed = 40. The ntal vanshng pont of the lne par s v j = l l j. The normalzed lne through v j and the mdpont of lne segment L k s gven by l ( 1) 2 ( 2) 2 jk = l jk l jk + l jk, where l jk = vj m k. Then, lne segment L k s consdered to support v j and s added to the consensus 1 1 set C j when the perpendcular dstance, djk = l jk ek, from one endpont e k of L k to l jk s no larger than D sup = 3 pxels and when the angle between these lnes s no larger than Θ sup = 3. All lne segments n the largest consensus set are used to estmate the fnal locaton of the vanshng * * pont, v. v s requred to mnmze the weghted sum, for all lnes L t n the consensus set, of * * the squared dstance of lne segment end ponts to the lne through v and m t. v s found usng standard nonlnear optmzaton routnes. Fgure 7 shows the results of usng ths algorthm to classfy the 3-space orentaton of the lne segments detected n the mage of fgure 6a. Fgure 8 shows the same results for the sx Ladybug camera mages shown n fgure 4 9

18 Fgure 7. Results of usng the vanshng pont algorthm to classfy the 3-space orentaton of the lne segments detected n the mage of fgure 6a. All lnes of the same color have the same 3-space orentaton. Fgure 8. Lne segments are detected n the ARL courtyard mages from fgure 4 and are color coded (ndependently n each mage) based on the estmated orentaton of the assocated 3-space lne. 10

19 In a typcal urban envronment, buldng facades are planar and orthogonal to the ground plane. Furthermore, markngs on most buldng facades consst of two sets of orthogonal 3-space lnes, one set that s orthogonal to the ground plane and one set that s parallel to the ground plane. The vanshng pont of the set that s orthogonal to the ground plane determnes the vertcal orentaton of the camera relatve to the ground plane. The vanshng ponts of 3-space lnes that are parallel to the ground plane determne the orentatons of the respectve façades when projected down onto the ground plane. An omndrectonal camera (the Ladybug) s used to ensure that the vertcal vanshng pont wll be detected. Ths s essental n order to determne the orentaton of the ground plane and, from ths, the orentaton of buldng facades. The vertcal vanshng pont s dentfed as the vanshng pont whose poston s nearest to the center of the overhead mage from the set of sx Ladybug mages. Ths vanshng pont defnes the ground plane normal, n. Gven n, the orentaton of the projecton onto the ground plane of any classfed lne segment s computed accordng to equaton Poston Estmaton For a calbrated camera (.e., a camera where the camera calbraton matrx n equaton 2 s known), every pxel n the mage corresponds to a specfc horzontal and vertcal vewng angle. All mage lne segments whose ground plane orentaton was determned, as descrbed n the prevous subsecton, are used to estmate the local buldng footprnt. For each horzontal vewng angle, the orentaton of the buldng footprnt n that drecton s assgned the domnant orentaton of all classfed lne segments n that drecton. The LFO vector conssts of the domnant orentaton for 36 drectons equally spaced over the 360 vewng plane. For each θ { 0, 10, 20,, 380 }, the average s computed over all vewng drectons n the range [ θ 5, θ + 5 ]. Gven the poston of a camera wth respect to the buldng footprnt terran model, the LFO vector s computed smlarly to the process descrbed n the prevous paragraph, except the terran model s used to compute the ground plane orentatons nstead of the mage lne segments. If the ray from the gven poston and n the specfc vewng drecton ntersects a buldng footprnt, then ths angle (whch s n the range 0 to 180 ) defnes the buldng footprnt orentaton n that vewng drecton. If the ray does not ntersect and buldng footprnt, then a value of 0 s assgned. Ths process s llustrated n fgure 9 for a regon of the buldng footprnt map llustrated n fgure 2. 11

20 Fgure 9. Computaton of the LFO vector for the locaton n the buldng footprnt terran model marked wth the blue dot. Thrty-sx footprnt orentatons are assgned from 36 equally spaced vewng drectons (left). For each of these vewng drectons, the ground plane orentaton s computed as the average over all 3-space lne orentatons that are ntersected by a ray wthn 5 of the gven vewng drecton. One such vewng drecton s shown n the rght mage. To determne the poston and orentaton of a camera from ts omndrectonal mages, we frst process the mages as descrbed prevously to generate the camera s LFO vector. Ths vector s then matched to those n the buldng footprnt terran model usng a gradent descent or some other smlar optmzaton scheme. The estmated locaton of the camera may be used, f avalable, to ntalze ths search. 3. Results Because software has not been completed to ntegrate all components of our algorthm, we evaluated the approach usng a smulaton. We assumed that, at any locaton n the terran model, our mage processng algorthms were able to correctly estmate, to a small error, 90% of the 36 elements of the LFO vector. That s, 10% of the 36 elements of any LFO vector were assgned random values rangng from 0 to 180. Furthermore, we assumed that the correctly estmated values had errors that were normally dstrbuted wth a mean of 0 and a standard devaton of 5. Fgures 10 and 11 show the true and estmate locaton of a camera for two dfferent trals. In all experments, the localzaton error was less than 0.5 m. The color at any pont n these fgures s proportonal to the dfference between the estmated LFO vector and the LFO vector at that pont. Ths gves an ndcaton of shape of the error surface and shows how close the ntal guess must be to the true poston n order for the algorthm to fnd an answer that s close to the true poston of the camera. It can be seen that the error surfaces are smooth wth farly large basns of attracton. Thus, the local optmzaton algorthm usually found very good solutons. Note also, that the global mnmum (over the entre regon of the terran model) s always very close to the true poston of the camera (ths was the case for all experments that 12

21 we ran wth the above-gven parameters). Because the search space was at most 3-D (two dmensons for x and y poston, and one dmenson for orentaton, f t s not known), t was easy and fast to perform a global search over a large regon of the terran model. Ths enables an even larger basn of attracton to the near-optmal soluton. Fgure 10. True and estmated camera poston for ARL courtyard experment 1. The localzaton error s approxmately 0.5 m. 13

22 Fgure 11. True and estmated camera poston for ARL courtyard experment 2. The localzaton error s approxmately 0.5 m. 4. Conclusons We have descrbed an effcent algorthm to determne the poston and orentaton of a camera n an outdoor urban envronment usng camera magery acqured from a sngle locaton on the ground. The locaton of the camera n the urban terran s determned by estmatng, from a sngle mage, the footprnt of vsble buldng facades and then regsterng ths local footprnt to the terran model. Both the local footprnt estmaton and the regstraton steps are fast. Local buldng footprnt estmaton s performed usng mage vanshng ponts to compute the 3-space orentatons on the ground plane of lne segments detected n an omndrectonal camera mage. The local footprnt orentaton vector s then regstered to the 2-D terran model to determne the camera s poston and orentaton. Based on ntal experments, we beleve our approach s an order of magntude more accurate than commercal GPS and t can be mplemented to run n real tme usng modest processor resources. These qualtes make the approach sutable for many applcatons of small platforms operatng n GPS-dened urban envronments such as navgaton, mappng, and survellance. Remanng work ncludes completng the real-tme software mplementaton and evaluatng the approach n real-world feld exercses. 14

23 5. References 1. Hartley, R. I.; Zsserman, A. Multple Vew Geometry n Computer Vson; 2 nd ed., Cambrdge Unversty Press, Campbell, R. J.; Flynn, P. J. A Survey of Free-Form Object Representaton and Recognton Technques. Computer Vson and Image Understandng February 2001, 81 (2), Chen, T.; Wu, K.; Yap, K.-H.; L, Z.; Tsa, F. S. A Survey on Moble Landmark Recognton for Informaton Retreval. Proceedngs of the Int. Conf. on Moble Data Management: Systems, Servces and Mddleware, May Woo, D.-M.; Nguyen, Q.-D.; Tran, Q.-D.N; Park, D.-C.; Jung, Y. K. Buldng Detecton and Reconstructon from Aeral Images. Proceedngs of the Int. Soc. for Photogrammetry and Remote Sensng, Bejng, Chna, July Cord, M.; Declercq, D. Three-dmensonal Buldng Detecton and Modelng Usng a Statstcal Approach. IEEE Transactons on Image Processng May 2001, 10 (5), San, D. K.; Turker, M. Automatc Buldng Detecton and Delneaton from Hgh Resoluton Space Images Usng Model Based Approach. Proceedngs of the ISPRS Workshop on Topographc Mappng from Space (wth Specal Emphass on Small Satelltes), Ankara, Turkey, February Verma, V.; Kumar, R.; Hsu, S. 3D Buldng Detecton and Modelng from Aeral LIDAR Data. Proceedngs of the IEEE Computer Socety Conference on Computer Vson and Pattern Recognton, vol. 2, Washngton, DC, June, Rottenstener, F.; Trnder, J.; Clode, S.; Kubk, K. Buldng Detecton Usng LIDAR Data and Mult-spectral Images. Proceedngs of the 7th Conf. on Dgtal Image Computng: Technques and Applcatons, Sydney, Australa, December Hathcoat, T. L.; Song, W.; Hpple, J. Buldng Footprnt Extracton and 3-D Reconstructon from LIDAR Data. Proceedngs of the IEEE/ISPRS jont Workshop on Remote Sensng and Data Fuson over Urban Areas, Rome, Italy, November Wang, O.; Lodha, S.; Helmbold, D. P. A Bayesan Approach to Buldng Footprnt Extracton from Aeral LIDAR Data. Proceedngs of the IEEE Thrd Internatonal Symposum on 3D Data Processng, Vsualzaton and Transmsson, June 2006, pp

24 11. Müller, S.; Zaum, D. Robust Buldng Detecton n Aeral Images. Proceedngs of the Internatonal Socety for Photogrammetry and Remote Sensng Workshop CMRT: Object Extracton for 3D Cty Models, Road Databases and Traffc Montorng - Concepts, Algorthms and Evaluaton, Venna, Austra, August Nevata, R.; Ln, C.; Huertas, A. A System for Buldng Detecton from Aeral Images. Proceedngs of the Conference on Automatc Extracton of Man-Made Objects from Aeral and Space Images (II), Basel, Swtzerland, pp , Vsual Learnng Systems, Inc. The LIDAR Analyst Extenson for ArcGIS Automated Feature Extracton Software for Arborne LIDAR Datasets, September 2005, (accessed 2009). 14. Canny, J. A Computatonal Approach to Edge Detecton. IEEE Trans. on Pattern Analyss and Machne Intellgence November 1986, Koves, P. D. MATLAB and Octave Functons for Computer Vson and Image Processng. School of Computer Scence & Software Engneerng, The Unversty of Western Australa, Fschler, M. A.; Bolles, R. C. Random Sample Consensus: A Paradgm for Model Fttng wth Applcatons to Image Analyss and Automated Cartography. Comm. Assocaton for Computng Machnery June 1981, 24,

25 6. Transtons We plan to transton ths software to the Communcatons-Electroncs Research Development and Engneerng Center (CERDEC)/Nght Vson and Electronc Sensors Drectorate (NVESD) n support of the Sensor Moblty and Percepton Technology Program Annex (TPA) (No. CE- CI ) and to the Safe Operatons of Unmanned systems for Reconnassance n Complex Envronments (SOURCE) Army Technology Objectve (ATO). 17

26 Lst of Symbols, Abbrevatons, and Acronyms 2-D two-dmensonal 3-D three-dmensonal ARL ATO CAD CCD CERDEC DEM GPS LFO LIDAR NVESD RANSAC SOURCE TPA U.S. Army Research Laboratory Army Technology Objectve computer-aded desgn charge-coupled devce Communcatons-Electroncs Research Development and Engneerng Center dgtal elevaton model global postonng system local footprnt orentaton Lght Detecton and Rangng Nght Vson and Electronc Sensors Drector/Drectorate Random Sample Consensus Safe Operatons of Unmanned systems for Reconnassance n Complex Envronments Technology Program Annex 18

27 No. of Copes Organzaton 1 ADMNSTR ELEC DEFNS TECHL INFO CTR ATTN DTIC OCP 8725 JOHN J KINGMAN RD STE 0944 FT BELVOIR VA CD US ARMY RSRCH LAB ATTN RDRL CIM G T LANDFRIED BLDG 4600 ABERDEEN PROVING GROUND MD CDS US ARMY RSRCH LAB ATTN IMNE ALC HRR MAIL & RECORDS MGMT ATTN RDRL CIM L TECHL LIB ATTN RDRL CIM P TECHL PUB ADELPHI MD HCS US ARMY RSRCH LAB ATTN RDRL CII A P DAVID (5 HCS) S YOUNG N FUNG ADELPHI MD TOTAL: 12 (1 ELEC, 7 HCS, 4 CDS) 19

28 INTENTIONALLY LEFT BLANK. 20