Real-time 2D Video/3D LiDAR Registration

Transcription

1 Real-ime 2D Video/3D LiDAR Regisraion C. Bodenseiner Fraunhofer IOSB M. Arens Fraunhofer IOSB Absrac Progress in LiDAR scanning has led o he availabiliy of large scale LiDAR daases for urban areas. We use such pre-acquired daa o deermine he poses of 2D monocular cameras highly accuraely in real-ime. This is achieved by firs correcly aligning key-frames of he muli-modal daa using a combinaion of feaure and inensiy-based 2D/3D regisraion mehods. The online pose is hen deermined in realime by densely sampling and racking feaures wihin he 2D video sream. The 3D coordinaes of hese feaures are deermined by a fas GPU-based backprojecion. The observed 2D/3D feaure daa is hen fused using a recursive Bayesian filer in order o exploi emporal coherency. The mehod is evaluaed using ground ruh camera rajecories and differen filer implemenaions. The proposed regisraion and filer framework execues a video-frame rae and i is up o 15% more accurae hen a regisraion only soluion. Applicaions are numerous and include, for insance, augmened-realiy applicaions, online georefereniaion or meric online 3D reconsrucion from monocular video daa. 1. Inroducion The accurae 3D mapping of he environmen using LiDAR sensors has seen grea progress in he las decade. This has led o he availabiliy of large scale Li- DAR daases for urban areas. In his work we describe a realime mehod for using such previously acquired LiDAR scans in combinaion wih a monocular 2D vision sysem. We use he backscaered laser inensiy informaion o generae synheic 2D views, which in urn enable muli-modal appearance-based regisraion of camera images w.r.. he coordinae sysem of he global scan. We provide insighs on how o pre-process raw LiDAR daases and how o achieve an accurae online mulimodal regisraion using Bayesian filer echniques. 2D Video/3D LiDAR Regisraion: There exiss a vas body of lieraure concerning 2D/3D regisraion mehods. Close o our applicaion scenario are he following works: Masin e al. [7] regiser heigh color coded 2D renderings wih camera images using muual informaion (MI) [10]. Vasile e al. [9] derive pseudo-inensiy images from LiDAR daa, including shadows, o allow for a 2D/3D regisraion wih aerial imagery. Feaure based approaches [5, 2] mosly rely on he deecion and alignmen of geomeric feaures like corners or line segmens in he camera image and projecions of hose from he 3D daa. For a good inroducion o filer based localizaion mehods we refer he reader o he work of Thrun e al. [8]. Conribuion: To our bes knowledge realime mulimodal video/lidar regisraion for meric online localizaion based on large-scale LiDAR daa have no been presened in he lieraure. We show how such a sysem can be buil. In paricular, we poin ou wha he enabling assumpions are and wha our design of he relevan probabiliy disribuion funcions is. We furher show how recursive Bayesian filer echniques are inegraed wihin he 2D/3D regisraion o exploi emporal coherency. Addiionally, we propose o use LiDAR poin clouds as a meric muli-modal calibraion body for an accurae inrinsic calibraion of he camera. The ouline of he paper is as follows: firs he key elemens of he mehod are described. The regisraion accuracy is hen evaluaed using ground ruh camera rajecories; in he evaluaion differen recursive filer echniques are evaluaed. Finally, he resuls and furher research direcions are discussed. 2. Mehod The proposed mehod comprises he following main elemens: (A) The raw LiDAR poin cloud daa is processed o allow for efficien rendering of 2D synheic views. This

2 involves he 3D regisraion of muliple erresrial and aerial laser scans and a muli-scale represenaion via ocree downsampling mehods using he PCL library. (B) The rendered views enable appearance-based regisraion wih keyframe camera images. For each camera keyframe, he regisraion is based on local feaure correspondences and a 2D/3D PnP solver [6]. The regisraion can hen be refined using muual informaion/gradien correlaion (MI/GC) [10, 4]. Wih muliple keyframes regisered, he inrinsic camera parameers can be also accuraely compued by opimizing he disance measure over he inrinsic parameers. (C) A dense se of feaures is exraced and racked in he camera images and heir corresponding 3D coordinaes are compued using he keyframe regisraions; for feaure racking we used Harris-Foersner corners in combinaion wih a sparse Lukas-Kanade opic flow mehod. The resuling dense 2D/3D correspondences are employed in a robus, RANSAC-based realime 2D/3D PnP solver [6] in combinaion wih a paricle filering backend. 2.1 (A) - LiDAR Scan Preprocessing We generae synheic inensiy images from he 3D LiDAR daa. The inensiy informaion for he synheic views sems from he backscaered laser pulse informaion. Local feaures are exraced from hese images using SURF descripors. The 3D-coordinaes of he feaures are deermined based on he GPU-deph buffer informaion a he feaure deecor posiions. 3D LiDAR Scans Regisraion: The regisraion of 2D images wih muliple raw LiDAR daases requires a common scan coordinae sysem and herefore he prior regisraion of all local laser scans. We choose he local coordinae sysem of a cenral LiDAR daase as reference sysem. We hen exrac local feaures from he virual views o auomaically find 2D/2D appearance based correspondences beween he scanning posiions. Corresponding feaures are backprojeced o obain 3D/3D correspondences for he esimaion of he rigid 3D ransformaion beween he scans. The ransformaion is robusly esimaed using Horns mehod. 2.2 (B) - Keyframe Pose Regisraion For he muli-modal 2D/3D keyframe regisraion we uilize a poin-based rendering approach o generae synheic 2D views from he 3D daase. 2D/2D correspondences wih he camera image are robusly idenified by searching for local regions of feaures where corresponding feaures have similar geomeric relaionships by employing a Generalized Hough Transform Figure 1. Combined airborne and erresrial LiDAR-scan 3D background model. [1]. The 3D posiions for he synheically generaed 2D feaures can easily be deermined using he GPU deph buffer informaion. The 2D/3D feaure-based regisraion is carried ou by he Ransac-based PnP solver [6] (inlier hreshold 3px,2500 ieraions). The resuling pose is refined by an inensiy-based regisraion. Inensiy-Based Pose Refinemen: To increase regisraion accuracy for regisered poses an inensiy-based similariy measure beween rendered views and query images is maximized. The convergence range of his opimizaion problem is usually small. However, he pose compuaion based on local feaures generally provides a sufficienly good saring poin. An imporan choice involves he selecion of an appropriae disance measure. The disance measure MI[10] is considered he gold sandard similariy measure for muli-modal maching. I measures he muual dependence of he underlying image-inensiy disribuions: D (MI) (I R, I Tθ ) = H(I R )+H(I Tθ ) H(I R, I Tθ ) (1) where H(I R ) and H(I Tθ ) are he marginal enropies and H(I R, I Tθ ) = p(x, Y )log( p(x, Y ) p(x)p(y ) ) X I Tθ Y I R (2) is he join enropy. p(x, Y ) denoes he join probabiliy disribuion funcion of he image inensiies X, Y in I R and I Tθ, and p(x) and p(y ) are he marginal probabiliy disribuion funcions. We linearly combine he MI measure above wih he gradien correlaion measure in [4] o enhance robusness and accuracy. To speed up compuaion ime, we resric he disance

3 compuaion o local regions around inlier feaures in he local feaure-based regisraion. 2.3 (C) - Recursive-Bayes Pose Filering The goal of his sep is o esimae he camera rajecory, represened by he poserior disribuion p(x 0: y 1: ) and he curren belief abou he camera pose a ime, represened by he marginal disribuion p(x y 1: ). The sae sequence {x : IN }, x X is assumed o be Markovian wih iniial disribuion p(x 0 ) and ransiion probabiliy p(x, x 1 ). The observaions {y : IN }, y Y consis of ses of racked 2D feaures in combinaion wih heir corresponding 3D poin posiions in he LiDAR background model. The observaions are assumed o be condiionally independen given he sae e.g. p(y x, y 1: 1 ) = p(y x ), which is approximaely rue in our daa se. Given he above menioned independence assumpions his leads o he well known recursive updae scheme [3]: p(x y 1: ) = αp(y x ) p(x x 1 )p(x 1 y 1: 1 ) x 1 (3) However, he inegral does no have a closed form soluion excep in is basic form. We es wo alernaives for overcoming hese resricions. One is o assume linear ransiions and observaions wih addiive Gaussian noise disribuions and arrives a he Kalman filer equaions. The oher is paricle filering. Paricle filering approximaes he soluion by a se of weighed paricles {x (i), π (i) }, where each paricle is an insance of a possible camera posiion a ime x (i) is is corresponding weigh, reflecing he confidence level on his posiion. The above inegral is hus approximae by: and π (i) p(x y 1: ) αp(y x ) n i=1 π (i) 1 p(x x 1 ) (4) In order o implemen he paricle filer one mus specify hree disribuions: 1) he dynamical disribuion p(x (i) 1 ) of he sae process, 2) he disribuion p(y ) of he observaion likelihood, and 3) a proposal disribuion p p (x (i) 1:, y 1:) for updaing he paricle se. The sae process models he moion of he camera, for which we use a consan acceleraion model p(x x 1 ) N (f(x 1, ẋ 1 ), Σ), where N (µ, Σ) denoes a normal disribuion wih mean µ and covariance Σ and : f(x 1, ẋ 1 ) = Ax 1 + Bẋ 1 (5) The consan coefficiens of he marices A and B have been deermined experimenally. The disribuion of he observaion likelihood is based on he reprojecion error beween he curren corresponding racked 2D image feaures m i and heir backprojeced 3D poins M i given he inrinsic parameers K of he camera and he roaion R and ranslaion parameers from he sae x : g(r, ) = i K(RM i + ) m i (6) For he compuaion of weighs π (i) we used he convex Huber cos funcion ρ(g(r, )) which inrinsically handles ouliers (wih reprojecion errors 5px) by a linear penaly. The proposal disribuion is modeled by 1:, y 1:) N (h(y ), Σ p ), where h(y ) p(x (i) refers o he calculaed pose parameers from he curren racked feaure se. A each ieraion in he recursive updae we use he sequenial imporance resampling scheme in [3], based 1:, y 1:) and finally reweigh on he curren paricle se {x (i), π (i) }. Firs we sample n paricles from he curren se according o a low variance sampler as described in [8]. We hen updae he curren paricle se wih paricles sampled from he proposal disribuion p(x (i) and normalize he paricles according o: π (i) p(y )p(x (i) 1 ) p p (x (i) 1:, y 1:) 3. Experimens and Numerical Resuls (7) To measure he performance of he proposed framework we used LiDAR scans of a small ciy and video sreams from hand-held cameras and miniuav sysems moving in he same ciy. We also generaed synheic video-sreams based on very dense LiDAR scans ( 10 9 daa poins) o generae ground ruh camera rajecories. We hen measured he increase in performance due o recursive filering mehods over he regisraion only soluion. Implemenaion Deails: The algorihms are implemened in C/C++ based on he OpenCV/PCL libraries; we noe ha many pars of our sofware framework can be furher runime-opimized. The repored ime measuremens herefore provide only an early esimae. The

4 es sysem is equipped wih an i5-2500k processor. Regisraion Accuracy: The measuremens resuls are based on he regisraion of 4500 camera frames. The regisraion accuracy is defined as he disance of he compued camera posiions o he ground ruh camera poses. The means and variances of he posiional accuracy in x,y,z coordinaes expressed in meers were 0, 18-1, 91, 0, 21-0, 77 and 0, 17-1, 00 respecively. The means and variances of he roaional accuracy in x,y,z- Euler angles expressed in degrees were 0, 12-0, 17, 0, 07-0, 50 and 0, 24-0, 21, respecively. Filer Performance: The figure below shows comparaive plos of he L 2 disances o he ground ruh camera ceners, when using a regisraion only, a Kalman filer and he proposed paricle filer for wo example sequences of 250/500 frames wih one iniial keyframe regisraion. The camera rajecories sem from low aliude miniuav flighs. Noe ha he bes regisraion resuls were obained using he paricle filer. The measured posiional regisraion accuracy (deviaion from a ground ruh pose) (x,y,z) for he kalman filer was 0, 23m (mean) and 1, 87 (variance), 0, 24 wih variance 0, 46 and 0, 18 wih variance 0, 33 respecive 0, 18m (mean) and 0, 27 (variance), 0, 20 wih variance 0, 19 and 0, 07 wih variance 0, 13 for he paricle filer. ransac ieraions ( ) and video resoluion (720x576px;1280x720px). Seing he parameers o maximal values led o frame raes of 20Hz-24Hz. The Kalman filer soluion produces no measurable runime overhead. Paricle filering however induces a runime overhead of 6-7fps based on an average paricle se size of 400. We noe ha his overhead is, a leas parly, caused by he fac ha our paricle filer implemenaion is no parallelized so far. 4. Conclusion And Oulook In his work we proposed and implemened a realime 2D/3D vision/lidar regisraion sysem. The proposed regisraion and filer framework works a video frame-rae and achieves up o 15% accuracy improvemens compared o a soluion based on regisraion only. Promped by he recen rend of dense mehods in compuer vision, we inend o invesigae windowed bundle adjusmen mehods in combinaion wih dense variaional opic flow for high accuracy vision based localizaion. We are currenly working on sparse filering mehods o invesigae he rade-off beween accuracy and compuaional speed for resource limied devices. References Runime Parameers: The runime of he regisraion algorihm depends on various parameers. Parameers ha ypically need o be considered include he number of racked feaures ( ), [1] C. Bodenseiner, W. Huebner, K. Juengling, J. Mueller, and M. Arens. Local muli-modal image maching based on self-similariy. In Proc. IEEE-ICIP, [2] M. Ding, K. Lyngbaek, and A. Zakhor. Auomaic regisraion of aerial imagery wih unexured 3d lidar models. In CVPR, [3] A. Douce, N. Freias, and N. Gordon. Sequenial Mone Carlo Mehods in Pracice. Springer-Verlag, [4] G. P. e. al. A comparison of similariy measures for use in 2-d-3-d medical image regisraion. IEEE TMI, 17(4): , [5] C. Frueh, R. Sammon, and A. Zakhor. Auomaed exure mapping of 3d ciy models wih oblique aerial imagery. In 3DPVT 2004, pages , [6] V. Lepei, F. Moreno-Noguer, and P. Fua. Epnp: An accurae o(n) soluion o he pnp problem. Inernaional Journal of Compuer Vision, 81: , [7] A. Masin, J. Kepner, and J. Fisher. Auomaic regisraion of lidar and opical images of urban scenes. In CVPR, [8] S. Thrun, W. Burgard, and D. Fox. Probabilisic Roboics. MIT Press, Cambridge, MA, [9] A. Vasile, F. R. Waugh, D. Greisokh, and R. M. Heinrichs. Auomaic alignmen of color imagery ono 3d laser radar daa. In AIPR, [10] P. Viola and W. Wells. Alignmen by maximizaion of muual informaion. Inernaional Journal of Compuer Vision, 24(2): , 1997.

5 Year: 2012 Auhor(s): Bodenseiner, C.; Arens, M. Tile: Real-ime 2D video/3d LiDAR regisraion 2012 IEEE. Personal use of his maerial is permied. However, permission o reprin/republish his maerial for adverising or promoional purposes or for creaing new collecive works for resale or redisribuion o servers or liss, or o reuse any copyrighed componen of his work in oher works mus be obained from he IEEE. Deails: Inernaional Associaion for Paern Recogniion -IAPR-; IEEE Compuer Sociey: 21s Inernaional Conference on Paern Recogniion, ICPR Vol.3 : Tsukuba, Japan, November 2012 Piscaaway/NJ: IEEE, 2012 ISBN: (Prin) ISBN: pp