Learning nonlinear appearance manifolds for robot localization

Transcription

1 Learning nonlinear appearance manifolds for robo localizaion Jihun Hamm, Yuanqing Lin, and Daniel. D. Lee GRAS Lab, Deparmen of Elecrical and Sysems Engineering Universiy of ennsylvania, hiladelphia, A {jhham, linyuanq, ddlee}@seas.upenn.edu Absrac We propose a nonlinear mehod for learning he low-dimensional pose of a robo from high-dimensional panoramic images. The panoramic images are assumed o lie on a nonlinear low-dimensional appearance manifold ha is embedded in a high-dimensional image space. We demonsrae ha he local geomery of a poin and is neares neighbors on his manifold can be used o projec he poin ono a low-dimensional coordinae space. Using his embedding, he unknown camera posiion can be esimaed from a novel panoramic image. We show how he image-based posiion measuremens can be inegraed wih odomery informaion in a Bayesian framework o yield an online esimae of a robo s posiion. Resuls from simulaed daa show ha he proposed mehod ouperforms oher appearance-based models based upon principal componens analysis and kernel densiy esimaion. Index Terms Appearance-based Localizaion, Manifold Learning, Nonlinear Feaure Exracion, Bayesian Filering I. INTRODUCTION Recenly, here has been much ineres in using vision sensors as a primary source of informaion o esimae he pose of a mobile robo. In his regard, here are wo disinc approaches o incorporaing visual informaion: geomery versus appearance-based mehods. The geomeric approach esimaes he ego-moion of he camera using geomeric models of he environmen as well as he camera. I ypically relies upon one or more calibraed cameras as well as preprocessing of he images. The calibraed images are hen analyzed o exrac and esimae he relaive posiions of environmenal feaures or landmarks [14]. Sereo cameras, or more recenly, omnidirecional cameras using srucure from moion have been used o rack he hree-dimensional posiions of hese feaures over ime in order o esimae egomoion [7], [2]. In conras, appearance-based mehods do no rely upon calibraed cameras o esimae pose. Raher han aemping o exrac localized feaures in he images, hese mehods consider he inpu images more holisically in relaion o oher images. Images wih a wide field of view are preferable for hese models, and recen work have used panoramic images for inpus [1], [10], [6], [3]. In paricular, we consider oriened cylindrical panoramic images as inpu images. In his case, he wo-dimensional ranslaional coordinaes of he camera will be refleced in he appearance of he resuling image. Convenional appearancebased mehods model his underlying low-dimensional srucure in he images by firs projecing a raining se of images wih known camera posiions ono a low-dimensional subspace spanned by a few principal componens of he images. Given a new image, he unknown camera posiion is esimaed by projecing he new image ono his principal componen subspace. The raining image wih he closes projecion o he new image is found, and he camera posiion of his closes raining image is used o esimae he unknown camera posiion [6]. The accuracy of his approach is limied by he spacing of he camera posiions during he acquisiion of he raining images. A recenly-developed mehod generaes a more refined esimae by inerpolaing beween raining images using kernel densiy esimaion [9]. The low-dimensional principal componen projecion allows he camera posiion esimae o be more robus o image noise and provides for efficien implemenaion. However, i is clear ha a linear subspace may be inappropriae for modeling he variabiliy in he appearance of he images. In his submission, we describe a mehod ha models he images as a low-dimensional nonlinear manifold embedded in a highdimensional Euclidean space R D, where D is he number of pixels in a picure. This appearance manifold is homeomorphic o he wo-dimensional space of camera posiions, and we learn he mapping from images o camera posiion by locally esimaing he srucure of his manifold. Our algorihm for Locally Linear rojecion (LL) approximaes he mapping from images o camera posiions using a locally weighed neighborhood. We show how LL can be easily incorporaed ino a Bayesian framework for online robo localizaion, and we compare is performance o convenional appearance-based localizaion mehods. The remainder of he paper is organized in he following manner. In Secion II, we briefly review how principal componens analysis and kernel densiy esimaion are currenly used for appearance-based localizaion. Secion III describes how he local srucure of nonlinear manifolds can be esimaed wih LL and used o map panoramic images o camera posiions. In Secion IV, we incorporae he LL measuremens ino a probabilisic model for robo localizaion. We hen compare he performance of our algorihm wih oher appearance-based localizaion algorihms in Secion V. Finally in Secion VI, some possible exensions and planned fuure work will be discussed.

2 Noaion In he following, x R 2 denoes a wo-dimensional posiion vecor and y R D an image in a D-dimensional pixel space. Dimensionaliy reducion on he images resuls in he low-dimensional vecor z R d where d D. A reference or raining se consiss of a se of T known posiions {x 1, x 2,, x T }, along wih heir corresponding images Y {y 1, y 2,, y T }, and possibly heir reduced dimensionaliy represenaion Z {z 1, z 2,, z T }. Thus, his daa is described by marices of size (2 T ), (D T ), and (d T ), respecively. II. RELATED WORK Much of he previous work in appearance-based localizaion preprocess he visual images using principal componens analysis (CA) o reduce heir dimensionaliy. robabilisic models for localizaion using kernel densiy esimaion (KDE) are hen consruced from hese principal componens. In his secion, we describe how CA and KDE are used in his manner for localizaion. A. Linear dimensionaliy reducion CA is a simple and efficien mehod for linear dimensionaliy reducion. Given a raining se of images Y, CA finds a low d-dimensional subspace of he pixel space spanned by orhogonal direcions wih maximum variance. The corresponding eigenimages Z are given by projecing he images ono he eigenvecors of he covariance marix C T /T wih maximal eigenvalues. In pracice, singular value decomposiion is ofen used o facor he image marix: Y USV T. The componens of Z are hen given by he projecion of he images ino he principal subspace: Z U T Y U T USV T SV T. This decomposiion can be shown o resul in he smalles linear reconsrucion error o Y. B. Kernel Densiy Esimaion Kernel Densiy Esimaion (KDE), ohewise known as arzen window densiy esimaion, is a nonparameric procedure o approximae a disribuion from a sample of poins. KDE esimaes an unknown disribuion by convolving he sample se wih a coninuous kernel densiy, and hus assumes very lile daa-specific srucure. For appearance-based mehods, KDE is usually performed no on he original images Y, bu raher on heir low-dimensional represenaion Z given by CA. In paricular, he join densiy relaing posiion o appearance is given by [9]: p(x, z, Z) 1 T T g x (x x n )g z (z z n ). n1 where g x and g z are Gaussian densiy funcions g x (x x n ) g z (z z n ) 1 2π Ψ x 1/2 exp { (x x n ) T Ψ 1 x (x x n )/2 }, 1 (2π) d/2 Ψ z 1/2 exp { (z z n ) T Ψ 1 z (z z n )/2 }. wih covariance parameers Ψ x and Ψ z. Since p(x, z, Z) is facorized, he likelihood p(z x,, Z) is given by he relaionship: p(z x,, Z) p(x, z, Z) p(x, Z) T n1 g x(x x n )g z (z z n ) T n1 g. x(x x n ) KDE is convenien in ha i exhibis asympoic convergence of he esimaed mean and covariance o he rue underlying disribuion. One disadvanages of KDE, however, is ha he parameers Ψ x and Ψ z need o be carefully chosen [9]. One mehod is o simply use a consan covariance Ψ x Ψ z σ 2 I. For our experimens, we opimize wo separae variances Ψ x σxi, 2 Ψ z σzi 2 o obain he bes resuls. III. AEARANCE MANIFOLD FOR LOCALIZATION In his secion, we describe wo relaed mehods for learning a nonlinear map from high-dimensional appearance images o a low-dimensional coordinae space. Locally linear embedding embeds he images in an unsupervised manner, as an alernaive o CA. We also describe locally linear projecion, which can be used o nonlinearly map images direcly o posiion coordinaes. A. Locally linear embedding The geomeric inuiion behind locally linear embedding (LLE) is quie simple. If he image daa Y is dense enough, we expec each daa poin and is neighbors o lie on or close o a locally linear pach of he underlying manifold [13]. The local geomery of he paches are characerized by linear coefficiens W ij ha reconsruc each daa poin from is neighbors. The weighs W ij are chosen o minimize he reconsrucion error C(W ) i y i j W ij y j 2, (1) given by he sum squared disances beween all he daa poins and heir local linear reconsrucions. In minimizing his cos, he weighs are consrained by j W ij 1, for all i. This condiion makes he weighs invarian o roaions, rescalings, and ranslaions; consequenly, he resuling weighs characerize inrinsic geomeric properies of each neighborhood ha do no depend upon a paricular reference frame. The

3 -posiion Y-posiion Rooms RD Map y y1 W1 Wk yk y~ W2 y2 CA x~ R2 x1 W1 Wk xk W2 LLE x2 Fig. 1. Comparison of CA projecion and LLE embedding of he image daa. Top row shows he environmen map and he grid posiions where camera images are acquired (see Secion V.) Middle row shows he firs wo CA componens and he boom row shows he firs wo coordinaes of he LLE embedding. d-dimensional embedding z Rd, is hen deermined by minimizing C(Z) kz i Wij z j k2, i j where Wij are he weighs previously deermined in (1). In CA, he principal componens are linear coordinaes in d-dimensional space which bes approximae he innerproducs Y T Y of he original daa. The LLE embeddings, in conras, are coordinaes in d-dimensional space which bes preserve he local geomery of he original daa. A demonsraion of he differences beween he CA and LLE embeddings of image daa is given in Fig. 1. The color mapping shows he correlaion beween he posiion of he camera and he resuling projecions. In he CA projecion, he firs componen varies almos monoonically wih he xaxis, however he second componen has no simple relaion o posiion. The images from differen rooms in he map become unresolvable in he wo CA componens. On he oher hand, he LLE embedding of he images preserves he opological srucure of he rooms quie well. Addiionally, he embedding coordinaes are almos homeomorphic o he rue camera posiions. These resuls demonsrae he imporance of nonlinear learning mehods o model he srucure of he appearance manifold. B. Locally linear projecion Fig. 2. Locally linear projecion of a es image y o he posiion space. Suppose he raining ses and Y are given, and he images Y lie close o a manifold parameerized by he camera posiions. The es poin y is projeced ino he affine space of K-neares neighbors, y 1, y 2,, y K,. and he weighs Wj are compued by he consrucing he projecion y The weighed mean x j Wj xj of he K-neares neighbor posiions x1, x2,, xk is used o esimae he posiion of image y. The uncerainy of his esimae is compued from he covariance of x1, x2,, xk. We modify he LLE algorihm o incorporae manifold srucure in order o direcly esimae he underlying camera posiion from he images. Insead of embedding he manifold in an unsupervised manner, locally linear projecion (LL) uses local linear paches o inerpolae he posiion of he camera. The geomerical idea is illusraed in Fig. 2. A given image y is firs approximaed by K-neares neighboring j Wj y j. images hrough local linear reconsrucion: y The error in his reconsrucion can be checked o ensure he locally linear assumpion is no violaed. The weighs Wj j W j xj. are hen used o esimae he unknown pose x This esimae explois he coninuiy of he map from image manifold o pose manifold. The uncerainy of his esimae is proporional o he sample covariance of he K neighboring poses. Thus, he uncerainy of he esimae is small if he neighboring poins are ighly clusered in posiion space. The resuling algorihm for LL is summarized below: Sep 1. Given an image y, find he K-neares neighbors A in he reference se Y. Sep 2. Compue he weighs W which linearly reconsruc he poin y from is neighbors: W arg min ky wj y j k2, W j A subjec o j A wj 1. Sep 3. Theposiion esimae is given by he weighed j wj xj. mean x

4 Sep 4. The uncerainy in he esimaed posiion is compued by scaling he sample covariance by a facor ρ: Ψ ρ (x j x)(x j x) T. K j A This procedure yields a probabilisic measuremen model for he posiion x corresponding o an image y: p(x y,, Y ) N ( x, Ψ). (2) The complexiy of finding he K-neares neighbors of an image using efficien algorihms is O(K log T ) while compuing he resuling weighs coss O(DK 3 ). Considering ha K is usually small (K < 20), his resuls in a compuaionally efficien procedure o esimae posiion. In comparison, KDE requires O(T ) complexiy o convolve he reference poins wih a kernel. Besides is algorihmic efficiency, LL is able o accuraely model he geomerical manifold srucure of he images. IV. BAYESIAN FILTERING We review Bayesian filering mehods used in robo localizaion, and show how LL can be combined wih odomery informaion for online localizaion. The uncerainy in he robo s posiion given he curren and all previous measuremens is described by he belief sae: Bel(x ) p(x curren and pas daa) p(x y,, y 1, u 1,, u 1 ). For mobile robos, wo condiions are usually assumed: 1) he curren sae x given he las odomery reading u 1 and previous sae x 1 is independen of oher pas daa, and 2) he measuremen y given he curren sae x is independen of oher daa. These Markov assumpions make sae esimaion a recursive Bayesian procedure: Bel(x ) ηp(y x ) p(x x 1, u 1 ) p(x 1 y 1,, y 1, u 1,, u 1 )dx 1 ηp(y x ) p(x x 1, u 1 )Bel(x 1 )dx 1, where η is a normalizaion consan independen of x: η p(y u 1,, u 1 ) 1. The wo disribuions p(y x ) and p(x x 1, u 1 ) are deermined by he measuremen and process model, respecively. The updae procedure can hen be divided ino wo seps. The process model is used o esimae: Bel(x ) p(x x 1, u 1 )Bel(x 1 ), (3) and hen measuremen is incorporaed ino: Bel(x ) ηp(y x )Bel(x ). (4) When he iniial belief is Gaussian and he wo process and measuremen models are boh linear-gaussian, he belief says Gaussian disribued for all ime. The resuling updaes become sraighforward marix operaions known as he Kalman filer. In our process model, a zero-mean Gaussian noise wih saionary covariance Q is used for he uncerainy in he odomery u: p(x x 1, u 1 ) N (x 1 + u 1, Q). However, in our measuremen model, he image y observed a he sae x is no a simple linear funcion of x. Thus, as in he Exended Kalman filer (EKF), he nonlinear model is linearized using he Jacobian of he ransformaion and he residuals are assumed o be Gaussian. Applying Bayes rule, we replace he generaive disribuion ηp(y x) in (4) wih he nonlinear projecion model p(x y): p(y x) p(x y), and furhermore, p(y x) p(x y), p(x) by assuming a nonspecific prior for p(x). Using he model p(x y,, Y ) in (2), he resulan updae equaions are effecively he same as he Kalman filer, hough compued in a slighly differen form: Le µ, Φ, x, Φ, ˆx, Ψ denoe he means and covariances of Bel(x ) N (µ, Φ ) Bel(x ) N (x, Φ ) p(x y,, Y ) N ( x, Ψ ). Sep 1. rocess updae x µ 1 + u 1 Φ Φ 1 + Q Sep 2. Measuremen updae Φ 1 Φ 1 + Ψ 1 µ Φ (Φ 1 x + Ψ 1 x ). Afer each image measuremen, he mean and he covariance of he model p(x y,, Y ) are firs compued from LL. These parameers are hen used o updae he curren belief over he robo posiion using he procedure above. V. EERIMENTS We esed he LL and localizaion algorihm using a virual environmen and robo simulaion. The 3D model of he environmen ha we consruced is shown in Fig. 3. The resuling scenes and cylindrical camera projecions were rendered by a ray-racer capable of modeling reflecive surfaces and realisic ligh sources [11]. The daa consised of he following images and camera posiions: a raining se (428), spli ino a reference (328) and cross-validaion (100) se, as well as a es se (135) o es localizaion performance. The reference posiions are on a regular grid of laice size 1m in

5 Simulaed Environmen True 1-NN: KDE: LL: A B C anoramic Views A B C Fig. 3. The upper figure displays he simulaed environmen (20m 20m) wih hree rooms A, B and C, which are opologically conneced hrough doorways. The environmen includes exured walls and obsacles ha occlude he rendered scenes. The boom images are examples of cylindrical camera projecions aken from he cener of rooms A, B, and C. he 20m 20m environmen, and he cross-validaion images and posiions were uniformly randomly sampled. The es se was generaed from he simulaed moion of a robo wih a sep size of approximaely 0.06m. Odomery readings are corruped by isoropic Gaussian noise of sd 0.06m added o each movemen sep. The panoramic images are gray-scale and down-sampled o pixel sizes , preserving he aspec raio beween heigh and radius of he cylindrical camera. Fig. 4. Localizaion resul for 1-neares neighbor (1-NN), Kernel Densiy Esimaion, and Locally Linear rojecion using only he curren image measuremen. Real lines indicae he rue rajecory of he robo. Arrows indicae he deviaion emeas of he esimaed posiion from he rue posiion. The average lenghs of deviaion E meas are also given. as follows. The emeas denoes he difference of he rue E[x y,, Y ]: posiion x and he esimaed posiion x meas emeas x x, and E is he average error " T #1/2 meas meas 2 E e /T. 1 Similarly, he error of he Bayesian-filered localizaion is compued by ef il x µ where µ E[Bel(x )], and he average error is " T #1/2 f il f il 2 E e /T. 1 A. erformance evaluaion The resuls of appearance-based localizaion were evaluaed in wo differen ways: deerminisic vs. probabilisic. Deerminisic localizaion uses only a single observed image y o esimae he robo posiion given ha image E(x y,, Y ). In conras, probabilisic localizaion uses he full model p(x y,, Y ) o accoun for uncerainy in esimaion, allowing for more accurae localizaion by incorporaing odomery readings u. Boh he convenional CA/KDE algorihm as well as LL can be used in he wo siuaions. We compue he localizaion error in he wo cases, where he posiion is esimaed 1) wih measuremen only and 2) afer Bayesian-filering boh image measuremens and odomery. The error of localizaion wih measuremen only is defined B. arameer selecion To do a fair comparison, we need o adjus he parameers of he various algorihms for bes performance. In order o do his, he cross-validaion raining se was used o opimize he dimensionaliy d for CA, and he widhs σx, σz for he KDE kernels. For LL, he cross-validaion se was used o opimize he number of neighbors K and he covariance scaling parameer ρ. When LL and KDE are used in he deerminisic comparison, only he parameers K, d, and σz are relevan. We found he minimum error for he wo algorihms was achieved using K 5, d 12 and σz For he Bayesian filering algorihm, he addiional parameers ρ 0.05 and σx 0.088

6 True Odomery only: Odomery+KDE: Odomery+LL: D environmen, LL demonsraes superior performance o mehods based upon CA and KDE. Evaluaions using acual panoramic images from a mobile robo plaform are currenly being performed. In his submission, we assume he roaional orienaion of he robo is known and considered only ranslaional degrees of freedom. When roaional angle is included in he pose space, he appearance manifold possesses he opology of a solid orus S R 2. Learning algorihms o esimae non-euclidean manifolds are sill very underdeveloped, and mehods ha use self-organized maps or neural neworks are no paricularly efficien wih appearance images [12], [5]. We are invesigaing how a mehod such as locally linear projecion may be exended o handle hese more challenging siuaions. ACKNOWLEDGMENT The auhors acknowledge he suppor he U.S. Naional Science Foundaion, he Army Research Office, as well as he Defense Advanced Research rojecs Agency. Fig. 5. Localizaion resul of CA/KDE and LL wih and wihou odomery. Lines indicae he rue robo rajecory, and he arrows indicae he deviaion e fil of he esimaed posiion from he rue posiion. The average lengh of he deviaion E fil is also given. were esimaed by minimizing E fil over he cross-validaion se. C. Resuls The performance of he algorihms for deerminisic localizaion is shown in Fig. 4. The naive 1-neares neighbor algorihm resuled in 0.46m average error, which is approximaely half he spacing of he reference images. CA/KDE achieves a comparable error of 0.281m, while LL produces an even smaller error of 0.208m. The figure also shows ha he larges errors occur near he doorways connecing he rooms in he environmen. Fig. 5 demonsrae he localizaion performance of CA/KDE and LL incorporaing odomery. When odomery is used alone, he small (sd 0.06m) noise added o each movemen amouns in a large drif afer a he end of he moion. Wih Bayesian filering, CA/KDE and LL reduce he localizaion error o 0.174m and 0.141m respecively, showing he advanage of using a nonlinear probablisic model. Errors in posiion near he doorways where measuremen-only localizaion produces large errors, were miigaed using he probabilisic model. VI. DISCUSSION A probabilisic localizaion algorihm which direcly maps high-dimensional appearance images o robo posiion via a nonlinear manifold is proposed. In ess on a simulaed virual REFERENCES [1] N. Aihara, H. Iwasa, N. Yokoya, and H. Takemura, Memory-based self-localizaion using omnidirecional images, in roceedings of Foureenh Inernaional Conference on aern Recogniion,2: , [2] R. Bunschoen and B. Kröse, 3-D scene reconsrucion from cylindrical panoramic images, In roceedings of 9h Inernaional Symposium on Inelligen Roboic Sysems (SIRS 2001), , LAAS-CNRS, Toulouse, France, July [3] D. Cobzas, and H. Zhang, Cylindrical panoramic image-based model for robo localizaion, in roceedings of IEEE/RSJ Inernaional Conference on Inelligen Robos and Sysems (IROS 2001), , Ocober 2001, Hawaii, USA. [4] J. L. Crowley and F. ourraz, Coninuiy properies of he appearance manifold for mobile robo posiion esimaion, in Image and Vision Compuing, 19(11): , [5] W. Duch, and R. Adamczak, Neural neworks in non-euclidean meric spaces, in roceedings of Inernaional Join Conference on Neural Neworks, Washingon, , [6] J. Gaspar, N. Winers and J. Sanos-Vicor, Vision-based navigaion and environmenal represenaions wih an omnidirecional camera, in IEEE Transacions on Roboics and Auomaion, 16(6): , [7] J. Gluckman and S. K. Nayar, Ego-moion and omnidirecional cameras, in roceedings of he Sixh Inernaional Conference on Compuer Vision (ICCV 98), , [8] M. Jogan and A. Leonardis, Robus localizaion using an omnidirecional appearance-based subspace model of environmen, in Roboics and Auonomous Sysems, 45(1):51 72, 2003, Elsevier Science. [9] B. J. A. Kröse, N. Vlassis, R. Bunschoen and Y. Moomura, A probabilisic model for appearance-based robo localizaion, in Image and Vision Compuing, 19(6): , 2001 [10] T. ajdla and V. Hlavac, Zero phase represenaion of panoramic images for image based localizaion, In roceedings of 8h Inernaional Conference on Compuer Analysis of Images and aerns CAI 99, ,1999. [11] ersisence of Vision y. Ld., Williamsown, Vicoria, Ausralia hp:// [12] H. Rier, Self-organizing maps in non-euclidean spaces, in Kohonen Maps, E. Oja and S. Kaski eds., , [13] L. K. Saul, and S. T. Roweis, Think globally, fi locally: unsupervised learning of low dimensional manifolds, in Journal of Machine Learning Research, 4: , [14] S. Se, D. Lowe, and J. Lile, Local and global localizaion for mobile robos using visual landmarks, In roceedings of he IEEE/RSJ Inernaional Conference on Inelligen Robos and Sysems (IROS 2001), , 2001.