Motion Estimation of a Moving Range Sensor by Image Sequences and Distorted Range Data

Transcription

1 Moion Esimaion of a Moving Range Sensor by Image Sequences and Disored Range Daa Asuhiko Banno, Kazuhide Hasegawa and Kasushi Ikeuchi Insiue of Indusrial Science Universiy of Tokyo Komaba, Meguro-ku, Tokyo Japan {vanno, k-hase, ki}@cvliisu-okyoacjp Absrac For a large scale objec, scanning from he air is one of he mos efficien mehods of obaining 3D daa In he case of large culural heriage objecs, here are some difficulies in scanning hem wih respec o safey and efficiency To remedy hese problems, we have been developing a novel 3D measuremen sysem, he Floaing Laser Range Sensor FLRS), in which a rage sensor is suspended beneah a balloon The obained daa, however, have some disorion due o he inrascanning movemen In his paper, we propose a mehod o recover 3D range daa obained by a moving laser range sensor; his mehod is applicable no only o our FLRS, bu also o a general moving range sensor Using image sequences from a video camera mouned on he FLRS enables us o esimae he moion of he FLRS wihou any physical sensors such as gyros and GPS A firs, he iniial values of camera moion parameers are esimaed by perspecive facorizaion The nex sage refines camera moion parameers using he relaionships beween camera images and he range daa disorion Finally, by using he refined parameer, he disored range daa are recovered We applied his mehod o an acual scanning projec and he resuls showed he effeciveness of our mehod Index Terms Srucure from Moion, Moving Range Sensor, Floaing Laser Range Sensor, Perspecive-Facorizaion I INTRODUCTION Nowadays, many researches on real objec modeling are making grea progress because of he availabiliy of accurae geomeric daa from hree dimensional laser sensors The echniques of real objec modeling conribue oward numerous applicaions in wide areas such as academic invesigaion, indusrial managemen, and enerainmen Among hem, one of he mos imporan and comprehensive applicaions is modeling culural heriage objecs Modeling hese heriage objecs has grea significance in many aspecs Modeling hem leads o digial archives of he objec shapes Uilizing hese daa enables us o resore he heriage objecs original shapes, even if he objecs have been desroyed due o naural weahering, fire, disasers and wars In addiion, we can provide images of hese objecs hrough he Inerne o people in heir homes or in heir offices Thus, he echniques of real objec modeling are available for many applicaions We have been conducing some projecs o model large scale culural heriage objecs such as grea Buddhas, hisorical buildings and suburban landscapes Basically, o scan hese large objecs, a laser range finder is usually used wih a ripod posiioned on sable locaions In he case of scanning a large scale objec, however, i ofen occurs ha some par of he objec is no visible from he laser range sensor on he ground In spie of such a difficuly, we have scanned large objecs from scaffolds emporally consruced nearby he objec However, his scaffold mehod requires cosly, edious consrucion ime In addiion, i may be impossible o scan some pars of he objec due o he limiaion of available space for scaffold-building We are now conducing a projec o model he Bayon Temple in Cambodia [7]; he emple s scale is abou 100 x 100 square meers Scanning such a huge scale objec from several scaffolds is unrealisic To overcome his problem, several mehods have been proposed For example, aerial 3D measuremens can be obained by using a laser range sensor insalled on a helicoper plaform [12] High frequency vibraion of he plaform, however, should be considered o ensure ha we obain highly accurae resuls To avoid irrevocable desrucion, he use of heavy equipmen such as a crane should be eschewed when scanning a culural heriage objec Based upon he above consideraions, we proposed a novel 3D measuremen sysem, a Floaing Laser Range Sensor FLRS) [4] This sysem digiizes large scale objecs from he air while suspended from he underside of a balloon plaform Fig1) Our balloon plaform is cerainly free from high frequency vibraion such as ha of a helicoper engine The obained range daa are, however, disored because he laser range sensor iself is moving during he scanning processes Fig 1 Floaing Laser Range Sensor and Bayon emple

2 In his paper, we propose a mehod o recover 3D range daa obained by a moving laser range sensor No only is his mehod limied o he case of our FLRS, bu i is also applicable o a general moving range sensor We obain disored range daa by using he range sensor, and obain image sequences simulaneously by using he video camera mouned on he FLRS Then he moion of he FLRS is esimaed by he obained images and disored daa wihou any physical sensors such as gyros and GPS We esimae he camera moion parameers imposing some consrains, which include informaion derived from he disored range daa iself In order o solve he non-linear opimizaion problem, we uilize a perspecive facorizaion mehod as he iniial value o avoid local minimums Then, using he refined camera moion parameer, he disored range daa are recovered This paper organized as follows In Secion II, we briefly explain he perspecive facorizaion, which is uilized as he iniial value for he camera moion In Secion III, we describe our proposed algorihm for refinemen of he parameers In Secion IV, we describe how we use his algorihm o model and recover he shape of he Bayon Temple in Cambodia To evaluae our mehod, he recovered shapes are compared wih oher daa obained by a range sensor on he ground Finally, we presen our conclusions and summarize our possible fuure works Using he horizonal coordinaes u fp, we can define an F P marix U Each column of he marix conains he horizonal coordinaes of a single poin in he frame order, while each row conains he horizonal coordinaes for a single frame Similarly, we can define an F P marix V from he verical coordinaes v fp The combined marix of 2F P becomes he measuremen marix as follow ) U W = 1) V Each frame f is aken a camera posiion f in he world coordinaes The camera pose is described by he orhonormal uni vecors i f, j f and k f The vecors i f and j f correspond o he x and y axes of he camera coordinaes, while he vecor k f corresponds o he z axis along he direcion perpendicular o he image plane Fig2) II PERSPECTIVE FACTORIZATION Esimaions of he shape of an objec or of camera moion by using images are called Shape from Moion or Shape from Srucure, and are main research fields in compuer vision The facorizaion mehod proposed by Tomasi and Kanade [13] is one of he mos effecive algorihms for simulaneously recovering he shape of an objec and he moion of he camera from an image sequence This mehod was originally limied o he orhographic model Then, however, he facorizaion was exended o several perspecive approximaions and applicaions [2][8][9][3] Han and Kanade [3] proposed a facorizaion mehod wih a perspecive camera model Using he weak-perspecive projecion model, hey ieraively esimaed he shape and he camera moion under he perspecive model Firs, we briefly explain weak-perspecive facorizaion, which is subsequenly exended o he perspecive facorizaion The soluion by he perspecive facorizaion is uilized as he iniial value for he opimizing problem described in Secion III A Weak-Perspecive Facorizaion Given a sequence of F images, in which we have racked P ineres poins over all frames, each ineres poin p corresponds o a single poin s p on he objec In image coordinaes, he rajecories of each ineres poin are denoed as {u fp,v fp ) f =1,, F, p =1,, P 2F P } Fig 2 he coordinae sysem: f denoes he posiion of he camera a ime of frame f The camera pose is deermined by hree uni basis vecors Under he weak-perspecive camera model, a single poin in he world coordinaes s p is projeced ono he image plane fasu fp,v fp ) u fp = f z f i f s p f ) 2) v fp = f z f j f s p f ) 3) z f = k f c f ) 4) The vecor c is he cener of mass of all ineres poins Wihou loss of generaliy, he origin of he world coordinaes can be placed a he cenroid, ha is c =0 Then his means ha z f = k f f o simplify he expansion of he following formulaion To summarize, u fp = m f s p + x f 5) v fp = n f s p + y f 6) where m f = f z f i f, x f = f z f i f f 7) n f = f z f j f, y f = f z f j f f 8)

3 To be expressed his equaion in a marix form: u 11 u 1P m 1 u 21 u 2P m 2 u F 1 u FP = v 11 v 1P m F s 1 s P ) n 1 v F 1 v FP n F x 1 x 2 + x F 1 1) 9) y1 y F Using ha he cener of all ineres poins is he origin, P P P u fp = m f s p + x f = P x f 10) similarly, P v fp = P y f 11) We obain he regisered measuremen marix W, afer ranslaion W = W x 1 x 2 x F y 1 y F ) 1 1 1) as a produc of wo marixes M and S W = M S 12) where M is a 2F 3 Marix and S is a 3 P Marix The above decomposiion, however, is no unique because any inverible 3 3 marix A makes a valid decomposiion of Ŵ as MA)A 1 S)=MAA 1 )S = MS = W 13) To ge rid of his ambiguiy, using he fac ha he marix M represens he axes of he camera coordinaes, he following consrains should be saisfied m f = n f, m f n f =0 14) These consrains give us he moion marix M and he shape marix S B Exension o Full-Perspecive Facorizaion The above formulaion is under he weak perspecive camera model, which is a linear approximaion of he perspecive model Nex, using an ieraive framework, we obain approximae soluions under he non-linear perspecive camera model Under he perspecive camera model, he projecive equaion beween he objec poin s p in 3D world and he image coordinae u fp,v fp ) is wrien as i u fp = f f s p f ) 15) v fp = f j f s p f ) 16) displacing z f = k f f, we obain he following equaion λ fp +1)u fp = f z f i f s p f ) 17) λ fp +1)v fp = f z f j f s p f ) 18) k where λ fp = f s p 19) z f Noe ha he righ hand sides are he same form under he weak-perspecive model see he Eq 2 and 3) This means, muliplying a image coordinae u fp,v fp ) by a real number λ fp changes coordinaes under he perspecive model ino coordinaes under he weak-perspecive model Solving he value of λ fp ieraively, we can obain moion parameers and coordinaes of ineres poins under he perspecive model in he framework of weak-perspecive facorizaion The enire algorihm of he perspecive facorizaion is as follows: Inpu: An image sequence of F frames racking P ineres poins Oupu:The posiions of P ineres poins s p The camera posiion f and poses i f, j f, kf a each frame f 1) giving λ fp =0 2) supposing he equaions 17 and 18, solve s p, f, i f, j f, k f and z f using he weak perspecive facorizaion 3) calculae λ fp by he equaion 19 4) subsiuing λ fp ino sep2; repea he above procedure unil λ fp s are close o he previous ieraion III REFINEMENT OF CAMERA MOTION Wihou noise in inpu, he above facorizaion mehod leads o he fine soluion As a resul, recovered 3D shape hrough he esimaed camera parameers is valid Real images, however, conain a bi of noise Therefore, i is no sufficien o recover range daa obained by he FLRS only hrough he facorizaion For he sake of more refined esimaion of camera parameers, we impose hree consrains racking, movemen, and range daa Refined camera moion can be found hrough he minimizaion of a global funcional To minimize he funcion, he soluion by he perspecive facorizaion is uilized as he iniial value o avoid local minimums A Consrain A Tracking consrain) As he mos fundamenal consrain, any ineres poin s p mus be projeced on each image plane a he coordinaes u fp,v fp ) This consrain conducs he following funcion: F P i F A = u fp f f s p f ) ) 2 f=1 j + v fp f ) f s p f ) ) 2 20) The minimizaion of F A leads o he correc racking of fixed ineres poins by a moving camera However, we can

4 see ha he presence of parameers we are rying o esimae in he denominaor makes his equaion a difficul one Then, suppose ha insead, we consider he following funcion: F P ) 2 F A = kf s p f )u fp f i f s p f ) f=1 ) ) 2 + kf s p f )v fp f j f s p f ) 21) B Consrain B Movemen consrain) One of he mos significan reasons for adoping a balloon plaform is o be free from high frequency vibraion differenly from a helicoper plaform In oher words, a balloon plaform is only under he influence of low frequency The balloon of our FLRS is held wih some wires swayed only by wind This means ha he movemen of he balloon is expeced o be smooh Cerainly, he movemen of he balloon is free from rapid acceleraion, rapid deceleraion or acue course changing Taking his fac ino accoun, we consider following funcion: 2 ) 2 f 2 q ) ) 2 f F B = w 1 + w2 2 2 d 22) Here, f denoes he posiion of he camera; is ime; w 1,w 2 are weighed coefficiens; and q f is a uni quaernion which represens he roaion of camera pose The presenaion by quaernion is obained immediaely by i f, j f and k f The firs erm of he above inegrand represens smoohness wih respec o he camera s ranslaion while he second represens smoohness wih respec o he camera s roaion When he moion of he camera is smooh, he funcion F B akes small value C Consrain C Range Daa consrain) Taking a broad view of range daa obained by he FLRS, he daa are disored by he swing of he camera We can find, however, ha hese daa conain precise informaion locally; ha informaion is uilized for refinemen of he camera moion The laser range sensor re-radiaes laser beams in raser scan order This leads ha we can insanly obain he ime when any pixel in he range image is scanned If he video camera coincides wih he range sensor, we can find he jus frame among he sequence when he pixel is scanned Wih he video camera calibraed wih he range sensor, we can obain he image coordinae of each ineres poin in he 3D world, in addiion o he disance from he sensor o i once and for all This consrain is very imporan since knowing he disances eliminaes ambiguiy wih respec o scaling The echniques of Shape from Moion or Srucure from Moion only by images remain ambiguous abou scaling On he oher hand, our mehod is able o derive some precise informaion abou scale even from he disored daa and remove he ambiguiy Then, we can conduc he hird consrain o be minimized as follows: P F C = xfp s p fp ) 2 23) Here, he index fp denoes he frame number when he range sensor scans he ineres poin p, and he measured disance by he range sensor a his momen denoes x fp As x fp = x fp,y fp,z fp ), he above funcion can be rewrien o saisfy more rigorous consrain: P xfp F C = i fp s p fp ) ) 2 + y fp j fp s p fp ) ) 2 + zfp k fp s p fp ) ) 2 ) 24) D Global consrain The weighed sum F = w A F A + w B F B + w C F C 25) leads o a global funcion To minimize his funcion, we employ he Flecher-Reeves mehod and he Polak-Ribiere mehod [10][5][11], which are ypes of he conjugae gradien mehod Then, we use he golden secion search o deermine he magniude of gradien direcions As menioned in he previous secions, we inpu he soluion by he perspecive facorizaion as he iniial value Minimizing he funcion F is basically quie difficul because his funcion has many local minimums By employing he soluion of he facorizaion as a fairly good approximaion, we ry o avoid hem IV EXPERIMENTS ON THE BAYON TEMPLE A Tracking In our sysem, we obain 72 frames a a single scanning process For ineres poin racking, we use he SIFT key [6], which is robus for scaling, ha is a movemen along he view direcion A simple window maching mehod around he poins races he rajecory of each poin hroughou all he frames Afer his procedure, we can derive abou one hundred ineres poins from a sequence of 72 frames B Evaluaion of he Recovered Shape To evaluae he accuracy of our shape recovery algorihm, we compare he recovered shape wih oher daa, which are obained by a range finder, he Cyrax2500, posiioned on he ground Aligning wo daa by using ICP algorihm [1][15], we analyze he overlapped area The resul is shown in Figure 3 The blue fine daa in boh images is a non-disored daa he correc daa) obained by he Cyrax2500 The whie coarse daa in he middle figure indicaes he original oupu daa obained by he FLRS, while he pink daa in he lower figure indicaes he daa recovered by our mehod One can easily find ha he recovered 3D shape is well-fied ono he Cyrax2500 s daa In paricular, aking noice of he area of ellipses in he middle figure, makes i obvious ha our algorihm is effecive In he upper figure, he cross secion,

5 Fig 4 The comparison beween he Cyrax 2500 s he correc daa) and he original disored daa lef), and ha beween he Cyrax 2500 s and he recovered daa: he green region indicaes where he disance of wo shapes is less han 60 cm, ha is, wo shapes coincide in his region Noe ha he green region is expanded afer he refinemen in he righ figure Fig 3 Range daa before and afer he recovery process: The middle figure illusraes he regisraion of he original daa from he FLRS whie) and he non-disored daa obained by he fixed Cyrax 2500 blue) Two shapes do no coincide, especially a he areas of ellipses in he figure On he oher hand, he recovered shape in he lower figure pink) is fied ono he correc daa The upper figure shows he cross secion on he whie doed line in he middle figure cu off on he whie doed line in he middle figure, also shows he effeciveness Anoher figure, Figure 4, shows he effeciveness of he mehod Tha figure indicaes he poin-o-poin disances beween Cyrax2500 s daa and he recovered daa The lef image shows a comparison beween he Cyrax2500 s and he original disored daa, while he righ shows a comparison beween he Cyrax2500 s and he recovered daa The region where he disances beween hem are less han 60 cm is colored greenin his sudy, we se he hreshold a 60cm) On he oher hand,he area where he disances are furher han 60cm is displayed in blue A a glance, he green region Fig 5 Anoher daa se by he FLRS: in his case, he original range daa are disored widely upper lef) The recovered shapes are shown in he upper righ and he resul of regisraion wih Cyrax 2500 s daa is shown in he lower is clearly expanded by he recovery algorihm Taking accoun of he fac he Cyrax 2500 could no measure he upper par of he emple because of occlusions, he mehod could recover he 3D shape correcly Above daa are in he case of ha he balloon s moion is raher moderae Figure 5 shows oher daa, which are obained by he balloon in wide moion One can see ha he range daa are disored widely he upper lef image in Fig5) In his case, we can recover he shape wihou inciden he upper righ) Aligning i wih he Cyrax2500 s daa, he recovered shape is well-fied wih he correc daa he lower image in Fig5) V CONCLUSIONS We have presened a mehod for esimaing camera moion In our framework, we move no only a video camera bu also a range sensor Wihou any physical sensors such as gyros and GPS, we ry o recover he shape of an objec and he camera

6 Fig 6 The overall daa of he Bayon Temple moion by using only an image sequence and he disored range daa Correc esimaion is essenial o recover he objec shape since, in our FLRS sysem, a range sensor swings during he measuremen process Using a video sequence on he FLRS and he disored range daa, we have esimaed he camera moion A firs, he perspecive facorizaion derived he iniial value for he esimaion Nex, we solved a nonlinear opimal problem under hree consrains In his process, we uilized disored daa which had been formerly sloughed Finally, by using esimaed camera moion parameers, we recovered he shapes Figure 6 shows he overall 3D daa of he Bayon Temple) This framework can be generally applied o a framework in which a range sensor moves during he scanning process, and is no limied o our FLRS We achieved high accuracy recovery of he disored range daa Our FLRS is expeced o be effecive for measuremen of large scale objecs Therefore, we have o improve he precision obained by his algorihm On he oher hand, several problems remain in his mehod For example, in he presen sysem, all racked ineres poins mus be visible hroughou a sequence The racking algorihm is so simple ha i would be raher difficul o rack poins when balloon s movemen is violen ACKNOWLEDGMENT This work was suppored in par by Minisry of Educaion, Culure, Spors, Science and Technology, under he program, Developmen of High Fideliy Digiizaion Sofware for Large-Scale and Inangible Culural Asses, Japan Science and Technology CorporaionJST) and Research Fellowships of Japan Sociey for he Promoion of ScienceJSPS) for Young Scieniss Scanning he Bayon Temple was conduced joinly wih Japanese Governmen Team for Safeguarding AngkorJSA) and JST [2] JCoseira and TKanade, A muli-body facorizaion mehod for moion analysis, Proc of ICCV, pp , 1995 [3] MHan and TKanade, Perspecive facorizaion mehods for Euclidean reconsrucion, CMU-RI-TR-99-22, 1999 [4] YHiroa, TMasuda, RKurazume, KOgawara, KHasegawa and KIkeuchi, Flying Laser Range Finder and is daa regisraion algorihm, Proc of ICRA, pp , 2004 [5] D A Jacobs, The Sae of he Ar in Numerical Analysis, London: Academic Press, 1977 [6] DGLowe, Disincive image feaures from scale-invarian keypoins, Inernaional Journal of Compuer Vision, Vol60, No2, pp91-110, 2004 [7] DMiyazaki, TOishi, TNishikawa, RSagawa, KNishino, TTomomasu, YYakase and KIkeuchi The grea buddha projec: Modelling culural heriage hrough observaion Proc of VSMM, pp , 2000 [8] TMoria and TKanade, A sequenial facorizaion mehod for recovering shape and moion from image sreams, IEEE Trans on PAMI, vol19, No8, pp , 1997 [9] CPoelmann and TKanade A paraperspecive facorizaion mehod for shape and moion recovery, IEEE Trans on PAMI, vol19, No3, pp , 1997 [10] E Polak, Compuaional Mehods in Opimizaion, New York: Academic Press, 1971 [11] J Soer and RBulirsh, Inroducion o Numerical Analysis, New York: Springer-Verlag, 1980 [12] SThrun, MDiel and DHaehnel, Scan alignmen and 3-D surface modeling wih a helicoper plaform, The 4h Inernaional Conference on Field and Service Roboics, 2003 [13] CTomasi and TKanade, Shape and moion from image sreams under orhography: a facorizaion mehod, Inernaional Journal of Compuer Vision, Vol 9, No2, pp , 1992 [14] JVisnovcova, LZhang and AGruen, Generaing a 3D model of a bayon ower using non-meric imagery, Proc of he Inernaional Workshop Recreaing he Pas -Visualizaion and Animaion of Culural Heriage, 2001 [15] ZZhang, Ieraive poin maching for regisraion of free-form curves and surfaces, Inernaional Journal of Compuer Vision, Vol13, pp , 1994 REFERENCES [1] PJBesl and NDMcKay, A mehod for regisraion of 3-D shapes, IEEE Trans on PAMI, vol14, pp , 1992