A Flexible Technique for Accurate Omnidirectional Camera Calibration and Structure from Motion

Transcription

1 A Flexble Technque for Accurate Omndrectonal Camera Calbraton and Structure from Moton Dade Scaramuzza, Agostno Martnell, Roland Segwart Swss Federal Insttute of Technology Lausanne (EPFL) CH-5 Lausanne, Swtzerland {dade.scaramuzza, agostno.martnell, Abstract In ths paper, we present a flexble new technque for sngle ewpont omndrectonal camera calbraton. The proposed method only requres the camera to obsere a planar pattern shown at a few dfferent orentatons. Ether the camera or the planar pattern can be freely moed. o a pror knowledge of the moton s requred, nor a specfc model of the omndrectonal sensor. The only assumpton s that the mage proecton functon can be descrbed by a Taylor seres expanson whose coeffcents are estmated by solng a two-step least-squares lnear mnmzaton problem. To test the proposed technque, we calbrated a panoramc camera hang a feld of ew greater than n the ertcal drecton, and we obtaned ery good results. To nestgate the accuracy of the calbraton, we also used the estmated omn-camera model n a structure from moton experment. We obtaned a 3D metrc reconstructon of a scene from two hghly dstorted omndrectonal mages by usng mage correspondences only. Compared wth classcal technques, whch rely on a specfc parametrc model of the omndrectonal camera, the proposed procedure s ndependent of the sensor, easy to use, and flexble.. Introducton Accurate calbraton of a son system s necessary for any computer son task requrng extractng metrc nformaton of the enronment from D mages, lke n ego-moton estmaton and structure from moton. Whle a number of methods hae been deeloped concernng planar camera calbraton [9,, ], lttle work on omndrectonal cameras has been done, and the prmary focus has been on partcular sensor types. For omndrectonal camera s usually ntended a son system prodng a 36 panoramc ew of the scene. Such an enhanced feld of ew can be acheed by ether usng catadoptrc systems, obtaned by opportunely combnng mrrors and conentonal cameras, or employng purely doptrc fsh-eye lenses [3]. As noted n [, 3, ], t s hghly desrable that such magng systems hae a sngle ewpont [4, 6]. That s, there exsts a sngle center of proecton, so that, eery pxel n the sensed mages measures the rradance of the lght passng through the same ewpont n one partcular drecton. The reason a sngle ewpont s so desrable s that t permts the generaton of geometrcally correct perspecte mages from the pctures captured by the omndrectonal camera. Moreoer, t allows applyng the known theory of eppolar geometry, whch easly permts to perform ego-moton estmaton and structure-from-moton from mage correspondences only. Preous works on omndrectonal camera calbraton can be classfed nto two dfferent categores. The frst one ncludes methods whch explot pror knowledge about the scene, such as the presence of calbraton patterns [5, 7] or plumb lnes [8]. The second group coers technques that do not use ths knowledge. Ths ncludes calbraton methods from pure rotaton [7] or planar moton of the camera [9], and self-calbraton procedures, whch are performed from pont correspondences and eppolar constrant through mnmzng an obecte functon [, ]. All mentoned technques allow obtanng accurate calbraton results, but prmarly focus on partcular sensor types (e.g. hyperbolc and parabolc mrrors or fsh-eye lenses). Moreoer, some of them requre specal settng of the scene and expense equpment [7, 9]. For nstance, n [7], a fsh-eye lens wth a 83 feld of ew s used as an omndrectonal sensor. Ths work was supported by the European proect COGIRO (the Cognte Robot Companon).

2 Here, the calbraton s performed by usng a halfcylndrcal calbraton pattern perpendcular to the camera sensor, whch rotates on a turntable. In [8, ], the authors treat the case of a parabolc mrror. In [8] t s shown that anshng ponts le on a conc secton whch encodes the entre calbraton nformaton. Thus, proectons of two sets of parallel lnes suffce for ntrnsc calbraton. Howeer, ths property does not apply to non-parabolc mrrors. Therefore, the proposed technque cannot be easly generalzed to other knds of sensors. Conersely, the methods descrbed n [,,, 4] fall n the self-calbraton category. These methods requre no calbraton pattern, nor a pror knowledge about the scene. The only assumpton s the capablty to automatcally fnd pont correspondences n a set of panoramc mages of the same scene. Then, calbraton s drectly performed by eppolar geometry by mnmzng an obecte functon. In [, ], ths s done by employng a parabolc mrror, whle n [, 4] a fsh-eye lens wth a ew angle greater than 8 s used. Howeer, besdes focusng on partcular sensor types, the mentoned self-calbraton technques may suffer n case of trackng dffcultes and of a small number of features ponts [6]. All preous calbraton procedures focus on partcular sensor types, such as parabolc and hyperbolc mrrors or fsh-eye lenses. Furthermore, they are strongly dependent on the omndrectonal sensor model they use, whch s sutable only when the sngle effecte ewpont property s satsfed. Although seeral panoramc son systems exst drectly manufactured to hae ths property, for a catadoptrc system ths requres to accurately algn the camera and the mrror axes. In addton, the focus pont of the mrror has to concde wth the camera optcal center. Snce t s ery dffcult to aod camera-mrror msalgnments, an ncorrectly algned catadoptrc sensor can lead to a quas sngle-ewpont optcal system []. As a result, the sensor model used by the mentoned technques could be suboptmal. In the case of fsh-eye lenses the dscusson aboe s analogue. Motated by ths obseraton, we propose a calbraton procedure whch uses a generalzed parametrc model of the sensor, whch s sutable to dfferent knds of omndrectonal son systems, both catadoptrc and doptrc. The proposed method requres the camera to obsere a planar pattern shown at a few dfferent locatons. Ether the camera or the planar pattern can be freely moed. o a-pror knowledge of the moton s requred, nor a specfc model of the omndrectonal sensor. The deeloped procedure s based on the assumpton that the crcular external boundary of the mrror or of the fsh-eye lens (respectely n the catadoptrc and doptrc case) s sble n the mage. Moreoer, we assume that the mage formaton functon, whch manages the proecton of a 3D real pont onto a pxel of the mage plane, can be descrbed by a Taylor seres expanson. The expanson coeffcents, whch consttute our calbraton parameters, are estmated by solng a two-step least-squares lnear mnmzaton problem. Fnally, the order of the seres s determned by mnmzng the reproecton error of the calbraton ponts. The proposed procedure does not requre any expense equpment. Moreoer, t s ery fast and completely automatc, as the user s only requested to collect a few mages of the calbraton pattern. The method was appled to a AIDA 36 One VR sngle-ewpont mrror mounted on a CCD camera. The system has a ertcal ew angle greater than and the mage sze s 9x pxels. After calbraton, we obtaned an aerage reproecton error of pxel. In order to test the accuracy of the method, we used the estmated model n a structure from moton problem, and we obtaned a 3D metrc reconstructon of a scene from two hghly dstorted omndrectonal mages, by usng mage correspondences only. The structure of the paper s the followng. The omndrectonal camera model and calbraton are descrbed n Sec. and 3. The results of the calbraton of a real system are gen n Sec. 4. Fnally, the 3D structure from moton experment and ts accuracy are shown and dscussed n Sec. 5.. Omndrectonal Camera Model We want to generalze our procedure to dfferent knds of sngle-ewpont omndrectonal son systems, both catadoptrc and doptrc. In ths secton we wll use the notaton gen n []. In the general omndrectonal camera model, we dentfy two dstnct references: the camera mage plane ( u ', ' ) and the sensor plane ( u '', ' '). In Fg. the two reference planes are shown n the case of a catadoptrc system. In the doptrc case, the sgn of u would be reersed because of the absence of a reflecte surface. All coordnates wll be expressed n the coordnate system placed n O, wth the z axs algned wth the sensor axs (see Fg. a). T Let X be a scene pont. Then, assume u'' [ u' ', ' '] be the proecton of X onto the sensor plane, and T u' [ u', ' ] ts mage n the camera plane (Fg. b and c). As obsered n [], the two systems are related by an affne transformaton, whch ncorporates the dgtzng process and small axes msalgnments; thus x x u' ' Au' t, where A and t. Then, let us

3 ntroduce the mage proecton functon g, whch captures the relatonshp between a pont u' ', n the sensor plane, and the ector p emanatng from the ewpont O to a scene pont X (see Fg. a). By dong so, the complete model of an omndrectonal camera s u'' gau' t PX, p g, () 4 where X s expressed n homogeneous coordnates; P s the perspecte proecton matrx. By 3x4 calbraton of the omndrectonal camera we mean the estmaton of the matrces A and t, and the non-lnear functon g, so that all ectors g Au' t satsfy the proecton equaton (). Ths means that, once the omndrectonal camera s calbrated, we are able to reconstruct, from each pxel, the drecton of the correspondng scene pont n the real world. We assume for g the followng expresson u'','' u'','', f u'','' g, () where f s rotatonally symmetrc wth respect to the sensor axs. For nstance, n the catadoptrc case, ths corresponds to assume that the mrror s perfectly symmetrc wth respect to ts axs. In general, such an assumpton s hghly reasonable because both mrror profles and fsh-eye lenses are manufactured wth mcrometrc precson. (a) (b) (c) Fgure. (a) Coordnate system n the catadoptrc case. (b) Sensor plane, n metrc coordnates. (c) Camera mage plane, expressed n pxel coordnates. (b) and (c) are related by an affne transformaton. Functon f can hae arous forms related to the mrror or the lens constructon [, 3, 4]. As mentoned n the ntroducton, we want to apply a generalzed parametrc model of f, whch s sutable to dfferent knds of sensors. Moreoer, we want ths model to compensate for any msalgnment between the focus pont of the mrror (or the fsh-eye lens) and the camera optcal center. We propose the followng polynomal form for f T f u'','',,,,,, a a a... a, (3) where the coeffcents a,,,,..., and the polynomal degree are the model parameters to be determned by the calbraton; s the metrc ds-,, tance from the sensor axs. Thus, () can be rewrtten as u' ' ' ' gau' t w' ' 3. Camera Calbraton A f u' t u ' ', ' ' P X, (4). By calbraton of an omndrectonal camera we mean the estmaton of the parameters [A, t, a, a, a,..., a ] so that all ectors g Au' t satsfy the equaton (4). In order to reduce the number of parameters to be estmated, we compute the matrces A and t, up to a scale factor, by transformng the ew feld ellpse (see Fg. c) nto a crcle centered on the ellpse center. Ths transformaton s calculated automatcally by usng an ellpse detector f the crcular external boundary of the sensor s sble n the mage. After performng the affne transformaton, an mage pont u' s related to the correspondng pont on the sensor plane u' ' by u' ' u'. Thus, by substtutng ths relaton n (4) and usng (3), we hae the followng proecton equaton (5) u'' u' u' '' ' g u' ' P X, '' ' w f... ' a a, where now u' and ' are the pxel coordnates of an mage pont wth respect to the crcle center, and ' s the Eucldean dstance. Also, note that the factor can be drectly ntegrated n the depth factor ; thus, only + parameters ( a, a, a,..., a ) need to be estmated. Durng the calbraton procedure, a planar pattern of known geometry s shown at dfferent unknown postons, whch are related to the sensor coordnate system by a rotaton matrx R [ r,r, r3 ] and a translaton t, called extrnsc parameters. Let I be an obsered mage of the calbraton pattern, M [ X, Y, Z ] the 3D coordnate of ts ponts n T the pattern coordnate system, and m u, ] the [

4 correspondent pxel coordnates n the mage plane. Snce we assumed the pattern to be planar, wthout loss of generalty we hae Z. Then, equaton (5) becomes (6) u p P X a... a X Y r r r t r r t 3 X Y Therefore, n order to sole for camera calbraton, the extrnsc parameters hae to be determned for each pose of the calbraton pattern. 3.. Solng for camera extrnsc parameters Before descrbng how to determne the extrnsc parameters, let us elmnate the dependence from the depth scale. Ths can be done by multplyng both sdes of equaton (6) ectorally by p p a p u... a p r X Y X r r t Y r t. (7) ow, let us focus on a partcular obseraton of the calbraton pattern. From (7), we hae that each pont p on the pattern contrbutes three homogeneous equatons ( r3x r3y t3) f ( ) ( rx ry t ) (8.) f ) ( r X r Y t ) u ( r X r Y t ) (8.) ( u ( rx ry t ) ( rx ry t) (8.3) Here X, Y and Z are known, and so are u,. Also, obsere that only (8.3) s lnear n the unknown r, r, r, r, t, t. Thus, by stackng all the unknown entres of (8.3) nto a ector, we rewrte the equaton (8.3) for L ponts of the calbraton pattern as a system of lnear equatons M H, (9) where T H [ r, r, r, r, t, t ], and X Y u X uy u M : : : : : : L X L LYL ul X L ulyl L ul A lnear estmate of H can be obtaned by mnmzng the least-squares crteron mn M H, subect to H. Ths s accomplshed by usng the SVD. The soluton of (9) s known up to a scale factor, whch can be determned unquely snce ectors r,r are orthonormal. Because of the orthonormalty, the unknown entres r3,r3 can also be computed unquely. To resume, the frst calbraton step allows fndng the extrnsc parameters r, r, r, r, r3, r3, t, t for each pose of the calbraton pattern, except for the translaton parameter t 3. Ths parameter wll be computed n the next step, whch concerns the estmaton of the mage proecton functon. 3.. Solng for camera ntrnsc parameters In the preous step, we exploted equaton (8.3) to fnd the camera extrnsc parameters. ow, we substtute the estmated alues n the equatons (8.) and (8.), and sole for the camera ntrnsc parameters a, a, a,..., a that descrbe the shape of the mage proecton functon g. At the same tme, we also compute the unknown t 3 for each pose of the calbraton pattern. As done aboe, we stack all the unknown entres of (8.) and (8.) nto a ector and rewrte the equatons as a system of lnear equatons. But now, we ncorporate all obseratons of the calbraton rg. We obtan the followng system () a A B A.. A.. : C D C.. C. u.. a : :.. : : :.. : : t3 A A.. A.. B : C C.. C.. u D t3

5 where A rx ry t, B r X r Y ), C rx ry t ( 3 3 and D u r X r X ). ( 3 3 Fnally, the least-squares soluton of the oerdetermned system s obtaned by usng the pseudonerse. Thus, the ntrnsc parameters a, a, a,..., a, whch descrbe the model, are now aalable. In order to compute the best polynomal degree, we actually start from =. Then, we ncrease by untary steps and we compute the aerage alue of the reproecton error of all calbraton ponts. The procedure stops when a mnmum error s found. 4. Expermental Results The calbraton algorthm presented n the preous sectons was tested on real data. The omndrectonal sensor to be calbrated s a catadoptrc system composed of a AIDA 36 One VR hyperbolc mrror and a SOY CCD camera hang a resoluton of 9x pxels. The calbraton rg s a checker pattern contanng 9x7 squares, so there are 48 corners (calbraton ponts) (see Fg. 4). The sze of the pattern s 4.3cm x 8.9 cm. Eleen mages of the plane under dfferent orentatons were taken, some of whch are shown n Fg.. calbraton ponts onto the mages. Then, we compute the Root of Mean Squared Dstances (RMS), n pxels, between the detected mage ponts and the reproected ones. The calculated RMS alues ersus the number of mages are plotted n Fg. 3 for dfferent polynomal degrees. ote that the error decreases when more mages are used. Moreoer, by usng a 4 th order polynomal to ft the model, we obtan the mnmum RMS alue, that s of about. pxels. A 3 rd order polynomal also prodes a smlar performance f more than four mages are taken. Conersely, by usng a nd order expanson, the RMS remans aboe pxels. Thus, for our applcatons we used a 4 th order expanson. As a result, the RMS error of all reproected calbraton ponts s. pxels. Ths alue s ery good f we consder that the mage resoluton s 9x pxels, and that corner detecton s less precse on omndrectonal mages than on conentonal perspecte pctures. In Fg. 4 you can see seeral corner ponts used to perform the calbraton, and the same ponts reproected onto the mage accordng to the ntrnsc and extrnsc parameters estmated by the calbraton. 7, 6, 5, 4, 3,,,, Fgure 3. RMS error ersus the number of mages of the pattern. The RMS alues are computed for dfferent polynomal degrees: nd order (black ), 3 rd order (blue ) and 4 th order (red ). Fgure. Some mages of the calbraton pattern taken under dfferent orentatons 4.. Performance wth respect to the number of planes and the polynomal degree Ths experment nestgates the performance of our technque wth respect to the number of mages of the planar pattern, for a gen polynomal degree. We ary the number of pctures from to, and for each set we perform the calbraton. ext, accordng to the estmated extrnsc parameters, we reproect the 3D Fgure 4. The corner ponts used for calbraton (red crosses) and the reproected ones (yellow rounds) after calbraton.

6 4.. Performance wth respect to the nose leel In ths experment, we study the robustness of our calbraton technque n case of naccuracy n detectng the calbraton ponts. At ths end, Gaussan nose wth mean and standard deaton s added to the nput calbraton ponts. We ary the nose leel from. pxels to.5 pxels. For each leel, we perform the calbraton and we compute the RMS error of the reproected ponts. The results obtaned usng a 4 th order polynomal are shown n Fg. 5. As t can be seen, the RMS alues reman under pxels., pattern n the orgnal mage (Fg. 6) appear straght after rectfcaton (Fg. 7). 5. Applcaton to Structure from Moton Our work on omndrectonal camera calbraton s motated by the use of panoramc son sensors for structure from moton and 3D reconstructon. In ths secton, we perform a 3D metrc reconstructon of a real obect from two omndrectonal mages, by usng the sensor model estmated by our calbraton procedure. In order to compare the reconstructon results wth a ground truth, we exploted a trhedral obect composed of three orthogonal checker patterns of known geometry (see Fg. 8).,9,8,7,6,5,4,3,,,,,,3,4,5,6,7,8,9,,,,3,4,5 Fgure 5. RMS error ersus the nose leel. Fgure 8. The sample trhedron used for the 3D reconstructon experment. Fgure 6. A sample mage before rectfcaton. Fgure 7. The sample mage of Fg. 6 after rectfcaton. ow the edges (hghlghted) appear straght Performance wth respect to mage rectfcaton In ths experment, we test the accuracy of the estmated sensor model by rectfyng all calbraton mages. Rectfcaton determnes a transformaton of the orgnal dstorted mage such that the new mage appears as taken by a conentonal perspecte camera. In general, s mpossble to rectfy the whole omndrectonal mage because of a ew feld larger than 8. Howeer, t s possble to perform rectfcaton on mage regons whch coer a smaller feld of ew. As a result, lnearty s presered n the rectfed mage. As you can see n Fg. 6, cured edges of a sample Two mages of the trhedron were taken by postonng our calbrated camera at two unknown dfferent locatons (see Fg. 9). Then, seeral pont matches were pcked manually from both ews of the obect and the eght pont algorthm [7] was appled. In order to obtan good reconstructon results, more than eght ponts (actually 35) were extracted. Then, the coordnates of the correspondent 3D ectors, back-proected nto the space, were normalzed accordng to the nonunform mrror resoluton. The results of the reconstructon are shown n Fg., where we used checker patches to ft the reconstructed 3D ponts (red rounds). In order to compare the results wth the ground truth, we computed the angles between the three planes fttng the reconstructed ponts. We found the followng alues: 94.6, 86.8 and Moreoer, the aerage dstances of these ponts from the ftted planes were respectely.5 cm,.75 cm and.7 cm. Fnally, snce we knew the sze of each checker to be 6. cm x 6. cm, we also calculated the dmenson of eery reconstructed checker, and we found an aerage error of.9 cm.

7 Fgure 9. Two pctures of the trhedron taken by the omndrectonal camera. The ponts used for the 3D reconstructon are ndcated by red dots. Fgure. Three rendered ews of the reconstructed trhedron. ote that the obect was reconstructed only from two hghly dstorted omndrectonal mages (as n Fg. 9). 6. Conclusons In ths paper, we presented a flexble new technque for sngle-ewpont omndrectonal camera calbraton. The proposed method only requres the camera to obsere a planar pattern shown at a few dfferent orentatons. o a-pror knowledge of the moton s requred, nor a specfc model of the omndrectonal sensor. The only assumpton s that the mage proecton functon can be descrbed by a Taylor seres expanson whose coeffcents are estmated by solng a two-step least-squares lnear mnmzaton problem. To test the proposed technque, we calbrated a panoramc camera hang a feld of ew greater than n the ertcal drecton, and we obtaned ery good results. To nestgate the accuracy of the calbraton, we also used the estmated omn-camera model n a structure from moton experment. We obtaned a 3D metrc reconstructon of a real obect from two omndrectonal mages, by usng mage correspondences only. The reconstructon results were also compared wth the ground truth. Wth respect to classcal technques, whch rely on a specfc parametrc model of the omndrectonal camera, the proposed procedure s ndependent of the sensor, easy to use and flexble. 7. References [] Baker, S. and ayar, S.. A theory of catadoptrc mage formaton. In Proceedngs of the IEEE Internatonal Conference on Computer Vson (ICCV 98), Bombay, Inda, 998, pp [] R. Swamnathan, M. D. Grossberg, and. S.. Caustcs of catadoptrc cameras. In Proceedngs of the IEEE Internatonal Conference on Computer Vson (ICCV ), Vancouer, Canada,. [3] B.Mcusk, T.Padla. Autocalbraton & 3D Reconstructon wth on-central Catadoptrc Cameras. In Proceedngs of Internatonal Conference on Computer Vson and Pattern Recognton (CVPR 4), Washngton US, 4. [4] S. Baker and S. ayar. A theory of sngle-ewpont catadoptrc mage formaton. Internatonal Journal of Computer Vson, 35(), oember 999, pp [5] C. Cauchos, E. Brassart, L. Delahoche, and T. Delhommelle. Reconstructon wth the calbrated syclop sensor. In Proceedngs of the IEEE Internatonal Conference on Intellgent Robots and Systems (IROS ), Takamatsu, Japan,, pp [6] T. Soboda, T. Padla, and V. Hlaac. Central panoramc cameras: Geometry and desgn. Research report. Czech Techncal Unersty - Center for Machne Percepton, Praha, Czech Republc, December 997. [7] H. Baksten and T. Padla. Panoramc mosacng wth a 8 feld of ew lens. In Proceedngs of the IEEE Workshop on Omndrectonal Vson,, pp [8] C. Geyer and. Danlds. Paracatadoptrc camera calbraton. PAMI, 4(5), May, pp [9] J. Gluckman and S.. ayar. Ego-moton and omndrectonal cameras. In Proceedngs of the IEEE Internatonal Conference on Computer Vson (ICCV 98), Bombay, Inda, 998, pp [] S. B. ang. Catadoptrc self-calbraton. (CVPR ),, pp. -7. [] B. Mcusk and T. Padla. Estmaton of omndrectonal camera model from eppolar geometry. In Proc. of CVPR 3, 3, pp

8 [] B.Mcusk, T.Padla. Para-catadoptrc Camera Autocalbraton from Eppolar Geometry. ACCV 4, orea, January 4. [3] J. umler and M. Bauer. Fsheye lens desgns and ther relate performance. [4] B.Mcusk, D.Martnec, T.Padla. 3D Metrc Reconstructon from Uncalbrated Omndrectonal Images. ACCV 4, orea, (January 4). [5] T. Soboda, T.Padla. Eppolar Geometry for Central Catadoptrc Cameras. IJCV, 49(), luwer, August, pp [6] S. Bougnoux. From proecte to Eucldean space under any practcal stuaton, a crtcsm of selfcalbraton. In Proceedngs of the 6th Internatonal Conference on Computer Vson, Jan. 998, pp [7] H.C. Longuet-Hggns. A computer algorthm for reconstructng a scene from two proectons. ature, Sept 98, 93: [8] R. I. Hartley. In defence of the 8-pont algorthm. In Proceedngs of the IEEE Internatonal Conference on Computer Vson (ICCV 95), 995. [9] Y. Ma, S. Soatto, J. osecka, S. Sastry, An ntaton to 3D son, from mages to geometrc models models, Sprnger Verlag, ISB [] Q.-T. Luong and O. Faugeras. Self-calbraton of a mong camera from pont correspondences and fundamental matrces. The Internatonal Journal of Computer Vson, (3), 997, pp [] Zhengyou Zhang. A Flexble ew Technque for Camera Calbraton, IEEE Transactons on Pattern Analyss and Machne Intellgence, Volume, Issue, oember, pp.: