Comprehensive Extrinsic Calibration of a Camera and a 2D Laser Scanner for a Ground Vehicle

Transcription

1 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile Hao Li, Fawzi Nashashibi o ite this version: Hao Li, Fawzi Nashashibi. Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile. [ehnial Report] R-0438, INRIA. 013, pp.4. <hal v> HAL Id: hal Submitted on 1 Jun 014 HAL is a multi-disiplinary open aess arhive for the deposit and dissemination of sientifi researh douments, whether they are published or not. he douments may ome from teahing and researh institutions in Frane or abroad, or from publi or private researh enters. L arhive ouverte pluridisiplinaire HAL, est destinée au dépôt et à la diffusion de douments sientifiques de niveau reherhe, publiés ou non, émanant des établissements d enseignement et de reherhe français ou étrangers, des laboratoires publis ou privés.

2 Lane Detetion (Part I): Mono-Vision Based Method 1 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile Hao LI, Fawzi NASHASHIBI N 438 July 013 R n o 438 Projet-eam IMARA ISSN ISRN INRIA/R FR+ENG

3

4 Lane Detetion (Part I): Mono-Vision Based Method 3 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile Hao LI *, Fawzi NASHASHIBI Projet-eam IMARA ehnial Report N o 438 July pages Abstrat: Cameras and laser sanners are two important kinds of pereptive sensors and both beome more and more ommonly used for intelligent ground vehiles; the alibration of these sensors is a fundamental task. A new method is proposed to perform COMPREHENSIVE extrinsi alibration of a SINGLE amera-d laser sanner pair, i.e. the proess of revealing ALL the spatial relationships among the amera oordinates system, the laser sanner oordinates system, the ground oordinates system, and the vehile oordinates system. he proposed method is mainly based on the onvenient and widely used hessboard alibration pratie and an be onveniently implemented. he proposed method has been tested on both syntheti data and real data based experiments, whih validate the effetiveness of the proposed method. Key-Words: Calibration, amera, D laser sanner, vehile, mobile robot * Hao LI: Imara, INRIA Paris-Roquenourt, hao.li@inria.fr Fawzi NASHASHIBI: Imara, INRIA Paris-Roquenourt, fawzi.nashashibi@inria.fr R n o 438

5 Calibration Extrinsèque Compréhensive d une Caméra et un Sanner Laser D pour un Véhiule errestre Résumé : La améra et le sanner laser sont deux types importants de apteurs pereptifs et tous les deux deviennent de plus en plus ommuns pour de nombreuses appliations des véhiules intelligents. La alibration de es apteurs est une tâhe fondamentale. Dans e rapport, on a propose une nouvelle méthode pour réaliser la alibration extrinsèque ompréhensive d une seule paire améra-sanner laser D, à savoir le proédé de révéler tous les relations spatiales parmi un système de oordonnées améra, un système de oordonnées sanner laser, un système de oordonnées terrestre, et un système de oordonnées véhiule. La méthode proposée se fonde prinipalement sur la pratique de abliration au damier et est faile à mettre en œuvre. Des tests des données réelles et des données synthétiques ont validé la performane de la méthode proposée. Mots-Clés : Calibration, améra, sanner laser D, véhiule, robot mobile

6 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile 5 1 Introdution Cameras and laser sanners are two important kinds of pereptive sensors and both beome more and more ommonly used for ground intelligent vehile appliations (or ground mobile robot appliations). Given a vehile equipped with a amera and a D laser sanner, it is sometimes needed to relate the data of one sensor to those of the other [1] [] [3], to relate the sensor data to the ground plane [4], and to relate the sensor data to the vehile [5] [6]. All these requirements onern a fundamental task of COMPREHENSIVE extrinsi alibration of a amera and a D laser sanner, i.e. the proess of revealing ALL the spatial relationships among the amera oordinates system, the laser sanner oordinates system, the ground oordinates system, and the vehile oordinates system. here are two kinds of needs for the alibration of these sensors: one kind is from manufaturers who fabriate the vehile platforms; the other kind is from researhers who use the vehile platforms in ad ho ways. he manufaturers normally possess speial advaned equipments whih an alibrate the installed sensors aording to strit manufaturing standards; the installation of the sensors alibrated in this way is not intended to be hanged after the alibration. On the other hand, the researhers might oasionally adjust the sensor installation for ertain ad ho tasks and thus need to re-alibrate the extrinsi parameters of the sensors. However, the researhers usually do not have the speial alibration equipments as the manufaturers do. herefore in this report, we only address alibration methods that are intended to satisfy the needs of the researhers. Comparatively more published works deal with the extrinsi alibration of MULIPLE amera-d laser sanner pairs [7] [8] [9] [10] [11], i.e. the alibration onerning multiple ameras (inluding stereo-amera), or multiple D laser sanners (inluding 3D laser sanner), or onerning both. In ontrast, the extrinsi alibration of SINGLE amera-d laser sanner pair only onerns one amera and one D laser sanner. Compared with the alibration of multiple amera-d laser sanner pairs, the alibration of single amera-d laser sanner pair is more diffiult, beause less geometri onstraints an be exploited to reover the extrinsi parameters. Besides, the alibration of single amera-d laser sanner pair is more general and basi: it an be diretly adapted for the alibration of multiple amera-d laser sanner pairs, whereas the onverse an not hold. Conerning the extrinsi alibration of single amera-d laser sanner pair, the number of published works is small; Zhang & Pless method [1] is the most widely used method, thanks to its onveniene and its generality: this method is based on the R n o 438

7 6 Hao LI, Fawzi NASHASHIBI onvenient hessboard alibration pratie that has almost beome a standard routine for amera intrinsi alibration sine the introdution of this pratie by Zhang Z. [13]; it does not require omplex alibration onditions, suh as IMU devies [7] or ompliated alibration boards [14]. Zhang & Pless method an be applied to the alibration of any general amera-d laser sanner pair, unlike some methods that work only for ertain speial kind of laser sanners (suh as visible laser sanner [19]). Sine this report also handles the extrinsi alibration of single amera-d laser sanner pair and intends providing a onvenient and general solution, Zhang & Pless method [1] serves as a proper referene method for our presented works. A new alibration method whih aims at performing omprehensive extrinsi alibration of a amera and a D laser sanner is proposed. he ontribution of the proposed method mainly lies in the following aspets: 1) he proposed method an reveal ALL the spatial relationships among the amera oordinates system, the laser sanner oordinates system, the ground oordinates system, and the vehile oordinates system, based on the hessboard alibration pratie with few extra measurements. ) he proposed method yields two improvements over the referene method in [1], even based exatly on the same hessboard alibration pratie. First, the proposed method an reveal more spatial relationships than the method in [1] does. More speifially, the method in [1] only reveals the spatial relationship between a amera and a D laser sanner, whereas the proposed method an not only reveal this spatial relationship but also that between the two sensors and the ground plane. Seond, the proposed method outperforms the method in [1] in terms of alibration auray, even only onerning the extrinsi alibration between the amera and the D laser sanner. Mathematial Fundaments and Denotations Several oordinates systems are relevant in the presentation of the proposed method: the amera oordinates system (CCS), the laser sanner oordinates system (SCS), the ground oordinates system (GCS), the vehile oordinates system (VCS), and the hessboard plane oordinates system (PCS). he origin and the oordinate axes of the CCS are denoted by {O,X,Y,Z }, where the O -X -Y plane is parallel to the image plane. he origin and the oordinate axes of the SCS are denoted by {O s,x s,y s,z s }, where the plane Z s =0 is the sanning plane of the D laser sanner. Let the vehile be stationary on the ground plane, the GCS and VCS are established as follows: the origin and the oordinate axes of the VCS are denoted by {O v,x v,y v,z v }, where the {X v,y v,z v } are respetively along the longitudinal diretion, the lateral Inria

8 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile 7 diretion, and the vertial diretion of the vehile; the O v is at the ground projetion of the rear wheel axle enter. he origin and the oordinate axes of the GCS are denoted by {O g,x g,y g,z g }, where the O g is at the ground projetion of the O, the Z g points from the O g to the O ; the X g is along the ground projetion of the Z. Given a pose of the hessboard plane, the origin and the oordinate axes of the PCS are denoted by {O p,x p,y p,z p }, where the plane Z p =0 is situated on the hessboard plane, the O p is at the hessboard left-bottom orner, the X p is along the hessboard bottom edge, and the Y p is along the hessboard left edge. he hessboard is plaed with several different poses in the pereption field of the amera and the D laser sanner; for eah pose, a sub-sript ( is used to distinguish the PCS. hus the different hessboard poses that are used for alibration are denoted by a set of PCS (, i.e. PCS (1) {O 1),X 1),Y 1),Z 1) }, PCS () {O ),X ),Y ),Z ) }, An illustration of these oordinates systems is given in Fig.1. Fig.1. he oordinates systems: CCS, SCS, GCS, VCS, and PCS ( It is worthy noting that these oordinates systems might be established differently; they are established in above way mainly for alibration onveniene and possible appliations assoiated with ground vehiles (or ground mobile robots). In this report, the {X a,y a,z a } also denote the unit vetors along orresponding oordinate axes. he R and generally denote 3 3 rotation matrix and 3 1 translation vetor respetively. he R ab and ab denote the rotation and translation R n o 438

9 8 Hao LI, Fawzi NASHASHIBI from the oordinates system {O a,x a,y a,z a } to the oordinates system {O b,x b,y b,z b }. For example, R s and s denote the transformation from the CCS to the SCS. he M denotes a point and M a =[x a,y a,z a ] denotes the oordinates of M in {O a,x a,y a,z a }. he following relationships always hold (a,b,f={, s, g, v, 1), ), }): M = R M + b ab a ab Dual relationship: Chain relationship: R ba ab ab ba ab = R ; = R (1a) fb af ab fb ab R R R = R + = ; (1b) af fb In the CCS, the N is used to denote the perpendiular vetor from the O to the plane PCS (. he N g is used to denote the perpendiular vetor from the O to the ground plane. Let N G =[N g,n 0 ] and let the ground plane be represented by equation N G [M,1] =0. A List of notations is summarized as follows: {O,X,Y,Z } Camera Coordinates System (CCS) {O s,x s,y s,z s } Laser Sanner Coordinates System (SCS) {O v,x v,y v,z v } Vehile Coordinates System (VCS) {O g,x g,y g,z g } Ground Coordinates System (GCS) R and Rotation and ranslation (general) R ab R from {O a,x a,y a,z a } to {O b,x b,y b,z b } ab from {O a,x a,y a,z a } to {O b,x b,y b,z b } M A point (general) M a =[x a,y a,z a ] A point M in {O a,x a,y a,z a }. N (In CCS) the perpendiular vetor from O to the plane PCS (. N g (In CCS) the perpendiular vetor from O to the ground plane. N G =[N g,n 0 ] Given a generi point M on the ground plane, then N G [M,1] =0. e 1, e, and e 3 [1,0,0], [0,1,0], and [0,0,1]. L-norm. Given an arbitrary vetor V, V =V V. Inria

10 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile 9 3 he Basi Version of the Comprehensive Extrinsi Calibration Method he whole alibration method onsists of three parts: 1) the method [1] to perform the alibration between the CCS and the SCS (briefly reviewed), based on the hessboard alibration pratie; ) a proposed method to perform the alibration between the CCS and the GCS, based on the same hessboard alibration pratie; 3) a proposed method to perform the alibration between the GCS and the VCS, with the help of few extra measurements in addition to the hessboard alibration pratie. 3.1 he Calibration Between the CCS and the SCS he amera intrinsi parameters are alibrated using the method in [13]; given several hessboard poses for alibration: PCS (1) {O 1),X 1), Y 1), Z 1) }, PCS () {O ),X ), Y ), Z ) }, et. For a pose PCS (, the N is omputed as: N = ( e3 R p ( ) R e 3 R and are omputed via the homography between the plane Z =0 and the image oordinates system [13]. Aording to the geometri onstraint that laser points should be loated on the hessboard plane, R s and s are optimized by minimizing the summed square of distanes of all the laser points to orresponding hessboard planes [1]: { R F 1 s =, i s } = arg min F j N [ N R, s s 1 R 1 s ( M s( i, j) s ) N ] () Where R s is parameterized by a 3-vetor using the Rodrigues formula [15]; M s(i,j) is the j-th laser point on the PCS (. he initial values of R s and s are estimated by solving a linear equation problem [1]. he R s and s an be omputed using the dual relationship (1a): R s =R s ; s =-R s s. 3. he Calibration Between the CCS and the GCS During the hessboard alibration pratie, one an hold the hessboard either on the ground or in the air, only if the amera and the D laser sanner an both pereive the hessboard. In pratie, it is more onvenient and more stable to hold the hessboard on the ground than in the air, as the hessboard might be large and heavy. R n o 438

11 10 Hao LI, Fawzi NASHASHIBI Besides, posing the hessboard on the ground brings one more geometri onstraint: ground plane onstraint. It means that the hessboard bottom edge, i.e. the line O +λx (λ is a salar), is situated on the ground plane. his onstraint is reasonable, beause a alibration field fairly flat ould always be found; for example, on the floor in a garage room. Let l be the length of the hessboard bottom edge; the orner points O and O +l X are hosen as ontrol points. he R and are omputed as mentioned in Setion III.A. In the CCS, the oordinates of O and X are respetively and R e 1. As the ground plane is denoted by N G [M,1] =0, a linear equation an be established: G N G = 0 (3)... p p ( + l R e G = ( he N G is the eigenvetor assoiated with the smallest eigenvalue of GG. Reall that N G =[N g,n 0 ] ; the 3-vetor N g is perpendiular to the ground plane. he R g and g, are omputed as follows: g n = N 0 g N g Ng e3 Ng e3 R ge1 = ( Ng + e3)/ N g + e N N R g e 3 = g g / g R ge = ( Rge3) ( Rge1) 3 g R g and g an be omputed using the dual relationship (1a): R g =R g ; g =-R g g. he spatial relationship between the SCS and the GCS an be omputed using the hain relationship (1b): R sg =R g R s ; sg =R g s + g. Proof: Lemma: Given a plane denoted as N p [M,1] =0, where N p =[N,n 0 ] and N is a 3- vetor; for an arbitrary point M a, the projetion of M a on this plane, denoted as M a(p), is omputed as: N Ma + n0 M a( p) = N + M N a Inria

12 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile 11 Lemma proof: As the 3-vetor N is perpendiular to the plane, the projetion M a(p) is in the form M a(p) =M a +λn where λ is a salar to-be-omputed. Substitute M a(p) =M a +λn for M in the equation N p [M,1] =0, i.e. N (M a +λn)+n 0 =0, and ompute the λ: N λ = M N a + n N 0 N Ma + n = N 0 Substitute the λ into M a(p) =M a +λn and the lemma is done. In the CCS, let the ground plane be denoted by equation N G [M,1] =0 and the N G =[N g,n 0 ]. Aording to the establishment of the GCS as speified in setion, the O g (i.e. g in the CCS) is the projetion of the O (i.e. 0 in the CCS) on the ground plane; then g an be omputed via the lemma: g g N 0 + n0 = N N g g n + 0 = N 0 g N g As the axis Z g points from O g to O, the unit vetor Z g (i.e. R g e 3 in the CCS) is omputed as: R e = / g 3 g g Selet a point on the axis Z (let it be e 3 in the CCS) and ompute its projetion on the ground plane: g N e3 + n0 P z = Ng + e N g 3 As the axis X g is along the projetion of the axis Z on the ground, the unit vetor X g (i.e. R g e 1 in the CCS) is omputed as: R g e = ( P 1 N = ( N z g )/ P z g e3 Ng e3 N )/ g + e3 N g + e3 g Ng g R n o 438

13 1 Hao LI, Fawzi NASHASHIBI Aording to the right-hand rule, the unit vetor Y g (i.e. R g e in the CCS) is omputed as: R ge = ( Rge3) ( Rge1) 3.3 he Calibration Between the GCS and the VCS he transformation between the GCS and the VCS is given by a rotation around the axis Z g and a translation along the ground plane, as follows: x y z v v v osθ = sinθ 0 sinθ osθ 0 0 x 0 y 1 z g g g t x + t y 0 (4) Given a PCS (, the O is hosen as a ground ontrol point. he oordinates of O in the GCS is omputed as: O g =R g + g. Choose some ground ontrol points O, ompute their oordinates O g =[x og(,y og( ] in the GCS, and measure their oordinates O v =[x ov(,y ov( ] in the VCS. Sine z v =z g always holds here, the third oordinate is omitted. With all the pairs of ontrol points, the objetive is to reveal the {θ,t x,t y } that satisfies (4) in the least mean squares sense. An initial value of {θ,t x,t y } an be estimated by solving the following linear equation:... x y... og( og(... y x og(... og( osθ... 0 sinθ = x 1 t x y... t y... ov( ov( (5) Afterward, an iterative refinement is arried out. At eah iteration step, the non-linear funtion osθ and sinθ are loally linearized with last estimate of θ; the inrement of θ and new {t x,t y } are omputed by solving a linear equation: Inria

14 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile x xog( = x y... og( sinθ osθ ov( ov( x x k 1 k 1 og( og( y y og( osθ sinθ og( sinθ k 1 k 1 osθ + y y k 1 k 1 og( og( sinθ osθ... Δθ 0 tx 1... t y k 1 k 1 k (6) After the {θ,t x,t y } onverge, the R gv and gv are obtained. A piee of pseudo-ode is given as follows: Initialization: ompute {θ (init),t x(init),t y(init) } using (5) Iteration: I. Linearize θ k-1 II. Compute {Δθ k,t x,t y } using (6); let θ k =θ k-1 +Δθ k By so far, all the spatial relationships among the CCS, the SCS, the GCS, and the VCS an be derived via (1). 4 he Improved Versions of the Comprehensive Extrinsi Calibration Method he basi version of the omprehensive extrinsi alibration method is introdued in the previous setion. Its performane depends on the auray of the amera intrinsi parameters whih are not preisely known in pratie. Conerning the alibration of the {R s, s }, Zhang and Pless [1] propose a global optimization strategy whih optimizes not only the {R s, s } but also the {A, R, } (A is the amera intrinsi matrix) in a joint objetive funtion: {,, A, R, } arg min F F R = (7) s = + α s i i j N [ N k m (i,k) R 1 s m( A, R Rs, s, A, R, p ( ( M s( i, j), s ) N, M i, k) ) ] where m (i,k) and m(a,r,,m i,k) ) are respetively the extrated and projeted image oordinates of the k-th ontrol point for the PCS (. his global optimization R n o 438

15 14 Hao LI, Fawzi NASHASHIBI strategy an be inorporated into the basi version of the alibration method to refine the alibration results. herefore, an improved version of the omprehensive extrinsi alibration method is formed and is named the improved version I in this report. he global optimization strategy in [1] over-adjusts the estimates of {R s, s, A, R, } slightly to fit them to sensor data affeted by noises; it results in a set of estimates that do not well satisfy the ground plane onstraint introdued in Setion III- B. o make the global optimization strategy more reasonable, the ground plane onstraint is proposed to be taken into aount as a term in the objetive funtion, i.e. the third term of F 3 in (8) whih stands for the summed square of distanes of all the O and O +l X to the ground plane: {,, A, R,, N } arg min F F R = (8) 3 s = + α + β s i i i j N [ N k m [ (i,k) R 1 s G m( A, R,1] N G ( M + [ Rs, s, A, R, p ( s( i, j), s NG ) N, M i, k) + l e 1 ) R 3 ],1] N G he Levenberg-Marquardt method [16] is used as the optimization tehnique. he α is a salar weight whih normalizes the relative ontribution of the laser error term and the amera error term [1]. he salar weight β is set to a omparatively large value and 100 in our implementation. he initial value of N G is omputed via (3), based on the initial estimates of {A, R, } he optimization strategy (8) is inorporated into the basi version of the alibration method, thus forming another improved version of the method whih is referred to as the improved version II in this report. 5 Experimental Results 5.1 Syntheti Data ests (Simulations) he ground-truths of the CCS, the SCS, the GCS, and the VCS are set as follows in a global referene: the orientation and the position of the CCS are [.50, -.50,.00] rads and [1.0, 0.0, 1.] meters; the orientation and the position of the SCS are [-0.01, 0.03, 0.00] rads and [.0, 0.0, 0.5] meters; the orientation and the position of the VCS are [0, 0, 0] rads and [0, 0, 0] meters. With these ground-truths, the ground-truths of the GCS pose and the ground-truths of the spatial relationships among these oordinates systems an be derived. Inria

16 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile 15 he amera is onfigured aording to an ideal pinhole model, with foal saling fator 750 and prinipal point (384, 88). he hessboard pattern onsists of squares of 100mm 100mm size; the position of the squares is well registered in the PCS. he hessboard poses are generated randomly while satisfying two onditions: first, the hessboard bottom edge is on the ground plane; seond, the hessboard an be pereived by both the amera and the D laser sanner. he hessboard orientation angle variation θ represents the angle between the hessboard plane and the image plane. Gaussian noise with mean 0 and standard deviation of 1.0 pixel is added to the projeted image points. he laser points are omputed based on the pose of the D laser sanner and the hessboard; they are ontaminated by uniform noise within ±5m, whih fairly represents the error distribution of the real laser sanner in our tests. In the experiments, the errors between the alibration results and the ground-truths are omputed. First, the influene of the number of hessboard poses, of the hessboard orientation angle variation θ, and of the number of ground ontrol points on the performane of the basi version of the alibration method will be examined. As the spatial relationships among the CCS, the SCS, and the GCS an be revealed based on the hessboard alibration pratie only, the tests examine how the number of hessboard poses and the hessboard orientation angle variation θ influene the alibrated spatial relationships among the CCS, the SCS, and the GCS. As ground ontrol points are neessary for revealing those spatial relationships assoiated with the VCS, the tests examine how the number of ground ontrol points influenes the alibrated spatial relationships assoiated with the VCS. Seond, a performane omparison among the basi version, the improved version I, and the improved version II of the omprehensive extrinsi alibration method will be presented. Performane w.r.t. the number of hessboard poses. he influene of the number of hessboard poses on the alibrated R g and g (CCS- GCS) and the alibrated R sg and sg (SCS-GCS) is demonstrated. he poses number is varied from 5 to 16. For eah poses number, 50 independent trials with θ=60 o are arried out; the RMS (root mean square) of the alibration errors of the 50 trials is omputed and is shown in Fig.. On the whole, the errors derease as the number of hessboard poses inreases. R n o 438

17 16 Hao LI, Fawzi NASHASHIBI 0.16 Camera-Ground 0.35 Camera-Ground Orientation error (deg) Position error (m) Number of poses Number of poses 0.7 Laser Sanner-Ground Laser Sanner-Ground Orientation error (deg) Position error (m) Number of poses Number of poses Fig.. he influene of the number of hessboard poses Performane w.r.t. the hessboard orientation angle variation he influene of the hessboard orientation angle variation θ on the alibrated R g and g (CCS-GCS) and the alibrated R sg and sg (SCS-GCS) is demonstrated. he θ is varied from 0 o to 60 o every o. For eah θ, 50 independent trials are arried out. In eah trial, 10 hessboard poses are randomly generated. he RMS of the alibration errors of the 50 trials is omputed and is shown in Fig.3. On the whole, the alibration results improve as the θ inreases until 50 o ; afterward, the alibration results have no notieable improvement. Inria

18 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile Camera-Ground 0.35 Camera-Ground Orientation error (deg) Position error (m) Orientation angle variation Orientation angle variation 1 Laser Sanner-Ground 40 Laser Sanner-Ground Orientation error (deg) Position error (m) Orientation angle variation Orientation angle variation Fig.3. he influene of the hessboard orientation angle variation θ Performane w.r.t. the number of ground ontrol points he influene of the number of ground ontrol points on the alibrated R v and v (CCS-VCS) and the alibrated R sv and sv (SCS-VCS) is demonstrated. he number of ground ontrol points is varied from to 10. For eah of these numbers, 50 independent trials with θ=60 o are arried out. In eah trial, 10 independent and randomly generated hessboard poses are used. For eah number, the RMS of the alibration errors of the 50 trials is omputed and is shown in Fig.4; on the whole, the alibration errors derease as the number of ground ontrol points inreases until 5; afterward, the alibration results have no notieable improvement. R n o 438

19 18 Hao LI, Fawzi NASHASHIBI 0.7 Camera-Vehile Camera-Vehile Orientation error (deg) Position error (m) Number of ground ontrol points Number of ground ontrol points 0.55 Laser sanner-vehile 1.6 Laser sanner-vehile Orientation error (deg) Position error (m) Number of ground ontrol points Number of ground ontrol points Fig.4. he influene of the number of ground ontrol points Performane omparison among the different method versions his test demonstrates a performane omparison among the three versions of the omprehensive extrinsi alibration method (namely the basi version, the improved version I, and the improved version II). During the test, 00 independent trials with θ ranging from 50 o to 60 o at random onditions are arried out. In eah trial, 10 independent and randomly generated hessboard poses and 3 ground ontrol points are used. he amera foal saling fator is orrupted by Gaussian noise with mean 0 and standard deviation 10 pixels; the amera prinipal point is orrupted by Gaussian noise with mean 0 and standard deviation 5 pixels. In eah trial, the three versions are applied to the same syntheti data (After some tuning aording to the empirial rule that the α is a salar weight whih normalizes the relative ontribution of the laser error term and the amera error term [1], the α is set to for all the tests. he β is always set to 100); the alibration results are reorded respetively. After all the trials, the RMS of the alibration errors for eah version is omputed. he orientation (ori.) error is evaluated by the L-norm error of the 3-vetor assoiated with orresponding rotation matrix; the position (pos.) error is evaluated by the L-norm error of orresponding translation vetor. he improvement of the amera intrinsi matrix is evaluated by the ratio of the Frobenius Inria

20 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile 19 norm of the differene between the estimated A and the ground-truth to the Frobenius norm of the differene between the orrupted A and the ground-truth (his ratio is onstantly 1 for the basi version). he results are listed in able I. able I. he Performane Comparison Among the hree Versions Basi version Improved version I Improved version II Ori. Error R s (deg) [1] [1] Pos. Error s (m) [1].373 [1].05 Ori. Error R g (deg) Pos. Error g (m) Ori. Error R sg (deg) Pos. Error sg (m) Ori. Error R v (deg) Pos. Error v (m) Ori. Error R sv (deg) Pos. Error sv (m) A Error ratio [1] [1] 0.10 Conerning all the error terms in able I, the improved version I yields improvements over the basi version and the improved version II yields further improvements over the improved version I. he method in [1], whih an only handle the alibration between the CCS and the SCS, forms the basis of this part of alibration for the basi version and the improved version I; its outputs are marked in able I. As an be seen (error R s, s, A), even only onsidering the alibration between the CCS and the SCS, the improved version II still outperforms the method in [1]. 5. Real Data ests An IBEO laser sanner and a 1394 amera have been set up at fixed positions on a Citroen vehile platform for tests. he angular resolution of the san is 0.5 degree per measurement; the range measuring error varies within ±5m. he amera image resolution is pixels. he hessboard panel has a pattern onsisting of squares of 100mm 100mm size; the position of the squares is well registered on the hessboard. Sine the squares are regularly arranged, this registration work an be easily performed. he alibration pratie is arried out on our garage floor. Sine the ground-truth for real data is laking, we an not diretly evaluate the alibration errors of eah trial. However, we follow a methodology of experimentation similar to those in [1] and [18]; the real-data tests are as follows: R n o 438

21 0 Hao LI, Fawzi NASHASHIBI 4 images of the hessboard with different poses are taken, together with orresponding range readings, i.e. totally 4 alibration frames. In eah trial, only 10 alibration frames are randomly seleted and the three versions of the alibration method are applied to the same seleted 10 alibration frames. We have arried out 00 independent trials. For eah method version, we an not diretly ompute the RMS error as shown in able I; instead, we ompute the variane of the alibration results of the 00 trials Despite that the trials variane is not stritly equivalent to their true RMS error; however, sine the number of trials is large, the variane of suh large amounts of trials an fairly reflet the error level of the alibration method and enables a reasonable omparison among the three method versions he results are listed in able II. able II. he varianes of the hree Versions Basi version Improved version I Improved version II Ori. Var R s (deg).506 [1] [1] Pos. Var s (m) [1] 4.5 [1] Ori. Var R g (deg) Pos. Var g (m) Ori. Var R sg (deg) Pos. Var sg (m) Ori. Var R v (deg) Pos. Var v (m) Ori. Var R sv (deg) Pos. Var sv (m) As shown in the olumn of the improved version II, the alibration results of the 00 trials are rather onsistent: the varianes of the orientation terms are no more than one degree (most of them are around or less than half a degree); the varianes of the position terms are no more than few entimeters. he alibration results of the improved version I are also rather onsistent, only slightly outperformed by the improved version II. he onsisteny of the alibration results reflets the effetiveness of the proposed omprehensive extrinsi alibration method to reveal all the spatial relationships among the CCS, the SCS, the GCS, and the VCS. Besides, even only onsidering the spatial relationships that the method in [1] an reveal (see variane R s and s ), the improved version II still outperforms the method in [1]. Inria

22 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile 1 Some intuitive results are also demonstrated to indiretly reflet the effetiveness of the proposed method (using the improved version II): First, the laser points and their ground projetions are mapped onto the orresponding image, respetively marked by red points and blue points, as shown in Fig.5-Left. hese mapped points are visually onsistent with the environment shown in Fig.5-Left. Seond, a bird-eye-view of the garage floor is generated based on the alibration results, as shown in Fig.5-Right. he squareness of the floor grids is well reovered. he ground projetions of the laser points an be positioned and the image of the garage floor an be inverse perspetive mapped onto the ground, thanks to the alibrated spatial relationships among the CCS, the SCS, and the GCS, whih are obtained by the proposed omprehensive extrinsi alibration method. It is worthy noting that these spatial relationships are revealed based on the ommon hessboard alibration pratie without any extra alibration pratie. Fig.5. Intuitive effets: (Left) laser points and their ground projetions; (Right) the bird-eye-view image of the garage floor 6 Conlusion We propose a new method to perform omprehensive extrinsi alibration of a amera and a D laser sanner, i.e. the proess of revealing all the spatial relationships among the CCS, the SCS, the GCS, and the VCS. As part of the method, the spatial relationships among the CCS, the SCS, and the GCS are alibrated based on the widely used hessboard alibration pratie only. With few extra measurements, the spatial relationships assoiated with the VCS an be further revealed. R n o 438

23 Hao LI, Fawzi NASHASHIBI he proposed method has been tested on both syntheti data and real data: both quantitative evaluation and intuitive effets are given. Experiments have shown that the introdued omprehensive extrinsi alibration method an effetively reveal all the spatial relationships among the CCS, the SCS, the GCS, and the VCS. Besides, even only onsidering the spatial relationships that the method in [1] an reveal, the new method (the improved version II) still outperforms the method in [1]. he proposed method an serve as a desirable solution of amera and laser sanner alibration for mobile roboti appliations; for example, the proposed method has been used for the alibration of the amera and the D laser sanner in the appliation presented in [17]. Reently, a new alibration method [18] has been proposed, whih improves Zhang & Pless method [1] by reduing the number of poses needed to guarantee a desirable initial estimate. Our method improves Zhang & Pless method by extending its alibration apability (i.e. our method an reveal more spatial relationships than Zhang & Pless method does with the same alibration pratie) and enhaning the alibration auray. his new method [18] and our method an omplement eah other and an be integrated, whih would be a diretion of further improvements. Referene [1] Gate G, Breheret A, Nashashibi F, Centralized fusion for fast people detetion in dense environment, IEEE Int Conf on Robotis & Automation, 009, pp.76~81 [] C. Premebida, O. Ludwig, U. Nunes, LIDAR and vision-based pedestrian detetion system, Journal of Field Robotis, vol.6, no.9, pp , 009 [3] Y. Yemez, C.J. Wetherilt, A volumetri fusion tehnique for surfae reonstrution from silhouettes and range data, Computer Vision & Image Understanding, vol.105, no.1, pp.30-41, 007 [4] Z. Kim, Robust lane detetion and traking in hallenging senarios, IEEE rans on Intelligent ransportation Systems, vol.9, no.1, pp.16-6, 008 [5] M. Bertozzi, A. Broggi, GOLD: A parallel real-time stereo vision system for generi obstale and lane detetion, IEEE rans on Image Proessing, vol.7, no.1, pp.6-81, 1998 [6] M. Bertozzi, A. Broggi, A. Fasioli, Stereo inverse perspetive mapping: theory and appliations, Image & Vision Computing Journal, vol.16, no.8, pp , 1998 [7] P. Nunez, P. Drews Jr, R. Roha, J. Dias, Data fusion alibration for a 3d laser range finder and a amera using inertial data, Pro. of 4th European Conf on Mobile Robots, 009, pp Inria

24 Comprehensive Extrinsi Calibration of a Camera and a D Laser Sanner for a Ground Vehile 3 [8] D. Saramuzza, A. Harati, Extrinsi self alibration of a amera and a 3d laser range finder from natural senes, IEEE Int Conf on Intelligent Robots & Systems, 007, pp [9] C. Gao, J.R. Spletzer, On-line alibration of multiple lidars on a mobile vehile platform, IEEE Int Conf on Robotis & Automation, 010, pp [10] H. Aliakbarpour, P. Nuez, J. Prado, K. Khoshhal, J. Dias, An effiient algorithm for extrinsi alibration between a 3d laser range finder and a stereo amera for surveillane, Int Conf on Advaned Robotis, 009, pp.1-6 [11] H. Zhao, Y. Chen, R. Shibasaki, An effiient extrinsi alibration of a multiple laser sanners and ameras sensor system on a mobile platform, IEEE Intelligent Vehiles Symposium, 007, pp.4-47 [1] Q. Zhang, R. Pless, Extrinsi alibration of a amera and laser range finder (improves amera alibration), IEEE/RSJ Int Conf on Intelligent Robots & Systems, 004, pp [13] Z. Zhang, A flexible new tehnique for amera alibration, IEEE rans on Pattern Analysis & Mahine Intelligene, vol., no.11, pp , 000 [14] G. Li, Y. Liu, L. Dong, X. Cai, D. Zhou, An algorithm for extrinsi parameters alibration of a amera and a laser range finder using line features, IEEE Int Conf on Intelligent Robots & Systems, 007, pp [15] O. Faugeras, hree-dimensional omputer vision: a geometri viewpoint, MI Press, 1993 [16] J.J. More, he Levenberg-Marquardt algorithm: implementation and theory, G.A. Watson editor, Numerial Analysis, Leture Notes in Mathematis, 1977, Springer-Verlag, vol.630, pp [17] H. Li, F. Nashashibi, Multi-vehile ooperative pereption and augmented reality for driver assistane: a possibility to see through front vehile, IEEE ISC, 011 [18] F. Vasonelos, J.P. Barreto, U. Nunes, A minimal solution for the extrinsi alibration of a amera and a laser-rangefinder, IEEE rans on Pattern Analysis & Mahine Intelligene, vol.34, no.11, pp , 01 [19] V. Caglioti, A. Giusti, D. Migliore, Mutual alibration of a amera and a laser rangefinder, Int Conf on Computer Vision heory & Appliations, 008, pp.33-4 R n o 438

25 4 Hao LI, Fawzi NASHASHIBI Publisher Inria Domaine de Volueau - Roquenourt BP Le Chesnay Cedex inria.fr Inria