Automated Stereo Camera Calibration System

Transcription

1 Automted Stereo Cmer Clibrtion System Brin J O Kennedy, Ben Herbst Deprtment of Electronic Engineering, niversity of Stellenbosch, Stellenbosch, 7, South Afric, brokenn@dspsuncz, herbst@ibissuncz Abstrct e present n novel technique for effectively clibrting binoculr stereo rig with clibrtion objects nd miniml humn intervention Intrinsic clibrtion is performed using clssicl techniques [1] nd novel method is developed for extrinsic clibrtion The extrinsic clibrtion provided by the clssicl techniques cn only be expected to be vlid in the spce spnned by the clibrtion object [2] e present technique to combine multiple views of the clibrtion object with unknown motion, resulting in virtully enlrged object sing priori knowledge of the clibrtion object we re then ble to upgrde our clibrtion to the euclidin spce The effects of noise on the system hs been exmined, nd our technique hs proven to be much less sensitive to noise thn clssicl method 1 Introduction Cmer clibrtion is n importnt tsk in computer vision The purpose of cmer clibrtion is to estblish the reltion between 3D world coordintes nd their corresponding imge coordintes This enbles us to infer 3D informtion from 2D informtion nd vice vers Thus cmer clibrtion is prerequisite for ny ppliction 3D informtion needs to be relted to 2D informtion The pplictions of cmer clibrtion re endless, rnging from militry to rcheology to medicl This pper focusses on the simultneous clibrtion of two cmers, which is referred to s stereo vision The method used for cmer clibrtion cn lso vry widely, depending on fctors like the ccurcy required, the level of humn interction nd whether reconstructions need to be metric Clssicl clibrtion lgorithms rely on clibrtion object with known dimensions which is plced in the cmer field of view This enbles us to clibrte the cmer up to metric scle, resulting in reconstructions which re up to scle with physicl mesurements Some more recent lgorithms rely on fetures extrcted from the scene to clibrte the cmers [3] These hve the dvntge tht there is no clibrtion object nd no humn intervention is needed, but these models cn only reconstruct the scene up to n unknown scle, nd relies on the ccurte extrction of fetures in the viewed scene, which re not lwys vilble e use clssicl lgorithm to clibrte the intrinsic prmeters of both cmers, nd then propose method to solve for the epipolr geometry of the cmers, which defines the inter-cmer trnsltion nd rottion These extrinsic prmeters re essentil to do 3D reconstruction of scene The clibrtion routine described in this rticle requires cler view of clibrtion object, on which known fetures re extrcted in the imge plne The extrction of these fetures is described in the next section, fter which intrinsic nd then extrinsic clibrtion is discussed It is followed by results nd noise nlysis of this method 2 Feture oction The cmer clibrtion routine tries to solve for set of prmeters which define the reltionship between 3D coordintes, nd the resultnt imge coordintes Thus for our method it is needed to hve set of imge coordintes of n object with known physicl dimension The ccurte extrction of these fetures from the imge plne plys fundmentl role in the ccurcy of the model extrcted e give here short description of our extrction method 21 Object oction The purpose of this implementtion is clibrtion method which cn esily be performed in the industry, thus we need miniml level of humn interction in the system Our clibrtion method relies on series of imges, t lest one of which must contin the clibrtion object This is obtined by tking series of imges with both cmers, until the object hs been locted in both imges This requires very robust extrction lgorithm, which must gurntee tht no flse mtches re extrcted, thus rther erring on the side of cution Our object consists of either one plne contining blck squres, or two fixed t n ngle of 9 degrees The extrction of both re very similr Our non coplnr object is shown if Fig 1 As previously stted the clibrtion routine requires very ccurte extrction of fetures The initil estimtes of corners in the bove routine my be incorrect by s much s 15 pixels e use these estimtes s the initil position for corner detector bsed on the Hrris-Plessey detector This extrcts the exct positions of the corners to sub-pixel ccurcy Note tht uncontrollble effects like lighting nd blooming on the CCD sensor cuse slight errors 3 Overview of Proposed Method e now give brief overview of our proposed method of clibrtion The clibrtion of binoculr system cn be divided into two stges The first consists of seprtely clibrting the intrinsic prmeters of both cmers This defines the trnsformtion of 3D coordintes in ech cmer s reference frme to their respective imge plnes The second stge consists of determining the epipolr geometry of the binoculr rig This is defined by rottion nd trnsltion, in which ccurcy is essentil in order to generte precise reconstructions of scene e solve for the intrinsic prmeters using clibrtion technique first proposed by Tsi This lgorithm ws chosen becuse of it s proven relibility in prcticl pplictions nd is

2 B [[[[[[[ G< 4 B B I I E E C E C E G 4 given by "!$%!$'(!$)!+%!+(!+,)!)-%!)(!)) (1) 8 nd 9 define the rottion nd trnsltion, respectively, from world coordintes to cmer coordintes The undistorted retinl coordintes re obtined s :; =<?>A@BDC ; =< (2) Figure 1: Corner extrction on clibrtion object described in section 4 sing this technique we determine intrinsic prmeters for cmer, s well s extrinsic prmeters relting the cmer s physicl position in reltion to clibrtion object In the cse tht both cmers cn simultneously see the clibrtion object, it is possible to extrct the epipolr geometry of the binoculr rig This is rrely the cse, especilly when the bseline between cmers is lrge In order for both cmers to view the object simultneously it is often required tht the object is uncceptbly fr wy from the cmers, resulting in inccurte feture extrctions, or n overly lrge clibrtion object e propose to solve this problem by tking multiple stereo views of clibrtion object with unknown motion prmeters, nd combining the dt to determine the epipolr geometry of the system The first order rdil coefficient is needed to solve this in order to trnsform the epipolr curve to n epipolr plne Section 5 demonstrtes our technique nd lso shows tht it is much more robust thn the technique proposed by [1] 4 Intrinsic Clibrtion Intrinsic clibrtion entils finding the prmeters defining the trnsformtion between 3D cmer coordintes nd the imge plne These re composed of the focl length, distortion coefficient, imge center nd scling fctors Here we describe the technique originlly proposed by Tsi This method extrcts the intrinsic s well s extrinsic prmeters of ech cmer The extrinsic prmeters re extrcted with the coordinte system centered t the position of the clibrtion object Thus the two cmers cn not be relted extrinsiclly This problem is solved in the next section 41 Cmer Model e use the well known pinhole cmer model [4] nd lso include the effect of rdil distortion This distortion effect is esily determined mthemticlly, nd cn hve significnt effect on the clcultion of the epipolr geometry The Eucliden trnsformtion of point, given in world coordintes into point in cmer centered coordintes is f denotes the cmer focl length in millimeters The reltion between rdilly undistorted nd distorted retinl coordintes HG is given bymno PQF :; :FE C ; C E I HG HG IKJ I HG MNO PQF IKJ I I : : e only use the first order rdil distortion coeffiecient In ddition the coordintes in the imge do not correspond to the physicl coordintes in the retinl plne ith CCD cmer the reltion depends both on the size nd shpe of the pixels nd on the position of the CCD chip in the cmer The imge coordintes re obtined through the following eqution :S T C S T VN4 4 Y 4 BF Y BF Thus the intrinsic prmeters re :FE C E Prmeter Focl ength Y 4 First order rdil distortion coefficient Y Imge center coordinte VN4 Imge center Y coordinte Horizontl scling fctor of frme grbber Horizontl size in mm of individul CCD elements Verticl size in mm of individul CCD elements Mnufcturers of CCD cmers supply the vlues V4 of B nd, however the dditionl uncertinty fctor is lso introduced to ccomodte for the imprecision in timing signls in the imge cquisition hrdwre 42 Clssicl Technique The intrinsic clibrtion is bsed on routine first proposed by Tsi Clibrtion is mde possible by single view of noncoplnr object, the imge coordintes of t lest 7 fetures re known, s well s the physicl dimensions of the object This routine extrcts both intrinsic nd extrinsic prmeters e give short lgorithmic overview of the method efer to [1] for detiled nlysis First convert the imge :ZE coordintes, :ZS T C S T, into distorted retinl coordintes, C E vi the inverse of eqution (3) The rdil lignment constrint llows us to set up the following set of liner equtions C EN\ \ C EN\ \ C EN\ \ C EN\ ^ :FEN\ \ ^ :FEN\ \ ^ :FEN\ \ C E-] ] C E-] ] C E-] ] C E-] ^ :FE-] ] ^ :FE-] ] ^ :FE-] ] ` [ [[[[[[ ) J b c d [ :FE :FE-e _ (3) (4)

3 ! [ G e G G \ S here for VN4KI,2 VN4KI 2 ) VN4KI)2 J VN b I J 2 c I b 2 d I c 2 is defined s (5) The solution of this system obtined by the Moore-Penrose inverse llows us to clculte the following prmeters 25 b c d () Determine the sign of 2 by projecting point on the object in world coordintes onto the retinl plne, nd compring the sign to the known retinl plne coordinte sign If there is sign difference, the sign of 2 will be negtive Determine the horizontl scling fctor V4 VN4 ) 25 (7) Compute IK, I, I), I J, I b, I c nd 24 from the solution to (4) nd (5) Now the third row of! cn be clculted s the crossproduct of the first two rows, using the orthonorml property of nd the right-hnded rule By ignoring rdil distortion it is possible to linerly com- < 27 pute nd by solving the following liner system I J \ I b \ I c \ 25 ^ C EN\ _ < 257 I J ] I b ] I c ] 25 ^ C E-] [ I d \ I \ I \ :FEN\ _ I d ] I ] I ] :FE-] Solving this set of liner equtions by mens of the Moore- Penrose inverse gives n initil estimte of < nd 2F7 which will be exct in the cse of idel cmer with no lens distortion The next step is to determine the first order rdil distortion coeffiecent, et be the world coordinte of feture point on the clibrtion object, nd S T :S T C S T be the corresponding imge point s found by the corner detector e compute the reprojection of ll the 3D world coordintes Y 4 Y using the cmer model described bove, ssuming to be the center of the frme buffer, nd The totl error is mesured s I I I S :S T ^ : "!$%' "!$% "!$% "!$% "!$% C S T ^ C "!$%' )( (9) is the reprojected imge coordinte This gives us n error mesure of how ccurtely our model fits the given trining < dt 27 The non-liner optimiztion is first performed on only, nd This is followed by optimiztion with ll the prmeters except the imge center (8) Figure 2: Multiple views of coplnr object The lst step is optimiztion with ll prmeters, including Y 4 nd Y e use Nelder-Med Simplex serch lgorithm, but ny stndrd multi-dimensionl unconstrined lgorithm such s evenberg-mrqudt should suffice This lgorithm llows us to very ccurtely extrct the intrinsic prmeters for single cmer The next section shows some results of the clibrtion 5 Extrinsic Clibrtion The Extrinsic prmeters define the trnsformtion from world centered coordintes to cmer centered coordintes It is essentil to define this ccurtely for both cmers in order to derive the epipolr geometry of the system This llows us to use mutul world coordinte system for both cmers, which we chose to lign with the left cmer reference system The epipolr geometry is defined by rottion nd trnsltion, s follows! et! 2,!!+ 2 nd, be respectively the left nd right cmer rottion nd trnsltion with reltion to their respective world coordinte systems The cmer bseline is clculted s!!!! ` 2 2 ^! ` 2! 51 Clssicl Technique (1) e give brief discussion of the extrinsic clibrtion technique of Tsi using coplnr dt This lgorithm ws developed for single cmer with single view of coplnr clibrtion object, such s figure 2 Extrinsiclly clibrting binoculr rig relies on the coplnr plne being visible in both cmers t the sme instnt This enbles us to relte both cmers extrinsiclly to the clibrtion object, nd thus to ech other through (1) This lgorithm is similr to the first stge of the technique described in section 42, nd the reder is referred to [1] for detiled derivtion The technique hs been proven to be very ccurte on synthetic dt, but the ddition of noise genertes serious distortions It hs lso been shown by [2] tht the cmer model cn only be expected to be ccurte over the volume

4 [[[[[[[[[[[[ ` O l p l e l P l P,T Figure 3: Epipolr Geometry encompssed by the clibrtion object This is due to inccurcies in feture extrction Our solution to this is to virtully enlrge the clibrtion object over multiple views in order to get trining dt over the entire rnge in which the cmers re expected to operte 52 Epipolr Geometry The geometry of stereo, known s epipolr geometry, is shown in Figure 3 The figure shows two pinhole cmers, their projection centers, nd! Ech cmer defines 3D reference frme, the origin of which coincides with the projection center, nd the Z-xis with the opticl xis The vectors nd! refer to the sme 3D point The vectors nd! refer to the projections of P r onto the left nd right imge plnes respectively The nme epipolr geometry is used becuse the points t which the line through nd! intersect the imge plnes re clled epipoles, denoted by nd! The definition of the epipolr plne, s in [4], sttes Given stereo pir of cmers, ny point in 3d spce,, defines plne,, going through nd the centers of projection of the two cmers The plne is clled the epipolr plne, nd the lines intersects the imge plnes conjugted epipolr lines The prcticl importnce of epipolr lines is tht the corresponding feture of point the the left imge must line on it s epipolr line in the right imge This constrins the serch for corresponding fetures to 1-dimensionl serch 53 Fundmentl Mtrix e will now derive the definition of the essentil nd fundmentl mtrices The eqution of the epipolr plne through is chrcterised by the coplnrity condition of the vectors, 9 nd ^ 9 ^ 9 ` 9 sing the reltion defined in (1) we obtin 8 `! ` 9 er p r O r (11) (12) riting the vector product s multipliction by rnkdeficient mtrix 9 (13) (11) now becomes ^ ^ 254 ^ `!$! (14) (15) with (1) sing the perspective projection defined in (2), (15) cn be written s `! (17) The mtrix is clled the essentil mtrix, nd estblishes link between the epipolr constrint nd the extrinsic prmeters of the stereo system It is this mtrix tht llows us to combine multiple views into one set of equtions to solve the extrinsic prmeters Note tht these clcultions re ll performed on the retinl coordintes Thus in order to use this, the trnsformtion from imge coordintes to retinl needs to be known Thus the cmer needs to be intrinsiclly clibrted In our cse the intrinsic prmeters re known, but we generlized the lgorithm for the cse it is not 531 Clcultion from Imge Mtches e will now show how the fundmentl mtrix cn be clculted directly from imge mtches et nd! be the pixel coordintes of vectors nd! Now we hve!!! (18) here nd! re the intrinsic mtrices from (3) of the left nd right cmer respectively `!! By substituting (18) into (17) we hve (19) (2) is known s the fundmentl mtrix nd ws first proposed by uong [5] The essentil nd fundmentl mtrix re very similr, the difference being tht the fundmentl mtrix is computed stright from pixel coordintes, nd does not require priori knowledge of the intrinsic prmeters of the cmers The fundmentl mtrix is clculted from the set of liner equtions defined in (19) which is defined for ech point mtch \! \ \! \! \ \! \ \! \! \ \ \ `S ]! ] ]! ]! ] ]! ] ]! ]! ] ] ] ` [ [[[[[[[[[[ ' ),) )- ) )) (21) (22)

5 4 B B B ) > ) ) Vrious solutions exist to clculte the fundmentl mtrix from liner equtions They re clled the 8-point lgorithms, s eight or more points re needed to solve for [] 532 Normliztion Trucco Verri [4] show tht the clcultion of the fundmentl mtrix is very sensitive to noise on the mesurements, nd tht by normliztion the results re gretly improved A simple procedure to void numericl instbility is to shift the points so tht the men is, nd scle them to norm of This scling is done by two 3x3 mtrices s follows, imge coordintes nd!!! (23)! re the normlized coordintes,! re the ^ ^ (24) is the vrince of the coordinte set, nd : C re respectively the men of the : nd C coordintes The normliztion mtrix for the right cmer,! is clculted in the sme wy we clculte using the bove liner method The coordintes re ll normlized using!, nd then is then obtined s! ` (25) The disdvntge of the liner method is tht the rnk-2 constrint is not enforced It ws shown by [5] tht this cn be constrined fter the clcultion by tking the singulr vlue decomposition of nd forcing the smllest singulr vlue to be zero This gives us good estimte of the fundmentl mtrix, but it hs been shown tht the clcultion cn be further refined by tking into ccount tht hs only 7 degrees of freedom This is enforced by the prmeteriztion of into 7 prmeters 533 Prmeteriztion This prmeteriztion of ws proposed by [5], nd is simple method to derive 7 prmeters from, perform non-liner optimiztion on these prmterers to constrin the degrees of freedom, nd reconstruct the optimized mtrix from the prmeters The mtrix is prmeterised into > B which re four coefficients of homogrphy between corresponding epipolr lines, the lrgest of which is used to normlize the system, resulting in only three degrees of freedom for the four prmeters The left nd right epipoles re represented by in this cse > ^ > ^ B ^ B ^ ^ > > ^ ^ nd (2) The prmeters of the prmeteriztion is obtined s follows ' ^ ^,) ' ^ ) ^ ' ) ^,) ^ ' ) ^ ) ^ ' )- ' ^ ) ^ ' (27) efer to [] for detiled nlisys of the prmeteriztion 534 Non iner Critere sing the prmeteriztion defined bove the ccurcy of the fundmentl mtrix cn be further enhnced by non-liner optimiztion such s the Nelder-Med Simplex serch The optimiztion is performed on the fundmentl mtrix with the following criterion proposed by uong [5], d ` d ` ` (28) B is distnce mesure in the imge plne This error criterion minimizes the distnce between points nd their corresponding epipolr lines nd lso ensures tht the two imges ply symmetric role 535 Fctoriztion Once the fundmentl mtrix hs been clculted ccurtely, it cn be converted to the essentil mtrix vi (2) Tking into ccount tht! is n orthonorml mtrix nd n ntisymmetric mtrix, we cn decompose into rottion nd trnsltion The trnsltion vector is determined up to n rbitrry scle fctor This mbiguity cn be solved by knowing ny metric distnce in the imges, which we obtin from ny view of our clibrtion object et be the three columns of The trnsltion vector is simply the cross product of ny two, (29) indictes the trnsltion up to n unknown scle fctor The mtrix of cofctors of is given by ` ` ` `) `! ` ` ` `! ` which cn be rewritten s It cn then be shown [7] tht 2 D `! ^ `! ` ` ^ 2 which yields! s function of nd (3) (31) (32)

6 9 54 Eucliden Clibrtion The trnsltion vector obtined by fctoriztion is only determined up to scle fctor, but cn be upgrded to metric by knowning ny physicl dimension in the imge e tke ll the views of our clibrtion object, nd by tringultion mesure known physicl distnces on these objects This enbles us to determine the scle fctor for ech mesurement of which we tke the medin, nd simply scle the trnsltion vector by this fctor Note tht our new world coordinte system hs it s origin centered on the left cmer s projection center with the xis extending long the opticl xis mm Tx mm Ty mm Tz 1 Intrinsic esults The intrinsic clibrtion hs been compred to ground truth dt supplied by the Clibrted Imging bortory t Crnegie Mellon niversity, nd consistently produced ccurte results, s would be expected from proven lgorithm 2 Extrinsic e used synthethic dt to investigte the noise sensitivity of our proposed method A Synthetic stereo rig ws generted, with relistic intrinsic prmeters, nd n extrinsic bseline with ^ ^ in millimeters, nd rottion vector 8 ^, the three rottion elements re given in rdins nd define yw, pitch nd roll ngles round the coordinte system e investigted the noise properties by dding vrying mounts of noise to the feture mesurements used to clculte the model For ech level of noise the extrinsic clcultion ws performed 2 times, nd the results verged This ws done for noise levels from (no noise) to pixels of rndom noise on ech mesurement Our method s dt consisted of 4 views of coplnr object t distnces between 2 metres nd 8 metres Our method is compred to the result when using Tsi s method with one of these views The resultnt trnsltion nd rottion vectors re shown in fig 4 nd fig 5 respectively Figure 4 shows the three trnsltion error components plotted ginst incresing noise levels The verticl xes re the trnsltionl errors in millimeters from the ground truth The upper dotted line is the result obtined by the clssicl routine proposed by Tsi [1], nd the lower solid line by our method It is cler tht our method is much more robust Figure 5 shows the three rottionl error components plotted ginst noise The verticl xes re the errors in rdins from the ground truth Once gin our method is denoted by solid line, which is clerly less sensitiveto noise It must be noted tht in the cse of! 7 the two method give very similr results 7 Conclusions e hve estblished simple nd robust method to determine the epipolr geometry of stereo cmer rig e demonstrted how intrinsic nd extrinsic prmeters of the cmer model is determined It ws shown tht this method is much more robust thn the method proposed by Tsi [1] 8 eferences [1] Tsi, A verstile cmer clibrtion technique for highccurcy 3-D mchine vision metrology using off-the-shelf Pixels Noise Pixels Noise Pixels Noise 2 4 Figure 4: Trnsltion Error Vector rdins x rdins y rdins z 2 4 Pixels Noise Pixels Noise Pixels Noise Figure 5: ottion Error Vector TV cmers nd lenses, in diometry (Physics-Bsed Vision), olff, S Shfer, nd G Heley, Eds Jones nd Brtlett, 1992 [2] Chen, J Dvis, nd Phillipp Slusllek, ide re cmer clibrtion using virtul clibrtion objects, Tech ep, Computer Grphics lb, Stnford niversity, CA, 1999 [3] QT uong nd O Fugers, Self-clibrtion of stereo rig from unknown cmer motions nd point correspondences, Tech ep -214, INIA, Sophi-Antipolis [4] E Trucco nd A Verri, Introductory techniques for 3-d computer vision, 1998 [5] Q uong nd O Fugers, The fundmentl mtrix: theory, lgorithms, nd stbility nlysis, in Interntionl Journl of Computer Vision, 199, vol 17, pp [] Z Zhng, Determining the epipolr geometry nd its uncertinty: A review, Tech ep -2927, INIA, Sophi- Antipolis, 199 [7] Olivier Fugers, Three-Dimensionl Computer Vision: A Geometric Viewpoint, MIT Press, Cmbridge, Msschusetts, 1993