Planebased Calibration of a Camera with Varying Foal Length: the Centre Line Constraint Pierre GURDOS and René PAYRISSAT IRITUPS TCI 118 route de Narbonne 31062 Toulouse Cedex 4 FRANCE PierreGurdjos@iritfr RenePayrissat@iritfr Abstrat This paper deals with the problem of alibrating a moving amera with varying foal length from views of a planar pattern with a known Eulidean struture The main issue under disussion is to find a new method whose omplexity does not dramatially inrease with the number of views ontrary to existing methods Our ontribution is to relate this alibration problem to the Centre Line CL onstraint that is the prinipal point lous when planar figures are in perpetive orrespondene in aordane with Ponelet s theorem We demonstrate that the CL equation is irrespetive of the foal length and holds for eah view with only three unknown parameters whose values are onstant in the images We define its analyti equation with oeffiients omputed from the worldplane to image homography matries An important aspet is that we an make use of it as a linear ost funtion that expresses a geometri error instead of algebrai errors in existing methods We explain why an optimal solution an be obtained when pixels are retangular The simulations on syntheti data and an appliation with real images onfirm the two strong points of our method with respet to existing ones: a lower omputation ost and a better system onditioning that permits to obtain more aurate results 1 Introdution At the highest level of flexibility the autoalibration of a projetive amera problem onsists in reovering the amera metri information from nonmetri information and selfonsisteny onstraints read [] or [1] for historial bakground Fundamentally one very determining result has been to highlight some imaginary projetive entities like the absolute oni or the dual absolute quadri enabling to define a generi framework based on properties of the diretion bases In partiular the problem of autoalibration from planar senes [8][4] ie from multiple views of at least one referene plane an be stated within this framework Speifially the issue at stake here is about planebased alibration [12][7][3] that an be seen as a speialization of the autoalibration from planar senes problem By 623 BMVC 2001 doi:105244/c1564
definition planebased alibration requires metri information about a referene plane eg oordinate system angle length ratio et An important advantage of planebased alibration is that the estimation of plane to image homographies are very stable and aurate Planebased alibration inludes the ase of a single view alibration from multiple planes where some knowledge about the amera internal parameters eg zeroskew and known aspet ratio are assumed providing that three mutually onjugated vanishing points are known [2] or in presene of parallelepipeds [10] In a seminal paper [12] about planebased alibration Z Zhang onsiders the same onstraints than those used in autoalibration from planar senes but substitutes the world to image homographies supposed to be known up to a similarity indued by a plane for the interimage homographies For varying intrinsi parameters Sturm and Maybank [7] inrease the sope of this method by showing that oeffiients depending on these varying parameters an be onsidered as additional unknowns of the problem In some ways these planebased alibration algorithms offer a ompromise solution between the high auray of photogrammetri methods and the flexibility of autoalibration These algorithms are easy to implement and fast beause the solution is obtained by solving linear equation systems to least squares by non iterative proedures They also prove to be effiient providing that there are no ritial amera displaements [7] The snag with these algorithms is that they minimize algebrai distanes and a good hoie of system normalization as we will disussed later is often neessary to obtain reliable results When only the foal length an vary an optimal solution attaining the CramerRao lower bound is proposed in [5] Nevertheless it requires an offline prealibration step not desribed and Newton iterations in order to only onsider the different unknown foal lengths unalibrated Motivation Within the framework of planebased alibration of a amera with varying internal parameters our aim is to find a new algorithm whose omplexity does not dramatially inrease with the number of views ontrary to the existing method [7] In the ase of a varying foal length we suggest to split planebased alibration problem in two The first subproblem is that of reovering the three onstant internal parameters aspet ratio and prinipal point oordinates The seond subproblem is then to ompute the foal length for eah partially alibrated amera We relate the first subproblem to the properties of the Centre Line CL that is the prinipal point lous when planar figures are in perspetive orrespondene We show that the geometri properties of the Centre Line allows us to define an optimal ost funtion based on orthogonal distanes in the images For the two subproblems this geometri formulation helps us intuitively to avoid the singularities already exhaustively listed in [7] Simulations show the very good performane of the CLbased algorithm vs Sturm & Maybank s algorithm regarding both omplexity and auray Notations A point or line is denoted by a sans serif letter and the vetor that represents it by the same lowerase bold letter A matrix is denoted by an upperase bold letter or a Greek bold letter We use for denoting the diagonal matrix of order with the elements on the main diagonal The olumn vetors of matrix are denoted by! and the elements of " are denoted by #%$& A set of matries of the form is denoted by ' * +' // +' 0 A salar is denoted by a nonbold letter 1 2 or 3 Notation 46587 means that 4 is equal to 7 up to a salar fator while notation ;:= means that approximately equal to 624
~ d 0/ %$ 0/ & 0/ 0/ 2 Planebased alibration onstraints Projetive Camera model The Eulidean oordinates 1 of a point and its imaged oordinates 2 are related by the projetion equation &2 5 1 The perspetive projetion matrix an be deomposed as where is the upper triangular matrix of the amera internal parameters and is a 3D rigid displaement We will use the following deomposition "!# by writing: $%'& & $%& & 2+* 23* * 1!2 * & 14#5 1 where is a skewness fator & is the foal length in pixels is an aspet ratio parameter unitless and 2 * * are the pixel oordinates of the prinipal point Pixels are very often supposed to be retangular in the literature so all along the paper we will assume hypothesis known as the zeroskew onstraint Let 6 87 be the dimensional = projetive spae and : a basis of 6 ;7 suh that the equation of the plane at infinity is >? The absolute oni @ has equation 1 BA 2AC >D with respet to basis : The matrix EFGH I'G represents the image of the absolute oni @ under Hene the reovery of is losely onneted to the intrinsi alibration of the amera ie the reovery of matrix The image of the absolute oni @ under is repre $% K 2+* 5 L * 2 2 * * 2* AM * A Let N $& PORQMS be the elements of Under the zeroskew assumption the following four Internal parameter onstraints sented by: equations an be used as autoalibration onstraints : N N B N 3 2 * N N * N N 4 Cirular points onstraints Let us onsider the ase of a plane with equation and adjoin to it two onjugate omplex points TU on the line at infinity V>W known as the irular points These points have oordinate vetors XYUZ[ 8 \O* where O Let ] be the worldtoimage ^_ homography matrix indued by Cirular points verify 1"A ` and therefore lie on the absolute oni @ Their images under ] lie on the image of @ whih means that P]'X U a ]'X U b Requiring that both real and imaginary parts of this equation be zero yields the two linear equations: 5 where and are the first two olumns of ] These onstraints are linear in the elements of and form the basis equations of the planebased alibration problem These onstraints are derived from the form ofe in 2 but to our knowledge the onstraints 4 have never been subjeted to a geometrial interpretation We suggest that they arise from a polepolar property In the projetive plane the line at infinity has a pole fg with respet to any proper oni h and this point is the entre of h In partiular for the absolute oni we an write ikj g1lnmpo where the vetor jqgrrts uvwuvxzy{ represents the entre of h and m o rs u vwuv}xy{ represents the line at infinity This matrix equation is transformed under into e2s g vp gvxy{_r m o sine m o is invariant under any affine transformation The transformed matrix equation yields only two independent equations that are stritly equivalent to 4 625
21 Related works To our knowledge there is no non iterative planebased alibration method other than those based on equations 5 and disussed in the introdution A Constant internal parameters By assuming onstant internal parameters Zhang [12] gives a losedform solution to the alibration problem : with images yielding homography matries ]!' there are equations taken from 5 while the unknown symmetri matrix has degrees of freedom Consequently at least images are required to reover Under the zeroskew assumption images suffie B Varying internal parameters In [7] Sturm & Maybank extend Zhang s work [12] to the ase of varying internal parameters In their paper pixels are supposed to be retangular Two ases are onsidered : [i] only the foal length & vary or [ii] & varies in onjuntion with the prinipal point These varying parameters are onsidered as additional unknowns of the basi planebased onstraints 5 Typially for images in the ase [i] resp [ii] the system will have equations with 1A resp 1A unknowns 22 Disussion From a omputational point of view the solution to the problem 21B seems to us to be questionable at least for two reasons Basially our ritiisms apply to the algorithm omplexity and system onditioning Complexity Regarding algorithms for planebased alibration inluding the varying foal length ase from images the problem is to solve a set of homogeneous equations of the form 4 where 4 is the unknown vetor with degrees of freedom and is the matrix of oeffiients omputed from matries ] ' in 5 When the system is overdetermined ie the task it to find the vetor 4 that minimizes the ost funtion 4 under the onstraint 4 F Suh a leastsquares solution requires the singular value deomposition of or equivalently the omputation of the spetral deomposition of With regard to problem 21B in the Sturm & Maybank s alibration algorithm for varying foal length this omputation applies to a matrix whose dimension is 6w A where is the number of views Suh omputation is extensive and does not suit to real timelike proessing of video image sequenes Indeed when solving a leastsquares problem from images the total number of flops for arrying out the of is 1A 1A AY omputed from [1] pp558 If we onsider for instane a sequene of images the matrix will have rows and olumns and the minimization will require Megaflops Conditioning The ost funtions used in [12][7] are based on algebrai distanes As a general rule it is known that suh quantities are not geometrially or statistially meaningful and the obtained solutions may not be those expeted intuitively [1] Although we do not provide the proof here we also laim that in the ase of a varying foal length input homographies orrupted with the same amount of noise may ontribute differently to the estimation problem depending on the foal length value whih leads to an undesirable row weighting Moreover the elements of matrix and vetor 4 may differ in magnitude in a very sizeable way That leads to a bad system onditioning that Sturm and Although their paper does not refer to Zhang s work Only the matries hhv appearing in the "! deomposition # r$bh% { are omputed 626
Maybank improve by olumn resaling suh as to have equal norms [7] The least we an say is that aording to what we have notied when running simulations this resaling is not optional at all In the following we want to define a linear ost funtion that does not reveal suh disadvantages It an be ahieved if we replae the algebrai onstraints on the imaged absolute oni by geometri onstraints dedued from two planar figures in perspetive orrespondene defining a penil of entre lines irrespetive of the foal length 3 The Centre Linebased onstraint 31 Geometrial bakground The geometri properties dedued from two planar figures in perspetive orrespondene and in partiular about the positions of the entres of projetion is known for a long time notably thanks to the works of V Ponelet in the middle of the XIX entury [6] Fundamentally Ponelet demonstrated the following theorem in a a purely geometri way: When a planar figure is a entral projetion of another planar figure these figures remain in perspetive when one rotates the plane of the first around its intersetion with the plane of the seond and the entre of projetion desribes a irle lying on a plane whih is perpendiular to this intersetion If we onsider the image of a plane this leads us to the following definition Definition 1 Centre line The entre line is the line segment obtained by the orthogonal projetion of the irle onto the image plane Property 2 Aording to Ponelet s theorem the entre line is the geometri lous of the prinipal point is orthogonal to the vanishing line of and the orientations of these two lines are invariant to variations of the foal length In the following we define expliitlythe analyti equation & of the entre line We demonstrate that this equation is irrespetive of the foal length and we use it as a onstraint in order to reover 2 * * ie the elements of matrix! in 1 This allows us to integrate a large amount of views in the system sine any additional view does not introdue a new unknown The foal length is then diretly omputed by solving the partially alibrated system 5 Proposition 3 Let ] be the worldtoimage homography matrix and let $& be the element O zs of ] The vetor t 1A where: ' ' ' G G ' represents the line oordinates of the entre line ' ' ' G G ' 6 with respet to the affine image frame Proof Let us initially eliminate algebraially N from equations 5 We obtain a new equation linear in the terms of : where obtain: M N zn N zn 7 is defined as in 6 Introduing relations 34 in 7 we 627
4 7 0/ 2 * * A Equation 8 tells us that the prinipal point with pixel oordinates 2H* * lies on a ertain line represented by R 2A in the image frame We now demonstrate that oinides with entre line Sine line ontains 2 * * using property 2 it suffies to show that line is orthogonal to the vanishing line of 3D plane It is known that the vetor representing line is equal projetively speaking to the third row of the adjoint matrix of ] Hene after some omputations we find: 5F With respet to the affine image frame it is equivalent to show that and are orthogonal with respet to @ 5 } This ondition holds sine it is straightforward to verify that k@ In onlusion the vetor represents the oeffiients of entre line with respet to the image frame 32 A twostep proedure for planebased alibration In the previous setion it has been shown that the equation of entre line defined in 8 results in a linear onstraint on the unknown vetor 4" 2 * * Its elements are supposed to be onstant in the images so the number of degrees of freedom in equation 7 is whatever the number A 1 of images This allows to estimate 2 * * in a first step from homographies The seond step is then to diretly reover the foal length for eah partially alibrated view We now desribe preisely these two steps 321 Reovering the onstant internal parameters With views providing worldtoimage homography matries ]6' find 4 suh that: 4 * 4 under the onstraint 4LF where C is the matrix formed by staking the row vetors ' R$# omputed from ]!' '3Q5 Q easily ompute 2 * * from equations 34 Searh of a geometri distane ' $%! " 8 the problem is to like explained in 6 One 4 has been reovered we an ' ' ' &% A very interesting aspet of our formulation using a entreline approah is that we an easily transform the ost funtion 4 into a sum of squares of Eulidean distanes It is worthy of note that under the assumption of Gaussian noise the minimization of a geometri ost funtion in the images is equivalent to a Maximum Likelihood Estimation +* [1] It is also known that the geometri desription of onstraints is quite advantageous beause system onditioning but also the detetion of degenerated onfigurations beomes learer The following proposition enables us transform the ost funtion into a geometri funtion based on orthogonal distanes in the images Proposition 4 Let the matrix obtained by normalizing eah row as _ 0 / When we seek 4 suh that: 32 4' 2 56 4 8 * ' 1* 22 1* :; IM 10 628
A & & where 8 * is the Eulidean distane from prinipal point 8 * to entre line This proposition means that the residual error in 10 is the sum of squared orthogonal distanes In pratie we know that : so this proposition is still aeptable Complexitywise the leastsquares problem 10 is then solved via the deomposition by omputing 4 where is the olumn of assoiated with the smallest singular value Atually only and are omputed so the amount of work A Y flops inluding row normaliza required by this step of the algorithm is about tions of proposition 4 where is the number of images 322 Reovering the foal lengths For eah view ' let G! ] ' where! is onstruted from 2 * *! '! 58 ' ' the matrix equation D A Let be the matrix obtained by normalizing the olumns $ of suh that 22 $ and let The expliit leastsquares solution obtain: D 3 G where 3 ^ A! A " The image of the absolute oni now modifies to _ be seen that the problem is to solve for where: as in 1 It an 11 22 an be omputed diretly and we For eah view the flop ount assoiated with this step of the algorithm is about flops 323 Singularities The singularities of alibration must our at two levels in our method ie when reovering the onstant internal parameters level 1 and when reovering the foal lengths level 2 The singularities are basially the same as those desribed in [7] but our geometrial framework enables us to give some justifiations We desribe in the following some of the most signifiant At level 1 degeneraies our if in two of the three required views vanishing lines assoiated with the observed plane are parallel It happens in partiular when the amera displaement is either a translation a rotation with axis parallel or perpendiular to or a omposition of these displaements Consequently the vanishing lines in different views must have different orientations in order to define the prinipal point as the vertex of the penil of entre lines In the majority of ases adding rotations around the axis an notieably & improve the results At level 2 regarding system 11 there is no solution for in two ases: [i] \ [ii] or These ases orrespond to partiular orientations of the world oordinate system with respet to the amera oordinate system : they are unneessary to desribe them sine they an easily vanish by applying a rotation around the axis ie the normal to of the world frame homographies are estimated up to a similarity For instane [ii] holds as soon as the 2 axis resp axis of the amera frame is parallel to the 1 axis resp axis of the world frame ie assoiated vanishing points are points at infinity 62
& ' 0 4 Experiments Simulation We arry out omparison tests about the Centre Linebased method CL vs Sturm & Maybank s method SM for a moving amera with variable foal length A simulation is run with ameras shooting a alibration objet The alibration objet is a planar grid ontaining points with dimension The image resolution is Y pixels All the internal parameters of the amera are supposed to be unknown The parameters 2 * * are onstant We divide the set of ameras in subsets of to whih we assign a single foal length value in pixels respetively equal to & & see figure 1 Figure 1: Samples of images assoiated with and Regarding the external parameters for eah amera we randomly hoose its position in a sphere whose the entre is the entre of the grid and radius is equal to The fixated point by the ameras is roughly the entre of the grid A random rotation around the optial axis is eventually added with mean and standard deviation We also restrit the orientation between the 3D plane and the optial axis to the range in order to put the simulation in the best situations for the Sturm & Maybank s algorithm The omparisons fous on error omputations at different levels of Gaussian noise with mean and standard deviation varying from to pixels with a step of This noise is added to the projeted points in the image Eah world to image homography matrix is estimated from the sole visible points in the view using the normalized Diret Linear Transformation DLT algorithm [1] We ompute absolute errors in pixels for 23* * relative errors for and mean relative errors & & Eah result presented in table 1 is the mean of indepen dent trials It is worth mentioning that the absolute pixel error funtion on 2H* * roughly behaves like with the CLbased method and like with Sturm & Maybank s one The estimations about & & are better with our method even if the error on is always very low % at the worst The fat that relative errors on & are higher than errors on & an be likely explained by the way we introdue the noise Indeed relative distanes between points are smaller in images assoiated with & than in images assoiated with & see figure 1 Consequently for the same amount of noise the estimation of the homographies is less aurate for than for Real images We also hek our algorithm with real images of a alibration grid taken at a distane of about from different positions Images are taken by a Nikon Coolpix R 800 in a jpeg format with the maximal setting of zoom We know that the bigger zoom is used the less distortion is present in the images The amera operate small variations of the foal length during the autofousing In eah image the orresponding entre line is represented by a solid line The prinipal point is drawn with a ross The reovered internal parameters are 2*`! *b F ' & ' & ' D & ' & & ' * '#" % The error vetor $ & ie whose elements are orthogonal distanes from the estimated prinipal point to the entre line in eah image has mean pixels with standard deviation It is In the following setion notationwise `r*u u+ x}u / distributed means has a mean value uu with standard deviation u of a / distribution If / is omitted the distribution is assumed to be uniform Indeed in their own simulations they notied that foal length estimates are slightly better within this range 630
worth mentioning that in image 3 the distane is pixels It onfirms the diffiulty we had when extrating the grid points in this image ; this is probably due to a bad fousing during the aquisition Error abs rel & % & 2 * * LC SM LC F SM Y LC SM LC SM Ÿ LC 3 SM F F Table 1: Comparison tests The CLbased method CL vs Sturm & Maybank s SM method Figure 2: Planebased alibration from views of a planar grid stiked on a CD jewel ase Centre lines are displayed The prinipal point ross is the point suh that the sum of squared orthogonal distanes from it to eah enter line is minimum 5 Conlusion In this paper we propose a new onstraint for planebased alibration from images It partiularly applies to ase of a varying foal length sine this onstraint is irrespetive of it The alibration is arried out using a twostep method: at first the aspet ratio and the prinipal point are estimated after whih eah foal length is diretly omputed Our method offers two strong points: [i] the required amount of flops is in while it was in in the existing one [ii] the results have a better auray at least in simulations This is probably due to the fat that our ost funtion has a geometri interpretation involving orthogonal distanes and that its minimization is equivalent to a MLE estimation Regarding the future prospets we have to ompare our experimental results with those of any maximumlikelihood algorithm in order to get a baseline performane Our ap 631
proah shows some obvious limits notably beause the prinipal point is supposed to be onstant The geometri properties of our linear system enables us to envisage a Kalmanlike reursive estimation with a ost funtion integrating the evolution of prinipal point oordinates and foal lengths taking into aount eah additional view as soon as it is available Referenes [1] R Hartley and A Zisserman Multiple view geometry in omputer vision Cambridge University Press Cambridge UK 2000 [2] D Liebowitz A Criminisi and A ZissermanCreating arhitetural models from images In Pro EuroGraphis volume 18 pages 350 September 1 [3] D Liebowitz and A ZissermanCombining sene and autoalibration onstraints In Pro 7th International Conferene on Computer Vision ICCV Kerkyra Greee September 1 [4] E Malis and R Cipolla Selfalibration of zooming ameras observing an unknown planar struture In Pro 15th International Conferene on Pattern Reognition ICPR 00 volume 1 pages 8588 Barelona Spain September 2000 [5] C Matsunaga and K Kanatani Calibration of a moving amera using a planar pattern: optimal omputation reliability evaluation and stabilization by model seletion In Pro 6th European Conferene on Computer Vision ECCV 00 Dublin Ireland uly 2000 [6] V Ponelet Traité des propriétés projetives des figures Imprimerie de Mallet Bahelier Paris 1862 [7] PF Sturm and S Maybank On planebased amera alibration: a general algorithm singularities appliations In Pro Computer Vision and Pattern Reognition Conferene CVPR pages 432 437 Fort Collins Colorado USA une 1 [8] B Triggs Autoalibration from planar senes In Pro 5th European Conferene on Computer Vision ECCV 8 pages 8 105 Freiburg Germany une 18 [] B Triggs Géométrie d images multiples" Thèse de l Institut National Polytehnique de Grenoble November 1 [10] M Wilzkowiak E Boyer and P Sturm Camera alibration and 3D reonstrution from single images using parallelepipeds In Pro 12th International Conferene on Computer Vision ICCV 2001 Vanouver Canada uly 2001 [11] Z Zhang Parameter estimation tehniques: a tutorial with appliation to oni fitting Reseah report INRIA N 2676 Otober 15 [12] Z Zhang A flexible new tehnique for amera alibration IEEE Transations of Pattern Analysis and Mahine Intelligene PAMI volume 22 number 11 pages 1330 1334 November 2000 mainly written in English 632