Simultaneous Object Pose and Velocity Computation Using a Single View from a Rolling Shutter Camera

Transcription

1 Smultaneous Object Pose and Velocty Computaton Usng a Sngle Vew from a Rollng Shutter Camera Omar At-Ader, Ncolas Andreff, Jean Marc Lavest, and Phlppe Martnet Unversté Blase Pascal Clermont Ferrand, LASMEA UMR 6602 CNRS Omar.AIT-AIDER@unv-bpclermont.fr, WWW home page: Abstract. An orgnal concept for computng nstantaneous 3D pose and 3D velocty of fast movng objects usng a sngle vew s proposed, mplemented and valdated. It takes advantage of the mage deformatons nduced by rollng shutter n CMOS mage sensors. Frst of all, after analysng the rollng shutter phenomenon, we ntroduce an orgnal model of the mage formaton when usng such a camera, based on a general model of movng rgd sets of 3D ponts. Usng 2D-3D pont correspondences, we derve two complementary methods, compensatng for the rollng shutter deformatons to delver an accurate 3D pose and explotng them to also estmate the full 3D velocty. The frst soluton s a general one based on non-lnear optmzaton and bundle adjustment, usable for any object, whle the second one s a closed-form lnear soluton vald for planar objects. The resultng algorthms enable us to transform a CMOS low cost and low power camera nto an nnovatve and powerful velocty sensor. Fnally, expermental results wth real data confrm the relevance and accuracy of the approach. 1 Introducton In many felds such as robotcs, automatc nspecton, road traffc, or metrology, t s necessary to capture clear mages of objects undergong hgh velocty moton wthout any dstorton, blur nor smear. To acheve ths task, there s a need to mage sensors whch allow very short exposure tme of all the matrx pxels smultaneously. Ths functonalty requres a partcular electronc desgn that s not ncluded n all camera devces. Full Frame CCD sensors, wthout storage memory areas, requre mechancal obturator or stroboscopc lght source, ntroducng more complexty n the vson system. Frame Transfer CCD sensors may not reach the desred frame rate or may be costly because of addtonal sllcum n storage areas [9]. Standard CMOS Rollng Shutter sensors are consdered as low cost and low power sensors. They are becomng more frequently used n cameras. They enable adequate exposure tme wthout reducng frame rate thanks to overlappng exposure and readout. Ther drawback s that they dstort mages of movng

2 objects because the pxels are not all exposed smultaneously but row by row wth a tme delay defned by the sensor technology (fgure 1). Ths dstorton may represent a major obstacle n tasks such as localzaton, reconstructon or default detecton (the system may see an ellpse where n fact there s a crcular hole). Therefore, CMOS Rollng Shutter cameras could offer a good compromse between cost and frame rate performances f the problem of deformatons s taken nto account. Fg. 1. An example of dstorton of a rotatng ventlator observed wth a Rollng Shutter camera: statc object (rght mage) and movng object (left mage). 2 Related works and contrbutons Ths work, s related to our prevous one presented n [1], whch focused on the development of a method whch mantans accuracy n pose recovery and structure from moton algorthms wthout sacrfcng low cost and power characterstcs of the sensor. Ths was acheved by ntegratng, n the perspectve projecton model, knematc and technologcal parameters whch are both causes of mage deformatons. The resultng algorthm, not only enables accurate pose recovery, but also provdes the nstantaneous angular and translatonal velocty of observed movng objects. Rollng shutter effects whch are consdered as drawbacks are transformed nto an advantage! Ths approach may be consdered as an alternatve to methods whch uses mage sequences to estmate the knematc between vews snce t reduces the amount of data and the computatonal cost (one mage s processed rather than several ones). In a parallel work by Mengast [7] (publshed after the submsson of ths paper), the projecton n rollng shutter cameras s modelled n the case of fronto-parallel moton obtanng equatons whch are smlar to those of Crossed-Slts cameras [13]. To our knowledge, there s no work n the vson communty lterature on takng nto account effects of rollng shutter n pose recovery algorthms nor on computng velocty parameters usng a sngle vew. Indeed, all pose recovery methods ([6], [8], [2], [3], [10]) make the assumpton that all mage sensor pxels are exposed smultaneously. The work done by Wlburn et al. [11] concerned the correcton of mage deformaton due to rollng shutter by constructng a sngle mage usng several mages from a dense camera array. Usng the knowledge of the tme

3 delay due to rollng shutter and the chronograms of release of the cameras, one complete mage s constructed by combnng lnes exposed at the same nstant n each mage from the dfferent cameras. Two man contrbutons are presented n ths paper. Frst, the perspectve projecton model of rollng shutter cameras presented n [1] s mproved by removng the assumpton of small moton durng mage acquston. Ths makes the model more accurate for very fast movng objects. A novel non-lnear algorthm for pose and velocty computaton s then descrbed. It generalzes the bundle adjustment method to the case of movng ponts. Indeed, t s based on non-lnear least-square optmzaton of an error functon defned n mage metrc and expressed wth respect to both pose and velocty parameters (rather than to only pose parameters n classcal approaches). Second, a lnear algorthm for pose and nstantaneous velocty computaton s developed n the partcular case of planar objects. Ths lnear soluton provdes an ntal estmate of the pose and velocty parameters and serves to ntalze the non-lnear algorthm. Secton 3 of ths paper descrbes the process of mage acquston usng a CMOS Rollng Shutter mager. In secton 4, a general geometrc model for the perspectve projecton of 3D pont on a sold movng object s presented. Image coordnates of the pont projectons are expressed wth respect to object pose and velocty parameters and to the tme delay due to mage scannng. Secton 5 deals wth the problem of computng pose and velocty parameters of a movng object, maged by a Rollng Shutter camera, usng pont correspondences. Fnally, experments wth real data are presented and analyzed n secton 6. 3 What s rollng shutter? In dgtal cameras, an mage s captured by convertng the lght from an object nto an electronc sgnal at the photosenstve area (photodode) of a sold state CCD or CMOS mage sensor. The amount of sgnal generated by the mage sensor depends on the amount of lght that falls on the mager, n terms of both ntensty and duraton. Therefore, an on-chp electronc shutter s requred to control exposure. The pxels are allowed to accumulate charge durng the ntegraton tme. Wth global shutter mage sensors, the entre mager s reset before ntegraton. The accumulated charge n each pxel s smultaneously transferred to storage area. Snce all the pxels are reset at the same tme and ntegrate over the same nterval there s no moton artfacts n the resultng mage. Wth a CMOS mage sensor wth rollng shutter, the rows of pxels n the mage are reset n sequence startng at the top and proceedng row by row to the bottom. The readout process proceeds n exactly the same fashon and the same speed wth a tme delay after the reset (exposure tme). The beneft of rollng shutter mode s that exposure and readout are overlappng, enablng full frame exposures wthout reducng the frame rate. Each lne n the mage has the same amount of ntegraton, however the start and end tme of ntegraton s shfted n tme as the mage s scanned (rolled) out of the sensor array as shown n Fg.2.

4 In ths case, f the object s movng durng the ntegraton tme, some artfacts may appear. The faster the object moves the larger s the dstorton. Fg. 2. Reset and readng chronograms n rollng shutter sensor (SILICON IMAGING documentaton). 4 Projectng a pont wth rollng shutter camera Let us consder a classcal camera wth a pnhole projecton model defned by ts ntrnsc parameter matrx [10] K = α u 0 u 0 0 α v v Let P = [X, Y, Z] T be a 3D pont defned n the object frame. Let R and T be the rotaton matrx and the translaton vector between the object frame and the camera frame. Let m = [u, v] T be the perspectve projecton of P on the mage. Notng m = [ m T, 1 ] T and P = [ P T, 1 ] T, the relatonshp between P and m s: s m = K [R T ] P (1) where s s an arbtrary scale factor. Note that the lens dstorton parameters whch do not appear here are obtaned by calbraton [5] and are taken nto account by correctng mage data before usng them n the algorthm. Assume now that an object of known geometry, modelled by a set of n ponts P = [X, Y, Z ] T, undergong a moton wth nstantaneous angular velocty Ω around an nstantaneous axs of unt vector a = [a x, a y, a z ] T, and nstantaneous lnear velocty V = [V x, V y, V z ] T, s snapped wth a rollng shutter camera at an nstant t 0. In fact, t 0 corresponds to the nstant when the top lne of the sensor s exposed to lght. Thus, the lght from the pont P wll be collected wth a delay τ proportonal to the mage lne number on whch P s projected. As

5 llustrated n fgure 3, τ s the tme delay necessary to expose all the lnes above the lne whch collects the lght from P. Therefore, to obtan the projecton m = [u, v ] T of P, the pose parameters of the object must be corrected n equaton 1 by ntegratng the moton durng the tme delay τ. Snce all the lnes have the same exposure and ntegraton tme, we have τ = τv where τ fp v max s the tme delay between two successve mage lne exposures. Thus τ = where fp s the frame perod and v max s the mage heght. Assumng that τ s short enough to consder unform (but not necessarly small) moton durng ths nterval, the object rotaton durng ths nterval s obtaned thanks to the Rodrgues formula: δr = aa T (1 cos (τv Ω)) + Icos (τv Ω) + âsn (τv Ω) where I s the 3 3 dentty matrx and â the antsymetrc matrx of a. The translaton durng the same nterval, expressed n the statc camera frame, s: δt = τv V Thus, equaton 1 can be rewrtten as follows: s m = K [δr R T + δt ] P (2) where R and T represent now the nstantaneous object pose at t 0. Equaton 2 s the expresson of the projecton of a 3D pont from a movng sold object usng a rollng shutter camera wth respect to object pose, object velocty and the parameter τ. One can note that t contans the unknown v n ts two sdes. Ths s due to the fact that coordnates of the projected pont on the mage depend on both the knematcs of the object and the mager sensor scannng velocty. Fg. 3. Perspectve projecton of a movng 3D object: due to the tme delay, ponts P 0 and P 1 are not projected from the same object pose.

6 5 Computng the nstantaneous pose and velocty of a movng object In ths secton, we assume that a set of rgdly lnked 3D ponts P on a movng object are matched wth ther respectve projectons m measured on an mage taken wth a calbrated rollng shutter camera. We want to use ths lst of 3D-2D correspondences to compute the nstantaneous pose and velocty of the object at t Non-lnear method for 3D objects In the general case, the scale factor of equaton 2 can be removed as follows: R u = α (1) P +T (x) u R (3) R v = α (2) v R (3) P +T (z) P +T (y) P +T (z) + u 0 = ξ (u) (R, T, Ω, a, V ) + v 0 = ξ (v) (R, T, Ω, a, V ) (3) where T (x,y,z) are the components of T = T + δt and R (j) s the j th row of R = δr R. Subsdng the rght term from the left term and substtutng u and v by mage measurements, equaton 3 can be seen as an error functon wth respect to pose and velocty (and possbly τ) parameters: u ξ (u) (R, T, Ω, a, V ) = ɛ (u) v ξ (v) (R, T, Ω, a, V ) = ɛ (v) We want to fnd (R, T, Ω, a, V ) that mnmze the followng error functon: ɛ = n =1 [ ] 2 [ 2 u ξ (u) (R, T, Ω, a, V ) + v ξ (v) (R, T, Ω, a, V )] (4) Ths problem wth 12 unknowns can be solved usng a non-lnear least square optmzaton f at least 6 correspondences are avalable. Ths can be seen as a bundle adjustment wth a calbrated camera. Note that, n our algorthm, the rotaton matrx R s expressed by a unt quaternon representaton q(r). Thus, an addtonal equaton, whch forces the norm of q(r) to 1, s added. It s obvous that ths non-lnear algorthm requres an ntal guess to converge towards an accurate soluton. 5.2 Lnear method for planar objects In ths secton, a lnear soluton whch may yeld an ntal guess of the pose and velocty parameters that can ntalze the non-lnear algorthm s developed. Assumng that τ s short enough to consder small and unform moton durng ths nterval, equaton 1 can be rewrtten, as n [7], as follows: [( ) ] s m = K I + τv ˆΩ R T + τv V p (5)

7 where ˆΩ s the antsymetrc matrx assocated to Ω = [ Ω (x), Ω (y), Ω (z)] T. When ponts p are all coplanar, the projecton equaton 1 becomes a projectve homography. By choosng an adequate object frame, all ponts can be wrtten p = [X, Y, 0] T. Notng p = [X, Y, 1] T, the classcal projecton equaton s ([12]): s m = H p (6) where H = K [r 1 r 2 T ] wth r j the j th column of R. As for the 3D object case, the velocty parameters are ntegrated n the projecton equaton as follows: s m = H p + τv D p (7) where D = K [ω 1 ω 2 V ] wth ω j the j th column of ω = ˆΩR. From equaton 7 one can derve a cross product whch must be null: whch yelds the followng equaton: m (H p + τv D p ) = 0 Ax = 0 (8) where A = [ ] T p 0 T u p T τv p T τv 0 T τv u p T 0 T p T v p T τv 0 T τv p T τv 2 p T s a 18 2n matrx and x = [ h 1 T h 2 T h 3 T d 1 T d 2 T d 3 T ] T s the unknown vector wth h j, d j beng the j th columns of respectvely H and D. Equaton 8 can be solved for x usng sngular value decomposton (SVD) as explaned n [4]. Once x s computed, the pose parameters are derved, followng [12], as follows: r 1 = λ h K 1 h 1, r 2 = λ h K 1 h 2, r 3 = r 1 r 2, T = λ h K 1 h 3 (9) where λ h = 1 K 1 h 1. The translatonal velocty vector s obtaned by: V = λk 1 d 3 (10) and angular velocty parameters are obtaned by frst computng columns 1 and 2 of matrx ω: ω 1 = λ d K 1 d 1, ω 2 = λ d K 1 d 2 (11) 1 where λ d = K 1 d 1, and then extractng Ω as follows: Ω (x) = ω 12R 12 ω 22 R 11 R 32 R 11 R 31 R 12, Ω (y) = ω 11R 22 ω 21 R 21 R 31 R 22 R 32 R 21, Ω (z) = ω 11R 32 ω 21 R 31 R 31 R 22 R 32 R 21 (12)

8 6 Experments The am of ths expermental evaluaton s frst to llustrate our pose recovery algorthm accuracy n comparson wth classcal algorthms under the same acquston condtons, and second, to show ts performances as a velocty sensor. The algorthm was tested on real mage data. A reference 3D object wth whte spots was used. Sequences of the movng object at hgh velocty were captured wth a Slcon Imagng CMOS Rollng Shutter camera SI1280M-CL, calbrated usng the method descrbed n [5]. Acquston was done wth a resoluton and at a rate of 30 frames per second so that τ = s. Image pont coordnates were accurately obtaned by a sub-pxel accuracy estmaton of the whte spot centers and corrected accordng to the lens dstorton parameters. Correspondences wth model ponts were establshed wth a supervsed method. The pose and velocty parameters were computed for each mage usng frst our algorthm, and compared wth results obtaned usng the classcal pose recovery algorthm descrbed n [5]. In the latter, an ntal guess s frst computed by the algorthm of Dementhon [2] and then the pose parameters are accurately estmated usng a bundle adjustment technque. Fgure 4 shows mage samples from a sequence where the reference object was moved followng a straght ral, forcng ts moton to be a pure translaton. In the frst and last mages of the sequence, the object was statc. Pose parameters correspondng to these two statc vews were computed accurately usng the classcal algorthm. They serve as ground-truth values to valdate our algorthm when velocty s null. The reference object trajectory was then assumed to be the 3D straght lne relatng the two extremtes. Table 1 shows the RMS pxel reprojecton error obtaned usng the pose computed wth the classcal algorthm and a classcal projecton model from the one hand-sde, and the pose computed wth our algorthms and the rollng shutter projecton model from the other hand-sde. Column 2 shows results obtaned wth the lnear algorthm usng only nne coplanar ponts of the pattern. Note that these results are obtaned usng the mnmum number of correspondences requred from the lnear algorthm and can thus be mproved. Anyhow, even under these condtons, the method remans accurate enough to correctly ntalze the non-lnear algorthm. Results n columns 3 and 4 show errors obtaned usng respectvely a classcal algorthm and our non-lnear algorthm. One can see that errors obtaned wth statc object vews are smlar. However, as the velocty ncreases, the error obtaned wth the classcal algorthm becomes too mportant whle the error obtaned wth our algorthm remans small. Let us now analyze pose recovery results shown n fgure 5. The left-hand sde of ths fgure shows 3D translatonal pose parameters obtaned by our non-lnear algorthm and by the classcal algorthm (respectvely represented by square and *-symbols). Results show that the two algorthms gve apprecably the same results wth statc object vews (frst and last measurements). When the velocty ncreases, a drft due to the dstortons appears n the classcal algorthm results whle our algorthm remans accurate (the 3D straght lne s accurately reconstructed by pose samples) as t s llustrated on Table 2 where are represented

9 Fg. 4. Image samples of pure translatonal moton. Table 1. RMS re-projecton error (pxel). Lnear algorthm Classcal algorthm Non lnear algorthm Image number RMS-u RMS-v RMS-u RMS-v RMS-u RMS-v dstances between computed poses wth each algorthm and the reference trajectory. Table 3 presents computed rotatonal pose parameters. Results show the devaton of computed rotatonal pose parameters from the reference orentaton. Snce the moton was a pure translaton, orentaton s expected to reman constant. As one can see, a drft appears on classcal algorthm results whle our algorthm results show a very small devaton due only to nose on data. Table 2. Dstances from computed poses to reference trajectory (cm). Image number Classcal algorthm Our algorthm Another result analyss concerns the velocty parameters. Fgure 5 shows that the translatonal velocty vector s clearly parallel to the translatonal axs (up to nose nfluence). Table 4 represents magntude of computed velocty vectors n comparson wth measured values. These reference values were obtaned by dvdng the dstance covered between each two successve mages by the frame perod. Ths gves estmates of the translatonal velocty mean value durng each frame perod. Results show that the algorthm recovers correctly acceleraton, deceleraton and statc phases. Table 5 represents computed rotatonal velocty

10 z z y y x x Fg. 5. Pose and velocty results: reconstructed trajectory (left mage), translatonal velocty vectors (rght mage). Table 3. Angular devaton of computed poses from reference orentaton (deg.). Image number Dementhon s algorthm our algorthm parameters. As expected, the velocty parameter values are small and only due to nose. Table 4. Computed translatonal velocty magntude n comparson wth measured velocty values (m/s) Image number Measured values Computed values In the second experment, the algorthm was tested on coupled rotatonal and translatonal motons. The prevously descrbed reference object was mounted on a rotatng mechansm. Its crcular trajectory was frst reconstructed from a set of statc mages. Ths reference crcle belongs to a plan whose measured normal vector s N = [0.05, 0.01, 0.98] T. Thus, N represents the reference rotaton axs. An mage sequence of the movng object was then captured. Fgure 6 shows samples of mages taken durng the rotaton, where rollng shutter effects appear clearly. The left part of fgure 7 represents the trajectory reconstructed wth a classcal algorthm (*-symbol) and wth our algorthm (square symbol). As for the pure translaton, results show that the crcular trajectory was correctly reconstructed by the poses computed wth our algorthm, whle a drft s observed

11 z z Table 5. Computed rotatonal veloctes (rad/s). Image number our algorthm on the results of the classcal algorthm as the object accelerates. The rght part of the fgure shows that translatonal velocty vectors were correctly orented (tangent to the crcle). Moreover, the manfold of nstantaneous rotaton axs vectors was also correctly orented. Indeed, the mean value of the angles between the computed rotaton axs and N s 0.50 degrees. Results n table 6 shows a comparson of the computed rotatonal velocty magntudes and the values estmated from each two successve mages. Fg. 6. Image samples of coupled rotatonal and translatonal motons y x y x Fg. 7. Pose and velocty results for coupled rotatonal and translatonal moton: reconstructed trajectory (left mage), rotatonal and translatonal veloctes (rght mage).

12 Table 6. Computed and measured rotatonal velocty magntudes (rad/s) Image number Measured values Computed values Concluson and perspectves An orgnal method for computng smultaneously the pose and nstantaneous velocty (both translatonal and rotatonal) of rgd objects was presented. It profts from an nherent defect of rollng shutter CMOS cameras consstng n exposng one after the other the rows of the mage, yeldng optcal dstortons due to hgh object velocty. Consequently, a novel model of the perspectve projecton of a movng 3D pont onto a rollng shutter camera mage was ntroduced. From ths model, an error functon equvalent to collnearty equatons n camera calbraton was defned n the case of both planar and non-planar objects. In the planar case, mnmzng the error functon takes the form of a lnear system, whle n the non-planar case t s obtaned through bundle adjustment technques and non-lnear optmzaton. The approach was valdated on real data showng ts relevance and feasblty. Hence, the proposed method n the non planar case s not only as accurate as smlar classcal algorthms n the case of statc objects, but also preserves the accuracy of pose estmaton when the object s movng. However, n the planar case, the expermental results were only accurate enough to ntalze the non-planar method but these results were obtaned wth the mnmal number of ponts. In addton to pose estmaton, the proposed method gves the nstantaneous velocty usng a sngle vew. Thus, t avods the use of fnte dfferences between successve mages (and the assocated constant velocty assumpton) to estmate a 3D object velocty. Hence, carefully takng nto account rollng shutter turns a low cost mager nto a powerful pose and velocty sensor. Indeed, such an orgnal tool can be useful for many research areas. For nstance, nstantaneous velocty nformaton may be used as evoluton models n moton trackng to predct the state of observed movng patterns. It may also have applcatons n robotcs, ether n vsual servong or dynamc dentfcaton. However, n the latter case, accuracy needs to be quantfed by ndependent means on accurate ground-truth values wthn an evaluaton framework, such as laser nterferometry or accurate hgh-speed mechansms, before the proposed method can serve as a metrologcal tool. From a more theoretcal pont of vew, several ssues open. Frst, the proposed method uses a rollng shutter camera model based on nstantaneous row exposure, but t should be easly extendable to more general models where each pxel has a dfferent exposure tme. One could also magne that an uncalbrated verson of ths method could be derved for applcatons where Eucldean nformaton s not necessary (vrtual/augmented realty or qualtatve moton

13 reconstructon, for nstance). Fnally, another pont of nterest could be the calbraton of the whole system (lens dstorton + ntrnsc parameters + rollng shutter tme) n a sngle procedure. References [1] O. At-Ader, N. Andreff, J. M. Lavest, and P. Martnet. Explotng rollng shutter dstortons for smultaneous object pose and velocty computaton usng a sngle vew. In Proc. IEEE Internatonal Conference on Computer Vson Systems, New York, USA, January [2] D. Dementhon and L.S. Davs. Model-based object pose n 25 lnes of code. Internatonal Journal of Computer Vson, 15(1/2): , June [3] M. Dhome, M. Rchetn, J. T. Lapreste, and G. Rves. Determnaton of the atttude of 3-d objects from a sngle perspectve vew. IEEE Transactons on Pattern Analyss and Machne Intellgence, 11(12): , December [4] R. Hartley and A. Zsserman. Multple Vew Geometry n Computer Vson. Cambrdge Unversty Press, [5] JM. Lavest, M. Vala, and M. Dhome. Do we really need an accurate calbraton pattern to acheve a relable camera calbraton. In Proceedngs of ECCV98, pages , Freburg, Germany, June [6] D. G. Lowe. Fttng parameterzed three-dmensonal models to mage. IEEE Transactons on Pattern Analyss and Machne Intellgence, 13(5): , May [7] M. Mengast, C. Geyer, and S. Sastry. Geometrc models of rollng-shutter cameras. In Proc. of the 6th Workshop on Omndrectonal Vson, Camera Networks and Non-Classcal Cameras, Bejng, Chna, October [8] T. Q. Phong, R. Horaud, and P. D. Tao. Object pose from 2-d to 3-d pont and lne correspondences. Internatonal Journal of Computer Vson, pages , [9] A. J. P. Theuwssen. Sold-state magng wth chargecoupled devces. Kluwer Academc Publshers, [10] R. Y. Tsa. An effcent and accurate camera calbraton technque for 3d machne vson. In Proc. IEEE Conference on Computer Vson and Pattern Recognton, pages , Mam Beach, [11] B. Wlburn, N. Josh, V. Vash, M. Levoy, and M. Horowtz. Hgh-speed vdeography usng a dense camera array. In IEEE Socety Conference on Pattern Recognton (CVPR 04), [12] Z. Zhang. A flexble new technque for camera calbraton. IEEE Transactons on Pattern Analyss and Machne Intellgence, 22(11): , [13] A. Zomet, D. Feldman, S. Peleg, and D. Wenshall. Mosacng new vews: The crossed-slts projecton. IEEE Transactons on Pattern Analyss and Machne Intellgence, 25(6): , 2003.