New dynamic zoom calibration technique for a stereo-vision based multi-view 3D modeling system

Transcription

1 New dynamc oom calbraton technque for a stereo-vson based mult-vew 3D modelng system Tao Xan, Soon-Yong Park, Mural Subbarao Dept. of Electrcal & Computer Engneerng * State Unv. of New York at Stony Brook, Stony Brook, NY, USA ABSTRACT A new technque s proposed for calbratng a 3D modelng system wth varable oom based on mult-vew stereo mage analyss. The 3D modelng system uses a stereo camera wth varable oom settng and a turntable for rotatng an object. Gven an object whose complete 3D model (mesh and texture-map) needs to be generated, the object s placed on the turntable and stereo mages of the object are captured from multple vews by rotatng the turntable. Partal 3D models generated from dfferent vews are ntegrated to obtan a complete 3D model of the object. Changng the oom to accommodate objects of dfferent ses and at dfferent dstances from the stereo camera changes several nternal camera parameters such as focal length and mage center. Also, the parameters of the rotaton axs of the turntable changes. We present camera calbraton technques for estmatng the camera parameters and the rotaton axs for dfferent oom settngs. The Perspectve Projecton Matrces (PPM) of the cameras are calbrated at a selected set of oom settngs. The PPM s decomposed nto ntrnsc parameters, orentaton angles, and translaton vectors. Camera parameters at an arbtrary ntermedate oom settng are estmated from the nearest calbrated oom postons through nterpolaton. A performance evaluaton of ths technque s presented wth expermental results. We also present a refnement technque for stereo rectfcaton that mproves partal shape recovery. And the rotaton axs of mult-vew at dfferent oom settng s estmated wthout further calbraton. Complete 3D models obtaned wth our technques are presented. Keywords: Stereo vson, 3D modelng, dynamc oom calbraton, mult-vew rotaton axs estmaton. INTRODUCTION Recent advances n consumer dgtal cameras have made low-cost 3D modelng systems feasble. Conventonal 3D modelng technques use a fxed oom settng n 3D reconstructon. For objects at dfferent dstances and/or of dfferent ses, a vson system wth varable oom s crtcal for 3D modelng. In the case of fxed oom settng, the relatve postons of lens components are statc. When the oom settng changes, the camera parameters also vary. To extend the fxed oom settng camera model to adjustable oom settngs, several algorthms have been presented. Wlson and Shafer, 2 ntroduced an teratve tral and error procedure n whch four camera parameters are selected. These camera parameters are -- the effectve focal length f, the mage center ( u, v ), and the translaton along the optcal axs T 3. Up to a 5th degree polynomal s used to estmate the camera parameters from fxed sampled ponts. Atena and Zelnsky 3 extended ths calbraton technque to gae detecton under the assumpton that the orentaton of the camera coordnate remans unchanged durng oom change. However when the optcal confguraton of a vson system changes, ths assumpton s not vald, and a tral and error procedure wll be needed to determne the crtcal parameters. There are several man problems n employng varable/dynamc oom n 3D modelng. Frst, many nternal camera parameters vary nonlnearly wth dfferent oom settngs. Ther varatons are too complex to be expressed analytcally, even for a smple lens system. Second, the relaton between the camera coordnate system and the turntable rotaton axs * E-mal: {txan, parksy, mural}@ece.sunysb.edu; Tel: ; WWW:

2 changes as t wobbles. Another dffculty arses from the naccurate and non-lnear mechancal control mechansm of a consumer camera. The resdual error n postonng the oom lens by the drvng motor cannot be gnored. The partal shape of a sngle vew can be reconstructed from stereo mages usng a stereo matchng technque. Stereo mage rectfcaton utles the eppolar geometry to reduce the search dmenson of stereo matchng from 2D to D, and also decreases the possblty of msmatches. In our research, we adopted a compact rectfcaton algorthm for stereo pars proposed by Fusello et al. 4. The rectfcaton s based on the camera s ntrnsc parameters, mutual poston, and orentaton. However, rectfcaton mposes a hgher accuracy requrement on camera parameters. Due to the nonlnearty of lenses and the naccuracy of mechancal parts, parameters from dynamc estmaton are not accurate enough for a perfect rectfcaton. A refnement based on vertcal profle SSD s presented to reconstruct the partal shape from estmated projecton matrx. For regsterng and ntegratng partal shapes, the rotaton axs s estmated wthout further mult-vew calbraton. In ths paper, a full oom calbraton s presented to avod the emprc tral procedure. The error of estmated camera parameters from dynamc oom are analyed. A new rectfcaton refnement technque s proposed to obtan a better partal shape. And the rotaton axs of mult-vew at dfferent oom settng s also estmated wthout further calbraton. Full 3D models usng the estmated rotaton axs are demonstrated. 2. CAMERA CALIBRATION THROUGH PERSPECTIVE PROJECTION Camera calbraton s to fnd a mappng from 3D world frame to 2D mage plane. It can be dvded nto two parts: frst a rotaton and translaton between world frame and camera frame, then a perspectve projecton from camera frame to mage plane. Some of the calbraton algorthms are drect calbraton, Tsa s calbraton 5, and Zhang s calbraton 6. In ths research, the calbraton based on perspectve projecton matrx s adopted 7. However the dynamc oom calbraton method should be of general value, and can be extended to other algorthms. A 3D pont ( X, Y, Z ) n world coordnate s projected to a pont ( u, v ) n the mage plane; the correspondng perspectve projecton matrx (PPM) P can be expressed as: wth t [ u v ] P [ X Y Z ] t x u = f y v = f where x, y, ) s the correspondng 3D pont n the camera frame. ( = () In the case that the calbraton pattern contans more than 6 correspondng world-pont matches, P can be solved. Snce there are 8*6 world-pont matches n our oom calbraton pattern, the least-square mnmaton technque s used to reduce the estmaton error. 3. DYNAMIC ZOOM CALIBRATION The PPM can be decoupled nto an ntrnsc matrx that descrbes the projecton from camera coordnate to mage plane, and an extrnsc matrx whch descrbes the transform from the world coordnate system to the camera coordnate system. The factoraton s expressed as: P = I[ R t] (3) The ntrnsc matrx I depends on the ntrnsc parameters, and has the followng format: (2)

3 where f u, f u α u I = f v (4) v f are focal length n effectve pxel se under u and v drecton of the mage plane, u, ) s the v ( v coordnate of the mage center, andα s aspect rato. The extrnsc matrx descrbes the rotaton and translaton of the camera coordnate system, and can be expressed by a 3*3 rotaton matrx R and a translaton vector t. In the rotaton matrx R, all the 9 elements are not ndependent. They wll be further reduced to 3 ndependent rotaton angles (roll, yaw, ptch) usng 6 orthonormal constrants. In dynamc oom calbraton, a seres of perspectve projecton calbratons are conducted at a set of base ponts. Then perspectve projecton matrces are decomposed as ntrnsc parameters, orentaton angles, and translaton vectors. These parameters change wth dfferent oom postons. They are plotted respectvely n Fg., Fg. 2 and Fg. 3. Camera parameters at an arbtrary ntermedate oom settng are estmated from the nearest calbrated oom postons by nterpolaton. In the fgures we observe that the changes of f, f, t are smlar to the result of Wlson and Atena u v 3, 2, 3. However, for a convergent stereo vson confguraton, the orentaton of the camera coordnate (roll, yaw, ptch) s not a constant anymore, as can be seen n Fg. 2. Moreover, the optcal center of the camera moves not only along the optcal axs, but also shfts n a plane that s perpendcular to the optcal axs. A nonlnear measure K s used as an ndex that ndcates the relatve error between the estmated parameters and the real ones: T p π (, α, α, mα n ) K = (5) N = p where p s the calbrated parameter and π (, α, α, mα n ) s the parameter estmated from dynamc oom. In Table, the nonlnearty measure K s calculated by comparng the estmated camera parameters and the parameters from real calbraton of left and rght cameras at 3 dfferent oom postons. It demonstrates that t, t2, u, v have relatvely large estmaton errors. Ths s caused by the non-lnearty of the lens desgn f u [mm] 4 f v [mm] (a) (b)

4 u [pxel] v [pxel] (a) (c) f (b) u Fgure. Intrnsc parameters vs. oom poston f (c) u and (d) v change wth dfferent oom postons v (d) roll [degree] yaw [degree] (a) (b) ptch [degree] (c) Fgure 2. Orentaton parameters vs. oom poston (a) Roll (b) Yaw and (c) Ptch change wth dfferent oom postons

5 t [mm] -46 t 2 [mm] (a) (b) t 3 [mm] (c) Fgure 3. Translaton parameters vs. oom poston (a) t (b) t and (c) 2 t 3 change wth dfferent oom postons K [%] f u f v u v roll yaw ptch t t 2 t 3 Left Rght Table. Non-lnear measure K for dfferent parameters Zoom settng ranges from step number 7 to 9 at ntervals of steps, N = 3 4. RECTIFICATION Whle matchng stereo mages, rectfcaton s used to reduce computaton and the possblty of msmatchng. Rectfcaton based on PPM s brefly ntroduced here 4. In order to have horontal eppolar lnes, the baselne must be parallel to the new X axs of both cameras. In addton, correspondng ponts must have the same vertcal poston (Y coordnate). Consequently, the poston of new optcal centers s the same as that n the old ones after sutable rotatons, and ntrnsc parameters are the same for both cameras. Therefore, the new projecton matrces wll dffer only n ther optcal centers. Let us wrte the new PPMs n terms of ther QR factoraton:

6 Pn = I[ R Rc] (6) Pn 2 = I[ R Rc2 ] The ntrnsc parameter matrx I s same for both new projectve matrces. The rotaton matrx R s the same for both PPMs. To rectfy the left mage and the rght mage, we need to compute a transformaton mappng of the mage plane P = [ Q q] onto the mage plane P n = [ Qn qn ]. We wll see that the sought transformaton s the collnearty gven by 3*3 matrx T Q n Q. The same result apples to the rght mage. = o For any 3D pont w, we can connect t to a correspondng pont m on the mage plane by a PPM P. Thus for the same 3D pont w, there are two ponts on the mage plane that correspond to before and after rectfcaton respectvely. mo = P ow mn = Pn w (7) Then the optcal rays that connect mage ponts m o, m n, and the optcal center are descrbed n parametrc form as: w = c + λqo mo λ R (8) w = c + λnqn mn λn R From Eqs. (8), we have: mn = λqn Qo mo λ R (9) where λ s an arbtrary scale factor. Reconstructon of 3D ponts by trangulaton can be performed from the rectfed mage drectly, usng P n, P. n2 5. RECTIFICATION REFINEMENT Rectfcaton uses PPM from calbraton or estmaton as a startng pont. In a successful rectfcaton, the vertcal dsparty between the left and the rght mage par should be ero. For ths, we need an accurate PPM. However, n the case of dynamc oom, due to the non-lnearty of the lens and mechancal mechansms, there wll be errors n the estmated camera parameters. When the projecton matrx for the dynamc oom case s not accurate enough for rectfcaton, the left and rght mage par may have vertcal shft of up to several pxels. Ths problem may be solved by ncreasng the search range of stereo matchng at the cost of dramatcally ncreasng the computaton and the possblty of msmatches. Then the advantage of rectfcaton s dmnshed. A rectfcaton refnement s needed to speed up computaton and reduce the stereo match error. An analytcal rectfcaton refnement s very dffcult due to the lack of constrants, f not mpossble. In Table, we see that the man error source of oom calbraton s t, t2, u, v due to optcal lens desgn. Based on ths observaton, a refnement technque s ntroduced based on vertcal profle Sum of Squared Dfference (SSD). The vertcal drecton profles of left and rght cameras are obtaned by projectng mages onto the Y-axs: Y ( ) = l Y ( ) = r N j= N j= I (, j) l I (, j) r where I (, j) s the gray level of mages. Then an SSD s computed n a wndow of wdth W. The vertcal mage shft s calculated by Eqs. (), (2). The refned rectfcaton s obtaned by movng one mage relatve to the other one n the vertcal drecton by d. W c ( ) = ψ ( Y ( ), Y ( + k)) () k = W l r ()

7 d = mn[c()] (2) Fg. 4 (a) and (b) show mages of a toy dog recorded by the left and the rght cameras. The se of mages s 96*28 pxels. Ther normaled vertcal profle s plotted n Fg. 4(c). The vertcal lne dfference from Y-SSD s shown n Fg. 4(d). The wdth of the SSD wndow s 8 pxels. There are three areas. Area and Area 3 are nose-domnated due to non-unform llumnaton. Area 2 s object-domnated, and the vertcal lne dfference s a constant (5 lnes). The partal shapes before and after rectfcaton refnement are shown n Fg. 5 (a) and (b) respectvely. (a) (b) (c) (d) Fgure 4. Rectfcaton refnement (a) Left mage (b) Rght mage (c) Vertcal profle of left and rght mage (d) SSD result

8 Fgure 5. Partal shapes before and after rectfcaton refnement 6. MULTI-VIEW ROTATION AXIS ESTIMATION For a complete 3D model, sngle partal shape from one vew s not enough. A rotaton stage s used n our stereo vson system to rotate the object. It s equvalent to fxng the object and rotatng the stereo camera. In Fg. 6, the full 3D model are ntegrated from 8 partal shapes, and each partal shape s obtaned from dfferent vews whch range from Vew to Vew 8. Snce the partal shapes are referenced to dfferent camera coordnate systems, t s necessary to regster the partal shapes. Mult-vew calbraton descrbes the poston and orentaton of the rotaton axs around whch the dfferent partal shapes are measured. The rotaton axs s expressed by a turntable matrx (4*4). A mult-vew calbraton method that s smlar to the camera perspectve projecton calbraton has been developed. The result from mult-vew calbratons can be expressed as a turntable matrx: R = cs t T cs (3) As descrbed n Sec. 3, the optcal center of cameras wobbles around the optcal axs wth dfferent oom settngs. As shown n Fg. 6, the optcal center of the left and rght camera shft from O,O to 2 O n, O wth the change of oom 2n poston. That means the turntable matrx, whch descrbes the rotaton axs wth respect to the optcal center, also changes. However the orgn of the world coordnate s stll fxed and can be used as the connecton between the eppolar geometry before and after oom change. The turntable matrx of the dynamc oom can be calculated from a calbrated rotaton matrx T, a calbrated PPM P, s and the estmated PPM P from dynamc oom. The new turntable matrx s obtaned by: Rn tn T = (4) n where: R n = ( Rs R ) R (5) cs t n = ( ts t ) + t (6) cs and R s, R, t s, t are decomposed from the calbrated PPM P and estmated PPM s P : Ps = I[ Rs t s ] (7) P = I[ R t ]

9 Fgure 6. Mult-vew ntegraton and rotaton axs estmaton 7. RESULTS Experments were conducted on the Stonybrook VIson System (SVIS-3). As shown n Fg. 7, SVIS-3 system s composed of a dgtal stereo camera, a rotaton stage, and lght sources. The dgtal stereo camera s made up of two vertcally-mounted Olympus C-4 dgtal cameras. Two checkerboard planes are mounted perpendcularly as a calbraton pattern. Olympus C-4 has 3 oom levels that range from 65 to 95 steps. The full 3D model after mult-vew calbraton estmaton s shown n Fg. 8. We see that the fnal full 3D models are qute good. Fgure 7. SVIS-3 3D modelng system

10 (a) (b) (c) Fgure 8. 3D models by estmated rotaton matrx (a) and (c) Mesh models of test objects. (b) and (d) Correspondng texture models of test objects 8. CONCLUSION We have presented a calbraton method for estmatng the nternal camera parameters that determne the perspectve projecton matrx under dynamc oom settng. We have also presented a calbraton method for estmatng the parameters of the rotaton axs of a turntable used for obtanng mult-vew stereo mages for 3D modelng. We have presented a method for refnng the results of stereo mage rectfcaton. Our methods are mplemented and evaluated on an actual camera system used n 3D modelng. Expermental results show that our method s very useful for enablng stereo based 3D modelng systems to ncorporate the varable oom feature. (d) REFERENCES. Reg G. Wllson, Steven A. Shafer, A perspectve projecton camera model for oom lenses, Proceedngs Second Conference on Optcal 3-D Measurement Technques, Zurch Swterland, Reg G. Wllson, Ph.D. thess, Modelng and calbraton of automated oom lenses, Department of Electrcal and Computer Engneerng, Carnege Mellon Unversty, January 994.

11 3. Rowel Atena, Alex Zelnsky, A practcal oom camera calbraton technques: an applcaton of actve vson for human-robot nteracton, Proceedngs 4th IEEE Internatonal Conference on Multmodal Interfaces, Pttsburgh, Pennsylvana, Fusello, E. Trucco, and A. Verr., A compact algorthm for rectfcaton of stereo pars, Machne Vson and Applcatons, Vol. 2(), pp. 6 22, R. Y. Tsa. A versatle camera calbraton technque for hgh-accuracy 3D machne vson metrology usng off-theshelf TV cameras and lenses, IEEE Journal of Robotcs and Automaton, RA-3(4), pp , Zhengyou Zhang, A flexble new technque for camera calbraton, IEEE Transacton on Pattern Analyss and Machne Intellgence, Vol. 22, pp , Emamuele Trucco, Alessandro Verr, Introductory technques for 3-D computer vson, Prentce Hall, 998.