Calibration of an Articulated Camera System

Transcription

1 Calbraton of an Artculated Camera System CHEN Junzhou and Kn Hong WONG Department of Computer Scence and Engneerng The Chnese Unversty of Hong Kong {jzchen, Abstract Multple Camera Systems (MCS) have been wdely used n many vson applcatons and attracted much attenton recently. There are two prncple types of MCS, one s the gd Multple Camera System (MCS); the other s the Artculated Camera System (ACS). In an MCS, the relatve poses (relatve -D poston and orentaton) between the cameras are nvarant. Whle, n an ACS, the cameras are artculated through movable jonts, the relatve pose between them may change. Therefore, through calbraton of an ACS we want to fnd not only the relatve poses between the cameras but also the postons of the jonts n the ACS. Although calbraton methods for MCS have been extensvely developed durng the past decades, the studes of ACS calbraton are stll rare. In ths paper, two ACS calbraton methods are proposed. The frst one uses the feature correspondences between the cameras n the ACS. The second one requres only the ego-moton nformaton of the cameras and can be used for the calbraton of the nonoverlappng vew ACS. In both methods, the ACS s assumed to have performed general transformatons n a statc envronment. The effcency and robustness of the proposed methods are tested by smulaton and real experments. In the real experment, the ntrnsc and extrnsc parameters of the ACS are calbrated usng the same mage sequences, no extra data capturng step s requred. The correspondng trajectory s recovered and llustrated usng the calbraton results of the ACS. To our knowledge, we are the frst to study the calbraton of ACS.. Introducton Calbraton of a Multple Camera System (MCS) s an essental step n many computer vson tasks such as SLAM (Smultaneous Localzaton and Map), survellance, stereo and metrology,, 7,. Both the ntrnsc and extrnsc parameters of the MCS are requred to be estmated before the MCS can be used. The ntrnsc parameters 9 descrbe the nternal camera geometrc and optcal character- Fgure. An obot wth Four Cameras Attached on It, Where the Cameras are Artculated. stcs of each camera n the MCS. In a gd Multple Camera System (MCS), the cameras are fxed to each other. The extrnsc parameters 5 of an MCS descrbe the relatve pose (the relatve -D poston and orentaton, totally, sx degrees of freedom) between the cameras n the MCS. Calbraton methods of the ntrnsc parameters of a camera are well establshed 5, 9. Calbraton methods for the extrnsc parameters of an MCS are also wdely studed. For nstance, Maas proposed an automatc MCS calbraton technque wth a movng reference bar whch can be seen by all cameras. Antone and Teller developed an algorthm whch recovers the relatve poses of cameras by overlappng portons of the outdoor scene. Baker and Alomonos presented MCS calbraton methods usng calbraton objects such as a wand wth LEDs or a rgd board wth known patterns, 4. Dornaka proposed a stereo rg selfcalbraton method by the monocular eppolar geometres and geometrc constrants of a movng MCS, n whch only the feature correspondences between the monocular mages of each camera are requred 8. In hand-eye calbraton, t s demonstrated that when a sensor s mounted on a movng robot hand, the relatonshp between the sensor coordnate system and hand coordnate system can be calculated by the moton nformaton of the hand and the sensor. One example of usng knematc nformaton of the cameras for MCS s dscussed by Casp and Iran

2 6, they ndcated that f the cameras of a non-overlappng vew MCS are close to each other and share a same projecton center, ther recorded mage sequences can be algned effectvely by the estmated transformatons nsde each mage sequence. However, n some types of MCS, the relatve poses between the cameras are not fxed, hence the calbraton methods for the MCS cannot be used drectly. In Fgure, a novel applcaton of lmb pose estmaton by attachng cameras on the arms of a robot s shown. On each arm of the robot, two cameras are artculated to each other through the elbow jont of the arm. When the robot moves, the relatve pose between the cameras may change, whle, the coordnate of the elbow jont refers to each camera attached on the correspondng arm s nvarant. In ths paper, such a type of MCS s named as Artculated Camera System (ACS). The jont of the elbow s named as the jont n the ACS. ACSs can be easly found n the real world, such as camera systems attached on human, robots and anmals. Before usng an ACS, t has to be calbrated. However, there are stll some unsolved problems: () In an ACS wth overlappng vew, tradtonal calbraton methods cannot estmate the postons of the jonts n the ACS. () In a nonoverlappng vew ACS, nether the postons of the jonts n the ACS nor the relatve poses between the cameras n the ACS can be estmated by tradtonal calbraton methods. These consderatons n mnd motvate us to develop the technologes n ths paper. The rest of ths paper are organzed as follows: Secton and analyss the constrants n a movng ACS. The correspondng calbraton methods are proposed. Secton 4 and 5 evaluate the proposed method by smulaton and real experment. In secton 6, a bref concluson and the future plan are presented.. Calbraton of ACS wth Overlappng Vews system of camera A, C B the coordnate system of camera B. Suppose there are enough feature correspondences between the cameras so that the pose of C A and C B referrng to the same coordnate system C W can be estmated. Therefore, the relatve pose between C A and C B s known. We want to fnd the poston of O n the ACS. Let H AW and H BW be the Eucldean transformaton matrxes descrbe the C A and C B refer to C W, so that for any pont P : P A = H AW P W = P B = H BW P W = AW T AW BW T BW P W () P W (), where s the rotaton matrx, T s vector, P W, P A and P B are the homogenous coordnates of the -D Pont P refer to C W, C A and C B respectvely. Accordng to equatons () and (): P W = H AW P A = H BW P B () H AW P A H BW P B = (4) T AW T AW T AW PA T BW T BW T BW PB = (5) T AW P A T BW P B = T AWT AW T BWT BW (6), where T s the transpose of. Suppose the ACS performed n transformatons, for the -th transformaton of the ACS, accordng to equaton (6): ( AW) T PA ( BW) T PB = ( AW) T TAW ( BW) T TBW (7) Let Õ = ŌA T ŌB T T, where Ō A and ŌB are the coordnates of the jont O refer to C A and C B respectvely. Equaton (7) can be rewrtten as: ( AW ) T ( BW )T Õ = ( AW) T T AW ( BW) T T BW (8) Fgure. An Artculated Camera System wth Overlappng Vews Suppose two rgd objects are artculated at jont O and two cameras (camera A and B) are fxed on the two rgd objects respectvely (See Fgure ). Let C A be the coordnate Snce camera A and B are fxed on the artculated rgd objects, Õ s nvarant durng the transformaton of the ACS. The transformatons ( AW, BW, T AW and T BW for... n) of the camera coordnate systems are calculated by the projected mage sequences. We propose that Õ can be estmated by a least squares method, when the ACS has moved to many dfferent postons and captured enough samples of AW, BW, T AW and T BW.

3 Let M A = ( A I)T, ( A I)T,..., ( n A I)T T, T A = (T A )T, (T A )T,..., (T n A )T T, we have: M A Ō A = T A () Snce the transformatons ( A and T A,...n) of camera A can be calculated by the projected mage sequence. We propose ŌA can be estmated by a least squares method. Smlarly, Ō B can also be estmated. Therefore, O A and O B are recovered. Fgure. A Non-overlappng Vew Artculated Camera System. Calbraton of Non-Overlappng Vew ACS In many stuatons, there s no overlappng vew between the cameras n an ACS. And the lack of common features makes the calbraton method proposed n secton become nvald (See Fgure ). Moreover, snce the relatve pose between the cameras n the ACS cannot be estmated by the overlappng vews, the calbraton of the relatve poses between the non-overlappng vew cameras s also requred. In ths secton, a calbraton method based on the ego-moton nformaton of the cameras n an ACS s dscussed... ecoverng the Poston of the Jont efers to the Cameras n the ACS Let CA nt and CB nt be the coordnate systems of camera A and B respectvely at the ntal state (tme t = ). Suppose the ACS performs n transformatons. Snce the coordnate of the jont O refers to camera A s fxed durng the transformaton of the ACS. At tme t =, we have: O A = H AO A = A T A O A (9), where H A s the Eucldean transformaton matrx of camera A at tme refers to CA nt. A and T A descrbe the orentaton and orgn of camera A at tme refer to CA nt. Also O A s the coordnate of pont O at ntal state refers to CA nt, and O A s the coordnate of pont O at tme refers to CA nt. If the poston of the jont O refers to CA nt s fxed durng the transformatons of the ACS, we have: OA = O A,,...,n. For -th transformaton of the ACS, accordng to equaton (9): O A = H A O A = A T A O A () ( A I)ŌA = T A ().. The Unqueness of the Jont Pose Estmaton If the dfferent segments of the artculated camera system (ACS) are connected by D rotatonal jonts (connected by pont rotatonal jonts) and the ACS can perform general transformatons, the soluton of the jont pose estmaton s unque: For the jont pose estmaton method usng specal moton (n secton.). Suppose the soluton of the jont pose estmaton s not unque, there must exst at lest two dfferent D ponts O and O satsfy equaton (). We have: M A O = T A and M A O = T A. Therefore, any pont P = so +( s)o wll also satsfy equaton (), where s s an arbtrary scalar. Accordng to the defnton of P, P s the pont on the lne passng through the ponts O and O. Snce P satsfy equaton () represents that the poston of the P refers to the camera n the ACS s nvarant durng the transformaton of the ACS, t means the dfferent segments of ACS are connected by the D rotatonal axs nstead of the D rotatonal jonts. The poston of the ponts on the D rotatonal axs refer to the camera n the ACS s nvarant durng the transformaton of the ACS. However, t conflcts wth the assumpton. Smlarly, the unqueness of the jont pose estmaton method usng overlappng vews (n secton ) can also be verfed... ecoverng the elatve Pose Between the Cameras of the Non-overlappng vew ACS Let H BA be the Eucldean transformaton matrx between CA nt and Cnt B, so that for any pont P : P B = H BA P A = BA T BA P A = H BA P A (), where P A and P B are the homogenous coordnate of Pont P refer to CA nt and Cnt B respectvely. The relatve pose ( BA and T BA ) between CA nt and CB nt s defned as: BA = T BA (4) T BA = T BA T BA (5)

4 Let OB be the coordnate of jont O at tme refer to CB nt. Snce the coordnate of the jont O refer to camera B s nvarant: OB = B TB O B = B TB BA T BA O A = B BA B T BA + TB O A (6) Accordng to equatons (9) and (): OB = H BAOA BA T = BA = A T A BA A BAT A + T BA Accordng to equatons (6) and (7): B BA B T BA + TB ŌA BA = A BATA + T BA ŌA B BA Ō A + B T BA + TB BA = AŌA + BA TA + T BA O A O A (7) (8) (9) B BA Ō A + BT BA BA AŌA BA T A + T B T BA = () Snce ŌA can be estmated by the method dscussed n secton., the BA and T BA can be estmated by a least square method, when the ACS perform enough general motons. In our smulaton and real experment, the estmated BA s refned by a method dscussed n 4. Then the roll, ptch and yaw correspondng to the BA are estmated accordng to the defnton of the rotaton matrx 9. Let BA = M(r, p, y), where r p and y are the correspondng roll, ptch and yaw of BA, M s a functon from roll, ptch and yaw to the correspondng rotaton matrx. Then, the r, p, y, T BA and ŌA are optmzed by mnmzng the nonlnear error functon: n E(r, p, y, T BA, O A ) = ( BM(r, p, y)ōa + BT BA = M(r, p, y) AŌA M(r, p, y)t A + T B T BA ) () usng a Levenberg-Marquardt method. Fnally, the BA s recovered from the optmzed r, p and y. The relatve pose between the CA nt and Cnt B s calculated by equatons (4) and (5). 4. Smulaton In ths secton, the proposed calbraton methods are evaluated wth synthetc transformaton data. 4.. Performance w.r.t. Nose n Transformaton Data Setup and Notatons: In each test, one ACS wth cameras and jont s generated randomly. In whch, O A meters, O B meters. The generated ACS performs random transformatons. Performance of the Calbraton Method for ACS wth Overlappng Vews: In the frst smulaton, the proposed algorthm s tested tmes. Zero mean Gaussan nose s added to the transformaton data of the cameras. The confguraton, nput and output of our smulaton system are lst as Table. Snce we assume there are overlappng vews between the two cameras, the relatve pose between them can be estmated by many exstng methods as dscussed n secton. Only the performance of jont pose estmaton s evaluated n our smulaton. The error of jont estmaton are computed by: Err = ŌA ˆŌ A ŌA + ŌB ˆŌ B ŌB (), where ŌA s the ground truth, ˆŌ A s the estmated poston of jont O refer to camera A. Smlarly, Ō B s the ground truth, ˆŌ B s the estmated poston of jont O refer to camera B. The correspondng results are shown n Fgure 4 and 5. Table. Confguraton, Input and Output Confguraton No. of Cameras n the ACS No. of Jonts n the ACS andom transformatons per test (n) Number of tests Input ( =...n) otatons of cameras ( AW, BW ) Translatons of cameras (TAW, T BW ) Zero Mean Gaussan nose: σ rot.4 and σ trans.meters Output Mean error of jont pose estmaton STD error of jont pose estmaton Performance of the Calbraton Method for Non- Overlappng Vews ACS: In the second smulaton, frstly, the pose of the jont s fxed refers to CA nt durng the transformatons of the ACS. The pose of the jont refers to the camera A (O A ) s calbrated by the transformatons of camera A. Smlarly, O B s calbrated. Then, the ACS performs several general transformatons (the jont s not needed to

5 Mean Error of Jont Pose Estmaton otaton Nose (degree) Mean Error of Jont Pose Estmaton Translaton Nose (meter) Fgure 4. Mean Error of Jont Poston Estmaton STD Error of Jont Pose Estmaton otaton Nose (degree) STD Error of Jont Pose Estmaton Translaton Nose (meter).. Table. Confguraton, Input and Output Confguraton No. of Cameras n the ACS No. of Jonts n the ACS andom transformatons per test (n) Number of tests Input ( =...n) Transformatons wth fxed jont pose: otatons of cameras ( A, B ) Translatons of cameras (TA, T B ) General transformatons: otatons of cameras ( A, B ) Translatons of cameras (TA, T B ) Zero Mean Gaussan nose: σ rot.4 and σ trans.meters Output Mean error of jont pose estmaton STD error of jont pose estmaton Mean error of relatve translaton estmaton STD error of relatve translaton estmaton Mean error of relatve rotaton estmaton STD error of relatve rotaton estmaton Mean Error of Jont Pose Estmaton Fgure 5. STD Error of Jont Poston Estmaton be fxed refer to CA nt ), the relatve pose between the cameras are calbrated usng the estmated jont pose and the transformatons of the cameras. The confguraton, nput and output of the smulaton system are lsted as Table. The error of jont pose, relatve rotaton, relatve translaton estmaton are calculated by equaton (), () and (4) respectvely. Fgure 6 and 7 show the results of jont pose estmaton. Compare wth the calbraton method usng the overlappng vews, the calbraton method usng specal motons s more accurate. The mean and STD error of the relatve rotaton and translaton estmaton are presented n Fgure 8, 9, and. The proposed algorthms are shown to be stable, when the zero mean Gaussan nose from to.4 s added to the roll, ptch and yaw of the rotaton data, and the zero mean Gaussan nose from to. meters s added to the translaton data. Err = roll roll + ptch ptch + yaw ŷaw () Mean Error of Jont Pose Estmaton otaton Nose (degree) Translaton Nose (meter) Fgure 6. Mean Error of Jont Poston Estmaton STD Error of Jont Pose Estmaton x otaton Nose (degree) STD Error of Jont Pose Estmaton Translaton Nose (meter).. Err = T AB ˆT AB T AB (4) Fgure 7. STD Error of Jont Poston Estmaton

6 Mean Error of elatve otaton Estmaton STD Error of elatve Translaton Estmaton Mean Error of elatve otaton Estmaton (degree).5.5 otaton Nose (degree) Translaton Nose (meter) STD Error of elatve Translaton Estmaton.5..5 otaton Nose (degree) Translaton Nose (meter) Fgure 8. Mean Error of elatve otaton Estmaton Fgure. STD Error of elatve Translaton Estmaton STD Error of elatve otaton Estmaton (degree) otaton Nose (degree) STD Error of elatve otaton Estmaton Translaton Nose (meter) Fgure 9. STD Error of elatve otaton Estmaton Mean Error of elatve Translaton Estmaton otaton Nose (degree) Mean Error of elatve Translaton Estmaton Translaton Nose (meter) Fgure. Mean Error of elatve Translaton Estmaton 5. eal Experment In the real experments, an ACS wth two cameras (Cannon PowerShot G9) s set up as Fgure (a). The ntrnsc parameters of each camera n the ACS are calbrated by Bouguet s mplementaton ( Camera Calbraton Toolbox for Matlab ) of 5. Snce the Bouguet s Toolbox can also estmate the pose nformaton of the camera, the transformatons of each camera are calculated usng the same mage sequence for the ntrnsc calbraton smultaneously. No addtonal mages nor manual nput s requred n the real experments Calbraton of the Pose of the Jont n Each Camera By Overlappng Vews (Algorthm I): In the frst real experment, the two cameras n the ACS observe the same checker plane and record mages smultaneously. The two cameras are free to move durng the transformaton of the ACS. Two mage sequences (Q and Q ) are recorded, each sequence conssts of 5 mages of sze 6. The estmated jont pose are lst n Table as algorthm I. By Fxed-Jont Motons (Algorthm II): In the second real experment, the jont of the ACS s fxed refers to the world coordnate system durng the transformaton of the ACS. The two cameras do not need to vew the same checker plane. And each camera records the mage sequence ndependently. Two mage sequences (Q and Q 4 ) are recorded, each sequence conssts of mages of sze 6. The camera pose of the frst mage s selected as the ntal pose to generate the transformaton sequence of each camera. The estmated jont pose are lst n Table as algorthm II. The poses of the jont refer to the two cameras n the ACS are also estmated manually for comparson purpose. Snce the camera pose of any mage n each mage sequence can be chosen as the ntal camera pose (see secton.), the proposed algorthm s also tested by choosng dfferent mages as the reference. The mean and standard dervaton of the correspondng calbraton results are presented n Table Calbraton of elatve Pose Between the Cameras n the Non-Overlappng Vew ACS (Algorthm III) In the thrd real experment, frstly, we use the nonoverlappng vew ACS calbraton method to process the mage sequences Q and Q. The jont pose (Ō A ) estmated by algorthm II s used as the nput for the relatve pose calbraton. Snce there are overlappng vews between Q and Q, we also calbrate the relatve pose between the two cameras by the feature correspondences for compar-

7 Table. esults Of Jont Pose Calbraton I: the algorthm usng overlappng vews. II: the algorthm usng fxed-jont motons. M: manual measurement(ground truth). O A s the coordnate of the jont refers to camera A, the same apples to O B. Algorthm Jont Pose (mm) X Y Z I O A O B II O A O B M O A ± 5± -4± O B -7± 5± -± Table 4. Mean and STD of the Jont Pose Calbraton Algorthm II Usng Dfferent eference Images. (O A s the coordnate of the jont refers to camera A, the same apples to O B.) Algorthm Jont Pose (mm) II X Y Z Mean O A O B STD O A O B son. The calbraton result are lsted n Table 5. After the jont pose refers to each camera n the ACS and relatve pose between the cameras n the ACS are calbrated, the trajectory of the ACS s recovered (see Fgure ). The proposed Table 5. esult of elatve Pose Calbraton III: our method. F: usng feature correspondences. Algorthm elatve otaton (Degree) oll Ptch Yaw III F Algorthm elatve Translaton (mm) T x T y T z III F calbraton method s also tested by non-overlappng vew mage sequences. Fgure (b), (c), (d) shows the confguraton of the non-overlappng vew ACS calbraton system n the real experment. Two mage sequences (Q 5 and Q 6 ) are recorded, each sequence conssts of 7 mages of sze 6. There s no overlappng vew between Q 5 and Q 6. Fgure 4 shows some samples of the recorded mages. We also manually measured the relatve pose between the two cameras for comparson. Snce no feature correspondence can be used, we only get a rough estmaton by a ruler. The calbraton results are shown n Table 6. After the relatve pose between the cameras at the ntal state s estmated, the trajectory of the non-overlappng vew ACS s recovered (see Fgure 5). Fgure. The Trajectory of the ACS ecovered from Q and Q (a) (c) (d) Fgure. The ACS wth Two Cannon PowerShot G9 Used n the eal Experment. (a) The ACS Used n the eal Experment. (b) The ACS and two Checker Planes. (c) In the Front of the ACS. (d) On the Top of the ACS. (b) Img Img 6 Img Img 7 (a) Images ecorded by Camera A Img Img 6 Img Img 7 (b) Images ecorded by Camera B Fgure 4. Images ecorded by the ACS 6. Concluson In ths paper, an ACS calbraton method s developed. Both the smulaton and real experment show that the pose of the jont n an ACS can be estmated robustly. When there s no overlappng vew between the cameras n an ACS, the jont pose and the relatve pose between the cameras can also be calculated. The trajectory of an ACS can be recov-

8 Table 6. esult of elatve Pose Calbraton Usng Non- Overlappng Vew Image Sequences. (III: our method. M: manual measurement.) Algorthm elatve otaton (Degree) oll Ptch Yaw III M ± 5 9 ± 5 ± 5 Algorthm elatve Translaton (mm) T x T y T z III M 9± ± 8 ± Fgure 5. The Trajectory of the ACS ecovered from Q 5 and Q 6 ered after the ACS s calbrated. The proposed calbraton method requres only the mage sequences recorded by the cameras n the ACS. In the real experment, the ntrnsc and extrnsc parameters of the ACS are calbrated usng the same mage sequences smultaneously. Our future plan may focus on usng an ACS attached on dfferent parts of human body to track the moton of the human. We foresee that f calbraton of artculated cameras become a smple routne, researchers wll fnd many novel and nterestng applcatons for such a camera system. Acknowledgement. We apprecate the revews for ther suggestons and menton of the relatve works. We would lke to thank Prof. Jaya JIA for dscusson, Ms. SHAO Lu n HKBU for checkng the Englsh, Mr. Gang LI, Hongnng DAI and other frends n CUHK for ther help. The research s supported by by a drect grant (code 55) from the Faculty of Engneerng, The Chnese Unversty of Hong Kong, Shatn, Hong Kong. eferences M. Antone and S. Teller. Scalable extrnsc calbraton of omn-drectonal mage networks. Internatonal Journal of Computer Vson, 49():4 74,. P. Baker and Y. Alomonos. Complete calbraton of a mult-camera network. Proc. IEEE Workshop on Omndrectonal Vson, :4 4,. P. Baker, A. Ogale, and C. Fermuller. The Argus eye: a new magng system desgned to facltate robotc tasks of moton. obotcs & Automaton Magazne, IEEE, (4): 8, 4. 4 P. T. Baker and Y. Alomonos. Calbraton of a multcamera network. Conference on Computer Vson and Pattern ecognton Workshop, 7:7,. 5 B. Caprle and V. Torre. Usng vanshng ponts for camera calbraton. Internatonal Journal of Computer Vson, 4():7 9, Y. Casp and M. Iran. Algnng Non-Overlappng Sequences. Internatonal Journal of Computer Vson, 48():9 5,. 7 S. Dockstader and A. Tekalp. Multple camera trackng of nteractng and occluded human moton. Proceedngs of the IEEE, 89():44 455,. 8 F. Dornaka. Self-calbraton of a stereo rg usng monocular eppolar geometres. Pattern ecognton, 4():76 79, I. Hartley and A. Zsserman. Multple vew geometry n computer vson. Cambrdge Unversty Press, ISBN: 55458, second edton, 4., 4. Horaud and F. Dornaka. Hand-eye calbraton. Internatonal Journal of obotcs esearch, 4():95, 995. M. Kaess and F. Dellaert. Vsual SLAM wth a Mult- Camera g. Techncal report, Georga Insttute of Technology, 6. T. Kanade, P. ander, and P. Narayanan. Vrtualzed realty: constructng vrtual worlds from real scenes. Multmeda, IEEE, 4():4 47, 997. H. G. Maas. Image sequence based automatc multcamera system calbraton technques. In Internatonal Archves of Photogrammetry and emote Sensng, (B5):76 768, Z. Zhang. A flexble new technque for camera calbraton. Techncal report, Techncal eport MS-T- 98-7, Mcrosoft esearch, Z. Zhang. A flexble new technque for camera calbraton. IEEE Transactons on Pattern Analyss and Machne Intellgence, (): 4,., 6