Hand Eye Calibration Applied to Viewpoint Selection for Robotic Vision Yuichi Motai, Member, IEEE, and Akio Kosaka, Member, IEEE

Transcription

1 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 55, NO. 10, OCTOBER Hand Eye Calbraton Appled to Vewpont Selecton for Robotc Vson Yuch Mota, Member, IEEE, and Ako Kosaka, Member, IEEE Abstract Vewpont calbraton s a method to manpulate hand eye for generatng calbraton parameters for actve vewpont control and object graspng. In robot vson applcatons, accurate vson sensor calbraton and robust vson-based robot control are essental for developng an ntellgent and autonomous robotc system. Ths paper presents a new approach to hand eye robotc calbraton for vson-based object modelng and graspng. Our method provdes a 1.0-pxel level of mage regstraton accuracy when a standard Puma/Kawasak robot generates an arbtrary vewpont. To attan ths accuracy, our new formalsm of hand eye calbraton deals wth a lens dstorton model of a vson sensor. Our most dstngushed approach of optmzng ntrnsc parameters s to utlze a new parameter estmaton algorthm usng an extended Kalman flter. Most prevous approaches dd not even consder the optmal estmates of the ntrnsc and extrnsc camera parameters, or chose one of the estmates obtaned from multple solutons, whch caused a large amount of estmaton error n hand eye calbraton. We demonstrate the power of ths new method for: 1) generatng 3-D object models usng an nteractve 3-D modelng edtor; 2) recognzng 3-D objects usng stereovson systems; and 3) graspng 3-D objects usng a manpulator. Expermental results usng Puma and Kawasak robots are shown. Index Terms Hand eye calbraton, Kalman flter, lens dstorton, multple vewponts, robot vson. I. INTRODUCTION WE REPORT here an mproved technque for calbratng hand eye robotc vson systems. Our approach allows the system to compute robotc calbraton parameters of the camera and the end-effector for multple vewpont generaton after the system performs the vson-based camera calbraton for only a small number of vewponts. Ordnarly, f one wshes to ntegrate multple vews for acqurng a 3-D object model, one would separately carry out camera calbraton for each vewpont. An alternatve approach conssts of carryng out calbraton for a certan number of desgnated vewponts and usng nterpolaton for other vewponts. The dsadvantages of ths approach are: 1) overall low accuracy (partally domnant errors) and 2) spatal lmtaton of the vewpont generaton. In our proposed scheme, we use multple vewponts (at least three vew- Manuscrpt receved December 20, 2006; revsed February 5, Frst publshed March 21, 2008; current verson publshed October 1, Ths work was supported by the Advanced Manufacturng Technology Department, Ford Motor Corporaton. Y. Mota s wth the School of Engneerng, Unversty of Vermont, Burlngton, VT USA (e-mal: ymota@uvm.edu). A. Kosaka s wth Purdue Unversty, West Lafayette, IN USA, and also wth Future Creaton Laboratory, Olympus Corporaton, Shnjuku , Japan (e-mal: kosaka@purdue.edu). Dgtal Object Identfer /TIE ponts) for calbraton of the camera mounted on the grpper and optmally estmate all the necessary parameters for actve control of vewponts and precse object graspng. The advantages of our method are to mnmze the camera calbraton error by: 1) applyng a lens dstorton model and 2) optmzng the camera parameters wth robotc arm knematcs. Snce camera calbraton s fundamental to all phases of research n robot vson, much work has been done so far. Some of the earlest references to camera calbraton n the computer vson lterature are [7] and [11]. These assume a pnhole model for the camera, and, therefore, ths assumpton restrcts the use of a wde range of vews for 3-D reconstructon snce the lens dstorton can be severe n the perpheral of the camera vew. Whereas few of these references deal wth lens dstorton of camera optcs [23], others address the ssue of effcent calculaton of varous camera parameters [25], [26]. Hand eye calbraton utlzes technques of camera calbraton and moton control of the robot hand. We have developed algorthms for hand eye calbraton by movng the camera to dfferent vewponts through robot arm moton and takng mages of 3-D calbraton patterns. The algorthms frst estmate ntrnsc and extrnsc camera parameters for each vewpont and select the ntrnsc camera parameters of one vewpont as the truth-value of the ntrnsc camera parameters for all vewponts. The algorthms then estmate varous transformaton matrces for hand eye calbraton, ncludng the transformaton from the camera to the end-effector and the transformaton from the robot base to the world coordnate frame. When the lens dstorton of the camera becomes severe, however, the estmated ntrnsc camera parameters can greatly dffer between vews due to the nstablty of ntrnsc camera parameter estmaton (ncludng lens dstorton parameters). The selecton of the ntrnsc camera parameters from a sngle vew leads to a large hand eye calbraton error, partcularly for arbtrary vewponts used n 3-D modelng of objects. In ths paper, to acheve accurate and relable performance of actve vson control, we have modeled a camera wth lens dstorton as well as optmally estmated camera calbraton parameters by ntegratng those estmates obtaned from multple vewponts usng the Kalman flter approach. Kalman flters have been wdely used n many ndustral electroncs, and some robotc applcatons smlar to ours are found n [4], [15], [16], and [28] [43]. Fg. 1 shows the mage regstraton error of the overall hand eye calbraton for vewpont generaton. Small black crcles are used for the camera calbraton patterns to deal wth lens dstorton, and whte cross-bars are supermposed to verfy the accuracy of the method. As shown n ths fgure, we attan /$ IEEE

2 3732 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 55, NO. 10, OCTOBER 2008 Fg. 1. Camera mounted on the grpper takes an mage of calbraton patterns of radus 2.0 mm at the dstance of 0.2 m. The estmated calbraton parameters from our method are used to produce the whte cross-hars, supermposed at the expected pattern centers. the 1.0-pxel level of mage regstraton accuracy (1-mm level of accuracy n the 3-D workspace) when the camera s moved to a poston that was not used for calbraton. Fg. 2 also shows the verfcaton of our hand eye calbraton for object graspng. The robot attempts to move the tp of the stylus mounted on the grpper to the center of calbraton patterns. As several early studes demonstrated, the soluton of the knematcs can be derved as a closed form usng an nverse transformaton matrx [10]. To determne the poston and the orentaton of the hand eye camera wth respect to the workspace, more recent ssues are to solve the unknown transformaton of the robot hand, whch s formed as a homogeneous matrx equaton AX = XB [6], [21], [24] and to solve ths equaton usng a quaternon approach [6], [8], [24]. In our method, we propose a modfed algorthm usng a nonlnear teraton method and then evaluate the new soluton usng our robot manpulaton systems. Here, we apply our hand eye calbraton technque to 3-D modelng and graspng of ndustral objects. Our robotc calbraton can be appled to develop a human-asssted model acquston system [17]. In ths project, objects are placed n the work area by a human, who then gudes the system nto establshng mage-to-mage and pose-to-pose correspondences. After a model s acqured n ths manner durng the test phase, the same object n a random pose s vewed from two vewponts for pose calculaton f the object s placed nsde a known workspace or n the vew of the sngle camera mounted on the robot hand. In the acqured model, the object surfaces that are approprate for graspng wth the robotc hand are marked. The computer vson method dentfes whch surface has graspng capabltes before usng the robotc hand to pck up the object. Lastly, robotc manpulaton s executed to grasp the object for whch the graspng ponts have already been acqured. We also seek an effcent vewpont selecton for magng, so that our robot vson system can relably compute an object s pose. The choce of an magng vewpont has been examned n the feld of actve vson [1], [2]. In these studes, a camera s manpulated to mprove the qualty of the perceptual results. Because we can compute calbraton parameters at any camera poston, havng actve control over the vewpont drecton ncreases machne percepton ablty [22]. In ths paper, we wll frst present our new approach for the hand eye calbraton system and wll then apply ths calbraton result to the human-asssted model acquston system. Lastly, expermental results that verfy the power of our approach wll be shown. Fg. 2. Verfcaton of hand eye calbraton usng a stylus s shown. (a) Puma and (b) Kawasak robots attempt to locate the tp of the stylus to the orgn of the world coordnate frame by usng the estmated calbraton parameters. II. PROBLEM STATEMENT OF HAND EYE CALIBRATION To generate a 3-D object model, we mount a monocular camera on the grpper of the robotc manpulator to capture vews of the object from multple vewponts. It s desred that all multple vews be automatcally captured usng the robot, followed by calbraton parameter calculaton for model acquston. For ths strategy n 3-D modelng, hand eye calbraton s an mportant task to perform. To formulate the hand eye calbraton problem, we defne the followng coordnate frames as shown n Fg. 3: Base (x B,y B,z B ), Tool (x T,y T,z T ), Approach (x A,y A,z A ), Graspng (x G,y G,z G ), Object (x O,y O,z O ), World (x W,y W,z W ), Camera (x C,y C,z C ), and Image (u, v). In our laboratory platform, robotc hand moton s fully determned usng a Puma 761 or Kawasak JS10 controller wth the postonng commands XY ZOAT (whch s a platformdependent control parameter smlar to XY Zϕθψ). We wll be able to determne the homogeneous transformaton Base from the robot tool to the robot base coordnate frames [see later n (33) and (34)]. Thus, our hand eye calbraton problems are, frst, to solve a camera (eye) calbraton problem and then to solve a robotc (hand) calbraton problem, descrbed as follows: 1) determne the camera-mage transformaton g, namely (u, v) =g(x C,y C,z C ) (1) 2) determne the homogeneous transformatons of Camera and H World Base.

3 MOTAI AND KOSAKA: HAND EYE CALIBRATION APPLIED TO VIEWPOINT SELECTION FOR ROBOTIC VISION 3733 Fg. 3. Coordnate transtonal llustraton for robot manpulaton. Please note that, n general, the camera mage coordnates (u p,v p ) under the pnhole camera mage coordnates are not observable. Lens dstorton adds to the normalzed pnhole camera mage coordnates a devaton of (u,v ).Let(ũ, ṽ) be the normalzed camera mage coordnate frame wth lens dstorton. Then, we can represent the relatonshp between (u,v ) and (ũ, ṽ) by the followng [25]: u = x C ũ + k 1 ũ(ũ 2 +ṽ 2 ) z C v = y C ṽ + k 1 ṽ(ũ 2 +ṽ 2 ) (3) z C Fg. 4. Camera dstorton model. A. Camera Dstorton Model Before dscussng the detals of our calbraton procedure, we wll present the process of camera mage formaton that deals wth lens dstorton (Fg. 4). Let us frst assume that the camera s modeled by a pnhole camera. In ths case, the camera mage coordnates (u p,v p ) can be mapped nto the normalzed pnhole camera mage coordnates (u,v ) by the ntrnsc camera parameters, the magnfcaton factors α u and α v, and the mage center coordnates (u 0,v 0 ) as follows: u = u p u 0 α u v = v p v 0 αv = x C z C = y C z C. (2) where the coeffcent k 1 for the devaton represents the parameter for the radal dstorton of the camera, and (ũ, ṽ) s computed from the actual camera mage coordnates (u, v) as follows: ũ = u u 0 α u ṽ = v v 0 α v. (4) In ths paper, we wll call s =[α u,α v,u 0,v 0,k 1 ] T the ntrnsc camera parameters. Note that these ntrnsc camera parameters wll reman constant durng the hand eye calbraton process. As shown n [25], tangental dstorton s not domnant for robotc applcatons. Therefore, we only model here a radal dstorton case. However, our method s applcable to lens dstorton models dealng wth radal and tangental cases. We now consder multple vewponts by movng a camera n the world coordnate frame. At vewpont, the camera locaton

4 3734 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 55, NO. 10, OCTOBER 2008 n the world coordnate frame s specfed by the homogeneous transformaton Camera, whch conssts of the rotaton matrx R =(rkm ) and the translaton vector t =[t x,t y,t z] T. Then from (3), the 3-D pont (x W,y W,z W ) n the world coordnate frame s mapped onto the camera mage frame as follows: u = r 11x W + r 12y W + r 13z W + t x r 31 x W + r 32 y W + r 33 z W + t z ũ + k 1 ũ(ũ 2 +ṽ 2 ) (5) v = r 21x W + r 22y W + r 23z W + t xy r 31 x W + r 32 y W + r 33 z W + t z ṽ + k 1 ṽ(ũ 2 +ṽ 2 ) Fg. 5. Calbraton steps. (6) where the rotaton matrx R s specfed by ndependent yaw ptch roll angles ϕ x, ϕ y, and ϕ z. We wll call e = [ϕ x,ϕ y,ϕ z,t x,t y,t z, ] T the extrnsc camera parameters for vewpont. Havng the ntrnsc camera parameters s and the vewdependent extrnsc camera parameters e, we can always determne the mappng of the world coordnate frame 3-D ponts nto the camera coordnate frame. To compute the cameramage transformaton,.e., 2-D mage coordnates (u, v) from the correspondng 3-D pont (x W,y W,z W ) n the world frame, we take the followng two steps. 1) Compute the normalzed camera mage coordnates (ũ, ṽ) by solvng the mplct forms of (5) and (6) usng an approprate teratve gradent method, for example, the Newton method, wth ntal values (ũ, ṽ) =(u,v ). 2) Compute the actual mage coordnates (u, v) from (ũ, ṽ) usng (4). III. SOLUTION TO HAND EYE CALIBRATION Most of the prevous work on hand eye calbraton dd not take nto account the lens dstorton for hand eye calbraton snce ther man nterest was vsual servong for robotc manpulaton [22]. In most cases, these methods only utlzed central regons of camera mages, where the pnhole camera s generally suffcent to model the 3-D 2-D pont projecton. When usng a camera wth a small focal length and wder vews for 3-D object modelng, lens dstorton becomes a sgnfcant ssue for calculatng precse 3-D measurements or for object modelng usng multple vew observatons. We propose here an effcent and accurate approach that can deal wth lens dstorton for the camera model whle stll producng precse determnaton of robotc hand and camera transformatons. In our hand eye calbraton strategy, as shown n Fg. 5, we take the followng three steps. Step 1) We locate a calbraton pattern board at dfferent heghts (z W values) n the world coordnate frame and then take snapshots of the calbraton pattern board from multple vewponts by movng the robot end-effector. We then ndependently estmate the ntrnsc and extrnsc camera parameters for each vewpont, assumng that ntrnsc camera parameters are not actually equal. Fg. 6. Radal lens dstorton llustraton by teratvely mnmzng (8). The left-hand sde shows the ntal estmaton correspondng to the pnhole camera model. The rght-hand sde shows how the radal dstorton s handled va nonlnear teratons. Step 2) We reestmate ntrnsc and extrnsc camera parameters by mposng the constrant that all ntrnsc camera parameters be equal. Step 3) We compute the homogeneous transformaton between the robot tool and the camera coordnate frames as well as that between the robot base and the world coordnate frames. Most of the prevous work on hand eye calbraton dd not take nto account the lens dstorton for hand eye calbraton snce ther man nterest was vsual servong for robotc manpulaton [22]. In most cases, these methods only utlzed central regons of camera mages, where the pnhole camera s generally suffcent to model the 3-D 2-D pont projecton (Fg. 6). A. Step 1: Intal Estmaton of Camera Parameters In Step 1, we generate multple vewponts (M vewponts n total) by movng the camera wth the robot and takng snapshots of the planar calbraton pattern board, whch s located at dfferent heghts z W specfed n the world coordnate frame. The calbraton patterns consst of small black crcles (N crcles n total), and the 3-D coordnates of the centrods of the calbraton patterns are measured n the world frame. By analyzng the snapshots of the calbraton patterns by a computer, we can estmate the 2-D mage coordnates of the centrods n the camera mage frame. For each vewpont and each crcle j, let (x j,y j,z j ) and (u j,v j ) be the 3-D coordnates of the calbraton pattern centrods n the world coordnate frame and the correspondng 2-D mage coordnates n the camera mage frame measured and accumulated

5 MOTAI AND KOSAKA: HAND EYE CALIBRATION APPLIED TO VIEWPOINT SELECTION FOR ROBOTIC VISION 3735 from varous heghts, respectvely. Then, from (5) and (6), we obtan r 11x W + r 12y W + r 13z W + t x r 31 x W + r 32 y W + r 33 z W + t z r 21x W + r 22y W + r 23z W + t y r 31 x W + r 32 y W + r 33 z W + t z =ũ j + k 1 ũ ( j (ũ j ) 2 +(ṽj) 2) =ṽj + k 1 ṽj ( (ũ j ) 2 +(ṽj) 2) ũ j = u j u 0 α u ṽ j = v j v 0 α v. In Step 1, we frst ndependently estmate ntrnsc camera parameters s =[α u, α v, u 0,v0,k 1] T and extrnsc camera parameters e =[ϕ x, ϕ y, ϕ z,t x, t y, t z] T for dfferent vewponts. To do ths, we apply Weng s algorthm, a nonlnear teratve method that s descrbed n detal n [25]. The man contrbuton of the algorthm s that ths problem s converted nto an optmzaton problem that mnmzes the objectve functon of the mage regstraton error between the 2-D measurement coordnates (u j, vj ) and the 2-D projected coordnates (u j, vj ) based on the parametrc representaton of ntrnsc and extrnsc camera models s and e, respectvely, computed from (7). The objectve functon s defned by f = j { (u j u j ) 2 + ( v j v j ) 2 } whch should be mnmzed wth respect to s and e. B. Step 2: Integraton of Camera Parameter Estmates From Multple Vews In the prevous step, ntrnsc camera parameters s ( = 1, 2,...,M) are not necessarly equal snce, for ths algorthm, the parameters are ndependently estmated for vewpont. In Step 2, we ntegrate all estmates to obtan an optmal estmate of ntrnsc and extrnsc camera parameters s and e by a nonlnear teraton algorthm. As for the ntal estmates for the teraton, we use the estmates s and e obtaned n Step 1. More specfcally, we attempt to mnmze f n (8) under the followng addtonal constrant: (7) (8) s = s 1 = s 2 = s M. (9) We also apply an teratve technque of nonlnear optmzaton to attan a robust and accurate estmate. In ths case, we have a good ntal estmate for s and e ( =1, 2,...,M); then, we apply an extended Kalman flter-based updatng scheme [13], so that sequental updatng of parameters can be attaned along wth outler elmnaton. In our mplementaton of the extended Kalman flterng, for each mage measurement pont (u j,v j ), we have a par of constrant equatons as follows: r11 x W+r 12 y W+r 13 z W+t x f r31 x W+r 32 y W+r 33 z ( ũ W+t j +k ( 1ũ j (ũ z j ) 2 +(ṽj )2)) r21 x W+r 22 y W+r 23 z W+t x r31 x W+r 32 y W+r 33 z ( ṽ W+t j +k ( =0 1ṽj (ũ z j ) 2 +(ṽj )2)) (10) where the mage measurement (u j, v j) s converted to the normalzed mage pont (ũ j, ṽ j) as (7). We defne the parameter vector p to be estmated as follows: p =[s e 1 e 2 e m ] (11) and measurement vector z =(u j, v j), and we assocate the mean p and the error covarance matrx Σ. Then, the extended Kalman flter updatng from (p, Σ) to (p new, Σ new ) s, for each teraton, expressed by where p new = p K(Mp y) (12) Σ new =(I KM)Σ (13) f = Mp y (14) K =ΣM T (W + MΣM T ) (15) M = f p (16) ( ) ( ) T f f W = R. z z (17) Note that y s obtaned as the lnearzaton of the constrant equaton f, K s the Kalman gan, and R s the error measurement covarance matrx. To stablze the estmaton, Kalman updatng s sequentally performed for each measurement pont (u j, v j) by randomzng the order of ndexes and j. Durng the sequental updatng, we also check whether outlers exst n the mage measurements, as descrbed n [13]. In our case, we have a large number of measurements of mage calbraton ponts (uj, vj) (j =1, 2,...,N), where N s typcally around The extended Kalman flter-based updatng can be mplemented for a small number of constrants for each teraton of updatng; more specfcally, only two degrees of freedom for the constrants are necessary as seen n the constrant equaton (10). Note that the parameters to be estmated are (s, e 1, e 2,...,e m ) for m multple vews, and the dmenson of the parameter vector p s 5+6m. The constrant equaton acqured from (10) s smply 2-D assocated wth the mage measurements (uj, vj) for each sequental updatng. Ths reducton of dmensonalty greatly helps the reducton of computatonal complexty. We would lke to menton here that our approach of optmzng ntrnsc parameters from multple vews s mportant. Compared to prevous approaches that do not consder the optmal estmates of the unque ntrnsc camera parameters and multple extrnsc camera parameters n multple vews, our proposed method s chosen as a good tool to move the camera to an arbtrary vewpont. Many studes on vsual servong

6 3736 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 55, NO. 10, OCTOBER 2008 for robotc manpulaton smply choose one of the ntrnsc parameters from multple solutons. Ths causes a large amount of an estmaton error for hand eye calbraton, partcularly for scenaros wth severe lens dstorton. As our expermental results later show, ths step sgnfcantly mproves the accuracy of camera calbraton. C. Step 3: Robotc Calbraton Once the expermental process has provded the camera parameters s and e ( =1, 2,...,M), the relatonshp between coordnates n Fg. 3 wll be obtaned as each component transformaton matrx such as Camera, H Base Tool, and H World Base.Note that H World Camera s derved from the extrnsc camera parameters e. The transtonal relatonshp between those transformaton matrces s shown n the followng: H World Camera = Camera H Base Tool H World Base. (18) Snce one camera settng s fxed n the tool poston, the mages of dfferent vewponts are produced by controllng the tool poston. Gven the postonng control parameters, [10]. The two unknown transformaton matrces n (18) are H World Base and Camera. Note that snce the transformaton from Tool coordnates to Camera coordnates s vewpont ndependent, we can denote an nvarant matrx as Camera. Usng the multple camera calbraton results, we solve H World Base frst, use ths matrx, and then compute the last unknown matrx Camera. Snce we approxmately know that the world z-axs and the robot base z-axs are almost parallel, H World Base s nearly equal to an dentty matrx n a rotatonal part and some values n a translatonal part. For smplcty, n (19), we show the two pars of vewpont cases for fve vewponts, denoted wth and j (thus, 5 C 2 combnatory possbltes), for the transtonal relatonshp,.e., we can specfy the transformaton matrx H Base Tool H World Camera /j = CameraH Base Tool /j H World Base. (19) We elmnate Camera from vewponts and j n (19) and obtan A j X XB j = O (20) where the notatons are defned as follows: X = H World Base (21) A j = ( H Base ) 1 Tool H Base Tool j (22) B j = ( H World Camera ) 1 H World Camera j. (23) In the above notaton, the 3 3 rotaton matrx R and the 3 1 translaton vector t can be separately expressed n a homogeneous form, so that the equaton can be decomposed as follows: R A jr = RR B j (24) R A jt + t A j = Rt B j + t. (25) Therefore, our frst problem s to estmate R. Although an analytcal dervaton of solutons to R n (24) s generally dffcult [6], [21], we can apply a smple teratve approach usng the followng method. Let ψ x, ψ y, and ψ z be the yaw ptch roll angles assocated wth the rotatonal transformaton from the world coordnate to the base coordnate assocated wth R defned by (26), shown at the bottom of the page. We apply the Broyden Fletcher Goldfarb Shanno optmzaton algorthm [5] to obtan the soluton of q =(ψ x,ψ y,ψ z ) T, whch mnmzes the followng objectve functon: f(q) =,j( j) R A j R RR B 2 j. (27) For ths nonlnear teraton method, we can approprately select an ntal estmate for q snce we know the approxmate poston of the robot base wth respect to the world coordnate frame. Our ntal estmate may not be suffcently close to the true value for us to apply a smple gradentdescent method lke the Newton method; therefore, we utlze the Broyden Fletcher Goldfarb Shanno optmzaton method, whch s known to have a stable convergence [3]. Once R s estmated from (27), we have the followng translaton equaton from (25): ( I R A j ) t = t A j Rt B j. (28) Then, we have the followng: Ct= d (29) where the followng notatons are used: I R A 1,2 t A 1,2 Rt B 1,2 C =. d =. (30) I R A m 1,m t A m 1,m Rt B m 1,m where m s the number of vewponts. The soluton to (29) wth respect to t s gven by t =(C T C) 1 C T d (31) whch s equvalent to the soluton that mnmzes Ct d 2. Ths completes the estmates of X n (20). R = cos ψ z cos ψ y cos ψ z sn ψ y sn ψ x sn ψ z cos ψ x cos ψ z sn ψ y cos ψ x +snψ z sn ψ x sn ψ z cos ψ y sn ψ z sn ψ y sn ψ x +cosψ z cos ψ x sn ψ z sn ψ y cos ψ z cos ψ z sn ψ x (26) sn ψ y cos ψ y sn ψ x cos ψ y cos ψ x

7 MOTAI AND KOSAKA: HAND EYE CALIBRATION APPLIED TO VIEWPOINT SELECTION FOR ROBOTIC VISION 3737 Note that the estmaton of Camera s derved as the same from (20) n the case of H World Base. When the robot tool s movng toward a specfc poston, we have the robot postonng control parameter and ts correspondng homogeneous transformaton H Base Tool. From (18), the three transformaton matrces Camera, H Base Tool, and H World Base wll produce the desred transformaton matrx H World Camera. Therefore, we can now generate both ntrnsc and extrnsc camera parameters for any tool postons f we have the end-effector postonng parameters. The output of our soluton s the optmzed ntrnsc camera parameters s and the computed extrnsc camera parameters e at any arbtrary tool poston at. Ths elmnates the need for recalbraton as the robot end-effector s moved to dfferent postons. IV. VIEWPOINT GENERATION BY ROBOTIC MOTION The purpose of hand eye calbraton s to generate vewponts and to obtan all parameters for transformatons that are assocated wth vewponts. In our case, we would lke to model 3-D objects located n the world coordnate frame by movng the camera to arbtrary vewng postons specfed n the world coordnate frame. To acheve ths, we need the nverse knematcs to locate the camera at arbtrary postons and orentatons n the world coordnate frame by movng the end-effector of the robot. Therefore, the problem s specfed as follows. Gven a homogeneous transformaton from the world coordnate frame to the camera coordnate frame, generate the robot moton commands. Therefore, the frst step of our vewpont generaton problem s how to compute Base usng H Camera World based on extrnsc camera parameters e. Because the robot hand s controlled by Base, ths can be done by computng Base = H World Base H Camera World Camera (32) where H World Base and Camera are determned usng the camera calbraton methods dscussed n Step 3 n Secton III. The second step of our vewpont generaton problem s to compute the robotc moton parameters from Base. Thss a well-known nverse knematcs problem for varous robotc manpulators. In our case, we apply our method wth the followng two robots. A. Puma 761 Puma 761 s a Puma robot hand, whose end-effector tool poston s controlled by the sx parameters XY ZOAT, and s expressed n terms of the homogeneous transformaton matrx n (33), shown at the bottom of the page. Gven Base, XY ZOAT parameters are computed for each element of (33). A detaled element computaton for the Puma robotc platform can be found n [10]. B. Kawasak js10 In the Kawasak JS10 Trells controller, the 3 3 rotaton matrx porton n (33) s represented by roll ϕ, ptch θ, and yaw ψ as gven n (34), shown at the bottom of the page. The parameters ϕ, θ, and ψ were derved by transformng the elements of the above matrx wth respect to the arc tangent. V. E XPERIMENTAL RESULTS We mplemented the proposed calbraton method on a SUN Workstaton (the mplemented source code n ths paper s avalable by request). We performed the evaluaton experments for the Puma and Kawasak platforms. Several mnutes of computaton were requred to optmally estmate the camera parameters s and e and homogeneous transformaton matrces Camera and H World Base. In ths experment, we used a Sony DC-47 monocular 1/3 n a charge-coupled devce camera wth a Pulnx lens of focal length 16 mm. We selected fve vewponts to estmate the hand eye calbraton parameters. All of fve vewponts were set up n such a manner that each was drected to the world orgn at the same heght n the world coordnate frame. One camera vewpont was located drectly above a pattern board, and the other vewponts were symmetrcally offset along the X and Y world coordnates, as shown n Fg. 7. For each vewpont, we took snapshots of a planar calbraton pattern board at three dfferent levels of z-values specfed n the world frame. The calbraton plane was moved n the z w axs to the z W values 0.001, , and m. The spacng of the ponts n the calbraton pattern spacng was m. Base = cos O sn T sn O sn A cos T cos O cos T +snosn A sn T sn O cos A X sn O sn T +cososn A cos T sn O cos T cos O sn A sn T cos O cos A Y cos A cos T cos A cos T sn A Z (33) Base = cos φ cos θ cos φ sn θ sn ψ sn φ cos ψ cos φ sn θ cos + sn φ sn ψ X sn φ cos θ sn φ sn θ sn ψ +cosφcos ψ sn φ sn θ cos ψ cos φ sn ψ Y sn θ cos θ sn ψ cos θ cos ψ Z (34)

8 3738 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 55, NO. 10, OCTOBER 2008 Fg. 7. Fve mage vewponts used for magng an object. As long as the noncoplanar ponts are located n the roughly 3 3 m 2 workspace, then all of the camera parameters can be estmated. Ths s dfferent from typcal self-calbraton because the z w moton s not necessarly parallel to the camera optc axs. We then ndependently estmated the ntrnsc camera parameters s and the extrnsc camera parameters e.next, we ntegrated multple vew-dependent camera parameters to estmate the optmal ntrnsc camera parameters s and to modfy ndvdual extrnsc camera parameters e. Last, the hand eye homogeneous transformatons, such as, were estmated. H World Base Camera and A. Evaluaton of Hand Eye Calbraton by Image Regstraton The accuracy of the estmated calbraton parameters was evaluated n the 2-D mage plane by generatng arbtrary robot hand motons and by projectng known 3-D coordnates of the calbraton patterns (whte cross-bars) onto the orgnally captured mages of calbraton patterns. Fg. 8 shows the results of mage regstraton accuracy. We numercally analyzed the error of calbraton parameters wth pxel devaton between the projected ponts and the actual mage ponts. In the Puma platform, the average error was 1.38 ± 0.79 pxel/marker, as the typcal mage shown n Fg. 1 (Secton I). In the Kawasak platform, several other vewpont mages were also evaluated as n Fg. 8. The mage regstraton errors for all vews are shown n Fg. 9 (the Kawasak robot platform). The mage regstraton results were compared to alternatve solutons n Table I, whch ncluded the method wthout lens dstorton. Modelng the regstraton results wth lens dstorton provdes the best accuracy, partcularly f wder camera mage vews are used, as shown n Fg. 8 and Table I. B. Evaluaton for 3-D Reconstructon The calbraton results were also evaluated for 3-D reconstructon by ntegratng vews from multple vewponts. We placed a calbraton pattern board wth 225 black markers at three dfferent heghts n the world coordnates and took snapshots of ths calbraton board from fve dfferent vewponts. We then extracted markers and estmated the 3-D coordnates of the black markers n the world coordnate frame usng the stereo Fg. 8. Verfcaton of supermposed maker patterns at two dfferent z values n the Kawasak platform. For (a) and (b), the projecton matrx s determned usng the results of the pnhole camera model. For (c) and (d), the results of the radal dstorton camera model are used. Fg. 9. Integraton of all supermposed mages usng estmated dstorton camera parameters n the Kawasak platform. The whte cross-bars + represent the supermposton of the pattern usng the estmated transformaton. The black overlaps represent the actual extracton of the patterns. TABLE I IMAGE REGISTRATION ERROR

9 MOTAI AND KOSAKA: HAND EYE CALIBRATION APPLIED TO VIEWPOINT SELECTION FOR ROBOTIC VISION 3739 Fg. 10. Three-dmensonal reconstructon of the maker patterns from three vewpont mages usng hand eye calbraton results (a) wth elmnatng dstorton and (b) affected by dstorton. The stereo correspondence pattern ponts n the three mages were dentfed and computed the world coordnate values (n meters) wth depth value z. The marker pont spacng n 3-D was z =0.001, , and m. The standard devatons of z were (a) m and (b) m, whch demonstrates a 21.1% mprovement when lens dstorton s taken nto account. Fg. 11. Standard devaton of z values versus the number of stereo mages used. reconstructon technque descrbed n [14]. Fg. 10 shows the 3-D reconstructon results from mergng three vews. Fg. 10(a) shows the results wth respect to lens dstorton, and Fg. 10(b) shows the results wthout takng lens dstorton nto account (usng the pnhole camera model). As we can see from Fg. 10, modelng the lens dstorton leads to better accuracy for 3-D reconstructon. We also evaluate the relatonshp between the number of vewponts and the 3-D reconstructon error. Fg. 11 shows the result. We conclude that fve vews are suffcent for modelng 3-D reconstructon of the object. VI. APPLICATIONS OF THE CAMERA AND ROBOT CALIBRATION We utlzed our hand eye calbraton results for 3-D object modelng and robotc bn pckng. As we have dscussed n Fg. 12. Stereo correspondence n human computer nteracton edtor. The feature extracton and correspondences n multple vews are establshed usng the human-n-the-loop approach. The features are computed n the 3-D world coordnate frame usng the camera parameters s and e by least-mean-square mnmzaton. Wth the calbraton results, each feature s represented n the 3-D world coordnates from multple vewpont mages. the prevous sectons, we have a good tool to move the camera to an arbtrary vewpont n the world coordnate frame where the object to be modeled or to be grasped s located. In the followng, we wll present the expermental results for: 1) modelng 3-D objects and 2) recognzng and graspng such objects by controllng the vewponts used by the camera. A. Object Model Acquston In our laboratory, vson models of 3-D objects were generated usng the human-asssted model-acquston system [17], [18] along wth the results from hand eye calbraton. In ths robot teachng system, shown n Fg. 12, the objects were placed n the work area by a human, who then taught the system to establsh the regstraton of mage-to-mage and pose-to-pose

10 3740 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 55, NO. 10, OCTOBER 2008 B. Object Localzaton and Graspng We also show other expermental results that verfy that our calbraton method for vewpont generaton can be appled to 3-D object recognton and pose estmaton tasks. After 3-D object models were generated wth the human-asssted model acquston system descrbed n Secton VI-A, the actual objects were randomly placed n a bn n the robot s workspace. The robot s task was to recognze and grasp the objects usng the vson model. To verfy and/or reject the pose hypotheses, a new vewpont was computed by followng the method n Secton IV, so that the desred vew could be obtaned at that vewpont. If the hypothess was not fully verfed, the system tred to capture an addtonal mage from a subsequent vewpont, and the object pose was recomputed to match the features obtaned at the new vewponts. We appled a model-based stereovson algorthm developed by Kosaka and Kak [14] to extract salent mage features for object localzaton n the world coordnate frame. In our strategy, the robot ntally took two vews of the object whose vewponts are preassgned. The robot then attempted to compute the pose hypotheses of the object based on the correspondence between the model features. To verfy the applcatons of our calbraton accuracy, we made experments for object localzaton and graspng tasks [19], [20]. Fg. 13. Some results of acqurng object models. For the polyhedral objects (a), (b) and (c), (d), and the oval objects (e), (f). The geometrcal shape was reconstructed usng the calbraton parameters. correspondences. We generated a 3-D geometrcal model of a 3-D object from salent features such as crcles, polygons, ellpses, and free-form curves. The computer extracted 2-D salent features from the mages taken from multple vews. The system then nteracted wth a human operator to represent the object wth the extracted features for each 2-D vew mage. Then, the system estmated the 3-D coordnates of features n the world frame from the 2-D features n the mage coordnates snce the system regstered the estmated 3-D features as a body of the 3-D object model. Durng the 3-D reconstructon process, the calbraton results, such as the lens dstorton parameters k 1 n (3), s, and e and robotc knematcs, were requred when the system ntegrated multple vews to generate 3-D object models, so that lnes and curves n each mage can be relably and accurately reconstructed n 3-D features. Based on our evaluaton experment, fve vews were used for ntegraton ths was verfed by Fg. 11. The example results of the two polyhedral object models are shown n Fg. 13(b) and (d). We tested our calbraton method by modelng an artfcally made object made of wood wth polygon surfaces snce the 3-D reconstructon for these objects easly evaluated the accuracy of our calbraton results. The modelng error was less than 3% (around 3 mm) compared to the true values of measurng the actual objects [17], [18]. The ndustral objects, ncludng oval curve shapes, were also examned through a human computer nterface edtor (Fg. 12). VII. CONCLUSION Ths paper presented a new method of hand eye robotc calbraton for vson-based object modelng and graspng. Our method provdes pxel-level mage regstraton accuracy for 3-D reconstructon when arbtrary vewponts are generated usng a standard Puma or a Kawasak robot. To attan ths accuracy, we developed a hand eye calbraton process that ncluded a lens dstorton camera model. Our method ntegrates estmates of camera parameters obtaned from multple vewponts by utlzng an extended Kalman flter. Based on modelng and manpulaton experments usng the robots, we showed that our vson system had suffcent power for actve vewpont generaton to acqure 3-D models of objects and to recognze and grasp these objects n the robotc work cell. ACKNOWLEDGMENT The authors would lke to thank J. B. Zurn for her comments. REFERENCES [1] J. Y. Alomonos, I. Wess, and A. Bandyopadhyay, Actve vson, Int. J. Comput. Vs., vol. 1, no. 4, pp , [2] D. H. Ballard, Anmate vson, Artf. Intell., vol. 48, no. 1, pp , Feb [3] C. G. Broyden, Quas-Newton methods, n Optmzaton Methods n Electroncs and Communcatons, vol. 1, Mathematcal Topcs n Telecommuncatons, K. W. Cattermole and J. J. O Relly, Eds. New York: Wley, 1984, pp [4] T. Bucher, C. Curo, J. Edelbrunner, C. Igel, D. Kastrup, I. Leefken, G. Lorenz, A. Stenhage, and W. von Seelen, Image processng and behavor plannng for ntellgent vehcles, IEEE Trans. Ind. Electron., vol. 50, no. 1, pp , Feb [5] E. K. P. Chong and S. H. Zak, An Introducton to Optmzaton. New York: Wley, 1996, pp

11 MOTAI AND KOSAKA: HAND EYE CALIBRATION APPLIED TO VIEWPOINT SELECTION FOR ROBOTIC VISION 3741 [6] F. Dornaka and R. Horaud, Smultaneous robot-world and hand eye calbraton, IEEE Trans. Robot. Autom., vol. 14, no. 4, pp , Aug [7] O. D. Faugeras and G. Toscan, Calbraton problem for stereo, n Proc. Int. Conf. Pattern Recog., 1986, pp [8] O. D. Faugeras and M. Herbert, Representaton, recognton, and locatng of 3-D objects, Int. J. Robot. Res., vol. 5, no. 3, pp , [9] M. A. Fschler and R. C. Bolles, Random sample consensus: A paradgm for model fttng wth applcatons to mage analyss and automated cartography, Commun. ACM, vol. 24, no. 6, pp , Jun [10] K. S. Fu, R. C. Gonzales, and C. S. G. Lee, Robotcs: Control, Sensng, Vson, and Intellgence. New York: McGraw-Hll, 1987, pp [11] S. Ganapathy, Decomposton of transformaton matrces for robot vson, n Proc. IEEE Int. Conf. Robot. Autom., 1984, pp [12] L. Grewe and A. C. Kak, Interactve learnng of a mult-attrbute hash table classfer for fast object recognton, Comput. Vs. Image Underst., vol. 61, no. 3, pp , [13] A. Kosaka and A. C. Kak, Fast vson-guded moble robot navgaton usng model-based reasonng and predcton of uncertantes, Comput. Vs. Graph. Image Process. Image Understandng, vol. 56, no. 3, pp , Nov [14] A. Kosaka and A. C. Kak, Stereo vson for ndustral applcatons, n Handbook of Industral Robotcs, S. Y. Nof, Ed. New York: Wley, 1999, pp [15] J. M. Lee, K. Son, M. C. Lee, J. W. Cho, S. H. Han, and M. H. Lee, Localzaton of a moble robot usng the mage of a movng object, IEEE Trans. Ind. Electron., vol. 50, no. 3, pp , Jun [16] J. Mangel and W. Leonhard, Vehcle control by computer vson, IEEE Trans. Ind. Electron., vol. 39, no. 3, pp , Jun [17] Y. Mota and A. C. Kak, An nteractve framework for acqurng vson models of 3D objects from 2D mages, IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 34, no. 1, pp , Feb [18] Y. Mota, Vsual-based human robotc nteracton for extractng salent features of an ndustral object for an automated assembly system, Int. J. Comput. Ind. Specal Issue on Machne Vson, vol. 56, no. 8/9, pp , Dec [19] Y. Mota, A new edge-groupng algorthm for multple complex objects localzaton, n Proc. Innovatons Appl. Artf. Intell., 2004, vol. 3029, pp [20] Y. Mota and A. Kosaka, Concatenate feature extracton for robust 3D ellptc object localzaton, n Proc. 19th ACM Symp. Appl. Comput., 2004, pp [21] Y. C. Shu and S. Ahmad, Calbraton of wrst-mounted robotc sensors by solvng homogeneous transform equatons of the form AX = XB, IEEE Trans. Robot. Autom., vol. 5, no. 1, pp , Feb [22] K. Tarabans, P. K. Allen, and R. Y. Tsa, A survey of sensor plannng n computer vson, IEEE Trans. Robot. Autom., vol. 11, no. 1, pp , Feb [23] R. Y. Tsa, A versatle camera calbraton technque for hgh-accuracy 3D machne vson metrology usng off-the-shelf TV cameras and lenses, IEEE J. Robot. Autom., vol. RA-3, no. 4, pp , Aug [24] C. C. Wang, Extrnsc calbraton of a vson sensor mounted on a robot, IEEE Trans. Robot. Autom., vol. 8, no. 2, pp , Apr [25] J. Weng, P. Cohen, and M. Hernou, Camera calbraton wth dstorton models and accuracy evaluaton, IEEE Trans. Pattern Anal. Mach. Intell., vol. 14, no. 10, pp , Oct [26] Z. Zhang, Flexble camera calbraton by vewng a plane from unknown orentatons, n Proc. Int. Conf. Comput. Vs., 1999, pp [27] Camera Calbraton Toolbox for Matlab. [Onlne]. Avalable: [28] G. We, K. Arbter, and G. Hrznger, Actve self-calbraton of robotc eyes and hand eye relatonshps wth model dentfcaton, IEEE Trans. Robot. Autom., vol. 14, no. 1, pp , Feb [29] F. Jatta, G. Legnan, and A. Vsol, Frcton compensaton n hybrd force/velocty control of ndustral manpulators, IEEE Trans. Ind. Electron., vol. 53, no. 2, pp , Apr [30] S. Katsura, Y. Matsumoto, and K. Ohnsh, Analyss and expermental valdaton of force bandwdth for force control, IEEE Trans. Ind. Electron., vol. 53, no. 3, pp , Jun [31] H. Hu and P. Woo, Fuzzy supervsory sldng-mode and neural-network control for robotc manpulators, IEEE Trans. Ind. Electron., vol. 53, no. 3, pp , Jun [32] Y. Matsumoto, S. Katsura, and K. Ohnsh, Dexterous manpulaton n constraned blateral teleoperaton usng controlled supportng pont, IEEE Trans. Ind. Electron., vol. 54, no. 2, pp , Apr [33] G. Cheng and K. Peng, Robust composte nonlnear feedback control wth applcaton to a servo postonng system, IEEE Trans. Ind. Electron., vol. 54, no. 2, pp , Apr [34] N. Kubota and K. Nshda, Perceptual control based on predcton for natural communcaton of a partner robot, IEEE Trans. Ind. Electron., vol. 54, no. 2, pp , Apr [35] M. Chen and A. Huang, Adaptve control for flexble-jont electrcally drven robot wth tme-varyng uncertantes, IEEE Trans. Ind. Electron., vol. 54, no. 2, pp , Apr [36] R. Wa and P. Chen, Robust neural-fuzzy-network control for robot manpulator ncludng actuator dynamcs, IEEE Trans. Ind. Electron., vol. 53, no. 4, pp , Jun [37] K. Sm, K. Byun, and F. Harashma, Internet-based teleoperaton of an ntellgent robot wth optmal two-layer fuzzy controller, IEEE Trans. Ind. Electron., vol. 53, no. 4, pp , Jun [38] S. Katsura, J. Suzuk, and K. Ohnsh, Pushng operaton by flexble manpulator takng envronmental nformaton nto account, IEEE Trans. Ind. Electron., vol. 53, no. 5, pp , Oct [39] K. Khayat, P. Bgras, and L. Dessant, A multstage poston/force control for constraned robotc systems wth frcton: Jont-space decomposton, lnearzaton, and multobjectve observer/controller synthess usng LMI formalsm, IEEE Trans. Ind. Electron., vol. 53, no. 5, pp , Oct [40] T. N. Chang, B. Cheng, and P. Srwlajaroen, Moton control frmware for hgh-speed robotc systems, IEEE Trans. Ind. Electron.,vol.53,no.5, pp , Oct [41] G. Km and W. Chung, Trpodal schematc control archtecture for ntegraton of mult-functonal ndoor servce robots, IEEE Trans. Ind. Electron., vol. 53, no. 5, pp , Oct [42] S. Katsura and K. Ohnsh, A realzaton of haptc tranng system by multlateral control, IEEE Trans. Ind. Electron., vol. 53, no. 6, pp , Dec [43] H. Hu and P. Woo, Fuzzy supervsory sldng-mode and neural-network control for robotc manpulators, IEEE Trans. Ind. Electron., vol. 53, no. 3, pp , Jun Yuch Mota (M 01) receved the B.Eng. degree n nstrumentaton engneerng from Keo Unversty, Tokyo, Japan, n 1991, the M.Eng. degree n appled systems scence from Kyoto Unversty, Kyoto, Japan, n 1993, and the Ph.D. degree n electrcal and computer engneerng from Purdue Unversty, West Lafayette, IN, n He s currently an Assstant Professor of electrcal and computer engneerng wth the Unversty of Vermont, Burlngton. Hs research nterests nclude the broad area of sensory ntellgence, partcularly of medcal magng, computer vson, sensor-based robotcs, and human computer nteracton. Ako Kosaka (S 86 M 88) receved the B.Eng. and M.Eng. degrees from the Unversty of Tokyo, Tokyo, Japan, n 1981 and 1983, respectvely, and the Ph.D. degree from Purdue Unversty, West Lafayette, IN, n He s currently an adjunct Assocate Professor of electrcal and computer engneerng wth Purdue Unversty, as well as the chef Research Scentst wth Future Creaton Laboratory, Olympus Corporaton, Shnjuku, Japan. He has conducted a wde varety of research projects, ncludng computeraded neurosurgcal navgaton systems, 6DOF haptc nterfaces, wearable user nterfaces, content-based mage retreval systems, and 3-D dgtal camera systems. Hs research nterests nclude computer vson, 3-D medcal magng, dstrbuted sensor networks, ntellgent envronments, and moble robot navgaton.