Extrinsic Camera Calibration with Minimal Configuration Using Cornea Model and Equidistance Constraint

Transcription

1 [DOI: /psjtcva.8.20] Regular Paper Extrnsc Camera Calbraton wth Mnmal Confguraton Usng Cornea Model and Equdstance Constrant Kosuke Takahash 1,a) Dan Mkam 1,b) Marko Isogawa 1,c) Akra Kojma 1,d) Receved: March 30, 2015, Accepted: January 14, 2016 Abstract: In ths paper, we propose a novel algorthm to extrnscally calbrate a camera to a 3D reference object that s not drectly vsble from the camera. We use the sphercal human cornea as a mrror and calbrate the extrnsc parameters from ts reflecton of the reference ponts. The key contrbuton of ths paper s to present a cornea-reflectonbased calbraton algorthm wth mnmal confguraton; there are three reference ponts and one mrror pose. The proposed algorthm ntroduces two constrants. Frst constrant s that the cornea s vrtually a sphere, whch enables us to estmate the center of the cornea sphere from ts projecton. Second s the equdstance constrant, whch enables us to estmate the 3D poston of the reference pont by assumng that the center of the camera and reference pont are located the same dstance from the center of the cornea sphere. We demonstrate the advantages of the proposed method wth qualtatve and quanttatve evaluatons usng syntheszed and real data. Keywords: camera calbraton, cornea, mrror, equdstance constrant 1. Introducton Determnng the geometrc relatonshp between a camera and a 3D reference object s called extrnsc camera calbraton, and has been a fundamental research feld n computer vson for many years [7], [23]. Ths technque s wdely used as an essental element of varous applcatons, such as 3D shape reconstructon from mult-vew mages [1], [13], and augmented realty [3]. Conventonal extrnsc calbraton technques have a fundamental assumpton: the camera should observe the 3D reference object drectly. Dsplay-camera systems such as laptop computers, smart phones, and dgtal sgnage have become popular and thus ganed much attenton as a useful devce for many tasks n computer vson. For example, Hrayama et al. [9] estmate the nterest of users who are watchng a dgtal sgnage. They assume that the user s gaze ponts represent hs/her nterests n the contents on dsplay. As another example, Kuster et al. [12] propose a gaze correcton method wth a dsplay-camera setup for home vdeo conferences. For these applcatons, they have to know the relatve posture and poston of the camera aganst the dsplay. However, the fundamental assumpton of extrnsc camera calbraton, the camera should observe the 3D reference object drectly, dose not hold n some cases of dsplay-camera system calbraton. In ths paper, we focus on extrnsc camera calbraton where the reference object les out of the camera s feld of vew. If the reference object s hdden from the camera, mrrors can be used to offset the occluson. Some studes on calbraton wth a mrror have descrbed setups to smplfy calbraton [2], [5], [8], [11], [16], [19], [20], [21]. Technques nclude decreasng the number of requred reference ponts or mrror poses, because smple setup offers many advantages for easy calbraton and low computaton complexty. Takahash et al. [21] and Hesch et al. [8] proposed calbraton algorthms wth three reference ponts and three poses of a planar mrror, whch s the mnmal setup for planar mrrors. To decrease the number of mrror poses, Agrawal [2] proposed an algorthm wth one pose of a sphercal mrror and eght reference ponts. As a calbraton method wth no addtonal hardware, Ntschke et al. [17] used the cornea as a sphercal mrror. Ths method needs three reference ponts and both cornea spheres,.e., two sphercal mrror poses. In ths paper, we focus on cornea-reflecton-based extrnsc camera calbraton for occluded reference objects (Fg. 1). The contrbuton of ths paper s to present a calbraton algorthm wth mnmal confguraton, that s three reference ponts and one sphercal mrror pose (cornea sphere). In the proposed algorthm, we ntroduce two constrants. Frst constrant s the cornea sphercal model. The shape and radus of the cornea have been modeled and the poston of the cornea sphere can be estmated from the projected lmbus mage [15]. Second s the equdstance con- 1 NIPPON TELEGRAPH AND TELEPHONE CORPORATION, NTT Meda Intellgence Laboratores, Yokosuka, Kanagawa , Japan a) takahash.kosuke@lab.ntt.co.jp b) mkam.dan@lab.ntt.co.jp c) sogawa.marko@lab.ntt.co.jp d) kojma.akra@lab.ntt.co.jp Fg. 1 Cornea-reflecton-based extrnsc camera calbraton. The goal of ths paper s to calbrate the camera and the reference object whch les out of the camera s fled of vew. c 2016 Informaton Processng Socety of Japan 20

2 strant. Ths constrant assumes that the center of the camera and each reference pont are located at the same dstance from the center of the cornea sphere (ths s satsfed by adjustng cornea sphere poston). The rest of ths paper s organzed as follows. Secton 2 provdes a revew of conventonal technques usng mrrors for calbraton and clarfes the novelty of proposed method. Secton 3 descrbes a measurement model for calbraton frstly, and then ntroduces key constrants and the algorthm. Secton 4 provdes evaluatons wth syntheszed data and real data to demonstrate the performance of our method. Secton 5 dscusses the performance of our method and optmzaton technques, and Secton 6 concludes ths paper wth an elucdaton of future work. 2. Related Work Ths secton revews conventonal mrror-based-calbraton and clarfes the contrbuton of ths paper. Mrror-based calbraton algorthms that use ndrect observatons of 3D reference objects can be categorzed n terms of mrror shape, the number of mnmal reference ponts and mrror poses (See Table 1). Frst, we categorze them nto two groups n terms of mrror shape: (1) Planar mrrors [8], [11], [16], [19], [20], [21], and (2) Sphercal mrrors [2]. Planar mrrors: The conventonal methods n ths group can be categorzed based on whether the mrror duplcates the camera (mrrored camera approach) or the reference ponts (mrrored pont approach). Hesch et al. [8] takes the mrrored camera approach. They estmate the extrnsc parameters between the mrrored camera and the true reference ponts (not reflectons) by solvng the P3P problem [6]. They used them for estmatng the extrnsc parameters between the camera and the true reference ponts wth the confguraton of three reference ponts and three mrror poses. On the other hand, Takahash et al. [21] adopts the mrrored pont based approach. They ntroduce an orthogonalty constrant that should be satsfed by all famles of reflectons of a sngle reference pont and utlzed t to estmate extrnsc parameters wth the same confguraton. Note that Sturm and Bonfort [20] revealed that at least three mrror poses are requred to unquely determne the extrnsc parameters f mrror s planar. Therefore, three reference ponts and three mrror poses s the mnmal confguraton for planar mrror based methods. Sphercal mrrors: Agrawal [2] proposes a sphercal mrror based calbraton method. They obtan an E matrx smlar to an essental matrx, by usng a coplanarty constrant wth eght pont correspondences and retreve the extrnsc parameters from the matrx. Ntschke et al. [17] proposes a method for calbratng dsplay- camera setups from the reflectons n the user s eye (cornea) wth no addtonal hardware. They estmate 3D postons of the reference ponts by fndng the ntersecton of two rays connectng a reference pont to the center of the eye ball. Ther method needs three reference ponts and both eyes, that s two sphercal mrrors. Our novel calbraton method s also based on cornea reflectons because the feature of no addtonal hardware for calbraton s mportant for casual dsplay-camera system, such as webcams and smartphones. In ths paper, we propose an algorthm to calbrate wth a mnmal confguraton where three reference ponts and one pose of a sphercal mrror (cornea sphere) by ntroducng a cornea sphere model and an equdstance constrant. 3. Extrnsc Calbraton usng Corneal Reflecton Ths secton ntroduces our cornea reflecton based calbraton algorthm; t determnes the extrnsc parameters representng the geometrc relatonshp between the camera and an obscured reference object. Ths secton frst ntroduces the measurement model used n ths paper and descrbes a method that can estmate the extrnsc parameters n Secton 3.1. And then t detals the unknown parameters needed for calbraton and descrbes the key constrants of ths approach n Secton Measurement Model As llustrated by Fg. 2, we assume that reference object X (Dsplay) s located out of camera C s feld-of-vew and there are three reference ponts p ( = 1, 2, 3) on X. These reference ponts p are mrrored by the eye ball and projected onto mage plane I as q. Extrnsc parameters (rotaton matrx R and translaton vector T) whch translate the reference object coordnate system {X} nto the camera coordnate system {C} satsfy the followng equaton. p = Rp {X} + T, (1) where p {X} denotes the 3D poston of p n {X}. We assume that {C} s the world coordnate system n ths paper and omt ths superscrpt f vector p s represented n {C}. Our goal s to estmate extrnsc parameters R and T from the projectons of the reference ponts. Determnng extrnsc parameters between two coordnate sys- Table 1 Mnmal confguraton for each method: shape of mrror, number of reference ponts and number of mrror poses. Shape Ponts Poses Kumar et al. [11] Plane 5 3 Rodrgues et al. [19] Plane 4 3 Hesch et al. [8] Plane 3 3 Takahash et al. [21] Plane 3 3 Agrawal [2] Sphere 8 1 Ntschke et al. [17] (Corneal) Sphere 3 2 Proposed (Corneal) Sphere 3 1 Fg. 2 Reflecton model of sphercal mrror. c 2016 Informaton Processng Socety of Japan 21

3 tems, such as {C} and {X}, through the use of a set of correspondng ponts n each coordnate system s known as the Absolute Orentaton Problem. Assume that there exsts two correspondng pont sets p ( = 1,, N p )andp {X}, that satsfy Eq. (1). By defnng, p = 1 N p p {X} = 1 N p N p p =1 N p =1 p {X} p C = p p, p {X} C = p {X} p {X}, where N p denotes the number of ponts, the correlaton matrx H s defned by N p H = =1 (2) p C p {X} C. (3) If the sngular value decomposton of H s gven by H = UΛV, then the optmal R and T are as follows: R = VU (4) T = p R p {X}. By usng ths method, we can obtan the extrnsc parameters from at least three pont correspondences. Snce the 3D postons of reference ponts p {X} n {X} are supposed to be gven a pror, we estmate the 3D postons of reference pont p n {C} as follows. 3.2 Estmaton of 3D Postons of Reference Ponts n Camera Coordnate System Ths secton descrbe our method for estmatng 3D postons of reference ponts p ( = 1, 2, 3) from ther projectons q. As llustrated by Fg. 3, the human eyeball can be modeled as two overlappng spheres. Snce the reflectons of reference ponts can be seen at the cornea, we utlze the cornea sphere as a sphercal mrror whose poston s S and radus s r. As llustrated by Fg. 2, m denotes the reflecton pont of reference pont p on the cornea sphere. A unt vector from the camera center O to m and unt vector from m to p are expressed as u and u, respectvely. p s expressed as follows: p = k m p u + m, (5) where k m p denotes the dstance between m and p. Based on the laws of reflecton, u s expressed as, u = u + 2( u n )n, (6) where n denotes the normal vector at m. Snce normal vector n s a unt vector connectng the center of cornea sphere S to m, n s expressed as, n = (m S)/ m S. (7) Wth unt vector u, m s expressed as, m = k Om u, (8) where k Om denotes the dstance between O and m. By usng projecton q, we obtan: u = q q = K 1 q I K 1 q I. Matrx K denotes the ntrnsc parameters and can be obtaned beforehand. Snce the m s on the cornea sphere, m satsfes the followng equaton: m S = r. (10) By substtutng Eq. (8) for Eq. (10) and multplyng by tself, we have k 2 Om u 2 2k Om u S + S 2 r 2 = 0. (11) Solvng Eq. (11) yelds two solutons as follows: u (u S ± S)2 u 2 ( S 2 r 2 ) k Om =. (12) u 2 Snce m s a closer pont to the camera among the ntersectons of u and sphere surface, the smaller k Om represents the dstance between O and m. From the above, we can obtan p from Eq. (5) by estmatng k m p, S and r. In order to estmate them, we ntroduce the followng two constrants, a geometrc model of the cornea sphere and the equdstance constrant Estmatng Parameters of Cornea Sphere Based on Its Geometrc Model In ths secton, we descrbe a method to estmate the center of the cornea sphere, S, from lmbus projecton by ntroducng a geometrc model [15]. The average radus of the cornea sphere, r, and the average radus of the cornea lmbus, r L, are 7.7mm and 5.6 mm respectvely [18]. As llustrated n Fg. 4, the lmbus projecton s modeled as an ellpse represented by fve parameters: the center, L, the major and mnor rad, r max and r mn, respectvely, and rotaton angle φ. (9) Fg. 3 (a) Cross secton, (b) Geometrc eye model based on Ref. [15]. Fg. 4 Estmatng the center of the cornea sphere from lmbus projecton. c 2016 Informaton Processng Socety of Japan 22

4 Algorthm 1 Extrnsc camera calbraton algorthm usng corneal model and equdstance constrant Input: Image I, q ( = 1, 2, 3), p {X}, K Output: R, T Compute ellpse parameters ( L,φ,r max, r mn ) from projecton of lmbus n mage I. Compute the center of corneal sphere S from ellpse parameters ( L,φ,r max, r mn ). for each q ( = 1, 2, 3) do Compute an unt vector u from Eq. (9). Compute a reflecton pont m of a reference pont p from Eq. (8) Compute a normal vector n at reflecton pont m from Eq. (7). Compute an unt vector u from Eq. (6). Compute 3D poston of reference pont p from Eq. (5). end for Solve Absolute Orentaton Problem wth p {X}, p and obtan extrnsc parameters R and T. Snce the depth of a tlted lmbus s much smaller than the dstance between camera and the cornea sphere, we assume weakly perspectve projecton. Under ths assumpton, the 3D poston of the center of lmbus L s expressed as L = dk 1 L, (13) where d denotes the dstance between the center of camera, O, and the center of lmbus L, and s expressed as d = f r L /r max. f and K represent the focal length n pxels and ntrnsc parameters, respectvely. Gaze drecton g s approxmated by the optcal axs of the eye, and s theoretcally determned by g = [sn τ sn φ, sn τ cos φ, cos τ], (14) where τ = ± arccos(r mn /r max ); τ corresponds to the tlt of the lmbus plane wth respect to the mage plane. Snce the center of cornea sphere, S, s located at dstance d LS (= r 2 rl 2 = mm), the radus of the cornea sphere from L, we compute S as follows, S = L d LS g. (15) In ths way, we estmate S from the ellpse parameters of the lmbus projected onto the mage plane, that s ( L,φ,r max, r mn ) Equdstance Constrant To obtan k m p, we ntroduce the Equdstance Constrant. The Equdstance Constrant states that the dstance from reference pont p to the center of cornea sphere S s equal to the dstance from the center of the camera, O, tos. If ths equdstance constrant s satsfed, trangle OSp s an sosceles trangle that satsfes O S = p S. Let the unt vector representng the bsector of OSp be denoted as l and a pont on ths bsector be denoted as a. Trangle Oa p s also an sosceles trangle that satsfes O a = p a. Addtonally, among the pont set O, a and p, the laws of reflecton can be establshed at a where l s used as the normal vector. Therefore, when a s the ntersecton of the bsector and the surface of cornea sphere, a s equal to the m of reference pont p. Therefore, when user sets hs/her center of cornea sphere such that the equdstance constrant does hold, trangle Om p should be an sosceles trangle that satsfes O m = p m, that s k m p = k Om. From the above, we compute 3D postons of reference pont p from Eq. (5) by ntroducng two constrants, and obtan extrnsc parameters R and T by solvng the Absolute Orentaton Problem. 4. Evaluatons In ths secton, we evaluate the performance of the proposed algorthm wth syntheszed data and real data. 4.1 Syntheszed Data To evaluate the performance of our method, we frst descrbe experments wth syntheszed data because obtanng ground truth data of extrnsc parameters s dffcult due to the dffculty of observng the precse camera center Confguraton In order to synthesze the data, we use the followng confguratons. The matrx of ntrnsc parameters, K, conssts of ( fx, f y, cx, cy); fx and f y represent the focal length n pxels, and cx and cy represent the 2D coordnates of the prncpal pont. We set them to (1400, 1400, 960, 540) n ths evaluaton respectvely. We set the camera coordnate system as the world coordnate system and set the center of camera to O = (0, 0, 0). The 3D postons of the reference pont are defned as p 1 = (0, 250, 0), p 2 = (125, 125, 0), p 3 = ( 125, 125, 0). The center of the cornea sphere s set to S = (0, 125, 50), whch satsfes the equdstance constrant, and ts radus s set to 5.6 mm on the bass of [18]. The dstances from S to O and to each reference pont, p ( = 1, 2, 3), are We added the Gaussan nose wth zero mean and standard devaton σ p (0 σ p 1) to projectons q as observaton nose. The ground truth of rotaton matrx R and translaton vector are set to R = I and T = (0, 250, 0), respectvely. Throughout ths experment we evaluated the dfference between estmated parameter and ts ground truth, and reprojecton error. Here, parameter subscrpt g ndcates ground truth data. The dfference between R and R g, D R (R, R g ), s defned as the Remannan dstance [14]: D R (R, R g ) = 1 2 Log(R R g ) F (16) LogR 0 (θ = 0), = θ 2snθ (R R ) (θ 0). (17) where θ = cos 1 ( trr 1 2 ). The dfference between T and T g, c 2016 Informaton Processng Socety of Japan 23

5 IPSJ Transactons on Computer Vson and Applcatons Vol (Apr. 2016) Fg. 5 Estmaton error under Gaussan nose for q wth standard devaton σ p. Fg. 7 A flow of estmatng ellpse parameters (L, φ, rmax, rmn ) from projecton of lmbus. Fg. 6 Confguraton for experments wth real data. The equdstance constrant s satsfed n a casual confguraton as dscussed n Secton 5.2. Notce that we used only three ponts p ( = 1, 2, 3) of chessboard pattern as the reference ponts for calbraton. DT (T, T g ), s defned as RMS: DT (T, T g ) = T T g 2 /3. The reprojecton error s defned as follows: 3 1 E p = q q (R, T, S). 3 =1 (18) (19) where q (R, T, S) denotes the reprojected pont calculated from estmated parameters. In ths smulaton, we compared our method aganst the stateof-the-art of planar mrror based method proposed by Takahash et al. [21]. For far comparson, we made sure that the projectons of reference ponts usng ether sphercal or planar mrrors occupy a smlar pxel area n the mage n the same way as n [2] Results wth Syntheszed Data Fgure 5 shows the evaluaton results of R, T and reprojecton error, from left to rght. In each fgure, the vertcal axs shows the average value over 100 trals and the horzontal axs denotes standard devaton of nose. Our estmaton errors, DR and DT, are sgnfcantly smaller than those of Takahash et al. [21] (59% and 96%, respectvely) and these results quanttatvely prove that our method outperforms Takahash et al. [21]. However, our reprojecton error almost matches that of Takahash et al. [21]. We explan ths by notng that Takahash et al. [21] employ non-lnear refnement for estmatng the extrnsc parameters, whch optmzes the reprojecton error, whle our method doesn t. For further analyss and dscusson of ths optmzaton see Secton 5.4. c 2016 Informaton Processng Socety of Japan Fg. 8 Images of cornea reflecton n three llumnaton envronments. The dstances between each q are about pxels. 4.2 Real Data Confguraton Fgure 6 overvews the confguraton. We used a Logcool HD Pro Webcam C920t and captured frames had the resoluton of As llustrated n Fg. 6, we used a chessboard pattern on dsplay as the reference ponts p ( = 1, 2, 3). The length of one sde of ths chessboard pattern s 125 mm. In order to satsfy our newly proposed equdstance constrant, we set the camera, reference ponts and user s cornea center as n Fg. 6. Whether ths constrant s satsfed or not was verfed by vsual judgment. The dstance between the user s cornea center and the dsplay s about 300 mm. The ntrnsc parameter was estmated beforehand followng [23]. In order to estmate ellpse parameters (L, φ, rmax, rmn ) from lmbus projecton, we bnarze the nput mage, apply the Canny detector, and ft an ellpse [4] as shown n Fg. 7. Addtonally, we confrmed that we can detect the boundares of the projected lmbus and estmate ellpse parameters under varous llumnaton envronments expected where the proposed method wll be used. The envronments had llumnaton ntenstes of 25 lux (dark, only dsplay lght used), 600 lux (mddle, under dstant fluorescent lght) and 1200 lux (brght, near to fluorescent lght). We show mages of cornea reflecton captured n these envronment n Fg Results wth real data Fgure 9 (a) llustrates the geometrc relaton of camera and reference ponts. The camera s set at the top of dsplay whch has three reference ponts. Fgure 9 (b) renders the estmated postons of the reference ponts. It s dfficult to obtan the ground 24

6 Table 2 Comparson of each dstance functon and reprojecton error followng [21]. Person1 Person2 Tral 1 Tral 2 Tral 3 Tral 1 Tral 2 Tral 3 D R D T E p Fg. 9 (a): Vew of confguraton. (b): Estmated postons of reference ponts by proposed method (red), by Ref. [21] (blue). truth of extrnsc parameters n any real confguraton, so we used Ref. [21] as the reference parameters. From ths result, we can see that the postons estmated by the proposed method are almost dentcal to those of Ref. [21]. Notce that the dfference n these rotaton matrces for x-axs, y-axs and z-axs are 2.44, 6.88, and 0.49 degrees, respectvely (D R = ), and D T s mm n Fg. 9. Table 2 quanttatvely compares the parameters estmated by the proposed method and Ref. [21]. We evaluated three trals for each of two users. From these results, we confrmed that our method works properly n real envronments snce the dfferences n the extrnsc parameters estmated by each method are small. Whle ths precson s not enough for eye gaze trackng, t s acceptable for applcatons that do not need hgh precson, such as gaze correcton [12] usng a dsplay and attached web camera system. Addtonally, these results ndcate that the estmaton results estmated by the proposed method depend on the user. We nvestgate ths dependency n the followng secton. 5. Dsucusson In ths secton, we dscuss the effects of nose on the constrants of our proposed method, and the method for optmzaton of extrnsc parameters. 5.1 Effects of Dfferences among Indvduals Ths paper makes two assumptons about the cornea model n developng our method. The frst assumpton s the radus of cornea sphere r. Whle we use the average radus of the cornea sphere, that s r = 7.7 mm [18], t can vary wth the ndvdual. The second one s the radus of cornea lmbus r L. In ths paper we use the average sze r L = 5.6 mm, but n practce the model parameters can be talored to sut the ndvdual. To more closely examne the effects of these assumptons, we created syntheszed data and nvestgated the effects of nose on these two rad. We used the same confguraton as n Secton 4.1. We added random nose wth unform dstrbuton n r and n rl to r and r L, respectvely, (0 n r 2, 0 n rl 1) based on Refs. [18] and [10]. Fg. 10 A case example where the equdstance constrant holds. Fgures 11 and 12 show the results of the averages of each dstance functon and reprojecton error. From Fgs. 11 and 12, we can see that r and r L strongly mpact the estmaton error of extrnsc parameters and reprojecton error. Ths because addng nose to r and r L affects the precson of S estmates based on Eq. (15) and d LS = r 2 rl 2, and the drecton and locaton of the reflecton on the cornea sphere changes sgnfcantly dependng on S and r. As to the results n Secton 4.2, we can say that these varatons do mpact estmaton performance. To solve ths problem, t s useful to calbrate the user s eye parameters beforehand. 5.2 Equdstance Constrant n a Real Stuaton Our proposed method need to satsfy the equdstance constrant. Although, t s dffcult to satsfy the proposed equdstance constrant strctly n a real stuaton. However, ths constrant s not such a strong assumpton for calbraton because t can easly be satsfed roughly even n a real stuaton. For example, as llustrated n Fg. 10, f the reference ponts p and the camera center are on the same crcle O, and the sphere center S s on a vector l equ that s orthogonal to the crcle plane and has ts footpont on the center of the crcle, the equdstance constrant s satsfed. If one assumes a dsplay-camera system, such as a smartphone or a web-camera attached on the dsplay (Fg. 6), the reference ponts and camera center are on almost the same plane and locatons of the reference ponts can be controlled by takng advantage of usng Dsplay as the reference object. Moreover, we can easly set our eyes on l equ by dsplayng the center of the crcle as a gude. We evaluate how well non-experts could satsfy ths constrant. We set the camera and reference ponts n the above manner. Snce t s dffcult to measure the dstances from the center of subject s cornea sphere to each reference pont and the camera by a ruler, we asked them to set an object on l equ n 3D space wth a trpod where t satsfed the equdstance constrant and measured the dstances from the object to each reference pont and the camera by a ruler. Notce that the ablty of placng one s eye so that t satsfes the equdsance constrant may not be same as the ablty of placng object to satsfy the constrant. However we thnk that f they can set an object, such as ther fngure, so that t c 2016 Informaton Processng Socety of Japan 25

7 Table 3 The standard devaton of dstances from the object to each reference pont and the camera σ equ, the average of dstances D equ and the mean of placement error e equ for 10 non-experts. S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 σ equ (mm) D equ (mm) e equ (mm) Fg. 11 Estmaton error under random nose for radus of cornea sphere r wth unform dstrbuton. Fg. 12 Estmaton error under random nose for radus of cornea lmbus r L wth unform dstrbuton. Fg. 13 Estmaton error under Gaussan nose for reference ponts p ( = 1, 2, 3) wth standard devaton σ m. satsfes the constrant, they can achve to set ther eye to satsfy the constrant by usng the object as a gude. Table 3 shows the standard devaton of dstances from the object to each reference pont and the camera σ equ, the average of dstances D equ and the mean of placement error e equ for 10 subjects. Notce that D equ = (Σ 4 =1 p o p )/4, where p o represents the 3D poston the object and p represents the 3D poston of each reference pont and camera, and e equ = (Σ 4 =1 p o p D equ )/4. From Table 3, we can see that non-experts can satsfy the equdstance constrant wth σ equ < 10 mm. Addtonally, we nvestgate the effects of nose on the equdstance constrant quanttatvely. We evaluate the performance of the proposed method by generatng syntheszed data that ncludes Gaussan nose wth zero mean and standard devaton σ m (0 σ m 10) to p ( = 1, 2, 3). Fgure 13 shows the results of the averages of each dstance functon and reprojecton error. From these results, the equdstance constrant does not have any severely negatve effects on extrnsc parameter estmaton, even when t s not strctly satsfed. For example, the estmaton error of the translaton vector s 3 mm at most. We consder that such precson s suffcent for dsplay-camera system applcatons. 5.3 Effects of Detecton Error of Projected Lmbus In order to estmate the center of cornea sphere, our proposed c 2016 Informaton Processng Socety of Japan 26

8 Fg. 14 Estmaton error under nose for rad of projected lmbus r max and r mn. Fg. 15 Estmaton error of optmzed extrnsc parameters usng N p reference ponts under Gaussan nose for q wth standard devaton σ p = 1. method need to detect the projected lmbus as the ellpse wth bnarzaton, canny edge detector, and ellpse detecton as n Secton However, the rad of detected ellpse mght contan pxel errors due to the observaton nose, parameters for detecton process, and so on. Here, we nvestgate the effects of these noses wth real data used n Secton We add the pxel error n re to r max and r mn respectvely (0 n re 4(pxel)) and computed reprojecton error and dfference functons between result wth orgnal rad and result wth nosed rad. Fgure 14 shows the averages of the dstance functons and reprojecton error. From ths fgure, we can see that the errors on r max and r mn can affect the translaton vector especally. Future work ncludes mprovng the accuracy of detectng the rad of projected lmbus. 5.4 Optmzaton In general, estmated extrnsc parameters can be refned by non-lnear optmzaton [22]. Here we dscuss the non-lnear optmzaton for extrnsc parameters estmated by the proposed method. In order to optmze the estmated extrnsc parameters R and T, we mnmze E(R, T, S) whch has reprojecton error term and equdstance constrant term as follows, E(R, T, S) = 3 { q q (R, T, S) + D (R, T, S)}. (20) =1 The term of D (R, T, S) s expressed as follows, D (R, T, S) = p (R, T) S O S + p (R, T) m (R, T, S) O m (R, T, S), (21) where p (R, T) andm (R, T, S) denote the 3D poston of reference pont p and the reflecton pont m computed wth estmated parameters respectvely. We evaluated the performance of ths optmzaton usng syntheszed data that ncludes Gaussan nose wth standard devaton σ p = 1 as n Secton 4.1. Fgure 15 plots the optmzaton performance under varous numbers of reference ponts. Whle the dstance functons D R and D T decrease after non-lnear optmzaton f more than fve ponts are used, they ncrease wth the mnmal confguraton. Ths s consdered to be due to the lack of redundancy n our method. Future work ncludes optmzaton of extrnsc parameters for the mnmal confguraton. 6. Concluson In ths paper, we proposed a new algorthm that calbrates a camera to a 3D reference object va cornea reflecton wth mnmal confguraton. The key features of our method are ts ntroducton of two constrants: cornea reflecton model and equdstance constrant. In evaluatons, our method outperformed a stateof-the-art of plane mrror based method wth both syntheszed and real data. Future work ncludes studes on the non-lnear optmzaton of extrnsc parameters whle keepng the mnmal confguraton and nteractve calbraton the locaton of reference ponts s optmzed on the dsplay. References [1] Agarwal, S., Furukawa, Y., Snavely, N., Curless, B., Setz, S.M. and Szelsk,R.: Reconstructng Rome,IEEE Computer, Vol.43, pp (2010). [2] Agrawal, A.: Extrnsc Camera Calbraton wthout a Drect Vew Usng Sphercal Mrror, Proc. ICCV, pp (2013). c 2016 Informaton Processng Socety of Japan 27

9 [3] Azuma, R., Ballot, Y., Behrnger, R., Fener, S., Juler, S. and MacIntyre, B.: Recent advances n augmented realty, Computer Graphcs and Applcatons, IEEE, Vol.21, No.6, pp (2001). [4] Ftzgbbon, A.W., Fsher, R.B., et al.: A buyer s gude to conc fttng, DAI Research paper (1996). [5] Francken, Y., Hermans, C. and Bekaert, P.: Screen-Camera Calbraton usng a Sphercal Mrror, 4th Canadan Conference on Computer and Robot Vson, 2007, CRV 07, pp (2007). [6] Haralck, B.M., Lee, C.-N., Ottenberg, K. and Nölle, M.: Revew and analyss of solutons of the three pont perspectve pose estmaton problem, IJCV, Vol.13, pp (1994). [7] Hartley, R.I. and Zsserman, A.: Multple Vew Geometry n Computer Vson, Cambrdge Unversty Press, 2nd ed. (2004). [8] Hesch, J.A., Mourks, A.I. and Roumelots, S.I.: Algorthmc Foundaton of Robotcs VIII, Sprnger Tracts n Advanced Robotcs, Vol.57, chapter Mrror-Based Extrnsc Camera Calbraton, pp , Sprnger-Verlag (2009). [9] Hrayama, T., Dodane, J.-B., Kawashma, H. and Matsuyama, T.: Estmates of user nterest usng tmng structures between proactve content-dsplay updates and eye movements, IEICE Trans. Inf. Syst., Vol.93, No.6, pp (2010). [10] Iyamu, E. and Osuoben, E.: Age, gender, corneal dameter, corneal curvature and central corneal thckness n Ngerans wth normal ntra ocular pressure, Journal of optometry, Vol.5, No.2, pp (2012). [11] Kumar, R., Ile, A., Frahm, J.-M. and Pollefeys, M.: Smple calbraton of non-overlappng cameras wth a mrror, Proc. CVPR, pp.1 7 (2008). [12] Kuster, C., Popa, T., Bazn, J.-C., Gotsman, C. and Gross, M.: Gaze Correcton for Home Vdeo Conferencng, ACM Trans. Graph. (Proc. ACM SIGGRAPH ASIA), Vol.31, No.6, p. to appear (2012). [13] Matsuyama, T., Nobuhara, S., Taka, T. and Tung, T.: 3D Vdeo and Its Applcatons, Sprnger Publshng Company, Incorporated (2012). [14] Moakher, M.: Means and Averagng n the Group of Rotatons, SIAM J. Matrx Anal. Appl., Vol.24, pp.1 16 (2002). [15] Nakazawa, A. and Ntschke, C.: Pont of Gaze Estmaton Through Corneal Surface Reflecton n an Actve Illumnaton Envronment, Proc. ECCV, ECCV 12, Berln, Hedelberg, Sprnger-Verlag, pp (2012). [16] Nayar, S.: Catadoptrc omndrectonal camera, Proc. CVPR, pp (1997). [17] Ntschke, C., Nakazawa, A. and Takemura, H.: Dsplay-camera calbraton usng eye reflectons and geometry constrants, Computer Vson and Image Understandng, Vol.115, No.6, pp (2011). [18] Rchard, S. and Snell, M.A.L.: Clncal Anatomy of the Eye, Wley- Blackwell, 2nd ed. (1997). [19] Rodrgues, R., Barreto, P. and Nunes, U.: Camera Pose Estmaton Usng Images of Planar Mrror Reflectons, Proc. ECCV, pp (2010). [20] Sturm, P. and Bonfort, T.: How to Compute the Pose of an Object wthout a Drect Vew, Proc. ACCV, pp (2006). [21] Takahash, K., Nobuhara, S. and Matsuyama, T.: A new mrror-based extrnsc camera calbraton usng an orthogonalty constrant, Proc. CVPR, pp (2012). [22] Trggs, B., McLauchlan, P., Hartley, R. and Ftzgbbon, A.: Bundle Adjustment A Modern Synthess, Trggs, B., Zsserman, A. and Szelsk, R. (eds.), Vson Algorthms: Theory and Practce, Lecture Notes n Computer Scence, Vol.1883, pp , Sprnger Berln Hedelberg (2000). [23] Zhang, Z.: A flexble new technque for camera calbraton, TPAMI, pp (2000). Dan Mkam receved hs B.E. and M.E. degree from Keo Unversty, Kanagawa, Japan n 2000 and 2002, respectvely. He has been workng for Nppon Telegraph and Telephone Corporaton from He receved hs Ph.D. from Tsukuba Unversty n Hs current research actvtes are manly focused on computer vson and vrtual realty. He was awarded the Meetng on Image Recognton and Understandng 2009 Excellent Paper Award 2009, the IEICE Best Papar Award 2010, the IEICE KIYASU- Zen t Award 2010, and the IPSJ SIG-CDS Excellent Paper Award 2013, the IEICE Human Communcaton Award He s a senor member of IEICE and a member of IPSJ and IEEE. Marko Isogawa receved her B.S. and M.S. degrees from Osaka Unversty, Japan, n 2011 and 2013, respectvely. She has been workng for Nppon Telegraph and Telephone Corporaton from Her research nterests nclude multmeda content handlng. Akra Kojma Senor Research Engneer, Supervsor, Vsual Meda Project, NTT Meda Intellgence Laboratores. He receved the B.E. and M.E. degree n Mathematcal Engneerng and Informaton Physcs from the Unversty of Tokyo n 1988 and 1990, respectvely. Snce jonng NTT n 1990, he has been engaged n research and development on vdeo database, dgtal lbrary, multmeda nformaton retreval, vdeo survellance and hgh-realty vsual communcaton. He s a member of the Insttute of Electroncs, Informaton and Communcaton Engneers (IEICE), the Insttute of Image Electroncs Engneers of Japan (IIEEJ) and ACM. (Communcated by Rugang Yang) Prze n CVPR Kosuke Takahash receved hs B.Sc. degree n engneerng and M.Sc. n nformatcs from Kyoto Unversty, Japan, n 2010 and 2012, respectvely. He s currently a researcher at NTT Meda Intellgence Laboratores. Hs research nterest ncludes computer vson. He receved Best Open Source Code award Second c 2016 Informaton Processng Socety of Japan 28