A Study on Eye Gaze Estimation Method Based on Cornea Model of Human Eye

Transcription

1 A Study on Eye Gze Estimtion Method Bsed on Corne Model of Humn Eye Eui Chul Lee 1 nd Kng Ryoung Prk 2 1 Dept. of Computer Science, Sngmyung University, 7 Hongji-dong, Jongro-Ku, Seoul, Republic of Kore Biometrics Engineering Reserch Center (BERC) oryong@smu.c.kr 2 Division of Digitl Medi Technology, Sngmyung University, 7 Hongji-dong, Jongro-Ku, Seoul, Republic of Kore Biometrics Engineering Reserch Center (BERC) prkgr@smu.c.kr Abstrct. In this pper, we propose new gze estimtion method by nlyzing the corne surfce model which is estimted through three dimensionl nlysis of humn eye in HMD (Hed Mounted Disply) environments. This pper hs four dvntges over previous works. First, in order to obtin ccurte gze position, we use corne sphere model bsed on Gullstrnd eye scheme. Second, we clculte the 3D position of the corne sphere nd gze vector by using cmer, three collimted IR-LEDs nd one illuminted IR- LED. Third, three coordintes such s cmer, monitor nd eye coordintes re unified, which cn simplify the complex 3D converting clcultion nd llow for clcultion of the 3D eye position nd gze position on HMD monitor. Fourth, simple user dependent clibrtion method is proposed by gzing t one position of HMD monitor bsed on Kpp compenstion. Experimentl results showed tht the verge gze estimtion error of the proposed method ws 0.89 degrees. 1 Introduction Gze estimtion is the procedure of detecting the position on monitor plne tht user is looking t. Gze estimtion for desktop monitor environments hs been widely reserched. However, comprtively little hs been done bout gze estimtion for HMD (Hed Mounted Disply) environment. We propose new vision-bsed gze detecting method by using smll cmer, n illuminted IR-LED (Infr-Red Light Emitting Diode) nd three collimted IR-LEDs in HMD. Reserches bout gze estimtion cn be clssified into two ctegories: the twodimensionl (2D) nd the three-dimensionl (3D) pproches. The 2D pproch does not consider the 3D motion or the 3D structure of the eyes. This pproch trnsforms the coordintes of the eye fetures or eye intensity vlues in 2D imge frme into the coordintes of the gze point in the 2D monitor plne by using mpping functions [1][17]. Recently, severl 3D pproches hve been reported, which utilize high resolution eye imge cptured by zoom-in cmer [2][3]. Since the cmer A. Gglowicz nd W. Philips (Eds.): MIRAGE 2007, LNCS 4418, pp , Springer-Verlg Berlin Heidelberg 2007

2 308 E.C. Lee nd K.R. Prk cptures only one eye, it produces enough resolution to nlyze 3D movements. The dvntges of the bove 3D pproch re more ccurte with simple user-dependent clibrtion compred to those of 2D pproch. All the 2D nd 3D techniques mentioned bove were developed in desktop monitor nd they re difficult to be used in HMD monitor due to the weight limittion of cmer system [15]. In desktop monitor environments, there is no physicl limittion to the cmer systems. So, focus nd zoom control cmers, multiple lenses, pn/tilt equipment nd sometimes stereo cmers cn be used [2][3]. However, in HMD environments, the cmer system ttched on HMD must be s light s possible becuse users wer it. It is imprcticl to employ complicted cmer systems like stereo cmers or multiple lenses. To overcome such problems, we propose new gze estimtion method bsed on 3D pproch by using smll USB (Universl Seril Bus) cmer, n illuminted IR- LED nd three collimted IR-LED illumintors in HMD environment. 2 The Proposed Method Fig.1 shows our gze detection system in HMD environment bsed on following policies. First, we use monoculr cmer with single lens. Second, we Model corne sphere for clculting the eqution of humn eyebll. Third, we use 3D pproch to eye gze estimtion bsed on user-dependent clibrtion by gzing t only one position of HMD monitor. Fig. 1. Overview of proposed gze detection system in n HMD environment We used monoculr cmer without complex lenses to ensure tht the system ws lightweight. Also, our system did not require the complicted user dependent clibrtion. For tht, we used corne sphere model to clculte the eqution of humn eyebll. And, we used three collimted IR-LEDs to mesure the ccurte 3D positions of corne sphere nd finl gze point on virtul monitor plne of HMD.

3 A Study on Eye Gze Estimtion Method Bsed on Corne Model of Humn Eye Proposed Gze Detection System As shown in Fig.1, our gze detection system is vision-bsed one which consists of HMD system [4], smll sized USB cmer [5], one illumintion IR-LED (850nm) nd three collimted IR-LEDs (880nm) [6]. Our HMD system hs the disply resolution of 800(horizontl) pixels by 600(verticl) pixels nd the field of view of 26 degrees digonl. Also, virtul imge size is sme to the size of when user sees the 70inch t 3937mm wy from right viewing window of Fig. 1. In our system, we use infrred LEDs of two types. One of them is the collimted IR-LEDs. Their illumintion ngle is so nrrow(2.9 ) tht we could generte smll bright spots on corne surfce in order to estimte the corne sphere eqution. In section 2.4, we will describe the method of estimting corne eqution by using these collimted IR-LEDs. Another infrred LED is the illumintion IR-LED. Becuse the illumintion ngle of this IR-LED is so wide(36 ), we could cpture the pproprite eye imges hving cler edge between pupil nd iris through illuminting the entire eye region. In used cmer, by ttching n IR-pss filter in front of the cmer lens nd using IR illumintors, our system is robust to environmentl lighting conditions. In ddition, since we used invisible IR lights, we were ble to reduce the dzzling effect on the user's eye. Also, since we used USB cmer, frme grbber bord ws not required. This helped to keep the weight of the system be minimum. For eye imge cpturing, we used the sptil resolution of pixels with frme rte of 15 frmes per second. 2.2 Humn Eye nd Corne Sphere Model First, we nlyze the 3D structure of the humn eye. Bsed on the Gullstrnd humn eye scheme which represents the structure of the humn eye very ccurtely [9], we cn pply this scheme to our gze estimtion method. Bsed on tht eye scheme, Fig. 2. Humn eye nd corne sphere model [10]

4 310 E.C. Lee nd K.R. Prk he reported tht the nterior corne ws sphere whose rdius ws 7.8mm. Since this nterior corne is the exterior prominent prt of humn eye, we cn observe it by using the cmer ttched on HMD s shown in Fig. 1. Also, since the nterior corne surfce is sphere with the rdius of 7.8mm, we cn estimte the eqution of corne sphere by using three collimted IR-LEDs s shown in Eq.(6) (see section 2.4). 2.3 Cmer Clibrtion In our system, we defined three coordinte: monitor, cmer nd eye. Becuse the cmer ws ttched below the monitor plne (in HMD environment), it ws necessry to consider the pnning nd tilting ngle of the cmer coordintes in reltion to the monitor coordintes. Also, there re the trnsltionl vector between the monitor coordinte nd the gimbls of the cmer nd tht between the gimbls nd the CCD center. Tht is becuse users gze t point on the monitor, nd then the eye feture point is projected to the cmer coordinte. So, in order to obtin the 2D perspective point (c h ) of w h of the cmer coordinte, we first ligned the cmer nd monitor coordintes s shown in Eq.(1) [7][18]. c = PCRGw (1) h Here, P is the perspective trnsform mtrix, nd C is the trnsltion mtrix between the gimbl center nd the CCD center. R is the rottion mtrix cused by pnning nd tilting ngle of the cmer coordinte in reltion to the monitor coordinte. G is the trnsltion mtrix between the monitor coordinte nd the gimbl center of the cmer. To determine (clibrte) the internl nd externl cmer prmeters, we defined Eq. (1) s c h = Aw h (here, A = PCRG) by homogeneous representtion s shown in Eq. (2) [7] [18]. c c c c h1 h 2 h3 h 4 = h X Y Z 1 From Eq.(2), we obtined two equtions with 12 unknown coefficients, s shown in Eq.(3) X + 12 X + Y Y + Z Z 41 xx 42 yx xy yy xz yz x y + 24 = 0 = 0 Becuse we obtined two equtions (from Eq.(3)) per pir of 2D-3D clibrtion points (x, y) - (X, Y, Z), we obtined the 2D-3D clibrtion points by clibrtion pnel ccording to different Z positions. From Eq.(3), we were ble to obtin the reltionship between the 2D nd 3D points (x, y) - (X, Y, Z), s shown in Eq.(4). x = ( y = ( X + Y + 12 X Y + Z Z ) /( ) /( X X + Y Y + Z Z + 44 ) ) (2) (3) (4)

5 A Study on Eye Gze Estimtion Method Bsed on Corne Model of Humn Eye 311 With the clculted 11 ~ 44 nd Eq.(2), we were ble to obtin the 2D position (c h ) of the cmer coordinte from the 3D points (w h ) of the monitor coordinte. In ddition, with the 11 ~ 44, Eq.(2) nd the known Z distnce, we were ble to obtin the 3D points (w h ) in the monitor coordinte from the 2D position (c h ) in the cmer coordinte. 2.4 Clculting Corne Sphere Eqution by Three Points on the Corne Surfce In this pper, we use Z-distnce mesuring method tht uses only one gze detection cmer nd collimted IR-LED. This method is shown in Fig. 3 [8][18]. Fig. 3. Mesuring Z-distnce between cmer nd eye by cmer nd collimted IR-LED[8] In Fig. 3, b is the distnce between the collimted IR-LED nd the center of the cmer lens, θ is the ngle between the lens plne nd the direction of the collimted IR-LED, x' is the distnce between the center of the bright spot in the imge nd the center of the imge. Here, b nd θ did not chnge fter the initil setup of cmer nd the collimted IR-LED. So, we obtined the vlues by initil cmer clibrtion nd used them s constnt vlues. (x p, y p ) is the imge coordinte point of the bright spot center which is shped by the collimted IR-LED. From tht, the Z distnce (D) of the bright spot ws clculted s follows [8]: b λ D = λ cotθ + x' (5)

6 312 E.C. Lee nd K.R. Prk So, by using three bright spots mde by three collimted IR-LEDs, we could obtin ccurte eqution of the corne sphere bsed on the mesured three Z-distnce(D)s nd Eq.(6). In cptured eye imges, there were three bright spots in the input imge, which were generted by the three collimted IR-LEDs, respectively. The collimted IR- LEDs (880 nm) hve nrrow illumintion ngles nd re used for mesuring the Z distnce between the eye nd the cmer. On the other hnd, the Illuminted IR-LED (850 nm) hs wide illumintion ngle nd is used for illuminting the entire eye region. In ddition, the pupil center is used for clculting the user's gze vector. So, we detected the pupil region by using the illumintion IR-LED nd we detected the bright spots by using three collimted IR-LEDs. To detect the position of the pupil center in the imge, we firstly binrized the input imge (by using threshold of 30). Then, we obtined the ccurte position by component lbeling nd ellipse Hough trnsform in the serching region. We then binrized the input imge gin (by using threshold of 200) in order to detect the bright spots produced by the three collimted IR-LEDs. In this cse, we defined three serching regions in the input imge. Then, we were ble to obtin the ccurte center position of the bright spots by component lbeling nd ellipse Hough trnsform [14] in ech serching region. Becuse we used n IR-Pss filter (which psses IR light over wvelength of 750nm) in front of the cmer lens with n IR-LED illumintor, the vrince of the imge brightness (ffected by environmentl lighting) ws not gret. In ddition, the HMD monitor (which ws closely positioned in front of the user's eye) ws ble to block some mount of environmentl light. Due to those resons, we were esily ble to determine the threshold vlue for binriztion. Then, bsed on the three obtined 3D points on the corne surfce, we clculted the eqution of corne sphere, s shown in Eq.(6). ( = x xc ) + ( y y c ) + ( z zc ) 7.8 (6) Becuse we knew the three 3D positions of bright spot on corne surfce, we were ble to determine the three unknown prmeters (X c, Y c, Z c ) in Eq.(6). We used LMS (Lest Men Squre) lgorithm to estimte the prmeters [16]. To clculte the user's gze position on HMD monitor, we obtined the 3D position of the pupil center. For tht, we detected the pupil center in the 2D nd obtined the 3D line which psses through the pupil center nd the center of the cmer lens bsed on Eq.(1). Then, we obtined the 3D position of the pupil center which is the intersected point of the 3D line nd the clculted corne sphere eqution. Bsed on the detected 3D positions of the pupil center nd the center of corne sphere, we were ble to obtin the gze direction in the HMD monitor coordinte. 2.5 Compensting Angle Kpp However, we should compenste for the discrepncy between the pupillry nd the visul xis in order to obtin more ccurte gze position. We define the "ppillry

7 A Study on Eye Gze Estimtion Method Bsed on Corne Model of Humn Eye 313 xis" s the line pssing through the center of the pupil perpendiculrly to the pupil plne [10]. When humn gzes t n object, the eye rottes to mke light rys fll on the fove. The fove is smll re in the retin where most cones re locted. The line pssing through the pupil center nd the fove is clled the line of sight or "visul xis", which represents true gze vector. Generlly, the visul xis does not coincide with the pupillry xis. There exists n ngulr discrepncy usully known s ngle Kpp between the two xes. However, the "visul xis" is more difficult to mesure becuse the fove is invisible [12]. The Kpp vrince is so gret [13] nd it cn increse finl gze error much. To overcome such problem in our system, when ech person uses our gze system, he or she is requested to gze t the one point of monitor plne t initil user-dependent clibrtion stge [18]. From tht, the ngle Kpp is mesured bsed on offset between the position of pupillry rrivl point nd the one point of monitor. Through user dependent clibrtion like this, we could compenste the discrepncy hving individul vrition cused by ngle Kpp. 2.6 Clculting Gze Position on Virtul Monitor Screen of HMD Conventionl HMD systems use convex lens in front of physicl monitor screen nd the user cn see the mgnified virtul monitor screen. From the HMD specifictions [4], we know the size of the virtul monitor screen nd the distnce between the physicl nd virtul monitor screen. In ddition, the size of the virtul monitor screen corresponds to the monitor resolution of our HMD virtul screen ( pixels). Bsed on the HMD spec. [4], we obtined the 3D position of the virtul monitor screen s Z = -3,937 from the right viewing window plne of Fig. 1. We then clculted the gze position on the virtul monitor screen from the intersection point between the gze vector nd Z = -3,937. In this cse, s mentioned in section 2.5, the directionl ngle of gze vector is previously compensted by the mesured ngle Kpp, horizontlly nd verticlly. 3 Experimentl Results Our gze detection lgorithm ws tested with Pentium-IV 2.4 GHz CPU. To mesure ccurcy, we performed following test. A totl of 50 users were sked to gze t 12 specific points, s shown in Fig. 4. The test ws iterted 20 times. Experimentl results showed tht the verge gze detection error ws bout In the first experiment, we compred the ccurcy of our method to tht of conventionl 3D method [2]. In this experiment of mesuring the ccurcy of the conventionl 3D pproch (which ws pplied to our HMD environment), we compenste the ngle Kpp by using our method s shown section 2.5. Fig. 4 (b) shows n exmple of the conventionl gze point estimtion result. Experimentl results show tht the gze error of the conventionl 3D method ws 1.1 degrees, which ws greter thn ours. Tht is becuse conventionl 3D methods [2] re optimized for gze estimtion

8 314 E.C. Lee nd K.R. Prk () (b) Fig. 4. Exmples of gze detection results when using the proposed method compred to those when using conventionl 3D pproch [2]. () Exmple of gze results when using the proposed method. (b) Exmple of gze results when using conventionl 3D pproch [2] (Reference points re mrked with " " nd estimted gze points re mrked with " "). in desktop environments. The dvntge of our proposed method is tht since it cn utomticlly compute the 3D position of the corne surfce by using three collimted IR-LEDs nd clibrtion mtrix, the estimted 3D position of the eyebll nd the finl gze position re very ccurte hving the gze error of 0.89 degrees. Also, our method requires user to gze t only one point for user-dependent clibrtion. In the next experiment, we compred the ccurcy of our method to tht of the 2D method. The 2D method uses functions which mp 2D eye feture points into the gze position in the virtul screen of HMD monitor without clculting the 3D position or the direction of the eyebll nd the monitor [1][11]. Â

9 A Study on Eye Gze Estimtion Method Bsed on Corne Model of Humn Eye 315 () (b) Fig. 5. Exmples of gze detection results using liner interpoltion nd the geometric trnsform method. () Exmple of the gze result using liner interpoltion (b) Exmple of the gze result using the geometric trnsform method. The 2D method uses two kinds of mpping functions, which re the liner interpoltion mpping function [1] nd the geometric trnsform mpping function [10][11]. The liner interpoltion method requires tht the user minimlly gzes t two positions on monitor (such s the upper-right corner nd the lower-left corner positions of the virtul screen). Bsed on the two-gze position informtion, other gze positions cn be clculted by liner interpoltion. To increse gze detection ccurcy, four positions (such s the upper-right, lower-right, upper-left nd lower-left positions of the virtul screen) my be used in the user clibrtion stge [1]. However, since the virtul screen is rectngle, they use the geometric trnsform method for mpping points between the distorted qudrngle nd rectngle [10][11]. Experimentl results showed tht the RMS errors of gze detection by liner interpoltion nd geometric

10 316 E.C. Lee nd K.R. Prk trnsform were 1.8 nd 1.2 degrees, respectively, which were greter thn ours. Fig. 5 shows exmples of the gze point estimtion results. In experimentl result, the gze error of 0.89 corresponds to tht of bout 34 pixels (27 pixels on the X xis nd 20 pixels on the Y xis) on HMD virtul screen of pixels. Also, our method does not tke much time to clculte the user's gze position on monitor (26 ms with Pentium-IV 2.4GHz). 4 Conclusion In this pper, we present new method for estimting gze points with requiring user to gze t only one point for user-dependent clibrtion. In order to estimte the 3D direction of n eyebll, we used corne sphere model nd clculted the 3D position of the corne eyebll by using cmer, three collimted IR-LEDs nd one illuminted IR-LED. Experimentl results showed tht the gze estimtion error of our proposed method ws 0.89 degree. In future work, we pln to mesure gze detection ccurcy in vrious environments with more field tests. Acknowledgements This work ws supported by the Kore Science nd Engineering Foundtion (KOSEF) through the Biometrics Engineering Reserch Center (BERC) t Yonsei University. References 1. J. Kim et l., "Intelligent process control vi gze detection technology," Engineering Appl ictions of Artificil Intelligence, vol. 13, no. 5, pp , Aug J. G. Wng nd E. Sung, "Study on eye gze estimtion," IEEE Trnsctions on Systems, Mn nd Cybernetics, Prt B, vol. 32, no. 3, pp , June S. W. Shih et l., "A Novel Approch to 3-D Gze Trcking Using Stereo Cmers," IEEE Trns. on SMC, Prt B, vol. 34, no. 1, pp , Feb (ccessed on October 20th, 2005) 5. (ccessed on October 20th, 2005) 6. (ccessed on September 20th, 2005) 7. Mubrk Shh, "Fundmentls of Computer Vision", pp. 9-14, Rmesh Jin, Rngchr Ksturi, Brin G.Schunck, "Mchine Vision", McGrw-Hill Inte rntionl Editions, pp , Gullstrnd A, "Helmholz's physiologicl optics", Opticl Society of Americ, pp , Rfel C. Gonzlez, Richrd E. Woods, "Digitl Imge Processing", Prentice-Hll, Inc., Se cond Edition, Jeong Jun Lee et l., "Gze detection system under HMD environments for user interfce", the Joint Conference of ICANN/ICONIP, Istnbul, Turkey, June 26-29, Eskridge JB et l., "The Hirschberg test: double-msked clinicl evlution", Americn j ournl of optometry nd physiologicl optics, September, F. Scheffel, "Kpp nd Hirschberg Rtio Mesured with n Automted Video Gze Trc ker", Optometry & Vision Science. Vol. 79(5), pp , My 2002

11 A Study on Eye Gze Estimtion Method Bsed on Corne Model of Humn Eye Rfel C. Gonzlez, Richrd E. Woods, "Digitl Imge Processing", Prentice-Hll, Inc., Se cond Edition, pp , Jeong Jun Lee, "Three Dimensionl Eye Gze Estimtion in Werble Monitor Environme nt", Ph.D Thesis, Yonsei University, Steven C. Chpr et l., "Numericl Methods for Engineers", McGrw-Hill, K.R. Prk et l., "Gze Position Detection by Computing the 3-Dimensionl Fcil Positio ns nd Motions", Pttern Recognition, Vol. 35, No.11, pp.2559~2569, Eui Chul Lee, "A Study on Gze Trcking Method bsed on Three Dimensionl Anlysis o f Humn Eye, Smsung Humn-Tech Pper Awrd, submitted.