An Automatic Personal Calibration Procedure for Advanced Gaze Estimation Systems

Transcription

1 BME An Automatc Personal Calbraton Procedure for Advanced Gaze Estmaton Systems Dmtr Model Moshe Ezenman Abstract Gaze estmaton systems use calbraton procedures to estmate subject-specfc parameters that are needed for the calculaton of the pont-of-gaze. In these procedures, subjects are requred to fxate on a specfc pont or ponts n space at specfc tme nstances. Advanced remote gaze estmaton systems can estmate the optcal axs of the eye wthout any personal calbraton procedure, but use a sngle calbraton pont to estmate the angle between the optcal axs the vsual axs (lne-of-gaze). hs paper presents a novel calbraton procedure that does not requre actve user partcpaton. o estmate the angles between the optcal vsual axes of each eye, ths procedure mnmzes the dstance between the ntersectons of the vsual axes of the left rght eyes wth one or more observaton surfaces (dsplays) whle subjects look naturally at these dsplays (e.g., watchng a vdeo clp). heoretcal analyss computer smulatons show that the performance of the proposed procedure mproves when the range of angles between the vsual axes vectors normal to the observaton surfaces ncreases. Experments wth 4 subjects show that the subject-specfc angles between the optcal vsual axes can be estmated wth a rootmean-square error of.5. Index erms emote gaze estmaton, pont of gaze, personal calbraton, calbraton free, optcal axs, vsual axs. I. INODUCION HE pont-of-gaze (PoG) s the pont wthn the vsual feld that s maged on the hghest acuty regon of the retna, known as fovea. Eye gaze trackng (EG) systems that estmate PoG are used n a large varety of applcatons such as studes of cogntve processes [1], drver behavor [], psychology psychatry [3, 4], marketng advertsng [5], plot tranng [6] human-computer nterfaces [7, 8]. Manuscrpt receved Aprl 3, 9. hs work was supported n part by a grant from the Natural Scences Engneerng esearch Councl of Canada (NSEC), n part by scholarshps from NSEC the Vson Scence esearch Program (oronto Western esearch Insttute, Unversty Health Network, oronto, ON, Canada). D. Model s a Ph.D. student wth the Department of Electrcal Computer Engneerng, Unversty of oronto, oronto, ON M5S 3G4, Canada (phone: ; fax: ; e-mal: dmtr.model@utoronto.ca). M. Ezenman s wth the Department of Electrcal Computer Engneerng, the Department of Ophthalmology Vson Scences, the Insttute of Bomaterals Bomedcal Engneerng, Unversty of oronto, oronto, ON M5S 3G9, Canada (e-mal: ezenm@ecf.utoronto.ca). Copyrght (c) 9 IEEE. Personal use of ths materal s permtted. However, permsson to use ths materal for any other purposes must be obtaned from the IEEE by sendng an emal to pubs-permssons@eee.org. All EG systems use personal calbraton procedures to estmate subject-specfc parameters that are needed for the calculaton of the pont-of-gaze. Durng these calbraton procedures, subjects are requred to fxate on a specfc pont or ponts n space at specfc tme nstances. EG systems that do not use physologcal model of the eye, use calbraton procedures wth multple calbraton ponts [9] to determne subject-specfc mappngs between locatons of eye features (such as pupl center, corneal reflectons, etc.) to gaze locatons on the observaton surface [1]. Systems that employ model of the eye to estmate PoG [11-13] use calbraton procedures to estmate subject-specfc eye-model parameters. As was shown n [14, 15], a system that uses a stereo par of vdeo cameras could estmate the center of curvature of the cornea the optcal axs of the eye wthout any user calbraton. However, snce human gaze s not drected along the optcal axs, but rather along the vsual axs [16], a onepont calbraton procedure s stll requred for the estmaton of the angle between the optcal vsual axes. Durng ths calbraton procedure, the subject s requred to look at a calbraton pont so that ts mage wll fall on the fovea. However, n some applcatons, such as covert montorng or studes wth young chldren or mentally challenged subjects, t s mpossble to relably perform even a one-pont calbraton routne. herefore, one of the most mportant goals n the feld of gaze trackng technology s to estmate human PoG wthout calbraton procedures that requre actve user partcpaton. hs paper descrbes a new methodology to estmate the 3D angle between the optcal vsual axes of the eye. hs methodology does not requre subjects to fxate on any specfc calbraton pont(s). Instead, t reles on the assumpton that the vsual axes of both eyes ntersect on an observaton surface uses ths constrant to calculate the subject-specfc angles between the optcal vsual axes [17, 18]. hs assumpton holds true for the vast majorty of subjects vewng condtons, but can be volated for certan group of patents (e.g., patents wth strabsmus). he paper s organzed as follows. he next secton descrbes the algorthm for the estmaton of subject-specfc angles between the optcal vsual axes. he propertes of the proposed algorthm are dscussed n Secton III. Secton IV presents both smulaton expermental results. Fnally, conclusons are presented n Secton V.

2 BME Fg. 1. A smplfed schematc dagram of the eye (not to scale). he optcal axs of the eye connects the center of the pupl wth the center of curvature of the cornea. Gaze s drected along the vsual axs, whch connects the center of the regon of hghest acuty of the retna (fovea) wth the center of curvature of the cornea. II. AUOMAIC CAIBAION MEHODOOGY A. Model Fg. 1 presents a smplfed schematc dagram of the eye [1]. he lne connectng the center of curvature of the cornea wth the center of the pupl defnes the optcal axs. he lne connectng the center of the fovea wth the center of curvature of the cornea defnes the vsual axs or the lne-of-gaze. he average magntude of the angle between the optcal vsual axes s 5. hs angle has both horzontal (nasal) vertcal components, whch exhbt consderable nter-personal varaton [16]. o develop an algorthm that estmates the angle between the optcal vsual axes, two coordnate systems are defned. he frst coordnate system s a statonary rghthed Cartesan World Coordnate System (WCS) wth the orgn at the center of the man dsplay, the X w -axs n the horzontal drecton, the Y w -axs n the vertcal drecton the Z w -axs perpendcular to the dsplay (see Fg. 1). he second coordnate system s a non-statonary rght-hed Cartesan Eye Coordnate System (ECS), whch s attached to the eye, wth the orgn at the center of curvature of the cornea, the Z eye axs that concdes wth the optcal axs of the eye X eye Y eye axes that, n the prmary gaze poston, are n the horzontal vertcal drectons, respectvely. he X eye -Y eye plane rotates accordng to stng s law [19] around the Z eye axs for dfferent gaze drectons. In the ECS, the unknown 3D angle between the optcal the vsual axes of the eye can be expressed by the horzontal 1, α, vertcal, β, components of ths angle (see Fg. 1). he unt vector n the drecton of the vsual axs wth respect to the ECS, ECS, s then expressed as: sn( )cos( ) ECS(, ) sn( ) (1) cos( )cos( ) he unt vector n the drecton of the vsual axs wth respect to the WCS,, can be expressed as: (, ) (, ) () ECS where s the rotaton matrx from the ECS to the WCS (ndependent of α β), whch can be readly calculated from the orentaton of the optcal axs of the eye stng s law [19]. Because the vsual axs goes through the center of curvature of the cornea, c, the PoG ( ψ ) n the WCS s gven by: ψ(, ) c k(, ) (, ) ck(, ) (, ) (3) where k s a lne parameter defned by the ntersecton of the vsual axs wth the observaton surface. For example, f the observaton surface s a plane defned by x n x h, then k, wll be gven by: h nc k, n, where n s the normal to the dsplay surface. ECS (4) 1 he angle between the projecton of the vsual axs on the X eye -Z eye plane the Z eye axs. It s equal to 9 f the vsual axs s n the X eye drecton. he angle between the vsual axs ts projecton on the X eye -Z eye plane. It s equal to 9 f the vsual axs s n the +Y eye drecton.

3 BME Note that snce EG systems that use a stereo par of vdeo cameras can estmate the center of curvature of the cornea the optcal axs of the eye wthout any user calbraton [1, 15, ], c are known. B. Automatc Calbraton Algorthm he proposed automatc calbraton algorthm for the estmaton of α, β, α β s based on the assumpton that at each tme nstant the vsual axes of both eyes ntersect on the surface of an observaton surface (dsplay). he superscrpts are used to denote parameters of the left rght eyes, respectvely. he unknown angles α, β, α β can be estmated by mnmzng the dstance between the ntersectons of the left rght vsual axes wth that surface (left rght PoGs). he objectve functon to be mnmzed s then F(,,, ) ψ (, ) ψ (, ) (5) where the subscrpt dentfes the -th gaze sample. he above objectve functon s non-lnear, thus a numercal optmzaton procedure s requred to solve for the unknown angles α, β, α β. However, snce the devatons of the unknown angles α, β, α β from the expected average values,, are relatvely small, a lnear approxmaton of (3) can be obtaned by usng the frst three terms of ts aylor s seres expanson: ψ(, ) ψ(, ) ψ(, ) ( ), (6) ψ(, ) ( ) et, ψ ψ(, ) c k, (7) where (, ) k k(, ). hen, usng (3), ψ(, ) a k k (8), ψ(, ) b k k (9), where, from (1)-(), cos( ) cos( ) (, ) (1), sn( )cos( ) sn( )sn( ) (, ) cos( ) (11), cos( )sn( ), from (4), k k k(, ) k(, ) k,, k n n n n (1) (13) Usng the above lnear approxmatons, the sum of the squared dstances between the left rght PoGs n the objectve functon (5) can be expressed as (,,, ) M x y (14) F where M a b a b s a 3x4 matrx, y ψ ψ s a 3x1 vector,, x ( ) ( ) ( ) ( ) s a 4x1 vector of unknown angles ( denotes transpose). he subscrpt " s used to explctly ndcate the correspondence to the specfc tme nstance or -th gaze sample. he soluton to (14) can be obtaned n a closed form usng least squares as 1 x M M M y, (15) opt ( ) where the optmzaton over several tme nstances s acheved by stackng the matrces on top of each other: M y 1 1 M y M ; y MN yn (16) Fnally, the estmates of the subject-specfc angles s gven by ˆ ˆ ˆ opt ˆ x (17) Snce the objectve functon (14) s a lnear approxmaton of the objectve functon (5), several teratons of (7)-(17) mght be needed to converge to the true mnmum of the objectve functon (5). In the frst teraton,,, are set to zero. In subsequent teratons, are set to the values of ˆ, ˆ,, ˆ, ˆ from the precedng teraton. he above methodology to estmate the angle between the optcal vsual axes s sutable for on-lne estmaton as a new matrx M s added to M a new vector y s added to y for each new estmate of the centers of curvature of the corneas the optcal axes that are provded by the EG.

4 BME III. POPEIES OF HE OBJECIVE FUNCION In a nose-free case (.e., c are nose-free), the mnmum of the objectve functon (5) s acheved when,, ψ ψ (18) In ths case, two PoGs on the observaton surface are suffcent to estmate the four unknown angles. However, when the algorthm uses estmates of the centers of curvature of the corneas estmates of the optcal axes that nclude nose, more than two PoGs are requred for a stable soluton. Moreover, the performance of the algorthm depends on the spatal dstrbuton of these PoGs. o smplfy the analyss, eq. (18) can be nterpreted as an mplct defnton of, as a functon of, :, (19) f g, () he soluton, ˆ, ˆ, ˆ, ˆ, s determned by the ntersecton of 3D surfaces 3, f g, for all gaze ponts (=1,,N). In general, the range of angles between the surfaces at the pont of ntersecton determnes the stablty of the soluton (.e., when the range of all the angles between the surfaces s small, all surfaces are approxmately parallel to each other the soluton s susceptble to nose). he purpose of the followng analyss s to defne the parameters that affect the range of angles between the surfaces f, =1,,N g, =1,..,N, at the soluton pont. he angle between two surfaces f ˆ ˆ, s gven by: f j at the soluton f 1 f j cos, (1) f f j f f where f,, 1. By substtutng f wth g, one can obtan a smlar expresson for the angle between surfaces g g j. By usng eq. (7)-(11) employng the mplct dervaton of (18) wth respect to, the partal dervatves f f g g,, can be calculated by the followng two equatons: k k () k k k k k k (3) k k k k he angles between two surfaces can be calculated by substtutng the estmated partal dervatves nto (1). o study parameters that affect the angles between surfaces, f f j, we wll make use of several examples. In the frst example, a subject s sttng 75 cm n front of an observaton surface (a plane at Z=, Fg. ), whle lookng at two arbtrary ponts, say, p 1 =[-1,,] p =[1,,] 4. Fg. shows the two surfaces, f 1 f, that correspond to these pont Fg. (c) shows a cross-secton of the functons f 1 f at ˆ. Fg. (c) demonstrates clearly, that even though the two curves ntersect at ˆ, the angle between the two surfaces at the soluton s relatvely small (1. ). he analyss of ths example n the Appendx provdes a close form expressons for the partal dervatves of each surface at the soluton pont. he partal dervatves of f, are gven by: f x n 1 (4) k n f x n (5) k n Equatons (4) (5) show that the partal dervatves,, depend on the followng parameters: 1) the dstance between two eyes, x c c ; ) the lne parameter, k (defned n (4)); 3) the angle between the normal to the observaton surface at the pont-of-gaze, n, the gaze drecton, ; 4) the angle between the normal to the observaton surface, n, the dervatve of the gaze drecton:, n (4) or,, n (5). For the above example n 1, (confguraton descrbed n Fg. ): 1, 1 1, so 1 for both p 1 p. Note that snce the partal dervatves n Equatons (4) (5) are smlar for the two ponts, the angle between the two surfaces wll be relatvely small. 3 Equatons (19) () gve rse to parametrc surfaces,, (, ),, (, ) f surfaces are referred to as g, respectvely. hese f g. 4 All coordnates n ths paper are specfed n centmetres.

5 BME (c) Fg.. op vew. Subject looks at a observaton surface (a plane Z=). are vectors n the drecton of the lne-of-gaze of the left rght eyes respectvely. n s the normal to the plane. x s the horzontal separaton between to eyes; Plot of, whch le on a plane Z= n the WCS. (c) Plot of two cross-sectons:, ˆ f, =1,. f for two PoGs: p 1 =[-1,,] p =[1,,], Based on Eq. (4), larger dfferences between f / 1 f / can be acheved f the normal to the observaton surfaces, n, the dervatve of the gaze drecton,,, are at larger angles to each other f the nner products, n1, 1 n,, have dfferent sgns. hs stuaton can be realzed f the same two ponts of gaze, p 1 p, are on two dfferent observaton planes wth dfferent orentatons, say, Z-X-1 = Z+X-1 =, respectvely, (Fg. 3). In ths case, the normal to the observaton surface, n, the dervatve of the gaze drecton,,, are at approxmately 45 to each other the nner products, n1, 1 n,, have dfferent sgns, the angle between the surfaces f 1 f becomes 6.5 (see Fg. 3 compare t to Fg. Fg. 3. op Vew. Subject look at a surface consstng of two planes wth normals n 1 n. 1 s the drecton of gaze towards p 1, whle s the drecton of gaze towards p. Sold lnes: plot of f, ˆ from eq. (19) for two PoGs: p 1 =[-1,,] whch les on a plane Z-X-1= p =[1,,] whch les on a plane Z+X-1=. Dashed lnes: lnear approxmaton based on the slope n (4). (c)). Fg. 3 shows the actual cross-secton of f, ˆ along wth the lnear approxmaton based on Eq. (4). Note the excellent agreement between the two curves. Smlarly, based on Eq. (5), the dfference between the f slopes of the surfaces f n the drecton of,, can be ncreased by selectng ponts for whch n, are at relatvely large angles to each other the products n1,1 n have dfferent sgns. For example, ponts p 1 =[,-, 1,] p =[,1,] on the planes Z-Y-1 = Z+Y-1 =, respectvely, satsfy the above requrements. Fnally, to maxmze the dfferences n the angles between the surfaces f =1,,N, n the drectons of both, one can combne the above examples use a pyramd wth a pnnacle facng the subject as an observaton surface. Under the assumpton that the subject wll look at all four facets of the pyramd, the condtons smlar to those dscussed above f, wll be acheved. Fg. 4 shows plots of, of the pyramd. he ntersecton of, g for four PoGs that le on the four dfferent facets provdes an estmate of to determne f, =1,,4, ˆ ˆ,, whch s suffcent ˆ ˆ accordng to (). As expected from the analyss n the Appendx, the surfaces, g =1,,4, g are almost overlappng, as both g are approxmately constant for central ponts of gaze (.e., PoGs

6 BME for whch the drecton of gaze s approxmately along the Z wcs -axs) regardless of the orentaton of the observaton plane. Fg. 4. p 1 =[-1,,], p =[-1,,], p 3 =[,1,] p 4 =[,-1,] that le on the correspondng facets of the pyramd. f,, =1,..,4; g,, =1,..,4. he above analyss, whch was developed for the central gaze, demonstrates that the stablty of the soluton mproves as the range of angles between n (as well as,, ) for the varous ponts of gaze ncreases. In many practcal applcatons, the observaton surface conssts of a sngle plane (n =n, ). In ths case, a stable soluton requres that the ponts of gaze wll span a wde range of gaze drectons (e.g., by lookng at larger areas of the observaton plane). he performance of the algorthm wth complex observaton surfaces as well as wth a sngle observaton plane wll be dscussed n the next secton. MAAB. One thous PoGs were used to estmate the angle between the optcal vsual axes for each set of eye parameters (,, ). he ntal value for the angle between the optcal vsual axes was set to zero (the ntal value dd not affect the results as long as t was wthn ± of the actual value). he frst update of the algorthm was done after 1 PoGs to prevent large fluctuatons durng startup. he smulatons were performed wth four dfferent observaton surfaces: a) a pyramd (3 cm x 3 cm base, 15 cm heght) that was used n the theoretcal analyss n the experments that are descrbed n the next secton, b) a plane that provdes vewng angles n the range of ±14.9º horzontally ±11.3º vertcally (4 cm x 3 cm observaton surface that s smlar n sze to a montor), c) a plane that provdes vewng angles n the range of ±8º horzontally ±1.8º vertcally (8 cm x 6 cm observaton surface that s smlar n sze to a 4 montor), d) a plane that provde vewng angles n the range of ±46.8º horzontally ±38.7º (16 cm x 1 cm). For each observaton surface each level of nose n the estmaton of the optcal axs or n the estmaton of the center of curvature of the cornea the smulatons were repeated 1 tmes. IV. NUMEICA SIMUAIONS AND EXPEIMENS A. Numercal Smulatons In the frst set of numercal smulatons, the performance of the algorthm was evaluated as a functon of the nose n the estmaton of the optcal axs the center of curvature of the cornea of each eye as a functon of the sze shape of the observaton surface. In the frst smulaton, the head was fxed (.e., no head movements) the poston of the center of curvature of the cornea of the subject s rght eye, c, was set to [3 75] (.e., approxmately 75 cm from the dsplay s surface) whle the center of curvature of the cornea of the subject s left eye, c, was set to [-3 75] (.e., nter-pupllary dstance = 6 cm). he angles between the optcal vsual axes were romly drawn for each smulaton from a unform dstrbuton wth a range of (-5, ) for, (, 5) for (- 5, 5) for. he PoGs were romly drawn from a unform dstrbuton over the observaton surfaces the optcal axes of the two eyes were calculated. Nose n the estmates of the center of curvature of the cornea the optcal axs was smulated by addng ndependent zero-mean whte Gaussan processes to the coordnates of the centers of curvature of the cornea (X,Y Z) to the horzontal vertcal components of the drecton of the optcal axs. he automatc calbraton algorthm was mplemented n Fg. 5. Estmaton results wth a 4 cm x 3 cm observaton surface (e.g., montor). SD of nose n the angles of the optcal axs was set to.1 n the coordnates of the center of curvature of the cornea to.5 mm. Estmated subject-specfc angles; oot-mean-square Error (MSE) of PoG estmaton (top) the dstance between eft PoG - ght PoG (bottom). Fg. 5 shows an example of one smulaton wth a plane that provdes vewng angles n the range of ±14.9º horzontally ±11.3º vertcally (4cm x 3cm observaton surface that s smlar n sze to a montor). he stard devaton (SD) of the nose n the angles of the optcal axs was set to.1 the SD of the nose n the coordnates of the center of curvature of the cornea was set to.5 mm. As can be seen from Fg. 5, after approxmately 5 PoGs the soluton converges to wthn ±.5 of the true values of,,. Fg. 5 shows the root-mean-square (MS) error n the PoG estmaton (top) the value of the objectve functon (bottom). he objectve functon was effectvely mnmzed after the frst update (.e., wth 1 PoGs) whle the MS error contnues to declne untl teraton 5. hs s due to the fact that even though after 1 teratons the dstance between the ponts-of-gaze of the left rght eyes was

7 BME mnmzed, PoGs of the both eyes had smlar bases relatve to the actual PoGs. From teraton 1 to 5 the PoGs of the two eyes drfted smultaneously towards ther true locatons. Fg. 6 Fg. 7 show the MS error n the estmaton of as a functon of nose n the optcal axs n the center of curvature of the cornea, respectvely. he estmaton error had no bas (.e., zero-mean) n all the above smulatons. Fg. 6 Fg. 7 show that: a) the pyramdal observaton surface provdes the best performance (.e., smaller MS errors as compared to the planes) b) the MS error decreases as the range of vewng angle or equvalently the sze of the observaton surface ncreases. hs fndng s consstent wth the theoretcal analyss n the prevous secton that showed that by ncreasng the range of angles between the gaze vectors the normal to the observaton surface the soluton becomes less senstve to nose. he MS errors for the largest observaton plane (16 cm x 1 cm) the pyramd are smlar as they provde approxmately the same range of angles between the normals to the observaton surfaces the gaze vectors. Fg. 6 Fg. 7 show that for the system that was descrbed n [14] (SD of the nose n the estmaton of the optcal axs s.4 SD of the nose n the estmaton of the center of curvature of the cornea s 1. mm) nose n the estmaton of the optcal axs contrbutes more sgnfcantly to the MS error n the estmaton of the angle between the optcal vsual axes than nose n the estmaton of the center of curvature of the cornea. senstvty of the algorthm to head poston. For these smulatons, the coordnates of the center of curvature of the cornea of the rght eye, c, were set at 7 dfferent postons (all the combnatons of X= -7 cm, 3 cm, 13 cm; Y= -1 cm, cm, 1 cm; Z= 85 cm, 75 cm, 65 cm, e.g., c = [-7 75], [3 75], [13 75], etc). For each eye poston nose level n the optcal axs (the nose level n the coordnates of the center of the curvature of the cornea was set to 1 mm) the smulatons were repeated 1 tmes. Fg. 8 shows the aggregate (all head postons) MS error n the estmatons of as a functon of the nose level n the optcal axs. When the data n Fg. 8 are compared to the data n Fg. 6 t s clear that changes n head poston ntroduced bases that ncreased the MS errors n the estmaton of the angle between the optcal vsual axes. o underst these bases, t s mportant to note that the dstance between the left rght PoGs on the observaton surface s proportonal to the nose level n the drectons of the optcal axes of the left rght eyes to the dstance between the eyes the PoG. hs mples that for a gven level of the nose n the drecton of the optcal axes, the algorthm wll attempt to mnmze the dstance between the locaton of the PoGs on the observaton surface the subject s eyes. For example, when the head s movng up relatve to the locaton of the PoG on the observaton surface, there wll be a postve bas n the estmaton of. herefore, n the presence of head movements (or non-symmetrc dstrbuton of gaze ponts relatve to the head poston), the mnmum of the objectve functon represents a balance between an attempt to decrease the error between left rght PoGs due to a shft away from the actual values of an attempt to mnmze the aggregate dstance between the PoGs on the observaton surface the eyes. Fg. 6. MS errors for the angle between the optcal vsual axes, for four dfferent observaton surfaces, as a functon of nose n the estmaton of the components of the drecton of the optcal axs. ;. Fg. 8. MS errors for the angle between the optcal vsual axes, for four dfferent surfaces, as a functon of the nose n the estmaton of the components of the drecton of the optcal axs. 7 head postons (all the combnatons of X head = -1 cm, cm, 1 cm; Y head = -1 cm, cm, 1 cm; Z- head = -1 cm, cm, 1 cm). ;. Fg. 7. MS errors for the angle between the optcal vsual axes, for four dfferent surfaces, as a functon of the nose n the estmaton of the center of the curvature of the cornea. ;. Addtonal smulatons were performed to study the Fg. 8 shows that the performance of the algorthm mproves when the range of angles between the subject s gaze vectors, as he/she looks at the dsplays, vectors normal to the observaton surfaces ncreases. For a plane that provdes vewng angles n the range of ±14.9º horzontally ±11.3º vertcally (4 cm x 3 cm observaton surface), a nose level of

8 BME º n the components of the drecton of the optcal axs results n a MS error of.6º n the estmaton of. For the same nose level n the optcal axs, the MS error n the estmaton of s reduced to only.5º when a plane that provdes vewng angles n the range of ±46.8º horzontally ±38.7º vertcally (16 cm x 1 cm) s used. hs s smlar to the performance of the algorthm wth the pyramdal observaton surface. B. Experments wth real subjects Gaze Estmaton System A two-camera emote EG (EG) system [14] was used for experments wth four subjects. he SD of the nose n the estmaton of the components of the drecton of the optcal axs n ths system s.4 the SD of the nose n the estmaton of the coordnates of the center of curvature of the cornea s 1. mm. Subjects were seated at a dstance of approxmately 75cm from a computer montor (plane Z=) head movements were not restraned. Gven the lmted trackng range of the system (approxmately ±15º horzontally vertcally), only a relatvely small observaton planes (< montor) or more complcated pyramd observaton surfaces could be used for the experments. Because the expected MS errors wth a observaton surface ( >.5º > 3º, see Fg. 8) are larger than the expected errors when,, are set a-pror to -.5º,.5º,,, respectvely, there was no pont n evaluatng the performance of the algorthm wth such an observaton surface. herefore, the experments were performed wth a pyramd observaton surface (3 cm x 3 cm base, 15 cm heght, pnnacle towards the vewer). he base of the pyramd was mounted on the plane Z=9 (cm) the pnnacle was located at [,- 1.5,4]. Durng the experment, each subject was asked to look at the four facets of the pyramd (n any order) thous estmates of the center of curvature of the cornea the drecton of the optcal axs were obtaned for each eye. he subject-specfc angles between the optcal vsual axes were estmated, off-lne, usng the automatc calbraton algorthm. Next, the pyramd was removed the subject was asked to complete a stard one-pont calbraton procedure [14]. he values of the estmated subject-specfc angles between the optcal vsual axes wth the automated calbraton procedure a one-pont calbraton procedure are shown n able I. Snce the estmates of the subject-specfc angles between the optcal vsual axes that are obtaned by the one-pont calbraton procedure mnmze the MS error between the estmated PoG the actual PoG, the values obtaned by the one-pont calbraton procedure wll serve as a reference ( gold stard ) for the calculatons of the errors of the automatc calbraton procedure. he estmaton errors of the automated calbraton methodology are -.8±.59 n (left rght combned) -.19±.43 n (left rght combned). Gven the nose characterstcs of the gaze estmaton system, these estmaton errors are n the range predcted by the numercal smulatons (see Fg. 6 Fg. 8). ABE I HOIZONA ( ) AMD VEICA ( ) COMPONENS OF HE ANGE BEWEEN HE OPICA AND VISUA AXES Subject-specfc angles, ( ) Subjects: AutoCalb 1-pont calb AutoCalb estmated wth the automatc calbraton algorthm usng a pyramdal surface. 1-pont calb estmated wth a onepont calbraton procedure. Followng the estmaton of the angles between the optcal vsual axes, the subjects looked at a grd of nne ponts (3x3, 8.5 apart) dsplayed on a computer montor. Ffty PoGs were collected at each pont. he MS errors n the estmaton of the PoG for all four subjects are summarzed n able II. For the automatc calbraton procedure the average MS error n PoG estmaton for all four subjects s 1.3. For the one-pont calbraton procedure the average MS error s.8. ABE II POG ESIMAION EO MS error, ( ) Subjects: AutoCalb 1-pont calb eft Eye ght Eye eft Eye ght Eye AutoCalb estmated wth the automatc calbraton algorthm usng a pyramdal surface. 1-pont calb estmated wth a onepont calbraton procedure. V. DISCUSSION AND CONCUSIONS A novel calbraton procedure to estmate the angles between the optcal vsual axes that does not requre accurate fxaton on known ponts at specfc tme ntervals was presented. he procedure s based on the assumpton that at each tme nstant both eyes look at the same pont n space. Usng the novel procedure wth a EG system [14] that estmates the components of the drecton of the optcal axs wth an accuracy of.4 show that the angles between the optcal vsual axes can be estmated wth an MS error of.5. hese results, whch were acheved wth an observaton surface that ncluded four planes, are consstent wth the numercal analyss n Secton IV.A. Based on ths numercal analyss (see Fg. 8), an eye-trackng system that estmates the components of the drecton of the optcal axs wth accuracy that s four tmes better than the accuracy of the EG system

9 BME n [14] (.e.,.1 ) can use the novel calbraton procedure wth a relatvely small sngle observaton plane (3 cm x 4 cm, e.g., computer montor) to acheve a smlar MS error. By ncreasng the sze of the observaton plane to 6 cm x 8 cm, a smlar MS error can be obtaned wth a EG system that can estmate the components of the drecton of the optcal axs wth accuracy that s only two tmes better than the accuracy of the system descrbed n [14] (.e,. ). he use of a sngle observaton plane wll mprove consderably the utlty of the novel automatc personal calbraton procedure. User-calbraton-free gaze estmaton systems that estmate the PoG from the ntersecton of the optcal axs of one of the eyes wth the dsplay (e.g., [1]) or from the mdpont of the ntersectons of the optcal axes of both eyes wth the dsplay (e.g., []) can exhbt large, subject dependent, gaze estmaton errors. For the experments descrbed n Secton IV.B (see able II), calculatons of the PoG by the ntersecton of the optcal axs of one eye wth the computer montor results n MS errors rangng from 1 to 4.5 (average.5 ). By usng the mdpont, the MS errors for the four subjects are reduced to a range between 1.5 to 3. (average. ). he automatc personal calbraton procedure that was presented n ths paper further reduced the MS errors for the four subjects to a range between.8 to 1.7 (average 1.3 ). he experments wth a two-camera gaze estmaton system [14] show that wth the automatc user calbraton procedure the MS error of the PoG estmaton s 1.3 (see able II). When the same two-camera system s used wth a sngle pont calbraton procedure, the MS error s reduced to.8. Smlar MS error (.8 ) s obtaned when a state-of-the-art one-camera eye-trackng system [1] multple (>4) pont calbraton procedure are used. By ncreasng the complexty of the eye-trackng system from one camera to two cameras, the number of calbraton ponts can be reduced (from multple ponts to one pont) wthout sacrfcng accuracy. he algorthm developed n ths paper elmnates the need for any calbraton ponts, but further ncreases system complexty (the need to track both eyes) reduces the accuracy of the pontof-gaze estmaton. For specfc applcatons, one should consder the trade-offs between system complexty, system accuracy the number of calbraton ponts. APPENDIX Approxmate expressons for,, (mplctly defned n equatons () (3)) when subjects look at centrally located ponts on a plane n front of them (as n Fg. ) are derved. In ths case: k k k k k k k k k (A.6) 1 xˆ 1 yˆ (A.7) k k (A.8) k k o calculate, multply both sdes of eq. () by yˆ from the left. Usng approxmatons (A.6)-(A.8), all terms except for the last one wll vansh, yeldng: ˆ y k => k => (A.9) o calculate, multply both sdes of eq. () by xˆ from the left use (A.9): x ˆ k ˆ k ˆ k ˆ x x k x. Usng (A.6) : xˆ xˆ xˆ ( k k ) ( k k ) ( k k ) Snce xˆ k ( ) k k 1 x ˆ ( ) k k k (A.3) xˆ xˆ c k ψ ψ c k, usng (A.6) (A.7) : k( ) c c. Fnally, c c x x ˆ (A.31) k k ( ) (where x c c s the horzontal separaton between the two eyes). Now we can further smplfy (A.3): x 1 k (A.3) k hen, usng eq. (1) : x n 1 (A.33) k n o calculate, multply both sdes of eq. (3) by yˆ from the left. Usng the approxmatons (A.6)-(A.8), all terms except for the second one on the left-h sde the last one on the rght-h sde of (3) can be neglected, yeldng:

10 BME yˆ k ˆ y k => k k => 1 (A.34) o calculate, multply both sdes of Eq. (3) by xˆ from the left use (A.34): xˆ ˆ ˆ ˆ k x k k k x x k ˆ ( ) ( k ˆ kˆ x x x ) k ˆ x ( ) (A.35) ( k ˆ kˆ x x ) Usng (A.31) the approxmaton: xˆ k ˆ x k k, (A.35) can be smplfed to: x k (A.36) k hen, usng eq. (13) we fnally get: x n (A.37) k n EFEENCES [1] K. ayner, "Eye movements n readng nformaton processng: years of research," Psychologcal Bulletn, vol. 14, no. 3, pp. 37-4, Nov [] J.. Harbluk, Y. I. Noy, P.. rbovch, M. Ezenman, "An on-road assessment of cogntve dstracton: Impacts on drvers' vsual behavor brakng performance," Accdent Analyss & Preventon, vol. 39, no., pp , Mar. 7. [3] M. Ezenman,. H. Yu,. Grupp, E. Ezenman, M. Ellenbogen, M. Gemar,. D. evtan, "A naturalstc vsual scannng approach to assess selectve attenton n major depressve dsorder," Psychatry esearch, vol. 118, no., pp , May 3. [4] C. Karatekn. F. Asarnow, "Exploratory eye movements to pctures n chldhood-onset schzophrena attenton-defct/hyperactvty dsorder (ADHD)," Journal of Abnormal Chld Psychology, vol. 7, no. 1, pp , Feb [5] G. oshe, "Consumer eye movement patterns of Yellow Pages advertsng," Journal of Advertsng, vol. 6, no. 1, pp , [6] P. A. Wetzel, G. Krueger-Anderson, C. Poprk, P. Bascom, "An eye trackng system for analyss of plots scan paths," Unted States Ar Force Armstrong aboratory ech. ep. A/H , Apr [7] A.. Duchowsk, "A breadth-frst survey of eye-trackng applcatons," Behavor esearch Methods Instruments & Computers, vol. 34, no. 4, pp , Nov.. [8] J. H. Goldberg X. P. Kotval, "Computer nterface evaluaton usng eye movements: methods constructs," Internatonal Journal of Industral Ergonomcs, vol. 4, no. 6, pp , Oct [9] Z. Zhu Q. J, "Eye gaze trackng for nteractve graphc dsplay," Machne Vson Applcatons, vol. 15, no. 3, pp , 4. [1]. E. Hutchnson, K. P. Whte, Jr., W. N. Martn, K. C. echert,. A. Frey, "Human-computer nteracton usng eye-gaze nput," Systems, Man Cybernetcs, IEEE ransactons on, vol. 19, no. 6, pp , [11] A. Vllanueva. Cabeza, "Models for gaze trackng systems," J. Image Vdeo Process., vol. 7, no. 4, pp. 1-16, 7. [1] E. D. Guestrn M. Ezenman, "General theory of remote gaze estmaton usng the pupl center corneal reflectons," IEEE ransactons on Bomedcal Engneerng, vol. 53, no. 6, pp , Jun. 6. [13] W. Jan-Gang E. Sung, "Study on eye gaze estmaton," Systems, Man, Cybernetcs, Part B: Cybernetcs, IEEE ransactons on, vol. 3, no. 3, pp ,. [14] E. D. Guestrn M. Ezemnan, "emote pont-of-gaze estmaton wth free head movements requrng a sngle-pont calbraton," n Proc. of Annual Internatonal Conference of the IEEE Engneerng n Medcne Bology Socety, 7, pp [15] S. W. Shh J. u, "A novel approach to 3-D gaze trackng usng stereo cameras," IEEE ransactons on Systems, Man, Cybernetcs, Part B: Cybernetcs, vol. 34, no. 1, pp , Feb. 4. [16]. H. S. Carpenter, Movements of the eyes. ondon, UK: Pon, [17] D. Model, E. D. Guestrn, M. Ezenman, "An Automatc Calbraton Procedure for emote Eye-Gaze rackng Systems," n 31st Annual Internatonal Conference of the IEEE EMBS, Mnneapols, MN, USA, 9, pp [18] M. Ezenman, D. Model, E. D. Guestrn, "Covert Montorng of the Pont-of-Gaze," n IEEE IC-SH, oronto, ON, Canada, 9, pp [19] H. Helmholtz, Helmholtz's treatse on physologcal optcs. ranslated from the 3 rd German ed. Edted by J. P. C. Southall ochester, NY: Optcal Socety of Amerca, 194. [] E. D. Guestrn M. Ezenman, "emote pont-of-gaze estmaton requrng a sngle-pont calbraton for applcatons wth nfants," n Proc. of the 8 Symposum on Eye rackng esearch & Applcatons Savannah, GA, USA: ACM, 8, pp [1] S.-W. Shh, Y.-. Wu, J. u, "A calbraton-free gaze trackng technque," n In Proceedngs of 15th Int. Conference on Pattern ecognton, pp []. Nagamatsu, J. Kamahara, N. anaka, "Calbraton-free gaze trackng usng a bnocular 3D eye model," n Proceedngs of the 7th Internatonal Conference on Human factors n Computng Systems, Boston, MA, USA, 9. Dmtr Model was born n Moscow, ussa, n 198. He receved the B.Sc. (Cum aude) M.Sc. degrees n electrcal engneerng from echnon Israel Insttute of echnology n 6, respectvely. He s currently workng toward the Ph.D. degree at the Department of Electrcal Computer Engneerng, Unversty of oronto. Hs current research nterests nclude eye-trackng gaze estmaton technology applcatons, sgnal mage processng, numercal optmzaton. Moshe Ezenman was born n el-avv, Israel, n 195. He receved the B.A.Sc., M.A.Sc., Ph.D. degrees n electrcal engneerng from the Unversty of oronto, oronto, ON, Canada, n 1978, 198, 1984, respectvely. Snce 1984, he has been wth the Faculty of the Unversty of oronto, where he s currently an Assocate Professor n the Department of Electrcal Computer Engneerng, the Department of Ophthalmology Vson Scences, also at the Insttute of Bomaterals Bomedcal Engneerng. He s also a esearch Assocate at the Eye esearch Insttute, Canada, the Hosptal for Sck Chldren, oronto. Hs current research nterests nclude detecton estmaton of bologcal phenomena, eye trackng gaze estmaton systems, vsual evoked potentals, the development of vson.