University of Bristol - Explore Bristol Research. Peer reviewed version. Link to published version (if available): /ICIP.2016.

Transcription

1 Anantrasirihai, P., Gilhrist, I., & Bull, D. (2016). Fixation identifiation for low-sample-rate mobile eye trakers. In IEEE International Conferene on Image Proessing (ICIP), 2016 Institute of Eletrial and Eletronis Engineers (IEEE). Peer reviewed version Link to published version (if available): /ICIP Link to publiation reord in Explore Bristol Researh PDF-doument This is the author aepted manusript (AAM). The final published version (version of reord) is available online via IEEE at Please refer to any appliable terms of use of the publisher. University of Bristol - Explore Bristol Researh General rights This doument is made available in aordane with publisher poliies. Please ite only the published version using the referene above. Full terms of use are available:

2 FIXATION IDENTIFICATION FOR LOW-SAMPLE-RATE MOBILE EYE TRACKERS N. Anantrasirihai, Iain D. Gilhrist and David R. Bull Bristol Vision Institute, University of Bristol, Bristol BS8 1UB, UK ABSTRACT This paper presents a novel method of fixation identifiation for mobile eye trakers. The most signifiant benefit of our method over the state-of-the-art is that it ahieves high auray for low-sample-rate devies worn during loomotion. This in turn delivers higher quality datasets for further use in human behaviour researh, robotis and the development of guidane aids for the visually impaired. The proposed method employs temporal harateristis of the eye positions ombined with statistial visual features extrated using a deep onvolutional neural network, inspired by models of the primate visual system, through the fovea and peripheral areas around the eye positions. The results show that the proposed method outperforms existing methods by up to 16 % in terms of lassifiation auray. Index Terms eye data, fixation, saade, onvolutional neural network, lassifiation 1. INTRODUCTION Eye trakers are frequently employed in human behaviour researh to apture instantaneous eye positions, revealing what visual information is important for different visual ontexts. The major events of eye motion patterns onsist of fixations of different temporal durations, saades of different spatial length, diretion and temporal length, and blinks. Fixations are generally onsidered to be the most interesting of the eye movement harateristis, sine they indiate what and when visual information is most probably registered to the brain [1]. Most researh on eye movements has been based on the presentation of stati images or video lips with partiipants being asked to look at a sequene of images on a omputer sreen. The eye trakers developed for this purpose urrently support ultra high sampling rates (up to 2000 Hz [2]), leading to high auray haraterisation of eye movements. In ontrast, the sampling-frequeny apability of mobile eye trakers is limited by additional onstraints, suh as light weight size and low power onsumption. With urrent tehnology, mobile eye trakers an provide sampling rates of up to 60 Hz [2,3], although those with higher rate are expensive. Cheaper or older models provide a sampling rate of up to 30 Hz, whih is lower than the requirement for fixation identifiation, sine saadi movements an be shorter than 50 ms (20 Hz) [4,5], of whih Nyquist frequeny for this saade traking is 40 Hz. Also, the traditional fixation-based algorithms annot be applied beause the head movement and traking behaviour (smooth downward movements to maintain fixation on an environmental feature during loomotion) ause dispersion to be too high. A large number of fixation identifiation methods have been proposed (as desribed in Setion 2). However, these methods an produe widely varying results, espeially for data from mobile eye trakers [6, 7]. Fixation patterns ourring during loomotion are different to those when viewing a sreen. Humans use oulomotor fixations when they are walking as the surrounding sene appears to be moving relative to them. These fixations enompass the ability to suppress oular drifts whilst maintaining a steady retinal image of a single target of interest. The main ontribution of this paper is a method to ahieve fixation identifiation, or noise removal, for low-sample-rate mobile eye trakers. We employ a learning method using a deep onvolutional neural network (CNN) as it resembles the organization of information in the human visual system. Saades, blinks and noise due to light interferene and large head movements are lassified as one lass against another for fixations. First, the eye positions and temporal relationships between them are employed to selet the fixation points to be used in the training proess, as the number of fixations is signifiantly larger than that of saades and noise ombined. Then the areas around the seleted points are employed to extrat key features used in the final lassifier. The remainder of this paper is organised as follows. Setion 2 presents existing work on fixation identifiation. Subsequently, we desribe our proposed method in Setion 3 and the results are shown in Setion 4. Finally the onlusions and future work are set out in Setion FIXATION IDENTIFICATION FOR EYE DATA The most ommon algorithm for fixation identifiation is veloity-based thresholding (I-VT) [8]. This lassifies eye movements based on the veloity (visual degrees per seond /s) of the diretional shifts of the eye. If the veloity is lower than the threshold, the point is lassified as a fixation. Subsequently, the lassifiation result is further ompared to that of the previous samples within the same lass to reate a

3 fixation or saade. Komogortsev et. al. suggests a veloity threshold of 30 /s in [9]. Tobii Tehnology applies a window length of 20 ms to average veloity when signals ontain high noise [8]. An adaptive threshold using mean and standard deviation was proposed in [6]. An alternative approah, using dispersion-based thresholds (I-DT) typially identifies gaze samples as belonging to a fixation if the samples are loated within a spatially loalised region (about 0.5 ) for a minimum period of time: the minimum allowed fixation duration ( mse) [10]. This simply follows the assumption that fixation points generally our near one another. Saades are then deteted impliitly as everything else. Probability-based algorithms, suh as Hidden Markov model [8], an be used to determine the most likely identifiations for a given protool. A two-step spatial dispersion threshold is proposed in [11]. The methods above exhibit poor performane in the detetion of fixations and saades in dynami senes. To address this, some fixation identifiation algorithms have been proposed for mobile eye trakers. These inlude a Bayesian mixture model for traffi hazard pereption [12] and the ombination of I-VT and duration sensitivity to trak fixations [13]. 3. PROPOSED SCHEME The proposed framework is depited in Fig. 1, where two lassifiers are employed for fixation-point seletion, desribed in Setion 3.1, and fixation identifiation, desribed in Setion 3.2. We identify eah event as one of two lasses: i) fixation or ii) saade and noise. In the training proess, the data from the eye trakers during eye blinks, ourring two to four times per minute under normal onditions, are removed Fixation-point seletion Generally, training a lassifier requires a balaned number of samples from eah lass; otherwise, the predited results an be biased towards the lass with more training samples. Eye traking data always ontains a signifiantly larger proportion of fixation samples, due to the nature of the human visual system. Therefore, if the number of training saades and noise is N t, we also use N t fixations. Here, we selet these N t fixations using a support vetor mahine [14], whih reates a hyperplane between the randomly seleted fixation points and the saade points (plus noise). Then, the distanes to the hyperplane are measured. We employ eight features, F 0, produed from eye positions and their temporal relationships as follows. Eye position at X t = {x t, y t }: two features indiate that eye positions obtained from the mobile eye traker exhibit entre-bias behaviour sine the head is often moved to improve vision. The eye positions near the edge of the image are often noise ourring when the light outside interferes with the amera used for deteting pupils. Eye positions (X t =[x t, y t ], t=[1..n]) Feature extration for fixation point seletion SVM lassifier gaze points and seleted fixation points 3-layer onvolutional neural network Feature extration for event predition SVM lassifier predited event at t Fig. 1. Diagram of the proposed method for event predition. Minimum distanes to D(X t 1 ) or D(X t+1 ) (in both horizontal and vertial diretions (two features)), where D(m n ) is a warp funtion from position m on frame n to the urrent frame s geometry: if the eye position in the urrent frame is near to that of the adjaent frame, it is likely to be a fixation. 2-point angular veloities between X t and D(X t 1 ), v (t 1,t) = Xt D(Xt 1) t, where t = 1 30 se for a 30- Hz eye traker: if the speed relative to the previous eye position mapped to the urrent frame s geometry is small, the urrent point is likely to be a fixation. 2-point angular veloities between X t and D(X t+1 ), v (t,t+1) = Xt D(Xt+1) t : if the speed relative to the next frame s eye position mapped to the urrent frame s geometry is small, the urrent point is likely to be a fixation. Angular aeleration, a t = v (t,t+1) v (t 1,t) t : high aelerations indiate small saades. ( ) v Interframe angle hange, θ t = aros (t,t+1) v (t 1,t), v (t,t+1) v (t 1,t) where v (m,n) = Xm Xn m n : if the angle between the vetors direted from the urrent eye position to the eye positions in the adjaent frames is lose to 180, the urrent eye position is likely to be a saadi movement. In this paper, ten randomly seleted fixation groups were used for training and the average distane from eah sample to these ten hyperplanes was omputed. The distane of sample x from the hyperplane is omputed as shown in Eq. 1, where w and b are the hyperplane normal vetor and the bias, respetively. d x = wt Fx 0 + b (1) w N t fixation points are seleted by ombining the N in fixation points inorretly lassified as saades with the N t N in points that have the smallest distane to the hyperplane

4 peripheral area resize to fovea area normalisation normalisation normalisation X t f p olour hannel ombine max-pooling ReLU Convolutional layer max-pooling ReLU Convolutional layer max-pooling ReLU Convolutional layer f F 4 fovea area R f R p G f G p B f B p F 1 = { μ, σ } 1 F 2= { μ, σ } 2 F 3= { μ, σ } 3 Fig. 2. Feature extration from 3-layer CNN. (sine generally N in < N t ). These points are the weakest lassifiers at this stage, so are ideal training data for the CNN. The orretly lassified points with large distanes to the hyperplane are easier to lassify Loal visual features by deep learning The final event lassifier employs the features F 0 and the statistial visual features extrated using the proess shown in Fig. 2. The fovea (3 ) and peripheral (6 ) areas around the seleted eye positions are used. The peripheral area is resized to the same size as the fovea area, h f h f pixels. Then, these two areas, 6 olour hannels in total, are ombined into one three-dimensional input to train the CNN Convolutional neural network A CNN is a biologially-inspired arhiteture that omprises multiple layers of neuron olletions that have learnable weights and biases. Their results are tiled so that they overlap to obtain a better representation of the original image. The CNN reates its filters values based on the task. Generally the CNN learns to detet edges from the raw pixels in the first layer, then uses the edges to detet simple shapes in the next layer. The higher layers produe higher-level features. Here, we develop our network using the Caffe [15] framework. The network onsists of three layers of onvolution, maxpooling, retified linear unit (ReLU), and loal normalisation, followed by a fully onneted layer and a linear lassifier at the top as shown in Fig. 2. Deeper networks may be used whih generally give better performane. The three-layer network is employed here beause of the limitations of our omputational system Feature extration We ompute the mean µ and the variane σ 2 of eah hannel of the output of eah layer. Following the parameters reommended in Krizhevsky s ConvNet struture [15], the input with size of (6 h f h f ) produes the outputs of layers 1 to 3 with the sizes of (32 h f 2 h f 2 ), (32 h f 4 h f 4 ) and (64 h f 8 h f 8 ), respetively. That means, F 1 = {µ 1 1, σ1, 1..., µ 1 32, σ32}, 1 F 2 = {µ 2 1, σ1, 2..., µ 2 32, σ32} 2 and F 3 = {µ 3 1, σ1, 3..., µ 3 64, σ64}. 3 The output vetors of the CNN are also employed, i.e. F 4 with the length of 10. Inluding F 0 desribed in Setion 3.1, the set of features used for fixation identifiation is shown in Eq. 2 a total of 274 dimensions, whih are subsequently employed in the final SVM lassifier. F = {F 0, F 1, F 2, F 3, F 4 } (2) 4. RESULTS AND DISCUSSION We first tested features F 0 using a high-sample-rate eye traker to examine whether these features are suitable to be used for fixation-point seletion. Subsequently, our framework was tested with a low-sample-rate mobile eye traker, using sequenes olleted from partiipants during loomotion on different terrain types. We employ a support vetor mahine (LIBSVM) [14] to perform linear lassifiation. The linear kernel is robust to overfitting and gives better speed than a non-linear kernel. We ompare our method to i) veloity-based threshold (I-VT) [10], ii) hidden Markov model identifiation [16], and iii) the EyeMMV [11] Hz EyeLink 1000 The features for fixation-point seletion were first tested with the down-sampled data of the 1000-Hz EyeLink eye traker [17]. The events marked by this devie were used as ground truth. 30-Hz samples were reated by subsampling the 1000Hz data using an average low-pass filter with the length of This repliates how ameras funtion by opening and losing the shutter for the duration of exposure. Four sequenes were aptured while 4 subjets were wathing a short film. One sequene was used for training, and the other three were used for testing. The average number of fixations and saades for one sequene are approximately 10,000 and 1,100 respetively, so 1,100 fixation points were randomly seleted to ensure that the number of both lasses were balaned. There were 40 different random fixation sets reating 40 tests in total and the results were averaged. The preision, reall and lassifiation auray of fixation identifiation are shown in Table 1. High preision means that fixations are rarely misidentified as saades, whilst high reall means all the true fixations are orretly identified. High auray implies that most fixations and saades are predited orretly. Table 1 shows that our simple feature set F 0 provides exellent performane both fast (less than

5 Fig. 3. Eye traking sequenes ontaining a variety of terrain types. The yles show eye positions whih may be fixations or saades or noises Table 1. Classifiation performane (%) for down-sampled data (30 Hz) from the 1000-Hz Eye link eye traker method I-VT I-HMM EyeMMV F0 preision reall auray Table 2. Classifiation performane (%) for 30-Hz SMI mobile eye traker method I-VT I-HMM EyeMMV F0 CNN (F 4 ) F 1 -F 4 F F w/o fix-sel preision reall auray µs/frame using Matlab on i7 CPU 2.8GHz) and highly aurate. Inorret preditions were observed to our at the beginning of the fixation, where our eyes are slightly shaky whilst fixating Hz SMI mobile eye traker The test sequenes were aquired with the SensoMotori Instruments (SMI) Eye Traking Glasses. These produe a point of view video at a resolution of pixels (W H) at 30 fps. The system provides a sene field of view of 60 horizontally and 46 vertially. Hene, the foveal and peripheral areas used in our method are approximately pixels (hf = 64) and pixels, respetively. We tested the proposed sheme using 12 sequenes of mobile eye traking data from 6 partiipants. Six sequenes ontaining 4 distint terrain types were used: flat onrete, slanted obbles, stepping stones and roks (shown in Fig. 3 olumns 1 4). The other 6 sequenes ontain mixed materials inluding dirt, roks, grass and wood on sloped terrain as shown in Fig. 3 olumns 5 and 6. The ground truths were manually marked. The average fixations and saades for one sequene are approximately 8,850 and 950, respetively. All sequenes are between 4-6 minutes in duration. A 2-fold ross validation was employed - 6 sequenes were used for training and the other 6 were used for evaluation. We ran 10 tests with randomly seleted training sequenes and the results were averaged. Table 2 shows the performanes of fixation identifiation. Our proposed method (F ) gives the best results with improvements in preision, reall and lassifiation auray of approximately 18 %, 10 % and 16 %, respetively. The performanes of the I-VT, IHMM and EyeMMV are dramatially lower than those of the stati eye traker (Setion 4.1). This learly shows that these methods are not suitable for mobile eye trakers. Predition on the mobile eye traker is signifiantly more diffiult than on the stati eye traker beause of outdoor onditions and trak and return behaviour [18], where our eyes fixate at a partiular loation traking it bak as walking forwards, then saading ahead again to fixate at the next loation. When the features F 1 -F 4 were inluded, the preision, reall and lassifiation auray were improved from using only F 0 by up to 14 %, 7 % and 13 %, respetively. This implies that the loal visual features play a very important role in attrating human attention during loomotion. The last row shows the results when the fixation-point seletion proess (i.e. sample point balaning in Setion 3.1) was not applied, i.e. the fixated points fed to the CNN were randomly seleted. This reveals that our fixation-point seletion method improves the lassifiation performane up to 2 %. 5. CONCLUSIONS AND FUTURE WORK This paper presents a new supervised learning method for fixation identifiation of the eye data. The proposed method employs temporal harateristis of the eye positions and loal visual features extrated by a deep onvolutional neural network (CNN), and then lassifies the eye events via a support vetor mahine (SVM). Our method offers higher lassifiation auray, preision and reall of fixations than the existing methods. The loal visual features learly enhane the performane of the fixation identifiation method for lowsample-rate mobile eye trakers. The fixation points predited by our method an then be used in human behaviour researh and to improve the performane of bio-inspired mahines. Temporal visual features and a deeper CNN will be employed in future work.

6 6. REFERENCES [1] K Rayner, Eye movements in reading and information proessing: 20 years of researh, Psyhologial Bulletin, vol. 124, no. 3, pp , [2] Eye Traking, Hardware: Eye traking systems, Teh. Rep., Traker-List, [3] SensoMotoni Instruments, Smi eye traking glasses, Teh. Rep., [4] B. Fisher and E. Ramsperger, Human express saades: extremely short reation times of goal direted eye movements, Experimental Brain Researh, vol. 57, pp , [5] H. Collewijn, C. J. Erkelens, and R. M. Steinman, Binoular o-ordination of human vertial saadi eye movements, Journal of Physiology, vol. 404, pp , [6] Marus Nyström and Kenneth Holmqvist, An adaptive algorithm for fixation, saade, and glissade detetion in eyetraking data, Behavior Researh Methods, vol. 42, no. 1, pp , [7] C. Berger, M. Winkels, A. Lishke, and J. Hoppner, Gazealyze: a matlab toolbox for the analysis of eye movement data, Behavior Researh Methods, vol. 44, no. 2, pp , [8] Dario D. Salvui and Joseph H. Goldberg, Identifying fixations and saades in eye-traking protools, in Proeedings of the 2000 Symposium on Eye Traking Researh & Appliations, New York, NY, USA, 2000, ETRA 00, pp , ACM. [9] O.V. Komogortsev, D.V. Gobert, S. Jayarathna, Do Hyong Koh, and S.M. Gowda, Standardization of automated analyses of oulomotor fixation and saadi behaviors, IEEE Transations on Biomedial Engineering, vol. 57, no. 11, pp , Nov [10] Anneli Olsen, The Tobii I-VT fixation filter: Algorithm desription, Teh. Rep., Tobii, [11] V. Krassanakis, V. Filippakopoulou, and B. Nakos, Eyemmv toolbox: An eye movement post-analysis tool based on a two-step spatial dispersion threshold for fixation identifiation, Journal of Eye Movement Researh, vol. 7, no. 1, pp. 1 10, [12] Enkelejda Tafaj, Thomas C. Kubler, Gjergji Kasnei, Wolfgang Rosenstiel, and Martin Bogdan, Online lassifiation of eye traking data for automated analysis of traffi hazard pereption, Artifiial Neural Networks and Mahine Learning, vol. 8131, pp , [13] Susan M. Munn, Leanne Stefano, and Jeff B. Pelz, Fixation-identifiation in dynami senes: Comparing an automated algorithm to manual oding, in Proeedings of the 5th Symposium on Applied Pereption in Graphis and Visualization, 2008, pp [14] C.C. Chang and C.J. Lin, LIBSVM: A library for support vetor mahines, ACM Transations on Intelligent Systems and Tehnology, vol. 2, pp. 27:1 27:27, [15] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshik, Sergio Guadarrama, and Trevor Darrell, Caffe: Convolutional arhiteture for fast feature embedding, arxiv preprint arxiv: , [16] O. V. Komogortsev and A. Karpov, Automated lassifiation and soring of smooth pursuit eye movements in presene of fixations and saades, Journal of Behavioral Researh Methods, pp. 1 13, [17] SR Researh, The eyelink, Teh. Rep., [18] B.M. Hart and Einhäuser, Mind the step: omplementary effets of an impliit task on eye and head movements in real-life gaze alloation, Experimental Brain Researh, vol. 223, pp , 2012.