Gaze tracking in multi-display environment

Transcription

1 HSI 2013 Sopot, Poland, June 06-08, 2013 Gaze tracking in multi-display environment Tomasz Kocejko and Jerzy Wtorek Department of Biomedical Engineering, Gdansk University of Technology, Narutowicza 11/12, Gdańsk, Poland, Abstract. This paper presents the basic ideas of eye and gaze tracking in multiple-display environment. The algorithm for display detection and identification is described as well as the rules for gaze interaction in multi display environment. The core of the method is to use special LED markers and eye and scene tracking glasses. Scene tracking camera registers markers position which is then represented as a cloud of points. Analyzing the mutual positions of detected points the algorithm estimates screen/displays position. Display number is assigned based on special information marker. Described project shows the possibility of hands free interaction in MDE. It also presents how to register visual attention when information is dispersed on several screens. Keywords: multi display environment, MDE, gaze communication, eye tracking, gaze interaction with multiple displays, visual attention registration. I I. INTRODUCTION N general, eye tracking systems can be categorized as a table and head mounted ones. They differ from each other by design, technology and implemented algorithms. The head mounted systems were designed to enable tracking a visual attention in so called a real environment and they are equipped with one [1], two (simultaneous tracking of the eye and a scene) [2]-[6], or even more than two cameras [7][8]. The multiple-camera head mounted systems enables performing tasks or experiments outside the lab. Similar to head mounted systems the table mounted ones can be equipped with one, two or more cameras. Despite a number of the cameras there are systems containing the pan/tilt units which allows the camera to be rotated [9] and/or the lens focus can automatically adjusted [10]. The eye trackers are widely used in many scientific disciplines including medicine e.g. neurology, psychology, cognitive science, and many others [16]-[18]. Apart from the above mentioned utilization the eye trackers are used in systems developed for data collection from a region of interest. They can also be used for communication of a user with computers [13]-[15]. It is achieved in many ways, e.g. by displaying on computer s monitor a keyboard and the user is selecting a desired letter by gazing at it. Rapid development of information technologies influences on improvement of diagnostic techniques. However, this improvement is associated with an increased amount of information and necessity of its This work has been partially supported by European Regional Development Fund concerning the project: UDA-POIG / Home assistance for elders and disabled -DOMESTIC, Innovative Economy , National Cohesion Strategy. transfer/presentation to a user. E.g., an amount and variety of information generated/transferred through Hospital Information Systems (HIS) causes that the workstations are equipped with multiple displays or even consist of multiple independent modules. It follows from a fact that it is easier to absorb a knowledge when a complex information is presented on several monitors/screens [11]. A cloud computing technology improves an information transfer between different systems and devices. It implies that increasing number of HIS providers offer their products for both desktops and tablets. Dedicated frameworks are developed to enable the user a better managing of information and its better distribution among PDAs, laptops, tablets and smartphones [12]. Thus, an objective evaluation of working effectiveness in such environment demands a special tools to be used. However, there are a very few technologies that allow for eye/gaze tracking in constantly changing multi display environment (MDE) Despite all the frameworks, the development of functional interface that enable controlling the multiple screens, devices or systems maybe beneficial for understanding the rules of interaction in MDE as well for getting more knowledge about process of obtaining information and/or communication. II. METHOD In general, idea of tracking the visual attention in multiple displays environment relies on marking the displays of interest with IR LED markers. The concept of gaze tracking presented in "Eye Mouse for disabled" [13] paper was used as the core of this project. Although the hardware for gaze and scene tracking remain the same the software architecture and algorithms were redesigned. The main improvement was adding a detection of display candidates from a points cloud and numerating them according to IR identifying markers (Fig. 2b). A. General Concept The project aimed at creating the user interface (UI) that allows acquiring a distribution of the visual attention from different, also mobile devices or, in general, from multi-display environment. The main task was to develop rules enabling data exchange between the eye tracking interface and devices forming a multiple display environment. To achieve this a client-server architecture was chosen. Fig. 1 presents the general concept of the project.

2 Exemplary markers configuration is presented in Fig. 2a, the identifying marker is presented in Fig. 2b: Fig. 1. The general concept of gaze interaction in multi display environment To enable tracking of the visual attention in multiple devices environment, every particular device had to be assigned an ID number and registered in the database software. The registration rule did not apply for multiple screen gaze tracking situation. In such a case there was only one device to exchange data with. Despite the number of devices all displays of interest had to be assigned with ID number and stored in the database. The operation of numerating and registering the devices allowed for their integration. From now on, integrated devices were able to provide an information about their configuration (screen size and resolution) to the server application when called. Equipped with eye and scene tracking camera, the eyemouse interface [13] was used for data registration. The scene camera constantly observed the environment registering the position of each display and correlating this information with a data provided by the eye tracking camera. According to the results of screen tracking and pupil center detection algorithms the approximate fixation point was calculated. When software classified that the gaze is within certain device/screen, the point of regard was calculated for the display of the interest. Selected device broadcasted their basic parameters and proper information was exchange with the device. In presented concept, the server application covered the data exchange and organization between tracked devices. Proper pupil detection was possible due to the algorithms applied in earlier projects [5][6][13]. However, device detection and differentiation relied on new algorithm detecting displays from the cloud of points created by IR LED markers. B. IR markers Each display's corners are marked with IR LED's. To distinguish particular devices/displays from each other special identifying markers were designed and mounted between the top corners of a display. This additional marker informed about the device/screen ID number in configured MDE (multi display environment). Two different way of numbering the display of interest were designed: traditional - number of lighten LEDs correspond to the display number binary - the number of display of interests is represented as binary code (lighten LEDs represents a number in binary code) Fig. 2. a) Exemplary configuration of LED markers b) the model of identifying LED marker Identifying markers were designed to be powered from voltage battery and contained switches enabling presentation the ID number by switching the LEDs ON/OFF. However, markers placed at the display corners were designed to be powered either from battery or USB source. C. Display detection and identification Displays detection was based on fitting of the quadrangle shapes to the given cloud of points. The scene camera was equipped with IR filters mainly enabling markers registration. Attached to the displays the LED markers created a cloud of bright points on the captured images. Position of each captured marker was estimated with contour detection algorithm. Every marker was represented by its contour properties (size, position of a center etc. ). Markers position were converted into the cloud of points and stored in a matrix. The next step was to apply a mesh to connect points with each other. Every two points joint with a line created a section. Sections with common point were paired regarding the ratio of their lengths and angle between them. The pairs with opposite vertices that shared two points were merged to a quadrangles. The number of created quadrangles was then reduced so every four points belonged only to one quadrangle. The detected quadrangles were identified and numbered according to the information obtained from indentifying markers. It was assumed that this markers were always placed at the center of upper edge of each display. Depends of the configuration, the algorithm checked either the binary code created by lighten LEDs or simply LED's status (on/off). The block diagram of the algorithm is presented in Fig. 3. This algorithm (display detection and identification algorithm) allowed for estimating the actual position of each marked screen or device.

3 corresponding pupil positions. For calculating the transform matrix Ω 1 from M1 and M2, the perspective transform was used. From now on, two most important parameters could be calculated: virtual point of regards and true point of regards The virtual point of regards (VPoR) referred to the position of pupil center correlated with images and information captured by the scene camera. It also could be represented in the same matrix space as a captured image of a scene. The accuracy of the VPoR estimation was related to the precision of the calibration performance and thereby to the transformation matrix: T VPoR = Ω 1 P (1) where: Ω 1 - is transformation matrix computed from pupil positions stored in matrix M1 related with calibration points stored in M2 P - is a vector containing absolute pupil center position registered by eye camera VPoR - Virtual Point of Regards vector containing the fixation position correlated and represented in the same space as images captured by scene camera. Fig. 3. Block diagram of multiple display tracking algorithm D. Gaze estimation The concept of gaze estimation used for eyemouse interface was partially implemented for gaze tracking in multiple display environment. The basic idea of estimating the gaze position is presented in Fig. 4. Beside the pupil center detection and establishing of the primary screen (the ID number of primary display was setup before the calibration procedure) the calibration procedure was first important step in gaze estimation. Regarding the fact that the calibration had to be done in regards to the constant set of points, only one display (the primary one) was used during this procedure. The calibration led to the transform matrix computation - Ω 1. During calibration user watched at the points of constant position in the scene while the software registered his/her pupil position. The calibration points were stored in matrix "M2" while matrix "M1" contained Knowing the VPoR it was possible to validate if it was within the area of particular labeled display. Along the VPoR, detected displays position were also stored in the matrix containing information about the image captured by the scene camera. To establish the "true" point of regard TPoR (actual position of fixation within particular display) the perspective transform was used one more time to calculate another transform matrix - Ω 2. However, in this case the transform matrix was calculated dynamically. Unlike the first transform matrix - Ω 1, which was calculated only once during the calibration procedure and for static data sets, the second one - Ω 2 was calculated every time the software detected displays in the image captured by the scene camera. Using perspective transformation, the Ω 2 matrix was computed with regards to matrix "Mi" (representing current display's of interest position) and corresponding matrix "MRi" containing markers position represented in particular display pixel space (with regards to its resolution). With the transform matrix Ω 2 and the VPoR the fixation on actual object of interest was calculated according to the formula: TPoR = Ω T 2 VPoR (2) where: Ω 2 - is transformation matrix computed from virtual positions of IR LEDs markers dynamically captured by scene camera stored in matrix Mi and their position represented in pixel space - MRi VPoR - Virtual Point of Regards vector containing the fixation position correlated and represented in the same space as images captured by scene camera.

4 TPoR - the fixation on real object of interest represented in particular displays pixel space Rendering the virtual representation of potential multi display environment allowed to assume the very wide angle of scene tracking camera and check different orientation of displays on a large scene. The results for different configuration of three screens in the scene is presented in Fig. 6: Fig. 4. Block diagram of gaze interaction in multi display environment The position of TPoR registered for particular device was transferred to the server and jointed with other information passed by this device as its number, screen-shot and/or performed action (e.g. a mouse click) III. RESULTS To check how the display detection and identification algorithm works the set of exemplary images of potential scene was generated according to the render 3D images. The set of images containing up to three displays setup in different positions in space was prepared. The exemplary pictures of render image and the result of the display detection algorithm are presented in Fig. 5. (c) Fig. 6. Different configuration of identified displays The situation for the detection result in case two displays overlying each other is presented in Fig. 7. Fig. 7. Display detecting algorithm applied on overlying screen The rotated (and different size) displays detection is in Fig. 8. Fig. 5. a) Rendered image of exemplary MDE b) result of display detection All detected displays were numerated according to the information obtained from the identifying markers. To check every aspect of the algorithm (including upper edge detection and vertex orientation detection) the important features were marked on the image (edge with green and bottom right corner of the screen with yellow color). Fig. 8. Result of rotated display detection However, the correct work of the algorithm was possible when the LED markers were attached properly. It means that the each angle of possible quadrangle must be within

5 certain range (in this experiment the angle range was set to degrees). Fig. 9 presents results for a wrong marker placement. Fig. 9. Wrong LEDs configuration Moreover, the algorithm was tested in real multi display environment. Two devices were setup: a laptop (as primary device - number 1) and a tablet (additional device - number 3). Possible situation of tracking visual attention from two different devices is presented in Fig. 10 (Fig. 10a - image captured by scene camera, Fig. 10b - result of display detection algorithm) Fig. 10. The example of a) MDE configuration and b) result of display detection and identification Presented example shows the situation where user moves his visual attention between configured devices. Another example (Fig. 11) presents results of implemented algorithm for user focusing his attention on only one of the devices. Fig. 11. The example of a) MDE configuration and b) result of display detection and identification IV. DISCUSSION We elaborated a method of display labeling based on IR LEDs markers which creates the cloud of bright points when captured by the IR sensitive scene camera. In our approach we assumed that displays have quadrangle shape. Then we applied the distance and angular dependencies to estimate their position on a detected cloud of points. On this basis we developed an algorithm for detecting and identifying labeled displays. Then we implemented this algorithm in previously created eye tracking software and we establish a method for tracking the visual attention in multi-display environment. We conducted a validation study which indicates that the algorithm performs well on images captured from scene camera of head-mounted eye tracking interface. The algorithm seems to work correctly even for large cloud of points. The experiment was conducted in natural light conditions (in the regular office) including the illumination of tracked displays (Fig. 11a). The problem of disturbing reflections from behind of operator was partially limited due to the implementation of IR light pass filters on both of the tracking cameras. The rules for differentiation between IR markers and other light sources also has been implemented in the algorithm. However, the misdetection episodes would rarely happened. Nevertheless, this problem has been considered very seriously and attempt to solve it both by hardware modifications and software improvements has been made. According the rendered objects, all of them were detected and identified correctly even when the display's position was changed (Fig. 6). Presented concept of gaze tracking in multi-display environment might improve tracking visual attention in multiple devices. The slight disadvantage of proposed solution is necessity of wearing the eye tracking glasses and pre-configuration of the devices belonging to monitored multi-display environment (MDE). However, the great advantage is that only one eye tracking interface is needed. After calibration user can freely moves his attention from one device to another as well as interact with different devices without the need of recalibration. Although the algorithm can detect and identify quit a number of devices, its efficiency is decreasing with every additional set of markers. When the cloud of the points grow, the number of section increases which influence on the speed of data processing. It was noticed that the algorithm may even collapse for large number of markers. Although the view angle of scene tracking camera enables for continuous tracking up to three different displays that are not overlying each other (for example laptop, tablet and Smartphone), this part still must be improved. The important fact is that the algorithm correctly identifies the

6 current display of interest (Fig. 11a). Moreover, the algorithm works well even for rotated displays (but only to a limited value of angle rotation). However, certain assumption was made according the angles value of detected quadrangles. This value depends on correct IR markers placement as well as on perspective projection of captured image on the camera plane. Therefore we are planning to investigate the range of head movement rotation during work within MDE. Depending on the results the problem will be solved mathematically or simply ignored (if the head rotation will not influence on the results of the display tracking algorithm). Still, more studies have to be made in this case to accurately verify the possible angle of display rotation. It happened that the empty space between marked displays was qualified as a quadrangle however algorithm correctly identified displays due to the specially designed markers. Described concept of gaze tracking in multi-display environment can also be applied for hands free communication with multiple devices clustered in one group of interest. We have not measured the accuracy and spatial resolution of the interface in MDE but this parameters were measured for the eye tracking device we designed (eyemouse). The tests were conducted for 14" display and the result was 0,7 of accuracy and 0,4 of spatial resolution. Still, more tests will be conducted in the MDE. Gaze communication would require installing additional applications at this devices to enable receiving and translating the commands given be means of eye tracking software. REFERENCES [1] Fejtova, M.; Fejt, J.; Novak, P. & Stepankova, O. (2006), System I4Control: Contactless control PC, in 'Intelligent Engineering Systems, INES '06. Proceedings. International Conference on', pp [2] cts/iview-x-hed.html [3] ware/tobii-glasses-eye-tracker/technical-specifications/ [4] nts/smi_etg_flyer.pdf [5] Kocejko T., Bujnowski A., Wtorek J, Complex human computer interface for LAS patient HSI'09, Proc. 2nd International Conference on Human System Interaction, Catania, Italy, May , pp [6] Kocejko T., Bujnowski A., Wtorek., Dual camera based eye gaze tracking system, 4th European Conference of the International Federation for Medical an Biological Engineering; IFMBE Proceedings, Antwerp, Belgium November, Berlin: Springer, (IFMBE Proceedings, vol pp [7] Nishant Kumar, Stefan Kohlbecher, E. S. (2009), 'A Novel Approach To Video-Based Pupil Tracking', Proceedings of the 2009 IEEE International Conference on Systems, Man, and Cybernetics, [8] Vockeroth, J.; Dera, T.; Boening, G.; Bartl, K.; Bardins, S. & Schneider, E. (2007), The combination of a mobile gaze-driven and a head-mounted camera in a Hybrid perspective setup, in 'Systems, Man and Cybernetics, ISIC. IEEE International Conference on', pp [9] Liu, R.; Zhou, X.; Wang, N. & Zhang, M. (2009), Adaptive Regulation of CCD Camera in Eye Gaze Tracking System, in 'Image and Signal Processing, CISP '09. 2nd International Congress on', pp [10] Poitschke, T.; Bay, E.; Laquai, F.; Rigoll, G.; Bardins, S.; Bartl, K.; Vockeroth, J. & Schneider, E. (2009), Using liquid lenses to extend the operating range of a remote gaze tracking system, in 'Systems, Man and Cybernetics, SMC IEEE International Conference on', pp [11] Poder, T.; Godbout, S. & Bellemare, C. (2011), 'Dual vs. single computer monitor in a Canadian hospital Archiving Department: a study of efficiency and satisfaction.', HIM J 40(3), [12] Biehl, J; Baker, W; Bailey, B;, Tan, D; Inkpen, K; Czerwinski, M IMPROMPTU: A New Interaction Framework for Supporting Collaboration in Multiple Display Environments and Its Field Evaluation for Co-located Software Development, CHI 2008, April 5 10, 2008, Florence, Italy [13] Kocejko, T.; Bujnowski, A. & Wtorek, J. (2008), Eye mouse for disabled, in 'Human System Interactions, 2008 Conference on', pp [14] Magee, J.; Scott, M.; Waber, B. & Betke, M. (2004), EyeKeys: A Real-Time Vision Interface Based on Gaze Detection from a Low-Grade Video Camera, in 'Computer Vision and Pattern Recognition Workshop, CVPRW '04. Conference on', pp [15] Li, D.; Winfield, D. & Parkhurst, D. (2005), Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches, in 'Computer Vision and Pattern Recognition - Workshops, CVPR Workshops. IEEE Computer Society Conference on', pp. 79. [16] Pfeiffer UJ, Schilbach L, Jording M, Timmermans B, Bente G, Vogeley K. Eyes on the mind: investigating the influence of gaze dynamics on the perception of others in real-time social interaction. Front Psychol. 2012;3:537. doi /fpsyg Epub 2012 Dec 3. [17] Balslev T, Jarodzka H, Holmqvist K, de Grave W, Muijtjens AM, Eika B, van Merriënboer J, Scherpbier AJ. Visual expertise in paediatric neurology. Eur J Paediatr Neurol Mar;16(2): doi: /j.ejpn Epub 2011 Sep 8. [18] Tylén K, Allen M, Hunter BK, Roepstorff A. Interaction vs. observation: distinctive modes of social cognition in human brain and behavior? A combined fmri and eye-tracking study. Front Hum Neurosci. 2012;6:331. doi: /fnhum Epub 2012 Dec 19.