A Method to Measure Eye-Hand Coordination for Extracting Skilled Elements-Simultaneous Measurement of Eye-Gaze and Hand Location-

Transcription

1 Computer Technolog and Application 5 (214) D DAVID PUBLISHING A Method to Measure Ee-Hand Coordination for Extracting Skilled Elements-Simultaneous Measurement of Ee-Gaze and Hand Location- Atsuo Murata and Kosuke Inoue Department of Intelligent Mechanical Sstems, Division of Industrial Innovation Sciences, Graduate School of Natural Science and Technolog, Okaama Universit, Okaama 7-853, Japan Abstract: The purpose of this stud was to construct bases for exploring the process of skill acquisition from the viewpoint of ee-hand coordination. The information obtained from ee-gaze is closel related to phsical movements in an activit. It is important to establish a method to measure ee-hand coordination for extracting skilled elements and understanding the skill acquisition process. Using a sstem which consists of an ee mark recorder and a three-dimensional location measurement device, a method for measuring ee-hand coordination was proposed on the basis of the simultaneous measurement of ee-gaze and brush tip locations. After describing the measurement algorithm, the ee-hand coordination during calligraph was exemplified. More concretel, using such a sstem, an attempt was made to show that the relationship between the line of ee-gaze and the brush tip stroke was different between a novice and an expert. In such a wa, we suggested that the proposed method is promising for exploring the process of skill acquisition. Ke words: Skill, ee-hand coordination, ee-gaze, brush tip location, algorithm for measuring ee-hand coordination. 1. Introduction An analsis of skill acquisition process is ver important in the field of cognitive and movement sciences [1-5]. In a variet of tasks, the information obtained b visual information processing is essential for the determination and the execution of phsical movements. It is assumed that the relationship between the ee movement and the phsical movement differs according to the skill level. In order to learn and master the traditional skills efficientl, the research focusing on the cognitive processes and motor process is necessar and indispensable. Therefore, the exploration of the difference of ee-hand coordination between skilled and non-skilled participants would contribute to the realization of effective tradition method of skill. Corresponding author: Atsuo Murata, Ph.D., professor, research fields: ergonomics, cognitive science. murata@iims.ss.okaama-u.ac.jp. Vickers [1] clarified different features of ee-hand coordination between skilled and non-skilled free-throw shooters. Murata et al. [2-4] suggested that the abilit of ee-hand coordination and the abilit of recognizing and predicting the work area are important elements in skilled tasks. Sano and Ukida [5], appling image processing techniques to the evaluation of characters in calligraph, showed that the writing of characteristic components of calligraph differed between skilled and non-skilled participants. In Ref. [2-4], the simultaneous measurement technique of ee-gaze and hand location is halfwa, and is not sstematicall described so that a similar measurement sstem of ee-hand coordination can be made easil and with high measurement accurac. Therefore, a sstem that can simultaneousl measure the ee-gaze and the brush tip locations has been developed, and its algorithm has been sstematicall described. The purpose of this stud

2 74 A Method to Measure Ee-Hand Coordination for Extracting Skilled Elements-Simultaneous was to construct bases for clarifing the process of skill acquisition from the viewpoint of ee-hand coordination. The paper consists of the following steps. First (in Section 2.1), we explained apparatus for measuring the ee-hand coordination, that is, an ee mark recorder and a three-dimensional magnetic movement tracking sstem. Second (Sections ), a method for measuring ee-hand coordination was proposed and full described. This method consisted of the following three processes: (1) calculation of pen tip location (Section 2.2), (2) calculation of ee-gaze location after correction of binocular parallax (Section 2.3), and (3) transformation of ee-gaze location to the same coordination sstem with pen tip location (Section 2.4). Third (in Section 3), calligraph was selected as a demonstrative application of simultaneous measurements of ee-gaze and brush tip location. 2. Method for Measuring Ee-Hand Coordination which are made of wood. In Fig. 3a, Receiver 1 of FASTRAK is attached to a brush, and in Fig. 3b, Receiver 2 of FASTRAK is attached to EMR-9. Fig. 4 corresponds to the camera screen coordination sstem viewed from a view camera. The ee-gaze points after the correction of view difference between right and left ees were used as a representative of ee location. The overview of measurement sstem and coordination sstems (transmitter coordination sstem, Receiver 1 coordination sstem, Receiver 2 coordination sstem, view camera coordination sstem, and camera screen coordination sstem) is shown in Fig. 5. In the ee-hand coordination measurement algorithm, these coordination sstems must be integrated to a common coordination sstem. The algorithm is mentioned below. View camera 2.1 Apparatus The sstem consisted of a wearable tpe ee mark recorder (Nac Image Technolog, EMR-9) (Fig. 1) and a three-dimensional magnetic motion tracker (Polhemus, FASTRAK) (Fig. 2). Both ee mark location and brush tip location were sampled with a sampling frequenc of 6 Hz. The ee mark recorder EMR-9 consists of a head unit equipped with a view camera and a camera for measuring line of gaze, and is not influenced b head swa. FASTRAK can obtain relative location between the transmitter and Receivers 1 or 2 with an accurac of.8 mm, if Receiver 1 or 2 is located within 76 mm from the transmitter. This sstem can measure the coordinates,, and Z and aw φ (rotation angle along Z-axis), pitch φ p (rotation angle along -axis), and roll φ r (rotation angle along -axis) on the transmitter coordination sstem. In order to restrain the effects of magnetic field on the measurement as much as possible, we used a desk and a chair for calligraph Camera for measuring ee movement Fig. 1 View camera and camera for measuring ee movement (EMR-9). Transmitter Receiver Fig. 2 FASTRAK, transmitter, and receiver for measuring three-dimensional locations.

3 A Method to Measure Ee-Hand Coordination for Extracting Skilled Elements-Simultaneous 75 Receiver1 (FASTRAK) View camera Receiver2 (FASTRAK) (a) (b) Fig. 3 (a) Attachment of Receiver 1 of FASTRAK to a brush; and (b) attachment of Receiver 2 of FASTRAK to EMR-9. Left ee mark Right ee mark Ee-gaze point after correction of view difference between right and left ees Fig. 4 Camera screen coordination sstem viewed from a vie camera. Z EMR-9 Z Z Receiver1 coordination sstem (FASTRAK) Receiver 2 coordination sstem (FASTRAK) Z view camera coordination sstem Marker coordination sstem Transmitter coordination sstem (FASTRAK) Z camera screen coordination sstem Fig. 5 Overview of measurement sstem and coordination sstem.

4 76 A Method to Measure Ee-Hand Coordination for Extracting Skilled Elements-Simultaneous 2.2 Calculation Method of Pen Tip Location The Receiver 1 of FASTRAK is attached so that the brush tip faces toward -Z. The brush location vector V p is represented b Eq. (1), where m is the length of brush. V p (1) m Next, the Receiver 1 coordination sstem is transformed to the transmitter coordination sstem using the location and posture data (,, Z, φ, φ p, φ r ) from FASTRAK output. The transformation matrix t T r is given b t T r = R z R R x P (2) where the matrices R x, R, R z are the rotation matrices around x-, -, and z-axis, respectivel. The matrix P is the translation matrix. The matrices R x, R, R z, and P are given b 1 cos sin r r R x (3) sin cos r r 1 cos p sin p 1 R (4) sin cos p p 1 cos sin sin cos R (5) z P (6) Z 1 1 Thus, the brush tip vector V pt using the transmitter coordination sstem can be calculated using Eq. (7). V pt = t T r V p (7) The value of m was set to 28 mm. As the soft brush was used in the experiment, the flexure of the brush must be taken into account. Therefore, using the coordinates of the tail end of the brush and the brush tip, the brush location is described using a three-dimensional line. The working plane of the world coordination sstem is nearl the same with that of the transmitter coordination sstem with = -27 mm. The coordinate of intersection of the two planes was calculated to derive the actual contact location of the brush tip b taking the flexure of the brush into account. Thus, setting the coordinate of end of brush, the coordinate of brush tip, and the coordinate of the actual contact location of the brush to ( 1, 1, Z 1 ),( 2, 2, Z 2 ),and ( w, w, Z w ), respectivel, Eq.(8) is obtained. x x1 1 z z1 (8) x2 x1 2 1 z2 z1 Putting = ( 1 )/( 2 1 ) and setting = -27, 1 = 1, and 2 = 2, Eq. (9) is obtained and can be expressed using known values (9) 2 1 From Eqs. (8) and (9), the following equations are obtained. w ( x2 x1 ) x1 (1) Z w ( z2 z1) z1 (11) Thus, the actual contact location ( w, w, Z w ) in the world coordination sstem is given b w ( x2x1 ) 1 27 w (12) Z ( z2 z1) w 2.3 Calculation Method of Ee-Gaze Location after Correction of Binocular Parallax It is described how the ee-gaze point (x, ) after the correction of binocular parallax is calculated using both right ((x r, r )) and left ((x l, l )) ee marks output from EMR-9. As shown in Fig. 6, the right or the left ee mark corresponds to the intersection of the ee-gaze vector that connects the ee-gaze point and the right or left ee and the calibration plane. The

5 A Method to Measure Ee-Hand Coordination for Extracting Skilled Elements-Simultaneous 77 camera screen coordination sstem is shown in Fig. 6. The ee-gaze point (x, ) after the correction of binocular parallax can be calculated b deriving the intersection of the ee-gaze vector that connects the ee-gaze point and the view camera and the calibration plane. As shown in Fig. 6, the ee-gaze points (x, ) after the correction of binocular parallax appears on the perpendicular bisector of the line connecting right and left ee marks. It must be noted that the right (x r, r ) and left (x l, l ) ee marks corresponds with the ee-gaze point (x, ) after the correction of binocular parallax on the camera screen coordination sstem when the ee-gaze point is on the calibration plane. As shown in Figs. 7 and 8, there are two cases, that is, when the gaze-point is beond the camera screen coordination sstem calibration plane, and when the gaze-point is before the calibration plane. First, the calculation method of the ee-gaze point (x, ) after the correction of binocular parallax is described when the gaze-point is beond the calibration plane. The upper view of Fig. 7 (when Fig. 7 is viewed from the upper and perpendicular direction) is shown in Fig. 9. The side view of Fig. 7 (when Fig. 7 is viewed from the horizontal direction) is shown in Fig. 1. From Fig. 9, the binocular parallax Δv is given b v : w L l : L (13) Thus, L is given b Eq. (14). wt L (14) w v Ee-gaze point after correction of view difference between right and left ees (x, ) Left ee mark (x l, l ) Right ee mark (x r, r ) Fig. 6 (64, 48) Relationship between ee-gaze point and right or left ee mark in camera screen coordination sstem. Calibration plane Location of view camera d Height of view camera h Ee-gaze point Pupil distance w Calibration distance l Fig. 7 Distance between ee and ee-gaze point L Geometric relationship when the gaze-point is beond the calibration plane.

6 78 A Method to Measure Ee-Hand Coordination for Extracting Skilled Elements-Simultaneous Calibration plane Location of view camera d Height of view camera h Ee-gaze point Pupil distance w Calibration distance l Distance between ee and ee-gaze point L Fig. 8 Geometric relationship when the gaze-point is before the calibration plane. x Ee-gaze point Binocular parallax Δv Pupil distance w Calibration distance l Distance between ee and ee-gaze point L Fig. 9 Geometr when Fig. 7 is viewed from the upper and perpendicular direction. Calibration plane Location of view camera d Ee-gaze point r Height of view camera h Calibration distance l Fig. 1 Distance between ee and ee-gaze point L Geometr when Fig. 7 is viewed from the horizontal direction.

7 A Method to Measure Ee-Hand Coordination for Extracting Skilled Elements-Simultaneous 79 The binocular parallax Δv can be represented b 2 2 v x (15) From Fig. 1, Eq. (16) is obtained. r : h L l : L d (16) Solving Eq. (16) with respect to r, Eq. (17) is obtained. L l r h (17) L d Using Eq. (15) and Eq. (17), Eq. (18) is obtained. hlv r (18) w( l d) dv Setting the coordinate of the middle point between the right and the left ee mark (p, q), the ee-gaze point (x, ) after correction of binocular parallax is given b Eqs. (19) and (2). x p r sin (19) q r cos (2) 2.4 Calculation Method for Ee-Hand Coordination Sstem-Transformation of Ee-Gaze Location after Correcting Binocular Parallax to Transmitter Coordination Sstem of Brush Tip Location- The method to transform the ee-gaze point after correcting the binocular parallax (x cs, cs, z cs ) to the transmitter coordination sstem (x t, t, z t ) b which the brush tip location is expressed. The ee-gaze point in the camera screen coordination sstem is transformed to the ee-gaze point according to the following procedure. Using the inner-camera parameter cs T c and outer-camera parameter c T m, the transformation matrix from the marker coordination sstem to the camera screen coordination sstem cs T m is calculated. cs T m = cs T c c T m (21) where cs T m is a 3 4 matrix. When transforming the marker coordination sstem to the camera screen coordination sstem, the information on Z-axis is missing. Therefore, the 3-rd row of cs T m was deleted. Calculation of the inverse matrix cs T -1 m enabled us to obtain the ee-gaze point on the x- plane in the marker coordination sstem as follows. x z cs T 1 m x 1 cs cs (22) Normalizing the values of the marker coordination sstem obtained b Eq. (22) and adding z m = to the vector given b Eq. (22) leads to Eq. (23). xm x / s 1 / m cs s T m (23) z m 1 1 The ee-gaze point in the marker coordination sstem given b Eq. (23) is transformed into the transmitter coordination sstem using the transformation matrix t T m given b Eq. (24). The transformation matrix t T m is represented as follows. As the center of AR marker is set to the coordinate (2, -27, 15) of the transmitter coordination sstem, the transformation matrix t T m can be defined as the following matrix that moved 2 mm along x-axis, -27 mm along -axis, and 15 mm along z-axis, and rotated 9 degrees around x-axis. 1 2 cos9 sin 9 27 t T (24) m sin 9 cos The transmitter coordination sstem is transformed into the Receiver 2 coordination sstem attached to EMR-9 in Fig. 5. The transformation matrix r2 T t can be calculated as the inverse matrix that transforms the transmitter coordination sstem to the receiver coordination sstem. On the basis of the transformation above, the ee-gaze point (x cs, cs ) in the camera screen coordination sstem can be transformed into the ee-gaze point (x r2, r2, z r2 ) in the Receiver 2 coordination sstem. Thus, the ee-gaze point in the Receiver 2 coordination sstem (x r2, r2, z r2 ) can be represented b

8 8 A Method to Measure Ee-Hand Coordination for Extracting Skilled Elements-Simultaneous xr 2 xm r 2 r 2 t m T t Tm (25) z r 2 zm 1 1 As both view camera and Receiver 2 are attached (fixed) to EMR-9, the location and the posture between these two sstems does not change even if the head location of the participant changes. Therefore, the transformation matrix that transforms the ee-gaze point in the camera screen coordination sstem into the ee-gaze point in the Receiver 2 coordination sstem is unchanged. The location of the Receiver 2 changes according to the head movement of the participant. The change of the ee-gaze point (x r2, r2, z r2 ) due to head movement is transformed into the ee-gaze point in the transmitter coordination sstem. Setting the transformation matrix to t T r2, the ee-gaze point (x t, t, z t ) in the transmitter coordination sstem is given b xt xr 2 t t r 2 T r 2 (26) z t zr B moving (x t, t, z t ) along -axis b -27 mm, (x t, t, z t ) corresponds with the world coordination sstem of the location of the brush tip. 3. Example of Application of the Measurement Sstem to Skill Evaluation in Calligraph The measurement was carried out during the calligraphic task for both novice and expert. The developed real-time ee-hand coordination sstem was used to measure ee movements and brush tip location during the calligraphic task. Examples of the positions of each brush tip stroke and ee-gaze are plotted shown in Fig. 11 (skilled participant) and Fig. 12 (non-skilled participant). The (a) 永 (Forever) (b) 大 (Big) Fig. 11 (c) 明 (Bright) Trajector of brush tip and ee-gaze point (skilled participant). (d) 上下 (ups and downs)

9 A Method to Measure Ee-Hand Coordination for Extracting Skilled Elements-Simultaneous (a) 永 (Forever) 5 (b) 大 (Big) Fig. 12 (c) 明 (Bright) Trajector of brush tip and ee-gaze point (non-skilled participant). (d) 上下 (ups and downs) 5 Outline Outline Ee-gaze 25 3 Ee-gaze Fig. 13 Comparison of brush tip and ee-gaze locations between skilled and non-skilled participant (5-th stroke of 永 (forever)). Skilled participant Non-skilled participant brush tip and ee-gaze locations when writing the 5-th stroke of Kanji character forever are compared between a skilled and a non-skilled participant in Fig. 13. As for the skilled participant, the brush tip location is distant from the ee-gaze location. As far as the novice is concerned, the brush tip exists at nearl the same location with the ee-gaze location. On the basis of the finding that the ee movement is preceding the brush tip for the skilled participant, it is possible that such a propert might be one of the important skills necessar for acquiring skills in calligraph. In such a wa, the proposed method for real-time measurement sstem of ee-hand coordination is promising for clarifing the process of skill acquisition in skilled movement such as calligraph. In future research, such a skilled element pointed out

10 82 A Method to Measure Ee-Hand Coordination for Extracting Skilled Elements-Simultaneous in the demonstrative experiment must be verified b a sstematic experiment using the developed real-time measurement sstem of ee-hand coordination. 4. Conclusions The aim of this stuff was to propose a method for measuring ee-hand coordination which consisted of (1) calculation of pen tip location, (2) calculation of ee-gaze location after correction of binocular parallax, and (3) transformation of ee-gaze location to the same coordination sstem with pen tip location so as to enable us to view simultaneousl the ee-gaze and the brush tip locations. Calligraph was selected as an application of the proposed measurement sstem of ee-hand coordination. As far as our experiment is concerned, it has been suggested that experts might be different from novices in the ee movement preceding brush tip, and such a propert might be an important skilled element for acquiring a higher skill. References [1] Vickers, J Gaze Control in Basketball Foul Shooting. In Ee Movement Research: Mechanisms, Processes, and Applications, edited b Findla, J., Walker, R., and Kentridge, R. Amesterdam: Elsevier, [2] Murata, A., and Moriwaka, M. 29. Skill of Ee-Hand Coordination in Calligraph-Difference of Skill of Ee-Hand Coordination between Expert and Novice-. In Proceedings of 5th International Workshop on Computational Intelligence & Applications, [3] Murata, A., Inoue, K., and Moriwaka, M Real-Time Measurement Sstem of Ee-Hand Coordination in Calligraph. In Proceedings of SICE211, [4] Murata, A., Ioue, K., Haami, T., and Moriwaka, M Real-Time Measuring Sstem of Ee-Gaze Location and Writing Pressure in Calligraph. In Proceedings of AHFE212, [5] Sano, T., and Ukida, H. 21. Measurement of Handwriting Skills for Japanese Calligraph. In Proceedings of Instrumentation and Measurement Technolog Conference,