Investigating Multi-touch Gestures as a Novel Biometric Modality

Transcription

1 Investigating Multi-touch Gestures as a Novel Biometric Modality Napa Sae-Bae, Nasir Memon, and Katherine Isbister Computer Science Department, NYU-Poly Six Metrotech Center, Brooklyn, NY, nsae-b01@students.poly.edu, isbister@poly.edu, memon@poly.edu Abstract We propose a new behavioral biometric modality based on multi-touch gestures. We define a canonical set of multitouch gestures based on the movement characteristics of the palm and fingertips being used to perform the gesture. We developed an algorithm to generate and verify multi-touch gesture templates. We tested our techniques on a set of 22 different gestures. Employing a matching algorithm for a multi-touch verification system with a k-nn classifier we achieved 1.28% Equal Error Rate (EER). With score-based classifiers where only the first five samples of a genuine subject were considered as templates, we achieved 4.46 % EER. Further, with the combination of three commonly used gestures: pinch, zoom, and rotate, using all five fingers, 1.58% EER was achieved using a score-based classifier. These results are encouraging and point to the possibility of touch based biometric systems in real world applications like user verification and active authentication. I. INTRODUCTION Since their introduction, multi-touch phones and tablets have grown exponentially. More then 60 million mobile devices with multi-touch capabilities are expected to ship in Multi-touch interfaces bring new capabilities to existing devices and enable their use in completely new contexts. Different low cost technologies allow them to be used as part of familiar surfaces such as fabric, paper, glass and blend in our daily lives. Soon, these prototypes currently viewed as novelties, will join mainstream multi-touch devices such as smartphones and tablets already in the market, as coffee tables in our living rooms, and desks in our offices. The attractive user interaction features and the tactile/visual experience provided by multi-touch interfaces, make them strong contenders for becoming the dominant human computer interface, possibly replacing the keyboard, mouse and stylus. Typically, there has been a correspondence between a user interface and biometric modalities, as the sensing mechanisms that underlie the interface can also be used to capture user characteristics. Past research, for example, has used keystroke dynamics and mouse movements as candidate biometric modalities. In this paper, we investigate the potential of multitouch gestures as a biometric modality. Prior research has shown that biometric data can be gleaned from physiological characteristics of hand geometry [1]. Research has also been done to show that behavioral biometric traits can be captured using hand drawn signatures [2], [3] using a pen or stylus. Our study in this paper shows that multi-touch gestures contain sufficient biometric information, resulting from variation in hand geometry and muscle behavior, to allow discrimination between users. Even though this paper studies multi-touch gestures on a personal tablet device, the techniques presented could be ported to multi-touch tabletops, wall displays, etc., where multiple users may share space in a dynamic context. The rest of the paper is organized as follows. First, related work is summarized in section 2. In section 3, we present a taxonomy of multi-touch gestures and identify a set of 22 gestures that we consider in this study. In section 4, we describe details of a classification algorithm for multi-touch gestures. In section 5 we present experimental results. In section 6, we conclude our study and discuss future work. II. RELATED WORK Biometric systems are an effective way to identify or authenticate users based on the something they are property. Based on the practicality and availability of input sensors, a large number of biometric systems have been proposed. The characteristics used in a biometric system are classified into two types: physiological and behavioral. Some examples of physiological traits include retina[4], iris[5], fingerprint[6], face[7], vein[8] and palm[9]. Some behavioral biometric traits include dynamic signature[2], [10], voice[11], and gait[12]. Many such traits have been well studied, and can potentially be employed for authentication once there are sensors available in end user devices. Some techniques, such as iris scans, require specialized sensors. For computing devices, which generally come equipped with a keyboard and mouse, a lot of research on keystroke and mouse dynamics has been performed [13], [14], [15]. One desirable characteristic of such biometric systems is they simultaneously capture physical as well as behavioral traits. Recently, Hamdy, et al[16], have proposed a novel mouse-based biometric system using a virtual keyboard. Their approach differs from previous work since their system homogeneously captures physiological and behavioral biometrics via mouse movement where the features are derived from physiological properties such as visual scan and detection, and short-term memory as well as a type of behavioral property called dynamic mouse movement. Motivated by a recent paradigm shift of user interface from the keyboard and mouse to a touch interface, we propose a novel touch-based biometric system using five finger multitouch gestures /12/$ IEEE 156

2 III. GESTURE TAXONOMY This paper explores the possibility of recognizing an individual based on physiological and behavioral biometric gleaned from hand geometry and muscle movement of five finger multi-touch gestures. Are multi-touch gestures sufficiently discriminative to provide biometric information? If yes, then what types of gestures can be used and which ones provide biometric distinguishability? Clearly not all multitouch gestures would be equally discriminative. For example a simple one finger drag gesture appears to contain less information about a user than a pinch or zoom gesture. In order to study these questions we first created a gestural taxonomy, based on movement of the two major components of the hand, the palm and the fingertips, as described below. 1) Palm movement: Some gestures involve placing one s hand in a static position for the entire gesture, whereas others require the hand to traverse while executing the gesture. Thus, we can divide gestures into two classes: Static palm: For examples, pinch or zoom gesture. Dynamic palm: For example, a drag or swipe gesture. 2) Fingertip movement: Most of the distinguishing features of multi-touch gestures derive from the movement of fingertips. We divide fingertip movement into four categories: Parallel: All fingertips are moving in the same direction during the gesture. For example, a five-finger swipe, in which all five fingers move from left to right. Close: All fingertips are moving inward toward the center of the hand. For example, a pinch gesture. Open: All fingertips are moving outward from the center of the hand. For example, a reverse pinch gesture. Circular: All fingertips are rotating around the center of the hand. For example, clockwise rotation. 3) Dynamic fingertips: In a gesture there may be one or more fingertips resting in a fixed position on the screen. This can help stabilize the gesture for the user. This yields an additional classification in our taxonomy. All fingertips moving: Example the pinch gesture. One or more fingertips resting: For example, resting a thumb while rotating the remaining fingers. The above classification results in a set of 22 gestures, noting that some of the combinations are not possible to perform such as dynamic palm with at least one fixed-finger. In addition, some commonly used gestures may have the same movement characteristic but they differ in direction of the palm movement or the speed, for example, swipe, flick and drag. The resulting set of 22 gestures is described in Table I. IV. MULTI-TOUCH GESTURE RECOGNITION A. Pre-processing A multi-touch gesture is basically a time-series of the set of x-y coordinates of finger touch points. Each set consists of five touch points where each touch point is generated from one fingertip. However, we do not know which fingertip corresponds to which touch point, as the system orders them based on how users lay their fingertips down. The first finger of the user to touch the surface may vary from one instance to another for the same gesture. And this order keeps fluctuating from one set within the time series to another within the same gesture. As a result, a set of touch points ordered by the system cannot be directly compared with another. Hence, the first step is to label each of the touch points with the corresponding fingertip. Specifically, for every touch point set S t = {(x i, y i ) i = 1,..., 5} at time instance t, we have to compute the unique bijective mapping from the set of five touch points to the set of fingertip labels I = {1,..., 5}. That is, we have to compute f t : S t I. Fig. 1: The illustration of a simple polygon generated from fingertip order We note that the closed path drawn from the five fingertip points in circular order is always a simple polygon, as shown in Figure 1. Therefore, we propose to compute the mapping f t, at each time instance as follows. Starting with the first set, S 0, the x-y coordinates of the five points in this set are used to compute their centroid. Then the points are sorted in ascending order by the angle they make with this centroid. Next, the thumb is identified by exploiting the property observed from typical hand geometry that the thumb is the furthest away from its immediate neighbors (in circular order) among all the fingers. Once we find the thumb, we find the index finger by following the x-y coordinates of the remaining points in a circular arc. Noting that, if the gesture is left handed the detected order will be thumb, pinky, ring, middle, and Index fingers respectively. However, this does not affect matching performance as the order is always consistent within the subject. Once we have computed f 0, that is, we have mapped the first set of points to the corresponding fingertips, the mapping for the subsequent remaining touch point sets can be modeled as an optimization problem where the possible mappings are constrained by the natural characteristic of hand geometry which dictates that when touch points corresponding to physically adjacent fingers are connected and the pinky is finally connected to the thumb, we get a simple polygon. In other words, if the label assignment is valid, connecting the points in S t ordered by its label in circular order ( ) will form a simple polygon. Given f t 1, the objective function in the optimization is to 157

3 TABLE I: Gesture descriptions and the number of subjects from whom data was collected per gesture Annotation Palm movement Fingertip movement Dynamic fingertips of subjects CCR Static Circular(CCW) All 32 CR Static Circular(CW) All 33 Closed Static Close All 34 Drag Dynamic( ) Parallel All 33 DDC Dynamic( ) Close All 33 DUO Dynamic( ) Open All 30 FBD Static Parallel( ) Fixed thumb and pinky 30 FBSB Static Parallel( shape) Fixed thumb and pinky 26 FBSA Static Parallel( shape) Fixed thumb and pinky 27 FPCCR Static Circular(CCW) Fixed pinky 28 FPC Static Close Fixed pinky 31 FPO Static open Fixed pinky 26 FPP Static Parallel( ) Fixed pinky 28 FTCCR Static Circular(CCW) Fixed thumb 30 FTCR Static Circular(CW) Fixed thumb 30 FTC Static Close Fixed thumb 31 FTO Static Open Fixed thumb 30 FTP Static Parallel( ) Fixed thumb 30 Flick Dynamic( ) Parallel All(Quick) 33 Opened Static Open All 34 Scrawl Dynamic(Customized) Parallel All 30 Swipe Dynamic( ) Parallel All 34 search for the current mapping function f t, such that the sum of the Euclidean distances between corresponding fingertips from the previous assigned sequence f t 1 to f is minimized. That is, we search for, f t = argmin f i I f 1 t 1 (i) f 1 (i) (1) such that {e i, i = 1,..., 5} constructs a simple polygon, where e i = ( f 1 (i), f 1 (i + 1) ) is the line segment drawn from the touch point corresponding to index i to the touch point corresponding to index i + 1 of the mapping function f : S t I. Noting that process may start from the last touch point set of a time-series input if the last set creates a larger simple polygon than the first set. B. Feature transformation According to differences in the hand s position and device s orientation within the same gesture, different trials from the same subject may result in different 2-D patterns on the touch surface. To cope with these variations, we extract higher order features by computing distances between pairs of the labeled points obtained after pre-processing. With different orientations and positions of the plane, the distance between any two points remains unchanged. This ensures that the features are rotation and translation invariant. To describe a polygonal curve < p 1, p 2,..., p n > of a gesture input, which is a sequence of n touch point sets, we calculate a feature vector for each touch point set to describe a high dimensional point p i as, 1) Euclidean distance feature: Let S t = {(x i, y i ) i = 1,.., 5} be a sorted set of touch points, each of the feature attributes F t (i, j) are calculated as, F t (i, j) = (x i x j ) 2 + (y i y j ) 2, (2) where i and j = 1,...,5. The number of distinct features for each touch point set in this case is 5 C 2 or 10. Afterward, F t is linearized, using only the upper triangle of the matrix (since the matrix is symmetric with all 0 s along the diagonal) to form a 10-dimensional point p i. 2) First order Euclidean distance feature: With the simple Euclidean distance feature set above, the movement direction and distance between two consecutive sets is not captured. Consequently, it could result in lower system accuracy. Therefore, we derive additional features as the following. Let S t = {(x ț i, yt i ) i = 1,.., 5} and S t+1 = {(x t+1,i, y t+1 i ) i = 1,.., 5} be two consecutive sorted sets of touch points, the additional feature attributes are calculated as, F t(i, k) = (x t i xt+1 i+k )2 + (y t i yt+1 i+k )2, k = 0, 1 (3) where i is the fingertip label in circular order ranging from 1 to 5. The number of distinct features for two consecutive sets is 5 C 2 or 10. To form a a point in 20-dimensional space, p i, the vector of 10 Euclidean distance from the first feature set is concatenated with F t(i, 0) and F t(i, 1). C. Distance function Taking a curve similarity approach, we use two different functions to measure the distance between a pair of polygonal curves representing touch sequences which may have different lengths. Let π =< p 1, p 2,..., p m > and σ =< q 1, q 2,..., q n > be two polygonal curves, M = {(p i, q i )} be an orderpreserving complete correspondence between π and σ, and d(p, q) is a matching cost between p and q, the following distances are defined as the following, 1) Discrete Frèchet distance: The discrete Frèchet distance is defined as [17], ( ) δ D (π, σ) = min M max d(p, q) (p,q) M (4) 158

4 2) Dynamic Time Warping distance: The dynamic time warping, which is a well-known matching algorithm to measure a similarity between two sequences which may vary in length, is defined as, DT W (π, σ) = min M (p,q) M d(p, q) min(m, n) D. Matching cost of curve distance functions Measuring curve similarity involves computing matching cost between two points on a fixed dimensional space. In our experiment, the performance of our system was evaluated using the following three cost functions, Let p and q be two points in l-dimensional Euclidean space, 1) Manhattan distance defined as, d M (p, q) = (5) l p i q i (6) i=1 2) Euclidean distance function is defined as, 3) Cosine distance is defined as, E. Decision rules d E (p, q) = l (p i q i ) 2 (7) i=1 d C (p, q) = 1 p q p q In this paper, we investigate two decision rules which are suitable for different applications and require different training sets. 1) A variant of k-nn based rule: In a variant of the k- NN based rule, the weight of each of k nearest instances is assigned as the inverse function of its distance to a test instance. Let d j be the distance from a test instance to its j th nearest neighbor among k nearest ones, the weight w j is given by [18], (8) w j = d k d j d k d 1 (9) In the verification case, the voting score needs to be normalized in order to cope with the differences between intrauser variations. To achieve that, the total weight of all nearest neighbors are normalized to 1. The normalized weight w j is defined as, w j w j = k i=1 w (10) i The verification score is then the sum of all weights of the neighbors belonging to the subject. The major disadvantage for k-nn is that it could only be used to classify a sample from a known class. That is, training the classifier requires samples from all classes. Practically, in some scenario, we may not have complete control over the input. In other words, the input might be from an unknown user. In this case, the performance of k-nn is unpredictable. 2) Score-based rule: For the situation where only positive samples (from an enrolled user) are known to the system, the score can only be computed based on these samples. In this case, we compute the score using a min rule given by, Score (G, T ) = min i (D (T i, G)) (11) Where Score (G, T ) is the distance from the input sample G to the set of training samples T, Ti is i th training sample, and D (Ti, G) is the curve distance between T i to G. V. EXPERIMENTAL RESULT AND ANALYSIS A. Data collection To collect user gesture data we implemented an ipad application using version 3.2 of ios, which has multi-touch capability to track up to 5 points at a time, to capture users gestures. The screen resolution was We recruited 34 participants of which 30 were right-handed, 28 had some multi-touch device experience, while only six had prior experience with the ipad. Their ages ranged from 15 to 50 years. Out of these, 24 were male. The data collection session began by explaining to the subject the purpose of study without informing them anything about the template creation or matching mechanisms. Next the participant filled out a brief pre-survey with demographic questions. Then, each subject was asked to practice performing the gesture a few times before supplying 10 samples and giving the feedback related to the gesture. In total, each of them was asked to supply 10 samples for each of the 22 multi-touch gestures identified above. They could skip any gesture they did not feel comfortable performing. We show in Table 1 the number of users that provided data for each gesture. Finally, the participants answered a set of questions about the overall experience. To evaluate performance of a multi-touch gestures based verification system, the samples of the gestures from a particular user were used to test against the samples from all other subjects. Specifically, each test sample was used N times, where N was the number of subjects. One time was to test as a positive sample label against its own class and each of (N- 1) times was to test as a negative sample label against (N-1) other classes. For k-nn classifier of each class, the positive training set composed of the sample gestures from a particular subject and the negative training set composed of the sample gestures from all other subjects. For score-based rule, only the sample gestures from a particular subject were used to train the classifier. To evaluate the gesture performance in verification mode, the Equal Error Rate (EER), the rate at which False Acceptance Rate(FAR) and False Rejection Rate(FRR) are equal, was adopted as a measurement matrix. B. A single gesture In this mode, the performance of each gesture was evaluated individually. First, we conducted the evaluation using a k-nn classifier trained on the first five samples from all subjects. Table II shows the performance of each gesture in terms of 159

5 TABLE II: EER for different curve distance functions and cost functions with k-nn classifier where k = 5 when the training set consists of the first five samples from all the subjects, and EER is an average of EER differences from the best EER at each gesture 10 feature set 20 feature set Gesture DTW Frèchet DTW Frèchet d M d E d C d M d E d C d M d E d C d M d E d C CCR CR Closed Drag DDC DUO FBD FBSB FBSA FPCCR FPC FPO FPP FTCCR FTCR FTC FTO FTP Flick Opened Scrawl Swipe EER EER when different distance functions, and the associated cost functions were applied. In total, there are six gestures that gave EER ranging from 5% to 10%, and 16 gestures that gave EER ranging from 0% to 5%. Overall, the average performance of each gesture was 3.63%EER, when the lowest EER for each gesture was considered. By evaluating the effect of curve distance functions, the results show that DTW distance outperformed Frèchet distance more than 2% on the average, for both feature sets. However, the results are very similar when using three different cost functions. In addition, the 20- feature set led to a better average EER than the set of 10 features in all cost functions using DTW distance. Next, we did experiments with the score-based approach in two settings. For the first setting, only the first five samples from a particular subject were used as the training. The test set consisted of the last five sample of a particular subject as a positive label and all ten samples from all others as a negative label. For the second one, we performed the experiment using the leave-one-out technique. In this test, one sample from a verifying class was used as a test sample with a positive label exactly once. When it was used as a test instance, all the others from the same subject were used as the training. The results from DTW distance using three cost functions are given in Table III. It is seen that Manhattan distance performed similar to Euclidean one and both outperformed the cosine distance. Since the final score in this work is basically the minimum distance from an input to the templates, this performance disparity could result from differences between inter-user as well as intra-user variations of different approaches. In other words, Manhattan and Euclidean distance functions produce smaller variations than the cosine function does. Therefore, using the global threshold approach, they reported the lower TABLE III: EER for DTW distance function of 20 features set with three different cost functions based on score-based classifier Gesture First 5 samples training Leave-one-out test d M d E d C d M d E d C CCR CR Closed Drag DDC DUO FBD FBSB FBSA FPCCR FPC FPO FPP FTCCR FTCR FTC FTO FTP Flick Opened Scrawl Swipe Average EER EER. Moreover, in leave-one-out setting, EER for each cost function was decreased by 3% by average when compared to the first five sample setting. This implies that the number of training samples have an impact on system s accuracy. C. Multiple gestures In this mode, multiple defined gestures from the same subject, one sample per gesture, were combined to form an input. The objective here is to evaluate the possibility of improving the system performance by combining different information extracted from multiple gestures. To test out the strategy, we 160

6 carried out the experiment where we used only the first five samples of all selected gestures from a particular subject. In each setting, the test set consisted of all possible permutations. For example, with two gestures combination, for the first setting, the number of test samples with a positive label for each subject was 25 which are all permutations from the last five samples from each gesture. Figure 2 shows the Detection Error Trade-off curves (DET) of different combinations using the first five samples as the training. The gestures included static counter-clockwise rotation, closed and opened, all with five fingertips, which are commonly used to interact with the multi-touch devices. It shows that more number of gestures generally provide more discrimination power to the system. As seen that, specifically, at 2% FAR, the system achieved 1.15% FRR when this 3-gesture applied compared to greater than 3% when only 2 gestures applied and greater than 13% when a single gestures applied. In addition, the best EER at 1.58% obtained when all three gestures are applied. Fig. 2: Detection error trade-off curve of different combinations and the base line curves from a single gesture using first five samples as the templates VI. CONCLUSION AND FUTURE WORK In this paper, we have investigated the feasibility of a new behavioral biometric modality based on multi-touch gestures. We first defined a multi-touch gesture taxonomy specifically for biometric verification purpose and identified a set of 22 gesture candidates for our experiments. In order to design a classifier we developed a feature set that is invariant to translation and orientation. Based on the designed feature set, we tested 2 different curve distance functions and 3 different matching cost functions. The result showed that the normalized DTW outperformed the Frèchet distance for our feature set. For the different matching cost functions, the results were all similar for the k-nn classifier. However, for the score based classifier, the cosine function performed poorer than Euclidean and Manhattan distances. We also showed that multiple gestures strategy resulted in lower EER. This indicates that complementary information can be extracted from different gestures. Our results show that multi-touch interaction has a potential not only in the design of a novel user-interface, but also to develop a biometric based authentication application. For future work, we plan to further evaluate the property of a multi-touch gesture by investigating intra-user variation over a longer time span as well as collecting the data from more subjects to confirm the level of accuracy achievable. We also plan to investigate a fusion method to combine different classifiers or other biometric modalities in order to obtain higher accuracy. In addition, we will investigate how to incorporate other information like pressure, to improve performance. ACKNOWLEDGMENT The authors would like to thank Kowsar Ahmed for helping collect the data throughout our experiment and Umut Topkara for his insightful comments early in the project. REFERENCES [1] D. Zhang, W. Zuo, and F. Yue, A comparative study of palmprint recognition algorithms, ACM Comput. Surv., vol. 44, no. 1, pp. 2:1 2:37, Jan [2] M. Faundez-Zanuy, On-line signature recognition based on vq-dtw, in Pattern Recogn., March 2007, pp [3] A. Kholmatov and B. Yanikoglu, Biometric authentication using online signatures, in Proc. ISCIS, Springer LNCS-3280 (2004) Springer-Verlag, 2004, pp [4] R. B. Hill, Retina identification, pp , [5] R. Sanchez-Reillo and C. Sanchez-Avila, Iris recognition with low template size, in Proceedings of the Third International Conference on Audio- and Video-Based Biometric Person Authentication, ser. AVBPA 01. London, UK: Springer-Verlag, 2001, pp [6] A. K. Jain and D. Maltoni, Handbook of fingerprint recognition, [7] M. Turk and A. Pentland, Face recognition using eigenfaces, in Computer Vision and Pattern Recognition, Proceedings CVPR 91., IEEE Computer Society Conference on, jun 1991, pp [8] C.-L. Lin and K.-C. Fan, Biometric verification using thermal images of palm-dorsa vein patterns, Circuits and Systems for Video Technology, IEEE Transactions on, vol. 14, no. 2, pp , feb [9] A. Jain, A. Ross, and S. Pankanti, A prototype hand geometry-based verification system, in 2nd Int l Conference on Audio- and Video-based Biometric Person Authentication, Washington D.C., [10] A. Kholmatov and B. Yanikoglu, SUSIG: an on-line signature database, associated protocols and benchmark results, Pattern Analysis & Applications, [11] D. A. Reynolds, T. F. Quatieri, and R. B. Dunn, Speaker verification using adapted gaussian mixture models, Digital Signal Processing, vol. 10, no. 1-3, pp , [12] D. Gafurov, K. Helkala, and T. Sndrol, Biometric gait authentication using accelerometer sensor, Journal of Computers, vol. 1, no. 7, [13] M. Obaidat and B. Sadoun, Verification of computer users using keystroke dynamics, Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, vol. 27, no. 2, pp , apr [14] R. Maxion and K. Killourhy, Keystroke biometrics with number-pad input, in Dependable Systems and Networks (DSN), 2010 IEEE/IFIP International Conference on, july , pp [15] C. Shen, Z. Cai, X. Guan, H. Sha, and J. Du, Feature analysis of mouse dynamics in identity authentication and monitoring, in Communications, ICC 09. IEEE International Conference on, june 2009, pp [16] O. Hamdy and I. Traoré, Homogeneous physio-behavioral visual and mouse-based biometric, ACM Trans. Comput.-Hum. Interact., vol. 18, pp. 12:1 12:30, August [17] B. Aronov, S. Har-Peled, C. Knauer, Y. Wang, and C. Wenk, Frèchet distance for curves, revisited, in Proceedings of the 14th conference on Annual European Symposium - Volume 14, 2006, pp [18] S. A. Dudani, The distance-weighted k-nearest-neighbor rule, Systems, Man and Cybernetics, IEEE Transactions on, vol. SMC-6, no. 4, pp , april