Finger alphabets recognition with multi-depth images for developing their learning system

Transcription

1 Finger alphabets recognition with multi-depth images for developing their learning system Daisuke TAKABAYASHI Yoto TANAKA Akio OKAZAKI Nobuko KATO Hideitsu HINO Kazuhiro FUKUI This paper proposes a method for recognizing Japanese finger alphabets using sets of multi-viewpoint depth images. The proposed method can recognize 41 finger alphabets with high accuracy. Based on the recognition method, we develop an interactive system for learning Japanese finger alphabets. The most remarkable characteristic of our system is that it can give appropriate real-time feedbacks, such as a useful guidance on how to correct the wrong hand shape inputted by a learner, according to the recognition results. The effectiveness of our recognition method and learning system are demonstrated through evaluation experiments. Keywords: finger alphabet, depth image, hand shape recognition 1. Introduction Finger alphabets shown in Fig. 1 is an indispensable communication tool for people with hearing impaired. The hand shapes for representing the finger alphabets are very complex and some of them are very similar to each other, hence it is difficult for beginners to learn them on their own. Therefore, an interactive system which tells the mistakes to a learner is desirable. This paper presents a novel method for recognizing various and complex hand shapes. By using this method, an interactive system for learning finger alphabets is proposed. It is difficult to recognize complex hand shapes used for finger alphabets automatically. In addition, the large number of alphabets to be recognized makes the recognition more difficult. Many methods for hand shape recognition have been proposed. They are classified into two types; methods using a contact type device of data grove (1) or color grove (2), and methods using a non-contact type device such as RGB camera (3), depth sensor or both of them (4) (5). In the learning system for finger alphabets, it is troublesome for learners to wear such devices which can also disturb learners hand movement. Therefore, the first type of the methods is not appropriate for the learning system of finger alpha- Graduate School of Systems and Information Engineering University of Tsukuba 1-1-1, Tennodai, Tsukuba, Ibaraki, Japan , takabayashi@cvlab.cs.tsukuba.ac.jp Graduate School of Technology and Science, Tsukuba University of Technology , Amakubo, Tsukuba, Ibaraki, Japan Faculty of Industrial Technology, Tsukuba University of Technology , Amakubo, Tsukuba, Ibaraki, Japan Fig. 1. Japanese finger alphabets. bets, and we develop a learning system based on the second type of the methods. Among the second type of the methods, the one using a depth sensor is preferable because the information from a depth image preserves the shapes of target, while it is invariant to the changes in illumination conditions. This property makes the system applicable under various lighting conditions. However, the shape obtained from a single depth image is only a portion of information contained in a 3D object, which is visible from a single view point. Since the appearance of a hand shape having complex 3D shape often changes depending on the view point, it is difficult to get a full shape information from only a single depth image. To address the above issue, our method performs recognition using a set of sequential depth images of a hand. They include more information about the hand shape compared to a single depth image. To handle the sequential images effectively, we represent each set of multi-depth images by a nonlinear subspace (called volume subspace (6) ), which is generated by Kernel PCA (KPCA (11) ) and calculate the similarity between two FCV2014 1

2 Fig. 2. A conceptual diagram of recognition using multi-view depth images. volume subspaces based on the canonical angles between them. By considering these subspaces, we can take full 3D shape information into account. Equipped with the highly accurate subspace-based classification method, we develop an interactive learning system for finger alphabets based on the method using sequential depth images. Our interactive learning system can recognize 41 finger alphabets in real time and provide information of the most matched finger alphabet to the learner. Moreover, our method can distinguish minor differences in shapes more accurately than that of the method using gray scale images (3). By using this high classification ability, we realize the function of error information feedback. The rest of the paper is structured as follows. Section 2 describes the proposed method for hand shape recognition. Section 3 presents our system constructed based on the proposed method. Section 4 demonstrates the performance of the proposed method through experiments. We draw conclusions in Section Robust finger alphabet recognition with multi-view depth images In this section, we describe how to recognize complex hand shapes used for finger alphabets in our system. Figure 2 shows a conceptual diagram of our recognition method with multi-view depth images. As shown in this figure, our method performs classification based on the structural similarity between two distributions of input and reference data patterns. To measure the similarity between two distributions, in our method, each distribution is represented by a nonlinear subspace, which is generated by applying KPCA to sets of data generated from theses distributions, and then the similarity between two nonlinear subspaces is measured by the canonical angles between them. Although this process can be done under the framework of Mutual Subspace Method (MSM) (7), to achieve higher performance, we apply Kernel Orthogonal Mutual Subspace Method (KOMSM) (8), which is one of the nonlinear extensions of MSM. 2.1 Orthogonal Mutual Subspace Method In the following, we first explain the algorithm of MSM as a base line, and then we explain its extension, Orthogonal Mutual Subspace Method (OMSM). OMSM improves the classification performance by enlarging angles between different class subspaces. Finally we describe the algorithm of Kernel OMSM (KOMSM) (9), which is a nonlinear extension of OMSM Measurement the similarity between two subspaces First, reference and input subspaces are generated by applying PCA to the set of multiple depth images. Then, the canonical angles between the reference and the input subspaces are calculated as follows. Given an m- dimensional subspace P and an n-dimensional subspace Q, m canonical angles (0 θ 1... θ m π/2) are defined. For convenience, we assume m n. The first canonical angle θ 1 is the smallest angle between P and Q. The second canonical angle θ 2 is the smallest one along the direction orthogonal to θ 1. The i-th canonical angle θ i (i = 3,..., m) is calculated in the same manner. A practical method to find the canonical angles is by computing singular values of the matrix U T V, where U = [u 1, u 2,...], V = [v 1, v 2,...]; u i and v i are the i-th orthonormal basis vectors of the subspace P and Q, respectively. In the recognition phase, the similarity S is defined by S = 1 m cos 2 θ i. m i= Orthogonalization of subspaces In OMSM, subspaces are orthogonalized using the framework of Fukunaga and Koontz s (10) before measuring the canonical angles between them. In their framework, the orthogonalization is performed by applying a whitening transform matrix O to reference subspaces. Let P i (i = 1,..., k) be the projection matrix to the subspace of each class, where k is the number of classes. If we define the matrix G = k i=1 P i as the sum of the projection matrices corresponding to the projections, O is calculated as O = Λ 1 2 B T, where Λ is the diagonal matrix with the i-th largest eigenvalue of matrix G as the i-th diagonal component, and B is a matrix whose i-th column vector is the eigenvector of the matrix G corresponding to the i-th largest eigenvalue. 2.2 Kernel Orthogonal Mutual Subspace Method In KOMSM, the process of OMSM is performed on a high-dimensional feature space. To perform PCA in high-dimensional vector space, it is necessary to calculate (Φ(x) Φ(y)), where Φ(x) and Φ(y) are the mapped vectors of x and y to the feature space. Since their dimensions can be extremely high, it is difficult to compute the inner product of them directly in general. Therefore, the method called kernel trick (11) is applied. Kernel trick replaces the dot product by a kernel function k(x, y). The Gaussian kernel shown bellow is one of the commonly used kernel functions. k(x, y) = exp ( x y 2 2σ 2 ), 2 FCV2014

3 Finger alphabets recognition with multi-depth images for developing their learning system Fig. 3. Process flow of the system. The numbers in the figure correspond to the process number shown in Sec Fig. 4. Screen shot of the interface system : (1) Inputted depth image, (2) Region of hand, (3) Recognition result, (4) Similarities indicated by a bar plot, (5) Similarities indicated by polygonal lines, (6) The sequential recognition result, (7) Position of input hand shape. where σ is a bandwidth parameter. In our system, we use the Gaussian kernel as the kernel function. Details of KOMSM can be found in (8). 3. Interactive system for learning finger alphabet In this section, we explain the flow of our system with Fig Flow of the proposed system ( 1 ) Depth images are captured by using a depth sensor (DS325). The DS325 sensor can output highly accurate depth images by the technique of time of flight (TOF), in spite of its inexpensive price. ( 2 ) The hand region is extracted from each captured depth image by using depth value, assuming that the object closest from the sensor is the hand. ( 3 ) The size of the extracted hand region is converted to the size of pixels. Intensities of the depth image is normalized to the range in [0, 1.0]. ( 4 ) The normalized depth images are vectorized. Then KOMSM with acceleration (3) is applied to make the system work in real time. ( 5 ) Various sorts of information from the recognition process are displayed on the screen as feedbacks for the learner. A screen shot of user interface of our system is shown in Fig. 4. Sub-windows in Fig. 4 represent the following information: ( 1 ) Depth image obtained from DS325 sensor. ( 2 ) Region of hand extracted from the input depth image. ( 3 ) Recognition result : Showing the image of finger alphabet. ( 4 ) Similarities indicated by bar plots : the x-axis is class of finger alphabet and the y-axis is similarity. ( 5 ) Similarities indicated by polygonal lines : The x-axis is time and the y-axis is similarity. ( 6 ) The sequential recognition result : Output the text string. ( 7 ) The relative position of input shape in the space of finger alphabet. The three similar shapes to the input shape are plotted at the three vertices and the input shape is plotted among them. 3.2 Error information feedback Figure 5 shows an example of the process flow of the proposed error feedback function. As mentioned previously, our recognition method using KOMSM has the ability of detecting minor differences between similar shapes. This ability enables our system to provide appropriate feedbacks to the learner in real time. Although most of small mistakes can be seen as the habits of the user, some of them cause serious misunderstandings. Figure 6 shows an example of such case that the input hand shape for Japanese alphabet a has been confused with that of sa. In this case, the difference between the positions of the thums of the correct and input shape is large. Hence, the system regards the input shape as an incorrect shape and provides a feedback message. With the aid of this function, learners FCV2014 3

4 Fig. 5. Process flow of mistake information feedback. Fig. 7. The feedback for representing the correctness of the hand shape by a user. Fig. 6. Example of a minor mistake. The hand shape in the middle figure is the input image given by a learner intended to express the finger alphabet a, while it is more closer to the correct shape of sa. Fig. 8. Recognition rate for each person. can correct the very minor but risky mistakes. To realize the error information feedback function, we collect the learning patterns of hand shapes with minor errors and register them as error reference hand shapes. Figure 7 shows the feedback for representing the correctness of hand shape by a user. In this figure, the green dot indicates relative position of the inputted hand shape to other similar hand shapes. The correct and two wrong shapes are placed at the top vertex and two bottom vertices, respectively. The position vector, P s, of the inputted hand is calculated by using the following formula: P s = S 1 P 1 + S 2 P 2 + S 3 P 3, where S 1, S 2 and S 3 are the similarities normalized to satisfy S 1 + S 2 + S 3 = 1.0, and P 1, P 2 and P 3 are the position vectors of the three vertices. In addition, the function to enlarge the symbol closest to the dot is implemented to understand the result easily as shown in the figure. 4. Experiments To evaluate the performance of our system, we conducted the following two experiments. Experiment 1 : Recognize 41 finger alphabets Experiment 2 : Recognize minor error shape The dataset contains 2,000 samples from each shape (finger alphabet and mistake shape), recorded from 10 different subjects (non-native to sign language). The dataset contains multi-view depth images of slightly rotated hand shapes. We tuned KOMSM parameters as follows: Dimension of reference subspaces : 50 Dimension of input subspace : 4 The number of input data : Experiment 1 In this experiment, we evaluated the recognition performance of the proposed method. We divided the data in two ways. First, we divided the data of each subject into two parts and used the first half as training samples and the rest as test samples. Thus, the evaluation was performed by using the data captured from the same person. Second, to evaluate the performance of our method in more practical situation, we used the data of one subject as test sample and the remaining nine subjects as training samples Results of Experiment 1 Figure 8 shows the experimental results using same person s data. From these results, we can see that recognition rate is high when learning from the data of same subject. On the other hand, the recognition rate using data of different person is 88.66%. The reason of decreasing recognition rate may be due to the influence of variation of hand shapes and the way of showing the hand. It may be solved by increasing the number of training samples. The recognition rate of each finger alphabet is shown in Table. 1. This table shows that recognition rate for finger alphabet i and chi are 4 FCV2014

5 Finger alphabets recognition with multi-depth images for developing their learning system Table 1. Recognition rate of each finger alphabets. very low. This low recognition rate can be attributed to the fact that unlike RGB images, depth images have information on only the boundary of the hand shape. In our future work, we will consider to combine depth and RGB images to improve the recognition rate. 4.2 Experiment 2 In this experiment, we examined whether the feedback of error information is properly presented. We selected five kinds of shapes from error shapes which are difficult to judge for learners and tend to be misunderstood to other finger alphabets. Among the five finger alphabets (Fig. 10), we conducted three class classification experiments (one correct shape and two error shapes). In the data captured from 10 people, the data of one person was used as the test sample, and the rest were used as training samples Results of Experiments 2 Figure 11 shows the result of the experiment 2. From this table, we can see that our method can detect slight differences in hand shape in most the cases. These results suggest that we can construct the error information feedback by using our recognition method. However, in the case of finger alphabet e, the recognition rate is low compared with other cases. The reason can be explained as follows: In the above successful cases, the differences are due to the variation of finger angles and the opening and closing of figures. In contrast, in this case, the whole shapes of the three hand shapes are very similar except to their sizes. Nevertheless, the information of the sizes is lost because of the normalization of the size of the hand region. To deal with this problem, we have to consider how to keep the information of the size. 5. Conclusion In this paper, we proposed a method for hand shape recognition by applying KOMSM to multi-view depth images. Besides, we developed a real time feedback system for learning finger alphabets based on the method. The experiments validated the effectiveness of our proposed method. In future works, we will consider the possibility of using CG technique (12) to synthesize a lot of training samples automatically. Development of more effective and user friendly interface to provide error feedback to the learner is also our important future work. Fig. 9. A pair of the hand shapes which are often confused each other. Fig. 10. Shapes used in the experiment 2. Ackowledgements This work is supported by JSPS KAKENHI Grant Numbers , We show our special thanks for Chendra Hadi Suryanto for valuable comments on early version of the manuscript. References ( 1 ) Y. Tabata, T. Kuroda, Y. Manabe, K. Chihara : A Study on Finger Character Education System based on Hand Posture Recognition, Transactions of Japanese Society for Information and Systems in Education 18(2), pp (2001).(in Japanese) ( 2 ) RY. Wang, J. Popovic : Real-Time Hand-Tracking with a Color Glove, ACM Transactions on Graphics(TOG), vol. 28, issue 3(2009). ( 3 ) Y. Ohkawa, K. Fukui : Hand Shape Recognition Using the Distributions of Multi-Viewpoint Image Sets,IEICE Transactions, Vol.E95-D,No.6,pp (2012). ( 4 ) M. V. den Berg, L. V. Gool : Combining RGB and TOF cameras for real-time 3d hand gesture interaction, Proc. of the FCV2014 5

6 Fig. 11. Confusion matrices obtained in the experiment. IEEE Workshop on Applications of Computer Vision (WACV 2011), (2011). ( 5 ) N. Pugeault, R. Bowden : Spelling it out: Real-time ASL fingerspelling recognition, ICCV Workshops, pp (2011). ( 6 ) M. Peris Martorell, K. Fukui : Both-hand Gesture Recognition Based on KOMSM with Volume Subspaces for Robot Teleoperation, IEEE-Cyber2012, pp (2012). ( 7 ) O. Yamaguchi, K. Fukui, K. Maeda : Face recognition using temporal image sequence, Automatic Face and Gesture Recognition, pp (1998). ( 8 ) K. Fukui, O. Yamaguchi : The Kernel Orthogonal Mutual Subspace Method and its Application to 3D Object Recognition, Asian Conference on Computer Vision,Vol. 2,pp (2007). ( 9 ) T. Kawahara, M. Nishiyama, T. Kozakaya, O. Yamaguchi : Face Recognition based on Whitening Transformation of Distribution of Subspaces., Workshop on ACCV2007, Subspace2007, pp (2007). (10) K. Fukunaga, Introduction to statistical pattern recognition. Academic Press Professional, (1990). (11) S. Taylor, N. Cristianini : Kernel Methods for Pattern Analysis, Cambridge University Press, (2004). (12) K. Taji, K. Setoyama, Y. Ohkawa, N. Kato, A. Okazaki, K. Fukui : Finger alphabet recognition using Adaboost based of CG data,wit, (2012).(in Japanese). Daisuke TAKABAYASHI received his Bachelor s degree in 2012 from Tsukuba University. He is currently a master student at Tsukuba University. His research interest includes the hand shape recognition using multi depth images. Yoto TANAKA received his Bachelor s degree in engineering in 2012 from Tsukuba Unibersity of Technology. He is now a master student at Tsukuba University of Technology. He is interested in training system for learning finger spellings using image recognition. Akio OKAZAKI received his Bachelor s degree, Master s degree and PhD in engineering from Nagoya University, in 1975, 1977 and 1982, respectively. He worked for Toshiba Corporatin from 1980 to He is currently a professor in the Department of Industrial Information, Faculty of Industrial Technology, Tsukuba University of Technology. His research interests include image pattern recognition, human interface and their applications. He is a member of IEEE, IEICE and IPSJ. Nobuko KATO received her Bachelor s degree and Master s degree in Science from University of Tsukuba, and she joined Toshiba Corporate Research and Development Center. She received her PhD in 2000 from University of Tsukuba. She is currently a professor in the Department of Industrial Information, Faculty of Industrial Technology, Tsukuba University of Technology. Her interests include the assistive technology, communication support, human interface and their applications. Hideitsu HINO received his Bachelor s degree in engineering in 2003, and Master s degree in Applied Mathematics and Physics in 2005 from Kyoto University, Japan. He joined Hitachi s Systems Development Laboratory and worked as a research staff from April 2005 to August He earned Doctor s degree in engineering in 2010 from Waseda University. He worked for Waseda University from April 2010 to March 2013, and from April 2013, he is an Assistant Professor at University of Tsukuba. His research interest includes the analysis of learning algorithms from the view point of geometry. He is also interested in time series analysis, kernel methods, distance metric learning, ranking models and their applications. He is a member of IEEE, IEICE and IPSJ. Kazuhiro FUKUI received his B.E. and M.E. (Mechanical Engineering) from Kyushu University in 1986 and 1988, respectively. In 1988, he joined Toshiba Corporate Research and Development Center and served as a senior research scientist at Multimedia Laboratory in He received his Ph.D. from Tokyo Institute of Technology in He is currently a professor in the Department of Computer Science, Graduate School of Systems and Information Engineering at University of Tsukuba. His interests include the theory of computer vision, pattern recognition, and applications of these theories. He has been serving as a program committee member at many pattern recognition and computer vision conferences including an Area Chair of ICPR 12 and ICPR FCV2014