Yi Zhang and Shuo Zhang

Transcription

1 104 Int. J. Computer Applications in Technology, Vol. 49, o. 2, D hand gesture tracking and recognition for controlling an intelligent wheelchair Xiaodong Xu* School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu , China *Corresponding author Yi Zhang and Shuo Zhang Engineering Research & Development Centre of Information Accessibility, Chongqing University of Posts and Telecommunications, Chongqing , China Huosheng Hu School of Computer Science and Electronic Engineering, University of Essex, Colchester CO4 3SQ, UK Abstract: Hand gesture recognition is a user-friendly and intuitive means for human machine interaction. This paper proposes a novel 3D hand gesture recognition method for controlling an intelligent wheelchair based on both colour and depth information. Image depth information of human palm is obtained by a 3D Kinect vision sensor and then its position is obtained through the hand analysis module in OpenI. The improved Centroid Distance Function is used to extract 3D hand trajectory features, while hidden Markov model (HMM) is applied to train samples and recognise hand gesture trajectories. Finally, the recognition results are converted into control commands through an ad hoc network and sent to an intelligent wheelchair for its motion control. Experiment results show that the proposed method has good invariance to lighting changes, hand rotation and scaling conditions and is very robust to background interference. Keywords: 3D gesture tracking; HMM; hidden Markov model; pattern recognition; intelligent wheelchair. Reference to this paper should be made as follows: Xu, X.D., Zhang, Y., Zhang, S. and Hu, H. (2014) 3D hand gesture tracking and recognition for controlling an intelligent wheelchair, Int. J. Computer Applications in Technology, Vol. 49, o. 2, pp Biographical notes: Xiaodong Xu is currently a PhD student at University of Electronic Science and Technology of China. He also gets involved in the multi-modal HRI research at ational Engineering Research & Development for Information Accessibility. His research interests include intelligent control system and its application, multi-modal HRI, service robotics. Yi Zhang received his PhD from Huazhong University of Science and Technology, Wuhan, China in 2002, and received post-doctorate from Southeast University, anjing, in ow he is a Professor in Chongqing University of Posts and Telecommunications. He has published over 150 papers in journals, books and conferences in these areas, applied for 12 national patents in which five patents have been awarded. He has also published one monograph and three text books His research interests mainly include robotics and its applications, data fusion. Shuo Zhang obtained her MSc in the Centre for ational Engineering Research & Development for Information Accessibility at Chongqing University of Posts and Telecommunications, China. Her research interests include intelligent wheelchair and its application, digital image processing. Huosheng Hu is a Professor in School of Computer Science & Electronic Engineering at the University of Essex, UK, leading the human-centred robotics research. His research interests include behaviour-based robotics, human-robot interaction, embedded systems, mechatronics, Copyright 2014 Inderscience Enterprises Ltd.

2 3D hand gesture tracking and recognition for controlling an intelligent wheelchair 105 learning algorithms and pervasive computing. He has published over 380 papers in journals, books and conferences in these areas, and received a number of best paper awards. He is a Fellow of Institute of Engineering and Technology, a Fellow of Institute of Measurement and Control, a senior member of IEEE and ACM. He has been a Chair or committee member for many international conferences such as IEEE ICRA, IROS, ICMA, ROBIO conferences. He currently serves as an Editor-in-Chief for International Journal of Automation and Computing, Editor-in-Chief for Robotics Journal and an Executive Editor for International Journal of Mechatronics and Automation. 1 Introduction To promote the construction of a harmonious society and improve the quality of life for elderly and disabled people, advanced technology needs to be deployed. Many countries began to research the intelligent wheelchair, which provides mobility to the elderly and people with disabilities. According to different disabilities, researchers have developed many human machine interfaces for intelligent wheelchairs. Recently, hand gesture recognition plays an important role in human computer interaction (Kao and Li, 2010; Tang et al., 2013; Zhang et al., 2009, 2012). Hand gesture recognition in human computer interaction has the advantages of being easy to implement and user-friendly (Van den Bergh and Van Gool, 2011; Pang et al., 2010). Hand trajectory is a simple and robust movement feature, and widely applied to the hand motion analyses. For instance, the upper arm motion trajectory is used for rehabilitation therapy in Zhang et al. (2008) and Wang et al. (2012), where local hand gesture trajectory variables are combined with global ones to improve the efficiency of gesture recognition. A method of using the HMM model to recognise the Chinese sign language was proposed in Zhang and Zhang (2010). Silanon and Suvonvorn (2011) put forward a method of gesture movement analysis and identification of the Thai alphabet using hand gesture trajectory features. ote that all these methods are identified based on the 2D trajectory. All the hand gesture recognition systems with one camera have apparent limitations. First, different gestures may share the same 2D trajectory due to the projection, and the same gesture may generate various projections from different viewpoints in one plane. Secondly, viewpoint problems can be solved by adding more categories with different viewpoints, which increases the computation burden and decreases the recognition rate (Holte, 2008). Therefore, the 3D hand gesture trajectory recognition becomes necessary. Hahn et al. (2009) proposed a 3D gesture track recognition method in an industrial environment. Yuan et al. (2010) proposed a same perspective 3D trajectory recognition algorithm that calculates main plane and maps 3D trajectory to it by using the least squares method. At present, hand gesture recognition research is mainly focused on the skin colour model, as well as continuous dynamic gestures based on the image attributes of the robust feature extraction (Chen and Zhang, 2009). o depth information has been widely deployed in hand gesture recognition yet. Therefore, this paper presents a novel method for 3D gesture trajectory feature extraction and recognition based on a 3D Kinect vision sensor. Depth image information is used for hand segmentation, which is the distance between the camera and the hand. So the background interference can be greatly reduced and the system is robust and real time. In this paper, a 3D hand gesture tracking and recognition method is proposed for controlling an intelligent wheelchair. The method of hand gesture detection is explained. The depth data are obtained from a 3D Kinect vision sensor and the palm of the hand position is obtained through the intermediaries hand analysis module in OpenI. The improved centroid distance function (CDF) is proposed to extract 3D hand gesture trajectory feature. Therefore this system is less susceptible to light changes, background noises and other factors, and the stability and robustness of the hand gesture trajectory recognition are much improved. In addition, the starting point of hand trajectory is firstly located and then the end point is obtained. This solves the problem of time delay in the previous methods and improves the real-time performance of the system. The rest of the paper is organised as follows. In Section 2, the method of hand gesture detection is introduced. Section 3 presents the recognition method of 3D hand gesture trajectory based on hidden Markov model (HMM). Section 4 is the experiment results of our new method. A brief conclusion and future work is illustrated in Section 5. 2 Hand gesture detection 2.1 Getting hand gesture trajectory information A 3D Kinect vision sensor is used in this research to capture images, which has a CMOS infrared sensor for perceiving the world. The sensor perceives the environment through the way of black and white spectrum, pure black represents infinity and grey area represents physical distance of object from the sensor. The Kinect sensor collects every point within sight, and then the gathering point forms the depth of field image about the surrounding environment. The sensor generates the image stream at 30 frames per second, and obtains 3D data from its surrounding environment in real-time. In human computer interaction, interactive hand gestures are often located in front of the camera, so the hand gesture region can be segmented according to different

3 106 X.D. Xu et al. depth value. The hand gesture segmentation method in this paper combines the hand analysis module in OpenI and the visual development kit in OpenCV. Figure 1 describes the hand gesture segmentation process. Figure 1(a) and (b) shows the ordinary image and depth image. Figure 1(c) and (d) shows the binary image and the region centre of the hand gesture. When the hand moves, the palm of the hand may shake and results in the hand jitter which is to be addressed in this paper presents. We compare the previous frame with the current frame in a sequence of hand images. If the difference between two frames is within the threshold range, the position of the previous frame is saved. If the difference is outside the threshold range, the location of the current frame is then saved. Therefore, the jitter with the small amplitude can be avoided. The threshold value is set at 2.4 mm after trial and error. Figure 1 Hand gesture segmentation and the palm of the hand location image: (a) an ordinary image; (b) depth image; (c) binary image and (d) the centre of hand (see online version for colours) so that the estimation accuracy can be effectively improved. This paper proposes a novel hand tracking method that combines Camshift algorithm with Kalman filtering for tracking hand gestures, fully utilising the advantages of the two algorithms. 2.3 Hand gesture spotting Lee and Kim (1999) proposed a HMM-based threshold model approach to hand gesture recognition. This model plays a key filtering role in the identification of non-gesture, and also effectively detects the start and end points of the predefined gestures contained in a trajectory sequence. On the basis of their work, this paper puts forward a new method, i.e. the start point is detected first and then the end point. So the time delay problem is solved, and both the real-time performance and the recognition rate are improved. The starting point is judged by the speed. When the hand speed is zero in a small area for a moment, its position is considered as a starting point. After the start point is detected, hand gesture trajectory recognition is started. The gesture trajectory recognition is completed when the end point is obtained by using Equation (1). ( λ ) ( λ ) p = P O P O, (1) i T (a) (c) (b) (d) where p is the competitive differential observation probability value between maximal gestures and nongesture. When p changes from negative to positive, the gesture ends and the end point is obtained. More than one reference model may meet the condition of p changes from positive to negative in a different location since the test sample contains many reference gestures. The first transition point cannot be simply treated as the end point since it cannot determine if this gesture is indeed part of a complete gesture or within more complex gestures. Therefore, this paper proposes that the interval between the two location points must be more than 25 frames. The identification of a complete hand gesture trajectory is achieved through the Viterbi algorithm. 2.2 Hand tracking Camshift algorithm is an improved method of Meanshift algorithm and can be effectively deployed for real-time visual tracking. Although Camshift algorithm has much reduced calculation time and better tracking performance than Meanshift algorithm in simple background, it remains vulnerable to environmental disturbance once the background becomes complex. So the estimated gestures motion parameters are introduced here to avoid these problems. Kalman filtering is an optimum estimate method under the rule of minimum error covariance, which needs small amount of calculation and therefore achieves real-time performance. Kalman filter uses the actual sensor information to correct the estimate of the motion state 3 3D hand gesture trajectory recognition based on HMM 3.1 Trajectory feature extraction Hand gesture trajectory feature can be mapped to the CDF observation-based invariants (Bashir et al., 2006). CDF has affine transformation invariance and is widely used in image retrieval. CDF represents the shape of the gesture trajectory observations from a global perspective, which is used to describe gesture trajectory. In essence, CDF is the distance of each point to the centre points of the trajectory. The nearest integer value of CDF is deployed as the HMM observation vector. CDF can besimply expressed as follows:

4 3D hand gesture tracking and recognition for controlling an intelligent wheelchair 107 [] [] [] 2 2 Ct = xt xc + yt yc, t= 0,1,, 1, (2) where x 1 1 yt 1 1 c [], y = yt [] t 0 c = = are the x, y t= 0 coordinates of the centre, respectively. 3.2 Improved CDF for feature extraction of 3D hand gesture trajectory At present, the basic CDF feature is 2D. In fact, most hand gesture trajectories are 3D. Different trajectories may have the same 2D trajectory if they have the different angle projection. The same gesture in the same plane through a different perspective projection could generate different gestures. To classify these gestures is time consuming. Therefore, this paper adds the third dimension information into the CDF algorithm to improve the classification speed. The improved CDF is used to extract 3D hand gesture features by using the formula: [] [] [] [] c c c, Ct = xt x + yt y + zt z (3) where t = 0,1,, 1; The x, y, z coordinates of the centre are xc = yt []; y []; [], t 0 c = yt z t 0 c zt = = = t= 0 respectively. The improved CDF needs translation, rotation and scaling invariance. Re-sampling of the trajectory points solves the problem of scaling. Translation invariance can be achieved through coordinate standardisation. So this paper only verifies the rotation invariance of 3D CDF features. As this paper deals with the data are three-dimensional curves, rotation transformation can be decomposed into twice 2D rotations. For the first time rotation in plane: G1 = W0 G, (4) where = x() t y() t z() t = [ x () t y () t z () t 1] T G [ 1] T is the original matrix; G is the matrix after the rotation transform; cosα sinα 0 W 0 = is the rotation matrix. 0 sin α cos α For the second rotation in the XOZ plane: G = WG (5) 1 1, where G is the matrix after the twice rotations; cos β 0 sin β W 1 = is the rotation matrix. sin β 0 cos β α and β are obtained through the rotation angle and meet the conditions: cosγ = cosα + cos β. Figure 2 shows the rotated 3D trajectories and the corresponding CDF eigenvalue, which verifies the rotation invariance. This method greatly simplifies the classification of the HMM model and improves the speed of computation. 3.3 Hand gesture recognition As we know, HMM is a very mature match time-varying data technology (Elmezain et al., 2008). It is also used in pattern recognition and human machine interaction (Zhang et al., 2010). The use of the HMM can be divided into two stages: training and classification. Observation vector sequence and coding sequence of symbols in hand gesture recognition system are referred as the observation sequence and denoted by O, which is a random sequence. HMM of -state (denoted by Sl, S2,, S) is represented by three parameters λ = { π, AB, }, where: = [ π π π ] π 1, 2,, is used to describe the state probability distribution of the observed sequence O in π = P q = S, i = 1,2,,, π = 1. time T i, i.e., ( ) i= 1 = { aij (, i j = 1,2,, ) } i i i i A is the state transition probability matrix, in which aij = P( qi = Sj qi 1 = Si, qi 2 = Sk, ), for the Markov sequence, a = P q = S q = S, it satisfies: a = 1. ( 1 ) ij i j i i i= 1 ij B is the state output probability matrix. It is each state observed probability space distribution of random variables or random vectors. B = { bj ( k), j = 1,2,, }, b k = 1, k = 1, 2,, M. { ( ) k = 1 j } Evaluation, training, decoding are three main problems in HMM, and can be solved by forward and backward algorithm, namely the Baum Welch algorithm and the Viterbi algorithm, respectively. This paper takes left-right model. In theory, with the increased number of states, the training model can describe the trajectory fast and accurately. However, this imposes a high calculation cost. The real-time requirement will be affected. So the selection of state number needs a compromise. In this paper, the state number is chosen from 3 to 10 according to the trajectory complexity. 4 Analysis of experimental results 4.1 Robustness verification To verify the robustness, we do the following experiments. As shown in Figure 3, (a) are the hand gesture recognition images in the normal lighting condition, (b) are the hand gesture recognition images in the low lighting condition, (c) are the hand gesture recognition images in the complex background condition. In this paper, 150 experiments were performed in three different conditions, and Table 1 shows gesture recognition rate for different conditions. Through experimental analysis, the average recognition rate is 97.8% under the normal lighting condition, the

5 108 X.D. Xu et al. average recognition rate is 97% under the low lighting condition, the average recognition rate is 96.9% under the complex background condition. The difference of three average recognition rates is <1%. It is clear that the recognition rate of hand gestures is relatively stable in different lighting conditions. The proposed dynamic gesture recognition system is robust to illumination changes and complex background. Figure 2 (a) and (b) show the 3D trajectory of the number 3 rotating and its CDF eigenvalue; (c) and (d) show the 3D trajectory of the number 3 rotating and its CDF eigenvalue, (e) and (f) show the 3D trajectory of the number 3 rotating and its CDF eigenvalue Figure 3 Gesture detection comparison images in different light and complex background condition (see online version for colours)

6 3D hand gesture tracking and recognition for controlling an intelligent wheelchair 109 Table 1 Command The gesture recognition rates under different conditions Light condition Low light condition Complex background condition Correct Accuracy Correct Accuracy Correct Accuracy number rate (%) number rate (%) number rate (%) Forward Backward Turn left Turn right Stop Hand gesture spotting verification To validate the hand gestures spotting method, the end point is got by calculating the observation sequence, which is used to match the gesture model with the threshold model. Figure 4 shows the likelihood diagram change over time. In the diagram, the abscissa is the time axis, the coordinate origin is starting point of the gesture, and the vertical axis represents the value of the likelihood. 4.3 Effectiveness verification In this research, a number of experiments are conducted to verify the system effectiveness. We capture the hand gesture trajectory of the Arabic numerals hand gesture trajectories are collected from seven individuals, in which 420 samples are used for training and 280 samples are used for testing. Figure 5 shows a comparison of the recognition rates from the proposed method and the traditional method (Elmezain et al., 2010), respectively, in which the traditional location coordinates are used as hand trajectory features. Through the calculation, the average recognition rate of our method can reach 97.5%. The comparative results show that our method has a high recognition rate and high reliability, which can be widely applied to the field of human computer interaction. 4.4 Intelligent wheelchair control system In this paper, hand gesture trajectory recognition is used to control an intelligent wheelchair. Five different hand gesture trajectories are selected to control five basic movement of intelligent wheelchair, namely forward, backward, turn left, turn right and stop. The system uses a Kinect sensor for extracting images. The software tools used in this research include VS2008, OpenI and OpenCV. In the experiment, the users put their hands in front of the Kinect vision sensor. When the palm position of hand is detected, the gesture trajectory is abstracted. The recognition results are obtained by using the HMM Viterbi algorithm. Figure 6 shows the system flowchart. The intelligent wheelchair has an ARM9 controller embedded, plus the DSP-based driver module and the sensor module. The upper computer system communicates with the sensor module via a RS232 serial bus and the drive module. Figure 7 show the desired trajectory for the intelligent wheelchair to travel in experiments. The entire experimental environment is 6 m long and 5 m wide. The wheelchair moves from point A to point B. The representatives of the two rectangles are obstacles, and the arrows in the middle represent the planned route and direction of wheelchair movement. To apply same rules for both control methods, the testing site is designed to have adequate space for the user to manoeuvre. Each task is designed to deploy full control functions including turning left, and then turning right, following strait lines to dock into a narrow place. For the hand gesture-based control method, the wheelchair speed is tuned to an optimised condition for each subject so that she or he can have the best performance with the new control method during testing. The acceleration, angular velocity and linear velocity are adjustable from zero to the maximum wheelchair speed. Figure 4 Likelihood diagram changes over time (see online version for colours)

7 110 X.D. Xu et al. Figure 5 A comparison of recognition rates (see online version for colours) Figure 6 The system flowchart (see online version for colours) Figure 7 The trajectory used in experiments (see online version for colours)

8 3D hand gesture tracking and recognition for controlling an intelligent wheelchair 111 Figure 8 shows experimental results for five subjects, i.e., from Subject A to Subject E. Each diagram shows 10 recorded time durations corresponding to 10 tasks in which five tasks are finished by the joystick control and five by the hand gesture control. As can be seen, for all the five subjects completing the same tasks, the joystick control needs less time than the hand gesture-based control. Figure 9 shows a comparison of experimental results between the joystick control and the hand gesture control in terms of time duration for five subjects, where J represents the joystick control, H represents the hand gesture control. From the average time duration and variance analysis in Figure 9, we can see that the smoothness and stability of the hand gesture control is similar to the joystick control. The proposed method can effectively control a real wheelchair motion to move in the specified routes and achieve real time performance. Figure 8 A comparison of two systems for five subjects: (a) Subject A; (b) Subject b; (c) Subject C; (d) Subject D and (e) Subject E (see online version for colours) (continued) (b) 5 Conclusions and future work In this paper, after the depth data are collected by a 3D Kinect vision sensor, the hand analysis module in OpenI is deployed so that the palm position of a human hand can be accurately located. This system is not vulnerable to the illumination change and complex background. As the improved CDF feature extraction method is introduced in the extraction process of 3D hand gesture trajectory features, real-time hand gesture recognition is achieved by using HMM algorithm and the effectiveness of the system is improved. The experimental results show that improved CDF has rotation, scaling, translation invariance. The robustness of the recognition of the gesture trajectory is improved, and the intelligent wheelchair is controlled through the hand gesture in real-time. Since dynamic hand gestures contain both movements and form changes, our future work will take hand form changes into account so that the recognition performance of the proposed method can be further improved. (c) Figure 8 A comparison of two systems for five subjects: (a) Subject A; (b) Subject b; (c) Subject C; (d) Subject D and (e) Subject E (see online version for colours) (d) (a) (e)

9 112 X.D. Xu et al. Figure 9 A comparison of experimental results (see online version for colours) Acknowledgements This work was supported by ational atural Science Foundation of China (o ), International Science & Technology Cooperation Program of China (o. 2010DFA12160) and the Program for Science and Technology Research Project of Chongqing (o. CSTC, 2010AA2055). We thank the Engineering Research & Development Centre of Information Accessibility at Chongqing University of Posts and Telecommunications, Chongqing, China and School of Computer Science & Electronic Engineering, University of Essex, UK, for their continuous support. Out thanks also go to Liu Jiao, Li Dingjie and Xu Xinli for their participation in experiments. References Bashir, F.I., Khokhar, A.A. and Schonfeld, D. (2006) View-invariant motion trajectory-based activity classification and recognition, Journal of Multimedia Systems, Vol. 12, o. 1, pp Chen, Y.M. and Zhang, Y.H. (2009) Research on human-robot interaction technique based on hand gesture recognition, Journal of Robot, Vol. 31, o. 4, pp Elmezain, M., Al-Hamadi, A. and Michaelis, B. (2008) Real-time capable system for hand gesture recognition using hidden Markov models in stereo colour image sequences, International Journal of Computer Graphics, Visualization & Computer Vision, Vol. 16, o. 1, pp Elmezain, M., Al-Hamadi, A., Sadek, S. and Michaelis, B. (2010) Robust methods for hand gesture spotting and recognition using hidden Markov models and conditional random fields, Proceedings of IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), December, pp Hahn, M., Krüger, L., Wöhler, C. and Kummert, F. (2009) 3D action recognition in an industrial environment, Human Centred Robot Systems, Springer, Berlin, Heidelberg, pp Holte, M.T. (2008) View invariant gesture recognition using 3Dmotion primitives, Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 4 April, pp Kao, M.C. and Li, T.S. (2010) Design and implementation of interaction system between humanoid robot and human hand gesture, Proceedings of IEEE, SICE Annual Conference, pp Lee, H. and Kim, J. (1999) An HMM-based threshold model approach for gesture recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 21, o. 10, pp Pang, Y.Y., Ismail,.A. and Gilbert, P.S. (2010) A real time vision-based hand gesture interaction, Mathematical Analytical Modelling and Computer Simulation: IEEE, pp Silanon, K. and Suvonvorn,. (2011) Hand motion analysis for Thai alphabet recognition using HMM, International Journal of Information and Electronics Engineering, Vol. 1, o. 1, pp Tang, C., Zhou, C.L., Pan, W., Xie, L.D. and Hu, H. (2013) Fusing mixed visual features for human action recognition, International Journal of Modelling, Identification and Control, Vol. 19, o. 1, pp Van den Bergh, M. and Van Gool, L. (2011) Combining RGB and ToF cameras for real-time 3D hand gesture interaction, Proceeding of IEEE International Conf. on Application of Computer Vision, pp Wang, X., Xia, M., Cai, H., Gao, Y. and Cattani, C. (2012) Hidden-Markov-models-based dynamic hand gesture recognition, Journal of Mathematical Problems in Engineering, Vol. 2012, pp Yuan, R., Cheng, J., Li, P., Chen, G., Xie, C. and Xie, Q. (2010) View invariant hand gesture recognition using 3D trajectory, Proceedings of the 8th World Congress on Intelligent Control and Automation (WCICA), pp Zhang J., Liu, Z.J. and Zhou H. (2010) Real-time gait classification based on fuzzy associative memory, International Journal of Modelling, Identification and Control, Vol. 10, os. 3 4, pp Zhang, S.L. and Zhang, B. (2010) Using HMM to sign language video retrieval, 2nd International Conference on Computational Intelligence and atural Computing (CIC), pp Zhang, S.M., Hu, H. and Zhou, H.Y. (2008) An interactive Internet-based system for tracking upper limb motion in home-based rehabilitation, Journal of Medical & Biological Engineering & Computing, Vol. 46, o. 3, pp Zhang, Y., Liu, J., Luo, Y. and Hu, H. (2012) Hybrid lip shape feature extraction and recognition for human-machine interaction, International Journal of Modelling, Identification & Control, Vol. 18, o. 3, pp Zhang, J.J., Lin, H. and Zhao, M.G. (2009) A fast algorithm for hand gesture recognition using relief. The 6th International Conference on Fuzzy Systems and Knowledge Discovery, pp.8 12.