A Method of Sign Language Gesture Recognition Based on Contour Feature

Transcription

1 Proceedings o the World Congress on Engineering and Computer Science 014 Vol I WCECS 014, -4 October, 014, San Francisco, USA A Method o Sign Language Gesture Recognition Based on Contour Feature Jingzhong WANG, Meng LI. Abstract The paper gives a method or sign language gesture recognition using contour eatures. First, the sign language gesture images pre-processing such as background image dierence is given. Then the method o extracting edge eatures and contour characteristics is introduced. Finally, the match method o weighted eatures with the sample image eature values is given. Aiming to sign language gestures, create sample library and validate the method with plenty o experiments. The results show that this method can carry out classiication eiciently o 30 sign language gestures, and its recognition rate reaches 93%. Index Terms sign language gesture recognition, YCrCb skin color detection, contour detection, eature extraction s I. INTRODUCTION ign language is the language used by the dea, and Chinese inger language is one kind o it. Chinese "phonetic alphabet inger chart" has a total o 30 dierent inger styles, representing all Pinyin. Sign language recognition research can make to improve the lives o the dea. With the extensive application o computer technology and matures o image processing and pattern recognition technology, the human-computer interaction has become a need o development. Thereore gesture recognition, an integrated multi-product o multi-disciplinary ield, has ar-reaching signiicance and application prospects [1] []. The edge inormation is one o the most important eatures o the image, so that many scholars have come to research gesture recognition based on the edge detection. Some gesture recognition methods use the image edge inormation directly as a eature [3][4][5], but when the edge detection caused ineective by light and other eects, the recognition rate will be aected. Thereore rely on the image edge inormation as the eature is not high stability. Combined with edge detection and image geometry moment method [6], geometric moment eatures have stability, but the process will lose the original edge inormation. Gestures contour and convex deects [7] as a eature, it can be applied gesture is Manuscript received July 8, 014; revised July 8, 014. This work was supported in part by China the National Natural Science Funds under Grant No and Beijing innovation team building promotion plan under Grant ID HT J.-Z. W. Author was with Inner Mongolia University, Inner Mongolia, China. He is now working in the College o Inormation Engineering, North China University o Technology, Beijing, China ( jingzhongwang@163.com). M. L. Author is now with the College o Inormation Engineering, North China University o Technology, Beijing, zip code China. (Corresponding author, phone : ; @ qq.com). ISBN: ISSN: (Print); ISSN: (Online) limited and requires a gesture parallel to the screen. This paper researches 30 static gestures recognition, using image processing and recognition methods, while applying the image edges and contours inormation and calculating multi-eature matching gesture recognition. II. FEATURE SELECTION A. Edge Detection According to a Gaussian noise model, the desired eect o edge detection operators need to meet three conditions: the irst is a low probability o ailure, that is a real edge points lost as little as possible, while avoiding as much as possible to retain the pseudo-edge points; second is the high position accuracy, edge detection as close to real edge; third, each point o edge is a single pixel edge. For this edge detection should ollow three guidelines: SNR Criterion The larger SNR is, the higher the quality o the extracted edge has. SNR is deined as ollow: w G( - x) h( x) dx SNR (1) w d hxdx ( ) = ò ò In the ormula, Gx ( ) represents an edge unction; hx ( ) represents the impulse response o the ilter when the width is w ; d represents the variance o the Gaussian noise. Positioning Accuracy Standards Edge positioning accuracy o L is deined as ollows: w ' ' G (- x) h( x) dx L w ' d h ( x) dx = ò ò In the ormula, G ' ( x) represents a derivative o Gx ( ); h ' ( x) represents a derivative o hx ( ); the larger L is, the higher positioning accuracy is. Single-Edge Response Criteria The average distance o detection operator impulse ' response derivative's zero crossing point D( ) should be met: () WCECS 014

2 Proceedings o the World Congress on Engineering and Computer Science 014 Vol I WCECS 014, -4 October, 014, San Francisco, USA In the ormula, D ' ( ) ì + ' h ( x) dx ü ïò- ï = p í + ý '' ï h ( x) dx ï îò- þ 1 h '' ( x) is the second derivative o hx ( ). Canny designed an edge detection algorithm. Its concrete steps are as ollows: --First, use convolution o D Gaussian ilter template to smooth the image. --Second, calculate gradient's magnitude and direction using the dierential operator. --Third, take non-maxima suppression o Gradient amplitude. That is to say, take traversal images and combine with the gradient direction and gradient strength to remove the pseudo edge points around the real edges. --Fourth, use dual-threshold algorithm to detect and connecting edges. By setting the dual-threshold makes edge coherent [8]. Reerences [9] proposed American Sign Language (ASL) use o calculating the image edge gradient identiication algorithm, which compared ive kind o edge detection methods including the canny operator, Prewitt operator, Sobel operator, Robert operator and the Laplace operator, proved canny operator is best suited or edge detection and gesture recognition. B. Features o Contour Geometric characteristics o the image are principal basis or image analysis and recognition. Recognition method based on geometric eatures can reduce the complexity o the algorithm, save storage space needed and time consuming. Due to ease o access to, storing and processing o binary images, a convenient extraction o the image geometric eatures usually requires image binary process [10]. Obtain the contour inormation rom binary images, in turn, can calculate the centroid, perimeter, area, and other geometric characteristics, eventually combined with a plurality o eatures to eectively improve the recognition rate. Center o Mass [11] For a continuous image o a D, ( x, y ) ³ 0, p+q-order moments m pq is deined as ollows: p q =òò x y (x, y)dxdy (4) mpq - - In the ormula, p and q are non-negative integers, or discretization digital image, above changes into ollows: N N p q m =å å i j (i, j) (5) pq j= 1 i= 1 When the order p + q 3 pq is a real physical meaning: 0-order moments ( m 00 ), that is the mass o the object; 1-order moments ( m ) represents the centroid o the object; -order moments ( m ) represents the (3) radius o rotation; 3-order moments ( m ) describes the object orientation and slope. These moments have the invariant eatures in translation, rotation and scaling, so it has a wide range o applications on the image eature extraction and recognition. Obtain the centroid coordinate o images by 0-order moments and 1-order moments: i = m / m, j = m / m (6) c c Area Area, a measure o the image total size, is a basic eature to describe the size o the image area, and only has relationship with the edge region, no relationship with the internal gradation value. For binary image, the easiest statistical method o area is measure o the number o pixels in the target proile. Assuming that target area ( x, y ) s size is M N, and a logical 1 indicates target portion, logic 0 indicates the background portion, the area is calculated as ollows: M N Area =å å ( x, y) (7) x= 1 y= 1 III. GESTURE RECOGNITION SYSTEM The process ramework o gesture recognition system implementation is showed in Figure 1: Fig. 1. Gesture recognition system lowchart As shown in Figure 1, the design o sign language recognition system is divided into two parts. For the irst part: create the gesture samples, images preprocess and eature extraction to the sample set, and inally construct a gesture eature library. The second part is built on the basis o the irst part, or a single gesture image. Gesture image through the same image preprocessing and eature extraction, and then make the gesture eature compare with gesture eature library to obtain the recognition result. A. Images pre-processing The purpose o gesture image preprocessing is to detect hand position o the image rom camera, thus can obtain gesture area o the image as a target area. Its lowchart has been shown in Figure : Fig.. Gesture recognition system ramework lowchart. ISBN: ISSN: (Print); ISSN: (Online) WCECS 014

3 Proceedings o the World Congress on Engineering and Computer Science 014 Vol I WCECS 014, -4 October, 014, San Francisco, USA --First, skin color detection or both the background image which was saved beore and the collected gesture image by YCrCb color space image segmentation. For color detection, a common approach is changing the image rom RGB color space to HSV color space [1] or YUV color space [13]. Due to the HSV color space segmentation is very sensitive to light, it is better to choose YUV color space to detection segmentation. --Second, ater skin segmentation, process the images by image dierence and binarization, thereby removing the inluence o background. As a result, obtain the binary image o skin color region except or the background. --Thirdorphological dilation. That is to eliminate noise and smooth image. --Fourthaximum contour detection. According to the outline inormation rom obtained binary image, compare with the contained area o each contour shape and elect the outline with the largest area, then record its boundary inormation. --Fith, obtain the target area. According to the boundary conditions on the step o recording, remove the area outside the boundary and obtain the hand area. Both original and binary gesture images need to area division. According to the direction o gesture, remove the arm region by using improved method o projection. According to the position o inlection points, remove the upper arm. Then ind the wrist position using the horizontal projection or vertical projection to cut the lower arm. --Sixth, image reduction normalized. Due to the size o each hand region is not same, each o the resulting images should be normalized to a bitmap which size is 64 * 64. That is can reduce the amount o image data and redundancy; enhance its smoothness and sharpness or urther processing. --Seventh, gray processing. Ater the image pre-processing, obtain the gray-level image which size is 64 * 64, as shown in Figure 3: a b c d e g h i j k l m n o p q r s t u v w x y z zh ch sh ng Fig. 3. Gesture images ater image preprocessing Ater the process o image preprocessing, obtain a gray-level image and a binary hand-type region image which size is 64 * 64. The gray-level images is used to edge detection o eature extraction, contour extraction o binary image is used to calculate the geometric characteristics. B. Feature Extraction Features o this paper researches are based on the contour and the edge inormation. Calculate the eigenvalues o sample images to generate the corresponding eature library that edge eature library (denoted as SL1) and geometric eatures o the library (denoted as SL), calculated as ollows: Based on Edge Inormation Process the gray-level image ater image preprocessing by canny edge detection and 3 * 3 area template dilation twice, making the boundary point spread to the neighborhood. The expanded images as an edge template, which is an edge eature library SL1 o gesture eature library. As shown in Figure 4 is a partial edge eature database: a b c d e g h i j k l m n o p q r s t u v w x y z zh ch sh ng Fig. 4. Partial gesture images o edge eature library Have a logical AND operation between images ater edge detection in test set (denoted as (, ) test xy ) and edge template images in eature library (denoted as sl1 ( xy, ) ): gxy (, ) t est ( xy, ) sl1( xy,) = Ù (8) In the ormula, gxy (, ) represents two-dimensional unction o the image result o the operation. The intersection o two sub-images can be obtained by the logical AND operation, calculate the coincide proportion between image o test set and the edge o the sample set image as a eature (denoted as 1 ), the ormula is deined as ollows: = åå y= 0 x= åå y= 0 x= 0 gxy (, ) test ( xy, ) Based on contour inormation --First, obtain the contour o binary image ater image preprocessing. --Second, calculate the gesture images' geometrical eature set (denoted as S), including: region area contained by contour(denoted as area); the centroid coordinate o the image(denoted as pc(x, y)); the longest distance rom the centroid to point on the contour as radius (denoted as R); the distance rom centroid o the contour to the horizontal secant(denoted as d1); the distance rom centroid point to the let contour point at the same height o the centroid (denoted as d); the longitudinal distance rom the centroid to the point at above section contour (denoted as d3). Above steps to sign O as an example, shown in Figure 5: Fig. 5. The process o geometric eature extraction (9) ISBN: ISSN: (Print); ISSN: (Online) WCECS 014

4 Proceedings o the World Congress on Engineering and Computer Science 014 Vol I WCECS 014, -4 October, 014, San Francisco, USA Following the steps above or all the samples, write the eigenvalues o S in a XML ile as the geometric eature database (denoted as SL). According to the test set, calculate the geometric eigenvalues using the same way and obtain a test eature set (denoted as Stest). Then calculate the eigenvalue distance rom the test image to each image o SL one by one, inally obtain the overall spatial distance as the eature (denoted as ), the equation is deined as ollows: = _1 3 _4 _5 _ 6 (10) Among the equation, d( pctest - pcsl ) _1 = 64 (11) ( xtest - xsl ) + ( ytest - ysl ) = 64 areatest - area 64*64 rtest - rsl _3 64 d1test -d1sl _4 64 dtest dsl _5 - d1 d1 6 sl = (1) = (13) = (14) = (15) test d3test -d3 64 sl sl = (16) Each eigenvalue has been normalized to a value within the range 0 to 1 in the process o calculation, so that reduce the impact rate. IV. THE EXPERIMENTAL RESULTS According to 30 alphabet gestures or the gesture recognition system, create three sets o gesture library (each set o images come rom dierent people's hand respectively). There are 450 gesture images totally in three set o gestures library. Each gesture has recorded ive letters in a set which size is 640*480 as a bitmap. Among them, choose one set as a sample set and two sets as test set. According to this algorithm, take a lot o experiments to test their impact on the recognition rate or 30 Pinyin gestures. A. Gesture Detection In order to complete the detection o the hand-type region, using a series o image processing algorithm mentioned above. Interception hand-type region accurately is the basis or eature extraction; especially those have complex background or arm in the gesture images. Thus, or the gesture recognition based on vision, accurate detection o a gesture needs to be solved or a long-term. The mainly result images o pre-processing as shown in Figure 6: C. Gesture recognition match In gesture recognition system, test images calculate the eigenvalues with the corresponding value o eature library by the method introduced beore. Then, obtain N (the number o sample in the eature library) matching values (denoted as match). The ormula is deined as ollows: match = max( w 1 1+ w ) (17) In the ormula above, w1 and w are the weight values o the edge eature and geometrical eature respectively, and need to meet w1 + w = 1. Due to the bigger edge coincidence rate ( 1 ) is and the smaller the spatial distance ( ) to the sample is, the closer to the sample the test image is. The parameters should be met w1 > 0, w < 0. In N matching values obtained, the maximum value o the corresponding sample number is result o gesture recognition. Fig.6. The process o detection o the hand-type region ISBN: ISSN: (Print); ISSN: (Online) WCECS 014

5 Proceedings o the World Congress on Engineering and Computer Science 014 Vol I WCECS 014, -4 October, 014, San Francisco, USA B. Logic operation o the edge As or using the image edge as a part o eatures, either to curve itting or to calculate the Euclidean distance or the edge, have lost its original inormation. This paper gives a method to have a logical AND operation between gesture edge image and edge image o sample, and use the edge inormation directly without any loss. Then get the intersect sub-image o two edge images, can intuitive relect the coincidence o two gestures. Using the edge image ater eature extraction o the test image as the original image and the image o eature library SL1 as the background image,then have the AND operation between original image and background image respectively. Using gestures U as an example, the ollowing calculation results shown in Figure 7: Gesture U Template o Gesture U Operational Result TABLE I RECOGNITION RESULTS OF TEST SETS ω 1 ω recognition rate o test set 1(%) recognition rate o test set (%) recognition rate o all test set(% ) As can be seen rom the Table 1, there is the best recognition rate o integral test sets reaches 93.33% when ω 1 = 0.3 and ω = For the test set, the recognition rate is 94.0%. Using geometric moments and edge histogram as eatures to recognize 30 letters gesture, its recognition rate is 90% rom the reerences 6 mentioned. Compared with it, the paper's method improves the gesture recognition rate eectively. Template o Gesture B Template o Gesture J Operational Result Operational Result Fig.7. The calculation process o edge coincidence rate Ater calculation, the edge's coincidence rate o test image (gesture U) and the template edge corresponding gesture G in eature library o sample is 70.%. As to the gesture B which is similar to the gesture U, its edge's coincidence rate is 60.8%. As to another gesture such as gesture J, it likes ar rom gesture U, and its edge's coincidence rate is just 6%. Thus, the operator can do its work or the gesture recognition. For the images which edges are similar to each otheray appear result in similar coincidence rate, so using this method to recognize while calculating the geometric eatures o images to aid identiication. C. Gesture Recognition From the experiments, in the process o the calculating eatures, 1 is larger than almost an order o magnitude. In order to balance the impact o the value gap between two characteristic values, the corresponding weight in the equation (17) need to meet w1 < w. Dierent weight values has great impact on the recognition rate, thereore, need to determine the optimal weights determined by experiment. Under the dierent weights, test set gesture recognition rate as shown in Table 1: V. CONCLUSION This paper researches the 30 inger gestures recognition o Chinese phonetic alphabet, and gives an algorithm based on multi-eature matching, which through a series o processing methods o digital image processing or image eature extraction and classiication. Experiments show that this method makes ull use o the inormation o the edge and outline o image, and thus provides an easy and eicient way to the gesture recognition, and it also has stability o the panning and zooming image. The light and excessive rotation gestures has a certain inluence on the recognition, just because o this the ollow-up research will ocus on the adaptive gesture recognition system. REFERENCES [1] W. Gao, J. Guo, B.-K. Zeng. The direction and status o the sign language research [J]. Application o Electronic technique, 00, 11: 000. [] X. Wu, Q. Zhang, Y.-X. Xu. An Overview o Hand Gestures Recognition [J]. Electronic Science and Technology, 013, 6(6): [3] L.-G. Zhang, J.-Q. Wu, W. Gao, et al. Hand Gesture Recognition Based on Hausdor Distance [J]. Journal o Image and Graphics (A), 00, 7(11). [4] Q. Yang, M. Wang. Hand Gesture Recognition Algorithm Based on Euclidean Distance [J]. Microcomputer Inormation, 007, 9: [5] L.-J. Sun, L.-C. Zhang. Static Sign Language Recognition Based on Edge Gradient Direction Histogram [J]. MICROEL ECTRONICS & COMPUTER, 010, 7(3). [6] Y.-Q. He, Y. Ge, L.-Q. Wang. Gesture Recognition Algorithm Based on Invariant Moment and Edge Detection [J]. Computer Engineering, 005, 31(15): [7] H.-L. Weng, Y.-W. Zhan. Vision-Based Hand Gesture Recognition with Multiple Cues [J]. Computer Engineering & Science, 01, 34(): [8] Canny J. A computational approach to edge detection [J]. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 1986 (6): [9] Thepade S D, Kulkarni G, Narkhede A, et al. Sign Language Recognition using Color Means o Gradient Slope Magnitude Edge Images [J]. Intelligent Systems and Signal Processing (ISSP), 013, ISBN: ISSN: (Print); ISSN: (Online) WCECS 014

6 Proceedings o the World Congress on Engineering and Computer Science 014 Vol I WCECS 014, -4 October, 014, San Francisco, USA [10] W.-X. Feng, M. Tang, B. He. Visual C++ digital image pattern recognition technology [M]. China Machine Press, 010. [11] WANG Bing, Zhi Qinchuan, Zhang Zhongxuan, et al. Computation o Center o Mass or Gray Level Image Based on Dierential Moments Factor[J]. JOURNAL OF COMPUTER-AIDED DESIGN & COMPUTER GRAPHICS, 004, 16(10): [1] Zhang X N, Jiang J, Liang Z H, et al. Skin color enhancement based on avorite skin color in HSV color space [J]. Consumer Electronics, IEEE Transactions on, 010, 56(3): [13] D. Qiu. Research about skin color detection based on HSV and YCrCb color space [J]. Computer Programming Skills & Maintenance, 01 (10): ISBN: ISSN: (Print); ISSN: (Online) WCECS 014