Real Time Hand Gesture Recognition Including Hand Segmentation and Tracking

Transcription

1 Real Tme Hand Gesture Recognton Includng Hand Segmentaton and Tracng Thomas Coogan 1, George Awad 2, Junwe Han 3, Alstar Sutherland 4 Dubln Cty Unversty, Dubln 9, Ireland { 1 tcoogan, 2 gawad, 3 jhan, 4 Abstract. In ths paper we present a system that performs automatc gesture recognton. The system conssts of two man components: () A unfed technque for segmentaton and tracng of face and hands usng a sn detecton algorthm along wth handlng occluson between sn objects to eep trac of the status of the occluded parts. Ths s realzed by combnng 3 useful features, namely, color, moton and poston. () A statc and dynamc gesture recognton system. Statc gesture recognton s acheved usng a robust hand shape classfcaton, based on PCA subspaces, that s nvarant to scale along wth small translaton and rotaton transformatons. Combnng hand shape classfcaton wth poston nformaton and usng DHMMs allows us to accomplsh dynamc gesture recognton. 1 Introducton The prmary goal of any automated gesture recognton system s to create an nterface that s natural for humans to operate or communcate wth a computerzed devce. Furthermore we am to develop our system wthout usng data gloves or colored gloves. Such a system could be used n, vrtual realty, robot manpulaton or gamng. In fact gesture recognton could be used to mprove the ntutveness of any Human-Computer Interacton (HCI). We routnely use hand gestures when communcatng, descrbng and drectng durng our everyday actvtes. Incorporatng gestures wth HCI could be an extremely benefcal development. In recent years varous approaches to gesture recognton have been proposed, Gupta et al [1] presented a method of performng gesture recognton by tracng the sequence of contours of the hand usng localzed contour sequences. Chen et al [2] developed a dynamc gesture recognton system usng Hdden Marov Models (HMMs). Patwardhan et al [3] recently ntroduced a system based on a predctve egentracer to trac the changng appearance of a movng hand. Kadr et al [4] descrbe a technque to recognze sgn language gestures usng a set of dscrete features to descrbe poston of the hands relatve to each other, poston of the hands relatve to other body locatons, movement of the hand, shape of the hand. Whle some of these approaches dsplay mpressve results, many explot controlled envronments and a compromse between vocabulary sze and recognton rate.

2 To acheve accurate gesture recognton over a large vocabulary we need to extract nformaton about the hand shape. Accomplshng ths entals detectng the hands, segmentng them, dfferentatng them and classfyng them. Ths requres usng sn detecton technques and handlng occluson between sn objects to eep trac of the status of the occluded parts. We present a unfed system for segmentaton and tracng of the face and hands n a gesture recognton usng a sngle camera. Unle much related wor that uses color gloves [5], we detect sn by combnng 3 useful features: color, moton and poston. These features together, represent the sn color pxels that are more lely to be foreground pxels and are wthn a predcted poston range. Also, unle other wor that avod occlusons entrely by choce of camera angle, sgn vocabulary, or by performng unnatural sgns [6,7], we handle occluson between any of the sn objects usng a Kalman flter based algorthm. Once the hand s segmented classfcaton s requred. In hand shape recognton, transformaton nvarance s ey for successful recognton. We propose a system that s nvarant to small scale, translaton and shape varatons. Ths s acheved by usng a-pror nowledge to create a transformaton subspace for each hand shape. Transformaton subspaces are created by performng Prncpal Component Analyss (PCA) on mages produced usng computer anmaton. We ntroduce our method that enables us to tran ths appearance based method usng computer anmaton mages. Also presented s the ncorporaton of ths hand shape classfer nto a dynamc gesture recognton system. Poston nformaton s combned wth hand shape nformaton to construct a feature vector that s passed to a DHMM for dynamc gesture recognton. The remander of ths paper s organzed as follows: The method used to segment the face and hands s reported n secton 2. The gesture recognton technque ncludng hand shape recognton s descrbed n secton 3. In secton 4 we detal some experments and fnally we offer some conclusons n secton 5. 2 Segmentaton and Tracng In gesture recognton we need to segment and trac three objects of nterest: the face and the two hands. The sn segmentaton module s responsble for segmentaton of sn objects, smlarly the object tracng module s responsble for matchng the resultng sn blobs of the segmentaton component to the prevous frame blobs whle eepng trac of the occluson status of the three objects. In the next sectons we wll explan the detals of these two components. 2.1 Sn Segmentaton In order to robustly detect the sn objects, we combne three useful features: color, moton and poston. Color cue s useful because the sn has a dstnct color that helps to dfferentate t from other colors. The moton cue s useful n dscrmnatng foreground from bacground pxels. Fnally, the predcted poston of objects usng Kalman flter helps to reduce the search space.

3 2.1.1 Color Informaton In order to collect canddate sn pxels, we use two smple classfers. Frst, a general sn model (color range) s appled on small search wndows around the predcted postons of sn objects. As the fxed color range can mss some sn pxels, we propose another color dstance metrc dst(c sn, X j ) to tae advantage of the pror nowledge of the last segmented object. Ths metrc s the Eucldean dstance between the average sn color C sn n the prevously segmented sn object and the current pxel X j n the search wndow at postons and j. Fnally, we normalze the values of the pror nowledge color metrc P col Moton Informaton Fndng the movement nformaton taes two steps. Frstly, moton detecton, then next step, fndng canddate foreground pxels. The frst step examnes the local graylevel changes between successve frames by frame dfferencng: D ( x, y ) = W ( x, y ) W 1 ( x, y ) (1) Where W s the th search wndow and D s the absolute dfference mage. We then normalze D to convert t to probablty values. The second step assgns a probablty value P m ( x, y) for each pxel n the search wndow to represent how lely ths pxel belongs to a sn object. Ths s done by loong bacward to the last segmented sn object bnary mage n the prevous frame search wndow and applyng the followng model on the pxels n : D OBJ 1 P m ( x, y ) = 1 D ( x, y ) D ( x, y ) f OBJ otherwse 1 ( x, y ) 1 (2) In ths way, small values (statonary pxels) n D that were prevously segmented as object pxels wll be assgned hgh probablty values as they represent sn pxels that were not moved, and new bacground pxels wth hgh D wll be assgned small probablty values. So smply, ths model gves hgh probablty values to canddate sn pxels and low values to canddate bacground values Poston Informaton To capture the dynamcs of the sn objects, we assume that the movement s suffcently small between successve frames. Accordngly, a Kalman flter model can be used to descrbe the x and y coordnate of the center of the sn objects wth a state vector S that ndcates the poston and velocty. The model can be descrbed as: S Z + 1 = = S A S + V + G Where A s a constant velocty model, G, V represents the state and measurement nose respectvely and Z s the observaton. Ths model s used to eep trac of the poston of the sn objects and predct the new poston n the next frame. ( 3 ) ( 4 )

4 Gven that the search wndow surrounds the predcted center, we translate a bnary mas of the object from the prevous frame to be centered on the new predcted center. Then the dstance transform s computed between all pxels n the search wndow and pxels of the mas. The nverse of ths dstance values assgns hgh values to pxels that are belongng to or near the mas and low values to far pxels. The dstance values are then converted to probabltes by normalzaton Informaton Combnaton. After collectng the color, moton and poston features, we combne them logcally usng an abstract fuson formula to obtan a bnary decson mage F (x,y): P pos 1 f ( Pcol ( x, y ) > τ ) OR (( P g ( x, y ) == 1) F ( x, y ) = AND ( Pm ( x, y ) > υ ) AND ( P pos ( x, y ) > σ )) 0 otherwse (5) Where P, P, and P s the decson probablty values of the color metrc, col m pos moton, and poston respectvely. Pg s the output of the general sn model, and τ, υ, and σ are thresholds where σ s determned adaptvely by the followng formula: σ = sze (( P m ( x, y ) > υ ) AND sze ( P m ( P pos ( x, y ) > υ ) ( x, y ) 1)) (6) The threshold σ determnes the margn that we are searchng nto around the predcted object poston. In Eq. (6) ths s formulated by fndng the overlappng between the predcted object poston and the foreground pxels above certan threshold value. The other thresholds values are determned emprcally. 2.2 Tracng and Occluson Detecton Occluson Detecton. In ths paper, we propose a Kalman flter based algorthm to detect occluson between any of the face and the two hands. In general, the algorthm uses the Kalman flter model to trac the four corner ponts of the boundng box around the face and two hands. Ths model can predct n the next frame the postons of these four corner ponts. Accordngly, we chec to see f there s any overlap between any of the boundng boxes n the next frame. If there s an overlap, we rase an occluson alarm correspondng to the two boundng boxes that wll overlap. If n the next frame, the number of detected sn objects s less than the current frame objects and an occluson alarm was rased prevously, we conclude that occluson happened. On the other hand, f the number of detected sn objects decreases and no occluson alarms were rased, then one or more sn objects have left the frame Tracng. The tracng process starts by frst constructng search wndows around each of the predcted postons of the traced objects. When two or more objects are occluded,

5 they are treated as one object and one search wndow s constructed around ther poston. Next, connected regons are labeled after removng nosy small regons. Usng the number of detected sn objects and the occluson alarms as dscussed n secton 2.2.1, we mantan a hgh-level understandng of the status of the current frame wth respect to the occluson status. For example, f we detected 1 object and occluson alarm between the face and left hand s rased, then we conclude that the face and left hand are occluded and the rght hand s hdng. Ths technque can be extended to handle all 7 stuatons of occluson status: separate face and two hands, face and 2 hands occluded, separate hand wth face and hand occluded, face and hdng hands, face and hands all occluded. The fnal step n the tracng part s the blob matchng where the prevous frame blobs are matched aganst the new frame blobs usng the nowledge of the hgh-level occlusons status we descrbed above. The matchng s done usng the dstance between the prevous objects centers and the new objects centers. 3 Gesture Recognton 3.1 Hand Shape Recognton To date many approaches have been proposed for hand shape recognton. These nclude: () Shamae [8] ntroduced a PCA based approach. () Just et al [9] ntroduced a hand shape system based on the Modfed Census Transform MCT. () A method to classfy hand postures aganst complex cluttered bacground was proposed by Tresch & von der Malsburg [10] usng elastc graph matchng. (v) Yuan et al [11] developed ther system by determnng a new Actve Shape Model (ASM) ernel based on the shape contours. (v) Chen et al [2] present a method of classfyng the statc hand poses by usng the Fourer Descrptor to characterze the spatal features of the hands boundary. Many of these approaches for hand shape recognton dsplay sub-optmal results due to the hghly deformable nature of the hand. Once agan a compromse s determned between small vocabulary and accurate recognton. Due to the complex temperament of the hand, any hand shape classfer needs to be able to cope wth small rotatons and translaton transformatons. We propose a transformaton subspace technque to combat these ssues. Our proposed method creates the nvarant subspaces from a sampled subset of all possble transformaton mages. These mages are produced systematcally usng the commercally avalable POSER computer anmaton (CA) software and ncludes 3D hand transformaton. Performng PCA on these mages wll generate a subspace that accurately represents the complex transformaton hyper-plane. Usng ths method of a-pror nowledge to construct the subspaces means we can elmnate the process of automatc subspace segmentaton as proposed by [12]. It also allows us to dsmss the need for managng outlers or mssng data n our subspaces [12]. Ths means we can create more accurate transformatons subspaces to what was prevously possble usng the smple PCA method. PCA provdes M orthogonal egenvectors {u1,,um} of the covarance matrx, that correspond to the frst M largest egenvalues, n order to mantan a mnmum

6 energy of the dataset. In order to classfy test mages, a dstance metrc needs to be ntroduced. We project the test mage nto the subspace and fnd the perpendcular dstance of the projected pont to the egenvectors representng the subspace. We now have the bacbone for hand shape recognton system. ISL contans 28 statc fngerspellng gestures. The system s constructed as follows: Tranng - Generatng a transformaton subspace for each hand shape. Testng Project the test mage nto each of the subspaces to fnd the subspace wth the nearest perpendcular dstance. Ths subspace wll be representatve of one partcular hand shape. 3.2 Hand Image Pre-Processng If ths technque s to be worthwhle we need to be able to accurately classfy mages of real hands when we tran wth CA mages. In order to do these we need to neutralze the dfferences between CA hand mages and mages of a human hand. Such a system would nherently be mult-user as t would be traned and tested by dfferent users. We have developed a detaled pre-processng step to counteract ssues such as: sn color, llumnaton varaton, hand sze and dstance from the camera Hand Scalng and Algnment. Once the hand has been dentfed and segmented t should be scaled and algned to ensure the system can deal wth users wth dfferent szed hands and users at dfferng dstances from the camera. We explot a smple but effectve custom of scalng the hand objects so that they occupy a predetermned area n a 32*32 reszed mage. Algnment s accomplshed by repostonng the hand object so that the center of the boundng box les n the center of the mage Sn Color and Illumnaton Varaton Removng sn color and color varance due to llumnaton s essental n an appearance based mult user hand shape recognton system. Frst the hand mage s converted to grayscale, ths reduces the space at whch color can be represented. In order to color normalze each hand mage n grayscale space we have ncorporated a color hstogram equalzaton approach nto our system. Color hstograms are graphs that depct the color dstrbuton of pxels n an mage. Hstogram Equalzaton s the process of redstrbutng the color values n the mage so that the mage hstogram taes a predetermned form. We now from the hand-scalng step that all hand objects are reszed to occupy the same area wthn an mage. Wth ths n mnd, a common hstogram can be defned that can represent all hand mages. It contans a large spe that represents the bacground of the mage; ths s located at the begnnng of the color scale because the bacground pxels are set to 0. Then a gaussan-shaped pulse exsts towards the end of the color scale represents object pxels. Ths postonng s mportant to maxmze the contrast of the normalzed hand mage.

7 3.2.3 Image Flterng We have found that t s useful to apply a smple gaussan flter to an mage before classfcaton. Ths flter can help smooth out nose n an mage. The hand mage s convolved wth a 9*9 gaussan ernel wth a small standard devaton to ensure the flterng doesn t blur mportant nformaton n the mage. 3.3 Dynamc Gesture Recognton Dynamc gesture recognton requres both temporal and spatal recognton of the hand movement and hand shape. We have devsed a smple system usng Hdden Marov Models (HMMs) to recognze dynamc gestures. A HMM s a tool for representng probablty dstrbutons over a sequence of observatons. For our dynamc gesture recognton system the sequence of observatons are feature vectors. These feature vectors consst of two elements, both of whch are postve ntegers. The frst denotes the group to whch the statc hand shape has been classfed. It should be noted that 40 statc hand shape groups are currently used to dstngush the gestures n ths system. The second defnes the poston of the hand n the mage and wll be n the range 1-9. The poston of the hand s classfed by determnng the secton of the mage that the center of the hand les n. Ths center s calculated by fndng the center of the hands boundng box. The mage s dvded nto 9 sectons. These sectons are created by dvng the mage vertcally by drawng two lnes ether sde of the head. The 1st of the horzontal lnes, H1, s located drectly under the head. The 2nd, H2, s placed M pxels below H1, where M s the length of the head object. Usng ths technque poston can be calculated nvarantly to the dstance of the user from the camera. A gesture s then represented as a sequence of tuples, contanng both shape and poston nformaton. A DHMM s traned for each possble gesture usng many dfferent examples. A gesture s classfed onlne, by manually dentfyng ts start and stop pont, then fndng the DHMM wth the hghest probablty for the feature vector of the test sequence. 4 Experments 4.1 Statc Hand Shape Classfcaton In order to classfy real hand mages we create a subspace for each hand shape as n secton 3.1. A subspace for each hand shape s now created by performng PCA on 3969 mages as depcted n equaton 7. 1 Orgn 3 rotatons mage 1 n ptch 49 translat drecton 4 ons 2 9 rotatons 3 rotatons n yaw drecton n roll drecton = 3969 mages. 1 Orgn hand mage that can be defned as beng the perfect orentaton of the hand shape. 2 Orgn hand shapes translated n all drectons usng combnatons of 2, 4 and 6 pxels. 3 9 rotatons are used n the yaw drecton as ths s the drecton that contans most sgnfcant devaton. These rotatons are 3 degrees apart coverng a total ptch of 24 degrees. 4 3 rotatons n the roll drecton, each at 10 degrees coverng a total ptch of 20 degrees. 5 3 rotatons n the roll drecton, each at 10 degrees coverng a total ptch of 20 degrees. 5 3 (7)

8 All test and tranng mages are pre-processed usng the technques descrbed n secton 3.2. We developed a test set n order to test the amount of energy we need to retan n each of these subspaces. Ths test set contaned 560 mages, 20 occurrences of each of the 28 hand shapes been used. All these mages were acqured from one traned user of the system over 4 separate sttngs on 2 dfferent days. In theses mages the assumpton has been made that the user s wearng long sleeves coverng the arms. The frst objectve of our experments was to dentfy the energy retenton value that gves superor recognton. We have found that when 80% of the energy s retaned the lowest error rate s acheved. One explanaton for ths s once we go over 80% the subspaces attempt to retan nformaton that s local to that of the ndvdual user,.e. local characterstcs of the computer anmaton mages. It s mportant to fnd ths balance between retanng as much nformaton as possble wthout ntroducng nose n our subspaces. 80% energy retenton entals eepng of the most sgnfcant egenvectors, dependng on the hand shape. Havng a low number of egenvectors s also mportant to mantan effcency. Table 1. Confuson matrx for the 28 statc handshapes U R T A E Fg. 1. Sample Statc gestures Preservng 80% energy n the subspace we acheve 94.5% recognton accuracy for our test set. The performance accuracy of each ndvdual statc gesture can be observed n Table 1. Ths table exhbts the confuson matrx for the statc gesture

9 recognton vocabulary. Most confuson s caused where gestures are very smlar. The two gestures that gve hghest confuson are U and R. These gestures only dffer slghtly as can be seen n Fg 1. When performng U, the ndex and mddle fnger lay parallel, whle performng R the ndex and mddle fnger are crossed. These dfferences become partcularly mnute once the mages are scaled to 32* Dynamc Gesture Recognton In order to test the accuracy of our dynamc gesture recognton system we have generated a vocabulary of 17 dynamc sngle-handed gestures. Ths lexcon was created to ensure gestures exst that dffer only n ether hand shape or hand poston. 20 samples of each solated gesture were recorded employng 2 dfferent users, of dfferent racal orgns, over 4 dfferent days. The vdeos are captured n an offce envronment wth addtonal lghtng to the front of the user. In our experments the samples are dvded nto test and tranng sets by random samplng. The proporton of test and tranng data was then vared over the recognton experments. A DHMM was traned for each gesture usng the selected tranng data. We then tested the recognton accuracy usng the remanng unseen data. Recognton accuracy was calculated by computng the average performance for dfferent samplng of the tranng data. Table 2 dsplays ths performance for each of the dfferent number of data samples used n tranng. As expected the performance ncreases as the number of tranng samples ncreases. It s also nterestng to note that reasonably hgh classfcaton rates can be acheved usng only one tranng sample for the DHMM. Ths classfcaton has been acheved on a standard PC usng the matlab nterpreter wth non-optmzed code n real tme at 10 frames per second. Table 2. llustrates the performance for each of the dfferent number of data samples used for tranng No. Tranng samples Average Performance Conclusons In ths paper a detaled framewor s presented for accurate real tme gesture recognton. A unfed approach for segmentng and tracng sn color objects has been descrbed. Tracng helps to reduce the search space for segmentaton whle accurate segmentaton helps to accurately enhance the tracng performance. Also accurate segmentaton asssts mprovng hand shape recognton usng the subspace classfer descrbed. A novel approach of tranng a subspace classfer usng mages generated from computer anmaton s also llustrated. Usng mage-processng technques we

10 have shown that accurate recognton s possble for human hands. Combnng ths hand shape nformaton wth the poston nformaton a gesture recognton system was generated. Successful classfcaton was acheved for solated gestures even wth lmted tranng. To mprove performance over a larger lexcon we ntend to ntroduce a more detaled poston gauge, ncrease the ban of allowable hand shapes along wth ntroducng new features such as hand moton. Acnowledgements Ths wor s partly funded by the Irsh Research Councl for Scence Engneerng and Technology, (IRCSET). References 1. Gupta, L., Ma S.: Gesture-Based Interacton and Communcaton: Automated Classfcaton of Gesture Contours, IEEE Transactons on Systems, Man and Cybernetcs Part C: Applcatons and revews, Vol 31, No 1, Chen, F.-S., Fu, C.-m., Huang, C.-L.: Hand Gesture Recognton Usng a Real-tme Tracng Method and Hdden Marov Models, Image and Vson Computng, Vol 21, pages , Patwardhan, K. S. Dutta Roy, S.: Hand gesture modelng and recognton nvolvng changng shapes and trajectores, usng a Predctve EgenTracer, Pattern Recognton, (Artcle n Press), Kadr,T., Bowden,R., Ong,E. J., Zsserman, A.:Mnmal Tranng, Large Lexcon, Unconstraned Sgn Language Recognton, In Proc BMVC 04, Vol 2, pp , Shamae, A., Sutherland, A.: Hand Tracng n Bmanual Movements, Image and Vson Computng, 23, pp , Huang, C.-L., Jeng, S.-H.: A Model-Based Hand Gesture Recognton System, Machne Vson and Applcaton, 12(5), pp , Terrllon, J.-C, Pplr, A., Nwa, Y., Yamamoto, K.: Robust Face Detecton and Japanese Sgn Language Hand Posture Recognton for Human-Computer Interacton n an Intellgent Room, In Proc. Int l Conf. Vson Interface, pp , Shamae, A.: Hand Tracng and Bmanual Movement Understandng, PHD Thess, Dubln Cty Unversty Just, A., Rodrguez, Y., Marcel, S.: Hand Posture Classfcaton and Recognton usng the Modfed Census Transform, n Proc. IEEE Internatonal Conference on Face and Gesture, pp , Tresch, J., Von der Malsburg, C.: Classfcaton of hand postures aganst complex bacgrounds usng elastc graph matchng, Image and Vson Computng Vol 20, pp , Yuan, Y., Barner, K.: An Actve Shape Model Based Tactle Hand Shape Recognton wth Support Vector Machne, n Proc 40th Annual Conf Informaton Scences and systems, Bshoff, H., Leonards, A., Maver, J.: Multple Egenspaces, Pattern Recognton, Vol 35, pp , 2002.