Turkish Fingerspelling Recognition System Using Axis of Least Inertia Based Fast Alignment

Transcription

1 Turkish Figerspellig ecogitio System Usig Axis of Least Iertia Based Fast Aligmet Oğuz Altu, Sogül Albayrak, Ali Ekici, ad Behzat Bükü Yıldız Techical Uiversity, Computer Egieerig Departmet, Yıldız, İstabul, Türkiye {oguz, Abstract. Figerspellig is used i sig laguage to spell out ames of people ad places for which there is o sig or for which the sig is ot kow. I this work we describe a Turkish figerspellig recogitio system that recogizes all 9 letters of the Turkish alphabet. A sigle represetative frame is extracted from the sig video, sice that frame is eough for recogitio purposes of the letters metioed. Processig a sigle frame, istead of the whole video, icreases speed cosiderably. The ski regios i the represetative frame are extracted by color segmetatio i YCrCb space before clearig oise regios by morphological opeig. A ovel fast aligmet method that uses the agle of orietatio betwee the axis of least iertia ad y axis is applied to had regios. This method compesates small orietatio differeces but icreases big oes. This is desirable whe differetiatig the figerspellig sigs, some of which are close i shape but differet i orietatio. Also the use of miimum boudig square is advised, which helps i resizig without breakig the aligmet. Biary values of this miimum boudig square are directly used as feature values, ad that allowed experimetig with differet classificatio schemes. Features like mea radial distace ad circularity are also used for icreasig success rate. Classifiers like knn, SVM, Naïve Bayes, ad BF Network are experimeted with, ad 1NN ad SVM are foud to be the best two of them. The video database was created by 3 differet sigers, a set of 90 traiig videos, ad a separate set of 174 testig videos are used i experimets. The best classifiers 1NN ad SVM achieved a success rate of 99.43% ad 98.83% respectively. Keywords: Turkish Figerspellig ecogitio, Fast Aligmet, Agle of orietatio, Axis of Least Iertia, Miimum Boudig Square, Classificatio. 1 Itroductio Sig Laguage is a visual meas of commuicatio usig gestures, facial expressio, ad body laguage. Sig Laguage is used maily by deaf people ad people with hearig difficulties. There are two major types of commuicatio i sig laguage. The first oe has word based sig vocabulary, where gestures, facial expressio, ad body laguage are used for the most commo words. The secod oe has letter based vocabulary, ad is called figerspellig, which is a method of spellig words usig had movemets. Figerspellig is used i sig laguage to spell out ames of people A. Sattar ad B.H. Kag (Eds.): AI 006, LNAI 4304, pp , 006. Spriger-Verlag Berli Heidelberg 006

2 474 O. Altu et al. ad places for which there is o sig ad ca also be used to spell words for sigs that the siger does ot kow the sig for, or to clarify a sig that is ot kow by the perso readig the siger [1]. Sig laguages develop specific to their commuities ad are ot uiversal. For example, ASL (America Sig Laguage) is totally differet from British Sig Laguage eve though both coutries speak Eglish []. I the automatic sig laguage recogitio, there are successful systems for America Sig Laguage (SL) [3], Australia SL [4], ad Chiese SL [5]. Previous approaches to word level sig recogitio rely heavily o statistical models such as Hidde Markov Models (HMMs). A real-time ASL recogitio system developed by Starer ad Petlad [3] used colored gloves to track ad idetify left ad right hads. They extracted global features that represet positios, agle of axis of least iertia, ad eccetricity of the boudig ellipse of two hads. Usig a HMM recogizer with a kow grammar, they achieved a 99.% accuracy at the word level for 99 test sequeces. For TSL (Turkish Sig Laguage) Haberdar ad Albayrak [6], developed a TSL recogitio system from video usig HMMs for trajectories of hads. The system achieved a word accuracy of 95.7% by cocetratig oly o the global features of the geerated sigs. The developed system is the first comprehesive study o TSL ad recogizes 50 isolated sigs. This study is improved with local features ad performs perso depedet recogitio of 17 isolated sigs i two stages with a accuracy of 93.31% [7]. For figerspellig recogitio, most successful approaches are based o istrumeted gloves, which provide iformatio about figer positios. Lamar ad Bhuiyat [8] achieved letter recogitio rates ragig from 70% to 93%, usig colored gloves ad eural etworks. More recetly, ebollar et al. [9] used a more sophisticated glove to classify 1 out of 6 letters with 100% accuracy. The worst case, letter U, achieved 78% accuracy. Isaacs ad Foo [10] developed a two layer feed-forward eural etwork that recogizes the 4 static letters i the America Sig Laguage (ASL) alphabet usig still iput images. ASL figerspellig recogitio system is with 99.9% accuracy with a SN as low as. Feris, Turk ad others [11] used a multi-flash camera with flashes strategically positioed to cast shadows alog depth discotiuities i the scee, allowig efficiet ad accurate had shape extractio. Altu et al. [1] icreased the effect of figers i Turkish figerspellig shapes by thick edge detectio ad correlatio with pealizatio. They achieved 99% accuracy out of 03 sig videos of 9 the Turkish alphabet letters. I this work, we have developed a siger idepedet figerspellig recogitio system for Turkish Sig Laguage (TSL). The represetative frames are extracted from sig videos. Had objects i these frames are segmeted out by ski color i YCrCb space. These had objects are aliged usig the ovel agle of orietatio based fast aligmet method. The, the aliged object is moved ito the ceter of a miimum boudig square, ad resized. The biary values of the miimum boudig square are used as classificatio features, i additio to the biary object features like mea radial distace ad circularity. We experimeted with differet classificatio schemes ad reported their success rate. The remaiig of this paper is orgaized as follows: I Sectio we describe the represetative frame extractio, our fast aligmet method, ad extractio of

3 Turkish Figerspellig ecogitio System 475 additioal object features. Sectio 3 covers the video database we use. We listed the classificatio schemes we used i Sectio 4. Fially, coclusios ad future work are addressed i Sectio 5. Feature Extractio Cotrary to Turkish Sig Laguage word sigs, Turkish figerspellig sigs, because of their static structure, ca be discrimiated by shape aloe by use of a represetative frame. To take advatage of this ad to icrease processig speed, these represetative frames are extracted ad used for recogitio. Fig. 1 shows represetative frames for all 9 Turkish Alphabet letters. Fig. 1. epresetative frames for all 9 letters i Turkish Alphabet I each represetative frame, had regios are determied by ski color. From the biary images that show had ad backgroud pixels, the regios we are iterested i are extracted, aliged ad resized. I additio to aliged biary pixel values, biary object features are extracted to support maximum correlatio based matchig. Each process is summarized below:.1 epresetative Frame Extractio I a Turkish figerspellig video, represetative frames are the oes with least had movemet. Hece, the frame whose distace to its successor is miimum ca be chose as a represetative frame. Distace betwee successive frames f ad f+1 is give by the sum of the city block distace betwee correspodig pixels:

4 476 O. Altu et al. (a) (b) Fig.. (a) Origial image ad detected ski regios after pixel classificatio, (b) result of the morphological opeig. (a) (b) (c) (d) Fig. 3. (a)-(b) The 'C' sig by two differet sigers. (c)-(d) The 'U' sig by two differet sigers. D f f f f = Δ + ΔG + ΔB, (1) where f iterates over frames, iterates over pixels,, G, B are the compoets of the pixel color, f f +1 f Δ =, f f +1 f ΔG = G G, ad f f +1 f ΔB = B B.. Ski Detectio by Color For ski detectio, YCrCb color-space has bee foud to be superior to other color spaces such as GB ad HSV [13]. Hece we covert the pixel values of images from GB color space to YCrCb usig (). I order to decrease oise, each of the Y, Cr ad Cb compoets of the image are smoothed with the D Gaussia filter give by (3), where σ is its stadard deviatio Y = G B, = B Y, = Y () 1 x + y F( x, = exp( ), (3) πσ σ Chai ad Bouzerdom [14] report that pixels that belog to the ski regio have similar Cr ad Cb values, ad give a distributio of the pixel color i Cr-Cb plae. Cosequetly, we classified a pixel as ski if the Y, Cr, Cb values of it falls iside the rages 135 < Cr < 180, 85 < Cb < 135 ad Y > 80 (Fig..a). After clearig small ski colored regios by morphological opeig (Fig..b), ski detectio is completed. C r C b

5 Turkish Figerspellig ecogitio System Fast Aligmet for Maximum Correlatio Based Template Matchig Template matchig is very sesitive to size ad orietatio chages. A scheme that ca compesate size ad orietatio chages is eeded. Elimiatig orietatio iformatio totally is ot appropriate however, as depicted i Fig. 3. Fig. 3a-b show two 'C' sigs that we must be able to match each other, so we must compesate the small orietatio differece. I Fig. 3c-d we see two 'U' sigs that we eed to differetiate from 'C' sigs. 'U' sigs ad 'C' sigs are quite similar to each other i shape, luckily orietatio is a major differetiator. As a result we eed a scheme that ot oly ca compesate small orietatio differeces of had regios, but also is resposive to large oes. Fig. 4. Axis of least secod momet ad the agle of orietatio We propose a fast aligmet method that works by makig the agle of orietatio (θ ) zero. Agle of orietatio, give by (4), is the agle betwee y axis ad the axis of least momet (show i Fig. 4). M11 θ = arcta M 0 M where (I(x, = 1 for pixels o the object, 0 otherwise), M ) = xi( x, y, ad = 0 x y x M y I( x, 0 y. 0 (4) M = xyi( x, 11 x y, (a) (b) (c) (d) (e) Fig. 5. Stages of fast aligmet. (a) Origial frame. (b) Detected ski regios. (c) egio of Iterest (OI). (d) otated OI. (e) esized boudig square with the object i the ceter.

6 478 O. Altu et al. Let's defie boudig square as the smallest square that ca completely eclose all the pixels of the object. After puttig images i the ceter of a boudig square, ad tha resizig the boudig square to a fixed, smaller resolutio, the fast aligmet process eds (Fig. 5)..4 Additioal Biary Object Features Istead of usig oly pixel values i the boudig square, additioal biary object features are extracted to support decisio process. These features iclude area, ceter of area, perimeter [15], agle of orietatio (defied above), ad circularity (defied as perimeter /area). I additio, mea radial distace μ is extracted: 1 μ = ( x, y ) ( x, (5) N where iterates over all pixels, N is the umber of pixels, ( x, y ) is the ceter of area, ( x, y ) is the coordiate of the th pixel, ad. deotes the Euclidea distace betwee two pixels. Aother feature is the stadard deviatio of radial distace σ, defied as 1 1 ( [( x, y ) ( x, μ ] ) N σ =. (6) As the last biary object feature, a secod circularity measure C is computed by C = μ σ. (7) To summarize, 9 biary object features are added to the 30x30 biary values of the miimum boudary square. 3 Video Database The traiig ad test videos are acquired by a Philips PCVC840K CCD webcam. The capture resolutio is set to 30x40 with 15 frames per secod (fps). While programmig is doe i C++, the Itel OpeCV library routies are used for video capturig ad some of the image processig tasks. We have developed a Turkish Sig Laguage figerspellig recogitio system for the 9 static letters i Turkish Alphabet. The traiig set was created usig three differet sigers. For traiig, they siged a total of 10 times for each letter, which sums up to 90 traiig videos. For testig, they siged a total of 6 times for each letter, which sums up to 174 test videos. Table 1 gives a summary of the distributio of the trai ad test video umbers for each siger. Notice that traiig ad test sets are totally separated.

7 Turkish Figerspellig ecogitio System 479 Table 1. Distributio of trai ad test video umbers for each siger Siger 1 Siger Siger 3 Total Trai Test Trai Test Trai Test Trai Test A Z Total Table. Success rates of most successful classifiers o figerspellig data Classifier Success ate (%) 1NN [16] SVM [17] adom Forest [18] BF Network [19] Multiomial Naive Bayes [0] Naive Bayes [1] J48 [] Classificatio Compariso A set of differet classificatio algorithms are applied to the features extracted as explaied i Sectio ad obtaied results are sorted accordig to their success rates. These classificatio results are summarized i Table. The most successful classifiers are oe earest eighbor (1NN) ad support vector machie (SVM). These methods classified more tha 98% successfully, as see i Table. The biggest problem is i the classificatio of the letter 'S', which is cofused by ' '. A secod problem letter was 'G', which is cofused by ' '. The cofused characters are very similar to each other i shape, as see i Fig. 6. Fig. 6. Two difficult cases where our methods may fail. Left to right: 'S' ad ' ', 'G' ad ' '. 5 Coclusios ad Future Work A Turkish figerspellig recogitio system is tested ad foud to have more tha 99% accuracy. Testig ad traiig sets is created by multiple sigers, as a cosequece the developed system is siger idepedet. Accuracy is the result of the

8 480 O. Altu et al. fast aligmet process we applied. This process brigs objects with similar orietatio ito same aligmet, while brigig objects with high orietatio differece ito differet aligmet. This is a desired result, because for figerspellig recogitio, shapes that belog to differet letters ca be very similar, ad the orietatio ca be the mai differetiator. After the aligmet, to resize without breakig the aligmet, the object is moved ito the ceter of a miimum boudig square. The biary values i miimum boudig square are used as the features. I additio, we used biary object features like circularity ad mea radial distace, which helped icreasig success rate. Our method is robust to the problem of occlusio of the hads, because the fast aligmet method allows us to process a esemble of oe or more coected compoets i the same way. The system is fast due to represetig the sig video by a sigle frame, the speed of fast aligmet process, ad resizig the boudig square to a smaller resolutio. The amout of resizig ca be arraged for differet applicatios. Sice we used biary pixel values as ordiary features, we are able to experimet with differet classificatio algorithms, amogst which are knn, SVM, BF Network, Naïve Bayes, adom Forest, ad J48 tree. The 1NN ad SVM give the best success rates, 99.43% ad 98.85% respectively. Not all letters i Turkish alphabet are represetable by oe sigle frame, ' ' beig a example. The sig of letter ivolves some movemet that differetiates it from 'S'. I fact, this letter is the oe that preveted us achievig a 100% success rate. Still, represetig the whole sig by oe sigle frame is acceptable sice this work is actually a step towards makig a full blow Turkish Sig Laguage recogitio system that ca also recogize word sigs. That system will icorporate ot oly shape but also the movemet, ad the research o it is cotiuig. The importace of successful segmetatio of the ski ad backgroud regios ca ot be overstated. I this work we assumed that there is o ski colored backgroud regios ad used color based segmetatio i YCrCb space. The systems' success depeds o that assumptio, ad research o better ski segmetatio is ivaluable. The fast aligmet ad classificatio schemes preseted would work equally well i the existece of a face i the frame, eve though i this study we used oly had regios whe creatig the figerspellig video database. Although we demostrated the fast aligmet method i the cotext of had shape recogitio, it is equally applicable to other problems where shape recogitio is required, for example to the problem of shape retrieval. efereces Starer, T., Weaver, J., Petlad, A.: eal-time America sig laguage recogitio usig desk ad wearable computer based video. Ieee Trasactios o Patter Aalysis ad Machie Itelligece 0 (1998) Holde, E.J., Lee, G., Owes,.: Australia sig laguage recogitio. Machie Visio ad Applicatios 16 (005) 31-30

9 Turkish Figerspellig ecogitio System Gao, W., Fag, G.L., Zhao, D.B., Che, Y.Q.: A Chiese sig laguage recogitio system based o SOFM/SN/HMM. Patter ecogitio 37 (004) Haberdar, H., Albayrak, S.: eal Time Isolated Turkish Sig Laguage ecogitio From Video Usig Hidde Markov Models With Global Features. Lecture Notes i Computer Sciece LNCS 3733 (005) Haberdar, H., Albayrak, S.: Visio Based eal Time Isolated Turkish Sig Laguage ecogitio. Iteratioal Symposium o Methodologies for Itelliget Systems, Bari, Italy (006) 8. Lamar, M., Bhuiyat, M.: Had Alphabet ecogitio Usig Morphological PCA ad Neural Networks. Iteratioal Joit Coferece o Neural Networks, Washigto, USA (1999) ebollar, J., Lidema,., Kyriakopoulos, N.: A Multi-Class Patter ecogitio System for Practical Figerspellig Traslatio. Iteratioal Coferece o Multimodel Iterfaces, Pittsburgh, USA (00) 10. Isaacs, J., Foo, S.: Had Pose Estimatio for America Sig Laguage ecogitio. Thirty-Sixth Southeaster Symposium o, IEEE System Theory (004) Feris,., Turk, M., askar,., Ta, K.: Exploitig Depth Discotiuities for Visio- Based Figerspellig ecogitio. 004 IEEE Computer Society Coferece o Computer Visio ad Patter ecogitio Workshops(CVPW'04) (004) 1. Altu, O., Albayrak, S., Ekici, A., Bükü, B.: Icreasig the Effect of Figers i Figerspellig Had Shapes by Thick Edge Detectio ad Correlatio with Pealizatio. PSIVT 006 (006) 13. Sazoov, V., Vezhevetsi, V., Adreeva, A.: A survey o pixel vased ski color detectio techiques. Graphico-003 (003) Chai, D., Bouzerdom, A.: A Bayesia Approach To Ski Colour Classificatio. TENCON- 000 (000) 15. Umbaugh, S.E.: Computer Visio ad Image Processig: A Practical Approach Usig CVIPtools. Pretice Hall (1998) 16. Aha, D.W., Kibler, D., Albert, M.K.: Istace-Based Learig Algorithms. Machie Learig 6 (1991) Keerthi, S.S., Shevade, S.K., Bhattacharyya, C., Murthy, K..K.: Improvemets to Platt's SMO algorithm for SVM classifier desig. Neural Computatio 13 (001) Breima, L.: adom forests. Machie Learig 45 (001) Fritzke, B.: Fast Learig with Icremetal bf Networks. Neural Processig Letters 1 (1994) McCallum, A., Nigam, K.: A Compariso of Evet Models for Naive Bayes Text Classificatio. AAAI-98, Workshop o Learig for Text Categorizatio (1998) 1. Joh, G.H., Lagley, P.: Estimatig Cotiuous Distributios i Bayesia Classifiers. Eleveth Coferece o Ucertaity i Artificial Itelligece. Morga Kaufma, Sa Mateo (1995) Quila,.: C4.5: Programs for Machie Learig. Morga Kaufma Publishers, Sa Mateo, CA (1993)