Visual Interface System by Character Handwriting Gestures in the Air

Transcription

1 9th IEEE Iteratioal Symposium o Robot ad Huma Iteractive Commuicatio Pricipe di Piemote - Viareggio, Italy, Sept. -5, Visual Iterface System by Character Hadwritig Gestures i the Air Toshio Asao ad Sachio Hoda Hiroshima Istitute of Techology, Japa tasao@cc.it-hiroshima.ac.jp Abstract A visual iterface system that recogizes hadwritig of Japaese katakaa characters i the air has bee developed. Characters writte i a sigle stroke have both o-strokes ad off-strokes. Thus, the shapes of the had-gesture characters are differet from the shapes of characters writte o paper. It is difficult to trace the shapes of characters i air because the writer caot see the trajectories. I this study, a light emissio diode (LED) pe ad a TV camera are used to capture the LED light trajectory, ad the movemets of the light are coverted ito directio codes correctig the slat of the hadwritig character. The codes are ormalized to data items to elimiate the effect of writig speed. The directio codes are compared with model data i which the directio codes of 6 Japaese characters are defied. Next, the system has expaded to a multi-camera system. Two of four cameras are selected ad the 3-D positios of the gesture trajectories are calculated by the stereo method, ad the positio data are coverted ito frot view data. I the experimets, we attaied a recogitio rate of 9.9% for the sigle-camera system. The multi-camera system has the advatage that it ca recogize gestures regardless of the origi directios of the gestures. The system also has the ability to recogize the directios of the gesture commads with a accuracy of 9º. Key words: Gesture recogitio, Had gesture, Huma iterface, Image processig, Character recogitio I. INTRODUCTION Computers are used i a variety of situatios, but huma iterface methods are curretly limited to iput devices such as keyboards ad mice. If computers with TV cameras ca uderstad huma gestures, the people will be able to easily iterface with computers ad robots [-3]. Gesture ad shape recogitio is very popular i the field of image recogitio [-6]. Gestures are doe by movig, for example, the head, the body, or the hads. Amog these, hads ca express iformatio most easily. Shakig ad rotatig are usually used i had gestures [7,8], but if a system ca recogize a word, the may kids of iformatio ca be set by gestures to computers ad robots. Research i which alphabets are used has bee reported [9]. Shapes of some alphabets were deformed for recogitio. These alphabets are ot useful i Japa. Rather, katakaa or hiragaa would be more useful for Japaese. Japaese laguage uses three kids of characters: hiragaa, katakaa ad kaji. The katakaa characters are based o Japaese prouciatio ad ca express all Fig. マエ Character had gesture recogitio system for robot cotrol. Japaese words ad seteces. There are 6 basic katakaa characters ad some cosoats for certai characters. The difficult poits of had gesture recogitio of letters are as follows. ) Gestures used to write i the air usig a sigle stroke cosist of o-strokes ad off-strokes of the characters. ) The trajectories are ivisible to the writer, so the letter loci are very bad. The gesture trajectory cotais o-strokes ad off-strokes. The o-strokes cotaied i the gesture trajectory are the strokes that would be writte o paper, ad the off-strokes are the strokes that do ot represet writig o paper but rather the retur lies of pe movemet. I this paper, two visual iterface systems, a sigle-camera system ad a multi-camera system, that eable recogitio of characters writte i a sigle stroke usig a light emissio diode (LED) pe are preseted. A computer recogizes the hadwritte katakaa letters usig their directio codes. This system ca be used as a iterface tool for computers ad a istructio tool for robots. Fig. shows a example as a robot cotrol commad. The character commad meas Go forward. II. DETECTION METHOD The developed system uses a TV camera to detect movemets of a LED pe. A butto o the pe is pressed whe writig a character. Fig. shows the positios of the LED light for each frame. The x, y positio data of poits of light are coverted ito directio codes represetig directio vectors (see Fig. 3). The movemet distace ρ i ad agle θ i are calculated for each light positio by the followig equatios: ( x x ) + ( y y ) ρ i = i i i i, () ad y i yi θ = arcta, i xi xi () //$6. IEEE 6

2 where (x i,y i ) is the coordiate pair of the i-th light positio P i, ρ i is the distace betwee P i (x i,y i ) ad P i- (x i-,y i- ) ad θ i is the agle of the vector from (x i-,y i- ) to (x i,y i ). The agles are each coverted ito oe of the eight directio codes that are show i Fig.. The agle rages of the directio codes have differet widths. Agle rages for vertical ad horizotal directios are arrow (3 ) ad those for slat directios are wide (6 ). This is because the agles used for slat strokes are more varied tha those for vertical ad horizotal strokes. Fig. 5 shows a katakaa ad its directio codes. O-strokes (,,,) ad off-strokes (7,5) are mixed. X[pixel] Fig. 6 Rejectio of tremblig poits. Y[pi xel] III. RECOGNITION ALGORITHM A. Rejectio of Tremblig Poits Sometimes, at the begiig of writig, light poits are crowded ad the movemet distace ρ is very small. A example is show i Fig. 6. We call these poits tremblig poits, ad it is reasoable to reject these poits from the locus of the character. If the movemet distace ρ is smaller tha pixels, the directio code data for the light poit is rejected. Fig. Pi-( xi-, yi- ) Detectio of LED pe tip ρi Fig θi Directio vector. 8 Fig Pi ( xi, yi ) 5 3 Eight directio codes. 5-5 B. Code Normalizatio by Speed The writig speeds are differet for each perso. These speeds also chage durig the strokes of a character. However, the image capture speed is costat (3 frames/secod). Therefore, to remove the ifluece of writig speed, the directio codes are ormalized to data accordig to the displacemet ρ i. The origial directio code series h j ( j= to max) is ormalized based o the displacemets to the code series H k (k = to ) havig directio codes. The H k is determied by the followig equatio. H k = h j, ρ k ρsum j j ρi ρi < k ρsum ρsum for j=, for where ρ sum is the total legth of strokes, give by j max, (3) 7 max ρ sum = ρ i 5 Fig. 5 Example: the directio codes Fig. 7 shows x, y positios of light poits of a character for each frame. The origial directio codes ad the speeds for each frame are show i Fig. 8. It is clear that the speed of the LED pe chages drastically, from to 6 pixels/frame, over the course of drawig the character. Fig. 9 shows the result of the speed ormalizatio. The ormalizatio is especially effective at the begiig of a character. 63

3 X[pixel] Test data Model data Y[pixel] Fig. 7 x, y positios of light poits for each frame of a character. Fig. Partial Matchig of Katakaa コ (ko). Lightig poit speed [pixels/frame] Directio Code Speed Directio code 3 Frame Number 8 6 Fig. 8 Speed of the LED pe ad the origial directio codes. Origial Normalized 6 8 Normarized Number Fig. 9 Coversio to speed-ormalized directio codes. Directio code Equatio (5) defies the mea of the errors. The fial decisio is doe by fidig the smallest E from amog all characters. δd = d test -d model ; () if δd > the δe = 8-δd ; Ne ( δei ) E =, (5) Ne where Ne is the umber of elimiated data. D. Oblique Characters It is difficult for a perso lyig i bed to write a character so that the character is upright with respect to a camera. I this case, the perso writes a straight lie that is horizotal from his or her perspective. The lie is writte right to left, as show i Fig.. The system calculates the deviatio σ of agle θ usig equatio (6), ad if the deviatio σ is smaller tha threshold σ t, the system chages the defiitio of the horizotal lie agle to θ ave defied by equatio (7). The rage for each directio code show i Fig. is rotated by agle θ ave. Fig. shows the ew horizotal lie X for the oblique character. C. Partial Matchig The ormalized directio codes of a had gesture are compared with directio codes of model data. Fig. shows the matchig of a gesture with model data. Some mismatchig occurs at directio code chagig poits. This is because the percetages of each stroke differ slightly accordig to the writer. Directio code data deviatig by +/- directios codes from the model, idicated by gray i Fig., are elimiated from the matchig-test data. The differeces of directio codes (δd) betwee test dataad model data are calculated by equatio (). Because the directio codes correspod to a circle, as show i Fig., the maximum possible differece is. That is, if the differece of the directio code umbers is 7, the real differece is. N σ = ( θ i θ ave ), N (6) where N θi θ ave =. N (7) θn θn- θ θ Fig. Detectio of horizotal agle θ ave. 6

4 Y X θave camera camera Fig. Oblique character ad the ew horizotal lie X. LED Pe IV. MULTI-CAMERA SYSTEM I the multi-camera system, a perso performs a character hadwritig gesture i the air, ad four TV cameras are used to capture the data of the LED light poits. Because the sigle-camera system used oly oe camera, the operator was required to write the letters i view of the camera. However, it is sometimes difficult to fid a camera for a character hadwritig gesture. Therefore, we developed a system that does ot deped o the gesture directio. Fig. 3 shows a example of the proposed multi-camera system. A. Selectio of Two Cameras Four TV cameras detect the movemets of a LED pe. The surface of the LED is modified to icrease light diffusio. A butto o the LED pe is pressed whe writig a character. Two cameras are selected for use i calculatig the 3-D positios of the light poits. This selectio is performed by comparig the areas of the light-poit trajectories obtaied from the four cameras. The trajectory area is defied as the area of the smallest rectagle that ecloses the trajectory. First, the camera havig the largest trajectory area is selected. The, the trajectory areas of the two adjacet cameras are compared, ad the camera havig the larger area is selected. Fig. shows the images obtaied by the four cameras. I Fig., cameras ad are selected for the calculatio of the 3-D positios of the light poits. B. Detectio of 3-D Positio The 3-D positios of light poits are calculated usig two cameras. The coordiates of the light poits for the left-had camera are represeted as (x c,y c ) ad those for the right-had camera are represeted as (x p,y p ). The 3-D coordiates (X,Y,Z) of the light poits are calculated as follows: camer Fig. 3 Gesture recogitio system with multi-camera. camera 3 camera Fig. Trajectories of light poits for four cameras. C C3xc C C3xc C3 C33xc X x c C, (8) C C3yc C C3yc C3 C33yc Y = yc C P P3xp P P3xp P3 P33xp Z xp P where C ij ad P ij are camera parameter values of right-had ad left-had cameras, respectively, which are obtaied from camera calibratio procedure []. C. Coversio to the Frot View The character drawig plae is calculated from the 3-D coordiates (X,Y,Z) of the light poits. The plae is expressed by the followig equatio: ax + by + cz + d = (9) where a, b, ad c are the compoets of a vector ormal to the plae. The values of a, b, ad c are obtaied by solvig the followig equatio: X i i = X iyi i = X i Z i i = X iyi i = Yi i = Yi Z i i = X i Z i X i i = a i = = Yi Z i b Yi i = c i = Z i Z i i = i = () where is the total umber of light poits. To obtai a frot view of the character, the drawig plae is rotated so that the ormal vector is matched to the Z-axis. The rotatio agles (θ,φ) are calculated by the followig equatios: X-axis rotatio agle: b θ = si, () b + c ad Y-axis rotatio agle: a φ = si. () a + b 65

5 Y φ X Z θ Fig. 5 Rotatio agles. B. Sigle Camera System Results ad Discussios ) Test ( Perso) Had gestures were repeated times for 6 characters ad cosoat diacritics. The model data are prepared for the specific perso. The average recogitio success rate was 9.9%. The success rate was 79.% without the partial matchig algorithm. This is the improvemet of 3.5%. ) Test (5 People) The 5 test subjects (all male studets) tried katakaa had gestures. They drew each character times. The directio codes of the model data were previously determied from stadard katakaa fots. The recogitio success rate depeded very much o the text subject. The highest success rate amog the 5 subjects was 9.6%, the lowest rate was 7.%, ad the mea rate was 8.7%. Table shows the success rates for each character. The diacritic gestures ad were recogized at high rates. It was difficult to distiguish betwee some characters due to similarities i their directio code patters. TABLE RECOGNITION RATES FOR THE 5 SUBJECTS (%) ア 55.3 カ 96.7 サ 88 タ 96 ナ 98 イ 7.7 キ 88 シ 89.3 チ 9 ニ 8 ウ 68 ク 93.3 ス 86.7 ツ 96.7 ヌ 7 エ 8.7 ケ 66 セ 6 テ 83.3 ネ 98.7 オ 86.7 コ 96 ソ 63.3 ト 87.3 ノ Fig. 6 Coversio to the frot view. Fig. 5 shows the rotatio agles. Fig. 6 shows the procedure for the coversio to the frot view. Ⅴ. EXPERIMENTS A. Experimetal Set-Up ) Sigle Camera System A CCD TV camera (/ ich CCD, f = 6 mm) picks-up the LED movemets. A image processor IP7 (Hitachi) is used for image capture ad recogitio. The image size is 5 x pixels. ) Multi-Camera System Fig. 7 shows the experimetal set-up. The distace betwee the cameras is 3 m. The camera calibratio was performed usig eight corer poits of a 9 cm cubic agle iro jig. Fig Operator m 7 Experimetal set-up for the multi-camera system. ハ 96.7 マ 6.7 ヤ 6.7 ラ 76.7 ワ 75.3 ヒ 9 ミ リ 8 ヲ 86 フ 97.3 ム 68 ユ 89.3 ル 96.7 ン 78 ヘ メ レ 9.7 ホ 76 モ 78 ヨ 9.7 ロ C. Multi-Camera System Results ad Discussios ) Recogitio Rate The recogitio rates of the sigle-camera system ad the multi-camera system were compared. The writig agles were varied i icremets of 5º, ad each character recogitio rate was measured. Gestures for each character were performed times. Figs. 8 ad 9 show the success rates for each character. I the sigle-camera system, a high recogitio rate was obtaied oly i frot of the camera. O the other had, high recogitio rates were obtaied for ay directios usig the multi-camera system. The processig time of the image processor was. sec/character. Fig. shows the overall success rate for four successive characters. ) Directio Distictio Accuracy The directios i which the characters were writte ca be detected by the Y-axis rotatio agle φ give i Equatio (). The stadard deviatio of the agle φ was tested for each directio ad was foud to be.5. Therefore, this system has the ability to distiguish directio i icremets of 9 (σ). This meas that directios ca be distiguished. 66

6 Oe applicatio of this techique could be for remote cotrol for home appliaces. There are may ifrared remote cotrollers i use, for example, for TV sets, air coditioers, hi-fi sets, ad video-recorders. The umber of cotrollers could be reduced by adoptig gesture recogitio systems. V. CONCLUSION Fig. 8 Recogitio rates for character ヒ (hi). We have developed two visual iterface systems, a sigle-camera system ad a multi-camera system, that eable recogitio of katakaa characters writte i a sigle stroke usig a LED pe. TV cameras are used to capture the light trajectories ad these trajectories are coverted ito directio codes. The directio codes are ormalized to remove variatios i writig speed. The code series are compared with model data i which the katakaa directio codes are defied. I the experimet, we have achieved a mea recogitio rate of 9.9% for oe subject ad 8.7% overall for the 5 test subjects. It is possible to exted these systems to iclude hiragaa characters, simple kaji characters, ad Eglish alphabets. The multi-camera system ca recogize hadwritig character gestures from ay agles. The 3-D positios of the light poits are calculated, ad the data are trasformed ito data represetig the frot view positio. The success rate for recogitio was 9%. The multi-camera system also has the ability to distiguish directio i icremets of 9º. Future work will be to improve the reliability of the systems ad to develop a robot system that ca recogize had character gestures ad ca talk with a perso. REFERENCES Fig. 9 Recogitio rates for character ロ (ro). Fig. Recogitio rates for characters ヒロシマ (hiroshima). [] Tomoari Sooda ad Yoichi Muraoka, "A Letter Iput System of Hadwritig Gesture," IEICE Tras. Vol.J86-D-Ⅱ, No.7, pp.5-5, 3 (i Japaese). [] Masaji Katagiri ad Toshiaki Sugimura, "Persoal Autheticatio by Sigatures i the Air with a Video Camera," Techical Report of IEICE, PRMU-3, pp.9-6, (i Japaese). [3] Hee-Deok Yag, A-Yeo Park, Seog-Wha Lee, "Gesture Spottig ad Recogitio for Huma-Robot Iteractio," IEEE Tras.ROBOTICS, Vol.3, No., pp.56-7, 7 [] Thierry Artieres, Saparith Marukatat, Patrick Galliari,"Olie Hadwritte Shape Recogitio Usig Segmetal Hidde Markov Models," IEEE Tras. PAMI, Vol.9, No., pp.5-7, 7 [5] Jae-Wa Park, Jog-Gu Kim, Dog-Mi Kim, Mi-Yeog Chog, Chil-Woo Lee, "Learig Touch Gestures usig HMM o Tabletop Display, " Proc. 6th Korea-Japa Joit Workshop o Frotiers of Computer Visio, pp.7-3, [6] Hyeo-Kyu Lee ad Ji H. Kim, "A HMM-Based Threshold Model Approach for Gesture Recogitio," IEEE Tras. PAMI, Vol., No., pp , 999. [7] Kota IRIE, Kazuori UMEDA,"Detectio of Wavig Hads from Images - Applicatio of FFT to time series of itesity values -",Proc. 3rd Chia-Japa Symposium o Mechatroics, pp.79-83,. [8] Kota Irie, Naohiro Wakamura, Kazuori Umeda, "Costructio of a Itelliget Room Based o Gesture Recogitio," Proc. IEEE/RSJ It. Cof. o Itelliget Robots ad Systems, pp.93-98,. [9] Ho-Sub Yoo, Jug Soh, Byug-Woo Mi, ad Hyu Seug Yag, " Recogitio of Alphabetical Had Gestures Usig Hidde Markov Model," IEICE Tras. Vol.E8-A, No.7, pp ,999. [] Tim Morris, Computer visio ad image processig, Palgrave macmilla, p.6,. 67