Efficient Generation of Large Amounts of Training Data for Sign Language Recognition: A Semi-automatic Tool

Transcription

1 Efficient Generation of Lare Amounts of Trainin Data for Sin Lanuae Reconition: A Semi-automatic Tool Ruiduo Yan, Sudeep Sarkar,BarbaraLoedin, and Arthur Karshmer University of South Florida, Tampa, FL, 60, USA University of South Florida, Lakeland, FL, 80, USA Abstract. We have developed a video hand sementation tool which can help with eneratin hands round truth from sin lanuae imae sequences. This tool may reatly facilitate research in the area of sin lanuae reconition. n this tool, we offer a semi automatic scheme to assist with the localization of hand pixels, which is important for the purpose of reconition. A candidate hand enerator is applied by usin the mean shift imae sementation alorithm and a reedy seeds rowin alorithm. After a number of hand candidates is enerated, the user can reduce the candidates by simple mouse clicks. The tool also provides a hand trackin function for faster processin and a face detection function for roundtruthin non manual sinals. n addition, we provided a twopasses roundtruthin scheme unlike other tools that only does one-pass. Our first pass processin is automatic and does not need user interaction. The experiment results demonstrate that based on the first pass s result, one can roundtruth 0,000+ frames of sin lanuae within 8 hours. ntroduction With the rowin capacity of computer vision technoloy, there is more interest bein directed towards the automatic reconition of sin lanuae based solely on imae sequences or videos. We refer thereaderto[]forareviewofthe applications of related vision technoloies. Briefly, if an accurate and reliable method can be developed to reconize and translate from sin sentences to spoken or written lanuaes, this will help alleviate communication difficulties that Deaf people experience daily. This method can be incorporated into an assistive device carried by the Deaf person or available within the computer systems of stores, banks and other places where a Deaf person needs to communicate with non-siners. n a vision-based sin lanuae reconition system, enerally a model will be trained on either a phoneme, sin or sentence level. This requires accurate feature extractions from the imae frames. For example, one may need to extract the facial expression, hand position and hand shape parameters. However, the automatic hand sementation is a challenin problem and unwanted noise will be enerated for real world imae sequences; hence the accuracy of the model K. Miesenberer et al. (Eds.): CCHP 006, LNCS 406, pp , 006. c Spriner-Verla Berlin Heidelber 006

2 66 R. Yan et al. can be decreased. For example, an automatic scheme for sementin the hand can be based on hand color model [], stereo depth [], colored loves [4] or motion [5] information. Nevertheless, a hand color model is not sufficient for real world imaes where a lare variation of lihtin condition exists. Alternatively usin depth imae is slow and will fail when the hand moves to the same plane as the body. n addition, colored loves miht help durin trainin but it still chanes the appearance of hand and makes the siner feel unnatural. And finally, althouh usin motion information is usually robust, it cannot accommodate subtle hand shape chanes and a movin backround. Due to these reasons, a hand roundtruthin step is often taken with an annotation tool before we start the trainin. Sin lanuae trainin databases enerally consist of many video shots and frames, which makes it very hard to roundtruth quickly usin eneral imae processin tool (e.. Photoshop) frame by frame. Even with a vision based video annotation tool, this task could be labor intensive and time-consumin. For example, considerin a small trainin database that has 00 sins, each sin has 5 trainin samples and the averae lenth of one trainin sample is 50 frames. There are a total of 5x00x50=5000 frames. Suppose we were usin a eneral imae processin tool which can enerate roundtruth for one frame in 5 min. The time cost to roundtruth the whole database will be 44 days. n this paper, we offer a semi automatic scheme to assist with the localization of hand pixels, which is important for the purpose of reconition. A candidate hand enerator is applied by usin the mean shift imae sementation alorithm and a reedy seeds rowin alorithm. After a number of hand candidates is enerated, the user can reduce the candidates by simple mouse clicks. n addition, we provided a two-passes roundtruthin scheme unlike other tool that only do onepass. Our first pass processin is automatic and does not need user interaction. We are aware of many other video annotation tools. However, most focus on scene sementation or key frame detection (e.. BM EVA[6], ESP Game[7]). Some of them combine local feature extraction and temporal trackin toether. For example, the VPER annotation tool proposed by Pfund et.al [8] provides imae sementation, temporal sementation and event annotation toether; ViPer- GT proposed by Doermann et.al [9] can detect multiple objects and track them usin boundin boxes automatically, Marcoteui et.al proposed VOGUE [0], where a number of imae and video sementation techniques are incorporated for object annotation purpose. All of these tools are standalone applications providin semi automatic roundtruthin function with user-friendly interface. Our annotation tool is a side-product of our vision-based American Sin Lanuae(ASL) reconition system. t also provides a semi-automatic scheme for efficient round-truthin. However, its main purpose is to sement the hand pixels frame by frame. nstead of usin the eneral sementation and trackin alorithm as in the existin tools, we advocate a candidate hand enerator approach which is more reliable durin hand shape chane and hand crossin face situation. Unlike the existin tools where only one-pass is conducted, we offer a two-pass scheme for faster processin, where the first pass enerates candidate hands automatically.

3 Efficient Generation of Lare Amounts of Trainin Data 67 Groupin = { p, p = { p, p = { p, p = { p, p = { p, p = { p, p links between roups = { p4, p = { p, 5 4 p5 { 5 = p 6 = { p = { p, p { 4 = p = { p, 5 p4 = { p4, 4 = { p, p p5 5 5 = { p, p, p6 6 = { p5, p6 Semented Frames p 4 p p p 5 p p 4 p p p p p 4 p p 5 p 6 p Oriinal Frames Groupin layers and links between roups (a) First Pass : Groupin = { p, p = { p, p = { p4, p = { p, 5 4 p5 { 5 = p 6 = { p = { p, p = { p, p = { p, p { 4 = p = { p, 5 p4 = { p, p = { p, p = { p4, 4 = { p, p p5 5 5 = { p, p, p6 6 = { p5, p6 Oriinal Frames User Selection Trackin User Selection Trackin User Selection (b) Second Pass: Selection Fi.. Overview of the two-pass approach Fi. illustrates our two-pass scheme. n Fi. (a), ASL video frames are first semented into seed primitives; these primitives are rouped by a roupin enine to enerate overlapped candidate hand roups, where each roup may consist of one or more primitives. This step is automatic and no user interaction is involved. n Fi. (b), where the second pass is taken, the rouped results will be loaded back for user s examination. Note the number of enerated roups could be hue, hence we allow the user to mouse click the hand reion to reduce the number of candidates. At the same time a trackin method is also incorporated amon adjacent frames to improve efficiency. The paper is oranized as follows: in section we will discuss the alorithm used in the first-pass, Section describes the Graphic User nterface(gu), Section 4 shows us the experimental results and we conclude with Section 5. First-Pass: Sementation and Groupin The first-pass annotation can be done completely on the backround since there is no user interaction involved. The oal of this pass is to enerate the hand candidates and store them as a file to be reloaded in the second pass. For existin

4 68 R. Yan et al. annotation tools, these two passes are enerally workin at the same time, forcin the user to wait for the sementin and trackin result. With a complex scene, the sementation itself can be very time consumin. (e.. a sementation of one sinle frame with a hihly textured scene with mean shift method [] could take over 0 seconds). On the other hand, many trackin techniques take an iterative approach which could also be slow. Trackin methods also suffer from the requirement of ood initialization, which must be taken for each sin in the trainin dataset separately. For our case, since we tend to enerate as many candidates as possible to accommodate with the fast chanin appearance of the hand while sinin, the processin of one sinle frame will even be slower. For example, with a complex scene there will be over 500 candidate hands enerated, which result in a runnin time for one frame to exceed 0 seconds. Fortunately, unlike other trackin or hand sementation schemes, our approach does not require perfect results or user modification when the candidate hands are bein detected. Hence this step can be done solely in the backround.. Groupin of Primitives We adopt a reedy approach to form the roups. Let us denote the low-level reion patches in the k-th imae frame by S k = {p k,,p k N k. From this initial set of primitives, we select a subset of primitives that are likely to come from the hand, based on the size of the patch. These are our seed patches. Given some knowlede of the approximate size of the hands in the sequence, we can eliminate lare, non-homoeneous reion patches from further consideration. We use a list L to store the possible roups. This list is initialized by choosin each selected primitive to be a sinleton roup. These roups would be mered to form a larer conlomerate. The roupin process starts by pickin the first roup in L, denoted here by p, and searchin for its neihbors {Np i. Each neihbor N p i is considered for roupin with p to enerate a tentative larer roupin. We select the best local roupin, and denote it as. The roup is further tested to see if it can possibly represent a hand. f the roup passes this test, it is inserted into the final candidate roup list, C, otherwise it is inserted at the end of the list L, to be considered for further roupin.. Associatin Groups Across Frames We denote the jth roup detected in tth frame as G j t. The roups detected in each frame are associated with those detected in previous frames to result in a linked sequence of roups spannin all the frames. We define the predecessors set of each element in each roups set as Pre(G j t )=[G j t,,gjn t ],t, j k c t, () where G j k t is one possible predecessor of G j t. The predecessor relationship between the roups from different time instants is based on feature similarity. t

5 Efficient Generation of Lare Amounts of Trainin Data 69 captures how likely the roups are from the same underlyin cause in the imae. We will use the best successors as the trackin result. Second Pass: Graphic User nterface After the candidate roups and their links are enerated, user interaction is needed to select the best roup, with a default roup iven at each frame by the trackin method. We provide functionality that specifically works well for sin sentences. These functions are built upon the candidate enerator discussed in section, a simple trackin technique that work with the links between the candidate roups, a face detector for non-manual information analysis, a lossin tool and various elements that facilitate the hand roundtruthin. Fi.. The Graphic User nterface The application is coded under the Microsoft Visual Studio Environment, usin MFC class, OpenCV libraries and related windows APs. Fi. illustrate us the GU. mportant functions related to sin lanuae are supported as:. views of current frame: The first is the dominant hand view, the second is the non-dominant hand view, the third is the view for both.. Click to select: One can click on the hand area to reduce the roups to be shown. The shown roups must include the point where the mouse is clicked.. Missin hand checkbox: One can choose the current hand as missin if the hand is out of the scene. 4. Hand listboxes: The two listboxes below show the list of the candidate dominant hand and candidate non-dominant hands. The list is ranked by their boundary smoothness and the trackin result. The hihlihted item is the candidate hand that is bein selected. 5. Face detector: automatic face detector, shown as the blue boundin box.

6 640 R. Yan et al. 6. Play,Stop,Step Button: Pressin the play button will automatically track all the hands and save the result. Stop Button will stop the trackin. Step button is to allow the application to track one frame and wait for the user s response, which is most often used. 7. Redo Checkbox: Redo Checkbox allows one to re-detect the current sequence. 4 Experiment Result We tested our annotation tool with datasets with different parameters settins. Both of them consist of ASL sin sentences. The first dataset has a simple backround with 0675 frames and a resolution of 460x90. On averae there are 00 candidate hands enerated per frame for this dataset, yet it took us less than 8 hours to finish roundtruthin both hands. The second dataset has a complex backround with 640x480 resolution. There are 500 candidate hands enerated for each frame. We took 500 frames and established the roundtruth within hour. Note the time we refer to here is the user interaction time, that is, the time of the second pass. n Fi. (a), we show the time taken over the datasets of the two passes. We use a P4.4G CPU with 4G memory. The number shown is the time taken per frame in seconds. Our first pass takes relatively loner since we incorporated automatic sementation, face detection, and the reedy seeds rowin alorithm. However it s completed offline. The second pass is done by reloadin the candidate result, takin much shorter time. Fi.. (a) Time taken of first pass and second pass (in seconds). (b) Time taken with different method sets. (in seconds).

7 Efficient Generation of Lare Amounts of Trainin Data 64 (a) (b) (c) (d) (e) (f) Fi. 4. (a) Oriinal frame. (b) Semented frame (c) List of candidate roups in frame. (d) Oriinal frame. (e) Semented frame (f) List of candidate roups in in frame. On the other hand, Fi. (b) shows us the time taken for different method sets of the tool. We choose 500 frames from the simple backround set to do the experiment. Here we refer to C as the method that only uses the enerated result, C+C refers to usin the click-to-select method to reduce the candidate set, C+T refers to usin candidate with trackin method, C++ refers to the method of usin candidate and both click-to-select and trackin method. Trackin contributes a lot because it exploits the temporal relationship. The click-to-select method does help when trackin failed. For example, when a lare motion happens, drastic hand shape chanes and occlusion happens. At the same time, we did an experiment, usin wand tools in a imae editin software to roundtruth 0 frames randomly chosen from the dataset. The averae time taken for each frame exceeds min. Fi. 4 shows us some visual results of the enerated candidate hands. We can see our method can discern the handshape even when there is occlusion and overlaps. n particular, Fi. 4 (c) shows us the result where face crossed hand, and Fi. 4 (f) shows us where the two hands crossed each other. 5 Conclusion n this paper, we describe our semi-automatic tool for annotatin sin lanuae video sequences. The objective of this tool is to enerate lare amount of data with labeled hand pixels that can provide reliable trainin process. The tool uses sementation, trackin and offline processin method to considerably save the

8 64 R. Yan et al. time to et the reliable hand data. For future work, an assistin edit tool can be added to secure the situation when there are meres amon hand pixels and non-hand pixels where sementation and roupin fail. Acknowledment This work was supported in part by the National Science Foundation under rant S 099. Any opinions, findins and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect those of the National Science Foundation. References. B. L. Loedin, S. Sarkar, A. Parashar and A.. Karshmer.: Proress in Automated Computer Reconition of Sin Lanuae., Lecture Notes in Computer Science, 8(004), T. Starner, and J. Weaver and A. P. Pentland.: Real-time American Sin Lanuae Reconition usin Desk and Wearable Computer Based Video, EEE Transactions on Pattern Analysis and Machine ntellience, (998) L. V. Tsap and M. C. Shin.: Dynamic Disparity Adjustment and Historam-Based Filterin of Rane Data for Fast -D Hand Trackin, Journal of Diital Sinal Processin, 4 (004) A. Just and S. Marcel.: Two-Handed Gesture Reconition, DAP Research nstitute CH-90, 005, Martiny, Switzerland 5. M.H. Yan., N.Ahuja. and M. Tabb.: Extraction of D Motion Trajectories and its Application to Hand Gesture Reconition., EEE Transactions on Pattern Analysis and Machine ntellience, 4 (00) T. Volkmer and J. R. Smith and A. Natsev.: A Web-based System for Collaborative Annotation of Lare mae and Video Collections: An Evaluation and User Study, MULTMEDA 05: Proceedins of the th annual ACM international conference on Multimedia, (005) L. V. Ahn and L. Dabbish.: Labelin maes with a Computer Game. n Conference on Human Factors in Computin Systems (CH) (004) T. Pfund and S. M. Maillet. A Dynamic Multimedia Annotation tool. n G. Beretta and R. Schettini, editors, n Proceedins of SPE Photonics West, Electronic main 00, nternet main, 467 (00) D. Doermann and D. Mihalcik. Tools and Techniques for Video Performance Evaluation. n Proceedins of the EEE nternational Conference on Pattern Reconistion (CPR 00), 4 (000) B. Marcoteui, P. Correia,A Video Object Generation Tool Allowin Friendly User nteraction.,nternational Conference on mae Processin, 4 (999) D. Comanicu and P. Meer.: Mean shift: A Robust Approach Toward Feature Space Analysis., EEE Transactions on Pattern Analysis and Machine ntellience, 4 (00)