Gesture Recognition using a Probabilistic Framework for Pose Matching

Transcription

1 Gesture Recognition using a Probabilistic Framework or Pose Matching Ahmed Elgammal Vinay Shet Yaser Yacoob Larry S. Davis Computer Vision Laboratory University o Maryland College Park MD USA Abstract This paper presents an approach or view-based recognition o gestures. The approach is based on representing each gesture as a seuence o learned body poses. The gestures are recognized through a probabilistic ramework or matching these body poses and or imposing temporal constrains between dierent poses. Matching individual poses to image data is perormed using a probabilistic ormulation or edge matching to obtain a likelihood measurement or each individual pose. The paper introduces a weighted matching scheme or edge templates that emphasize discriminating eatures in the matching. The weighting does not reuire establishing correspondences between the dierent pose models. The probabilistic ramework also imposes temporal constrains between dierent pose through a learned Hidden Markov Model (HMM) or each gesture. 1 Introduction The recognition o human gestures has many applications in human computer interaction virtual reality and in robotics. Dierent approaches have been proposed recently or gesture recognition. These approaches can be classiied into three major categories: model based appearance based and motion based. Model based approaches ocus on recovering three-dimensional model parameters o articulated body parts [13 4]. Appearance based approaches uses two dimensional inormation such as gray scale images or body silhouettes and edges. In contrast motion based approaches attempt to recognize the gesture directly rom the motion without any structural inormation about the physical body or example [1]. In all these approaches the temporal properties o the gesture are typically handled using Dynamic Time Warping (DTW) or statistically using Hidden Markov Models (HMM). Our objective is to recognize arm gestures perormed by a human standing at a distance rom the camera. This might be to operate a robot or a vehicle driven by a robot or generally to control an environment using gestures. This paper presents an approach or view-based recognition o gestures. The approach is based on representing each gesture as a seuence o learned body poses through a probabilistic ramework or matching these body poses to the the image data. The probabilistic ramework also imposes temporal constrains between dierent pose through a learned Hidden Markov Model (HMM) o each gesture. Matching individual poses is perormed using a probabilistic ormulation or Chamer matching to obtain a likelihood measurement or each individual pose. The paper introduces a weighted matching scheme or edge templates that emphasize discriminating eatures in the matching. The weighting does not reuire establishing correspondences between the dierent pose models. The paper is organized as ollow. Section 2 gives an overview o the proposed system. Section 3 presents an overview about the problem o edge matching. Section 4 presents the proposed pose classiication approach. Section 5 describes the hidden Markov model used or gesture recognition. Section 6 illustrates some experimental results. 2 System Overview Figure 1 describes an overview o the proposed system. The system consists o three modules: Training Segmentation and Tracking and Gesture Recognition. The training module learns models or body poses. It also learns temporal models o each individual gesture as a seuence o the learned body poses through a Hidden Markov Model (HMM). Finally the training module learns models o the activities i.e the dierent gesture that can be recognized and their temporal relations. The segmentation and tracking module continuously segment and track the person perorming the gesture rom the rest o the background. The segmentation rom the background is perormed using coarse range data through a series o plane itting to the range data and a rule based system to determine which plane in the range corresponds to the person. The range data is registered to the video data thereore locating the person in the

2 5 3 transormation and thereore the matching algorithm has to compensate or that by searching a transormation space to achieve the best match. Also the image typically contains other eatures rom the cluttered background other objects in the scene and noise. The object itsel might be occluded or some o the eatures are not detected and thereore the matching algorithm has to be robust to these missing eatures. The problem becomes even harder i the object is non-rigid and/or contains articulated parts or example the case o humans which is considered in this paper. Figure 1. System Overview range image locates the person in the video data. Since the segmentation is perormed using a very coarse range data the output o this module is just the location o the person in the image as a rectangle. The uality o the range data in terms o accuracy is not enough to provide ine silhouette segmentation or to do gesture recognition. For an example o the uality o the result o the segmentation see igure 7. The system is supposed to be mounted on a moving vehicle thereore the person is continuously tracked in both the range and the video to provide the context inormation necessary or the recognition module. The gesture recognition module uses only the video data. It matches learned silhouette models through coarse to ine search around the person location provided by the segmentation and tracking module to register the learned poses to the video data. The recognition module then matches all learned pose models to each new image to obtain pose probability likelihoods. The gesture classiication part uses the learned HMM o each gesture to impose temporal constrains on the body poses and thereore determine the gesture class. The gesture recognition module also uses an HMM activity model to determine the beginning and the end o each gesture (gesture segmentation). 3 Edge Matching: Background The problem o matching a eature template corresponding to an object to an image is a classical problem in computer vision with many applications or object detection recognition and tracking. The objective is to match a eature template which is a inite set o eature points to an image. This problem is hard or many reasons. The occurrence o the object in the image undergoes dierent geometric Figure 2. Distance Transorm The distance transorm DT has been used in matching edge eature (and other eature) templates. Given a set o eatures detected in an image the distance transorm at pixel is deined to be the distance to the nearest eature point in the image i.e.! "$#&% ')( +* where * is a metric. Typically the Euclidean distance is used or the metric * and in this case the unction. deines the Voronoi surace o [6]. The distance transorm is deined with respect to a set o binary eature o the same type e.g. edges or corners. Given a template /0 1 2 where 43 s are the locations o the template eatures transormed into the image space through translation rotation scaling or other geometric transormations the matching can be achieved by averaging the distance transorm values at each transormed template eature location 3 i.e. the matching 5 score is 71 9 :< This orm o matching is called Chamer matching and the distance 5 6 is called the Chamer distance. The smaller this matching score the better the match and an ideal match will have the value 0 where the template exactly lie over its corresponding image location. Chamer distance has been used extensively in object detection or example in [3]. Note that Chamer matching is asymmetric (model to image) so additional eatures in the image will not contribute to the matching. Figure 2 shows an example o an image and the detected edge eatures and the distance transormed image according to these eatures. In [2] the matching was generalized to include multiple eature types (or example oriented edges) by matching 6 (1)

3 = = 3Y Figure 3. Example body poses rom two dierent Gesture each individual eature template with its corresponding distance transormed image and combining the results. Also the matching was generalized in [2 3] to match multiple templates through a hierarchical template structure. Another metric o similarity between the transormed template and the image eatures is the direct Hausdor distance [6] = 71 deined by 7 >"@?BA C ()D "E#&% ')( GF IHJ F The unction = 71 simply inds the nearest point in or each template point and thereater = 7 deines the arthest o these distances. In other words = 7 deines the most mismatch in the best match between the template and the image eatures. Thereore the smaller the distance = 71 the better the match. Obviously this distance is also not symmetric. Direct Hausdor distance can be computed using the distance transorm where = 71 can be rewritten in terms o KL as 71 M"@?BA C ()D Since Hausdor distance is deined based on the worst mismatch it is very sensitive to outliers. Any missing eature in the image (occluded or not detected) can aect the distance dramatically. Thereore a more robust measure is introduced in [6] and is called the partial Hausdor measure and is deined by =ON 6 QP CR C ()D "$#S% ')( GF THJ F Several extensions have been proposed to matching using Hausdor distance. This includes but are not limited to matching oriented edge pixels [10]. Also a probabilistic ormulation o Hausdor matching was introduced in [9 11]. Another extension is the use o Eigenspaces to approximate the Hausdor distance as was introduced in [7 5] 4 Pose Classiication 4.1 Pose Likelihood We represent each gesture as a seuence o body poses. The temporal relation between these body poses is enorced Figure 4. Pose template registered to an image by a hidden Markov model as will be presented in section 5. This section ocuses on matching individual body poses. The objective is to evaluate all dierent body pose models with respect to each new rame in order to obtain a probability estimate or each o these poses. Let UVW XZY-[Q \] be the set o all learned body poses or all the gestures to be recognized. Each pose X Y is represented as an edge template i.e. each pose is represented as a inite set o edge eature locations X Y / Y Y ^`_ Y where a Y is the number o edge eatures or pose template [. Figure 3 shows example pose templates or two dierent gestures. All the poses are registered to each other during the learning so registering one pose to any new image will thereore register the rest o the poses. Registering these poses to the images is done while the person is not perorming any gesture (idle). In this case the matching is perormed using an idle pose (shown in igure 3 irst pose on top) through a coarse to ine search. Figure 4 shows the registered poses to a new rame. At each new image it is desired to ind a probabilistic matching score or each pose. Let b7! be the distance transormed image given the set o edge eatures detected at image. For each edge eature 3 Y in pose model X Y the measurement 5 3 Y 7 is the distance to the nearest edge eature in the image. A perect model to image match will have 5 3 Y Vc or all model edge eatures. Let s consider the random variable associated with

4 5 Š Y ^ Š H 3 this distance measurement and let the associated probability density unction (PDF) be d 3 Y. We assume that these random variable are independent. This assumption was used in [9 11] based on the results obtained in [10]. Thereore the likelihood unction (the probability o the observation given the model X Y ) can be deined as the product o these PDFs as e X Y Qhg!i X Y j Since dierent templates have dierent numbers o eatures this likelihood euation needs to be normalized using the number o eatures in each modela Y. Taking the logarithm o this euation we obtain the log-likelihood unction l&m)n e X Y a Y 3SkI I all the poses are assumed to be euiprobable then the model probability given the observation is proportion to the likelihood i.e. X Y i po!i X Y. Thereore we can use this likelihood unction to evaluate dierent models. The PDF d 3Y or the distance between model eatures and nearest image eature location is deined or each eature in each pose model [. We use a PDF o the orm l&m)n r ts u 3wv Y xy7z{ 5 Y 3 5 Y 3 t}1~ S _ } The scale parameter u Y 3 is deined or each pose [ and each eature. The motivation behind this is that dierent variations (or uncertainty) are expected at dierent model eatures or example the edges corresponding to the hand are expected to have more variations in location than the upper arm or the shoulder location. These variations are learned during the learning o the pose models. Since the distance 5 can become arbitrary large the probability can become very small and thereore the constant r is used as a lower bound on the probability. This makes the likelihood unction robust to outliers. A similar PDF was used in [9] but with the same scale variable u or all the eatures. This probabilistic ormulation was irst introduced in [9] and was used in a Hausdor matching context to ind the best transormation o an edge template using maximum likelihood estimation. Euation 2 represents a probabilistic ormulation o Chamer matching as deined in euation 1. We use this probabilistic ormulation to evaluate the observation likelihood given each gesture state as will be described in section weighted matching Our objective is to match multiple pose templates to the same image location in order to evaluate the likelihood o (2) the observation given each o these poses. Typically the dierent pose templates are similar in some parts and dierent in another parts in the templates. For example the head torso and bottom parts o the body are likely to be similar in dierent pose templates while articulated body parts that are involved in the gesture such as the arm will be at dierent positions at dierent pose templates. For example see igure 4. Since the articulated part such as the arm is represented by a small number o eatures with respect to the whole pose templates the matching is likely to be biased by the major body parts. Instead it is desired to make the matching biased more by articulated parts involved in perorming the gesture since these parts will be more discriminating between dierent poses templates. To achieve this goal dierent weights are assigned to dierent eature points in each pose template. Thereore each pose template X Y is represented as a set o eature locations as well as a set o weights 2ƒ Y 4ƒ Y 4ƒ ^ Y _ corresponding to each eature where ƒ 3 Y. The likelihood euation 2 is then modiied to be a weighted one lsmn e X Y 3SkI In our case the set o all recognized poses does not have a common correspondence rame. For example some eatures in one pose might not have corresponding eatures in another poses. Also we do not restrict the pose templates to have the same number o eatures. Thereore we drive the weights with respect to the image locations. Let X be the set o all eatures in all registered poses in the training data i.e. X X Y ˆ ƒ Y 3 lsmn 5 Y 3 ) ^ where each 3 is the image location o an edge eature. Given this sample o w edge eature locations the edge probability distribution (the probability to see an edge at certain image location ) can be estimated using kernel density estimation [14] as w a P R Where P R is a kernel unction with a scale variable =. We used a Gaussian kernel P R 1Œ R ~ Ž } or this z { probability estimation. The weight assigned to each eature point is based on the inormation this eature provides. Given the estimated edge probability distribution w (3) at any image pixel the weight or a certain eature at a certain pose [ is the ratio o the inormation given by this eature to the total

5 3Y inormation by that pose i.e. ƒ Y 3 5 Gesture classiication 5.1 HMM Overview l&m)n Š ^_ l&m)n Š kt An HMM consists o a set U o distinct states U B 21 2) - representing a Markov stochastic process. A stochastic process is called a Markov process i the conditional probability o the current event given all the past events depends only on the th most recent events. In particular i the current event depends only on the previous event then this is called a irst order Markov process and the HMM is called a irst order HMM. The HMM is called hidden i the stochastic variable associated with the states is not observable. Instead the observation is a probabilistic unction o the state. For an overview o HMMs and their applications in speech recognition reer to [12]. We use the same notation as in this paper. HMM s have been used extensively in gesture recognition. They were used in [15] or American Sign Language recognition (ASL) by tracking the hands based on color. In [17 16] HMM s were also used or ASL based on shape and motion parameters. In [] HMM s were used to track head gestures. In [1] a parametric HMM was introduced to model parametric gestures. Generally an HMM is deined as the states U the transition probabilities between the states 2 3 where C œ ˆ i C ˆ 3 where C is the state at time and the initial state distribution y where y 3 ]@š ž 3. Finally the observation probability given the states Ÿ + i. 5.2 Gesture HMM Y Figure 5. Let-Right HMM We represent each gesture by a set o poses X Y 1[ -\] and an HMM where the hidden states correspond to the progress o the gesture with time. The HMM elements are as ollows: 1. A set o states UV B B 1 2) 1 2. We use C to denote the state at time. Note that the number o states is not necessarily the same as the number o pose models \ i.e. each state does not necessarily represent one pose. Instead one state can represent a mixture o pose models. 2. The state transition probabilities 3 where C œ i C / 3 «ª - 4. We use a let-right model or a Bakis model [12] as in igure 5 since the progress o the gesture is always orward in time. 3. The initial state distribution y where y W@š ª. 4. The probability o each pose X Y given the states 2r Y Q X Y i ª ª [E \] The actual observation C is the detected edge eatures at each new rame which is a probabilistic unction o the current state o the gesture. This probabilistic unction is deined using the set o recognized poses. That is the observation probability given the state can be written as Ÿ C > C i Y kt C i X Y Given the deinition o the variables rewritten as Ÿ C > C i Y kt Y X Y i above this can be C i X Y We can think o the set o poses I as a set o discrete symbols or an alphabet that is being emitted by the dierent states but the actual observation is a probabilistic unction o these symbols based on the mixture deined by the variables. The observation probabilities given the poses C i X Y are obtained using the likelihood euations 2 and 3 as was described in section 4 Given a set o observations ) D and given a set o HMM models corresponding to dierent gesture the objective is to determine the probability o this observation seuence given each o the models i.e. i ª. This is a traditional problem or HMM and can be solved eiciently through a procedure called the Forward-Backward procedure [12]. This procedure deines a set o orward variables ± C U 3 i by: ± C œ where ± C 1 C which can be updated recursively at each time step ³² ± C y 3 Ÿ 3 3 µ Ÿ C œ. Given these orward variables the observation likelihood can be calculated as i ± D + H

6 gesture cl gesture cr gesture dl gesture dr gesture el gesture er gesture l gesture r HMM results gesture cl gesture cr gesture dl gesture dr gesture el gesture er gesture l gesture r HMM results Gesture Likelihood Gesture Likelihood Frames Frames Figure 6. Gesture classiication results 6 Experimental Results The proposed approach was used to classiy eight arm gestures. Basically the eight recognized gestures are similar to the ones shown in igure 4 perormed with both arms in upward motion and downward motion. Figure 7 shows the segmentation results obtained rom the range data. The image is color coded so that each itted plane has a dierent graylevel and the segmented person is labeled white. Figure shows some pose classiication results or dierent people. The igures shows the pose with the highest likelihood score overlaid over the original image. learned body poses where the temporal relation is learned through a Hidden Markov Model. Individual poses are matched to the image data through a probabilistic ormulation o Chamer distance that yields a probability likelihood estimation or each pose that is eeded to the HMM to perorm the gesture recognition. A weighted matching scheme was introduced in the paper to enable matching multiple poses in a way that emphasizes the discriminating eatures or each pose. Experimental results were shown where the proposed approach was used to recognize eight dierent hand gestures. Reerences [1] Aaron F. Bobick and James W. Davis. The recognition o human movement using temporal templates. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(3): Figure 7. Segmentation result Figure 6 shows the gesture likelihood probabilities or the eight gesture classes. As can be noticed rom the graphs All the gestures were close in likelihood at the beginning o the action but as the gesture progresses with time the likelihood o the right gesture increases and the the likelihood o the other gesture decreases as a result o the temporal constrains imposed by the HMM or each gesture. 7 Conclusions The paper presented an approach or view-based arm gesture recognition based on a probabilistic ramework or pose matching. Any gesture is represented by a seuence o [2] D. Gavrila. Multi-eature hierarchical template matching using distance transorms. In In Proc. o the International Conerence on Pattern Recognition Brisbane Australia199. pages [3] D. Gavrila and V. Philomin. Real-time object detection or smart vehicles. In ICCV99 pages [4] D. Hogg. Model based vision: A program to see a walking person. Image and Vision Computing 1(1): [5] D. Huttenlocher and C. Olson R.H. Lilien. Viewbased recognition using an eigenspace approximation

7 [12] L. R. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings o the IEEE 77(2): February 199. [13] J. Rehg and T. Kanade. Model-based tracking o seloccluding articulated objects. In ICCV95 pages [14] David W. Scott. Mulivariate Density Estimation. Wiley-Interscience [15] T. Starner and A. Pentland. Real-time american sign language recognition rom video using hidden markov models. In SCV95 page 5B Systems and Applications [16] C. Vogler and D. Metaxas. Adapting hidden markov models or asl recognition by using three-dimensional computer vision methods. In SMC 97. pages [17] Christian Vogler and Dimitris N. Metaxas. Parallel hidden markov models or american sign language recognition. In ICCV (1) pages Figure. Pose matching results to the hausdor measure. IEEE Transactions on Pattern Analysis and Machine Intelligence (9): [1] Andrew D. Wilson and Aaron F. Bobick. Parametric hidden markov models or gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(9): [6] D. Huttenlocher D. Klanderman and A. Rucklige. Comparing images using the Hausdor distance. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(9):50 63 September [7] Daniel P. Huttenlocher Ryan H. Lilien and Clark F. Olson. Approximate hausdor matching using eigenspaces. In ARPA Image Understanding Workshop 1996 pages [] C. Morimoto Y. Yacoob and L. Davis. Recognition o head gestures using hidden markov models international conerence on pattern recognition. In International Conerence on Pattern Recognition Vienna Austria August 1996 pages [9] C. Olson. A probabilistic ormulation or hausdor matching. In CVPR9 pages [10] C. Olson and D. Huttenlocher. Automatic target recognition by matching oriented edge pixels. IEEE Transactions on Image Processing (1): [11] Clark F. Olson. Maximum-likelihood template matching. In IEEE International Conerence on Computer Vision and Pattern Recognition volume 2 pages