Action Recognition by Matching Clustered Trajectories of Motion Vectors

Transcription

1 Acton Recognton by Matchng Clustered Trajectores of Moton Vectors Mchals Vrgkas 1, Vasleos Karavasls 1, Chrstophoros Nkou 1 and Ioanns Kakadars 2 1 Department of Computer Scence, Unversty of Ioannna, Ioannna, Greece 2 Department of Computer Scence, Unversty of Houston, Houston, Texas, USA {mvrgkas, vkaravas, cnkou}@cs.uo.gr, oannsk@uh.edu Keywords: Abstract: Human acton recognton; Optcal flow; Moton curves; Gaussan mxture modelng (GMM); Clusterng; Longest common subsequence. A framework for acton representaton and recognton based on the descrpton of an acton by tme seres of optcal flow moton features s presented. In the learnng step, the moton curves representng each acton are clustered usng Gaussan mxture modelng (GMM). In the recognton step, the optcal flow curves of a probe sequence are also clustered usng a GMM and the probe curves are matched to the learned curves usng a non-metrc smlarty functon based on the longest common subsequence whch s robust to nose and provdes an ntutve noton of smlarty between trajectores. Fnally, the probe sequence s categorzed to the learned acton wth the maxmum smlarty usng a nearest neghbor classfcaton scheme. Expermental results on common acton databases demonstrate the effectveness of the proposed method. 1 Introducton Acton recognton s a preponderant and dffcult task n computer vson. Many applcatons, ncludng vdeo survellance systems, human-computer nteracton and robotcs to human behavor characterzaton, requre a multple actvty recognton system. The problem of categorzng a human acton remans a challengng task that has attracted much research effort n the recent years. The surveys n (Aggarwal and Ryoo, 2011) and (Poppe, 2010) provde a good overvew of the numerous papers on acton/actvty recognton and analyze the semantcs of human actvty categorzaton. Several feature extracton methods for descrbng and recognzng human actons have been proposed (Efros et al., 2003; Schuldt et al., 2004; Jhuang et al., 2007; Nebles et al., 2008; Fath and Mor, 2008). A major famly of methods reles on optcal flow whch has proven to be an mportant cue. In (Efros et al., 2003), human actons are recognzed from low-resoluton sports vdeo sequences usng the nearest neghbor classfer, where humans are represented by wndows of heght of 30 pxels. The approach n (Fath and Mor, 2008) s based on md-level moton features, whch are also constructed drectly from the optcal flow features. Moreover, Wang and Mor employed moton features as nputs to hdden condtonal random felds and support vector machne (SVM) classfers (Wang and Mor, 2011). Real tme classfcaton and predcton of future actons s proposed n (Morrs and Trved, 2011), where an actvty vocabulary s learnt through a three step procedure. Other optcal flowbased methods whch ganed popularty are presented n (Ln et al., 2009; Chaudhry et al., 2009). The classfcaton of a vdeo sequence usng local features n a spato-temporal envronment has been gven much focus. Schuldt et al. represent local events n a vdeo usng space-tme features, whle an SVM classfer s used to recognze an acton (Schuldt et al., 2004). In (Gorelck et al., 2007), actons are consdered as 3D space tme slhouettes of movng humans. They take advantage of the Posson equaton soluton to effcently descrbe an acton by utlzng spectral clusterng between sequences of features and applyng nearest neghbor classfcaton to characterze an acton. Nebles et al. address the problem of acton recognton by creatng a codebook of spacetme nterest ponts (Nebles et al., 2008). A herarchcal approach was followed n (Jhuang et al., 2007), where an nput vdeo s analyzed nto several feature descrptors dependng on ther complexty. The fnal classfcaton s performed by a mult-class SVM classfer. In (Dollár et al., 2005), spato-temporal features are proposed based on cubod descrptors. An acton descrptor of hstograms of nterest ponts, re-

2 lyng on (Schuldt et al., 2004) was presented n (Yan and Luo, 2012). Random forests for acton representaton have also been attractng wdespread nterest for acton recognton (Yao et al., 2010). Furthermore, the key ssue of how many frames are requred for acton recognton s addressed n (Schndler and Gool, 2008). In ths paper, we address the problem of human acton recognton by representng an acton wth a set of clustered moton trajectores. Moton curves are generated by optcal flow features whch are then clustered usng a dfferent Gaussan mxture (Bshop, 2006) for each dstnct acton. The optcal flow curves of a probe sequence are also clustered usng a Gaussan mxture model (GMM) and they are matched to the learned curves usng a smlarty functon (Vlachos et al., 2002) relyng on the longest common subsequence (LCSS) between trajectores. The LCSS s robust to nose and provdes an ntutve noton of smlarty between trajectores. Snce dfferent actors perform the same acton n dfferent manners and at dfferent speeds, an advantage of the LCSS smlarty s that t can handle wth moton trajectores of vared lengths. The man contrbuton of the paper s twofold. At frst, human moton s represented by a small set of trajectores whch are the mean curves of the mxture components along wth ther covarance matrces. The complexty of the model s consdered low, as t s determned by the Bayesan Informaton Crteron (BIC), but any other model selecton technque may be appled. Secondly, the use of the longest common subsequence ndex allows nput curves of dfferent length to be compared relably. The rest of the paper s organzed as follows: we represent the extracton of moton trajectores, the clusterng and the curve matchng n Secton 2. In Secton 3, we report results on the Wezmann (Blank et al., 2005) and the KTH (Schuldt et al., 2004) acton classfcaton datasets. Fnally, conclusons are drawn n Secton 4. 2 Acton Representaton and Recognton Our goal s to analyze and nterpret dfferent classes of actons to buld a model for human actvty categorzaton. Gven a collecton of fgure-centrc sequences, we represent moton templates usng optcal flow (Lucas and Kanade, 1981) at each frame. Assumng that a boundng box can be automatcally obtaned from the mage data, we defne a square regon of nterest (ROI) around the human. A bref overvew of our approach s depcted n Fgure 1. In the tranng mode, we assume that the vdeo sequences contan only one actor performng only one acton per frame. However, n the recognton mode, we allow more than one acton per vdeo frame. The optcal flow vectors as well as the moton descrptors (Efros et al., 2003) for each sequence are computed. These moton descrptors are collected together to construct moton curves, whch are clustered usng a mxture model to descrbe a unque acton. Then, the moton curves are clustered and each acton s characterzed by a set of clustered moton curves. Acton recognton s performed by matchng the clusters of moton curves of the probe sequence and the clustered curves n each tranng sequence. 2.1 Moton Representaton Followng the work n (Efros et al., 2003), we compute the moton descrptor for the ROI as a fourdmensonal vector F = ( F x +,Fx,F y +,Fy ) R 4, where = 1,..., N, wth N beng the number of pxels n the ROI. Also, the matrx F refers to the blurred, moton compensated optcal flow. We compute the optcal flow F, whch has two components, the horzontal F x, and the vertcal F y, at each pxel. It s worth notng that the horzontal and vertcal components of the optcal flow F x and F y are half-wave rectfed nto four non-negatve channels F x +,Fx,F y +,Fy, so that F x = F x + F x and F y = F y + F y. In the general case, optcal flow s sufferng from nosy measurements and analyzng data under these crcumstances wll lead to unstable results. To handle any moton artfacts due to camera movements, each half-wave moton compensated flow s blurred wth a Gaussan kernel. In ths way, the substantve moton nformaton s preserved, whle mnor varatons are dscarded. Thus, any ncorrectly computed flows are removed. 2.2 Extracton of Moton Curves Consder T to be the number of mage frames and C={c (t)},t [0,T], s a set of moton curves for the set of pxels =1,...,N of the ROI. Each moton curve s descrbed as a set of ponts correspondng to the optcal flow vector extracted n the ROI. Specfcally, we descrbe the moton at each pxel by the optcal flow vector F = ( F x +,Fx,F y +,Fy ). A set of moton curves for a specfc acton s depcted n Fgure 1. Gven the set of moton descrptors for all frames, we construct the moton curves by followng ther optcal flow components n consecutve frames. If there s no pxel dsplacement we consder a zero optcal flow vector dsplacement for ths pxel.

3 Fgure 1: Overvew of our approach. The set of moton curves descrbes completely the moton n the ROI. Once the moton curves are created, pxels and therefore curves that belong to the background are elmnated. Ths s accomplshed by dscardng curves whose ampltude of the optcal flow vector s below a predefned threshold. In order to establsh a correspondence between the moton curves and the actual moton, we perform clusterng of the moton curves usng a Gaussan mxture model. We estmate the characterstc moton whch s represented by the mean trajectory of each cluster. 2.3 Moton Curves Clusterng A moton curve s consdered to be a 2D tme sgnal c j (t)= ( F x j (t),f y j (t) ), t [0,T] where the ndex = 1,...,N represents the th pxel, for the j th vdeo sequence n the tranng set. To effcently learn human acton categores, each acton s represented by a GMM by clusterng the moton curves n every sequence of the tranng set. The p th acton (p=1,...,a) n the j th vdeo sequence ( j = 1,...,S p ) s modeled by a set of K p j mean curves (learned by the GMM) denoted by x p jk (t), k=1,...,kp j. The GMM s traned usng the Expectaton- Maxmzaton (EM) algorthm (Bshop, 2006), whch provdes a soluton to the problem of estmatng the model s parameters. However, the number of mxture components should be determned. To select the number of the Gaussans K p j, for the jth tranng sequence, representng the p th acton, the Bayesan Informaton crteron (BIC) (Bshop, 2006) s used. Thus, when EM converges the cluster labels of the moton curves are obtaned. Ths s schematcally depcted n Fgure 1, where a set of moton trajectores, representng a certan acton (e.g., p), n a vdeo sequence (e.g., labeled by j) s clustered by a GMM nto K p j = 2 curves for acton representaton. Note that, a gven acton s generally represented by a varyng number of mean trajectores as the BIC crteron may result n a dfferent number of components n dfferent sequences. 2.4 Matchng of Moton Curves Once a new probe vdeo s presented, where we must recognze the acton depcted, the optcal flow s computed, moton trajectores are created and clustered, and they are compared wth the learned mean trajectores of the tranng set. Recall that human actons are not unform sequences n tme, snce dfferent actors perform the same acton n dfferent manner and at dfferent speeds. Ths means that moton curves have vared lengths. An optmal matchng may be performed usng dynamc programmng whch detects smlar pars of curve segments. The longest common subsequence (LCSS) (Vlachos et al., 2002) s robust to nose and provdes a smlarty measure between moton trajectores snce not all ponts need to be matched. Let c 1 (t), t [0,T] and c 2 (t ), t [0,T ] be two curves of dfferent lengths. Then, we defne the affn-

4 ty between the two curves as: ( ) ( ) LCSS c α c 1 (t),c 2 (t 1 (t),c 2 (t ) ) =, (1) ( where the LCSS c 1 (t),c 2 (t ) mn(t,t ) ) (Eq. (2)) ndcates the qualty of the matchng between the curves c 1 (t) and c 2 (t ) and measures the number of the matchng ponts between two curves of dfferent lengths. Note that the LCSS s a modfcaton of the edt dstance (Theodords and Koutroumbas, 2008) and ts value s computed wthn a constant tme wndow δ and a constant ampltude ε, that control the matchng thresholds. The terms c 1 (t) T t and c 2 (t ) T t denote the number of curve ponts up to tme t and t, accordngly. The dea s to match segments of curves by performng tme stretchng so that segments that le close to each other (ther temporal coordnates are wthn δ) can be matched f ther ampltudes dffer at most by ε. When a probe vdeo sequence s presented, ts moton trajectores are clustered usng GMMs of varous numbers of components usng the EM algorthm. The BIC crteron s employed to determne the optmal value of the number of Gaussans K, whch represent the acton. Thus, we have a set of K mean trajectores y k, k=1,...,k modelng the probe acton. Recognton of the acton present n the probe vdeo sequence s performed by assgnng the probe acton to the acton of the labeled sequence whch s most smlar. As both the probe sequence y and the j th labeled sequence of the p th acton n the tranng set x p j are represented by a number of GMM components, the overall dstance between them s computed by: p K d(x p j j,y)= K k=1l=1 π p jk π l [ ( )] 1 α x p jk (t),y l(t ), (3) where π p jk and π l are the GMM mxng proportons for the labeled and probe sequence, respectvely, that s k π p jk = 1 and l π l = 1. The probe sequence y s categorzed wth respect to ts mnmum dstance from an already learned acton: [ j, p ]=argmnd(x p j,y). (4) j,p 3 Expermental Results We evaluated the proposed method on acton recognton by conductng a set of experments over publcly avalable datasets. At frst we appled the algorthm to the Wezmann human acton dataset Table 1: Recognton accuracy and executon tme over the Wezmann dataset. The results of (Blank et al., 2005; Seo and Mlanfar, 2011; Nebles et al., 2008; Chaudhry et al., 2009; Ln et al., 2009; Jhuang et al., 2007; Fath and Mor, 2008) are taken from the orgnal papers. Method Accuracy (%) Proposed Method 98.8 (Blank et al., 2005) (Chaudhry et al., 2009) 95.7 (Fath and Mor, 2008) (Jhuang et al., 2007) 98.8 (Ln et al., 2009) (Nebles et al., 2008) 90.0 (Seo and Mlanfar, 2011) 97.5 (Blank et al., 2005). The Wezmann dataset s a collecton of 90 low-resoluton vdeos, whch conssts of 10 dfferent actons such as run, walk, skp, jumpng jack, jump forward, jump n place, gallop sdeways, wave wth two hands, wave wth one hand, and bend, performed by nne dfferent people. The vdeos were acqured wth a statc camera and contan uncluttered background. To test the proposed method on acton recognton we adopted the leave-one-out scheme. We learned the model parameters from the vdeos of eght subjects and tested the recognton results on the remanng vdeo set. The procedure was repeated for all sets of vdeo sequences and the fnal result s the average of the ndvdual results. The optmal number of mxture components K p j for the j th vdeo sequence, j= 1,...,S p of the p th acton p=1,...,a s found by employng the BIC crteron. The value of BIC was computed for K p j = 1 to 10. In the recognton step, n our mplementaton of the LCSS (2), the constants δ and ε were estmated usng cross valdaton. Parameter δ, was set to 1% of the trajectores lengths, and parameter ε was determned as the smallest standard devaton of the two trajectores to be compared. As shown n Table 1, the average correct classfcaton of the algorthm on ths dataset s 98.8%. The performances of other state-of-the-art methods on the same dataset are shown n Table 1. As t can be observed, we acheve better results wth respect to four out of seven state-of-the-art methods. However, the proposed method provded only one erroneous categorzaton as one jump-n-place (pjump) acton was wrongly categorzed as run. It seems that n ths case the number of Gaussan components K p j computed by the BIC crteron was not optmal. Fgure 2 depcts the confuson matrx for ths experment. We have further assessed the performance rate of our method by conductng experments on the KTH

5 ( ) LCSS c 1 (t),c 2 (t ) = 0, f T = 0 or T = 0, 1+LCSS { max LCSS (c 1 (t) T t 1,c 2 (t ) T t 1 ), f c 1 (t) c 2 (t ) <ε and T T <δ, )},otherwse (c 1 (t) Tt 1,c 2 (t ) T t ), LCSS (c 1 (t) T t,c 2 (t ) T t 1 (2) Table 2: Recognton results over the KTH dataset. The results of (Fath and Mor, 2008; Jhuang et al., 2007; Ln et al., 2009; Nebles et al., 2008; Schuldt et al., 2004; Seo and Mlanfar, 2011) are taken from the orgnal papers. Method Accuracy (%) Proposed Method (Fath and Mor, 2008) 90.5 (Jhuang et al., 2007) 90.5 (Ln et al., 2009) (Nebles et al., 2008) (Schuldt et al., 2004) (Seo and Mlanfar, 2011) 95.1 Wezmann database, accuracy = 98.8% Fgure 2: Confuson matrx for the classfcaton results for the Wezmann dataset for the estmaton of the number of components usng the BIC crteron for both the tranng and probe sequences. dataset (Schuldt et al., 2004). Ths dataset conssts of 2391 sequences and contans sx types of human actons such as walkng, joggng, runnng, boxng, hand wavng, and hand clappng. These actons are repeatedly performed by 25 dfferent people n four dfferent envronments: outdoors (s1), outdoors wth scale varaton (s2), outdoors wth dfferent clothes (s3), and ndoors (s4). The vdeo sequences were acqured usng a statc camera and nclude a unform background. The average length of the vdeo sequences s four seconds, whle they were downsampled to the spatal resoluton of pxels. We tested the acton recognton capablty of the proposed method by usng a leave-one-out cross valdaton approach. Accordngly, we learned the model from the vdeos of 24 subjects whle we tested the algorthm on the remanng subject and averaged the recognton results. The confuson matrx over the KTH dataset for ths leave-one-out approach s depcted n Fgure 3. We acheved a recognton rate of 96.71%, whch to the best of our knowledge s a very hgh performance for ths dataset. In addton, comparson of the proposed method wth other stateof-the-art methods s reported n Table 2. As can be observed, the proposed method provdes the more ac- curate recognton rates. The proposed method attans hgh acton classfcaton accuracy as the BIC crteron determnes the optmal value of Gaussans K p j for ths dataset. The average recognton tme depends on the value of Gaussans K p j and t s approxmately 2 sec for both datasets, on a standard PC (dual core, 2 GHz RAM). KTH database, accuracy = 96.71% Fgure 3: Confuson matrx for the classfcaton results for the KTH dataset for the estmaton of the number of components usng the BIC crteron for both the tranng and probe sequences.

6 4 Concluson In ths paper, we presented an acton recognton approach, where actons are represented by a set of moton curves assumed to be generated by a probablstc model. The performance of the extracted moton curves s nterpreted by dscoverng smlartes between the moton trajectores, followed by a classfcaton scheme. Although a perfect recognton performance s accomplshed wth a fxed number of Gaussan mxtures, there are stll some open ssues n feature representaton. Our next step s to apply ths work to other benchmark databases wth rcher moton varatons and more nformaton to be modeled by a Gaussan mxture where more Gaussan components would be necessary. Moreover, an extenson of the acton classfcaton method s envsoned n order to ntegrate t nto a complete scheme consstng of moton detecton, background subtracton, and acton recognton n natural and cluttered envronments, whch s a dffcult and more challengng topc. REFERENCES Aggarwal, J. K. and Ryoo, M. S. (2011). Human actvty analyss: A revew. ACM Compututng Surveys, 43(3):1 43. Bshop, C. M. (2006). Pattern Recognton and Machne Learnng. Sprnger. Blank, M., Gorelck, L., Shechtman, E., Iran, M., and Basr, R. (2005). Actons as space-tme shapes. In Proc. 10 th IEEE Internatonal Conference on Computer Vson, pages , Begng, Chna. Chaudhry, R., Ravchandran, A., Hager, G. D., and Vdal, R. (2009). Hstograms of orented optcal flow and Bnet-Cauchy kernels on nonlnear dynamcal systems for the recognton of human actons. In Proc. IEEE Computer Socety Conference on Computer Vson and Pattern Recognton, pages , Mam, Florda, USA. Dollár, P., Rabaud, V., Cottrell, G., and Belonge, S. (2005). Behavor recognton va sparse spato-temporal features. In Proc. 14 th Internatonal Conference on Computer Communcatons and Networks, pages 65 72, Bejng, Chna. Efros, A. A., Berg, A. C., Mor, G., and Malk, J. (2003). Recognzng acton at a dstance. In Proc. 9 th IEEE Internatonal Conference on Computer Vson, volume 2, pages , Nce, France. Fath, A. and Mor, G. (2008). Acton recognton by learnng md-level moton features. In Proc. IEEE Computer Socety Conference on Computer Vson and Pattern Recognton, pages 1 8, Anchorage, Alaska, USA. Gorelck, L., Blank, M., Shechtman, E., Iran, M., and Basr, R. (2007). Actons as space-tme shapes. IEEE Transactons on Pattern Analyss and Machne Intellgence, 29(12): Jhuang, H., Serre, T., Wolf, L., and Poggo, T. (2007). A bologcally nspred system for acton recognton. In Proc. IEEE Internatonal Conference on Computer Vson, pages 1 8, Ro de Janero, Brazl. Ln, Z., Jang, Z., and Davs, L. S. (2009). Recognzng actons by shape-moton prototype trees. In Proc. IEEE Internatonal Conference on Computer Vson, pages , Mam, Florda, USA. Lucas, B. D. and Kanade, T. (1981). An teratve mage regstraton technque wth an applcaton to stereo vson. In Proc. 7 th Internatonal Jont Conference on Artfcal Intellgence, pages , Nce, France. Morrs, B. T. and Trved, M. M. (2011). Trajectory learnng for actvty understandng: Unsupervsed, multlevel, and long-term adaptve approach. IEEE Transactons on Pattern Analyss and Machne Intellgence, 33(11): Nebles, J. C., Wang, H., and Fe-Fe, L. (2008). Unsupervsed learnng of human acton categores usng spatal-temporal words. Internatonal Journal of Computer Vson, 79(3): Poppe, R. (2010). A survey on vson-based human acton recognton. Image and Vson Computng, 28(6): Schndler, K. and Gool, L. V. (2008). Acton snppets: How many frames does human acton recognton requre? pages 1 8, Anchorage, Alaska, USA. Schuldt, C., Laptev, I., and Caputo, B. (2004). Recognzng human actons: A local SVM approach. In Proc. 17 th Internatonal Conference on Pattern Recognton, pages 32 36, Cambrdge, UK. Seo, H. J. and Mlanfar, P. (2011). Acton recognton from one example. IEEE Transactons on Pattern Analyss and Machne Intellgence, 33(5): Theodords, S. and Koutroumbas, K. (2008). Pattern Recognton. Academc Press, 4th edton. Vlachos, M., Gunopoulos, D., and Kollos, G. (2002). Dscoverng smlar multdmensonal trajectores. In Proc. 18 th Internatonal Conference on Data Engneerng, pages , San Jose, Calforna, USA. Wang, Y. and Mor, G. (2011). Hdden part models for human acton recognton: Probablstc versus max margn. IEEE Transactons on Pattern Analyss and Machne Intellgence, 33(7): Yan, X. and Luo, Y. (2012). Recognzng human actons usng a new descrptor based on spatal-temporal nterest ponts and weghted-output classfer. Neurocomputng, 87: Yao, A., Gall, J., and Gool, L. V. (2010). A Hough transform-based votng framework for acton recognton. In Proc. IEEE Computer Socety Conference on Computer Vson and Pattern Recognton, pages , San Francsco, CA, USA.