Detection of Human Actions from a Single Example

Transcription

1 Detecton of Human Actons from a Sngle Example Hae Jong Seo and Peyman Mlanfar Electrcal Engneerng Department Unversty of Calforna at Santa Cruz 1156 Hgh Street, Santa Cruz, CA, {rokaf,mlanfar}@soe.ucsc.edu Abstract We present an algorthm for detectng human actons based upon a sngle gven vdeo example of such actons. The proposed method s unsupervsed, does not requre learnng, segmentaton, or moton estmaton. The novel features employed n our method are based on space-tme locally adaptve regresson kernels. Our method s based on the dense computaton of so-called space-tme local regresson kernels (.e. local descrptors) from a query vdeo, whch measure the lkeness of a voxel to ts spatotemporal surroundngs. Salent features are then extracted from these descrptors usng prncpal components analyss (PCA). These are effcently compared aganst analogous features from the target vdeo usng a matrx generalzaton of the cosne smlarty measure. The algorthm yelds a scalar resemblance volume; each voxel ndcatng the lkelhood of smlarty between the query vdeo and all cubes n the target vdeo. By employng non-parametrc sgnfcance tests and non-maxma suppresson, we accurately detect the presence and locaton of actons smlar to the gven query vdeo. Hgh performance s demonstrated on a challengng set of acton data [8] ndcatng successful detecton of multple complex actons even n the presence of fast motons. 1. Introducton A huge and growng number of vdeos are avalable onlne today. Human actons are one of the most mportant parts n moves, TV shows, and personal vdeos. Analyss of human actons n vdeos s consdered a very mportant component n computer vson systems because of such applcatons as content-based vdeo retreval, vsual survellance, analyss of sports events and more. The generc problem of nterest addressed n ths paper can be brefly descrbed as follows: We are gven a sngle query vdeo of an acton of nterest (for nstance a short Fgure 1. (a) A hand-wavng acton and possbly smlar actons (b) Gven a query vdeo, we want to detect/localze actons of nterest n a target vdeo T. T can be dvded nto a set of overlappng cubes ballet turn), and we are nterested n detectng smlar actons wthn other target vdeos. Detectng human actons from vdeo s a very challengng problem due to the fact that physcal body moton can look very dfferent dependng on the context: 1) smlar actons wth dfferent clothes, or n dfferent llumnaton and background can result n a large appearance varaton; 2) the same actons performed by two dfferent people may look dssmlar n terms of acton speed or frame rate of the vdeo (See Fg. 1 (a)). There have been many efforts to model and recognze human actons broadly by means of parametrc tme-seres approaches, frame-by-frame nonparametrc approaches, and volumetrc approaches. We refer the nterested reader to [13] and references theren for a good summary. Volumetrc approaches tend to outperform the other two approaches. These volumetrc methods do not requre background subtracton, moton estmaton, and complex models of body confguraton and knematcs. They tolerate varatons n appearance, scale, rotaton, and movement to some extent. Methods such as those n [5, 8] whch am at recognzng actons based solely on one query (what we shall call tranngfree) are very useful for vdeo retreval from the web. In these methods, a sngle query vdeo s provded by users and every gallery vdeo n the database s compared wth the sngle query, posng a vdeo-to-vdeo matchng problem. Inspred by ths trend toward tranng-free acton analyss, ths paper presents a novel tranng-free human acton IEEE 12th Internatonal Conference on Computer Vson (ICCV) /09/$ IEEE

2 Fgure 2. Acton detecton system overvew (There are broadly three stages.) detecton framework. Our proposed method s based on the calculaton and use of what we call space-tme local regresson kernels whch are local weghts computed drectly from the gven pxels n both the query and target vdeos. The orgnal motvaton to use these local regresson kernels s the earler successful work on adaptve kernel regresson for mage denosng, nterpolaton [9], deblurrng [10], and (2-D) generc object detecton [6]. Takeda et al. [11] extended the kernel regresson framework to super-resoluton by ntroducng space-tme local steerng kernels whch capture the essental local behavor of a spato-temporal neghborhood. The space-tme local steerng kernel (3-D LSK) s fundamentally based on the comparson of neghborng pxels n both space and tme, thus t mplctly contans nformaton about the local moton of the pxels across tme, thus requrng no explct moton estmaton. The space-tme local steerng kernel s defned as follows: { } det(cs) (xs x) T C s(x s x) K(x s x) = exp, (1) h 2 2h 2 where x s =[x 1,x 2,t] T s s the space-tme coordnates, s [1,,P ], hs a global smoothng parameter, P s the total number of samples n a space-tme local analyss wndow around a sample poston at x, and the matrx C s R (3 3) s a covarance matrx estmated from a collecton of frst dervatves along spatal (x 1,x 2 ) and temporal (t) axes. The covarance matrx C s modfes the shape and sze of the local kernel n a way whch robustly encodes the space-tme local geometrc structures present n vdeos. Normalzaton of ths kernel functon yelds robustness to llumnaton, contrast, and color dfferences. For a more n depth analyss on local steerng kernels, we refer the nterested reader to [6, 9, 10, 11]. Very recently, Shechtman and Iran [7] ntroduced a space-tme local self-smlarty descrptor for acton detecton and showed performance mprovement over ther prevous approach [8]. Ths (ndependently derved) local space-tme self-smlarty descrptor s a specal case of our space-tme local steerng kernel and s also related to a number of other local data adaptve metrcs such as Optmal Space-Tme Adaptaton (OSTA) [2] and Non-Local Means (NLM) [3] whch have been used very successfully for vdeo restoraton n the mage processng communty. Whle the method proposed by Shechtman and Iran [7] s related to our method, ther approach fundamentally dffers from ours n the followng respects: 1) Snce the calculaton of space-tme local steerng kernels s stable n the presence of uncertanty n the data [9], our approach s robust even n the presence of nose; 2) As opposed to [7] flterng out non-nformatve descrptors n order to reduce the tme complexty, we automatcally obtan the most salent feature volumes by applyng Prncpal Components Analyss (PCA) to a collecton of 3-D LSKs. From a practcal standpont, t s mportant to note that the proposed framework operates usng a sngle example of an acton of nterest to fnd smlar matches; does not requre any pror knowledge (learnng) about actons beng sought; and does not requre any pre-processng step or segmentaton of the target vdeo. Fg. 2 shows an overvew of our proposed framework for acton detecton. To summarze the operaton of the overall algorthm, we frst compute the normalzed space-tme local steerng kernels (3-D LSKs) W, W T from both and T. In the second stage, we obtan the salent feature volumes F, F T by projectng the descrptors W, W T to a projecton space A derved from W. In the thrd stage, we compare the feature volumes F T (=a chunk of F T at th poston) and F usng the Matrx Cosne Smlarty measure. The fnal output s gven after a sequence of sgnfcance tests, followed by non-maxma suppresson [4]. Ths paper s organzed as follows. In the next secton, we provde further techncal detals about the varous steps outlned above. In Secton 3, we demonstrate the performance of the system wth expermental results, and we conclude ths paper n Secton Techncal Detals As outlned n the prevous secton, our approach to detect actons conssts broadly of three stages. Assume that we are gven a target vdeo T and that we have a query vdeo, where s generally smaller than T. The task at hand s to detect and locate cubes of T that are smlar to. The frst step s to calculate space-tme local steerng kernels (3-D LSKs). To be more specfc, 3-D LSK functon K(x s x) s densely calculated and normalzed as follows: W I (x s x)= K I(x s x) P s=1 K I(x s x), { s =1,,P, I {, T }. Fg. 3 llustrates what the normalzed versons of 2-D LSKs and 3-D LSKs n varous regons look lke. (2) 1966

3 Fgure 3. (a) Examples of 2-D LSK n varous regons. (b) Examples of space-tme local steerng kernel (3-D LSK) n varous regons. Note that key frame means the frame where the center of 3-D LSK s located. In order to organze W (x s x) s and W T (x s x) s, whch are densely computed from and T, let W,W T be matrces whose columns are vectors w,w T, whch are column-stacked (rasterzed) versons of W (x s x),w T (x s x) respectvely: W = [w, 1, w] n R P n, W T = [wt 1,, w nt T ] RP nt. (3) where n and n T are the number of 3-D LSKs n the query vdeo and the target vdeo T respectvely. As descrbed n Fg. 2, the next step s to apply PCA to W for dmensonalty reducton and to retan only ts salent characterstcs. Applyng PCA to W we can retan the frst (largest) d prncpal components 1 whch form the columns of a matrx A R P d. Next, the lower dmensonal features are computed by projectng W and W T onto A : F = [f, 1, f]=a n T W R d n, F T = [ft 1,, f nt T ]=AT T W T R d nt. (4) Fg. 4 llustrates the prncpal components n A and shows what the features F, F T look lke for the ballet vdeo case. Very recently, Al and Shah [1] proposed a set of knematc features that extract dfferent aspects of moton dynamcs present n the optcal flow. They obtaned bags of knematc modes for acton recognton by applyng PCA to a set of knematc features. We dfferentate our proposed method from [1] n the sense that 1) moton nformaton s mplctly contaned n 3-D LSK whle [1] explctly computes optcal flow; 2) Background subtracton was used as a pre-processng step whle our method s fully automatc; 3) [1] employed multple nstance learnng to a set of all knematc modes n the dataset whle our proposed method does not nvolve any tranng phase. 1 Typcally, d s selected to be a small nteger such as 3 or 4 so that 80 to 90% of the nformaton n the LSKs would be retaned. (.e., d=1 λ P=1 0.8 (to 0.9) where λ λ are the egenvalues.) Fgure 4. Ballet acton : A s learned from a collecton of 3-D LSKs W, and Feature row vectors of F and F T are computed from query and target vdeo T respectvely. Egenvectors and feature vectors were transformed to volume and up-scaled for llustraton purposes. The next step n the proposed framework s the measurement of a dstance between the computed features F, F T. For ths purpose, we employ the nonparametrc detecton framework [6] based on Matrx Cosne Smlarty. The Matrx Cosne Smlarty (MCS) between two feature matrces F, F T whch consst of a set of vectors can be defned as the Frobenus nner product between two normalzed matrces as follows: F T ρ =< F, F T > F = trace( F T ) [ 1, 1], (5) F F F T F f 1 f n f T 1,, ] and F F F F T =[,, F F T F f n T F T F ]. where, F =[ Equaton 5 can be rewrtten as a weghted average of the cosne smlartes ρ(f, f T ) between each par of correspondng feature vectors (.e., columns) n F, F T as follows: n f l T f l T n ρ = = ρ(f F F F T, l f l f l T ) f T l. (6) F F F F T F l=1 l=1 The weghts are represented as the product of f l T f l F F and F T F whch ndcate the relatve mportance of each feature n the feature sets F, F T. Ths measure 2 not only generalzes the cosne smlarty, but also overcomes the dsadvantages of the conventonal Eucldean dstance whch 2 We compute ρ over M target cubes and ths can be effcently mplemented by column-stackng the matrces F, F T and smply computng the cosne smlarty between two long column vectors as follows: ρ n d l=1 j=1 n l=1 f (l,j) f(l,j) T dj=1 f (l,j) 2 nl=1 dj=1 f (l,j) T 2 = ρ(colstack(f ), colstack(f T )) [ 1, 1], where f (l,j),f(l,j) T are elements n l th vector f l and f T l respectvely, and colstack( ) means an operator whch column-stacks a matrx., 1967

4 f l f l T f l f l T Fgure 5. Left: Examples of A) ρ(f, l ft l ) : cosne smlarty, B) F F F T F : weghts, and A B) ρ(f, l ft l ) F F F T F : weghted cosne smlarty. Note that query and target are same as those n Fg. 2(Left). Rght: two sgnfcance tests and non-maxma suppresson [4]are descrbed. s senstve to outlers. Fg. 5(Left) shows examples of the computaton of the MCS, whch ndcate that t provdes a relable measure of smlarty. It s worth notng that Shechtman and Iran [8] proposed 3-D volume correlaton score (global consstency measure between query and target cube) by computng a weghted average of local consstency measures. The dffculty wth that method s that local consstency values should be explctly computed from each correspondng subvolume of the query and target vdeo. Furthermore, the weghts to calculate a global consstency measure are based on a sgmod functon, whch s somewhat ad-hoc. Here, we clam that our measure, MCS s better motvated, more approprate, and more general than ther global consstency measure for acton detecton. The next step s to generate a so-called resemblance volume (RV), whch wll be a volume of voxels ndcatng the lkelhood of smlarty between and T at each spatotemporal poston. As for the fnal test statstc comprsng the values n the resemblance volume, we use the proporton of shared varance (ρ 2 ) to that of the resdual varance (1 ρ 2 ). More specfcally, RV s computed usng the functon f( ) as follows: ρ2 RV : f(ρ )= 1 ρ 2, =0,,M 1. (7) From a quanttatve pont of vew, we note that f(ρ ) s essentally the Lawley-Hotellng Trace statstc [12], whch s used as an effcent test statstc for detectng correlaton between two data sets. Next, we employ a two-step sgnfcance test as shown n Fg 5 (Rght). The frst s an overall threshold (τ 0 )on the RV to decde whether there s any suffcently smlar acton present n the target vdeo at all. If the answer s yes at suffcently hgh confdence, we would then want to know how many actons of nterest are present n the target vdeo and where they are. Therefore, we need two thresholds: an overall threshold 3 τ o as mentoned above, and a threshold 4 τ to detect the (possbly) multple occurrences of the same acton n the target vdeo. After the two sgnfcance tests wth τ o,τ are performed, we employ the dea of non-maxma suppresson [4] for the fnal detecton. We take the volume regon wth the hghest f(ρ ) value and elmnate the possblty that any other acton s detected wthn some radus 5 of the center of that volume regon agan. Ths enables us to avod multple false detectons of nearby actons already detected. Then we terate ths process untl the local maxmum value falls below the threshold τ. Fg. 5 (Rght) shows a graphcal llustraton of sgnfcance tests and non-maxma suppresson [4]. For the sake of completeness, the overall pseudo-code for the algorthm s gven n Algorthm Expermental Results Our method detects the presence and locaton of actons smlar to the gven query and provdes a seres of bound- 3 In a typcal scenaro, we set the overall threshold τ o to be, for nstance, 0.96 whch s about 50% of varance n common (.e., ρ 2 =0.49). In other words, f the maxmal f(ρ ) s just above 0.96, we decde that there exsts at least one acton of nterest. 4 We employ the dea of nonparametrc testng. We compute an emprcal probablty densty functon (PDF) from M samples f(ρ ) and we set τ so as to acheve, for nstance, a 99 % (α =0.99) sgnfcance level n decdng whether the gven values are n the extreme (rght) tals of the dstrbuton. Ths approach s based on the assumpton that n the target vdeo, most cubes do not contan the acton of nterest (n other words, acton of nterest s a relatvely rare event), and therefore, the few matches wll result n values whch are n the tals of the dstrbuton of f(ρ ). 5 The sze of ths excluson regon wll depend on the applcaton at hand and the characterstcs of the query vdeo. 1968

5 Algorthm 1 Tranng-free generc acton detecton : uery vdeo, T : Target vdeo, τ o : Overall threshold, α : Confdence level, P : Sze of space-tme local steerng kernel (3-D LSK) cube. Stage1 : Compute Descrptors Construct W, W T whch are a collecton of normalzed 3-D LSK assocated wth, T. Stage2 : Feature Representaton 1) Apply PCA to W and obtan projecton space A from ts top d egenvectors. 2) Project W and W T onto A to construct F and F T. Stage3 1) Compute Matrx Cosne Smlarty for every target cube T, where [0,,M 1] do ρ =< F F T, > F F F T F and (RV) : f(ρ F )= ρ2 1 ρ 2. end for Then, fnd max f(ρ ). 2) Sgnfcance tests ) If max f(ρ ) >τ o, go on to the next test. Otherwse, there s no acton of nterest n T. ) Threshold RV by τ whch s set to acheve 99 % confdence level (α = 0.99) from the emprcal PDF of f(ρ ). 3) Non-maxma suppresson Apply non-maxma suppresson [4] to RV untl the local maxmum value s below τ. ng cubes wth resemblance volume embedded around detected actons. Note that no background/foreground segmentaton s requred n the proposed method. Ths method can also handle modest amount of varatons n rotaton (up to ±15 degree), and spatal and temporal scale change (up to ±20%). In practce, once gven and T, we downsample both and T by some factor of (3, here) n order to reduce the tme-complexty. We then compute 3-D LSK of sze 3 3 (space) 7 (tme) as descrptors so that every space-tme locaton n and T yelds a 63-dmensonal local descrptor W and W T respectvely. The smoothng parameter h for computng 3-D LSKs was set to 2.1. We end up wth F, F T by reducng dmensonalty from 63 to d =4and then, we obtan RV by computng the MCS measure between F, F T. The threshold τ for each test example was determned by the confdence level α =0.99. We appled our method to 3 dfferent examples :.e. detectng 1) walkng people, 2) ballet turn actons, and 3) multple actons n one vdeo. Shechtman and Iran [8] have tested ther method on these vdeos usng the same query and [5, 7] also tested ther methods on some of these vdeos. We acheved smlar (or even better) performance as compared to the methods n [5, 7, 8]. It s worth notng here that the other acton detecton methods [5, 7, 8] dd not provde ether threshold values or descrbe how they selected threshold values n reportng detecton performance. On the other hand, the threshold values are automatcally chosen n our algorthm wth respect to the confdence level as explaned earler. Fg. 6(A) shows the results of searchng for nstances of walkng people n a target beach vdeo (460 frames of pxels). The query vdeo contans a very short walkng acton movng to the rght (14 frames of pxels) and has a background context whch s not the beach scene. In order to detect walkng actons n ether drecton, we used two queres ( and ts mrror-reflected verson) and generated two RVs. By votng the hgher score among values from two RVs at every space-tme locaton, we ended up wth one RV whch ncludes correct locatons of walkng people n the correct drecton. Fg. 6(A) (a) shows a few sampled frames from. In order to provde better llustraton of T, we dvded T nto 3 non-overlappng sectons. Fg. 6(A) (b) and (c) represent each part of T and ts correspondng RV respectvely. Red color represents hgher resemblance whle blue color denotes lower resemblance values. Fg. 6(A) (d) and (e) show a few frames from T, wth RV and boundng boxes supermposed on them respectvely. Fg. 6 (B) shows the results of detectng ballet turnng acton n a target ballet vdeo (284 frames of pxels). The query vdeo contans a sngle turn of a male dancer (13 frames of pxels). Fg. 6(B) (a) shows a few sampled frames from. Next, Fg. 6(B) (b) and (c) represent each part of T and ts correspondng RV respectvely. Fg. 6(B) (d) and (e) show a few frames from T wth resemblance volumes supermposed on t respectvely. Most of the turns of the two dancers (a male and a female) were detected even though ths vdeo contans very fast movng parts and relatvely large varablty n spatal scale and appearance (the female dancer wearng a skrt) as compared to the gven query. We observed that one of the female dancer turnng actons was mssed because of large spatal scale varaton as compared to the gven. However, we can easly deal wth ths problem by ether adjustng the sgnfcance level or usng mult-scale approach as done n [6]. The detecton result of the proposed method on ths vdeo outperforms that n [5, 8] and compares favorably to that n [7]. Fg. 6(C) shows the results of detectng 4 dfferent actons ( walk, wave, clap, and jump ) whch occur smultaneously n a target vdeo (120 frames of pxels). Four query vdeos were matched aganst the target vdeo ndependently. Fg. 6(C) (a) and (b) show a few sampled frames from and T respectvely. Whte boxes n Fg. 6(C) (a) represent actual regons used for the query. The resultng RVs are shown n Fg. 6(C) (c). In all the above examples, we used the same parameters. It s evdent, based on all the results above, that the proposed tranng-free acton detecton based on 3-D LSK works well and s robust to modest varatons n spato-temporal scale. Our system s desgned wth detecton accuracy as a hgh prorty. A typcal run of the object detecton system takes a lttle over 1 mnute on a target vdeo T (50 frames of pxels, Intel Pentum CPU 2.66 Ghz machne) usng a query (13 frames of ). Most of the runtme s taken up by the computaton of MCS (about 9 seconds, and 16.5 seconds for the computaton of 3-D LSKs from and T respectvely, whch needs to be computed 1969

6 Fgure 6. Results searchng for (A) walkng person on the beach, (B) ballet turn on the ballet vdeo, and (C) multple actons. (A,B): (a) query vdeo (a short walk clp) (b) target vdeo (c) resemblance volumes (RV) (d) a few frames from T (e) frames wth resemblance volume on top of t. (C): (a) four dfferent short vdeo queres. Note that whte boxes represent actual query regons (b) target vdeo T (c) resemblance volumes (RV)s wth respect to each query. only once.) There are many factors that affect the precse tmng of the calculatons, such as query sze, complexty of the vdeo, and LSK sze. Our system runs n Matlab but could be easly mplemented usng mult-threads or parallel programmng as well as General Purpose GPU for whch we expect a sgnfcant gan n speed. 4. Concluson and Dscusson In ths paper, we have proposed a novel acton detecton algorthm by employng space-tme local steerng kernels (3-D LSKs); and by usng a tranng-free nonparametrc detecton scheme based on Matrx Cosne Smlarty (MCS). The proposed method can automatcally detect n the target vdeo the presence, the number, as well as locaton of actons smlar to the gven query vdeo. The proposed method s practcally appealng because t s nonparametrc. The proposed framework s general enough as to be extendable to acton categorzaton usng a nearest neghbor classfer along wth an automatc acton croppng method as smlarly done n [5]. Improvement of the computatonal complexty of the proposed method s also a drecton of future research worth explorng. 5. Acknowledgment Ths work was supported by AFOSR Grant FA References [1] S. Al and M. Shah. Human acton recognton n vdeos usng knematc features and multple nstance learnng. Accepted for publcaton n IEEE Transactons on Pattern Analyss and Machne Intellgence (PAMI), [2] J. Boulanger, C. Kervrann, and P. Bouthemy. Space-tme adaptaton for patch-based mage sequence restoraton. IEEE Transactons on Pattern Analyss and Machne Intellgence, 29: , June [3] A. Buades, B. Coll, and J. M. Morel. Nonlocal mage and move denosng. Internatonal Journal of Computer Vson, 76(2): , [4] F. Devernay. A non-maxma suppresson method for edge detecton wth sub-pxel accuracy. Techncal report, INRIA, (RR-2724), [5] H. Nng, T. Han, D. Walther, M. Lu, and T. Huang. Herarchcal space-tme model enablng effcent search for human actons. IEEE Transactons on Crcuts and Systems for Vdeo Technology, n press, [6] H. J. Seo and P. Mlanfar. Tranng-free, generc object detecton usng locally adaptve regresson kernels. Accepted for publcaton n IEEE Transactons on Pattern Analyss and Machne Intellgence, June [7] E. Shechtman and M. Iran. Matchng local self-smlartes across mages and vdeos. In Proc. of IEEE Conference on Computer Vson and Pattern Recognton (CVPR), pages 1 8, June [8] E. Shechtman and M. Iran. Space-tme behavor-based correlaton -or- how to tell f two underlyng moton felds are smlar wthout computng them? IEEE Transactons on Pattern Analyss and Machne Intellgence, 29: , November [9] H. Takeda, S. Farsu, and P. Mlanfar. Kernel regresson for mage processng and reconstructon. IEEE Transactons on Image Processng, 16(2): , February [10] H. Takeda, S. Farsu, and P. Mlanfar. Deblurrng usng regularzed locally-adaptve kernel regresson. IEEE Transactons on Image Processng, 17: , Aprl [11] H. Takeda, P. Mlanfar, M. Protter, and M. Elad. Superresoluton wthout explct subpxel moton estmaton. Accepted for publcaton n IEEE Transactons on Image Processng, [12] M. Tatsuoka. Multvarate Analyss. Macmllan, [13] P. Turaga, R. Chellappa, V. Subrahmanan, and O. Udrea. Machne recognton of human actvtes: A survey. IEEE Transactons on Crcuts and Systems for Vdeo Technology, 18: , November