Recognizing Primitive Interactions by Exploring Actor-Object States

Transcription

1 In IEEE Intenational Confeence on Patten Recognition (CVPR), Alaska, June Recognizing Pimitive Inteactions by Exploing Acto-Obect States Roman Filipovych Ealdo Ribeio Compute Vision and Bio-Inspied Computing Laboatoy Depatment of Compute Sciences Floida Institute of Technology, Melboune, FL 32901, USA Abstact In this pape, we pesent a solution to the novel poblem of ecognizing pimitive acto-obect inteactions fom videos. Hee, we intoduce the concept of acto-obect states. Ou method is based on the obsevation that at the moment of physical contact, both the motion and the appeaance of actos ae constained by the taget obect. We popose a pobabilistic famewok that automatically leans models in such constained states. We use oint pobability distibutions to epesent both acto and obect appeaances as well as thei intinsic spatio-tempoal configuations. Finally, we demonstate the applicability of ou appoach on seies of human-obect inteaction classification expeiments. 1. Intoduction In this pape, we focus on the poblem of ecognizing human-obect inteactions. Expeimental evidence suggests that motion infomation alone may not be sufficient to achieve highe-level easoning about activities that involve inteactions with obects. Indeed, appoaches fo ecognizing human acto-obect inteactions usually ely on additional contextual infomation povided in the fom of pedefined labels o landmak points [13, 4], o a numbe of electonic sensos [19]. Moeove, actions in such cases ae usually stongly constained and descibed with a pedefined set of semantic entities [16]. As an illustation of the main poblem addessed in this pape, let us conside the gasp a cup activity in Figue 1. In the figue, the hands appoach cups at diffeent speeds and having diffeent spatial popeties (e.g., clutched, in the fist sequence, and slightly open, in the second). The motion of diffeent actos pefoming the same inteaction activity may diffe consideably. Howeve, at the instant of physical contact, actos motions, appeaances, and acto-obect spatial configuations become constained by the taget obect. These constained motion and spatial configuations ae desciptive of the specific acto-obect inteaction. Figue 1. Constained acto-obect states. The liteatue on activity ecognition is extensive. In geneal, activity analysis methods focus on high-level analysis of activities. The analysis is usually semantic-based and aims at ecognizing complex activities such as geeting o pepaing a fench toast. Semantic level desciption can be accomplished by means of context-fee gammas [18], language-based models [16], and gaphical models [13, 12, 19, 3, 14]. An oveview of effots made in the aea of actions and inteactions ecognition fom a highlevel pespective can be found in [1]. Howeve, the ecognition of pimitive human-obect inteactions is still an open and elatively unexploed poblem. An effot in this diection was made by Gupta and Davis [10]. They pesented a Bayesian appoach that simultaneously estimates obect type, location, movement segments, and the effect of movements on obects. Howeve, inteactions hee ae limited to a pedefined sequence of motions (i.e., eaching, taectoy-like manipulation, and obect eaction). Peusum et al. [17] suggest the impotance of action undestanding in obect ecognition tasks. They use human activity to infe both the location and identity of obects. This idea was consistent with the esults obtained by Gupta and Davis [10]. Ou acto-obect inteaction ecognition method is inspied by ecent developments of pobabilistic constella- 1

2 tion models [15, 9]. We popose a pobabilistic gaphical model of pimitive acto-obect inteactions that combines infomation about the inteaction s dynamics, and actoobect static appeaances and spatial configuations. We tem these appeaances and oint acto-obect configuations as acto-obect states. In ou method, a spatio-tempoal pat-based model of inteaction s dynamics guides the pocess of discoveing consistent acto-obect states. Selected acto-obect states ae subsequently modeled within a static pat-based famewok. No manual input of obect o contextual infomation ae equied. Video sequences of pimitive inteactions ae the only input to the pogam. To the best of ou knowledge, this is the fist geneal visiononly method fo leaning pimitive human-obect inteactions without manual input of obect infomation. 2. Integating Acto-Obect Static States and Inteaction Dynamics We commence by defining the main components of ou model. An inteaction sequence V can be consideed to be the tempoal vaiation of a specific acto-obect static state. Let S = {S 1,..., S K } be a set of K discete static states sampled fom the space of all epesentative static states of an inteaction class. Let M be the spatio-tempoal infomation extacted fom the video. In this pape, this infomation is obtained using spatio-tempoal featues [7, 11]. Let X epesent simultaneously a paticula spatio-tempoal configuation of static states and inteaction dynamics. The tem acto-obect states will epesent the specific oint appeaance and spatial configuation of an acto and obect in an video fame. Let p(v X ) be the likelihood of obseving a paticula video sequence given that an inteaction is at some spatio-tempoal location. Fom Bayes theoem: p(x V) p(v X ) p(x ) p(s X ) }{{} p(m X }{{} ) p(x }{{} ) static states appeaance dynamics appeaance spatio-tempoal configuation The appeaance of both static states and dynamics ae assumed to be statistically independent. As a esult, the likelihood tem in (1) can be factoized into two components. Following the pat-based obect factoization suggested by Candall and Huttenloche [5], we assume that the pats spatio-tempoal aangement can be encoded into the pio pobability distibution while the likelihood function encodes thei appeaance. (1) {(a (i) 1, x(i) 1 ),..., (a(i) N Si, x (i) N Si )}, whee the pai (a (i), x(i) ) consists of the appeaance a and the spatio-tempoal location x of the subegion fo the model of static-state S i, espectively. Hee, N Si is the total numbe of subegions fo the static-state S i. The tempoal position of the static-state in the video sequence seves as the tempoal coodinate of the pats locations. Similaly, the dynamic infomation equied by ou model is epesented by a spase set of N M spatio-tempoal featues [7, 11] given by M = {(a (M) 1, x (M) 1 ),..., (a (M) N M, x (M) N M )}. Fo simplicity, we model both static-state and dynamic infomation using diected acyclic sta gaphs. This is simila to the pat-based obect model suggested by Fegus et al. [8]. Hee, a paticula vetex is assigned to be a landmak vetex (a (i), x (i) ) fo the static-state S i. A simila landmak vetex assignment is done fo the dynamics model, (a (M), x (M) ). The emaining vetices within each patial model ae conditioned on the coesponding landmak vetex. Figue 2(b) shows an example of pat-based models fo two static states of the gasp a cup inteaction. Finally, we obtain a multi-layeed tee-stuctued acto-obect inteaction model by conditioning the landmak vetices of the static-state model gaphs on the landmak vetex of the dynamics model gaph. An example of the inteaction model is shown in Figues 2(a,c). Gaph aows indicate vetices conditional dependences. The oint distibution of the spatial inteaction configuation can be deived fom the gaphical model in Figue 2(c): p(x ) = p(x (M) ) p(x (i) x (M) ) (2) S i S whee x (i) is the spatio-tempoal configuation of the static-state S i, and x (M) is the spatio-tempoal configuation of the dynamics model. The pobability distibutions that compose Equation 2 ae: p(x (M) ) = p(x (M) ) p(x(m) x (M) ) and p(x (i) x (M) ) = p(x (i) x (M) ) p(x(i) x(i) ). The patial models dependence is based solely on thei spatio-tempoal configuation within the global model (i.e., patial models appeaances ae statistically independent). Appeaance Model. Unde the appeaance independence assumption, the appeaance likelihood of the static-state S i can be witten as p(s i X ) = N Si p(a (i) x(i) ). Similaly, the dynamics model appeaance likelihood is p(m X ) = ). The likelihood tem in (1) becomes: NM p(a (M) x (M) Spatio-Tempoal Pio Model. We begin by assuming that a static-state S i S can be subdivided into a numbe of non-ovelapping subegions such that S i = p(v X ) = N K Si i p(a (i) NM x(i) ) p(a (M) x (M) ) (3)

3 Figue 2. (a) Spatio-tempoal pat-based model of inteaction. (b) Static states pat-based models; (c) Inteaction model gaph. 3. Leaning Inteactions The factoization in (2) and (3) allows fo a modula leaning pocedue given a set of unsegmented taining videos {V 1,..., V L }. The leaning steps ae the following (Figue 3). Leaning Step 1 - Leaning the Dynamics Model. We begin by modeling the pobabilities of subegion locations in p(x (M) ) fom (2). Conditional distibutions elating independent Gaussian distibutions ae also Gaussian [2]. As a esult, the conditional densities in the components of (2) take a paticulaly simple fom [2]. We extact a set of spatio-tempoal inteest points [7], and associate a spatiotempoal location with evey extacted subegion. We adopt the leaning pocess descibed by [5] to obtain an initial spatio-tempoal model and the optimal numbe of pats. An Expectation-Maximization (EM)-based pocedue is used to simultaneously efine the initial estimates of the appeaances and the spatial paametes. Figue 3. Inteaction leaning pocess. Leaning Step 2 - Ceating Initial Static-State Models. The input to this step ae inteest subegions extacted fom all fames of the taining sequences. Subegion locations ae given by the x- and y-coodinates of the subegion in the fame image, and the additional tempoal t-coodinate (i.e., fame-position). Howeve, some subegions may have simila appeaance acoss vaious fames (e.g., the appeaance of the toy ca emains unchanged duing the push a toy ca inteaction). Hence, simply clusteing appeaances would make indistinguishable some simila pats belonging to diffeent static states. Secondly, pats of a specific static-state model must have zeo tempoal vaiance as they natually belong to the same fame. We addess this initialization poblem by using the model of inteaction dynamics to estict the space of input subegions. Fo evey taining sequence, we obtain MAP locations of the dynamics model: ˆx (M) = ag max p(m x (M) )p(x (M) ) (4) x Fo evey sequence, the maximization in (4) esults in the location (ˆx, ŷ, ˆt) of the model s landmak pat. A set of initial samples t 0 = {t 1, t 2,..., t K } of tempoal displacements is ceated, whee t i is the static-state S i tempoal displacement with espect to the dynamics model s landmak node. Fo a given static-state S i, we select subegions fo which tempoal displacements ae between (t i t, t i + t), whee the constant t defines the fame ange containing the coesponding static state. These subegions ae subsequently used to obtain the candidate pats and initial paametes of the static-state S i models. Candidate pats selection is pefomed as in Step 1. We cluste pose subegions to fom initial pats of the undelying pose, and discad pats with low desciptiveness powe with espect to thei appeaance. We use the desciptiveness evaluation pocedue de-

4 scibed in [5]. Leaned pose pats ae oganized into a stagaph stuctue with the most desciptive pat as the landmak node. Initial spatio-tempoal paametes ae estimated fom the pats maximum likelihood locations as follows: ˆx (i) = ag max p(s i x (i) ) (5) x landmak nodes have the lagest tempoal vaiance when conditioned on the dynamics model. Also, when two intevals (t i t, t i + t) and (t t, t + t) ovelap, seveal instances of same static state may be leaned. Theefoe, we geedily select a subset of static-state models such that when conditioned on the landmak node of the dynamics model x (M) x (i), the mean locations of thei landmak nodes ae sepaated by a pedefined distance (e.g., one fame). 4. Classification and Infeence Inteaction ecognition can then be posed as a detection poblem. We seek fo the spatio-tempoal location in the video sequence that maximizes the posteio pobability of the inteaction s location: X = ag max p(x V) (6) X Figue 4. Initial static-state model. Dynamics model tempoal location ˆt is estimated fo each sequence. 2D inteest subegions ae extacted fom fames in the tempoal neighbohood ˆt + t i. Subegions ae clusteed to fom an initial static-state model fo the fist state. Subsequent static-state models ae similaly detemined. Leaning Step 3 - Ceating the Global Model. In this step, the initial static-state models ae combined with the model of inteaction dynamics into the global model of inteaction as in Figue 5(a). The initial paametes of the conditional distibutions p(x (i) x (M) ) that contibute to p(x (i) x (M) ) in Equation 2 ae estimated fom the MAP localizations of the static states in the taining sequences. Howeve, not only the initial static-state models contain noisy pats, but the paametes of the conditional distibutions p(x (i) x (M) ) ae vey inaccuate. We evise the paametes of the global model with the EM algoithm. The EM algoithm eestimates all model paametes including those of the dynamics model. Ou EM algoithm s Expectation-step MAP estimation consides only those model configuations whee static-state pats belong to the same fame. (See Section 4 fo details). Howeve, while the evised conditional distibutions become bette defined, the updated model still contains ovelapping pats. Ovelapping pats ae emoved and paametes evised with the EM algoithm once again. Figue 5(b) shows an example model with ovelapping pats emoved. Finally, fom the set of all leaned static-state models we etain only those that ae tempoally well-defined. This is done by puning the static-state model subgaphs whose We expect that pats epesenting the same static-state belong to the same video fame. As a esult, the global model MAP seach space can be significantly educed. Ou exact infeence algoithm is as follows: (i) Conside all states of the vaiable x (M) ; (ii) Conside all states of x (M) ; (iii) Conside all states of x (i). Fo evey state x (i) of x (i), conside only those states of x (i) that have the same tempoal coodinate as x (i) 5. Expeimental Results. Obtain the maximizing configuation. Inteactions Dataset. We acquied ou own dataset of videos with pimitive inteactions. Vision-based humanobect inteaction ecognition is a novel poblem with no widely available datasets. Complex scenaios wee delibeately chosen to motivate futue impovements of humanobect inteaction methods. Example fames fom ou inteaction dataset ae shown in Figue 6. The dataset consists of videos of eight diffeent acto-obect inteaction types pefomed by ten individuals in two diffeent scenaios. The inteactions ae gasp a cup, gasp a fok, touch a fok, gasp a spoon, touch a spoon, gasp a toy ca, touch a toy ca and push a toy ca. Evey individual pefomed inteactions with a unique set of obects (e.g., diffeent cups wee used by diffeent individuals in a gasp a cup inteaction). Sequences had clean and clutteed backgound, espectively. In the clutteed backgound scenaio, the backgound was changed fo evey individual and evey inteaction type. Any two inteaction types fom ou dataset diffe in one of the following thee aspects: (1) diffeent obects and diffeent motions (i.e., gasp a cup vs. touch a fok ); (2) simila obects and diffeent motions (i.e., gasp a fok vs. touch a fok ); and (3) diffeent obects and simila motions (i.e., gasp a fok vs. gasp a spoon ).

5 Figue 5. (a) Initial global model; (b) Nodes coesponding to ovelapping pats ae puned and only tempoally well-defined static states ae etained. Theefoe, we believe that this dataset is suitable fo evaluating ou method s validity. The above choice of inteactions was inspied by expeiments using functional neuoimaging in humans [6]. These expeiments evealed egions of the paietal lobes that ae specialized fo paticula visuomoto actions such as eaching and gasping. Videos wee acquied with a CCD camea at thity fames pe second. Fames wee downsized to pixels. opeato. The edge maps wee used to ceate the static-state models. Featues equied to ceate the dynamics model wee obtained using the spatio-tempoal inteest point detecto descibed in [7]. In all cases, the data dimensionality was educed using pincipal component analysis (PCA). (a) dynamics only (b) dynamics seveal-static states (c) dynamics only (d) dynamics seveal-static states (a) clean backgound Figue 7. Confusion matices. (a) clean scenaio (30.0% coect classification); (b) clean scenaio (70.0% coect classification); (c) clutteed scenaio (23.0% coect classification); (d) clutteed scenaio (54.0% coect classification). (b) clutteed backgound Figue 6. Sample fames fom ou inteaction dataset. Video Data Pepaation. In ou implementation, we began by obtaining a set of Gaussian smoothed edge-maps of squae patches centeed at the peviously detected inteest points. Inteest point locations wee detected using a Hais Classification. We pefomed fou expeiments. Fo each scenaio, we fist leaned a dynamics model of an inteaction. We show classification esults obtained fo each scenaio using dynamics infomation only as well as the combination of dynamics infomation and static-state infomation. A leave-one-out evaluation scheme was used fo classification. Labeling decisions wee made based only on best

6 (a) (b) (c) (d) (e) Figue 8. Leaned models of inteactions supeimposed at the detected locations. The plots epesent time-axis coss-sections of the dynamics-static state conditional distibutions. White bodes indicate static-state pats. Gayed ectangles indicate slices of the landmak spatio-tempoal subegion in the coesponding fames. Dak bode highlights the tempoal coodinate of the spatio-tempoal featue coesponding to the landmak node of dynamics. model match. The dynamics only model (i.e., no staticstate infomation included) achieved only 30% and 23% coect ecognition fo the clean and clutteed scenaios, espectively. Confusion matices fo these esults ae shown in Figue 7(a,c). Results suggest that motion alone was not sufficient to pefom accuate classification, and would have to be einfoced with the static-state models. Next, static-state infomation was included into the famewok. The initial static-state models wee obtained using t0 = {t1, t2,..., tk } as initial tempoal displacements (Leaning Step 2). The numbe of initial static-state models was set to five, and we selected t0 = { 14, 7, 0, 7, 14} and t = 4 fames. In ou appoach, the paamete that indiectly govens the numbe of static-state models is the

7 theshold on the tempoal distance between any two static states. This theshold was set to one fame. Consequently, stating fom the static-state with the lowest tempoal vaiance, we geedily etained static-state models while meeting the tempoal theshold. Confusion matices geneated by these classification esults ae shown in Figue 7(b,d). The figue also displays (in paentheses along side the inteaction types) the aveage numbe of static-state models etained in the global model fo a given inteaction. Oveall ecognition pefomance in these expeiments was 70.0% fo the clean backgound scenaio and 54.0% fo the clutteed backgound scenaio. This was significantly highe than the esults obtained by the dynamics-only model. Impovements ae still needed fo clutteed scenaios. Weak ecognition esults fo highly noisy and ambiguous inteactions wee expected. Figue 8 shows qualitative esults fo some inteaction models fom the latte expeiments. In the figue, models of seveal inteactions ae supeimposed on test sequences at the detected locations. The plots epesent the time-axis coss-sections of dynamics-static state conditional distibutions. Static-state pats ae epesented with the white bode. Gayed ectangles epesent slices of coesponding landmak spatio-tempoal subegion. The model pats appeaances shown in the Figue wee obtained using the closest vectos indices in the PCAeduced featue space. Dak bode highlights the tempoal coodinate of the spatio-tempoal featue coesponding to the landmak node of dynamics. 6. Conclusions We pesented a solution to a novel poblem of ecognizing pimitive acto-obect inteactions. The concept of acto-obect states was intoduced using a pobabilistic famewok. The poposed ecognition method combines static-states infomation with the video s motion dynamics to fom a global acto-obect inteaction model. Additionally, we intoduced a dataset of pimitive acto-obect inteactions and showed that ou appoach is effective fo human-obect inteaction classification. Ou cuent method is view-dependent. Howeve, single-view camea setup is a common scenaio fo many suveillance applications. In these scenaios, diection of motion can be quite simila (e.g., motion diection when opening a fidge is simila acoss diffeent agents). Also, ou appoach is independent of the type of inteest featues. Candidate pats could be extacted by sampling the video s spatio-tempoal subegions, and any existing inteest featue extaction method would wok. Finally, ou appoach is not limited to gestue-specific inteactions, and should wok well with full-body inteactions. Futue diections of investigation include a study and use of altenative appeaance models. Acknowledgments. Reseach suppoted by U.S. Office of Naval Reseach unde contact: N Refeences [1] J. K. Aggawal and S. Pak. Human motion: Modeling and ecognition of actions and inteactions. In 3DPVT, pages , Washington, DC, USA, [2] C. M. Bishop. Patten Recognition and Machine Leaning. Spinge-Velag New Yok, Inc., Secaucus, NJ, USA, [3] O. Boiman and M. Iani. Detecting iegulaities in images and in video. In CVPR, pages I: , [4] R. Chelappa, A. K. Roy-Chowdhuy, and S. K. Zhou. Recognition of Humans and Thei Activities Using Video. Mogan & Claypool Publishes, [5] D. J. Candall and D. P. Huttenloche. Weakly supevised leaning of pat-based spatial models fo visual obect ecognition. In ECCV (1), volume 3951 of LNCS, pages Spinge, [6] J. C. C. Culham and K. F. F. Valyea. Human paietal cotex in action. Cuent Opinion of Neuobiology, 2, Mach [7] P. Dollá, V. Rabaud, G. Cottell, and S. Belongie. Behavio ecognition via spase spatio-tempoal featues. In VS-PETS, Octobe [8] R. Fegus, P. Peona, and A. Zisseman. A spase obect categoy model fo efficient leaning and exhaustive ecognition. In CVPR (2), pages , [9] R. Filipovych and E. Ribeio. Combining models of pose and dynamics fo human motion ecognition. In ISVC 2007, Lake Tahoe, Nevada, USA, Novembe [10] A. Gupta and L. S. Davis. Obects in action: An appoach fo combining action undestanding and obect peception. In CVPR, pages 1 8, [11] I. Laptev and T. Lindebeg. Space-time inteest points. In ICCV, page 432, Nice, Fance, Octobe [12] B. Laxton, J. Lim, and D. Kiegman. Leveaging tempoal, contextual and odeing constaints fo ecognizing complex activities in video. In CVPR, pages 1 8, [13] N. Nguyen, S. Venkatesh, and H. Bui. Recognising behavious of multiple people with hieachical pobabilistic model and statistical data association. In BMVC, page III:1239, [14] J. Niebles, H. Wang, H. Wang, and L. Fei Fei. Unsupevised leaning of human action categoies using spatial-tempoal wods. In BMVC, Edinbugh, page III:1249, [15] J. C. Niebles and L. Fei-Fei. A hieachical model of shape and appeaance fo human action classification. In CVPR, Minneapolis, USA, June [16] S. Pak and J. K. Aggawal. Semantic-level undestanding of human actions and inteactions using event hieachy. In CVPRW, volume 1, page 12, Washington, DC, USA, [17] P. Peusum, G. West, and S. Venkatesh. Combining image egions and human activity fo indiect obect ecognition in indoo wide-angle views. In ICCV, volume 1, pages 82 89, [18] M. S. Ryoo and J. K. Aggawal. Recognition of composite human activities though context-fee gamma based epesentation. In CVPR, pages , [19] J. Wu, A. Osuntogun, T. Choudhuy, M. Philipose, and J. Rehg. A scalable appoach to activity ecognition based on obect use. In ICCV, 2007.