Joint Recognition of Multiple Concurrent Activities using Factorial Conditional Random Fields

Transcription

1 Jont Recognton of Multple Concurrent Actvtes usng Factoral Condtonal Random Felds Tsu-yu Wu and Cha-chun Lan and Jane Yung-jen Hsu Department of Computer Scence and Informaton Engneerng Natonal Tawan Unversty Abstract Recognzng patterns of human actvtes s an mportant enablng technology for buldng ntellgent home envronments. Exstng approaches to actvty recognton often focus on mutually exclusve actvtes only. In realty, people routnely carry out multple concurrent actvtes. It s therefore necessary to model the co-temporal relatonshps among actvtes. In ths paper, we propose usng Factoral Condtonal Random Felds (FCRFs) for recognton of multple concurrent actvtes. We desgned experments to compare our FCRFs model wth Lnear Chan Condton Random Felds (LCRFs) n learnng and performng nference wth the MIT House n data set, whch contans annotated data collected from multple sensors n a real lvng envronment. The expermental results show that FCRFs can effectvely mprove the F-score n actvty recognton for up to 8% n the presence of multple concurrent actvtes. Introducton Recognzng patterns of human actvtes s an essental buldng block for provdng context-aware servces n an ntellgent envronment, ether at home or n the work place. Take as an example, the applcaton of elder care n a home settng, actvty recognton enables the ntellgent envronment to montor an elder s actvtes of daly lvng and to offer just-n-tme assstance, playng the role of a responsble care gver. Automatc actvty recognton presents dffcult techncal challenges. To tackle the problem, one often makes the smplfyng assumpton by focusng on mutually exclusve actvtes only. In other words, most exstng approaches do not take nto account the co-temporal relatonshp among multple actvtes. Such solutons are not accurate enough n practce as people routnely carry out multple actvtes concurrently n ther daly lvng. House n s an ongong project by the Department of Archtecture at Massachusetts Insttute of Technology (Intlle et al. 2006b) that offers a lvng laboratory for the study of ubqutous technologes n home settngs. Hundreds of sensng components are nstalled n nearly every part of the home, whch s a one-bedroom condomnum. The lvng lab s beng occuped by volunteer subjects who agree to lve n Copyrght c 2007, Assocaton for the Advancement of Artfcal Intellgence ( All rghts reserved. the home for varyng lengths of tme. Sensor data are collected as the occupants nteract wth dgtal nformaton n ths naturalstc lvng envronment. As a result, the House n data set (Intlle et al. 2006a) s potentally a good benchmark for research on actvty recognton. In ths paper, we use the data set as the materal to desgn our experment. Our analyss of the House n data set reveals a sgnfcant observaton. That s, n a real-home envronment, people often do multple actvtes concurrently. For example, the partcpant would throw the clothes nto the washng machne and then went to the ktchen dong some meal preparaton. As another example, the partcpant had the habt of usng the phone and the computer at the same tme. As s shown n fgure 1, there are many cases n the House n data set n whch people would perform multple actvtes concurrently. Fgure 1: Annotated actvtes n the House n data set from 10 AM to 1 PM. Tradtonally, research on actvty recognton has focused on dealng wth mutually exclusve actvtes. In other words, they assumed that there s at most one actvty occurrng at every tme pont. For any gven tme pont, the prmary concern s to label wth the most probable actvty. Gven that multple concurrent actvtes do exst, we should no longer make the assumpton of mutually exclusve actvtes. In addton, we should not gnore the fact that multple actvtes nteract wth each other, as t could be of great help to take such relatonshp nto account. Condtonal Random Felds (CRFs) (Lafferty, McCallum, 82

2 & Perera 2001) provde a powerful probablstc framework for labellng and segmentng structured data. By defnng a condtonal probablty dstrbuton over label sequences gven a partcular observaton sequence, CRFs relax the Markov ndependence assumpton requred by Hdden Markov Models (HMMs). The CRF model has been appled to learnng patterns of human behavor (Cheu, Lee, & Kaelblng 2006; Smnchsescu, Kanauja, & Metaxas 2006; Lao, Fox, & Kautz 2007). Nevertheless, as mentoned above, prevous research gnored the possble nteractons between multple concurrent actvtes. To address ths problem, we advocate usng a factoral condtonal random felds (FCRFs) (Sutton & McCallum 2004) model to conduct nference and learnng from patterns of multple actvtes. Compared wth basc CRFs, FCRFs utlze a structure of dstrbuted states to avod the exponental complexty problem. In addton, the FCRF model accommodates the relatonshp among multple concurrent actvtes and can effectvely label ther states. Ths paper s organzed as follows. At frst, we ntroduce the CRFs and FCRFs model. And then we present FCRFs for multple concurrent actvtes recognton ncludng the model desgn, the nference algorthm, and the learnng method. Fnally, we descrbe our experment for performance evaluaton of ths model, followed by the concluson and future work. Condtonal Random Felds CRFs are undrected graphcal models condtoned on observaton sequences whch have been successfully appled to sequence labellng problems such as bonfomatcs (Sato & Y. 2005; Lu et al. 2005), natural language processng (Lafferty, McCallum, & Perera 2001; Sha & Perera 2003; Sutton & McCallum 2004; Sarawag & Cohen 2005) and computer vson (Smnchsescu, Kanauja, & Metaxas 2006; Vshwanathan et al. 2006). Unlke generatve models such as Dynamc Bayesan Networks (DBNs) and HMMs, CRFs are condtonal models that relax the ndependence assumpton of observatons and avod enumeratng all possble observaton sequences. Maxmum Entropy Markov Models (MEMMs) are an alternatve condtonal models, but they suffer from the label bas problem (Lafferty, McCallum, & Perera 2001) due to per-state normalzaton. In contrast, CRFs are undrected structures and globally normalzed, thereby solvng the label bas problem. Let G be an undrected graph consstng of two vertex sets X and Y, where X represents the set of observed random varables and Y represents the set of hdden random varables condtoned on X. Gven graph G, let C be the set of maxmum clques, where each clque ncludes vertces X c X and Y c Y. For a gven observaton sequence Y, CRFs defne the condtonal probablty by the potental functon Φ(X c, Y c ) such that P (Y X) = 1 Φ(X c, Y c ) Z(X) c C where Z(X) s the normalzaton constant. Fgure 2: An LCRF example for actvty recognton. Fgure 2 shows an example of applyng lnear chan CRFs (LCRFs) to actvty recognton. We can represent the states of actvtes as a tme sequence. Thus, we have a set of hdden varable nodes Y = {Y 1, Y 2, Y 3...} standng for the actvty sequence. Gven the observaton sequence X, the dependency can be defned by the feature functons upon the clques. Several researchers have appled CRFs to actvty recognton (Cheu, Lee, & Kaelblng 2006; Smnchsescu, Kanauja, & Metaxas 2006; Lao, Fox, & Kautz 2007), but they dd not address the ssue of multple concurrent actvtes. To recognze multple concurrent actvtes, t s possble to construct a unque model for each ndvdual actvty. However, ths approach gnores the relatonshp between dfferent actvtes. For example, the House n data set (Intlle et al. 2006b) shows that the partcpant often used the phone and the computer concurrently n the experment. In addton, a person typcally cannot sleep and watch TV at the same tme. To take the co-temporal relatonshp between multple actvtes nto account, we may treat each combnaton of multple actvtes as a new actvty. However, the model complexty of ths approach grows exponentally wth the number of actvtes to be recognzed. As a result, nference and learnng may become computatonally ntractable when the number of actvtes s large. Factoral CRFs (FCRFs) (Sutton & McCallum 2004), whch are lke Factoral HMMs (FHMMs) (Ghahraman & Jordan 1997), suggest a structure of dstrbuted states to avod the exponental complexty problem. In addton, FCRFs can model the dependency between multple concurrent actvtes by ntroducng co-temporal connectons. An FCRFs Model for Multple Concurrent Actvtes Recognton Model Desgn Let Y t be a random varable whose value represents the state of actvty at tme t and Y t = {Y1 t, Y2 t... YN t } where N s the number of actvtes to be recognzed. In a total tme nterval T, let Y = {Y 1, Y 2... Y T } be a sequence of vector Y t. We can defne the observaton sequence X = {X 1, X 2... X T } n the same way. Suppose that there s a graph G = {V, E}. Let each element n Y t 83

3 and X t be a vertex n G. Edges n G represent the relatonshps between these random varables. Fgure 3 shows a sample FCRF model for recognton of three concurrent actvtes. The FCRF s represented n a dynamc form by unrollng the structure of two tme slces. There are two sets of edges standng for dfferent relatonal meanngs. The edge set E c, whch ncludes pars of actvty varables n the same tme slce, represents the co-temporal relatonshp; the edge set E t, whch ncludes pars of actvty varables across tme slces, represents the temporal relatonshp. Fgure 3: An FCRF example of three concurrent actvtes. We defne par-wse potental functons Φ c (Y t, Y j t, X) for each edge (Y t, Y j t) n E c and Φ t (Y t, Y t+1, X) for each edge (Y t, Y t+1 ) n E t. And we defne the local potental functon Ψ(Y t t, X) for each vertex Y n G. The FCRFs wll be determned as ( T 1 p(y X) = 1 Z(X) t=1 N T N t=1,j Φ t (Y t, Y t+1, X) ) ( T k Φ c (Y t, Y t t=1 N j, X) Ψ t (Y t, X) where Z(X) s the normalzaton constant. The potental functons Φ c, Φ t and Ψ are then defned by a set of feature functons f = {f 1, f 2... f K } and ther correspondng weghts λ = {λ 1, λ 2... λ K } such that ( ) Φ c (Y t, Yj t, X) = exp λ k f k (Y t, Yj t, X) Φ t (Y t, Y t+1, X) = exp Ψ(Y t, X) = exp ( ( k k λ k f k (Y t, Y t+1, X) λ k f k (Y t, Y t, X) Now, we have formally defned the FCRFs. ) ) ) Inference Algorthm Gven any observaton sequence O, there are two knds of nference tasks of concern. One task s to compute the margnal probablty of each node par, whch wll be used n the learnng algorthm to be ntroduced later. The other task s to perform MAP (Maxmum A Pror) nference to nfer the most possble sequence of actvtes states. There are many nference algorthms for CRFs, ncludng the forward-backward algorthm, mean feld free energy, juncton tree, and loopy belef propagaton (LBP), whch s one of the most popular CRFs nference methods. Even though t can only approxmate the probabltes and does not guarantee convergence for graphs wth loops, LBP has been shown to be effectve (Sutton & McCallum 2004; Vshwanathan et al. 2006; Lao, Fox, & Kautz 2007). Sum-Product Algorthm To compute the margnal probablty, the LBP sum-product algorthm s adopted. We ntroduce a message m j (A j ) for each par of neghborng nodes A and A j, whch s a dstrbuton sent from node A to node A j about whch state varable A j should be n. All messages m j (A j ) are ntalzed as unform dstrbutons over A j. The message m j (A j ) sent from node A to ts neghbor A j s updated based on all the messages to A receved from ts neghbors A n except those from node A j. m j (A j ) = κ Ψ(A, O)Φ(A, A j, O) m k (A ) k,j where Ψ(A, O) s the local potental, Φ(A, A j, O) s the par-wse potental, and κ s the normalzaton constant. The messages propagate through the CRF graph untl they converge, when every message vares for less than a threshold. Gven that LBP does not guarantee convergence, we have to set a lmt on the maxmum number of teratons. Even though the update order of messages may affect the convergent speed, emprcal study (Sutton & McCallum 2004) showed that a random schedule s suffcent. After LBP converges, the margnal probablty of nodes A and A j s then determned as P (A, A j ) = κ Θ(A, A j, O) m k (A ) m lj (A j ) where k,j l,j Θ(A, A j, O) = Ψ(A, O)Ψ(A j, O)Φ(A, A j, O) k enumerates over all neghbors of A, l enumerates over all neghbors of A j and κ s the normalzaton constant. MAP Inference To do MAP nference, we replace summaton wth maxmzaton n the message update rule of the sum-product algorthm. We have m j (A j ) = κ max Ψ(A, O)Φ(A, A j, O) m k (A ), where κ s the normalzaton constant. k,j 84

4 After LBP converges, the MAP probablty of node A s defned as P (A ) = Ψ(A, O) j m j (A ), where A j s the neghbor of A. To do the nference, we can label every hdden varable by choosng the most lkely value accordng to the MAP probablty. Learnng Algorthm The purpose of learnng s to determne the weght λ k for each feature functon f k. We can do ths by maxmzng the log-lkelhood of the tranng data. Gven the tranng data D = {A(1), O(1); A(2), O(2)... A(M), O(M)}, the loglkelhood L(D λ) s defned as follows. L(D λ) = m log P (A(m) O(m)) The partal dervatve of the log-lkelhood wth respect to λ k s derved as L(D λ) = f k (A(m), A(m) j, O(m)) λ k m,j p(a(m), A(m) j λ)f k (A(m), A(m) j, O(m)), m,j where edge (A, A j ) can be ether n E c or E t. The former part of the partal dervatve s easy to compute, whle the latter part can be dffcult. The margnal probablty for the latter part can be computed by usng loopy belef propagaton ntroduced n the prevous subsecton. As solvng the equaton L(D λ)/ λ k = 0 does not yeld a closed-form soluton, λ may be updated teratvely usng some optmzaton technques such as teratve scalng, gradent descent, conjugate gradent or BFGS (Wallach 2003; Sha & Perera 2003) to fnd the weghts that maxmze the log-lkelhood. Prevous research has shown that L-BFGS works well n such an optmzaton problem for CRFs learnng and many current CRFs packages mplement ths method. L-BFGS s a varant of the Newton s method, whch s a second order optmzaton method. It s computatonally expensve to calculate the second order dervatve, Hessan matrx, Newton s method. BFGS provdes an teratve way to approxmate Hessan matrx by updatng the matrx n the prevous step. L-BFGS mplctly updates the Hessan matrx by memorzng the update progress and gradent nformaton n the prevous m steps. As a result, L-BFGS can reduce the amount of memory and computaton needs. In practce, L-BFGS requests the partal dervatve as well as the log-lkelhood at each teraton. So far, we have explaned how to compute partal dervatve. As to the loglkelhood, computng the normalzaton constant drectly s nfeasble, so we use Bethe free energy (Yedda, Freeman, & Wess 2003) to approxmate the normalzaton constant. In order to reduce over-fttng, we defne a zero mean Gaussan pror P (λ k ) = exp( λ 2 k /2σ2 ) wth varance σ 2 for each parameter λ k so that we maxmze the penalzed log-lkelhood L(λ D) = L(D λ) + k logp (λ k). As a result, the partal dervatve becomes L(λ D) λ k = L(D λ) λ k λ k σ 2. Experments We have ntroduced the FCRFs model for multple concurrent actvtes recognton. Let us fnd out how t works for the House n data set. In ths secton, we descrbe the expermental desgn as well as the expermental result for the evaluaton of our model. Expermental Desgn We extracted our expermental data from the MIT House n data set, whch s freely avalable for academc research. The data set was recorded on Frday March 4, 2005 from 9 AM to 1 PM wth a volunteer performng a set of common household actvtes. The data set recorded the nformaton from a varety of dgtal sensors such as swtch sensors, lght sensors, and current sensors, etc. The data set was then manually labelled wth ground truths of multple actvtes and locaton nformaton. Unfortunately, wth only four hours of recorded actvtes n the released data set, the sensor data were too sparse to be useful for effectve tranng. As the House n data are stll beng collected, cleaned, and labelled, so more comprehensve data set may be forthcomng. Meanwhle, we decde to utlze the locaton data as our prmary observatons. The locaton nformaton n the current House n data set s manually annotated and assumed to be qute accurate. Nevertheless, our system should perform n the same way should such data come from some (roomlevel) ndoor locaton trackng system. Now, let s explan the data used n our experments n more detal. The annotaton of the data recorded from 9 AM to 10 AM s somewhat unorganzed, so we decded to use only the data set recorded from 10 AM to 1 PM for a total of second. We labeled the actvtes for every second and dvde the data set nto 18 parts, each contanng 10 mnutes of data. Ths way, we can do 6-fold cross-valdaton for the evaluaton. For smplcty, we clustered the orgnal 89 actvtes nto 6 classes of actvtes, ncludng cleanng, laundry, dshwashng, meal preparaton, personal and nformaton/lesure. Because the 6 classes of actvtes may overlap wth each other, the property of multtaskng s sutable for our multple concurrent actvtes recognton. In addton, there are 9 mutually exclusve locatons that may be observed, ncludng lvng room, dnng area, ktchen, offce, bedroom, hallway, bathroom, powder room and outsde. Our FCRFs model s constructed n the followng way. Frst, for each actvty class, we buld one hdden varable node whch contans a Boolean state representng ether happenng or not-happenng. And then we let all the hdden varable nodes n the same tme slce to be fully connected, assumng that every actvty has cotemporal relatonshp wth each other. To represent the temporal relatonshp, we connect the two hdden varable nodes across two tme slces for the same actvty class. In addton, we connect 85

5 every hdden varable node wth one observed varable node whch represents the states of locaton. The FCRFs model defned are used to do learnng and nference for multple concurrent actvtes recognton n our experments. The varance of the Gaussan pror s 9. To evaluate the mportance of co-temporal relatonshp between actvtes, we constructed 6 lnear chan CRFs (LCRFs) models as for comparson. Each LCRFs model recognzes only one actvty class at a tme. Performance Evaluaton We want to test f the LCRFs model and the FCRFs model can correctly predct the actvty sequences n the testng data set. There are 6 actvtes labels at each tme stamp. The value of each actvty label can be ether postve (for happenng) or negatve (for not-happenng. In each part of the testng data set, there are 600 seconds and total 3600 actvtes labels. A label s consdered to be True Postve (TP) f the model correctly predcts ts value to be postve. Smlarly, a model s True Negatve (TN) f the model correctly predcts the value as negatve. On the contrary, the label s consdered to be False Postve (FP) f the model wrongly guesses the value as postve and False Negatve (FN) f the model wrongly guesses the value as negatve. To evaluate our performance, we calculate the recall, precson and F- score for these two models. The results are summarzed n Table 1, whch compares the recall, precson and F-score for the testng data set. Data set Recall Precson F-score FCRF(%) LCRF(%) Table 1: Performance Comparson of LCRFs and FCRFs. As we can see, the FCRFs model outperforms the LCRFs models. Even though the locaton nformaton may be ambguous for the recognton of actvty, the consderaton for co-temporal relatonshp n FCRFs complements ths defcency. As a result, the proposed FCRFs model mproves the F-score up to 8%. Ths expermental result provdes us the concluson that t s helpful to utlze the co-temporal relatonshp for actvty recognton. Concluson and Future Work Ths paper proposes the FCRFs model for jont recognton of multple concurrent actvtes. We desgned the experments based on the MIT House n data set and compared our FCRFs model wth the LCRFs model. The ntal experment showed that FCRFs mprove the F-score for up to 8% for recognton of multple concurrent actvtes n daly lvng. The current experment presents just an ntal step towards concurrent actvty recognton. As we mentoned earler, usng a sngle sensor such as locaton may not be suffcent for dsambguatng among the actvtes. To further mprove the recognton accuracy, data from multple heterogeneous sensors must be combned and taken nto consderaton n any practcal actvty recognton system. In addton, we may extend our actvty model to represent long-range temporal dependency as well as the relatonshp among dfferent actvtes across tme slces. In our current mplementaton, both learnng and nference are performed off-lne. To deploy actvty nference n real-world contextaware applcatons, we wll need to develop onlne nference such as the CRF-flter algorthm proposed n (Lmketka, Lao, & Fox 2007). References Cheu, H. L.; Lee, W. S.; and Kaelblng, L. P Actvty recognton from physologcal data usng condtonal random felds. Techncal report, SMA Symposum. Ghahraman, Z., and Jordan, M. I Factoral hdden markov models. Machne Learnng 29(2-3): Intlle, S. S.; Larson, K.; Tapa, E. M.; Beaudn, J. S.; Kaushk, P.; Nawyn, J.; and Rocknson, R. 2006a. House n placelab data set ( n/data /placelab/placelab.htm ). Intlle, S. S.; Larson, K.; Tapa, E. M.; Beaudn, J. S.; Kaushk, P.; Nawyn, J.; and Rocknson, R. 2006b. Usng a lve-n laboratory for ubqutous computng research. In Fshkn, K. P.; Schele, B.; Nxon, P.; and Qugley, A., eds., PERVASIVE 2006, volume l, Lafferty, J.; McCallum, A.; and Perera, F Condtonal random felds: Probablstc models. for segmentng and labelng sequence data. In Proceedngs of the Eghteenth Internatonal Confernence on Machne Learnng (ICML-01). Lao, L.; Fox, D.; and Kautz, H Extractng places and actvtes from gps traces usng herarchcal condtonal random felds. Internatonal Journal of Robotcs Research 26: Lmketka, B.; Lao, L.; and Fox, D Crf-flters: Dscrmnatve partcle flters for sequental state estmaton. In 2007 IEEE Internatonal Conference on Robotcs and Automaton. Lu, Y.; Carbonell, J. G.; P., W.; and V., G Segmentaton condtonal random felds (scrfs): A new approach for proten fold recognton. In Research n Computatonal Molecular Bology, 9th Annual Internatonal Conference. Sarawag, S., and Cohen, W. W Sem-markov condtonal random felds for nformaton extracton. In Advances n Neural Informaton Processng Systems 17. Sato, K., and Y., S Rna secondary structural algnment wth condtonal random felds. In ECCB/JBI (Supplement of Bonformatcs). Sha, F., and Perera, F Shallow parsng wth condtonal random felds. In Proceedngs of the 2003 Human Language Technology Conference and North Amercan Chapter of the Assocaton for Computatonal Lngustcs. Smnchsescu, C.; Kanauja, A.; and Metaxas, D Condtonal models for contextual human moton recognton. Comput. Vs. Image Underst. 104(2):

6 Sutton, C. A. Rohanmanesh, K., and McCallum, A Dynamc condtonal random felds: factorzed probablstc models for labelng and segmentng sequence data. In Proceedngs of the Twenty Frst Internatonal Confernence on Machne Learnng (ICML-04). Vshwanathan, S. V. N.; Schraudolph, N. N.; Schmdt, M. W.; and Murphy, K Accelerated tranng of condtonal random. felds wth stochastc meta-descent. In Proceedngs of the Twenty Thrd Internatonal Confernence on Machne Learnng (ICML-06). Wallach, H. M Effcent tranng of condtonal random felds. In Proceedngs of the 6th Annual CLUK Research Colloquum. Unversty of Ednburgh. Yedda, J. S.; Freeman, W. T.; and Wess, Y Understandng Belef Propagaton and ts Generalzatons. Kaufmann, M. chapter 8,