A New Network-based Algorithm for Human Activity Recognition in Videos

Transcription

1 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, A New Network-based Algorithm for Huma Activity Recogitio i Videos Weiyao Li, Yuazhe Che, Jiaxi Wu, Hali Wag, Bi Sheg, ad Hogxiag Li Abstract I this paper, a ew etwork-trasmissio-based (NTB) algorithm is proposed for huma activity recogitio i videos. The proposed NTB algorithm models the etire scee as a error-free etwork. I this etwork, each ode correspods to a patch of the scee ad each edge represets the activity correlatio betwee the correspodig patches. Based o this etwork, we further model people i the scee as packages while huma activities ca be modeled as the process of package trasmissio i the etwork. By aalyzig these specific package trasmissio processes, various activities ca be effectively detected. The implemetatio of our NTB algorithm ito abormal activity detectio ad group activity recogitio are described i detail i the paper. Experimetal results demostrate the effectiveess of our proposed algorithm. H I. INTRODUCTION uma activity recogitio is of icreasig importace i may applicatios icludig video surveillace, huma-computer iteractio, ad video retrieval [-4]. Automatically recogizig activities of iterest plays a key part i may of the existig video systems. Therefore, it is always desirable to develop ew activity recogitio algorithms with higher accuracy ad stroger capability for hadlig various scearios. May algorithms have bee proposed to recogize huma activities [-]. Aggarwal ad Ryoo [] gave a comprehesive review of huma activity aalysis. Nascimeto et al. [] detected huma actios usig a bak of switch dyamical models with a priori kowledge of the sceario. Rao et al. [] itroduced view-ivariat dyamic time warpig for aalyzig activities with trajectories. Zeliker et al. [4] created global trajectories by trackig people across differet cameras ad detected abormal activities if the curret global trajectory deviates from the ormal This work was supported i part by the Natioal Sciece Foudatio of Chia, uder Grats 646, 655, 67979, i part by the Chiese atioal 97 project, uder Grat CB74, i part by the Shaghai Pujiag Program, uder grat PJ44, i part by the Ope Project Program of the Natioal Laboratory of Patter Recogitio (NLPR), ad i part by the SMC grat of SJTU. W. Li ad Y. Che are with the Departmet of Electroic Egieerig, Shaghai Jiao Tog Uiversity, Shaghai 4, Chia ( {wyli, yzche45}@sjtu.edu.c). J. Wu is with the Natioal Key Laboratory for Novel Software Techology, Najig Uiversity, Najig, Chia ( wujx@ju.edu.c). H. Wag is with the Departmet of Computer Sciece ad Techology, Togji Uiversity, Shaghai 84, Chia ( haliwag@togji.edu.c) B. Sheg is with the Departmet of Computer Sciece, Shaghai Jiao Tog Uiversity, Shaghai 4, Chia ( shegbi@cs.sjtu.edu.c). H. Li is with the Departmet of Electrical ad Computer Egieerig, Uiversity of Louisville, Louisville 49, USA ( hogxiagli@gmail.com). Copyright (c) IEEE. Persoal use of this material is permitted. However, permissio to use this material for ay other purposes must be obtaied from the IEEE by sedig a to pubs-permissios@ieee.org. paths. However, these algorithms oly focus o recogizig the scee-related activities (i.e., activities oly cosiderig the relatioship betwee the perso ad his surroudig scee, such as a perso followig a regular path or a perso eterig uusual regios). Thus, it is very difficult to exted these algorithms ito the recogitio of group activities (i.e., activities icludig the iteractio amog people such as approach or people beig followed []). Furthermore, Kim ad Grauma [5] proposed to use a Mixture of Probabilistic Pricipal Compoet Aalyzers (MPPCA) to lear ormal patters of activities ad ifer a space-time Markov Radom Field (MRF) to detect abormal activities. This method ca effectively detect ad locate abormal activities which deviate from the leared ormal motio patters ad has the potetial to be exteded to detect group activities whe the group motio patter is suitably leared. However, sice this method is costructed based o the local-regio motio iformatio, it caot explicitly differetiate activities with motio patters i commo (e.g., differetiatig movig-back-ad-forth from movig-forward ad movigbackward). Also, the step of iferrig the MRF durig the detectio process is also time-cosumig. There are also a variety of researches o group activity recogitio. Zhou et al. [] detected pair-activities by extractig the causality features from bi-trajectories. Ni et al. [4] further exteded the causality features ito three types icludig idividuals, pairs ad groups. Cheg et al. [] used the Group Activity Patter for represetig ad differetiatig group activities where Gaussia parameters from trajectories were calculated from multiple people. Li et al. [] used group represetative to represet each group of people for detectig the iteractio of people groups such that the umber of people ca vary i the group activity. However, while these methods suitably hadle the iteractio amog people, may of them eglect the relatioship betwee people ad their surroudig scee. Thus, they may have limitatios whe detectig the scee-related activities. Furthermore, their abilities for detectig complex activities (such as people first approach ad the split) are also limited. Although some methods [7,, 7, 7] ca recogize the group activity as well as the scee-related activity by usig some pre-desiged graphical models such as the layered Hidde Markov Model (HMM) [], they ofte require large amout of traiig data i order to work well. Besides, the restricted graphical structure used i these methods may also limit their ability to hadle various uexpected cases. I this paper, we propose a ew etwork-trasmissio-based (NTB) algorithm for huma activity recogitio. The proposed framework first models the etire scee as a error-free etwork. I this etwork, each ode correspods to a patch of the scee ad each edge represets the activity correlatio betwee the

2 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, correspodig patches. Based o this etwork, we further model people i the scee as packages ad huma activities ca be viewed as the process of package trasmissio i the etwork. By suitably aalyzig these specific package trasmissio processes, huma activities ca be efficietly recogized. Our NTB algorithm is flexible ad capable of hadlig both the iteractios amog people (group activities) ad the iteractio betwee people ad the scee (scee-related activities). Experimetal results demostrate the effectiveess of our proposed algorithm. The rest of the paper is orgaized as follows: Sectio II describes the framework of our proposed NTB algorithm. Sectio III ad Sectio IV describe the implemetatios of our NTB algorithm i abormal evet detectio ad group activity recogitio i detail, respectively. The experimetal results are show i Sectio V ad Sectio VI cocludes the paper. perso ad the scee, our etwork-based model ca also be easily exteded for hadlig the iteractio amog people. For example, as i Fig., we ca costruct a relative etwork where oe perso is always located i the ceter of the etwork ad the movemet of aother perso ca be modeled as the package trasmissio process i this relative etwork based o his relative movemet to the etwork-ceter perso. I this way, the iteractio amog people ca also be effectively recogized by evaluatig differet trasmissio eergies i our etwork-based model. Based o the above discussios, we ca propose our NTB algorithm. The framework of our algorithm is described i the followig sectio. II. FRAMEWORK OF THE NTB ALGORITHM A. Basic idea of the algorithm The basic idea of our etwork-trasmissio-based (NTB) algorithm ca be described by Fig. ad Fig.. Our NTB algorithm first divides the etire scee ito patches where each patch is modeled as a ode i the error-free etwork (as i Fig. ). Based o this etwork, the process of people movig i the scee ca be modeled as the package trasmissio process i the etwork (i.e., a perso movig from oe patch to aother ca be modeled as a package trasmitted from oe ode to aother). I this way, various huma activity recogitio problems ca be trasferred ito the package trasmissio aalysis problem i the etwork. With this etwork-based model, oe key problem is how to use this model for recogizig activities. We further observe that if we model the process of perso movig amog patches as the eergy cosumed to trasmit a package, the activities ca the be detected with these trasmissio eergy features. For example, abormal activities ca be detected if its eergies deviate from the ormal activity trasmissio eergy by larger tha a pre-traied threshold (i.e., for example, if a perso moves to a uusual patch, the eergy used will be icreased ad this will be detected as a abormal activity). I this way, the abormal detectio problem ca be modeled as the eergy efficiet trasmissio problem i a etwork [9]. m u q u m q (a) (b) Fig. (a) Divide the scee ito patches. (b) Model each patch i (a) as a ode i the etwork ad the edges betwee odes are modeled as the activity correlatio betwee the correspodig patches. The red trajectory R(u, q) i (a) is modeled by the red package trasmissio route i (b). (Note that (b) ca be a fully coected etwork (i.e., each ode has edges with all the other odes i the etwork). I order to ease the descriptio, we oly draw the four eighborig edges for each ode i the rest of the paper) (best viewed i color). Furthermore, besides modelig the correlatio betwee the Network ceter perso Fig. Costructig relative etworks for modelig people iteractios. Upper: the locatios of the two approachig people i two differet frames (the dashed patches are divided by makig the red-circled grey perso at the etwork ceter). Dow: the trasferred etworks of the upper frames (the red-circled grey ode ad the blue-circled dotted ode are the locatios of the two people i the etwork). The locatio of the red-circled grey perso is fixed i the bottom etwork while the locatio of blue-circled dotted perso i the bottom etwork is decided by his relative locatio to the red-circled grey perso. B. The framework The framework of our proposed NTB algorithm is show i Fig.. I Fig., the part i the dashed rectagle is the traiig module while the three blocks o the top are the testig process. People Trajectories of the activities Divide the scee ito patches Traiig Module Calculate the eergy cosumptio for people activities Costruct etworks Calculate the DT eergy (activity correlatio) betwee patches Traiig data Aalyze the trasmissio eergy cosumptio Activity detectio rules Fig. The framework of the NTB algorithm. Recogized activities I the traiig module, the scee is first divided ito patches where each patch is modeled as a ode i the etwork. The the activity correlatios betwee patches are estimated based o the traiig data ad these activity correlatios will be used as the edge values i the etwork. With these odes ad edges, the trasmissio etworks ca be costructed. At the same time, the activity detectio rules are also derived from the traiig data for detectig activities of iterest durig the testig process. I the testig process, after obtaiig trajectories of people (which represet activities), their correspodig trasmissio eergies are first calculated based o the costructed etwork.

3 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, The, these trasmissio eergies are aalyzed ad the activity detectio rules will be applied for detectig the activities. Furthermore, several thigs eed to be metioed about our NTB algorithm. They are described i the followig: () We assume the etworks used i our algorithm are error-free (i.e., there are o iterfereces such as oises or package losses durig trasmissio). () Although there are other works [7, 5, 7] tryig to segmet the scee ito parts for activity recogitio, our NTB algorithm is differet from them i: (a) our NTB algorithm costruct a package trasmissio etwork over the patches while other works [7, 7] use graphical models for recogitio. While the fixed structures of the graphical models [7, 7] may limit their ability to hadle various uexpected cases, our fully-coected trasmissio etwork is more geeral ad flexible for hadlig various scearios; (b) With the trasmissio etwork model, our NTB algorithm is robust to the patch segmetatio styles (e.g., i this paper, we just simply segmet the scee ito idetical rectagular blocks as show i Fig. ). Comparatively, the graphical model-based methods ormally require careful segmetatio of the scee [5, 7]. () From Fig., it is clear that the steps of calculate the eergy betwee patches ad activity detectio rules are the key parts of our NTB algorithm. The implemetatio of these steps ca be differet for differet activity recogitio scearios. Therefore, i the ext two sectios, we will describe the implemetatios of our algorithm i two scearios (abormal evet or scee-related activity detectio, ad group activity recogitio), respectively. III THE IMPLEMENTATION OF NTB ALGORITHM IN ABNORMAL EVENT DETECTION I this sceario, we try to detect abormal activities such as people followig irregular paths ad people that move back ad forth. I the followig, we will describe the implemetatio of each step i Fig. i detail. Agai, ote that the two grey blocks i Fig. are the key parts of our algorithm. A. Divide the scee ito patches For the ease of implemetatio, we simply divide the scee ito idetical o-overlappig rectagular patches i this paper, as i Fig.. Note that other sematic-based segmetatio methods [5] ca also be easily used i our algorithm. B. Calculate the eergy cosumptio for people activities Let R(u, q) be the perso trajectory of the curret activity with u beig the startig patch ad q beig the perso s curret patch. Also defie the Direct Trasmissio (DT) eergy for the edge betwee patches i ad j as e(i, j) (i.e., the eergy used by directly trasmittig a package from patch i to j without passig through other patches, as will be described i detail i the ext sub-sectio). The total trasmissio eergy for the trajectory R ca be calculated by accumulatig the DT eergies of all patch pairs i the trajectory, as i Eq. (). E u,q ei, j i, j Ru, q For example, i Fig. (a), the total trasmissio eergy for the red trajectory R(u, q) equals to e(u, m)+e(m, )+e(, q). () C. Calculate the DT eergy (activity correlatio) betwee patches The edges betwee odes i the etwork are modeled by the Direct Trasmissio (DT) eergy betwee patches. I order to calculate the DT eergy, we first itroduce the activity correlatio AC betwee patches. That is, whe there are high chaces for people to perform activities betwee two patches i ad j (e.g., move across i ad j), a high correlatio AC(i, j) will appear betwee these patches. Otherwise, a low correlatio will be set. Thus, the activity correlatio ca be calculated by: i, j tw i, j AC () k k where AC(i, j) is the activity correlatio betwee patches i ad j. tw k (i, j) is the correlatio impact weight betwee i ad j from the k-th trajectory i the traiig data. From Eq. (), we ca see that the activity correlatio AC(i, j) is the summatio of correlatio impact weights tw k (i, j) from the traiig trajectories. If more traiig trajectories idicate a high correlatio betwee patches i ad j, a large activity correlatio AC(i, j) will be calculated. With the defiitio of AC(i, j), the DT eergy e(i, j) betwee patches ca be calculated by: i, j ACi, j e () From Eq. (), we ca see that the DT eergy is iversely proportioal to the activity correlatio. That is, whe the activity correlatio value AC(i, j) are larger betwee patches i ad j, it implies that a "higher" activity correlatio will appear betwee the patches, resultig i a "lower" DT eergy. I this way, we ca guaratee that ormal activities (ormally go across high-correlatio patches) ca result i smaller total eergies. From Eqs ()-(), we ca see that the correlatio impact weights tw k (i, j) are the key parts for calculatig the DT eergies. I this paper, a iterative method is proposed to calculate tw k (i, j), AC(i, j), ad e(i, j) ad the flowchart of this iterative method is show i Fig. 4. No Step : Iitialize twk (i, j), AC(i, j), e(i, j) Step : Give the curret values of twk (i, j), AC(i, j), e(i, j), update the detectio thresholds T ad T Step : Give the curret thresholds T ad T, update twk (i, j), AC(i, j), e(i, j) Coverge or reach the maximum iteratio time? Yes The fial DT eergy values e(i, j) ad the detectio thresholds T, T Fig. 4 The flowchart of the iterative method. From Fig. 4, we ca see that the proposed iterative method maily icludes three steps: I the first step, tw k (i, j), AC(i, j), ad e(i, j) are iitialized. I the secod step, give the curret values of tw k (i, j), AC(i, j), ad e(i, j), the thresholds for activity

4 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 4 detectio (T ad T ) are updated such that they ca achieve good detectio results with the curret DT eergy values e(i, j). I the third step, the values of tw k (i, j), AC(i, j), e(i, j) are further updated with the ewly updated detectio thresholds (T ad T ). Ad step ad step will be performed iteratively util the parameter values are coverged or the maximum iteratio time is reached. From Fig. 4, we ca see that the key parts of the iterative method are the three steps. Therefore, i the followig, we will describe the detailed process of the three steps i Fig. 4. Step : Iitialize tw k (i, j), AC(i, j), ad e(i, j). The values of tw k (i, j), AC(i, j), ad e(i, j) are iitialized by: tw k i, j if traiig trajectory k moves across otherwise AC i, j tw i, j ad i, j k k AC e patch i ad j (4) i, j where tw k (i, j), AC (i, j), ad e (i, j) are the iitialized values ad the superscript stads for the iteratio umber. From Eq. (4), we ca see that tw k (i, j) is iitialized to be if the k-th traiig trajectory moves across ode i ad ode j, or iitialized to be otherwise. This meas that the iitial DT eergy values e (i, j) are set to be the iverse of the total umber of traiig trajectories crossig the patches such that a large umber of crossig trajectories implies a high correlatio betwee the patches ad thus a low DT eergy value will be iitialized. This process ca reasoably iitialize the DT eergy values close to their optimal oes. Furthermore, ote that a large value will be set for iitializig e (i, j) if the trajectory-crossig time betwee i ad j is zero i order to avoid dividig by. Step : Update the detectio thresholds T ad T. I this step, the detectio thresholds T ad T are updated such that the updated thresholds ca achieve good detectio results with the curret DT eergy values e(i, j). The thresholds are updated by Eq. (9) ad this step will be described i detail i the ext sub-sectio (Sectio III-D). Step : Update tw k (i, j), AC(i, j), ad e(i, j). Give the ewly updated detectio thresholds T ad T, the values of tw k (i, j), AC(i, j), ad e(i, j) ca be updated by: twk i, j if k-th trajectory is correctly recogized Ek T l twk i, j twk i, j Ek if k is a false alarm Tl - E k twk i, j Tl if k is a miss detectio AC i, j twk i, j ad e i, j AC i, j k where the superscript ad + are the iteratio umbers ad the subscript k is the trajectory umber. tw k + (i, j), AC + (i, j), ad e + (i, j) are the updated values i the + iteratio. E k is the total trasmissio eergy for trajectory k calculated by Eq. () based o the DT eergy values i the -th iteratio e (i, j). T l (l= or ) are the detectio thresholds i the -th iteratio where the selectio of l depeds o which threshold is used for detectio. The calculatio of T l (=,,...) will be described i detail i the ext subsectio (Sectio III-D). (5) From Eq. (5), we ca see that durig each iteratio, the activity detectio results o the traiig data are used as the feedback for updatig the DT eergies. I this way, a false alarm will icrease the activity correlatio weight tw k (i, j) ad a miss detectio will decrease tw k (i, j). More specifically, if trajectory k is a false alarm (i.e., a ormal activity wrogly detected as abormal) ad it passes through patches i ad j, we will icrease tw k (i, j) accordig to Eq. (5) (i.e., icrease tw k (i, j) by multiplyig a factor of +(E k -T l )/E k. I this way, the DT eergy e(i, j) will be decreased such that the total trasmissio eergy for k will become smaller i the ext iteratio (ote that e(i, j) is iversely proportioal to tw k (i, j), makig k more likely to be detected as a ormal activity. O the cotrary, if the abormal activity k is detected as a ormal activity (i.e., a miss), the correlatio impact weight of k will be decreased (or the DT eergy e(i, j) is icreased) for icreasig the ability to detect abormal activities. Furthermore, ote that sice our etwork model allows the odes to be fully coected to each other (i.e., each ode ca also have edge with o-adjacet patches), our NTB algorithm is more geeral ad flexible of hadlig uexpected cases such as a perso uexpectedly jumpig to a o-adjacet patch due to the occlusio i the adjacet patches. D. Activity detectio rules As metioed, the basic idea of usig our NTB algorithm for abormal activity detectio is to evaluate whether the trasmissio eergy of the activity deviates from the ormal case by larger tha a pre-traied threshold. Therefore, oe of the key issues of our algorithm is to estimate the eergy cosumptio for ormal activities such that it ca be used as the referece for abormality detectio. I this paper, we propose to create the miimum-eergy route map for estimatig the total trasmissio eergies eeded for ormal activities. Based o this miimum-eergy route map, criteria ca be developed to detect abormal activities. The miimum-eergy route map ca be described by: map E m,,r m, m, S (6) ormal mi mi where S is the etire set of all patches, E mi (m, ) ad R mi (m, ) are the smallest possible trasmissio eergy ad its correspodig miimum eergy cosumptio route whe we wat to trasmit packages from patch m to, respectively. Sice i practice, the umber of etrace (or exit) patches i the scee is limited, we do ot eed to calculate E mi ad R mi for all (m, ) pairs. Istead, map ormal oly eed to iclude E mi (u, ) ad R mi (u, ) where u are the etrace (or exit) patches ad is ay patch i the scee (i.e., S). I this way, give ay patch i the scee, we ca kow the best route ad its correspodig miimum trasmissio eergy from the etrace u to this patch. Sice the calculatio of E mi (u, ) ad R mi (u, ) are similar to the eergy routig problem i etwork broadcastig [9], they ca be calculated by the eergy-efficiet-routig algorithms [9] used i wireless sesor broadcastig. However, sice we oly eed to calculate the miimum eergy ad route to the etrace patch u istead of betwee ay patches (i.e., u is fixed i E mi (u, ) ad R mi (u, )), the routig algorithm ca be simplified.

5 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 5 Therefore, i this paper, we use a Simplified Broadcast Icremetal Power (SBIP) algorithm for creatig the ormal trasmissio route map. It is described as i Algorithm. Fig. 5 shows the process of the SBIP algorithm i a example etwork. NE Algorithm The SBIP Method Iput: a udirected weighted complete graph G(N, V), where N is the set of its odes ad V={e(i, j)} is the set of its edges. The DT eergy of edge from N(i) to N(j) is stored i e(i, j). NE is the set of odes which have already bee added to the routig tree. Ad u is the start ode of the route tree. Output: miimum-eergy route tree ode set R mi ad accumulative eergy vector E mi. Iitializatio: Set all elemets of E mi (i, j) (i j) to a large umber L. Also set R mi (u, u)={u}; E mi (u, u)=; NE={u} where u is the start ode while NE N do Ec=L for i=: N ad ine for j =: N ad jne E_tmp=E mi (u, i)+ e(i, j) if E_tmp< Ec Ec=E_tmp; ic= i; jc =j; ed ed ed set E mi (u, jc)=ec; R mi (u, jc)={r mi (u, ic), jc}; add ode jc to NE; ed u b a c NE NE u a u a b 6 7 c b c where {e (i, j)} is the DT eergy set for all edges i the trasmissio etwork durig the -th iteratio (as calculated by Eq. (5)). err FA (t, t, {e (i, j)}) ad err miss (t, t, {e (i, j)}) are the false alarm ad miss detectio rates [] for detectig abormal activities i the traiig set whe the thresholds t, t as well as {e (i,j)} are used for detectio. It should be oted that the rules i Eq. (7) are oly oe way to detect abormal activities. I practice, other geeral classifiers (such as the Support Vector Machie (SVM) [8]) ca also be used to take the place of Eq. (7) ad to perform abormal activity recogitio based o our eergy features. Ad this will be further discussed i Sectio V. Furthermore, Fig. 6 shows the detailed detectio processes for 4 example activity trajectories. From Eq. (7) ad Fig. 6, we ca see that with our detectio rules, a activity will be detected as abormal if its total eergy deviates from the ormal activity trasmissio eergy by larger tha a pre-traied threshold T (such as Fig. 6 (b) ad (d)), or it is larger tha aother pre-traied threshold T (such as Fig. 6 (c)). Note that the secod detectio criterio is icluded such that: (a) The o-the-fly (or olie) abormal detectio is eabled such that we ca detect ormal/abormal activities i the curret patch rather tha waitig util the ed of the trajectory for detectio; (b) Some abormal activities with small absolute eergy values but large ratios over E mi (u, q) ca be effectively detected (as Fig. 6 (c)). R R mi(u, a)={u, a} E mi(u, a)= mi(u, a)={u, a} E mi(u, a)= R mi(u, a)={u, a} E mi(u, a)= R mi(u, b)={u, b} E mi(u, b)= R mi(u, b)={u, b} E mi(u, b)= R mi(u, c)={u, b, c} E mi(u, c)=5 Fig. 5 The process of the SBIP algorithm i a example etwork. (The odes iside the blue dash-dot circle are the set of odes which have already bee added to the routig tree (i.e., NE i Algorithm ); The dashed lies are the DT eergy values; The bold solid lies are the miimum eergy routes i the tree; Ad the lists at the bottom are the decided R mi ad E mi i each step) Based o the miimum-eergy route map, detectio rules ca the be developed to decide whether the iput activity trajectory is abormal. The proposed abormal detectio criteria are: The curret activity R (u, q) is abormal if: E(u, q)>t or E(u, q)>t (u, q) (7) where R(u, q) is the trajectory of the curret activity with u beig the etrace patch ad q beig the curret patch. E(u, q) is the total trasmissio eergy for the curret activity ad E mi (u, q) is the miimum possible eergy betwee u ad q ad it is calculated by Algorithm ad Eq. (6). T l (l= or ) are the thresholds for detectig abormal activities. Note that T is a costat value for all trajectories while T (u, q) is adaptive with the trajectories ad cotrolled by the parameter by: T u,q E u,q (8) mi Note that T ad T ca be automatically determied by the traiig data durig the same recursive traiig process as i Fig. 4 where i each iteratio, T ad T are updated by fidig a suitable set T ad T that miimize the squared summatio of two error rates err FA + err miss : T, T arg mi err t, t, e i, j FA err miss t, t, e i, j t, t (9) Total Eergy Total Eergy T T=αEmi E(u,i) (a) Normal activity T T=αEmi E(u,i) i j Abormal i E(u,j) Patch Locatio E(u,j) Patch j Locatio Total Eergy Total Eergy T Abormal E(u,j) E(u,i) i j T=αEmi Patch Locatio (b) Abormal climbig table T Abormal T=αEmi E(u,i) i E(u,j) j Patch Locatio (c) Abormal back ad forth (d) Abormal approach uusual regio Fig. 6 Examples of the activity trajectories (upper) ad their correspodig detectio processes (dow). (The red dashed lie ad the pik dash-dot lies are thresholds T ad T. The black circle-marker lie is the total trasmissio eergy. Ad the blue arrows are the patches where abormalities are detected) IV THE IMPLEMENTATION OF NTB ALGORITHM IN GROUP ACTIVITY RECOGNITION I the group activity recogitio scearios, we wat to recogize various group activities such as people approach each

6 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 6 other, oe perso leaves aother, ad people walk together. As metioed, whe recogizig the iteractio amog people, the relative etworks ca be costructed as i Fig.. At the same time, sice some group activities also iclude the relatioship betwee people ad their surroudig scee (e.g., we eed to recogize whether a perso is movig or stadig still i the scee i order to differetiate activities such as both people walk to meet or oe perso stad still ad aother oe approaches him), a scee-related trasmissio etwork similar to abormal evet detectio is also required. Therefore, i this sectio, we propose to use two types of etworks for represetig group activities. The detailed implemetatio of the key parts i Fig. for group activity detectio is described i the followig. A. Costruct etworks I this paper, we costruct three etworks for recogizig group activities: the scee-related etwork, the ormal relative etwork, ad the weighted relative etwork. The scee-related etwork is used to model the correlatio betwee people ad the scee ad it ca be costructed as i Fig.. The ormal relative etwork ad the weighted relative etwork are used for modelig the iteractio amog people ad they ca be costructed by fixig the locatio of oe perso i the etwork ad derive the locatios of other people based o their relative movemets to the locatio-fixed perso, as i Fig.. Besides, the followig poits eed to be metioed about the etworks. () The structures of the ormal relative ad the weighted relative etworks are the same. They oly differ i edge values. () Note that the scee-related etwork is a udirected etwork (i.e., the DT eergy cosumptio whe movig from patch i to j is the same as movig from j to i). However, the ormal relative etwork ad the weighted relative etwork are directed etworks (i.e., the DT eergy from i to j is differet from j to i). This poit will be further described i detail i the followig sub-sectios. () Sice the relative etworks oly focus o the relativity betwee people, whe costructig relative etworks, we radomly select oe perso to be the referece perso ad put him at the ceter of the relative etwork. (4) Besides the three etworks used i the sectio, our algorithm ca also be exteded to iclude other etworks with other motio features. Ad this poit will be further discussed i detail i the experimetal results (i.e., Sectio V-C ad Sectio V-D). B. Calculate the eergy cosumptio for people activities I this paper, we propose to calculate a set of trasmissio eergies from the three etworks for describig group activities. For the ease of descriptio, we use two-people group activity as the example to describe our algorithm. Multiple people scearios ca be easily exteded from our descriptio. The total trasmissio eergy set for two-people group activity ca be calculated by: [E (u,q ), E (u,q ), ENR(u -u,q -q ), EWR(u -u,q -q )] () where E (u,q ) ad E (u,q ) are the total trasmissio cosumptio for perso ad perso i the scee-related etwork, respectively. Ad they ca be calculated by Eq. (). ENR(u -u,q -q ) is the total trasmissio cosumptio i the ormal relative etwork where R(u -u,q -q ) is the relative trajectory of perso with respect to perso. Ad EWR(u -u,q -q ) is the total trasmissio cosumptio i the weighted relative etwork. ENR(u -u,q -q ) ad EWR(u -u,q -q ) ca be calculated by: ENR EWR u u, q q eri, j () i, j Ru u, q q u u, q q ewri, j i, j Ru u, q q where er(i, j) ad ewr(i, j) are the Direct Trasmissio (DT) eergy from patch i to j i the ormal relative etwork ad weighted relative etwork, respectively. The calculatio of er(i, j) ad ewr(i, j) will be described i detail i the ext sub-sectio. C. Calculate the eergy (activity correlatio) betwee patches The DT eergy for the three etworks is show by Fig. 7. For the scee-related etwork, sice we oly eed it to detect the movemet of the perso i our sceario, we simply set all the DT eergies to be, as i Fig. 7 (a). Note that if we wat to detect abormal group activities such as a group of people followig abormal paths, we ca also utilize the method described i Fig. 4 to automatically trai the DT eergies i the scee-related etwork istead of simply puttig all DT eergies to be. Furthermore, we ca also exted the scee-related etwork by usig directed etworks to hadle the scearios related to motio directios (e.g., the road traffic case). For the ormal relative etwork, three DT eergy values are used as show i Fig. 7 (b). For edges poitig toward the ceter ode, their DT eergy values er(i, j) will be (as the red dashed arrows i Fig. 7 (b)). For edges poitig outward the ceter ode, their DT eergy values will be - (as the blue dash-dot arrows i Fig. 7 (b)). Ad the DT eergy values will be for edges betwee odes havig the same distace to the ceter ode (as the black solid arrows i Fig. 7 (b)). Sice i the ormal relative etwork, perso is fixed at the ceter ode, the ormal relative eergy ENR(u -u,q -q ) is maily calculated by the movemet of perso with respect to perso. Based o our DT eergy defiitio, whe perso is movig close to the ceter ode (i.e., movig toward perso ), ENR(u -u,q -q ) will be icreased. O the cotrary, whe perso is leavig the ceter ode, ENR(u -u,q -q ) will be decreased. I this way, the relative movemet betwee people ca be effectively modeled by the trasmissio eergy. (a) Scee-related etwork (b) Normal relative etwork (c) Weighted relative etwork Fig. 7 The DT eergy values for the three etworks (ote that (a) is a udirected etwork while (b) ad (c) are directed etworks). - -4

7 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 7 The structure of the weighted relative etwork is the same as the ormal relative etwork. However, the DT eergy values ewr(i, j) are weighted as show i Fig. 7 (c). For edges either poitig toward or outward the ceter ode, the DT eergy values will become larger whe they are closer to the ceter ode. Their oly differece is that edges poitig toward the ceter ode are positive while edges poitig outward the ceter ode are egative. With this weighted relative etwork, we ca extract the history or temporal iformatio of the relative movemet betwee people. For example, whe perso moves from the red ode i Fig. 7 (c) toward perso ad moves back, the correspodig total weighted relative trasmissio eergy EWR(u -u,q -q ) will be a positive value. O the cotrary, EWR will be a egative value whe perso leaves perso from the red ode ad the comes back. If we take a more careful look at the three etworks i Fig. 7, we ca see that sice the scee-related etwork is costructed based o the scee without beig affected by the people movemets, it ca be viewed as a idetical field where packages eed to cosume eergy to move ad their movig distaces are proportioal to their cosumed eergies. Comparatively, sice the two relative etworks i Fig. 7 (b)-(c) are costructed based o perso, they ca be viewed as the repulsive fields where perso i the etwork ceter is creatig repulsive forces. Thus, packages eed to cosume eergy i order to approach perso while gai eergy whe leavig perso. At the same time, o eergy will be cosumed or gaied whe packages are revolvig aroud perso. Furthermore, ote that the three eergy etworks i Fig. 7 are ot fully coected (i.e., each ode is oly coected to its eight eighborig odes ad are ot coected with its o-adjacet odes). However, these etworks ca also be exteded to become fully coected. For example, we ca defie the DT eergy betwee two o-adjacet odes i ad j as the miimum possible eergy eeded to move from i to j. I this way, the DT eergy betwee ay odes ca be calculated ad the fully-coected etworks ca be costructed. D. Activity detectio rules With the three etworks ad their correspodig DT eergies, we ca calculate the total trasmissio eergy set for the iput group activity trajectories, as i Eq. (). The whe recogizig group activities, we ca view the total trasmissio eergy set i Eq. () as a feature vector ad trai classifiers for automatically achievig the detectio rules. I this paper, we use Support Vector Machie (SVM) [8] to lear the detectio rules from the traiig set ad use it for group activity recogitio. It should be oted that the proposed method ca be used with geeral classifiers. We choose SVM sice it is the commo choice for activity recogitio so that it is easy to be implemeted ad compared with our methods. Experimetal results demostrate that our NTB algorithm ca effectively recogize various group activities. V. EXPERIMENTAL RESULTS I this sectio, we show experimetal results for our proposed NTB algorithm. I the followig, we will show the results o four differet datasets icludig abormality detectio ad group activity recogitio. Fially, we will also discuss the computatio complexity ad memory storage requiremet of our NTB algorithm. Furthermore, ote that i our experimets, whe we map the trajectory of a object ito the patch-based route (such as i Fig. (b)), we check the locatio of the object i each frame. Ad as log as the object moves to a ew patch, this ew patch will be added to the object's patch-based route. I this way, all the patches that the trajectory passes through ca be added i the route. A. Experimets for abormal evet detectio i a abormality dataset First, we perform experimets o our multi-camera dataset. The dataset is created by a two-static-camera system as show i Fig. 8. From Fig. 8, we ca see that the etire room has 5 cubes (the grey blocks) ad oe etrace door (the dashed block). I ormal cases, people eter the room to their cubes, or exit the room from their cubes, or move from oe cube to aother. Therefore, these cubes ad the etrace door ca be viewed as the etrace (or exit) patches. Furthermore, two cameras are used to moitor the etire room where oe camera moitors the right part (the left blue dashed camera i Fig. 8) ad the other oe moitors the left part (the right red dash-dot oe i Fig. 8). I total, there are 6 sequeces i our dataset which icludes ormal activity sequeces ad 96 abormal activity sequeces. Note that each sequece icludes two videos from the two cameras. I our experimets, three types of abormal activities are defied as i Table. Fig. 9 shows part of the global trajectories that we extracted for ormal activities i the two-camera view where the trajectories from two cameras are first extracted by particle-filter-based method [6] ad the combied ito the global trajectory by the method of Prosser [6]. Besides, the colored blocks i Fig. are the patches that we divided i our experimet. Table Three types of abormal activities i our experimets (I) follow a irregular path (e.g., climb over the discussio table as i Fig. 6 (b)) (II) movig back ad forth oce or more tha oce i the room (e.g., i Fig. 6 (c)) (III) approach uusual regios (e.g., go to the left-bottom corer as i Fig. 6 (d)) Etrace Door Cube Discussio Table Cube Cube Cube 4 Cube 5 Fig. 8 The cofiguratio of cameras i our two-camera dataset. Fig. 9 Global trajectories for part of the regular activities. Furthermore, several thigs eed to be metioed about combig multiple camera views: () Our proposed algorithm is geeral ad other methods [9, ] ca also be used to achieve the activity trajectories; () Whe the patches from differet camera views overlap (i.e., patches from differet camera views

8 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 8 are for the same regio), we simply set the DT eergy betwee these patches to be sice movig betwee these patches do ot cosume ay eergy; () Besides creatig the global trajectories, other methods ca also be used to combie multi-camera views. For example, we ca first use image stitchig [] to stitch multi-camera images ito a large image of the etire scee. The, we ca divide patches o this large image ad apply our method. It should be oted that the sceario of our experimet is quite challegig ad complex because: (a) The two-camera view icludes both overlappig regios (i.e., regios covered by both cameras) ad o-overlappig regios (i.e., regios oly covered by a sigle camera). (b) There are oly about 5 sequeces available for traiig (i 75% traiig ad 5% testig case), which is difficult for costructig satisfactory detectio models. (a) (b) Fig. The miimum-eergy route map ad ormal trasmissio routes. (a) The miimum-eergy route map calculated by our SBIP algorithm. (b) The map for ormal trasmissio routes by deletig coectios with large DT values i (a). Fig. (a) shows the result of a miimum-eergy route map calculated by our SBIP algorithm uder 75% traiig ad 5% testig where five idepedet experimets are performed ad the results are averaged (ote that we trai o 75% of all the data where both ormal ad abormal samples are icluded). From this map, we ca achieve the miimum-eergy routes from the etrace patch to all the patches i the scee. Furthermore, by deletig the coectios with large DT values i Fig. (a) (ote that large coectios with large DT values refer to the routes to the uusual patches), a "ormal" route map ca be achieved which ca be roughly regarded as the map for ormal trasmissio routes, as i Fig. (b). From Fig. (b), we ca see that the calculated ormal trasmissio routes amog the cubes ad the etrace door are pretty close to the regular trajectories i Fig. 9. This implies the effectiveess of our algorithm i detectig abormal activities. Furthermore, ote that: () although some patches such as i ad j i Fig. (a) are ot coected (because the route map oly allows oe route from the etrace patch to each patch ad circle routes are ot allowed), it does ot mea that the DT eergy betwee i ad j is large. Rather, the DT eergy betwee i ad j is small. Therefore, durig the testig part, people movig dowward aroud the table to patch j ca also be detected as ormal activity sice its total trasmissio eergy is small. () Some o-adjacet patches (such as k ad i i Fig. (a)) are also coected i the miimum-eergy route map. This is because these patches are for the overlap regios (i.e., patches from differet camera views but represetig the same physical regio). Ad as metioed, the DT eergies betwee these patches are set to by our algorithm. I this way, patch i ca fid its best route to the etrace patch u by goig through k. Furthermore, Table compares the activity detectio results of the followig seve methods: () The baselie method which views the patches covered by the ormal traiig trajectories as "ormal" patches (i.e., patches wet through by the ormal traiig trajectories) ad the remaiig patches as "abormal" (or uusual) oes. Thus, i the testig part, trajectories goig through those "abormal" patches will be detected as abormal (Baselie i Table ). () The baselie method which uses kerel desity estimatio (KDE) [4] to costruct a occupacy probability based o the ormal traiig trajectories ad detects trajectories that eter ito low occupacy-probability areas as the abormal activities. (Baselie i Table ). () The trajectory-similarity-based method where abormalities are detected if there is a clear differece betwee the iput trajectory ad the pre-traied trajectory cluster [4] (TSB i Table ). (4) The spatio-temporal-aalysis-based method which first extracts dyamic istats from the global trajectory ad the utilizes view-ivariat dyamic time warpig for measurig trajectory similarities for detectio [] (STAB i Table ). (5) The probability-trasitio-matrix-based method which calculates the activity s coditioal probability based o the pre-traied probability trasitio matrix for activity detectio [9] (PTM i Table ). (6) The NTB+SVM method. That is, usig [E(u, q), E(u, q)/ E mi (u, q)] as a -dimetioal feature vector for describig the activities ad usig SVM to take the place of the rules i Eq. (7) for abormity detectio (NTB+SVM i Table ). Note that the traiig process of the NTB+SVM method is similar to the NTB algorithm as i Fig. 4. However, there are two major differeces for the traiig process of the NTB+SVM method: (a) Sice i the abormal activity recogitio sceario, the DT eergy values (i.e., e(i, j) which are used to calculated E(u, q) ad E mi (u, q)) also eed to be traied i the traiig process, the SVM classifier eeds to be re-traied durig each iteratio whe the DT eergy values are updated. That is, the SVM re-trai step is used to take the place of threshold updatig step (Step ) i Fig. 4. (b) Sice there are o thresholds i the NTB+SVM method, the correlatio impact weight tw k (i, j) ca be updated by Eq. () istead of Eq. (5): twk i, j if k-th trajectory is correctly recogized twk i, j twk i, j Pk if k is a false alarm twk i, j Pk if k is a miss () where P k is the activity detectio probability for the k-th trajectory calculated by the SVM i the -th iteratio. (7) Our proposed NTB algorithm (NTB i Table ). I Table, three rates are compared: false alarm rate (FA) [], miss detectio rate (Miss) [], ad total error rate (TER) []. The FA rate is defied by N θ fp /N θ where N θ fp is the umber

9 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 9 of false positive video clips for activity θ (i.e., the umber of ormal activities wrogly detected as abormal activities i this experimet), ad N θ is the total umber of egative video clips except activity θ i the test data (i.e., the total umber of ormal activities i this experimet) []. The miss detectio rate is defied by N θ f /N θ + where N θ f is the umber of false egative (misdetectio) sequeces for activity θ (i.e., the umber of abormal activities wrogly detected as ormal activities i this experimet), ad N θ + is the total umber of positive sequeces of activity θ i the test data (i.e., the total umber of abormal activities i this experimet) []. The TER rate is calculated by N t_r /N t_f where N t_r is the total umber of wrogly detected activities for both ormal ad abormal activities ad N t_f is the total umber of activity sequeces i the test set. TER reflects the overall performace of the algorithm i detectig both the ormal ad the abormal activities []. I order for more detailed compariso, we also iclude the Miss rate for each idividual abormal activity listed i Table (i.e., the miss rates for I, II, III i Table ). Note that sice the two baselie methods (Baselie ad Baselie ) are oly desiged to detect ormal ad abormal activities, they caot differetiate differet abormality types ad thus the miss rates for type I, II, III abormalities for these two methods are ot listed i Table. Furthermore, also ote that the abormal activities ca be simply differetiated i our NTB algorithm by: (a) detectig as type I abormality if the activity s total trasmissio eergy (TE) is larger tha both thresholds T ad T, (b) detectig as type II abormality if TE is smaller tha T ad larger tha T, ad (c) detectig as type III abormality if TE is larger tha T but smaller tha T. Some examples of the process for detectig these three abormality types are show i Fig. 6. Table Miss, FA ad TER rates of abormal activity detectio uder 75% traiig ad 5% testig Baselie Baselie TSB STAB PTM NTB+SVM NTB FA (%) (I) Miss (II) (%) (III) Total TER (%) From Table, we ca see that the performace of our NTB algorithm is obviously better tha the other methods (Baselie, Baselie, TSB, STAB, ad PTM). Besides, several observatios ca be draw from Table. () The performace of our NTB algorithm is obviously better tha the two baselie methods. This is because: (a) Sice the traiig data i this experimet are ot sufficiet, "ormal" regios are ot fully covered by the ormal traiig trajectories (i.e., some regios are ormal but o ormal traiig trajectory passes through them). Thus, if we simply use the limited ormal traiig data to model all the ormal routes (such as the two baselie methods), may ormal regios will be mis-regarded as the "abormal" oes ad the detectio performace will be greatly affected. Comparatively, our NTB algorithm utilizes a iterative method to costruct the DT eergies betwee patches by suitably itegratig both the ormal ad abormal traiig samples as well as the error rates o these traiig samples (i.e., Eqs (5) ad (9) i the paper). I this way, the isufficiet traiig data ca be more efficietly utilized to costruct a more reliable model. (b) More importatly, the two baselie methods also caot differetiate the abormalities whose etire trajectories are iside the ormal regios (e.g., movig back ad forth i the regular route or movig aroud the table i the regular route). Comparatively, our method ca effectively detect these abormalities by checkig the secod criteria i Eq. (7). () Our NTB algorithm also has better performace tha the trajectory-similarity-based methods such as TSB ad STAB. This is because: (a) the trajectory-similarity-based methods will easily cofuse large-deviatio ormal trajectories with small-deviatio abormal trajectories. For example, if a ormal trajectory keeps zigzaggig aroud the ormal route, its distace to the ormal-route cluster may be large. At the same time, if a abormal trajectory closely follows the ormal route most of the time but oly deviates to uusual regio at the ed, its distace to the ormal-route cluster may be eve smaller tha the ormal trajectory. I this case, the trajectory-similarity-based methods will fail to work. Comparatively, our NTB algorithm utilizes both the ormal ad abormal traiig samples as well as the error rates o these traiig samples to costruct suitable DT eergies betwee patches. I this way, trajectories movig ito uusual regios will be effectively detected due to the large DT eergies eterig these uusual patches. (b) The trajectory-similarity-based methods also have low efficiecy i detectig abormalities such as back ad forth whose trajectory overlaps. Comparatively, our NTB algorithm ca work effectively by checkig the secod criteria i Eq. (7). () Although the PTM method itroduces the trasmissio matrix ad multi-camera cosesus for hadlig activity detectio, its performace is still less satisfactory. This is because: (a) The trasmissio matrix may have limitatios i differetiatig differet trajectory behaviors. (b) The camera overlappig area i this experimet is small, which limits the capabilities of its multi-camera cosesus step. O the cotrary, our NTB algorithm is more flexible uder this sceario. (4) Our NTB algorithm is ot oly effective i detectig abormal activities, but also efficiet i differetiatig all the abormality activity types (i.e., I, II, III). Compared to our method, the abormal activity differetiatio ability for the other methods are much poorer. Furthermore, the compared methods (i.e., TSB, STAB, PTM) are extremely poor i differetiatig the back ad forth activity (i.e., (II)). This is because may back trajectory patches are overlapped with the forth trajectory patches, makig the etire trajectory very difficult to be efficietly represeted. However, this type of activities ca still be effectively detected by NTB as show i Fig. 6 (b). (5) Comparig NTB with NTB+SVM methods, we have see that both methods ca achieve similar performaces. This demostrates that: (a) our proposed eergy-based features (E(u, q) ad E mi (u, q)) are effective i differetiatig abormal activities; (b) Our eergy-based features from the etwork models are geeral ad other geeral classifiers ca also be utilized to perform abormal activity recogitio besides the criteria i Eq. (7). Furthermore, Fig. shows the total trasmissio eergies calculated by our NTB algorithm for oe set of testig sequeces where the first 4 values are for ormal activity sequeces ad the later values are for abormal activity sequeces. We ca

10 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, see from Fig. that most abormal activities have large total eergy values by our model ad thus ca be easily detected. Some abormal activities have relatively small absolute eergy values. For example, sequeces 9-4 correspod to activities goig back ad forth alog the ormal path. Sice most of their trajectories are o the ormal route where the DT eergies are small, the accumulated total eergies for these sequeces become small. However, sice their eergy differeces with the miimum possible eergy E mi are larger tha our pre-traied threshold, they still ca be successfully detected i our algorithm by checkig the secod criterio i Eq. (7). Fig. The total trasmissio eergies calculated by our NTB algorithm for activities i the test sequeces. Table Miss, FA ad TER rates of the detectio algorithms with differet patch sizes (Note that the results i Table ad Fig. are achieved by usig the patch size of 48 48) Patch Size TER Fially, we also perform aother experimet by usig differet patch sizes for abormality detectio. The results are show i Table ad we ca have the followig observatios. () We ca achieve stable results whe the patch sizes chage withi a wide rage (e.g., from 4 4 to i Table ). This implies that the iterative traiig method i our NTB algorithm ca adaptively achieve suitable DT eergy values for differet patch sizes whe the patch size is withi a reasoable rage. () The patch size caot be extremely large. Whe the patch size is extremely large (e.g., 7 7 i Table ), there will be few patches i the scee. This will make the algorithm difficult to differetiate the various activity patters. For example, i the extreme case, if the patch size is the etire image ad there is oly oe patch, it is impossible to perform recogitio. () Also, the patch size also caot be extremely small. Whe the patch size is extremely small (e.g., 9 9 i Table ), the umber of odes ad edges i the etwork will become obviously large. I this case, large umber of traiig samples is required i order for costructig reliable DT eergies. Otherwise, the performace will be poor. For example, if the patch size is (i.e., each pixel is a patch) ad we oly have traiig trajectories, there will be large umber of "ormal" patches where o traiig trajectory arrives. I this case, the DT eergy for these patches will be large ad trajectories passig these patches will be easily detected as abormal. (4) From the above discussios, i the experimets i our paper, we select patch sizes such that the etire image of oe camera scee ca have 7-4 patches i width ad 6 patches i height. Of course, whe more traiig samples are available, smaller patch sizes ca also be selected. B. Experimets for group activity recogitio o the BEHAVE dataset We further perform aother set of experimets for the group activity recogitio. The experimets are performed o the public BEHAVE dataset [8] where 8 activity clips are selected for recogitio. Eight group activities are recogized as show i Table 4. Some frames are show i Fig.. Table 5 compares the results of the four methods: () The group-represetative-based algorithm [] (GRAD i Table 5). () The pair-activity classificatio algorithm based o bi-trajectories aalysis which uses causality ad feedback ratios as features [] (PAC i Table 5). () The localized-causality-based algorithm usig idividual, pair, ad group causalities for group activity detectio [4] (LCC i Table 5). (4) Our proposed NTB algorithm with trasmissio eergy sets from three etworks (NTB i Table 5). Table 4 The group activities recogized o the BEHAVE dataset (I) meet: two people walk toward each other. (II) follow: two people are walkig. Oe people follow aother. (III) approach: oe people stad ad aother walk toward the first people. (IV) separate: two people escape from each other. (V) leave: oe people stad ad aother leave the first people. (VI) together: two people are walkig together. (VII) exchage: two people first gathered ad the leave each other. (VIII) retur: two people first separate ad the meet. (a) Leave (b) Follow Fig. Example frames of the BEHAVE dataset. Table 5 Miss, False Alarm, ad TER rates of the group recogitio algorithms uder 75% traiig ad 5% testig GRAD PAC LCC NTB Meet Miss(%) FA(%) Follow Miss(%) FA(%) Approach Miss(%) FA(%) Separate Miss(%) FA(%) Leave Miss(%) FA(%) Together Miss(%) FA(%) Exchage Miss(%) FA(%) Retur Miss(%) FA(%) TER(%) I Table 5, three rates are compared: the miss detectio rate (Miss), the false alarm rate (FA) [], ad the total error rate (TER). From Table 5, we ca see that our proposed NTB algorithm ca achieve obviously better performace tha the other three state-of-art algorithms. This demostrates that our NTB algorithm with the trasmissio eergy features ca precisely catch the iter-perso spatial iteractio ad the activity temporal history characteristics of the group activities. Specifically, our NTB is obviously effective i recogizig

11 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, complex activities (i.e., exchage ad retur). I Fig., (a) shows two example trajectories of the complex activities, (b) shows the values of the major features i the PAC algorithm [], ad (c) shows the trasmissio eergy (EWR) from the weighted relative etwork i our NTB algorithm. From Fig. (b), we ca see that the features i the PAC algorithm [] caot show much differece betwee the two complex activities. Compared to (b), our EWR eergy i (c) is obviously more distiguishable by effectively catchig the activity history iformatio. (a) (b) (c) Fig. (a) Example trajectories for complex group activities; (b) The major feature values for PAC algorithm; (c) The EWR eergy values by our NTB EWR Exchage Retur C. Experimets for group activity recogitio o the CASIA dataset I order to further demostrate the effectiveess of our NTB algorithm, we also perform aother experimet o the CASIA dataset [6]. The CASIA dataset cotais seve group activities as show i Table 6 [6]. Some example frames are i Fig.. Table 6 The group activities recogized o the CASIA dataset A (rob): perso P follows perso P, catches him, robs him, ad the rus away. A (fight): people P ad P approach each other ad fight with each other. A (follow): perso P follows perso P util the ed. A 4 (follow ad gather): perso P follows perso P ad the walks together A 5 (meet ad part): P ad P approach each other, meet, ad the depart. A 6 (meet ad gather): P ad P meet each other ad the walk together A 7 (overtake): perso P overtakes perso P. Sice the activities such as "rob" ad "fight" are related to perso's local motio itesities, may of the algorithms o this dataset [7] utilize both the trajectory ad the motio itesity features for detectio. Therefore, i order to have a fair compariso with these methods, we further exted our algorithm by itroducig a additioal motio-itesity etwork to iclude the motio itesity feature. The structure of the motio-itesity etwork is the same as the scee-related etwork i Fig. 7 (a). However, differet from the scee-related etwork whose DT eergies are a costat value, the DT eergies i the motio-itesity etwork are decided by the motio itesities [7] whe a object moves across patches: emi( i, j, t) s( i, j, t) () where emi(i, j, t) is the DT eergy betwee patches i ad j at time t i the motio-itesity etwork. s(i, j, t) is the motio-itesity i patches i ad j at time t ad it ca be calculated by: i, j, t vi, j t s( i, j, t) v (4) opticalflo w, where v opticalflow (i, j, t) is the magitude of the average optical flow speed iside patches i ad j at time t. Ad v(i, j, t) is the magitude of the object's global speed movig across patches i ad j at time t. Similar to [7], v opticalflow (i, j, t) ad v(i, j, t) ca be calculated from the Lucas Kaade algorithm [8] ad the object's trajectory, respectively [7]. Basically, sice v opticalflow icludes both the object's local ad global motios while v oly icludes the global motio, by removig v from v opticalflow, the object's local motio itesities ca be achieved [7]. Furthermore, ote that the DT eergy emi(i, j, t) is related to time t. This meas that people movig across patches with differet local motio patters will have differet DT eergies i the motio-itesity etwork. With this motio-itesity etwork, the followig feature vector is utilized i our NTB algorithm for group activity recogitio. [E (u,q ), E (u,q ), ENR(u -u,q -q ), EWR(u -u,q -q ), EMI (u,q ), EMI (u,q )] (5) where the defiitios of E (u,q ), E (u,q ), ENR(u -u,q -q ), ad EWR(u -u,q -q ) are the same as i Eq. (). EMI (u,q ) ad EMI (u,q ) are the total trasmissio cosumptio for perso ad perso i the motio-itesity etwork. Fig. 4 compares the experimetal cofusio metric results of differet methods o the CASIA dataset. I Fig. 4, (a)-(e) shows the results for the Hidde Markov Model (HMM) method [, 9], the Coupled HMM method (CHMM) [], the coupled observatio decomposed HMM method with cotiuous features (CODHMM_C) [7], the coupled observatio decomposed HMM method with some discretized features [7], ad our NTB algorithm, respectively. From Fig. 4, we ca see that our proposed NTB algorithm ca also achieve better performace tha the state-of-art algorithms [7] o the CASIA dataset. More specifically, our algorithm has obvious improvemets i detectig activities such as rob (A ), follow (A ), ad overtake (A 7 ). This further demostrates that: (a) our etwork-based models are very effective i differetiatig similar activities (such as follow ad overtake); (b) Besides trajectories, our algorithm ca also be exteded to iclude other motio features (such as local motio itesities) ad effectively hadle the complicated activities such as rob. This poit will be further discussed i the ext sub-sectio. (a) HMM (b) CHMM (c) CODHMM_C (d) CODHMM_CD (e) NTB (proposed) Fig. 4 Cofusio matrices for differet methods o the CASIA dataset. D. Experimets for group abormality detectio o the UMN dataset I this sectio, we perform aother experimet o the UMN dataset [] which cotais videos of differet scearios of a abormal "escape" evet i differet scees icludig both idoor ad outdoor. Each video starts with ormal behaviors ad eds with the abormal behavior (i.e., escape). Some example images of the UMN dataset are show i Fig. 5. I order to recogize the abormal "escape" evets, we simply

12 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, use a sigle ormal relative etwork (as i Fig. 7 (b)) ad put it i the ceter of the image scee (ote that i this experimet, the ormal relative etwork is fixed at the image ceter rather tha movig with some object). Furthermore, istead of extractig the object trajectories, we directly extract the optical flows [7] from the videos ad use them as the packages to trasmit i the ormal relative etworks. Whe detectig activities, we use a slidig widow to segmet the video ito small video clips [] ad the total trasmissio eergy of all optical-flow packages i the video clip is used to detect evets i this video clip (i.e., we simply compare the total trasmissio eergy with a threshold to detect the abormal "escape" evets). (a) Groud Truth Proposed Method (b) Groud Truth Proposed Method (c) Groud Truth Proposed Method (d) Fig. 5 (a) ROC results of differet methods o the UMN dataset; (b)-(d) The qualitative results of usig our NTB algorithm for abormal detectio i the UMN dataset. The bars represet the labels of each frame, black represets ormal ad red represets abormal. (best viewed i color) Fig. 5 (b)-(d) compares the ormal/abormal classificatio results of our algorithm with the groud truth. Furthermore, Fig. 5 (a) compares the ROC curves betwee our algorithm (Proposed) ad the other four algorithms: the optical flow oly method (Optical Flow) [, ], the Social Force Model (SFM) [4], the Iteractio Eergy Potetial method (IEP) [], ad the Velocity-Field Based method (VFB) []. The results i Fig. 5 (a) show that by usig the simple optical flow feature ad a sigle relative etwork, our algorithm ca achieve similar or better results tha the state-of-the-art algorithms [-4]. This further demostrates the effectiveess of our NTB algorithm. Furthermore, several thigs eed to be metioed about Fig. 5. () Basically, i our ormal relative etwork (as i Fig. 7 (b)), packages movig to differet directios will cause differet positive ad egative eergies. Therefore, i ormal evets, sice people walk radomly i various directios, the optical-flow packages will create similar amouts of positive ad egative eergies such that the total trasmissio eergy will be aroud zero. However, durig the abormal "escape" evets, sice people are coheretly movig "outside", most optical-flow packages will create egative eergies. Thus, the total trasmissio eergy will be a large egative umber ad the abormalities ca be effectively detected. () I this experimet, the low-level optical flow features are used. This implies that i cases whe reliable trajectory caot be achieved (e.g., i extremely crowded scees), we ca also exted our algorithm by usig the low-level motio features to take the place of the trajectories for recogitio. () Note that i this experimet, we oly use the basic optical flow feature ad a sigle relative etwork for detectio. The performace of our NTB may be further improved by: (a) usig more reliable motio features such as the improved optical flow [], (b) itroducig additioal etworks to hadle more complicated scearios. E. Computatio complexity ad memory storage requiremets Fially, we evaluate the computatio complexity ad memory storage requiremets of our algorithm. () Computatio complexity. Our algorithm is ru o a PC with.6 GHz -Core CPU ad 4 G RAM while the traiig ad testig processes are implemeted by Matlab. Table 7 Computatio costs of our NTB algorithm o differet datasets Abormality dataset NTB+SVM NTB BEHAVE CASIA UMN Learig mi 6 mi 5 sec sec < ms Evaluatio ms ms ms < ms < ms Table 7 shows the computatio costs of our NTB algorithm i the experimets of Sectios V-A to V-D. From Table 7, we ca see that for the abormality detectio dataset, give about 5 traiig trajectories, the traiig process took about 6 miutes to coverge. Ad the etire testig process oly took about ms to process over 8 iput trajectories sice we oly eed to calculate the trasmissio eergies by applyig Eq. (). Besides, whe combied with huma detector [] ad particle-based trackig [6, 6] (implemeted by C++), our algorithm ca still achieve about frames/sec i the testig process. Therefore, our algorithm has low computatio complexity ad is suitable for real-time applicatios. For group activity recogitio i the BEHAVE dataset, give about 6 group trajectories i the traiig set, the traiig process took about 5 secods sice the DT eergies i the group activity experimets do ot eed to be traied i a iterative way. Ad etire testig processig also oly took about ms to process over iput trajectories. Similarly, our algorithm's complexity cost o the CASIA ad UMN datasets are also low. Moreover, Table 8 compares the computatio costs of our algorithm with the other methods o the CASIA dataset [7]

13 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, (ote that the complexity costs of the other methods are achieved from their publicatios [7] i order for a fair compariso). From Table 8, we ca also see that the computatio complexity of our algorithm is lower tha the compared methods. Table 8 Computatio costs of differet methods o the CASIA dataset HMM CHMM CODHMM_C CODHMM_CD Proposed Learig < mi < mi mi mi sec Evaluatio ms ms ms ms < ms () Memory storage requiremets. As for the storage issue, for abormal evet detectio case, we eed a N N matrix to store the fully coected etwork (i.e., the N N DT eergies betwee odes where N is the total umber of odes) ad a N matrix to store the miimum possible eergy E mi from the etrace patch u to all the other patches. I the example of Figs 9-, we have totally 84 odes ad thus oly oe ad oe 84 float type matrixes are eeded, which is small load for memory. For group activity recogitio case, sice the etworks i Fig. 7 are pre-calculated. We eve do ot eed matrixes to store the DT values ad all the DT eergy values ca be derived accordig to the relative locatio betwee people. Thus, the storage requiremets of the proposed method are low. VI. CONCLUSION AND FUTURE WORK I this paper, a ew etwork-based algorithm is proposed for huma activity recogitio i videos. The proposed algorithm models the etire scee as a etwork. Based o this etwork, we further model people i the scee as packages. Thus, various huma activities ca be modeled as the process of package trasmissio i the etwork. By aalyzig the trasmissio process, various activities such as abormal activities ad group activities ca be effectively recogized. Experimetal results demostrate the effectiveess of our algorithm. I the future, our algorithm may be further exteded i the followig ways: (a) I this paper, we assume that the camera is fixed ad directly divide patches i the image of the scee. However, we ca exted our algorithm by settig up a global coordiate of the etire scee ad divide patches of the scee i this global coordiate. I this way, eve whe the camera is movig or zoom-i/zoom-out, we ca also hadle these cases by first mappig the perso's locatio ito this global coordiate [5] ad the performig our algorithm iside the global coordiate. (b) Although our algorithm ca perform olie detectio, this olie detectio capability will fiish whe the abormality is detected (i.e., after the abormality is detected, all the later parts will be detected as abormal). However, ote that our algorithm ca be exteded to further hadle the olie detectio task eve after abormality happes. For example, we ca use the slidig widows to segmet the video ito clips ad the perform detectio i each video clip idepedetly. Ad these will be oe of our future works. REFERENCES [] J. K. Aggarwal ad M. S. Ryoo, "Huma Activity Aalysis: A Review," ACM Computig Surveys (CSUR), vol. 4, o. 6, pp. -47,. [] J. Nascimeto, M. Figueiredo, ad J. Marques, Segmetatio ad classificatio of huma activities, It l Workshop Huma Activity Recogitio ad Modelig, pp , 5. [] C. Rao, M. Shah, T. Syeda-Mahmood, Actio recogitio based o view ivariat spatio-temporal aalysis, Proceedigs of ACM Multimedia, pp ,. [4] E. E. Zeliker, S. Gog ad T. Xiag, Global abormal behavior detectio usig a etwork of CCTV cameras, It l Workshop. Visual Surveillace, pp. -8, 8. [5] Jaechul Kim ad Kriste Grauma, "Observe Locally, Ifer Globally: a Space-Time MRF for Detectig Abormal Activities with Icremetal Updates," IEEE Cof. Computer Visio ad Patter Recogitio (CVPR), pp. 998, 9. [6] B. Prosser, S. Gog ad T. Xiag, Multi-camera matchig uder illumiatio chage over time, Workshop o Multi-camera ad Multi-modal Sesor Fusio Algorithms ad Applicatios, pp., 8. [7] C. C. Loy, T. Xiag ad S. Gog. Modellig activity global temporal depedecies usig time delayed probabilistic graphical model, It l Cof. Computer Visio (ICCV), pp 7, 9. [8] C.-C. Chag ad C.-J. Li, LIBSVM : a library for support vector machies, ACM Tras. Itelliget Systems ad Techology, vol., o., pp. 7,. [9] B. Sog, A. Kamal, ad C. Soto, Trackig ad activity recogitio through cosesus i distributed camera etworks, IEEE Tras.Image Processig, vol. 9, pp ,. [] D. Zhag, D. Gatica-Perez, S. Begio, ad I. McCowa, Modelig idividual ad group actios i meetigs with layered HMMs, IEEE Tras. Multimedia, vol. 8, o., pp. 59 5, 6. [] Z. Cheg, L. Qi, Q. Huag, S. Jiag, ad Q. Tia, Group activity recogitio by Gaussia process estimatio, It l Cof. Patter Recogitio (ICPR), pp. 8,. [] [] W. Li, M.-T. Su, R. Poovedra ad Z. Zhag, Group evet detectio with a varyig umber of group members for video surveillace, IEEE Tras. Circuits ad Systems for Video Techology, pp ,. [] Y. Zhou, S. Ya ad T. Huag, Pair-activity classificatio by bi-trajectory aalysis, IEEE Cof. Computer Visio Patter Recogitio (CVPR), pp. -8, 8. [4] B. Ni, S. Ya ad A. Kassim, Recogizig huma group activities with localized causalities, IEEE Cof. Computer Visio ad Patter Recogitio (CVPR), pp , 9. [5] J. Li, S. Gog, ad T. Xiag, Scee segmetatio for behavior correlatio, ECCV, pp. 8 95, 8. [6] R. Hess ad A. Fer, Discrimiatively Traied Particle Filters for Complex Multi-Object Trackig, IEEE Cof. Computer Visio ad Patter Recogitio (CVPR), pp. 447, 9. [7] J. Li, S. Gog, ad T. Xiag, Discoverig multi-camera behaviour correlatios for o-the-fly global predictio ad aomaly detectio, It l Workshop. Visual Surveillace, pp. -7, 9. [8] BEHAVE set: [9] J. E. Wieselthier, G. D. Nguye, ad A. Ephremides, O the costructio of eergy-efficiet broadcast ad multicast trees i wireless etworks, INFOCOM, pp ,. [] J. Wu, C. Geyer ad J. M. Rehg, Real-time huma detectio usig cotour cues, IEEE Cof. Robotics ad Automatio (ICRA), pp ,. [] Z. Kalal, J. Matas, ad K. Mikolajczyk, Olie learig of robust object detectors durig ustable trackig, IEEE O-lie Learig for Computer Visio Workshop (OLCV), pp , 9. [] G. Wu, Y. Xu, X. Yag, Q. Ya, K. Gu, Robust object trackig with bidirectioal corer matchig ad trajectory smoothess algorithm, IEEE Iteratioal Workshop o Multimedia Sigal Processig (MMSP), pp. 9498,. [] M. Brow, D. Lowe, "Automatic paoramic image stitchig usig ivariat features," Iteratioal Joural of Computer Visio, vol. 74, o., pp. 59-7, 7. [4] KDE Toolbox: [5] L. G. Mirisola, J. Dias, ad A. T Almeida, Trajectory recovery ad d mappig from rotatio-compesated imagery for a airship, IEEE It l Cof. Itelliget Robots ad Systems, pp. 98-9, 7. [6] CASIA Actio Database [Olie]. Available: [7] P. Guo, Z. Miao, X. Zhag, Y. She ad S. Wag, Coupled observatio decomposed hidde markov model for multiperso activity recogitio, IEEE Tras. Circuits ad Systems for Video Techology, vol., o. 9, pp. 6,.

14 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 4 [8] B. D. Lucas ad T. Kaade, A iterative image registratio techique with a applicatio to stereo visio, Proceedigs of Imagig Uderstadig Workshop, pp , 98. [9] L. McCowa, D. Gatica-Perez, S. Begio, G. Lathoud, M. Barard, ad D. Zhag, Automatic aalysis of multimodal group actios i meetigs, IEEE Tras. Patter Aal. Mach. Itell., vol. 7, o., pp. 5-7, 5. [] M. Brad, N. Oliver, ad A. Petlad, Coupled hidde Markov models for complex actio recogitio, i Proc. IEEE Cof. Comput. Vis. Patter Recogit., pp , 997. [] Uusual Crowd Activity Dataset: [] J. Zhao, Y. Xu, X. Yag, ad Q. Ya, Crowd Istability Aalysis Usig Velocity-Field Based Social Force Model, Visual Commuicatios ad Image Processig (VCIP), pp. -4,. [] X. Cui, Q. Liu, M. Gao, Metaxas ad D. N, Abormal Detectio Usig Iteractio Eergy Potetials, IEEE Cof. Computer Visio ad Patter Recogitio (CVPR), pp. 6-67,. [4] R. Mehra, A. Oyama, ad M. Shah, Abormal Crowd Behavior Detectio usig Social Force Model, IEEE Cof. Computer Visio ad Patter Recogtio (CVPR), pp , 9. Weiyao Li received the B.E. degree from Shaghai Jiao Tog Uiversity, Chia, i, the M.E. degree from Shaghai Jiao Tog Uiversity, Chia, i 5, ad the Ph.D degree from the Uiversity of Washigto, Seattle, USA, i, all i electrical egieerig. Curretly, he is a associate professor at the Departmet of Electroic Egieerig, Shaghai Jiao Tog Uiversity. His research iterests iclude video processig, machie learig, computer visio ad video codig & compressio. Yuazhe Che, received the B.S. degree from Shaghai Jiao Tog Uiversity, Shaghai, Chia, i i electrical egieerig. He is curretly workig toward the M.E. degree i electrical egieerig from both Shaghai Jiao Tog Uiversity, Chia ad Georgia Tech Uiversity, USA. His research iterests iclude image & video processig, machie learig, computer visio ad multimedia techologies. Hali Wag received the B.S. ad M.S. degrees i electrical egieerig from Zhejiag Uiversity, Hagzhou, Chia, i ad 4, respectively, ad the Ph.D. degree i computer sciece from the City Uiversity of Hog Kog, Kowloo, Hog Kog, i 7. He is curretly a Professor with the Departmet of Computer Sciece ad Techology, Togji Uiversity, Shaghai, Chia. He was a Research Fellow with the Departmet of Computer Sciece, City Uiversity of Hog Kog, from 7 to 8. From 7 to 8, he was a Visitig Scholar with Staford Uiversity, Palo Alto, CA. From 8 to 9, he was a Research Egieer with Precoad, Ic., Melo Park, CA. From 9 to, he was a Alexader vo Humboldt Research Fellow with the Uiversity of Hage, Hage, Germay. His curret research iterests iclude digital video codig, image processig, patter recogitio, ad video cotet aalysis. Bi Sheg received his BA degree i Eglish ad BE degree i Computer Sciece from Huazhog Uiversity of Sciece ad Techology i 4, MS degree i software egieerig from the Uiversity of Macau i 7, ad PhD degree i Computer Sciece from The Chiese Uiversity of Hog Kog i. He is curretly a assistat professor i the Departmet of Computer Sciece ad Egieerig at Shaghai Jiao Tog Uiversity. His research iterests iclude virtual reality, computer graphics, ad image based techiques. Hogxiag Li is a Assistat Professor with the Departmet of Electrical ad Computer Egieerig, Uiversity of Vouisville, Vouisville, KY, USA. His research iterests iclude broadbad mobile commuicatios ad wireless etworks. He received a B.S. degree from Xi a Jiaotog Uiversity, Chia i, a M.S. degree from Ohio Uiversity i 4, ad a Ph.D. degree from Uiversity of Washigto, Seattle i 8, all i electrical egieerig. From 8 to, he was a Assistat Professor with the Departmet of Electrical ad Computer Egieerig, North Dakota State Uiversity, Fargo, ND,USA. Jiaxi Wu received his BS & MS degree i computer sciece from Najig Uiversity, ad the PhD degree i computer sciece from Georgia Istitute of Techology. He is curretly a professor i Najig Uiversity, ad was a assistat professor i Nayag Techological Uiversity. His research iterests are computer visio ad machie learig.