Detecting Irregularities in Images and in Video

Transcription

1 Internatonal Journal of Computer Vson 74(1, 17 31, 2007 c 2007 Sprnger Scence + Busness Meda, LLC. Manufactured n the Unted States. DOI: /s Detectng Irregulartes n Images and n Vdeo OREN BOIMAN AND MICHAL IRANI Department of Computer Scence and Appled Math, The Wezmann Insttute of Scence, Rehovot, Israel Receved March 7, 2006; Accepted November 14, 2006 Frst onlne verson publshed n January, 2007 Abstract. We address the problem of detectng rregulartes n vsual data, e.g., detectng suspcous behavors n vdeo sequences, or dentfyng salent patterns n mages. The term rregular depends on the contet n whch the regular or vald are defned. Yet, t s not realstc to epect eplct defnton of all possble vald confguratons for a gven contet. We pose the problem of determnng the valdty of vsual data as a process of constructng a puzzle: We try to compose a new observed mage regon or a new vdeo segment ( the query usng chunks of data ( peces of puzzle etracted from prevous vsual eamples ( the database. Regons n the observed data whch can be composed usng large contguous chunks of data from the database are consdered very lkely, whereas regons n the observed data whch cannot be composed from the database (or can be composed, but only usng small fragmented peces are regarded as unlkely/suspcous. The problem s posed as an nference process n a probablstc graphcal model. We show applcatons of ths approach to dentfyng salency n mages and vdeo, for detectng suspcous behavors and for automatc vsual nspecton for qualty assurance. Keywords: detectng suspcous behavors, salency, detectng rregulartes, novelty detecton, anomaly detecton, acton recognton, automatc vsual nspecton 1. Introducton Detecton of rregular vsual patterns n mages and n vdeo sequences s useful for a varety of tasks. Detectng suspcous behavors or unusual objects s mportant for survellance and montorng. Identfyng spatal salency n mages s useful for qualty control and automatc nspecton. Behavoral salency n vdeo s useful for drawng the vewer s attenton. Prevous approaches to recognton of suspcous behavors or actvtes can broadly be classfed nto two classes of approaches: rule-based methods (e.g., Ivanov and Bobck (1999 and statstcal methods wthout predefned rules (e.g., Stauffer and Grmson (2000 and Zhong et al. (2004. The statstcal methods are more appealng, snce they do not assume a predefned set of rules for all vald confguratons. Instead, they try to automatcally learn the noton of regularty from the data, and thus nfer about the suspcous. Nevertheless, the representatons employed n prevous methods have Patent Pendng been ether very restrctve (e.g., trajectores of movng objects (Stauffer and Grmson, 2000, or else too global (e.g., a sngle small descrptor vector for an entre frame (Zhong et al., In ths paper we formulate the problem of detectng regulartes and rregulartes as the problem of composng (eplanng the new observed vsual data (an mage or a vdeo sequence, referred to below as query usng spato-temporal patches etracted from prevous vsual eamples (the database. Regons n the query whch can be composed usng large contguous chunks of data from the eample database are consdered lkely. The larger those regons are, the greater the lkelhood s. Regons n the query whch cannot be composed from the eample database (or can be composed, but only usng small fragmented peces are regarded as unlkely/suspcous. Our approach can thus nfer and generalze from just a few eamples, about the valdty of a much larger contet of mage patterns and behavors, even f those partcular confguratons have never been seen before. Local descrptors are etracted from small mage or vdeo patches (composed together to large ensembles of patches, thus

2 18 Boman and Iran allowng to quckly and effcently nfer about subtle but mportant local changes n behavor (e.g., a man walkng vs. a man walkng whle pontng a gun. Moreover, our approach s capable of smultaneously dentfyng a vald behavor n one porton of the feld of vew, and a suspcous behavor n a dfferent porton the feld of vew, thus hghlghtng only the detected suspcous regons wthn the frame, and not the entre frame. Such eamples are shown n Secton 6. Inference from mage patches or fragments has been prevously employed n the task of class-based object recognton (e.g. Bart and Ullman (2004, Felzenszwalb and Huttenlocher (2005 and Fergus et al.. A small number of nformatve fragments have been learned and preselected for a small number of pre-defned classes of objects. However, class-based representatons cannot capture the overwhelmng number of possbltes of composng unknown objects or behavors n a scene, and are therefore not sutable for our underlyng task of detectng rregulartes. Our approach can also be appled for detectng salency n mages and n vdeo sequences. For eample, gven a sngle mage wth no pror nformaton, we can measure the valdty of each mage regon (the query relatve to the remanng portons of the same mage (the database used for ths partcular query. An mage regon wll be detected as salent f t cannot be eplaned by anythng smlar n other portons of the mage. Smlarly, gven a sngle vdeo sequence (wth no pror knowledge of what s a normal behavor, we can detect salent behavors as behavors whch cannot be supported by any other dynamc phenomena occurrng at the same tme n the vdeo. Prevous approaches for detectng mage salency (e.g., Itt et al. (1998 proposed measurng the degree of dssmlarty between an mage locaton and ts mmedate surroundng regon. Thus, for eample, mage regons whch ehbt large changes n contrast are detected as salent mage regons. Ther defnton of vsual attenton s derved from the same reasonng. Nevertheless, we beleve that the noton of salency s not necessarly determned by the mmedate surroundng mage regons. For eample, a sngle yellow spot on a black paper may be salent. However, f there are many yellow spots spread all over the black paper, then a sngle spot wll no longer draw our attenton, even though t stll nduces a large change n contrast relatve to ts surroundng vcnty. Our approach therefore suggests a new and more ntutve nterpretaton of the term salency, whch stems from the nner statstcs of the entre mage. Our approach to spatal mage salency s more closely related to that of Honda and Nayar (2001. However, (Honda and Nayar, 2001 s restrcted to repettve structured mage patterns and s hghly dependent on the local surroundng mage propertes, whereas our approach s not. Eamples of detected spatal salency n mages and behavoral salency n vdeo sequences usng our approach are shown n Secton 6. Our paper therefore offers four man contrbutons: 1. We propose an approach for nferrng and generalzng from just a few eamples, about the valdty of a much larger contet of mage patterns and behavors, even f those partcular confguratons have never been seen before. 2. We present a new graph-based Bayesan nference algorthm whch allows to effcently detect large ensembles of patches (e.g., hundreds of patches, at multple spato-temporal scales. It smultaneously mposes constrants on the relatve geometrc arrangement of these patches n the ensemble as well as on ther descrptors. 3. We propose a new nterpretaton to the term salency and vsual attenton n mages and n vdeo sequences. 4. We present a sngle unfed framework for treatng several dfferent problems n Computer Vson, whch have been treated separately n the past. These nclude: attenton n mages, attenton n vdeo, recognton of suspcous behavors, recognton of unusual objects, automatc vsual nspecton (e.g., for qualty assurance, and more. A shorter verson of ths paper appeared n ICCV 2005 (Boman and Iran, Inference by Composton Gven only a few eamples, we (humans have a noton of what s regular/vald, and what s rregular/suspcous, even when we see new confguratons that we never saw before. We do not requre eplct defnton of all possble vald confguratons for a gven contet. The noton of regularty / valdty s learned and generalzed from just a few eamples of vald patterns (of behavor n vdeo, or of appearance n mages, and all other confguratons are automatcally nferred from those. Fgure 1llustrates the basc concept underlyng ths dea n the paper. Gven a new mage (a query Fg. 1(a, we check whether each mage regon can be eplaned by a large enough contguous regon of support n the database (see Fgs. 1(b and (c. Although we have never seen a man sttng wth both arms rased, we can nfer the valdty of ths pose from the three database mages of Fg. 1(c. Thus, regons n the new observed data/query (an mage or a vdeo sequence whch can be eplaned by large contguous chunks of data from the database are consdered very lkely, whereas regons n the query whch

3 Detectng Irregulartes n Images and n Vdeo 19 Fgure 1. The basc concept Inference by Composton. A regon n the query mage s consdered lkely f t has a large enough contguous regon of support n the database. New vald mage confguratons can thus be nferred from the database, even though they have never been seen before. Fgure 2. Detectng a matchng ensemble of patches. cannot be eplaned by large enough database peces are consdered unlkely or suspcous. When the vsual query s an mage, then those chunks of data have only a spatal etent. When the vsual query s a vdeo sequence, then those chunks of data have both a spatal and a temporal etent. 3. Ensembles of Patches Human behavors and natural spatal structures never repeat dentcally. For eample, no two people walk n the same manner. One may rase hs arms hgher than the other, or may just walk faster. We therefore want to allow for small non-rgd deformatons (n space and n tme n our peces of puzzle (chunks of data. Ths s partcularly true for large chunks of data. To account for such local non-rgd deformatons, large chunks are broken down to an ensemble of lots of small patches at multple scales wth ther relatve geometrc postons. Ths s llustrated n Fg. 1(d. In the nference process, we search for a smlar geometrc confguraton of patches wth smlar propertes (of behavor, or of appearance, whle allowng for small local msalgnments n the relatve geometrc arrangement. Ths concept s llustrated n Fg. 2. When the vsual query s an mage, then an ensemble of patches s composed of spatal patches (see Fg. 3(a. When the vsual query s a vdeo sequence, then the ensemble of patches s composed of spato-temporal patches (see Fg. 3(b, whch allows to capture nformaton about dynamc behavors. In our current mplementaton, a sngle ensemble typcally contans hundreds of patches, smultaneously from multple scales (multple spatal scales n the case of mage patches, and multple space-tme scales n the case of spato-temporal patches. Whle the dea of composng new data from eample patches was prevously proven useful for a varety of tasks (e.g., Efros and Leung (1999, Freeman et al. (2000 and Weler et al. (2004, these methods dd not mpose any geometrc restrcton on the eample patches

4 20 Boman and Iran used for constructon,.e., ther relatve postons and dstances n the database. Ths was not necessary for ther purpose. It s however crucal here, for the purpose of detectng rregulartes. Often, the only real cue of nformaton for dstngushng between a lkely and an unlkely phenomenon s the degree of fragmentaton of ts support n the database. For eample, the stretched arm of a man holdng a gun s smlar to an nstantaneous stretchng of the arm whle walkng, but ts regon of support s very lmted n tme. Capturng the geometrc relatons of patches was dentfed as beng mportant for the task of class-based object recognton (Bart and Ullman, 2004; Felzenszwalb and Huttenlocher, 2005; Fergus et al.,; Lebe et al. Those approaches are not sutable for our objectve for two reasons: ( Ther geometrc confguratons are restrcted to a relatvely small number of patches, thus cannot capture subtle dfferences whch are crucal for detecton of rregulartes. ( Those confguratons were pre-learned for a small number of pre-defned classes of objects, whereas our framework s applcable to any type of vsual data. Whle the geometrc constrants of Lebe et al. are more fleble, thus allowng to recognze new object confguratons from just a few eamples, ther method s stll lmted to a set of predefned object classes wth predefned object centers. Ths s not sutable for detectng rregulartes, where there s no noton of object classes. Vdeo Google (Svc and Zsserman, 2003 mposes geometrc constrants on large collectons of non classbased descrptors, and searches for them very effcently. However, those descrptors are spatal n nature and the search s restrcted to ndvdual mage frames, thus not allowng to capture behavors. In order for the nference to be performed n reasonable tmes, nformaton about the small patches and ther relatve arrangement must be effcently stored n and etracted from the database. For each small patch etracted from the eamples, a descrptor vector s computed and stored (see below, along wth the absolute coordnates of the patch (spatal or spato-temporal coordnates. Thus, the relatve arrangement of all patches n the mage/vdeo database s mplctly avalable. Later, our nference algorthm takes an ensemble of patches from the vsual query and searches the database for a smlar confguraton of patches (both n the descrptors and n ther relatve geometrc arrangement. To allow for fast search and retreval, those patches are stored n a mult-scale data structure. Usng a probablstc graphcal model (Secton 4, we present an effcent nference algorthm (Secton 4.2 for the ensemble search problem Patch Descrptors Fgure 3. Ensembles of patches n mages and vdeo. Patch descrptors are generated for each query patch and for each database patch. The descrptors capture local nformaton about appearance/behavor. Our current mplementaton uses very smple descrptors, whch could easly be replaced by more sophstcated descrptors: The Spatal Image Descrptor of a small (e.g., 7 7 spatal patch s constructed as follows: The spatal gradent magntude s computed for each pel n the patch. These values are then stacked n a vector, whch s normalzed to a unt length. Such descrptors are densely etracted for each pont n the mage. Ths descrptor etracton process s repeated n several spatal scales of the spatal Gaussan pyramd of the mage. Thus, a 7 7 patch etracted from a coarse scale has a larger spatal support n the nput mage (.e., n the fne scale. In some applcatons an RGB/ntensty-based descrptor may be more approprate than a gradent-based one. In general our overall framework s not restrcted to those partcular descrptors. Those could be easly replaced by more sophstcated spatal descrptors such as SIFT (Lowe, 2004 etc. The Spato-Temporal Vdeo Descrptor of a small (e.g., spato-temporal vdeo patch s constructed from the absolute values of the temporal dervatves n all pels of the patch. These values are stacked n a vector and normalzed to a unt length. Ths descrptor etracton process s repeated n several spatal and temporal scales of a space-tme vdeo pyramd. Thus, a patch etracted from a coarse scale has a larger spatal and larger temporal support n the nput sequence. Note that ths descrptor s nearly nvarant to a statc background, snce the temporal dervatve s always zero n any statc background. Therefore, usng ths spatotemporal descrptor, we can detect rregular actons n a new query sequence, regardless of the background. However, ths smple descrptor s dependent on spatal teture, whch may pose a problem wth people wearng hghly tetured clothes. Our approach, however, s not restrcted to the partcular choce of these smple descrptors. Those descrptors could be easly replaced by more sophstcated space-tme descrptors (whch are acton-senstve and more nvarant to appearance, such as Shechtman and Iran (2005 or Laptev and Lndeberg (2003.

5 Detectng Irregulartes n Images and n Vdeo The Basc Algorthm Gven a new vsual query (an mage or a vdeo sequence, we would lke to estmate the lkelhood of each and every pont n t. Ths s done by checkng the valdty of a large regon (e.g., regon n an mage, and regon n a vdeo sequence surroundng every pel. The large surroundng regon s broken nto lots (hundreds of small patches at multple scales (spatal or spato-temporal, and s represented by a sngle ensemble of patches correspondng to that partcular mage/vdeo pont. Let q 1, q 2,...,q n denote the patches n the ensemble (see Fg. 3(a. Each patch q s assocated wth two types of attrbutes: ( ts descrptor vector d, and ( ts locaton n absolute coordnates l.we choose an arbtrary reference pont c (e.g., the center of the ensemble see Fg. 3(a, whch serves as the orgn of the local coordnate system (thus defnng the relatve postons of the patches wthn the ensemble Statstcal Formulaton Let an observed ensemble of patches wthn the query be denoted by y. We would lke to compute the jont lkelhood P(, y that the observed ensemble y n the query s smlar to some hdden ensemble n the database (smlar both n ts descrptor values of the patches, as well as n ther relatve postons. We can factor the jont lkelhood as: P(, y = P(y P(. Our modellng of P(y resembles the probablstc modellng of the star graph of Felzenszwalb and Huttenlocher (2005. However, n the class-based settng of Felzenszwalb and Huttenlocher (2005 what s computed s P(y; θ, where θ s a prelearned set of parameters of a gven patch-constellaton of an object-class. In our case, however, there s no noton of objects,.e., there s no pror parametrc modellng of the database ensemble. Thus, θ s undefned, and P( must be estmated non-parametrcally drectly from the database of eamples. Let dy denote the descrptor vector of the -th observed patch n y, and ly denote ts locaton (n absolute coordnates. Smlarly, d denotes the descrptor vector of the -th hdden (database patch n, and l denotes ts locaton. Let c y and c denote the orgn ponts of the observed and hdden ensembles. The smlarty between any such par of ensembles y and s captured by the followng lkelhood: P(, y= P ( c, d 1,...,l1,..., c y, d 1 y,...,l1 y,... (1 In order to make the computaton of the lkelhood n Eq. (1 tractable, we make some smplfyng statstcal assumptons. Gven a hdden database patch and ts descrptor d, the correspondng observed descrptor d y s assumed to be ndependent of the other patch descrptors. (Ths s a standard Markovan assumpton, e.g., Freeman et al. (2000, whch s obvously not vald n case of overlappng patches, but s a useful appromaton. We model the smlarty between descrptors usng a Gaussan dstrbuton: P ( d y d = α1 ep ( 1 2( d y d T ( S 1 D d y d (2 where α 1 s a constant, and S D s a constant covarance matr, whch determnes the allowable devaton n the descrptor values. Other dstrbutons can be plugged n the model, correspondng to other descrptor smlarty functons. Gven the relatve locaton of the hdden database patch (l c, the relatve locaton of the correspondng observed patch (ly c y s assumed to be ndependent of all other patch locatons. Ths assumpton enables to compare the geometrc arrangement of two ensembles of patches wth enough fleblty to accommodate for small changes n vewng angle, scale, pose and behavor. Thus: P ( ( ly l, c, c y = α2 ep 1 (( ( l 2 y c y l T c S 1 (( ( L l y c y l c (3 where α 2 s a constant, and S L s a constant covarance matr, whch captures the allowed devatons n the relatve patch locatons. (In ths case the dependency n relatve locatons was modelled usng a Gaussan, however the model s not restrcted to that. So far we modelled the relatons between attrbutes across ensembles (descrptors: dy, d, and relatve locatons: ly c y, l c. We stll need to model the relatons wthn the hdden ensemble, namely, the relatons between a patch descrptor d to ts locaton l. In the general case, ths relaton s hghly non-analytc, and hence cannot be modelled parametrcally (n contrast to classbased approaches, e.g. Felzenszwalb and Huttenlocher (2005 and Fergus et al. (2003. Therefore, we model t non-parametrcally usng eamples from the database: P (d l = { 1 (d, l Database 0 otherwse (4 where d and l are an arbtrary descrptor and locaton. We assume a unform pror dstrbuton for c and c y (local orgn ponts,.e., no pror preference for the locaton of the ensemble n the database or n the query. The relaton between all the above-mentoned varables s depcted n the Bayesan network n Fg. 4. Thus, for an observed ensemble y and a hdden database ensemble, we can factor the jont lkelhood

6 22 Boman and Iran covarance matrces, each assocated wth a dfferent part n the model. Ths s a good approach when the recognton task s restrcted to a few known predefned classes, each wth ts pre-defned parts and parameters. Ths, however, s not the settng n our case, where there s no predefned noton of what we are lookng for, yet, we want to be able to detect subtle rregulartes compared to the eamples. Our model s therefore non-parametrc and ts generalzaton capabltes do not rely on parameter tunng, but rather on the dversty of the eamples n the database. In that sense, our non-parametrc modellng bears resemblance to the non-parametrc treatment of Lebe et al. In our mplementaton we have set the covarance matrces S D and S L to smple scalar varance determned emprcally. Ths smple settng was satsfactory for our eperments. Note that n ths settng, the sole purpose of these two parameters s to properly weght the costs of geometrc deformatons and appearance/descrptor deformatons. Moreover, note that these are the only parameters n the model, and therefore requres very lttle parameter tunng. Fgure 4. The probablstc graphcal model. The Bayesan dependences are llustrated usng the arrows between varables. The dependences are llustrated only for one patch n the ensemble (the -th patch. Observed varables are marked n orange ; Hdden varables are marked n blue. c and c y are the orgn of the hdden and observed ensembles, respectvely. l and l y are the locatons (n absolute coordnates of the -th patch n the hdden and observed ensembles; d and d y are the descrptor vectors of the -th patch n each ensemble. P(, y of Eq. (1 usng Eqs. (2 (4 as follows: P ( c, d 1,...,l1,...,c y, dy 1,...,l1 y = α P ( ly l, c (, c y P d y d ( P d l (5 For any hdden ensemble assgnment wth non-zero lkelhood, we defne the composton cost as the mnus log lkelhood functon: log P ( c, d 1,...,l1,...,c y, dy 1,...,l1 y = log P ( ly l, c, c y + log P ( d y d +α1 (6 Where α 1 = log(α s a constant. The frst term s the overall cost of local geometrc deformatons n the ensemble, whle the second term s the overall cost of appearance/descrptor deformatons n the ensemble. In our formulaton, the covarance matrces S D and S L are constant. The reason for ths s that our approach s non-parametrc and data-drven. Parametrc approaches to object recognton (such as Felzenszwalb and Huttenlocher (2005, and Fergus et al. allow for multple learnt 4.2. Belef Propagaton Inference Gven an observed ensemble, we seek a hdden database ensemble,whch mamzes ts MAP (mamum a- posteror probablty assgnment. Ths s done usng the above statstcal model, whch has a smple and eact Belef Propagaton algorthm (Yedda et al., Accordng to Eq. (5 the MAP assgnment can be wrtten as: ma X = α P ( c, d 1,...,l1,...,c y, dy 1,...,l1 y ma l P ( l y l, c, c y ma P ( d d y d P ( d l (7 Ths epresson can be phrased as a message passng algorthm n the graph of Fg. 4. Frst we compute for each patch the message m dl passed from node d to node l regardng ts belef n the locaton l : m dl( l = ma d P ( d y d P ( d l (8 Namely, for each observed patch, compute all the canddate database locatons l wth hgh descrptor smlarty. Net, for each of these canddate database locatons, we pass a message about the nduced possble orgn locatons c n the database: m lc (c = ma l P ( l y l, c, c y mdl ( l (9

7 Detectng Irregulartes n Images and n Vdeo 23 At ths pont, we have a canddate lst of orgns suggested by each ndvdual patch. To compute the lkelhood of an entre ensemble assgnment, we multply the belefs from all the ndvdual patches n the ensemble: m c (c = m lc (c (10 The nference performed by ths algorthm s a MAP nference. Therefore, somethng that occurred once n the eamples database s equally lkely as somethng that occurred many tmes. Ths formulaton s useful n many applcatons, however, there may be applcatons where we would lke the frequency of occurrence n the database to affect the lkelhood of an ensemble. A smple modfcaton of the above algorthm allows to compute lkelhood nstead of MAP, by transformng the nference algorthm from a ma-product to a sum-product Estmatng the Lkelhood of a Query Pont For each pont n the query, we try to compose a large regon around t. Ths s done by checkng the valdty of a large regon surroundng every pont, usng the above nference process (by computng a query regon lkelhood. Ths pont partcpates n many query regons. We defne the lkelhood of a query pont as the mamal lkelhood of a regon contanng that pont. Therefore, a pont n the query wll have a hgh lkelhood, f there ests a large regon contanng t, wth a correspondng smlar database regon. Ths way, we can compose queres wth partal occluson of objects, snce ponts whch are near the boundary are contaned n a large regon nsde the object. However, partal occlusons mght create small contguous regons of objects, whch cannot be composed wth hgh lkelhood usng our current nference algorthm. We would lke the regon that we compose around every pont to be as large as possble, because the larger the regon s, the hgher the evdence that the pont s not rregular. However, there are cases n whch a regular observed ensemble cannot be fully composed by a sngle database ensemble (e.g., due to partal occluson. In those cases (whch are not very frequent, we reduce the sze of the observed regon (e.g., by 25% and repeat the nference process wthout the dscarded patches. We penalze the overall ensemble lkelhood score for each patch we dscard. In terms of Eq. (6 we add a constant cost penalty for every patch we dscard. The magntude of the penalty term, reflects the mportance we attrbute to the composton regon sze. Handlng Ensembles of Dfferent Szes: In order to detect rregular regons n an entre observaton, we can smply threshold the composton cost n Eq. (6. However, there may be cases where the sze of the observed ensemble would be dfferent (e.g., because of non-nformatve regons, regons ecluded from analyss, data boundares, etc.. In order to compare composton cost of ensembles of dfferent szes, a normalzaton s requred. We use a normalzaton based on the statstcal sgnfcance of the composton cost. We defne the null-hypothess H 0 such that each observed ensemble was generated usng the statstcal model defned above. Therefore, the statstcal sgnfcance of a composton cost C 0 can be measured by the pvalue Pr(C > C 0 H 0. Assumng the null-hypothess, and gven the hdden ensemble, each term n the composton cost n Eq. (6 s dstrbuted χ 2 and the overall cost s also dstrbuted χ 2. These dstrbutons can be used to compute the pvalue whch normalzes the composton cost for ensembles of dfferent szes. 5. An Effcent Inference Algorthm A nave mplementaton of the message passng algorthm presented n Secton 4.2 s very neffcent, snce ndependent descrptor queres are performed for each patch n the observaton ensemble, regardless of answers to prevous queres performed by other patches. Ths results n a complety of O(Nk where N s the number of patches n the database (e.g., 100,000 patches for a onemnute vdeo database and k s the number of patches n the ensemble (e.g., 256. Moreover, we should scan the entre query (the new mage or new vdeo sequence, whch results n a total complety of O(Nkq, where q s the number of patches n the query. The complety s prohbtve for real applcatons, because each of the terms (N, k and q s not neglgble. In ths secton we show how to sgnfcantly reduce the complety wthout sacrfcng accuracy The Progressve Elmnaton Process The patches n the observed ensemble are related by a certan geometrc arrangement. We can use ths knowledge for an effcent search by progressve elmnaton of the search space n the database: We compute the message m dl for a small number of patches (e.g., 1. The resultng lst of possble canddate orgns nduces a very restrcted search space for the net patch. The net patch, n turn, elmnates addtonal orgns from the already short lst of canddates, etc. Ths process s llustrated n Fg. 5. In order to speed-up the progressve elmnaton, we use truncated Gaussan dstrbutons (truncated after 4σ, n Eqs. (2 and (3. Therefore, these dstrbuton gve a lkelhood of zero to hgh patch deformatons n terms of geometry or appearance/descrptor. The search of the frst patch costs O(N. We keep only the best c cand-

8 24 Boman and Iran Fgure 5. The progressve elmnaton process. Step 1: Searchng for the frst patch (blue square n the database yelds several possble locatons. Each locaton nduces probablty for the locaton of the hdden ensemble center (blue crcles. Step 2: We use the PDF of the hdden ensemble center nduced by the frst (blue patch n order to search for the second (red patch only n non-zero lkelhood regons (dotted red crcles. (c Step 3: Each detected locaton of the red patch nduces another PDF for the hdden ensemble center. Locatons where one of the PDFs s zero are elmnated. We proceed to process all the other patches n the ensemble n the same way. date orgns from the lst proposed by the frst patch (n our mplementaton, c = 50. The second patch s now restrcted to the neghborhoods of c locatons. The thrd wll be restrcted to a much smaller number of neghborhoods. Thus, n the worst case scenaro, our complety s O(N + kc O(N. In contrast, the complety of the nference process n Felzenszwalb and Huttenlocher (2005 and Lebe et al. s O(Nk, whle the complety of the constellaton model (Fergus et al. s eponental n the number of patches. The above proposed reducton n complety s etremely mportant for enablng vdeo nference wth ensembles contanng hundreds of patches. Note that lmtng the number of canddate orgn ponts to c canddates mght be problematc: For nstance, f the frst patch we choose s non-nformatve (.e., sngle edge, then choosng the best c canddates s arbtrary and we mght dscard the globally optmal ensemble. In practce, other components of our nference algorthm (mult-scale strategy, predctve search, and scannng the observaton elmnate ths rsk. Note that f we assume truncated Gaussan dstrbutons (or other fnte support dstrbutons, and f searchng for the frst few patches yelds less than c canddate locatons, then the progressve elmnaton process guarantees an eact soluton, because we only dscard canddates wth zero lkelhood. Note that ths entals that under such condtons we can offer an eact nference whch s equvalent to Belef Propagaton wth reduced complety. Moreover, we know durng the nference f the result s eact (optmal or f t s only an appromaton Mult-Scale Search To further speedup the elmnaton process, we use a coarse-to-fne strategy (both n space and n tme. We choose the frst searched patches from a coarse scale, for two reasons: ( There s a much smaller number of coarse patches n the database than fne patches (thus decreasng the effectve N n the frst most ntensve step, and ( coarse patches are more dscrmnatve because they capture nformaton from large regons. Ths elmnates canddate orgns of database ensembles very quckly. We proceed untl we process all the coarse scale patches n the observed ensemble. Then we project the canddate orgn ponts to the net fner scale and contnue to process patches n the fner scale (both n space and n tme. We proceed n ths mult-scale manner to process

9 Detectng Irregulartes n Images and n Vdeo 25 all the patches n the observed ensemble. The complety of the mult-scale search s O(N 0 + kc, where N 0 s the number of patches n the coarsest scale of the space-tme pyramd Effcent Database Storage and Retreval A smple mplementaton of the database would be to use an array of patch descrptors and search t lnearly. However, tme and space complety can be mproved sgnfcantly for database retreval and storage, respectvely. Storage space can be reduced sgnfcantly by keepng appromatons of the descrptor vectors. For nstance, all the descrptor vectors can be projected on a low dmensonal lnear space usng standard technques such as PCA and ICA. In addton, vector quantzaton technques (such as Kmeans, or Jure and Trggs (2005 can be used to cluster groups of descrptors. The result of projecton and quantzaton s that there are less descrptor types to store, and each descrptor vector s shorter. Another beneft s that database retreval tme s reduced. Note that projecton and quantzaton ntroduce errors n the descrptor vectors. We can elmnate the error f each compressed descrptor contans a lnk to the orgnal descrptor. In ths case, storage space would not be reduced, but the retreval tme would be reduced. A closely related approach to reduce database retreval tme s to use better data structures for storng the descrptor vectors, such as KD-trees and hash-tables for fndng appromate nearest neghbors. These data-structures enable fast range queres (fndng all elements n the database n a certan range around a gven element. The resultng tme complety s O(Range(N 0 + kc, where Range(N 0 N 0 s the cost of a range query n the database data structure wth N 0 elements (patches n t Usng Predctve Search So far we assumed that the composton algorthm descrbed above s appled to all the ponts n the observaton, ndependently of each other. Ths s usually wasteful as neghborng observed ensembles tend to have neghborng hdden ensembles n the database. We utlze ths fact to speed up the composton by predctng the values of hdden ensemble varables n space and n tme. By usng all the prevously composed ensembles n the vcnty of the current ensemble (n space and n tme, we predct the locaton of the hdden ensemble center and the dentty of the hdden patches n the database, usng knowledge obtaned for the overlappng observed patches. We use the smplest predcton: Gven a neghborng observed ensemble (ỹ and ts correspondng detected database ensemble we predct some of the hdden varables n hdden ensemble correspondng to a new observed ensemble y. We predct the hdden ensemble center c usng: c = c + c y c y (11 Moreover, for each observed patch (ly, d y, whch partcpated n the predctng ensemble (ly, d y = ( l y, j d y j we predct the correspondng hdden varables (l, d = ( l j, d j. The rest of the hdden varables, whch are not predcted, can be nferred very quckly usng the progressve elmnaton process. Note that for neghborng ensembles, most of the observed patches overlap, therefore the complety of composng a new ensemble s very low. In cases where the predcton s bad and hence results n a low qualty composton (.e., low lkelhood of the observed regon, we dscard the predcton results and use the usual nference over the entre database. Thus, the predctve search does not prevent detecton elsewhere n the database. However, n most cases the predctve search s qute accurate and reduces the nference tme consderably. Assume that there s a chan of vald predctons of length r. The cost of predctng an ensemble n ths chan s O(k. Therefore the total complety of such a chan s O(Range(N 0 + kc + kr nstead of O(Range(N 0 r + krc wthout predcton. Besdes sgnfcantly reducng total nference tme, predcton actually mproves the accuracy of nference. Ths s because regons where the composton was accurate, propagate nformaton to regons wth less certanty (e.g., the leg of a standng person has less certanty than the upper part of the body. 6. Applcatons The approach presented n ths paper gves rse to a varety of applcatons whch nvolve detecton of rregulartes n mages and n vdeos: 6.1. Detectng Unusual Image Confguratons Gven a database of eample mages, we can detect unusual thngs n a new observed mage (such as objects never seen before, new mage patterns, etc. An eample s shown n Fg. 6. Images of three dfferent poses are provded as a database (Fg. 6(a. Images of other poses are provded as queres (Fg. 6(b. New vald poses (e.g., a man sttng on the char wth both arms up, a man sttng on a char wth one arm up are automatcally nferred from the database, even though they have never been seen before. New pose parts whch cannot be nferred from the three database mages are hghlghted n red as beng unfamlar (Fg. 6(c. Fgure 6(d vsually ndcates the database mage whch provded most

10 26 Boman and Iran Fgure 6. Detecton of rregular mage confguratons. New vald poses are automatcally nferred from the database (e.g., a man sttng on the char wth both arms up, a man sttng on a char wth one arm up, even though they have never been seen before. New pose parts whch cannot be nferred from the three database mages are hghlghted n red as beng unfamlar. evdence for each pel n the query mages (.e., t tells whch database mage contans the largest most probable regon of support for that pel. Note, however, that these are not the regons of support themselves. Unform patches (wth neglgble mage gradents are assumed vald by default and dscarded from the nference process (for added speedup Spatal Salency n a Sngle Image Gven a sngle mage (.e., no database, salent mage regons can be detected,.e., mage regons whch stand out as beng dfferent than the rest of the mage. Ths s acheved by measurng the lkelhood of each mage regon (the query relatve to the remanng portons of

11 Detectng Irregulartes n Images and n Vdeo 27 Fgure 7. Identfyng salent regons n a sngle mage (no database; no pror nformaton. The Jack card was detected as salent. Note that even though the damond cards are dfferent from each other, none of them s dentfed as salent. the same mage (the database used for nferrng ths partcular regon. Ths process s repeated for each mage regon. (Ths process can be performed effcently by adaptvely addng and removng the approprate descrptors from the database when proceedng from the analyss of one mage regon to the net. Such an eample s shown n Fg. 7. Ths approach can be appled to problems n automatc vsual nspecton (nspecton of computer chps, goods, etc Detectng Suspcous Behavors Gven a small database of sequences showng a few eamples of vald behavors, we can detect suspcous behavors n a new long vdeo sequence. Ths s despte the fact that we have never seen all possble combnatons of vald behavors n the past, and have no pror knowledge of what knd of suspcous behavors may occur n the scene. These are automatcally composed and nferred from space-tme regons n the database sequence. An eample s shown n Fg. 8, whch shows a few sample frames from a 2-mnute-long vdeo clp, along wth detected suspcous behavors. For full vdeos see vson/irr egulartes.html. The result of our algorthm s a dense lkelhood map. In our vdeo eamples, a sngle threshold was selected for an entre vdeo sequence query. More sophstcated thresholdng methods (hysteress, adaptve threshold, etc. can be used. Note that because our space-tme patch descrptors were based on temporal dervatves (see Secton 3.1, the detecton results are nvarant to dfferent statc backgrounds n the query and eample database sequences. (In fact, because we detect suspcous dynamc behavors, we do not process the statc regons, whch reduces run-tme consderably. An mportant property of our approach s that we can ncrementally and adaptvely update the database when new regular/vald eamples are provded, smply by appendng ther raw descrptors and locatons to the database. No relearnng process s needed. Ths s essental n the contet of detectng suspcous behavors, should a detected suspcous behavor be dentfed as a false alarm. In such cases, the database can be updated by appendng the new eample, and the process can contnue Spato-Temporal Salency n Vdeo Usng our approach we can dentfy salent behavors wthn a sngle vdeo sequence, wthout any database or pror nformaton. For eample, one person s runnng amongst a cheerng crowd. The behavor of ths person s obvously salent. In ths case, salency s measured relatve to all the other behavors observed at the same tme. The valdty of each space-tme vdeo segment (the query s measured relatve to all the other vdeo segments wthn a small wndow n tme (the database for ths partcular vdeo segment. Ths process s repeated for each vdeo segment. Such an eample s shown n Fg. 9. For full vdeos see vson/irr egulartes.html. Vdeo salency can also be measured relatve to other temporal wndows. E.g., when the salency s measured relatve to the entre vdeo, behavors whch occur only once wll stand out. Alternatvely, when the salency s measured relatve to the past (all prevous frames, new behavors whch have not prevously occurred wll be spotted. Ths gves rse to a varety of applcatons, ncludng vdeo synopss Automatc Vsual Inspecton (Qualty Assurance Our approach can be used for automatc vsual nspecton. Automatc vsual nspecton s wdely used for qualty assurance n the manufacture of goods, electronc

12 28 Boman and Iran Fgure 8. Detecton of suspcous behavors. New vald behavor combnatons are automatcally nferred from the database (e.g., two men walkng together, a dfferent person runnng, etc., even though they have never been seen before. behavors whch cannot be nferred from the database clps are hghlghted n red as beng suspcous. For full vdeos see vson/irregulartes.html Fgure 9. Detectng salent behavors n a vdeo sequence (no database and no pror nformaton. Salency s measured relatve to all the other behavors observed at the same tme. In ths eample, all the people wave ther arms, and one person behaves dfferently. For full vdeos see vson/irregulartes.html prnted boards, wafers, etc. One of the man problems n automatc nspecton s descrbng all the possble correct patterns. In some cases, an eact reference for comparson can be suppled. In those cases automatc nspecton reduces to a smple problem of pattern matchng wth change detecton. However, there are many mportant comple cases where t s meanngless or mpossble to provde a reference for comparson, (e.g., because of the combnatoral complety of the space of good cases. We address such cases usng our approach for detectng rregulartes. By supplyng a few eamples of epected/normal patterns (for goods, prnted boards, wafers, photomasks, flat panel dsplays, ceramc tles, fabrc, fruts, etc. we can try to generalze from the eamples and compose new observatons that were never seen before. Regons wth low composton lkelhood

13 Detectng Irregulartes n Images and n Vdeo 29 Fgure 10. Detecton of defects n grapefrut mages. Usng the sngle mage (a as a database of hgh qualty grapefruts, we can detect defects n dfferent grapefruts at dfferent arrangements n mages (b,(c. In both mage pars the nput mage s to the left and the output mage s to the rght. Detected defects are hghlghted n red. Fgure 11. Detecton of defects n wafer mages (No database and no pror nformaton. Wafers tend to ehbt repeatng structures. Ths can be utlzed usng our salency approach to detect defects wthout any database. In each eample, the left mage s the nput, the rght mage s the output. Detected defects are hghlghted n red. wll be consdered as defects. One such eample s shown n Fg. 10 for frut nspecton. Often, nspected products ehbt repeatng patterns (e.g., wafers, fabrc, flat panel dsplays. In these cases we can use our salency approach to detect defects wthout any pror eamples. Ths s llustrated n Fg. 11 for wafer nspecton and n Fg. 12 for fabrc nspecton. For the eamples shown we have used patch descrptors based on RGB or gray levels values accordngly. We have used a Gaussan dstrbuton for modellng descrptor smlarty. Our approach, however, s not restrcted to ths partcular choce of descrptors. 7. Concluson We address the problem of detectng rregulartes n vsual data (mages or vdeo. The term rregular depends on the contet n whch the regular or vald are defned. Yet, t s not realstc to epect eplct defnton of all possble vald confguratons for a gven contet. We pose the problem of determnng the valdty of vsual data as a process of constructng a puzzle: We try to compose a new observed mage regon or a new vdeo segment ( the query usng chunks of data etracted from prevous vsual eamples ( the database. Regons

14 30 Boman and Iran Fgure 12. Detecton of defects n fabrc mages (No database and no pror nformaton. Fabrc tend to ehbt nearly repeatng tetures and patterns wth small non-rgd deformatons. Ths can be utlzed usng our salency approach to detect defects wthout any database. Detected defects are hghlghted n red. n the observed data whch can be composed usng large contguous chunks of data from the database are consdered very lkely, whereas regons n the observed data whch cannot be composed from the database (or can be composed, but only usng small fragmented peces are regarded as unlkely/suspcous. We refer to ths process as nference by composton. It allows to generalze from just a few eamples as to what s regular and what s not n a much larger contet. The composton process s mplemented as an effcent nference algorthm n a probablstc graphcal model, whch accommodates for small spato-temporal deformatons between the query and the database. Inference by composton can also be used to detect salency n vsual data wthout any pror eamples. For ths purpose we regard each mage regon as a query, and try to compose t usng the remander parts of the mage (the database. Ths s repeated n turn for all mage regons. Salent regons wll be detected as such whch cannot be eplaned (composed usng other parts of the mage. Ths leads to a new defnton of the term salency n vsual data. In the case of vdeo data, those regons are spato-temporal, and the salent vdeo regons correspond to salent behavors. Our nference by composton approach s general and can therefore address a wde range of problems n a sngle unfed framework. Its generalty stems from the fact that t does not resort to any pre-learned class-based models. We demonstrated applcatons of ths approach to detectng suspcous behavors, salent behavors, promnent mage regons, defects n goods and products. Our current algorthm has two man lmtatons: ( Although occlusons can be handled to some etent, t cannot handle etreme occlusons (such as when only small fragmented parts of the object are vsble. ( The tme and memory complety of our current nference algorthm s lnear n the sze of the eample database. Ths s obvously problematc for very large databases. These two problems are a topc of our future research. Acknowledgments Ths work was supported n part by the Israel Scence Foundaton (Grant No. 281/06 and by the Alberto Moscona Foundaton. The research was conducted at the Moross Laboratory for Vson and Motor Control at the Wezmann Insttute of scence. References Bart, E. and Ullman, S Class-based matchng of object parts. In Vdeo Regster04, p Boman, O. and Iran, M Detectng rregulartes n mages and n vdeo. In ICCV05, pp. I: Efros, A.A. and Leung, T.K Teture synthess by non-parametrc samplng. In ICCV, pp Felzenszwalb, P. and Huttenlocher, D Pctoral structures for object recognton. IJCV, 61(1: Fergus, R., Perona, P., and Zsserman, A Object class recognton by unsupervsed scale-nvarant learnng. In CVPR03. Freeman, W., Pasztor, E., and Carmchael, O Learnng low-level vson. IJCV, 40: Honda, T. and Nayar, S Fndng anomales n an arbtrary mage, pp. II: Itt, L., Koch, C., and Nebur, E A model of salency-based vsual attenton for rapd scene analyss. PAMI. Ivanov, Y. and Bobck, A Recognton of mult-agent nteracton n vdeo survellance. In ICCV.

15 Detectng Irregulartes n Images and n Vdeo 31 Jure, F. and Trggs, B Creatng effcent codebooks for vsual recognton. In ICCV05, pp. I: Laptev, I. and Lndeberg, T Space-tme nterest ponts. In ICCV03, pp Lebe, B., Leonards, A., and Schele, B Combned object categorzaton and segmentaton wth an mplct shape model. In ECCV04 Workshop on Statstcal Learnng n CV. Lowe, D Dstnctve mage features from scale-nvarant keyponts. IJCV, 60: Shechtman, E. and Iran, M Space-tme behavor based correlaton, pp. I: Svc, J. and Zsserman, A Vdeo google: A tet retreval approach to object matchng n vdeos. In ICCV. Stauffer, C. and Grmson, E Learnng patterns of actvty usng real-tme trackng. PAMI. Weler, Y., Shechtman, E., and Iran, M Space-tme vdeo completon. In CVPR04, pp. I: Yedda, J.S., Freeman, W.T., and Wess, Y Understandng belef propagaton and ts generalzatons, pp Zhong, H., Sh, J., and Vsonta, M Detectng unusual actvty n vdeo. In CVPR04, pp. II: