Detecting Irregularities in Images and in Video

Transcription

1 Detectng Irregulartes n Images and n Vdeo Oren Boman Mchal Iran Dept. of Computer Scence and Appled Math The Wezmann Insttute of Scence Rehovot, Israel Abstract We address the problem of detectng rregulartes n vsual data, e.g., detectng suspcous behavors n vdeo sequences, or dentfyng salent patterns n mages. The term rregular depends on the context n whch the regular or vald are defned. Yet, t s not realstc to expect explct defnton of all possble vald confguratons for a gven context. We pose the problem of determnng the valdty of vsual data as a process of constructng a puzzle: We try to compose a new observed mage regon or a new vdeo segment ( the query ) usng chunks of data ( peces of puzzle ) extracted from prevous vsual examples ( the database ). Regons n the observed data whch can be composed usng large contguous chunks of data from the database are consdered very lkely, whereas regons n the observed data whch cannot be composed from the database (or can be composed, but only usng small fragmented peces) are regarded as unlkely/suspcous. The problem s posed as an nference process n a probablstc graphcal model. We show applcatons of ths approach to dentfyng salency n mages and vdeo, and for suspcous behavor recognton. 1 Introducton Detecton of rregular vsual patterns n mages and n vdeo sequences s useful for a varety of tasks. Detectng suspcous behavors or unusual objects s mportant for survellance and montorng. Identfyng spatal salency n mages s useful for qualty control and automatc nspecton. Behavoral salency n vdeo s useful for drawng the vewer s attenton. Prevous approaches to recognton of suspcous behavors or actvtes can broadly be classfed nto two classes of approaches: rule-based methods (e.g., [7]) and statstcal methods wthout predefned rules (e.g., [10, 12]). The statstcal methods are more appealng, snce they do not assume a predefned set of rules for all vald confguratons. Instead, they try to automatcally learn the noton of regularty from the data, and thus nfer about the suspcous. Nevertheless, the representatons employed n prevous methods have been ether very restrctve (e.g., trajectores of movng objects [10]), or else too global (e.g., a sngle small descrptor vector for an entre frame [12]). In ths paper we formulate the problem of detectng regulartes and rregulartes as the problem of composng (explanng) the new observed vsual data (an mage or a vdeo sequence, referred to below as query ) usng spatotemporal patches extracted from prevous vsual examples (the database ). Regons n the query whch can be composed usng large contguous chunks of data from the example database are consdered lkely. The larger those regons are, the greater the lkelhood s. Regons n the query whch cannot be composed from the example database (or can be composed, but only usng small fragmented peces) are regarded as unlkely/suspcous. Our approach can thus nfer and generalze from just a few examples, about the valdty of a much larger context of mage patterns and behavors, even f those partcular confguratons have never been seen before. Local descrptors are extracted from small mage or vdeo patches (composed together to large ensembles of patches), thus allowng to quckly and effcently nfer about subtle but mportant local changes n behavor (e.g., a man walkng vs. a man walkng whle pontng a gun). Moreover, our approach s capable of smultaneously dentfyng a vald behavor n one porton of the feld of vew, and a suspcous behavor n a dfferent porton the feld of vew, thus hghlghtng only the detected suspcous regons wthn the frame, and not the entre frame. Such examples are shown n Secton 6. Inference from mage patches or fragments has been prevously employed n the task of class-based object recognton (e.g. [4, 1, 3]). A small number of nformatve fragments have been learned and preselected for a small number of pre-defned classes of objects. However, class-based representatons cannot capture the overwhelmng number of possbltes of composng unknown objects or behavors n a scene, and are therefore not sutable for our underlyng task of detectng rregulartes. Our approach can also be appled for detectng salency n mages and n vdeo sequences. For example, gven a sngle mage wth no pror nformaton, we can measure the valdty of each mage regon (the query ) relatve to

2 (a) A query mage: (b) Inferrng the query from the database: (c) The database wth the correspondng regons of support: (d) An ensembles-of-patches (more flexble and effcent): Fgure 1. The basc concept Inference by Composton. A regon n the query mage s consdered lkely f t has a large enough contguous regon of support n the database. New vald mage confguratons can thus be nferred from the database, even though they have never been seen before. the remanng portons of the same mage (the database used for ths partcular query). An mage regon wll be detected as salent f t cannot be explaned by anythng smlar n other portons of the mage. Smlarly, gven a sngle vdeo sequence (wth no pror knowledge of what s a normal behavor), we can detect salent behavors as behavors whch cannot be supported by any other dynamc phenomena occurrng at the same tme n the vdeo. Prevous approaches for detectng mage salency (e.g., [6]) proposed measurng the degree of dssmlarty between an mage locaton and ts mmedate surroundng regon. Thus, for example, mage regons whch exhbt large changes n contrast are detected as salent mage regons. Ther defnton of vsual attenton s derved from the same reasonng. Nevertheless, we beleve that the noton of salency s not necessarly determned by the mmedate surroundng mage regons. For example, a sngle yellow spot on a black paper may be salent. However, f there are many yellow spots spread all over the black paper, then a sngle spot wll no longer draw our attenton, even though t stll nduces a large change n contrast relatve to ts surroundng vcnty. Our approach therefore suggests a new and more ntutve nterpretaton of the term salency, whch stems from the nner statstcs of the entre mage. Examples of detected spatal salency n mages and behavoral salency n vdeo sequences are also shown n Secton 6. Our paper therefore offers four man contrbutons: 1. We propose an approach for nferrng and generalzng from just a few examples, about the valdty of a much larger context of mage patterns and behavors, even f those partcular confguratons have never been seen before. 2. We present a new graph-based Bayesan nference algorthm whch allows to effcently detect large ensembles of patches (e.g., hundreds of patches), at multple spatotemporal scales. It smultaneously mposes constrants on the relatve geometrc arrangement of these patches n the ensemble as well as on ther descrptors. 3. We propose a new nterpretaton to the term salency and vsual attenton n mages and n vdeo sequences. 4. We present a sngle unfed framework for treatng several dfferent problems n Computer Vson, whch have been treated separately n the past. These nclude: attenton n mages, attenton n vdeo, recognton of suspcous behavors, and recognton of unusual objects. 2 Inference by Composton Gven only a few examples, we (humans) have a noton of what s regular/vald, and what s rregular/suspcous, even when we see new confguratons that we never saw before. We do not requre explct defnton of all possble vald confguratons for a gven context. The noton of regularty / valdty s learned and generalzed from just a few examples of vald patterns (of behavor n vdeo, or of appearance n mages), and all other confguratons are automatcally nferred from those. Fg. 1 llustrates the basc concept underlyng ths dea n the paper. Gven a new mage (a query Fg. 1.a), we check whether each mage regon can be explaned by a

3 Fgure 2. Detectng a matchng ensemble of patches. (a) A spatal ensemble: (for queres on mages) (b) A space-tme ensemble: (for queres on vdeo) Fgure 3. Ensembles of patches n mages and vdeo. large enough contguous regon of support n the database (see Fgs. 1.b and 1.c). Although we have never seen a man sttng wth both arms rased, we can nfer the valdty of ths pose from the three database mages of Fg. 1.c. Thus, regons n the new observed data/query (an mage or a vdeo sequence) whch can be explaned by large contguous chunks of data from the database are consdered very lkely, whereas regons n the query whch cannot be explaned by large enough database peces are consdered unlkely or suspcous. When the vsual query s an mage, then those chunks of data have only a spatal extent. When the vsual query s a vdeo sequence, then those chunks of data have both a spatal and a temporal extent. 3 Ensembles of Patches Human behavors and natural spatal structures never repeat dentcally. For example, no two people walk n the same manner. One may rase hs arms hgher than the other, or may just walk faster. We therefore want to allow for small non-rgd deformatons (n space and n tme) n our peces of puzzle (chunks of data). Ths s partcularly true for large chunks of data. To account for such local non-rgd deformatons, large chunks are broken down to an ensemble of lots of small patches at multple scales wth ther relatve geometrc postons. Ths s llustrated n Fg. 1.d. In the nference process, we search for a smlar geometrc confguraton of patches wth smlar propertes (of behavor, or of appearance), whle allowng for small local msalgnments n the relatve geometrc arrangement. Ths concept s llustrated n Fg. 2. When the vsual query s an mage, then an ensemble of patches s composed of spatal patches (see Fg. 3.a). When the vsual query s a vdeo sequence, then the ensemble of patches s composed of spato-temporal patches (see Fg. 3.b), whch allows to capture nformaton about dynamc behavors. In our current mplementaton, a sngle ensemble typcally contans hundreds of patches, smultaneously from multple scales (multple spatal scales n the case of mage patches, and multple space-tme scales n the case of spato-temporal patches). Whle the dea of composng new data from example patches was prevously proven useful for a varety of tasks (e.g., [2, 5, 11]), these methods dd not mpose any geometrc restrcton on the example patches used for constructon,.e., ther relatve postons and dstances n the database. Ths was not necessary for ther purpose. It s however crucal here, for the purpose of detectng rregulartes. Often, the only real cue of nformaton for dstngushng between a lkely and an unlkely phenomenon s the degree of fragmentaton of ts support n the database. For example, the stretched arm of a man holdng a gun s smlar to an nstantaneous stretchng of the arm whle walkng, but ts regon of support s very lmted n tme. Capturng the geometrc relatons of patches was dentfed as beng mportant for the task of class-based object recognton [1, 4, 3, 8]. Those approaches are not sutable for our objectve for two reasons: () Ther geometrc confguratons are restrcted to a relatvely small number of patches, thus cannot capture subtle dfferences whch are crucal for detecton of rregulartes. () Those confguratons were pre-learned for a small number of pre-defned classes of objects, whereas our framework s applcable to any type of vsual data. Whle the geometrc constrants of [8] are more flexble, thus allowng to recognze new ob-

4 Fgure 4. The probablstc graphcal model. Observed varables are marked n orange ; hdden database varables are marked n blue. The drectons of arrows sgnfy Bayesan dependences (See text for more detals). ject confguratons from just a few examples, ther method s stll lmted to a set of predefned object classes wth predefned object centers. Ths s not sutable for detectng rregulartes, where there s no noton of object classes. Vdeo Google [9] mposes geometrc constrants on large collectons of non class-based descrptors, and searches for them very effcently. However, those descrptors are spatal n nature and the search s restrcted to ndvdual mage frames, thus not allowng to capture behavors. In order for the nference to be performed n reasonable tmes, nformaton about the small patches and ther relatve arrangement must be effcently stored n and extracted from the database. For each small patch extracted from the examples, a descrptor vector s computed and stored (see below), along wth the absolute coordnates of the patch (spatal or spato-temporal coordnates). Thus, the relatve arrangement of all patches n the mage/vdeo database s mplctly avalable. Later, our nference algorthm takes an ensemble of patches from the vsual query and searches the database for a smlar confguraton of patches (both n the descrptors and n ther relatve geometrc arrangement). To allow for fast search and retreval, those patches are stored n a mult-scale data structure. Usng a probablstc graphcal model (Secton 4), we present an effcent nference algorthm (Secton 5) for the ensemble search problem. Patch descrptors are generated for each query patch and for each database patch. The descrptors capture local nformaton about appearance/behavor. Our current mplementaton uses very smple descrptors, whch could easly be replaced by more sophstcated descrptors (e.g., SIFT ): The Spatal Image Descrptor of a small (e.g., 7 7)spatal patch s constructed as follows: The spatal gradent magntude s computed for each pxel n the patch. These values are then stacked n a vector, whch s normalzed to a unt length. Such descrptors are densely extracted for each pont n the mage. Ths descrptor extracton process s repeated n several spatal scales of the spatal Gaussan pyramd of the mage. Thus, a 7 7 patch extracted from a coarse scale has a larger spatal support n the nput mage (.e., n the fne scale). The Spato-Temporal Vdeo Descrptor of a small (e.g., 7 7 4) spato-temporal vdeo patch s constructed from the absolute values of the temporal dervatves n all pxels of the patch. These values are stacked n a vector and normalzed to a unt length. Ths descrptor extracton process s repeated n several spatal and temporal scales of a spacetme vdeo pyramd. Thus, a patch extracted from a coarse scale has a larger spatal and larger temporal support n the nput sequence. 4 Statstcal Formulaton Gven a new vsual query (an mage or a vdeo sequence), we would lke to estmate the lkelhood of each and every pont n t. Ths s done by checkng the valdty of a large regon (e.g., regon n an mage, and regon n a vdeo sequence) surroundng every pxel. The large surroundng regon s broken nto lots (hundreds) of small patches at multple scales (spatal or spato-temporal), and s represented by a sngle ensemble of patches correspondng to that partcular mage/vdeo pont. Let q 1,q 2,.., q n denote the patches n the ensemble (see Fg. 3.a). Each patch q s assocated wth two types of attrbutes: () ts descrptor vector d, and () ts locaton n absolute coordnates l. We choose an arbtrary reference pont c (e.g., the center of the ensemble see Fg. 3.a), whch serves as the orgn of the local coordnate system (thus defnng the relatve postons of the patches wthn the ensemble). Let an observed ensemble of patches wthn the query be denoted by y. We would lke to compute the jont lkelhood P (x, y) that the observed ensemble y n the query s smlar to some hdden ensemblex n the database (smlar both n ts descrptor values of the patches, as well as n ther relatve postons). We can factor the jont lkelhood as: P (x, y) =P (y x)p (x). Our modellng of P (y x) resembles the probablstc modellng of the star graph of [3]. However, n the class-based settng of [3] what s computed s P (y; θ), whereθ s a pre-learned set of parameters of a gven patch-constellaton of an object-class. In our case, however, there s no noton of objects,.e., there s no pror parametrc modellng of the database ensemble x. Thus, θ s undefned, and P (x) must be estmated non-parametrcally drectly from the database of examples. Let d y denote the descrptor vector of the -th observed patch n y, andly denote ts locaton (n absolute coordnates). Smlarly, d x denotes the descrptor vector of the - th hdden (database) patch n x, andlx denotes ts locaton. Let c y and c x denote the orgn ponts of the observed and hdden ensembles. The smlarty between any such par of ensembles y and x s captured by the followng lkelhood: P (x, y) =P (c x,d 1 x,.., l1 x,.., c y,d 1 y,.., l1 y,..) (1)

5 (a) The database mages (3 poses): (b) Query mages: (c) Red hghlghts the detected unfamlar mage confguratons (unexpected poses): (d) Color-assocaton of the nferred query regons wth the database mages (determned by MAP assgnment): (Unform patches are assumed vald by default for added speedup). Fgure 5. Detecton of rregular mage confguratons. New vald poses are automatcally nferred from the database (e.g., a man sttng on the char wth both arms up, a man sttng on a char wth one arm up), even though they have never been seen before. New pose parts whch cannot be nferred from the three database mages are hghlghted n red as beng unfamlar. In order to make the computaton of the lkelhood n Eq. (1) tractable, we make some smplfyng statstcal assumptons. Gven a hdden database patch and ts descrptor d x, the correspondng observed descrptor d y s assumed to be ndependent of the other patch descrptors. (Ths s a standard Markovan assumpton, e.g., [5], whch s obvously not vald n case of overlappng patches.) We model the smlarty between descrptors usng a Gaussan dstrbuton: P ( d ) y d x = α1 exp ( (d y d x )T S 1 D (d y d x )) (2) where α 1 s a constant, and S D s a constant covarance matrx, whch determnes the allowable devaton n the descrptor values. Gven the relatve locaton of the hdden database patch (l x c x), the relatve locaton of the correspondng observed patch (l y c y) s assumed to be ndependent of all other patch locatons. Ths assumpton enables to compare the geometrc arrangement of two ensembles of patches wth enough flexblty to accommodate for small changes n vewng angle, scale, pose and behavor. Thus: P ( ly l x,c ) x,c y = α2 exp ( ((ly c y) (lx c x)) T S 1 L ((l y c y) (lx c x)) ) (3) where α 2 s a constant, and S L s a constant covarance matrx, whch captures the allowed devatons n the relatve patch locatons. (In ths case the dependency n relatve locatons was modelled usng a Gaussan, however the model s not restrcted to that). So far we modelled the relatons between attrbutes across ensembles (descrptors: d y,d x, and relatve locatons: ly c y,lx c x ). We stll need to model the relatons wthn the hdden ensemble, namely, the relatons between a patch descrptor d x to ts locaton l x. In the general case, ths relaton s hghly non-parametrc, and hence cannot be modelled analytcally (n contrast to classbased approaches, e.g. [4, 3]). Therefore, we model t

6 (a) The nput mage: (b) The computed salency map (- log lkelhood): (c) The detected salent regons: Fgure 6. Identfyng salent regons n a sngle mage (no database; no pror nformaton). The Jack card was detected as salent. Note that even though the damond cards are dfferent from each other, none of them s dentfed as salent. non-parametrcally usng examples from the database: { 1 (dx,l P (d x l x )= x ) DB (4) 0 otherwse where d x and l x are an arbtrary descrptor and locaton. We assume a unform pror dstrbuton for c x and c y (local orgn ponts),.e., no pror preference for the locaton of the ensemble n the database or n the query. The relaton between all the above-mentoned varables s depcted n the Bayesan network n Fg. 4. Thus, for an observed ensemble y and a hdden database ensemble x, we can factor the jont lkelhood P (x, y) of Eq. (1) usng Eqs. (2,3,4) as follows: P (c x,d 1 x,..., l1 x,..., c y,d 1 y,..., l1 y )= α P (ly l x,c x,c y )P (d y d x)p (d x l (5) x) 5 The Inference Algorthm Gven an observed ensemble, we seek a hdden database ensemble whch maxmzes ts MAP (maxmum a-posteror probablty) assgnment. Ths s done usng the above statstcal model, whch has a smple and exact Vterb algorthm. Accordng to Eq. (5) the MAP assgnment can be wrtten as: max P ( c x,d 1 x,..., X l1 x,..., c y,d 1 y,..., ) l1 y = α max P ( ) ly l x,c x,c y max P ( ( ) d y dx) P d x l l x d x x Ths expresson can be phrased as a message passng (Belef Propagaton) algorthm n the graph of Fg. 4. Frst we compute for each patch the message m dl passed from node( d x) to node lx regardng ts belef n the locaton lx: m dl l x = max P ( ( d d y x) d P d x lx). Namely, for each x observed patch, compute all the canddate database locatons lx wth hgh descrptor smlarty. Next, for each of these canddate database locatons, we pass a message about the nduced possble orgn locatons c x n the database: m lc (c x) = maxp ( l l y l x,c ) ( ) x,c y mdl l x. At ths pont, x we have a canddate lst of orgns suggested by each ndvdual patch. To compute the lkelhood of an entre ensemble assgnment, we multply the belefs from all the ndvdual patches n the ensemble: m c (c x )= m lc (c x). The progressve elmnaton process: A nave mplementaton of the above message passng algorthm s very neffcent, snce ndependent descrptor queres are performed for each patch n the observaton ensemble, regardless of answers to prevous queres performed by other patches. These patches are related by a certan geometrc arrangement. We therefore use ths knowledge for an effcent search by progressve elmnaton of the search space n the database: We compute the message m dl for a small number of patches (e.g., 1). The resultng lst of possble canddate orgns nduces a very restrcted search space for the next patch. The next patch, n turn, elmnates addtonal orgns from the already short lst of canddates, etc. In order to speed-up the progressve elmnaton, we use truncated Gaussan dstrbutons (truncated after 4σ). Thus, f n s the number of patches n the ensemble (e.g., 256), and N s the number of patches n the database (e.g., 100, 000 patches for a one-mnute vdeo database), then the search of the frst patch s O(N). We keep only the best M canddate orgns from the lst proposed by the frst patch (n our mplementaton, M =50). The second patch s now restrcted to the neghborhoods of M locatons. The thrd wll be restrcted to a much smaller number of neghborhoods. Thus, n the worst case scenaro, our complexty s O(N) +O(nM) O(N). In contrast, the complexty of the nference process n [3, 8] s O(nN), whle the complexty of the constellaton model [4] s exponental n the number of patches. The above proposed reducton n complexty s extremely mportant for enablng vdeo nference wth ensembles contanng hundreds of patches. Mult-scale search: To further speedup the elmnaton process, we choose the frst searched patches from a coarse

7 (a) The database sequence contans a short clp of a sngle person walkng and joggng: (b) Selected frames from the query sequence: (Colored frames = nput; BW frames = output; Red=Suspcous) (c) More frames from the query sequence... (Colored frames = nput; BW frames = output; Red=Suspcous) Fgure 7. Detecton of suspcous behavors. New vald behavor combnatons are automatcally nferred from the database (e.g., two men walkng together, a dfferent person runnng, etc.), even though they have never been seen before. behavors whch cannot be nferred from the database clps are hghlghted n red as beng suspcous. For full vdeos see wezmann.ac.l/ vson/irregulartes.html scale, for two reasons: () There s a much smaller number of coarse patches n the database than fne patches (thus decreasng the effectve N n the frst most ntensve step), and () coarse patches are more dscrmnatve because they capture nformaton from large regons. Ths elmnates canddate orgns very quckly. Nevertheless, there are cases n whch a vald ensemble cannot be explaned n coarse scales (e.g., due to partal occluson). In these cases (whch are not very frequent), we repeat the elmnaton process wthout the coarsest scale, startng wth a fner-scale patch as the frst patch (but penalze the overall ensemble lkelhood score). Ths s done n order to dstngush between these knd of ensembles and rregular (nvald) ensembles. 6 Applcatons The approach presented n ths paper gves rse to a varety of applcatons whch nvolve detecton of rregulartes n mages and n vdeos: 1. Detectng Unusual Image Confguratons: Gven a database of example mages, we can detect unusual thngs n a new observed mage (such as objects never seen before, new mage patterns, etc.) An example s shown n Fg. 5. Images of three dfferent poses are provded as a database (Fg. 5.a). Images of other poses are provded as queres (Fg. 5.b). New vald poses (e.g., a man sttng on the char wth both arms up, a man sttng on a char wth one arm up) are automatcally nferred from the database, even though they have never been seen before. New pose parts whch cannot be nferred from the three database mages are hghlghted n red as beng unfamlar (Fg. 5.c). Fg. 5.d vsually ndcates the database mage whch provded most evdence for each pxel n the query mages (.e., t tells whch database mage contans the largest most probable regon of support for that pxel. Note, however, that these are not the regons of support themselves). Unform patches (wth neglgble mage gradents) are assumed vald by default and dscarded from the nference process (for added speedup). 2. Spatal Salency n a Sngle Image: Gven a sngle mage (.e., no database), salent mage regons can be detected,.e., mage regons whch stand out as beng dfferent than the rest of the mage. Ths s acheved by measurng the lkelhood of each mage regon (the query ) relatve to the remanng portons of the same mage (the database used for nferrng ths partcular regon). Ths process s repeated for each mage regon. (Ths process can be performed eff-

8 A few sample frames from an nput vdeo (top row), and the correspondng detected behavoral salency (bottom row): Fgure 8. Detectng salent behavors n a vdeo sequence (no database and no pror nformaton). Salency s measured relatve to all the other behavors observed at the same tme. In ths example, all the people wave ther arms, and one person behaves dfferently. For full vdeos see vson/irregulartes.html cently by adaptvely addng and removng the approprate descrptors from the database when proceedng from the analyss of one mage regon to the next). Such an example s shown n Fg. 6. Ths approach can be appled to problems n automatc vsual nspecton (nspecton of computer chps, goods, etc.) 3. Detectng Suspcous Behavors: Gven a small database of sequences showng a few examples of vald behavors, we can detect suspcous behavors n a new long vdeo sequence. Ths s despte the fact that we have never seen all possble combnatons of vald behavors n the past, and have no pror knowledge of what knd of suspcous behavors may occur n the scene. These are automatcally composed and nferred from space-tme patches n the database sequence. An example s shown n Fg. 7, whch shows a few sample frames from a 2-mnutelong vdeo clp, along wth detected suspcous behavors. For full vdeos see vson/irregulartes.html. The result of our algorthm s a contnuous lkelhood map. In our vdeo examples, a sngle threshold was selected for an entre vdeo sequence query. More sophstcated thresholdng methods (hysteress, adaptve threshold, etc.) can be used. An mportant property of our approach s that we can ncrementally and adaptvely update the database when new regular/vald examples are provded, smply by appendng ther raw descrptors and locatons to the database. No relearnng process s needed. Ths s essental n the context of detectng suspcous behavors, should a detected suspcous behavor be dentfed as a false alarm. In such cases, the database can be updated by appendng the new example, and the process can contnue. 4. Spato-Temporal Salency n Vdeo: Usng our approach we can dentfy salent behavors wthn a sngle vdeo sequence, wthout any database or pror nformaton. For example, one person s runnng amongst a cheerng crowd. The behavor of ths person s obvously salent. In ths case, salency s measured relatve to all the other behavors observed at the same tme. The valdty of each space-tme vdeo segment (the query ) s measured relatve to all the other vdeo segments wthn a small wndow n tme (the database for ths partcular vdeo segment). Ths process s repeated for each vdeo segment. Such an example s shown n fg. 8. For full vdeos see vson/irregulartes.html. Vdeo salency can also be measured relatve to other temporal wndows. E.g., when the salency s measured relatve to the entre vdeo, behavors whch occur only once wll stand out. Alternatvely, when the salency s measured relatve to the past (all prevous frames), new behavors whch have not prevously occurred wll be spotted. Ths gves rse to a varety of applcatons, ncludng vdeo synopss. References [1] E. Bart and S. Ullman. Class-based matchng of object parts. In VdeoRegster04, page 173, [2] A. A. Efros and T. K. Leung. Texture synthess by nonparametrc samplng. In ICCV, pages , [3] P. Felzenszwalb and D. Huttenlocher. Pctoral structures for object recognton. IJCV, 61(1):55 79, [4] R. Fergus, P. Perona, and A. Zsserman. Object class recognton by unsupervsed scale-nvarant learnng. In CVPR03. [5] W. Freeman, E. Pasztor, and O. Carmchael. Learnng lowlevel vson. IJCV, 40(1):25 47, October [6] L. Itt, C. Koch, and E. Nebur. A model of salency-based vsual attenton for rapd scene analyss. PAMI, [7] Y. Ivanov and A. Bobck. Recognton of mult-agent nteracton n vdeo survellance. In ICCV, [8] B. Lebe, A. Leonards, and B. Schele. Combned object categorzaton and segmentaton wth an mplct shape model. In ECCV04 Workshop on Statstcal Learnng n CV. [9] J. Svc and A. Zsserman. Vdeo google: A text retreval approach to object matchng n vdeos. In ICCV, [10] C. Stauffer and E. Grmson. Learnng patterns of actvty usng real-tme trackng. PAMI, [11] Y. Wexler, E. Shechtman, and M. Iran. Space-tme vdeo completon. In CVPR04, pages I: , [12] H. Zhong, J. Sh, and M. Vsonta. Detectng unusual actvty n vdeo. In CVPR04, pages II: , 2004.