Graffiti Detection Using Two Views

Transcription

1 Graffii Deecion Using wo Views Luigi Di Sefano Federico ombari Alessandro Lanza Sefano Maoccia Sefano Moni Deparmen of Elecronics Compuer Science and Sysems (DEIS), Universiy of Bologna Viale Risorgimeno 2, Bologna, Ialy Advanced Research Cener on Elecronic Sysems (ARCES), Universiy of Bologna Via offano 2/2, Bologna, Ialy Absrac his paper presens a novel video surveillance approach designed o deec vandal acs occurring on he background of he moniored scene, such as graffii paining on walls and surfaces, public and privae propery defacing or eching, unauhorized pos sicking. he aim of our approach is o deec his class of evens rapidly and robusly. We propose o use wo synchronized views o deploy synergically deph and inensiy informaion concerning he moniored scene. Our sysem can work wihin unsrucured environmens and wih geomerically unconsrained backgrounds. 1. Inroducion Nowadays vandal acs represen a serious problem in urban areas, wih housands of public and privae properies being damaged daily all around he world. Coss issued by his problem are huge: e.g. for he problem of graffii, ha relaes o he wide range of markings, echings and painings ha deface public and privae properies, an esimae $ 12 billion a year is spen for cleaning and prevenion in he Unied Saes [1]. Beside he expenses relaed o repairing, cleaning and/or subsiuing a vandalized propery, indirec coss arise due o he perceived insecuriy associaed wih he occurrence of vandal acs in a cerain area. his ypically resuls in a decrease of revenues for commercial aciviies or services aking place in he area, such as shops, house enures, and public ranspor, for which he uncleanness and perceived insecuriy lower passenger confidence in he ranspor sysem and consequenly end o decrease ridership. No less imporan are he social consequences ha repeiive vandal acs in a cerain urban area imply on he dwellers. he effor pushed o ackle - or a leas conrol - he diffusion of vandal acs in urban areas worldwide has ofen resored o he use of auomaic monioring sysems due o he huge amoun of public and properies in ciies. Commercial producs based on audio sensors [11, 3, 2, 15] ry o deec graffii by analyzing he sounds ypically occurring during hese acions. hese devices presen noable limiaions, since hey deec specific soundsandcan hardly generalize o differen or noiseless vandal acs. Moreover hey can be easily ricked by he presence of environmenal noise, and hey ypically need o sand very close o he moniored region. Due o hese reasons, vision-based approaches relying on auomaic video analysis have been recenly driving increasing aenion. On one hand, all he proposed sae-of-he-ar vision-based sysems [14, 1, 5] rely on a single-view approach, ha is hey ry o recognize vandal acs by processing a video sequence obained from a single camera. On he oher hand, wo differen classes of algorihms can be oulined. One approach consiss in applying behaviour and gesure analysis echniques for recognizing he high-level spaio-emporal paern corresponding o he person perperaing he vandal ac. For example, deecion of graffii is carried ou in [14] by searching for he paern corresponding o a person wriing on a moniored surface. However, such echniques require accurae raining of classifiers and generally perform much beer when a cerain degree of cooperaion from he subjec can be achieved, which is obviously no he case of vandal acs.

2 Anoher approach relies on comparing he curren appearance of he moniored objec wih ha of a background model of he scene. Hence, his class of mehods can deec only vandal acs which produce visible and saionary changes of he appearance of he moniored scene. We will refer o his class of evens as Saionary Visible Changes (SVC), which includes painings on walls and surfaces, public and privae propery defacing, eching or sealing, unauhorized pos sicking. his also concerns oher scenarios such as, e.g. for culural heriage environmens or museums, criminal acs such as earing, dirying, defacing, sealing of pars of an arwork. Deecion of vandal acs by recognizing SVC is carried ou in [1, 5]. In paricular, in [1] he auhors focus on graffii and propose a low-level approach for SVC recogniion based on single-view change deecion. his approach inherenly suffers from a false posiives problem when deployed as vandal acs deecors. In fac, SVC evens include mos of he common aforemenioned vandal acs, bu also oher frequen evens such as people sanding sill, parked vehicles (such as cars, moorbikes, bicycles), abandoned objecs. he same issue arises wih mehod [5], which concerns a fas singleview SVC deecor based on he analysis of higher-level evens occurring in he moniored scene. his problem is parially deal wih by limiing he deecion only on a subse of he camera field-of-view and by assuming ha he moniored scene is no crowded. Finally, robusness wih regards o sudden illuminaion changes occurring in he scene is no invesigaed. Our idea is o go beyond he visibiliy and saionariy cues in order o obain a finer classificaion of SVC. his should allow for a more effecive deecion of specific vandal acs based on he recogniion of heir peculiar effecs. o his purpose, we propose o deploy also deph informaion, so ha he class of SVC evens can be pariioned ino he wo following muually-exclusive sub-classes: a) Saionary Appearance Changes (SAC): saionary visible changes due o variaions of he appearance bu no he 3D geomery of he scene. b) Saionary Geomeric Changes (SGC): saionary visible changes due o variaions of he 3D geomery of he scene; I is clear ha mos of he vandal acs ha previous proposals ry o deec as SVC are indeed SAC, for hey deermine no (or small) variaion of he scene 3D geomery, while mos false posiives have o be ascribed o SGC. Hence, we propose o effecively deec vandal acs based on he abiliy of disinguishing beween SAC and SGC. In paricular, in his paper we presen a real-ime SAC deecion algorihm based on he use of wo synchronized views of he moniored scene and on a novel muli-view change deecion approach. his deploys on-line inensiy informaion coming from he wo image sensors ogeher wih knowledge of he 3D srucure of he moniored scene, which is obained once a iniializaion ime by means of a sereo maching process. his enables o deec effecively SAC evens even in presence of saic subjecs ha produce SGC. he proposed mehod can deec graffii-like evens wihin unsrucured environmens and does no pose any consrain on he appearance and geomery of he background of he moniored scene. Moreover, by means of a specific sage which is robus wih respec o non-linear phoomeric disorions, our approach can also handle srong sudden illuminaion changes and shadows. Finally, our sysem can aler he occurrence of vandal acs while hey are being commied. he paper is organized as follows. Secion 2 oulines some basic principles of muli-view change deecion, conexually reviewing he sae-of-he-ar in he field. he proposed vandal acs deecion algorihm is presened in Secion 3 by describing firs he novel muli-view change deecor devised for discriminaing Appearance Changes (AC) from Geomeric Changes (GC), hen he simple procedure used o evaluae saionariy of AC. Secion 4 presens some experimenal resuls obained on real scenes under challenging condiions. Finally, Secion 5 draws conclusions. 2 Principles of muli-view change deecion he inpu informaion o our approach is represened by wo synchronized video sequences of a scene characerized by a considerable overlap of field-of-views. Moreover, we assume saionariy of he capuring devices as well as of he scene background geomery, so ha geomeric regisraion of he background over he wo views, hereinafer denoed as lef view (L) and righ view (R), can be compued only once a iniializaion ime. Apar from saionariy, no furher assumpion is made abou geomery of he background surface which, in paricular, is no consrained o be planar. he goal of our approach is o compue in one of he wo views, referred o as primary, a binary mask highlighing he pixels which are sensing a SAC, ha is, a he even level, graffii. o his purpose, we use a novel muli-view change deecor o carry ou he wofold ask of robusly deecing VC and, among hese, discriminaing beween AC and GC. hen, a simple procedure is used o evaluae saionariy of AC. o beer illusrae our proposal, in his secion we ouline some basic principles concerning muli-view change deecion and, conexually, review he sae-of-he-ar in he field. As regards he way he inpu informaion (i.e. he wo synchronized video sequences) can be exploied for deecing changes wih respec o a reference scene, we define: a) emporal consisency consrain: for a given view-poin, he processed frames are images of he

3 LEF (PRIMARY) VIEW RIGH (AUXILIARY) VIEW LEF (PRIMARY) VIEW RIGH (AUXILIARY) VIEW B L B R F L, F R, F L, F L R, F R, d L R c ( BL, FL, ) c ( BR, FR, ) S c ( FL,, FL R, ) C R, S (a) emporal change deecion: only he VC super-class can be recognized (b) spaial change deecion: only he GC sub-class can be recognized Figure 1. emporal and spaial change deecion same scene aken a differen imes; b) spaial coherence consrain: for a given elaboraion ime insan, he processed frames are images of he same scene aken from differen view-poins; he emporal consisency consrain can be exploied o perform a emporal change deecion independenly in each view by a classical background subracion procedure. ha is, a each ime he curren frames F L, and F R, are compared by a suiable operaor c (, ) wih as many off-line generaed view-dependen appearance models B L and B R of he reference scene, ha we call emporal backgrounds. wo emporal change masks are hus obained, ha is wo binary masks CL, and C R, comprising he pixels which are currenly sensing a violaion of he emporal consisency consrain. emporal change deecion is illusraed in Figure 1(a) by means of a oy example consising of a planar background (ligh grey wih wo darker verical srips), a parallel-axis sereo sensor wih he wo opical axes perpendicular o he background, and AC (red) as well as GC (blue) evens being sensed in he curren frames. As one can easily undersand and as poined ou in Figure 1(a), generally speaking emporal change deecion allows for deecing, independenly in each view, he super-class of VC evens bu no for discriminaing beween he AC and GC sub-classes. his is due o he fac ha recovering deph informaion from a single view is in principle an ill-posed problem. Exploiaion of he spaial coherence consrain yields he simples muli-view change deecion approach, proposed in [7], ha we call spaial change deecion. he spaial background, unlike emporal ones, does no sore appearance bu geomeric informaion abou he moniored scene. In fac, i consiss in he dispariy map D L R (compued off-line) warping he moniored scene from he auxiliary o he primary view. Spaial background subracion is hus performed by a background dispariy verificaion. ha is, a each ime he auxiliary frame F R, is warped ino he primary view by he background dispariy map and hen compared by a suiable operaor c S (, ) wih he primary frame F L,. his allows o obain a spaial change mask, ha is a binary mask CL, S highlighing he pixels which are currenly sensing a violaion of he spaial coherence consrain. As illusraed in Figure 1(b), only he GC sub-class can be recognized by spaial change deecion. In fac, AC evens occur on he background surface and, hence, are coheren wih respec o he background dispariy map. Moreover, he mehod suffers from an inrinsic false posiives problem, called occlusion shadows. In fac, he background pixels in he primary view which are occluded by a foreground objec in he auxiliary view are inherenly deeced as changed. o deal wih his problem, in [7] he auhors propose o exploi more han one auxiliary view and o compue he inersecion of he binary masks obained by comparing he primary wih each of he auxiliary views. In [12] he problem is addressed from a sensor planning perspecive. In paricular, i is shown how occlusion shadows can be removed by using jus wo views if a suiable sensors configuraion is adoped. he combined exploiaion of boh he emporal consisency and he spaial coherence consrains is proposed in [8] and [10]. Essenially, boh approaches rely on he idea, illusraed in Figure 2(a), of firs performing emporal change deecion in each view and hen carrying ou spaial change deecion based on he obained emporal change masks. his should allow o obain he AC mask CL, A and, by subracion from he emporal change mask, he GC mask CL, G. However, as poined ou in Figure 2(a) and discussed in deail in Secion 3.2, hese mehods inherenly suffer from missed deecions in he GC mask and, dually, from false deecions in he AC mask.

4 LEF (PRIMARY) VIEW RIGH (AUXILIARY) VIEW LEF (PRIMARY) VIEW RIGH (AUXILIARY) VIEW B L B R B L F L, F R, F L, F L R, F R, d L R c ( BL, FL, ) c ( BR, FR, ) c ( BL, FL, ) S c ( FL,, FL R, ) C L R, C R, S d L R C L, CL R, A C C L G, L, \ A (a) join use of emporal and spaial change deecion as proposed in [8,10] S C L, CL, G C C L A, L, \ G (b) join use of emporal and spaial change deecion as proposed in our approach Figure 2. Join exploiaion of emporal and spaial change deecion 3 he proposed algorihm he proposed graffii deecion algorihm relies on a novel muli-view change deecion approach. he novely consiss in a simple ye clever way of combining emporal and spaial change deecion so as o perform an effecive discriminaion beween AC and GC. o beer illusrae he approach we will disinguish beween off-line and on-line elaboraion seps. Once he AC evens are deeced, he proposed procedure for he recogniion of saionary AC (SAC) can be regarded as a pos-processing sep. Hence, i will be described in a separae secion ogeher wih a simple binary morphology sage applied on he final SAC mask. 3.1 Off-line elaboraion he very firs sep concerns he calibraion of he sereo sensor, which aims a esimaing, for each view, he calibraion parameers, a se of opical disorion parameers and a recificaion homography. his informaion is condensed ino wo geomerical ransformaions g L ( ) and g R ( ) ha will be used a each processing ime - boh off-line and on-line - o compue he undisored and recified versions F L, and F R, of he capured frames FL, c and F R, c, respecively. In formulas: ( ) ( ) F L, = g L F c L, F R, = g R F c R, (1) Hence, for each view a shor boosrap sequence of N frames (N in he order of ens) is used o infer an appearance model of he reference scene, i.e. he emporal background: B L = b ( ) F L,1,...,F L,N B R = b ( ) F R,1,...,F R,N (2) wih b( ) denoing a generic, possibly robus, pixel-wise saisical esimaor. In he experimens shown in Secion 4 we have used he median operaor. he wo emporal backgrounds are hus fed o a dense sereo maching algorihm so as o compue he dispariy map warping he reference scene from he auxiliary (righ) o he primary (lef) view, i.e. he spaial background: D L R = m ( ) B L,B R (3) I is worh poining ou here ha his operaion aimed a obaining he spaial background needs o be obained once and for all a iniializaion ime, hence on-line sereo maching is no required by our mehod. herefore, wih our approach one should deploy an as accurae as possible, even hough slow, sereo maching algorihm, so as o maximize he accuracy of he warping funcion. In he experimens shown in Secion 4 we have used he algorihm described in [6]. 3.2 On-line elaboraion he main on-line processing seps performed by he proposed algorihm are illusraed in Figure 2(b) by means of he same oy example used in he previous secion.

5 Firs of all, emporal change deecion is performed in he primary view by background subracion, ha is by comparing he curren frame F L, wih he off-line generaed emporal background B L, so as o compue he emporal change mask CL, : CL, = ( ) c B L,F L, (4) In paricular, o achieve robusness wih respec o srong phoomeric disorions we apply a pixel-level he block-level approach presened in [9]. his algorihm is able o filer-ou illuminaion changes yielding locally order-preserving ransformaions of pixel inensiies. Spaial change deecion is hen performed. o his purpose, firs of all he auxiliary frame F R, is warped ino he primary view: ( ) F L R, = d L R FR, (5) wih d L R ( ) denoing he operaion of warping pixel by pixel he auxiliary frame according o he background dispariy map D L R. he spaial change mask CL, S is hen obained by comparing he primary frame F L, wih he warped auxiliary frame F L R, according o he operaor c S (, ): CL, S = ( ) cs F L,,F L R, (6) Differenly from emporal change deecion, here he compared frames are synchronized. Hence, under he assumpion of Lamberian surfaces, illuminaion changes occurring in he moniored scene affec in he same way he amoun of radiaion inciden ono he wo sensors. Neverheless, in general he wo sensors can produce differen measures (i.e. image inensiies) due o he presence of non-lamberian surfaces, o a differen foreshorening of he objecs in he wo views and differen camera parameers (e.g. gain, exposure). For his reason, also in his case a robus change deecion algorihm is desirable. We propose o use a block-based approach and he well-known Normalized Cross-Correlaion (NCC) measure, due o is simpliciy and is consan complexiy. his measure is invarian o linear phoomeric disorions. I is also worh o poin ou ha he compuaion of CL, S by means of he NCC measure can be efficienly performed using incremenal schemes [13, 4], so ha complexiy urns ou independen on block size. As discussed in he previous secion and clearly oulined in Figure 2(b), on one hand he emporal change mask comprises he super-class of pixels sensing a VC, while on he oher hand he spaial change mask conains he sub-class of GC pixels and he false posiives corresponding o occlusion shadows. Hence, by compuing he inersecion of he wo masks he geomeric change mask CL, G conaining GC pixels can be easily obained: C G L, = C L, CS L, (7) Finally, i is sraighforward o compue he appearance change mask CL, A by subracing he geomeric from he emporal change mask: C A L, = C L, \ C G L, (8) Summarizing, in principles his mask should include only hose pixels ha are currenly sensing a change of he background appearance relaed neiher o an illuminaion change nor o a variaion of he background geomery. Given he addressed applicaion domain, such changes can be ascribed o graffii. I is worh poining ou ha in he mehod of Figure 2(a) dispariy verificaion is carried ou on he binary emporal change masks. As a resul, he mehod will inherenly yield false AC in correspondence of he overlapping areas beween GC regions found in he primary view and in he warped auxiliary view. Differenly, wih our approach dispariy verificaion is performed on he original frames, as in 1(b). Hence, for he pixels belonging o he above menioned overlapping areas a decision is aken based on phoomeric similariy according o he NCC measure. Since in such areas overlapping beween differen pars of a foreground objec is likely o occur (e.g. he lef and righ shoulder of a person making graffii), unless he objec is unexured, i is likely ha phoomeric dissimilariy will allow for a correc classificaion as spaial changes. As a consequence, our mehod will unlikely yield false AC. 3.3 Saionariy and morphology Since graffiiyield permanenand saic modificaionsof scene appearance, we can exploi he furher consrain ha AC deeced by he proposed muli-view change deecor have o be saionary. o his purpose, we propose o use a simple procedure based on a pos-processing and pixel-wise approach. ha is, a each ime a pixel sensing an AC is classified as a SAC if he appearance change is persisen over a given inerval of k previousframes. In formulas: C SA (p) =C A (p) C 1 A A (p)... C k (p) (9) where C SA denoes he obained SAC binary mask and he subscrip L is drop for simpliciy. Similarly, a persisen absence of AC is required o swich off a SAC pixel in C SA. Finally, o refinehe compuedsac binarymask andremove small false posiives and false negaives we apply a simple wo-seps morphological filering consising of an area-opening and a morphological closing. he obained graffii blobs are hen labelled and heir bounding-boxes are exraced.

6 Figure 3. Resuls dealing wih he 3 Graffii sequences (o be viewed in color)

7 Figure 4. Resuls dealing wih he Saue sequence (o be viewed in color) 4 Experimenal resuls his secion presens experimenal resuls aimed a evaluaing he capabiliies of he proposed approach o deec ypical SAC evens under real condiions. In paricular, we have implemened he proposed algorihm in C code using off-he-shelf hardware which includes a PC wih an AMD Ahlon 2.21 GHz core processor and a very cheap sereo seup represened by wo web-cams. Figure 3 shows he resuls dealing wih hree video sequences (A, B and C) concerning graffii deecion. Sequences A, B refer o an oudoor environmen, while sequence C refers o an indoor scene. For all sequences, he op lef frame shows he idle appearance of he scene (i.e. he background), while remanining frames show he oupu of he sysem sampled every 10 or 5 seconds (depending on he dynamics of he even) saring from he beginning of he vandalic acion. In paricular, he oupu depics wih blue pixels hose poins currenly deeced as GC, while in red hose poins currenly deeced as AC. Finally, when a SAC even is deeced (i.e. afer pos-processing) a green bounding box wih a numbered label highlighs he area where he acion is aking place. In he Graffii A sequence he background is represened by hree exureless slaned walls a differen dephs on which a person poss up a flyer and draws some graffii, while in he Graffii B sequence he background is mainly composed by a exureless slaned wall. In boh sequences i is worh o noe ha our approach is able o accuraely deec he graffii evens a differen dephs while he acion

8 is occurring. Moreover i is worh poining ou he noable absence of false posiives hroughou he whole sequences. I is also ineresing o noe ha SGC evens are currenly discriminaed from SAC evens (e.g. in Graffii B sequence, he moorbike which is parked in fron of he background during he vandal ac). For wha concerns he Graffii C sequence, he background is represened by a whie slaned wall on which a person draws some graffii and poss up a flyer. Also in his case, graffii are correcly and on-line discriminaed from GC. Similarly o he previous sequence, false posiives are absen along all frames despie he noable presence of shadows on he background. In his case, he adoped robus emporal change deecion algorihm allows o rejec he majoriy of shadow poins as visible changes (frames 1-5, 7), he remaining ones being discarded by saionariy and morphology (frames 1-5). Fig. 4 refers o a more general case of vandal acs deecion over a complex background. In his case, referred o as Saue sequence, he background is consiued by a able and a small saue close o he sensor, plus a variegaed group of objecs a a furher disance. he background models for he wo views ogeher wih he corresponding dispariy maps are shown on he op of he figure. Similarly o he previous cases, frames 1-9 show he oupu of he sysem sampled every 5 seconds. Beside he complex background and he no perfecly synchronized sereo sensor, challenges are also inroduced by he evens aking place in he scene. ha is, differen people are moving simulaneously (frames 3, 5-7) even close o he camera (frame 5). Furhermore, a chair is placed in he scene (frames 5-8), his even being correcly no classified as SAC since i represens a SGC. SAC evens are represened by defacing of he saue (beween frames 6 and 7) and by swiching on a monior (beween frames 7 and 8). hese evens are correcly and accuraely deeced (frames 7-9). Besides, a person sanding sill (frame 8) does no produce any false posiive since, again, i correspond o a SGC. As for compuaional requiremens, our approach can efficienly process video frames a an average rae of 10 fps. 5 Conclusions We have presened an original real-ime approach for he on-line deecion of acs of vandalism, such as in paricular graffii, yielding SAC evens in video sequences. Our mehod relies on wo synchronized views and joinly explois emporal and spaial coherence concerning he moniored scene appearance by means of a novel muli-view change deecion algorihm. his allows us o discriminae effecively beween evens which only change he appearance of he scene, such as graffii, and hose which also affec is geomery. Overall, our mehod can deal wih ypically challenging aspecs such as crowded scenes, abandoned/removed objecs, saic inrusions, sudden illuminaion changes. he experimenal resuls allow us o claim ha he proposed algorihm is a robus and accurae soluion o deec ac of vandalism ha yield SAC evens in real complex scenes. I is also worh poining ou ha our approach can also work wih a cheap, common hardware represened by a sandard PC and wo web cams. References [1] D. Angiai, G. Gera, S. Piva, and C. Regazzoni. A novel mehod for graffii deecion using change deecion algorihm. In Proc. In. Conf. Advanced Video and Signal-based Surveillance (AVSS 05), pages , [2] hp:// Axium echnologies. [3] hp:// Broadband Discovery Sysems Inc. [4] F. Crow. Summed-area ables for exure mapping. Compuer Graphics, 18(3): , [5] M. Ghazal, C. Vazquez, and A. Amer. Real-ime auomaic deecion of vandalism behavior in video sequences. In Proc. IEEE In. Conf. on Sysems, Man and Cyberneics (ISIC 2007), pages , [6] H. Hirschmuller. Accurae and efficien sereo processing by semi-global maching and muual informaion. In Proc. Conf. on Compuer Vision and Paern recogniion (CVPR 2005), volume 2, pages , [7] Y. A. Ivanov, A. F. Bobick, and J. Liu. Fas lighing independen background subracion. Inernaional Journal of Compuer Vision, 37(2): , June [8] S. M. Khan and M. Shah. A muliview approach o racking people in crowded scenes using a planar homography consrain. In Proc. European Conference on Compuer Vision (ECCV 06), volume 4, pages , May [9] A. Lanza and L. D. Sefano. Deecing changes in grey level sequences by ML isoonic regression. In Proc. In. Conf. Advanced Video and Signal-based Surveillance (AVSS 06), pages 1 4, November [10] A. Lanza, L. D. Sefano, J. Berclaz, F. Fleure, and P. Fua. Robus muli-view change deecion. In Proc. Briish Machine Vision Conference (BMVC 07), Sepember [11] G. Lerg, A. Devine, D. Robers, and R. Johnson. Graffii deecion sysem and mehod of using he same. US Paen , July [12] S. N. Lim, A. Mial, L. S. Davis, and N. Paragios. Fas illuminaion-invarian background subracion using woviews: Error analysis, sensor placemen and applicaions. In Proc. IEEE In. Conf. Compuer Vision and Paern Recogniion (CVPR 05), volume 1, pages , June [13] M. Mc Donnel. Box-filering echniques. Compuer Graphics and Image Processing, 17:65 70, [14] C. Sacchi, C. Regazzoni, and G. Vernazza. A neural nework-based image processing sysem for deecion of vandal acs in unmanned railway environmens. In Proc. In. Conf. Image Analysis and Processing (ICIAP 01), pages , [15] hp:// rapec Inc.