Combining Keypoint Clustering and Neural Background Subtraction for Real-time Moving Object Detection by PTZ Cameras

Transcription

1 Combnng Keypont Clusterng and Neural Background Subtracton for Real-tme Movng Object Detecton by PTZ Cameras Danlo Avola 1, Marco Bernard 2, Lug Cnque 2, Gan Luca Forest 1 and Crstano Massaron 2 1 Department of Mathematcs, Computer Scence, and Physcs, Udne Unversty, Va delle Scenze 208, Udne, Italy 2 Department of Computer Scence, Sapenza Unversty, Va Salara 113, Rome, Italy Keywords: Abstract: Foreground Detecton, Keypont Clusterng, Neural Background Subtracton, Movng Objects, PTZ Cameras. Detecton of movng objects s a topc of great nterest n computer vson. Ths task represents a prerequste for more complex dutes, such as classfcaton and re-dentfcaton. One of the man challenges regards the management of dynamc factors, wth partcular reference to bootstrappng and llumnaton change ssues. The recent wdespread of PTZ cameras has made these ssues even more complex n terms of performance due to ther composte movements (.e., pan, tlt, and zoom). Ths paper proposes a combned keypont clusterng and neural background subtracton method for real-tme movng object detecton n vdeo sequences acqured by PTZ cameras. Intally, the method performs a spato-temporal trackng of the sets of movng keyponts to recognze the foreground areas and to establsh the background. Subsequently, t adopts a neural background subtracton to accomplsh a foreground detecton, n these areas, able to manage bootstrappng and gradual llumnaton changes. Expermental results on two well-known publc datasets and comparsons wth dfferent key works of the current state-of-the-art demonstrate the remarkable results of the proposed method. 1 INTRODUCTION Smart systems to automatcally perform montorng tasks (e.g., person re-dentfcaton) s playng a more and more mportant role. A man duty of these tasks s the movng object detecton, snce t allows the segmentaton of the foreground movng elements, thus facltatng the reconstructon of the background of a vdeo sequence. Anyway, the movng object detecton presents a wde range of challenges, whch are well reported n (Shakh et al., 2014). These challenges manly regard the management of the followng dynamc factors of the scene: bootstrappng: constructon of the background model s a complex task, especally when the ntal frames of a vdeo contan movng objects; llumnaton changes: gradual llumnaton changes can affect the movng object detecton, especally n outdoor envronments where the natural lght changes over tme; camouflage: foreground movng elements must be segmented from the scene, even f they have chromatc features smlar to those of the background; shadows: shadows of the movng objects must be consdered n the constructon of the background. In the last decades, several solutons have been proposed to face these ssues (Bouwmans, 2014; Sobral and Vacavant, 2014) and, recently, great attenton has been gven to movng object detecton algorthms based on artfcal neural networks (ANNs) (Maddalena and Petrosno, 2008; Maddalena and Petrosno, 2014). ANNs present dfferent advantages. In partcular, ther ablty n adaptng and learnng new stuatons has played a key role n usng these algorthms nstead of the tradtonal approaches n vdeo survellance. In addton, these algorthms are provng to be very sutable for the management of those dynamc aspects that are the focus of the present paper,.e., bootstrappng and llumnaton changes. 1.1 Foreground Detecton by usng PTZ Cameras Foreground detecton by usng PTZ cameras has a rch lterature. Intally, the frame-to-frame methods were the frst presented approaches. They consst n dentfyng the overlappng regons between two consecutve frames and n analysng the pxel nformaton nsde them. In (Kang et al., 2003), an adaptve background model s generated by consecutve frames, and subsequently algned by usng of a geo- 638 Avola, D., Bernard, M., Cnque, L., Forest, G. and Massaron, C. Combnng Keypont Clusterng and Neural Background Subtracton for Real-tme Movng Object Detecton by PTZ Cameras. DOI: / In Proceedngs of the 7th Internatonal Conference on Pattern Recognton Applcatons and Methods (ICPRAM 2018), pages ISBN: Copyrght 2018 by SCITEPRESS Scence and Technology Publcatons, Lda. All rghts reserved

2 Combnng Keypont Clusterng and Neural Background Subtracton for Real-tme Movng Object Detecton by PTZ Cameras metrc transform. The work proposed n (Zhou et al., 2013), nstead, utlzed a moton segmentaton algorthm, based on the methods reported n (Brox and Malk, 2010; Ochs and T.Brox, 2011), to analyse the pont trajectores, to segment them nto clusters, and to turn these clusters nto dense regons. Another soluton s shown n (Avola et al., 2017b), where a spato-temporal trackng of sets of keyponts s used to dstngush the background from the foreground. Subsequently, dfferent attempts n mprovng the exstng methods led the developers n explorng the frame-to-global approaches. An nterestng work s presented n (Xue et al., 2011), where the authors proposed a panoramc Gaussan mxture model to cover the camera feld of vew and to regster each new frame usng a mult-layered correspondence ensemble. Probablstc methods were also used, as reported by (Kwak et al., 2011; Elqursh and Elgammal, 2012), whch proposed solutons based on Bayesan flters. Dfferent approaches are those based on machne learnng technques. In the work presented by (Ferone and Maddalena, 2014), movng object detecton s performed by an orgnal extenson of a neural-based background subtracton approach. Lastly, n (Rafque et al., 2014), an algorthm based on Restrcted Boltzman Machne (RBM) to learn and to generate the background model s reported. 1.2 Man Contrbuton Unlke the exstng works, ths paper presents an algorthm for real-tme movng object detecton (n vdeo sequences acqured by PTZ cameras) based on the combnaton of keypont clusterng and neural background subtracton. The spato-temporal trackng of the keyponts s used to estmate the camera movements and the scale varatons (Avola et al., 2017b). In partcular, ths step s utlzed to dentfy the canddate areas of foreground and to manage the bootstrappng problem. Subsequently, on these areas, an ANN mplemented by usng self-organzng feature maps (SOFMs) (Kohonen, 1982) s used to perform a neural background subtracton and to handle the gradual llumnaton changes. The man contrbuton of the paper can be summarzed as follows: 1. a robust bootstrappng management, n real-tme, by usng an orgnal keypont clusterng strategy; 2. a varaton of the neural background subtracton method proposed n (Maddalena and Petrosno, 2014) through whch also the vdeo sequences acqured by PTZ cameras (and not only by statc cameras) can be managed; 3. an adaptve use of the neural background subtracton method proposed n (Maddalena and Petrosno, 2014; Ferone and Maddalena, 2014) through whch only canddate areas of the mage (and not the whole mage) can be analysed, thus reducng both computatonal tme and nose. The rest of the paper s structured as follows. Secton 2 descrbes the proposed method. Secton 3 presents two expermental sessons. In the frst, the Hopkns 155 dataset (Tron and Vdal., 2007) s used to compare the accuracy of the proposed method wth key works of the current state-of-the-art. In the second, the Arport MotonSeg dataset (Dragon et al., 2013) s used to evaluate the performance of the proposed method durng zoom operatons. Fnally, Secton 4 concludes the paper. 2 LOGICAL ARCHITECTURE As shown n Fgure 1, the system archtecture s dvded n dfferent modules. The frst module s the Background Module Intalzaton, where a model of the background composed by both a neural map and a set of keyponts, and lnked descrptors, s created. The keyponts and descrptors are contaned n two sets called K bt and D bt, respectvely, and where t s a tme nstant. The self-organzng neural network created n ths step s organzed as a 3D matrx, denoted wth β t. In the frst teraton, the sets K b0 and D b0 are extracted from f 0,.e., the frst frame acqured by the PTZ camera. The frames after f 0 are provded as nput to the Feature Matchng module, whch fnds the correspondences between K bt 1 and K t, where K t s the set of keyponts extracted from the frame f t. The correspondences are stored n a collecton contanng all the matches, called Φ t. By usng Φ t, the changes n the scene, due to the PTZ camera movements, are estmated n the Camera Movements & Scale Changes module. Ths module also analyses the dsplacement of the keyponts nsde the scene and dentfes the set of canddate keyponts that belongs to the foreground, called K Ft. In Keypont Clusterng & Foreground Area Detecton module, the keyponts n K Ft are grouped n clusters that ndcate the areas of the foreground elements. These areas of pxel, called A t, are used to perform the foreground segmentaton n the Neural Background Subtracton module. Movng objects are represented by a set of blobs, called Mask Ft. At each teraton, all components of the background model are updated by the Background Model Updatng module. The updatng of the weght vectors n β t s performed accordng to the poston of the pxels. If a pxel belongs to the foreground, ts weght vector s not updated. Ths last phase allows to obtan a robust model. 639

3 ICPRAM th Internatonal Conference on Pattern Recognton Applcatons and Methods Fgure 1: Logcal archtecture of the proposed system. 2.1 Background Model Intalzaton Gven a tme nstant t, the background model at the prevous teraton s composed by an RGB mage, called I bt 1, the set of keyponts K bt 1, the set of descrptors D bt 1, and a self-organsng neural network, called β t 1. The I bt 1 mage represents an approxmaton, at tme nstant t 1, of the scene contanng only the background elements (when t = 0, I b0 s equal to f 0 ). Notce that, f 0 can also contan foreground elements, n fact the use of the keypont clusterng (as shown n Secton 2.4) can manage the bootstrappng problem even f the ntal scene s populated by movng elements. In the proposed soluton, β t s represented by a 3D matrx wth N rows, P columns, and n layers, whose number,.e., n = 6, was chosen accordng to emprc tests. Each layer L, wth 1 n, contans, for each pxel x I bt, L N weght vectors, called m L (x). The whole set of layers, L, composes the map β t. At tme nstant t = 0, the weghts of the vectors m (x) n β 0 are ntalzed by usng the pxel brghtness values of f 0. Also the sets K b and D b are extracted from f 0 by usng the feature extractor A-KAZE (Alcantarlla et al., 2013), whch s able to compute and to descrbe the vsual features wth faster performance wth respect to other popular feature extractors, such as: SURF (Bay et al., 2008) or ORB (Rublee et al., 2011). 2.2 Feature Matchng In ths phase, nspred by the work proposed n (Avola et al., 2017a), the features of the background model are compared wth those extracted from the current frame. A set of keyponts, K t, wth ther descrptors, D t, are extracted from f t. In ths step, the features of the background model are separated to those that do not belong to t. The K-Nearest Neghbours (KNN) approach (Bshop, 2011) was chosen to perform the match between the descrptors n K bt 1 and those n K t. Consderng that A-KAZE extracts descrptors composed of bnary values, the Hammng dstance was used. Lke the work proposed n (Avola et al., 2017b), for each k K bt 1, the two best matches between k and K t were found by usng a KNN wth K = 2. Subsequently, the rato between these two matches s computed as follows: rato = hdst(k,k 1 ) hdst(k,k 2 ) (1) where, k,k K t are the keyponts that have a match wth k. The Hammng dstances from k are expressed by hdst(k,k 1 ) and hdst(k,k 2 ), respectvely. The rato n [0, 1] expresses the proxmty of Hammng dstances between two dfferent matches. If the rato value s hgh, the two dstances are close. When the rato s over a threshold r (where r defnes the maxmum closeness between the two dstances), all matches of k are dscarded. A low value of r mples a low presence of undesred matches. Based of several emprcal tests, we fxed the value of r to All vald best matches are nserted nto the Φ t set, whle all the keyponts, n K t, wthout a vald match wth K bt 1 are nserted n a set called K d f f. Unlke the work presented n (Avola et al., 2017b), the proposed method performs a comparson between the sets K d f f and K Ft 1, n addton to the only comparson between K bt 1 and K t. Notce that, ths task can be performed only when a foreground element s dentfed n the frame f t 1. The set of these matches are called Φ d. All keyponts n K d f f wth a match n Φ dt are nserted n the K Ft set, the latter represents the collecton of the canddate foreground keyponts nsde the frame f t. Ths last step has been added to 640

4 Combnng Keypont Clusterng and Neural Background Subtracton for Real-tme Movng Object Detecton by PTZ Cameras also dentfy the foreground elements that do not perform movements n a current frame. 2.3 Camera Movement and Scale Change Estmaton The foreground keyponts, the camera movements, and the scale changes are estmated by usng the matches nsde Φ t. The frst step s to dstngush the background keyponts from the movng object keyponts, by usng a 3x3 homography matrx, called H. Ths matrx descrbes the relaton between two consecutve frames and maps the coordnates of a pont x 1 n f t 1 nto the coordnates of a pont x 2 n f t : x 2 = Hx 1 (2) In our case, H s used to map the coordnates of the keyponts nsde I bt 1 n the coordnates of the keyponts nsde f t. Ths task s performed by the RANdom SAmple Consensus (RANSAC) algorthm (Fschler and Bolles, 1981) by usng the matches contaned n Φ t. The ntal estmated homography, H, s refned by usng the Levenberg-Marquardt optmzaton (Marquardt and Donald, 1963) that mnmzes the re-projecton error. For each match (k,k ) Φ t, the followng condton must be verfed: z = { 1 f (k k ) 2 (k k h ) 2 τ 1 0 otherwse. (3) where, k h = Hk s the estmated poston of k n f t, and τ 1 s a tolerance on the dfference between the estmated dstance by homography and the estmated dstance by Φ t. If z = 0, the keypont k s a background keypont and t s nserted n K bt. When τ 1 has a low value, a large number of keyponts results statc and ftted n the background, on the other hand, n ths case a large number of false postves nsde K Ft can occur. Instead, f τ 1 has a hgh value, the estmaton of the keypont movements s less restrctve, but a large number of false negatves can occur. Based on emprcal tests, the value of τ 1 has been fxed to 2.0. On the contrary, when k s a foreground keypont, t s nserted n K Ft. When all the canddate keyponts n K bt are found, ther matches n Φ t are used to compute the affne transformaton matrx between I bt 1 and f t, thus computng the nformaton requred to estmate the movements performed by the PTZ camera. Gven three pars of matches (k a,k b ), (k c,k d ), and (k e,k f ) Φ t wth k b, k d, and k f K t, the affne transformaton matrx, A, can be calculated as follows: A = λ x 0 t x [ ] 0 λ y t y xkb x kd x k = f x 1 k a x kc x ke y y kb y kd y ka y kc y ke (4) k f where, t x and t y are the translatons on the x and y axes, respectvely. Whle, λ x and λ y are the varaton scale on the same axes. Notce that, when a zoom s performed, we obtan that λ x = λ y. Subsequently, the weghts n β t 1 must be algned accordng to the new spatal poston of ts correlated pxel. By usng t x,t y and λ, the common porton of the scene between f t 1 and f t s estmated. Ths area s expressed by a boundng box R t 1 n f t 1 and a boundng box R t n f t. If t x 0 or t y 0, the weghts of the pxels n R t 1 must be moved to the new regon R t, for each layer L β t 1. Ths algnment generates a new self-organsng map, called β t. If λ 1, a zoom operaton occurs and an nterpolaton to update the layers of β t 1 s appled. For each layer L β t 1, the weghts of the pxels nsde R t 1 are nterpolated n R t of L β t. The pan, tlt, and zoom-out operatons generate new pxels n f t, whch requre an ntalzaton of ther weght vectors n β t. For each p R t s requred an ntalzaton, then L β t, L (p) = f t(p) s obtaned. When t x = t y = 0 and λ = 1, the algnment of the weghts n β t 1 s not necessary. 2.4 Keypont Clusterng and Foreground Detecton The areas that nclude the foreground elements n f t are obtaned by usng a clusterng algorthm appled on the keyponts contaned n K Ft. The Densty-Based Spatal Clusterng of Applcatons wth Nose (DB- SCAN) algorthm (Ester et al., 1996) s chosen for the reason that t does not requre to specfy a-prory the number of clusters, moreover t s sutable to manage the presence of nose. The DBSCAN algorthm requres two parameters. The frst, called τ 2, s the radus used to search the neghbourng keyponts. The second, called MnPts, s the mnmum number of neghbourng keyponts that are requred to form a sngle cluster. Wth a low value of τ 2, small clusters are obtaned, but a certan amount of nformaton can be lost. On the contrary, wth a hgh value of τ 2, large clusters are created, but a hgh level of nose can be ncluded. The result of ths step conssts of a set of clusters, C t = (c 1,c 2,...,c m ). Each c C t s a porton of f t and s assocated to a set of pxels, called α c. The entre collecton of these areas α c, called A t, ndcates all the regons that contan foreground elements n f t. 2.5 Neural Background Subtracton In ths stage, the pxels nsde the areas n A t are analyses to fnd foreground elements. The background subtracton process works as follows: 641

5 ICPRAM th Internatonal Conference on Pattern Recognton Applcatons and Methods For each pxel g α, and α A t, the value of Mask Ft (g) (.e., the value of the pxel g nsde the mask) s set to 0. For each pxel p α, and α A t : { 1 f Ω p Mask Ft (p) = H p 0.5; 0 otherwse. (5) where, H p = p : p p h, s a 2D spatal neghbourhood of p (ncludng p) wth a radus of length h (n the proposed method h = 2). Instead, Ω p s the set of pxels n H p that satsfes the followng condton: Ω p = {p H p : d(β t(p ), f t (p )) ε} (6) where, d(β t(p ), f t (p )) s the best match dstance between the value of pxel p n f t and the weghts of pxel p nsde β t. More detals on ths dstance are reported n Secton The threshold ε ndcates the maxmum dstance that a pxel p can have wth ts best match. If the best match dstance s greater than ε, then p s defned as a foreground pxel. Notce that, hgh values of the threshold allow to successfully process background pxels wth sgnfcant changes (n terms of speed). Anyway, these values can made the method short-sghted n capturng slow changes of the foreground scene. The Eq. 5 ndcates a measure of how many pxels n the neghbourhood of p have correspondence wth the model. The pxel p can be consdered a background pxel only f ts value s less than 0.5 (.e, less than half of the pxels n ts neghbourhood does not belong to the foreground). In general, the value of 0.5 can be consdered a balanced estmaton for background and foreground elements Fnd Best Match The best match of a pxel p wth ts model n β t s computed as follows: d(β t(p), f t (p)) = mn d(m =1,...,n L (p), f t (p)) (7) where, d(, ) s a metrc based on the used color space to construct the self-organsng map. In the proposed method, the Eucldean dstance between the value of pxel and ts weght vector of the layer L usng the HSV colour model, s used. The metrc can be expressed as follows: d(l (p), f t(p)) = (v 1 s 1 cos(h 1 ),v 1 s 1 sn(h 1 ),v 1 ) (v 2 s 2 cos(h 2 ),v 2 s 2 sn(h 2 ),v 2 ) 2 2 (8) where, v 1,s 1 and h 1 are the value, the saturaton, and the hue of the pxel p, respectvely. Instead, v 2,s 2 and h 2 are the same values of the weghts n m L (p) Cast Shadow Wth the am to manage complex scenaros, the proposed method nherts, from the current lterature, a remarkable approach to make better the background subtracton stage, thus mprovng the whole algorthm. Movng objects can generate shadows, whch requre to be managed and ncluded n the background model. Based on the work reported n (Cucchara et al., 2003), the proposed method performs a detecton of the cast shadow pxels, whch are consdered as background and updated by the Eq Background Model Updatng The renforcement of the self-organsng map s a necessary step to recognze movng objects and to dstngush them from the changes of the background. Durng ths task, the updatng of a pxel weght nfluences all the weghts belongng to the same layer. So a new updated self-organsng map, called β t, for the frame f t s obtaned. Moreover, a new set of nternal layers of β t, called L, for 1 n, s computed. The update relaton s defned as follows: m L (p) = (1 ϕ(p, p ))m L (p) + ϕ(p, p ) f t (p) p N p (9) where, p α (α A t ) and N p = {p f t : p p < w 2D } s a 2D squared spatal neghbourhood of p that ncludes p. The radus of N p s expressed by w 2D (here fxed as: w 2D = 1). Actually, ϕ(p, p ) s defned as follows: ϕ(p, p ) = γg(p p )(1 Mask Ft ) (10) where, g(p p ) s a 2D Gaussan low-pass flter (Burt, 1981) and γ s the learnng rate. A large value of γ produces a fast learnng step of the selforgansng map wth respect to the changes of the scene. In ths work, after dfferent emprcal tests, γ was set to On the contrary, a small value of the parameter reduces the false negatves snce the map learns less rapdly. For each pxel outsde the areas n the A t set, only the weght of every layer L n β t s processed as follows: { ft (p) f = n; m L (p) = m L (p) otherwse. (11) +1 In ths way, the most recent frame nformaton, n the n th layer, s stored, whle the older nformaton contaned n the 0 th layer s removed. The updatng of the weghts of these pxels s fundamental to manage the bootstrappng problem. Even f, n the background ntalzaton phase, foreground elements are ncluded, ther locaton s dentfed by the clusters. 642

6 Combnng Keypont Clusterng and Neural Background Subtracton for Real-tme Movng Object Detecton by PTZ Cameras (a) (b) (c) (d) (e) (f) Fgure 2: Examples of movng object detecton and background updatng. The vdeo sequences from the column (a) up to the column (e) belong to the Hopkns 155 dataset. The last sequence belongs to the Arport MotonSeg dataset. For each column (from the top to the bottom), the frst pcture s the generc frame, the second s the keypont clusterng stage, the thrd s the movng object mask, and the last s the background updatng. From the frst up to the last column the followng vdeo are shown: tenns, people2, cars6, camel01, people1, and bus. Ths means that each pxel outsde the clusters belongs to the background. The proposed method can estmate, snce from the frst frames, the movng objects wthout an ntal estmaton of the background. The latter can be consdered a concrete overcomng of the current state-of-the-art. 3 EXPERIMENTAL RESULTS Ths secton reports the expermental results performed on the Hopkns 155 dataset (Tron and Vdal., 2007) and on the Arport MotonSeg dataset (Dragon et al., 2013). The frst was used to perform a comparson, n terms of Precson (Prec), Recall (Rec), and F1 Measure (F1), wth selected key works of the current lterature. The second was used to prove the effectveness of the proposed method durng zoomn/zoom-out operatons. Notce that, the latter task s rather unusual snce the majorty of works n ths applcaton feld do not consder movng object detecton along wth zoom operatons. Observe also that, n the proposed experments, the values of ε and γ was set to and 0.05, respectvely, thanks to the prelmnary emprcal tests performed on both datasets. Conversely, the parameters τ2 and MnPts, that depend of several factors, ncludng the mage resolu- ton and the keypont dstrbuton, and whose values are reported n Table 1, were establshed on the bass of the observatons derved by the OPTICS algorthm (Ankerst et al., 1999). Table 1: Values of the τ2 and MnPts parameters on the bass of the OPTICS algorthm. The frst fve vdeos (from the top) belong to the Hopkns 155 dataset, the last belongs to the Arport MotonSeg dataset. Vdeo Camel01 Cars6 People1 People2 Tenns Bus Resoluton 680x x x x x x1080 τ MnPts Expermental Evaluaton In Fgure 2, vsual representatons of the results obtaned by the proposed method durng the dfferent stages of the archtecture are reported. The frst row shows an example of source frame for each of the fve challengng vdeo sequences, the second row presents the related foreground keypont clusterng. The computed clusters allow the method to reduce the nose n the background subtracton stage, as depcted n the thrd row. The clusters also lmt the computatonal 643

7 ICPRAM th Internatonal Conference on Pattern Recognton Applcatons and Methods Table 2: Comparson wth key works of the current lterature on the bass of the average of the Precson, Recall, and F1- Measure metrcs. The people1, people2, cars6, camel01, and tenns vdeo sequences belong to the Hopkns 155 dataset. The last vdeo sequence, bus, belongs to the Arport MotonSeg dataset. people1 people2 cars6 camel01 tenns bus Prec Rec F1 Prec Rec F1 Prec Rec F1 Prec Rec F1 Prec Rec F1 Prec Rec F1 Proposed method (Avola et al., 2017b) N.A. N.A. N.A N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A (Kwak et al., 2011) - wth NBP N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A (Kwak et al., 2011) - wthout NBP N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A (Elqursh and Elgammal, 2012) N.A. N.A. N.A. N.A. N.A. N.A N.A. N.A. N.A (Elqursh and Elgammal, 2012) N.A. N.A. N.A. N.A. N.A. N.A N.A. N.A. N.A (Ferone and Maddalena, 2014) N.A. N.A. N.A. N.A. N.A. N.A N.A. N.A. N.A. (Brox and Malk, 2010) N.A. N.A N.A N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A (Zhou et al., 2013) N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A. N.A (Shekh et al., 2009) N.A. N.A. N.A. N.A. N.A. N.A N.A. N.A. N.A tme requred by the movng object detecton stage consderng that only some portons of the frame f t are processed (and not the whole frame). The area analysed by the method s slghtly greater than the area delmted by the clusters, ths to be sure not to mss parts of the foreground elements. Typcally, the method consders an area of the 20% bgger. We chosen ths value by emprcal evaluatons, whch shown, n the worst case, an amount of pxels outsde the clusters of about the 18%. Fnally, the last row reports the reconstructon of the background, I bt, for each vdeo. The computatonal tme requred by the constructon of the clusters does not nfluence the performance of the method snce the number of keyponts that has to be analysed s sgnfcantly lower than the number of pxels that has to be processed wthout usng the clusterng based approach. The correctness of the movng object detecton algorthm was estmated by usng three well-known metrcs: Precson (Prec), Recall (Rec), and F1 Measure (F1). The metrcs were computed on the bases of the pxels correctly assgned to the background and foreground n relaton to the gven ground-truth, more specfcally: T P Rec = (12) T P + FN Prec = T P (13) T P + FP F1 = 2(Rec)(Prec) (14) Rec + Prec where, T P, FP, FN, and T N are the number of true postve, false postve, false negatve, and true negatve, respectvely, n terms of number of pxels nsde and outsde of the related porton of the mage (.e., background or foreground). In Table 2 the results of the proposed soluton are shown. We have chosen to perform the experments on those vdeo sequences snce they, almost all, are drectly comparable wth selected key works of the current state-ofthe-art. The results show that, as regards the recall values, the proposed method acheves good performance, especally n the people1 and tenns vdeo sequences, where t reaches the best results. In people2 and car6 vdeo sequences, even f the method does not obtan the hgher values, the recall measure s very close to the best works n the lterature. Moreover, n the tenns vdeo sequence the obtaned results, as regards the recall and F1 metrcs, exceed the extermnated key works. Notce that, hgh recall values obtaned by the proposed method hghlght that t s able to capture almost all the pxels that compose the foreground objects. Ths last factor s very mportant for the applcaton of addtonal processng, such as object classfcaton and people re-dentfcaton. No results were found n other works for the camel01 vdeo, anyway we have tested t because, n our opnon, t s a very nterestng sequence. In addton, we have observed that several works do not treat zoom-n and zoom-out operatons, make the comparson a very hard task. We have adopted a very challengng vdeo sequence of the Arport MotonSeg dataset,.e., bus, to test the proposed method durng these operatons. The obtaned results have been extremely satsfyng wth all the computed metrcs. The bus vdeo sequence does not provde the ground-truth of the foreground. To solve ths gap, the lkely foreground pxels were computed by a sem-automatc segmentaton process. Summarzng, the method has shown to work properly wth dfferent challengng vdeo sequences. The obtaned results have ponted out that the proposed strategy s hghly practcal and consstent. Fnally, the use of the keyponts allows the method to be used n real-tme applcaton felds. 4 CONCLUSIONS Ths paper presents a combned keypont clusterng and neural background subtracton method for realtme movng object detecton n vdeo sequences acqured by PTZ cameras. The expermental results performed on two well-known publc datasets demonstrate the effectveness of the proposed approach compared wth selected key works of the current state-of- 644

8 Combnng Keypont Clusterng and Neural Background Subtracton for Real-tme Movng Object Detecton by PTZ Cameras the-art. The reported soluton shows dfferent contrbutons wth respect to the current lterature, ncludng the management of the bootstrappng and llumnaton change ssues, the real-tme processng, an orgnal keypont clusterng strategy, and a novel ppelne based on the neural background subtracton. REFERENCES Alcantarlla, P., Nuevo, J., and Bartol, A. (2013). Fast explct dffuson for accelerated features n nonlnear scale spaces. pages Brtsh Machne Vson Conference (BMVC). Ankerst, M., Breung, M. M., Kregel, H.-P., and Sander, J. (1999). Optcs: Orderng ponts to dentfy the clusterng structure. SIGMOD Record, 28(2): Avola, D., Bernard, M., Cnque, L., Forest, G. L., and Massaron, C. (2017a). Adaptve bootstrappng management by keypont clusterng for background ntalzaton. Pattern Recognton Letters, 100: Avola, D., Cnque, L., Forest, G. L., Massaron, C., and Pannone, D. (2017b). A keypont-based method for background modelng and foreground detecton usng a ptz camera. Pattern Recognton Letters, 96: Bay, H., Ess, A., Tuytelaars, T., and Gool, L. V. (2008). Speeded-up robust features (surf). Computer Vson and Image Understandng, 110(3): Bshop, C. M. (2011). Pattern Recognton and Machne Learnng. Informaton Scence and Statstcs. Sprnger. Bouwmans, T. (2014). Tradtonal and recent approaches n background modelng for foreground detecton: An overvew. Computer Scence Revew, 11-12: Brox, T. and Malk, J. (2010). Object segmentaton by long term analyss of pont trajectores. pages European Conference on Computer Vson (ECCV). Burt, P. J. (1981). Fast flter transform for mage processng. Computer Graphcs and Image Processng, 16(1): Cucchara, R., Grana, C., Pccard, M., and Prat, A. (2003). Detectng movng objects, ghosts and shadows n vdeo streams. IEEE Transactons on Pattern Analyss and Machne Intellgence, 25(10): Dragon, R., Ostermann, J., and Gool, L. V. (2013). Robust realtme moton-splt-and-merge for moton segmentaton. pages German Conference on Pattern Recognton (GCPR). Elqursh, A. and Elgammal, A. (2012). Onlne movng camera background subtracton. pages European Conference on Computer Vson (ECCV). Ester, M., Kregel, H.-P., Sander, J., Xu, X., Smouds, E., Han, J., and Fayyad, U. (1996). A denstybased algorthm for dscoverng clusters n large spatal databases wth nose. pages Proceedngs of the Second Internatonal Conference on Knowledge Dscovery and Data Mnng (KDD). Ferone, A. and Maddalena, L. (2014). Neural background subtracton for pan-tlt-zoom cameras. IEEE Transactons on Systems, Man, and Cybernetcs: Systems, 44(5): Fschler, M. A. and Bolles, R. C. (1981). Random sample consensus: A paradgm for model fttng wth applcatons to mage analyss and automated cartography. Communcatons of the ACM, 24(6): Kang, S., Pak, J.-K., Koschan, A., Abd, B. R., and Abd, M. A. (2003). Real-tme vdeo trackng usng ptz cameras. volume 5132, pages Internatonal Conference on Qualty Control by Artfcal Vson. Kohonen, T. (1982). Self-organzed formaton of topologcally correct feature maps. Bologcal Cybernetcs, 43(1): Kwak, S., Lm, T., Nam, W., Han, B., and Han, J. H. (2011). Generalzed background subtracton based on hybrd nference by belef propagaton and bayesan flterng. pages IEEE Internatonal Conference on Computer Vson (ICCV). Maddalena, L. and Petrosno, A. (2008). A self-organzng approach to background subtracton for vsual survellance applcatons. IEEE Transactons on Image Processng, 17(7): Maddalena, L. and Petrosno, A. (2014). The 3dsobs+ algorthm for movng object detecton. Computer Vson and Image Understandng, 122: Marquardt and Donald (1963). An algorthm for leastsquares estmaton of nonlnear parameters. SIAM Journal on Appled Mathematcs, 11(2): Ochs, P. and T.Brox (2011). Object segmentaton n vdeo: a herarchcal varatonal approach for turnng pont trajectores nto dense regons. pages IEEE Internatonal Conference on Computer Vson (ICCV). Rafque, A., Sher, A. M., and Jeon, M. (2014). Background scene modelng for ptz cameras usng rbm. pages Internatonal Conference on Control, Automaton and Informaton Scences (ICCAIS). Rublee, E., Rabaud, V., Konolge, K., and Bradsk, G. (2011). Orb: An effcent alternatve to sft or surf. pages IEEE Internatonal Conference on Computer Vson (ICCV). Shakh, S. H., Saeed, K., and Chak, N. (2014). Movng Object Detecton Usng Background Subtracton. SprngerBrefs n Computer Scence. Sprnger Internatonal Publshng. Shekh, Y., Javed, O., and Kanade, T. (2009). Background subtracton for freely movng cameras. pages IEEE Internatonal Conference on Computer Vson (ICCV). Sobral, A. and Vacavant, A. (2014). A comprehensve revew of background subtracton algorthms evaluated wth synthetc and real vdeos. Computer Vson and Image Understandng, 122:4 21. Tron, R. and Vdal., R. (2007). A benchmark for the comparson of 3-d moton segmentaton algorthms. pages 1 8. IEEE Internatonal Conference on Computer Vson and Pattern Recognton (CVPR). Xue, K., Ogunmakn, G., Lu, Y., Vela, P. A., and Wang, Y. (2011). Ptz camera-based adaptve panoramc and mult-layered background model. pages IEEE Internatonal Conference on Image Processng (ICIP). Zhou, X., Yang, C., and Yu, W. (2013). Movng object detecton by detectng contguous outlers n the lowrank representaton. IEEE Transactons on Pattern Analyss and Machne Intellgence, 35(3):