An Intelligent Surveillance Model Using Three Dimensional Quasi Connected Components in Multi-resolution Wavelet Space

Transcription

1 An Intelligent Surveillance Model Using Three Dimensional Quasi Connected Components in Multi-resolution Wavelet Space CS 6910 Project Report/ Research Paper By Ankur Chattopadhyay Abstract The goals of most background subtraction techniques for detection in surveillance systems have been to attain high performance in extracting individual targets. However, in the process of isolating out just single targets, one particular area, which has been less worked upon, is the detection of group targets, in an effort to solve the problem of target fragmentation. In this paper, we concentrate on the issue of target defragmentation through the task of grouping individual targets into one single target component. This paper presents the example of a multi-resolution codebook model, which handles the challenge of target detection for both individual targets as well as group targets. The proposed multi-scale codebook framework, in this paper, employs a novel method of target segmentation at multiple resolution levels using wavelets. The proposed model also introduces the unique concept of applying three dimensional quasi-connected components, within the multi-resolution wavelet space, to achieve the objective of detection at an individual level, as well as at a group level. In the process, this paper demonstrates that, besides just being used for foreground cleanup, the idea of connected components can be extended to determine relationship beyond local spatial attributes. Apart from local spatial attributes, our proposed model includes external spatial & frequency features for connectivity computation in a wavelet domain. In the final analysis, we show that neighborhood, when it comes to images, can be designed as a function of both space and frequency to enhance background subtraction algorithms. 1. Introduction Before going into the discussion of our multiresolution codebook model, we first touch upon the idea of background subtraction in this section. Then, we focus our attention on the use of the codebook model as one of the better techniques of background subtraction, justifying the reason why we choose the codebook model in our proposed scheme Background Subtraction The ability to extract moving targets from a video sequence, captured using a static camera, is a typical first step in visual surveillance. That s where the process of background subtraction figures in the approach being to discriminate moving targets from background to detect them. So, the overall idea is to subtract or difference the current image from a reference background model, so that the subtraction identifies the non-stationary or new targets. There has been considerable research work done in this area based on different techniques, starting from simple methods using a single reference model to complex methods handling multiple backgrounds. The basic techniques, in references [12, 13], use single-mode models to deal with simple backgrounds, while the general mixture of Gaussians (MOG) model, as cited in reference [14], is used to handle complex, non-static backgrounds. However, most of the MOG-based methods face problems while coping with backgrounds having fast variations, as cited in reference [16]. Also, the MOG model encounters issues with adjustment of its learning rate when adapting to the fast background changes, as mentioned in reference [16]. These drawbacks of the MOG model were overcome by non-parametric methods, as in references [10, 15]. But most non-parametric methods fail, due to memory limitations, when long time periods are required to sample the background. Here is where the real-time foregroundbackground segmentation algorithm, using the codebook model, in reference [11], figures in. The codebook (CB) model employs a highly compressed background model, which solves the above problem of memory constraints. Two salient features responsible for the improved performance of this algorithm are layered modeling/detection and adaptive codebook updating. 1

2 1.2. Codebook Model The codebook (CB) model, as cited in reference [11], is a superior background subtraction algorithm. Sample background values at each pixel are quantized into codebooks which represent a compressed form of background model for a long image sequence. This enables capturing of structural background variations, due to periodic-like motion, over a long period of time under limited memory. Mixed backgrounds can be modeled by multiple codewords, without making parametric assumptions. During the initial training period, unconstrained training considers the moving foreground targets in the scene. The CB algorithm adopts a quantization technique to build a background model from lengthy observation sequences. For each pixel, it constructs a codebook consisting of one or more codewords. Samples at each pixel are clustered into the set of codewords based on a color distortion metric together with brightness bounds. The background is encoded on a pixel-by-pixel basis. Detection is done by testing the difference of the current image from the background model with respect to color and brightness differences. An incoming pixel is categorized as background if it meets the following two conditions - (1) the color distortion to some mapped codeword, for that pixel, is less than the detection threshold, and (2) its brightness lies within the brightness range of that codeword. Otherwise, it is classified as foreground. The codebook representation is efficient in memory and speed compared with other background modeling techniques. It can handle scenes containing both static and moving backgrounds, as well as illumination variations. All the above characteristics explain why we select the codebook method to implement our proposed model. 2. Motivation In this section, we explain the motivation behind the problem we are addressing through this paper. Most of the background subtraction methodologies, described in the previous section, are based on the extraction of only single targets. Individual targets are important, no doubt, and it cannot be denied that detection has a crucial part to play, as targets cannot be tracked without being detected. But often the other side is ignored that is, group target identification. Grouping of individual targets into one has not been taken up in most background subtraction techniques. The importance of target defragmentation cannot be underestimated. In fact, we believe that the capability to detect group targets, in addition to the usual individual targets, will make the future visual surveillance systems more powerful and versatile. For instance, if we take the example of an image comprising of a flock of birds, then with our suggested model we could detect the entire flock as one single group target, as well as pick out each bird separately as an individual target. 3. Proposed Multi-resolution CB Model In this section, we discuss our multi-resolution CB model that can detect both individual and group targets. Our proposed scheme is based on the idea of transforming the surveillance image space into a wavelet domain. We then construct separate codebooks at multiple resolutions, across different scales of the wavelet space, to carry out detection at those multiple layers. Once the backgroundforeground segmentation is done at those different resolution scales, using the respective codebooks, a fresh technique of three dimensional quasi-connected components is applied on the multi-resolution segmented images in order to obtain the desired level of target detection. The overall algorithmic steps of our system are depicted below in Figure 1. We next describe those basic steps. As seen in Figure 1, our model is designed for a visual surveillance system, which captures images on a continuous monitoring basis. The initial step is to use a two dimensional (2D) Discrete Wavelet Transform (DWT) to decompose the original image into multi-resolution parts, where each execution of DWT corresponds to a separate resolution level. So, DWT level N indicates N executions of the transform. Every time the DWT is applied, the image resolution is reduced by ¼, by the multi-scale decomposition, as shown in Figure 2. This iterative process of splitting the image frequency spectrum, through low pass (LP) and high pass (HP) filters, is a special property of DWT, as given in the references [18, 19, 23, 24]. We use this wavelet characteristic for splitting the original image frame into multi-resolution components. So, after decomposition up to DWT level N, we have N different resolutions, represented by M/4, M/4^2, M/4^3, M/4^ (N 1), M/4^ N, respectively, where M is the resolution of the original image frame. We next highlight our unique concept of employing wavelet-based segmentation, using sub-band codebooks, across multiple resolution levels to perform multi-scale target detection Multi-resolution Detection Using Wavelets As displayed in Figure 2, at the end of each application of DWT, we are left with four separate frequency bands 2

3 New image frame arrives Apply 2D DWT on image to break it down iteratively into N levels of multiresolution wavelet components Build multiple adaptive codebooks with the multi-resolution components of the image frame this includes four different codebooks, corresponding to the low frequency (LL, LH) & high frequency (HL, HH) components within each resolution level Based on each constructed codebook for every image component across the different resolution levels, do real-time background-foreground segmentation to extract targets; this includes separate target detection in each of the four image components (LL, LH, HL, HH) within every resolution level an iterative process; so each of the 4 sub-band image components are, in turn, recursively decomposed into four lower level sub-band images to create further lower resolution scales. We utilize this notion of sub-bands, as cited in references [21, 22], to create four sub-band codebooks within each resolution level, using the CB model. Then, in the next step, we perform real-time, adaptive background-foreground segmentation for each sub-band codebook at every resolution. Thus, at the end of this stage, we get 4 segmented images, corresponding to the 4 separate frequency bands at each resolution level within the wavelet space. The advantage of having the CB model as the supporting base for our system is that the segmentation performance delivered at each resolution scale is of proven quality, as explained in section 1.2. The underlying CB model leads to robust and sensitive detection for different types of changing environments. Apply 3D QCC, with hysteresis thresholding, to every sub-band image & represent dimensions in the 3D space by the other 3 sub-band images, thereby computing connectivity with local attributes & external spatial attributes from other frequency bands Combine the resultant image components, using logical operators, within each resolution level, and add up all the images across the multiple resolution layers to get the overall output image with desired level of target detection; the resultant low resolution (N>1) images contribute to group detection, while the resultant high resolution (N=0,1) images contribute to individual detection Move on to next frame Figure 1. Basic steps of our proposed model LL, LH, HL & HH, which indicate the low frequency and high frequency components at that resolution level. This is Figure 2. Sub-bands within image after application of DWT 3.2. Three Dimensional QCC Using Wavelets In this section, we discuss our novel notion of using three-dimensional (3D) Quasi-Connected Components (QCC) to achieve close association of targets, leading to our objective of group target detection. This algorithm of grouping individual targets into one is essentially both a gap-filling and region labeling method, which uses hysteresis thresholding in 3D wavelet space. Typically, in background-subtraction techniques, the last step, after the basic detection phase, is to discard the noisy areas in the segmented image, by employing general cleaning methods. Some popular methodologies used for this processing are the morphological operators, as cited in references [3, 25]. However, one of the better techniques, as given in references [3, 4, 5, 6, 7, 25], is to classify the distinct regions of interest in the image through the identification of connected components. The basic idea of the above methodology is to separate out areas within the image, lying above a given threshold value, thereby filtering out the small, insignificant noise regions. As mentioned, the fundamental connected components algorithm uses only one threshold to categorize regions. A 3

4 more efficient way is applying a dual-threshold system, by using an upper bound based on a higher threshold value ( ), and a lower bound based on a lower threshold value ( ). This is known as hysteresis thresholding, as cited in references [1, 2, 8]. The application of two thresholds, instead of one, is more effective as the added high threshold sheds off components, which don t contain enough information details or may have been noise, but was not filtered out by the earlier single threshold (low threshold value). Our proposed 3D QCC method uses the above technique of hysteresis thresholding as part of its algorithm. The real challenge, in successfully grouping single targets into one, is the establishment of a stronger degree of association amongst the connected components. For connecting individual targets with each other, we draw inspiration from the quasi-connected-components (QCC) technique, as cited in references [1, 2, 3]. The basic 2D QCC algorithm, as shown in Figure 3, combines hysteresis thresholding with region merging and area thresholding (filtering by minimum area). It typically uses a union-find data structure. We take a leaf out of the above concept of QCC, specially the role played by resolution reduction, as depicted in Figure 3. Our system implements the idea of gap filling by breaking down the original image to multiple lower resolution levels using wavelet transform. As explained earlier, with every level of DWT applied, we reduce the image by 4 times the parent image resolution, and therefore get rid of the holes in the parent image more efficiently. To further improvise, we create more powerful connections between the connected image regions by applying an innovative 3D QCC technique at each resolution level after the multi-scale detection as in section 3.1. Adding a third dimension to the basic QCC method enables us to consider more distant neighbors than what is covered by the basic QCC technique. So, our gap-filling method becomes more spatially varying since the connectivity analysis takes more options for distance between the neighbors 4 or 8 pixels for 2D connectivity, as well as 6, 10, 18 or 26 pixels for 3D connectivity. Our method constructs equivalence classes, using connected component labels. For further innovation, we apply the idea of 3D connected components, coupled with hysteresis thresholding, as in reference [8], and build a 3D wavelet space using the frequency sub-bands at each resolution level. For instance, when we apply 3D QCC to the LL image component, we do not restrict ourselves to just the local spatial attributes, within that image frequency band, but also take into account the external spatial attributes across the three other sub-band images (LH, HL & HH), that represent three different frequency bands. We refer to the three sub-band images as the three dimensions & refer to their respective spatial attributes, for processing connectivity. We repeat this process for each of the LH, HL and HH image components. This novel algorithm helps in building stronger connections between similar image regions by jumping over the gaps in the parent image pixels, and merging regions more effectively. Our 3D QCC technique is not only unique in the aspect of 3D connectivity analysis, but also for the fact that it goes beyond the constraints of local spatial attributes, to include external spatial characteristics, across three different frequencies. Figure 3. Idea of QCC, as taken from references [1, 2, 3] 3.3. Combining Multi-resolution Images The final step in our model is to combine the segmented and 3D QCC processed images across all the different resolution levels using logical operations. In this context, we refer to the multiscale-decomposition-based image fusion schemes, as cited in reference [9]. Our idea is inspired by the above mentioned paper, which talks about a different scenario. However, our originality lies in our area of application and the method of implementation. For our own implementation, we use the OR logical operator to sum up the sub-band image components within each resolution scale, and then add the multi-resolution images together to form the overall image. This process does involve conversion of the lower resolution images to higher resolution (parent image resolution), which takes us back to the parent image space. With this last step, we complete our entire process of image decomposition and 4

5 CS 6910 Research Paper Ankur Chattopadhyay merging within the wavelet space. A careful study will reveal that we have effectively brought both space & frequency into the whole picture, as factors for our connected components computation. As explained in the penultimate step of Figure 1, the lower resolution image components (at level N >1) leads to better connections between the related regions of the image. On the other hand, the higher resolution components (for DWT level N=0,1) enable single target detection. This adds to the capacity of our model, making it possible to do individual target detection along with group target detection, all within one single system. (a) and (b) of Figure 4 take into consideration one of the surveillance images, taken from the above cited video, and its corresponding ground truth image. As evident from (b), there are 4 actual targets in the given frame. Now what we see from part (c) is that for DWT level N=0, our system performs individual target detection successfully by segmenting each of the 4 targets separately. In case of part (d), there are only 3 targets extracted because of the merging of 2 targets. For part (e), only 2 components are generated, instead of the original four, as 3 targets have been grouped together. Finally, in part (f), we find that the 4 individual targets have been successfully grouped into one single component. From these results, we can see that our model is capable of separating out individual targets at the parent image resolution, and as we go lower down from the parent resolution, the model fine tunes itself for detecting the original individual targets as one group target. It is quite clear from the output images that more multi-scale wavelet decomposition leads to stronger association of targets in our system. Our initial evaluation shows an overall individual target detection rate, at N=0, of about 90% (ranging from 100% down to 80% approximately). The average successful target defragmentation rate, for group detection (at N>1) was found to be 75% (ranging from 100% down to 50% approximately), depending on the number and complexity of individual targets, and also on the level of wavelet decomposition performed. For example, at N>=2, the above success rate of group target detection went up to 80% (ranging from 100% down to 60% approximately), and it shot up to 85% (ranging from 100% down to 70% approximately) at N>=3. 4. Results (a) Captured image frame (b) Ground truth for evaluation (c) Single target detection (N=0) (d) Target association (N=1) 5. Conclusion and Future Scope of Work Our main contribution lies in the implementation and presentation of an innovative vision system that is flexible and capable of dual mode detection for both individual targets as well as group targets according to the demands of a situation. In addition, we exhibit our new idea of performing real-time background-foreground segmentation at multiple resolution scales, by multiple codebook construction using wavelets. We also introduce our unique concept of 3D QCC components within the multi-resolution wavelet space. The novelty of this method is based on the fact that we group the associated image regions together more meaningfully, based on local spatial attributes, as well as external spatial attributes across three different frequency sub-bands. In this process, we open up the opportunity of future investigation of neighborhood relationships in images, based on both space & frequency. (e) Grouping of targets (N= 2) (f) Group target detection (N=3) Figure 4. Results from our implemented system In this section, we discuss the results from our proposed system. The model implementation work has been done in OpenCV, using the CB model in reference [25]. We use Haar and Daubechies wavelets, as described in references [17, 19, 20], for building our DWT. Some of our results are shown in Figure 4. As displayed, one of the datasets we used to test our system is the Meet_Crowd video clip, and its ground truth data, taken from reference [26]. Parts 5

6 Overall, we tackle the issue of target defragmentation, by making it possible to group individual targets into one. We are able to demonstrate that the idea of connected components, when used with wavelets, can serve a whole new purpose in target detection, apart from just being used in foreground cleanup. An application of our proposed system is to identify multiple targets in an image (like the example of the flock of birds given earlier) together as a group, as well as separately in the form of single, individual targets. However, one limitation of our proposed model might be the added time consumption and expenses due to use of wavelet transform. We have used the very simple and inexpensive Haar wavelets in one version of our system implementation to deal with the probable issues of cost and complexity. Still one may argue that the load of DWT application for the image breakdown adds more effort and time to the real-time process of target detection. But that is one trade-off or compromise one has to make for the added performance offered by our system. Of course, exploring ways to improve upon the 3D QCC computation and speed up the entire process is part of our ongoing research work. Other future scopes of work include trying out our system design on other well-known, popular background subtraction techniques apart from the CB model, and maybe extend the idea to other algorithms of vision other than background modeling. References: [1] T. E. Boult, X. Gao, R. Micheals, M.Eckmann, Omnidirectional visual surveillance - Image & Vision Computing, Volume 22, Issue 7; July, 2004; Pages [2] T.E. Boult, R. Micheals, X. Gao, P. Lewis, C. Power, W. Yin, A. Erkan, Frame-Rate Omnidirectional Surveillance & Tracking of Camouflaged and Occluded Targets - pp.48, Second IEEE Workshop on Visual Surveillance, [3] T.E. Boult, R.J. Micheals, X. Gao, M. Eckmann, Into the woods: visual surveillance of non-cooperative and camouflaged targets in complex outdoor settings - IEEE Proceedings; October, [4] R. Strandh, Jacques-Olivier Lapeyre, An efficient union-find algorithm for extracting the connected components of a largesized image LaBRI Research Report; January, [5] Prasad Gabbur, Hong Hua, Kobus Barnard, A fast connected components labeling algorithm and its application to real-time pupil detection - Machine Vision & Applications Journal, Springer; January, [6] U. L. Jau, C. S. Teh, Real-Time Object-Based Video Segmentation Using Color Segmentation and Connected Component Labeling - LNCS Journal, Volume 5857, Springer; November [7] Ali Rahimi, Fast Connected Components on Images people.csail.mit.edu/rahimi/connected/. [8] G. Malandain, Connected Components Extraction www-sop.inria.fr/epidaure/personnel/ Gregoire.Malandain/segment/connexe.html. [9] Z. Zhang, R. S. Blum, A Categorization of Multiscale- Decomposition-Based Image Fusion Schemes with a Performance Study for a Digital Camera Application, Proceedings of IEEE; Vol. 87, No. 8, pp ; [10] Mittal A, Paragios N, Motion-based background subtraction using adaptive kernel density estimation - IEEE CVPR Conference, [11] Kim K, Chalidabhongse TH, Harwood D, Davis LS, Realtime foreground-background segmentation using codebook model - Real Time Imaging, 2005; 11(3): Pages [12] Wren CR, Azarbayejani A, Darrell T, Pentland A, Pfinder: Realtime Tracking of the Human Body - IEEE Transactions on Pattern Analysis & Machine Intelligence, 1997;19(7). [13] Horprasert T, Harwood D, Davis LS, A statistical approach for real-time robust background subtraction and shadow detection - IEEE Frame-Rate Applications Workshop, Kerkyra, Greece; [14] Stauffer C, Grimson Wel, Adaptive background mixture models for real-time tracking - IEEE International Conference on Computer Vision and Pattern Recognition 1999; 2: [15] Elgammal A, Harwood D, Davis LS, on-parametric model for background subtraction - European Conference on Computer Vision 2000; 2: [16] Toyama K, Krumm J, Brumitt B, Meyers B Wallflower: principles and practice of background maintenance International Conference on Computer Vision 1999; [17] G. Davis, A. Nosratinia, Wavelet-based image coding: An overview - Applied & Computational Control, Signals and Circuits, 1(1), [18] Amara Graps, An Introduction to Wavelets - IEEE Computational Sciences & Engineering; pp 50-61, [19] I. Daubechies, Ten Lectures on Wavelets - Philadelphia, PA: SIAM, [20] M. Antonini, M. Barlaud, P. Mathieu, I. Daubechies, Image coding using wavelet transforms - IEEE Transactions on Image Processing, [21] S. Mallat, A theory for multiresolution signal decomposition: the wavelet representation - IEEE Transactions Pattern Analysis & Machine Intelligence; Vol. 11, pp ; July [22] M. Vetterli, J. Kovacevic, Wavelets and Subband Coding Englewood Cliffs, NJ: Prentice Hall, [23] C. S. Burrus, R. A. Gopinath, H. Guo, Introduction to Wavelets and Wavelet Transforms: A Primer - Englewood Cliffs, NJ: Prentice-Hall, [24] C. Valens, A Really Friendly Guide to Wavelets clemens/wavelets/wavelets.html. [25] G. Bradski, A. Kaehler, Learning OpenCV: Computer Vision with the OpenCV Library - O'Reilly Publications. [26] PETS 2004 Dataset from EC Funded CAVIAR Project, I RIA Labs, Grenoble, France/IST TA1/. 6