Background subtraction by modeling pixel and neighborhood information

Transcription

1 Background subtraction by modeling pixel and information Shu-Jhen Fan Jiang 3, Kahlil Muchtar, Chih-Yang Lin, Li-Wei Kang and Chia-Hung Yeh 3 Asia University, Taichung, Taiwan andrewlin@asia.edu.tw Academia Sinica, Taipei, Taiwan lwkang@iis.sinica.edu.tw 3 National Sun Yat-sen University, Kaohsiung, Taiwan yeh@mail.ee.nsysu.edu.tw Abstract In applications of the computer vision field, a vision system is usually composed of several low level and high level components, stacked on top of each other. A better design of the lower level components usually results in better accuracy of higher level functions, such as object tracking, face recognition, and surveillance. In this paper, we focus on the low level component design, background construction, which is one of the most basic elements for a surveillance system. The proposed method eases the problems that usually occur in background construction, including aperture problem, vacillating background, and shadow removal. In conventional background construction methods, only the history information (vertical direction) of pixels is usually considered. In contrast, the proposed scheme not only uses the vertical direction but also the information (horizontal direction). Experimental results show that the proposed scheme can detect objects more delicate, alleviate the aperture problem, and identify shadow and discard it from detected objects. I. INTRODUCTION In video surveillance systems, background subtraction is strongly needed in order to detect moving objects. There are numerous papers that have proposed different approaches in order to achieve the best result. They can be classified in following types; basic background modeling [,, 3], statistical background modeling [4, 5, 6], fuzzy background modeling [7, 8, 9, 0, ], background clustering [, 3], neural network background modeling [4, 5], wavelet background modeling [6] and background estimation [7, 8, 9]. One of the promising papers, which employs a statistical approach and has inspired subsequent papers, is Stauffer and Grimson s method [5] that was proposed in 999. Unlike a single Gaussian approach [4], they use a mixture of Gaussian models for modeling backgrounds. Such an approach can cope with a multimodal background and successfully detect moving objects in a vacillating background. Nevertheless, since pixel processing was employed, shadows of moving objects will be classified as foreground and aperture problems are still unsolved. In addition moving objects will have a coarse texture in some cases when some border pixels of detected object will be regarded as background pixels or vice versa. Recently, Schindler and Wang [0] proposed a method, which considers not only the pixel itself but also its corresponding neighbors to acquire a smooth result. Furthermore, Liu, et al. [] proposed a hierarchical manner using MRF (Markov Random Field) to cast shadows from the foreground. Instead of using a likelihood function for getting information from neighbors, our proposed method uses a modest way to model certain neighbor pixels by using Gaussian mixture models to determine the probability of that pixel being in the background. The rest of paper is organized as follows. Firstly, we introduced the renowned Grimson and Stauffer s method that henceforth is called Gaussian Mixture Models (). Secondly, our proposed method by merging pixel and information will be unveiled in Section III. In Section IV, results and discussions will be depicted by providing quantitative and qualitative comparisons. In addition, conclusion will be provided in the last section. II. THE GAUSSIAN MIXTURE MODEL The Gaussian Mixture Model () was proposed by Grimson and Stauffer [5]. The authors proposed a time series method to model each background pixel into a certain number (K) of. Typically K is a small number from 3 to 5. The weight associated with each Gaussian represents the portion of the data accounted for that Gaussian. Formally, in the model, each pixel in the scene is modeled by a mixture of K Gaussian distributions. The probability that a pixel is regarded as having a value X t at a certain time is given as follows [5]: K PX ( ) ω * η( X, m, ), = () t jt, t jt, jt, j= where K is the number of Gaussian distributions, ω j,t is the weight estimation of the j th Gaussian in the mixture at time t, m j,t and j,t are the mean value and covariance matrix respec-

2 tively, of the j th Gaussian in the mixture at time t, and η is a Gaussian pdf (probability density function), defined in Eq. (). η( Xt, m, ) = e n ( π ) ( ) T Xt mt ( Xt mt ). For computational efficiency,, is defined as to represent the covariance of the k th model component. Since the probable background pixel values should stay longer and be more invariant than others, the corresponding Gaussians have the most supporting evidence and the least variance. This model was widely used in a real-time mode accompanied by an adaptive update process. When a new pixel comes in, it is checked against existing model components. The new pixel is said to match one of the weighted Gaussian distributions if its pixel value is within.5 standard deviations of the matched distribution. If any of the models are matched, the matched distribution will be updated. Otherwise (i.e., none of the K Gaussian distributions can be matched to the current pixel value), the distribution that has the minimum weight is replaced with a distribution using the current value as its mean value, an initially high variance, and a low prior weight. In the maintenance of the background model, the K distributions are sorted based upon the value ω/σ. The first B distributions are selected as the background model of a pixel for the scene and denoted as: b B = arg min ( ω > T ), (3) b k B k = where T is a predefined threshold that represents the minimum quantity of the data that must be accounted for in the background model. T is usually set to about 90% in many applications. After determining whether the incoming pixel will match one of the existing Gaussian distributions, the prior weights of K Gaussian distributions are updated as follows: ω = ( α) ω + α( M ), (4) kt, kt, kt, where α is the learning rate and M k,t is for the matched distribution and 0 for the unmatched distribution. Subsequently, weights for distributions are renormalized. If the incoming pixel matches a Gaussian distribution, the values of mean and variance of this distribution are updated as follows: () mt = ( ρ) mt + ρxt, (5) σ ( ) ( ) t = ρ σt + ρ Xt mt ( Xt mt), (6) where ρ = αη( X m, σ ). (7) t k k III. NEW BACKGROUND MODELING METHOD In this section, we describe the proposed method and shadow removal. From the previous section, conventional only considers individual pixels to determine background and foreground value. Apart from taking pixel modeling into account, our proposed method also considers certain number of neighbors that are directly adjacent to the observed pixel. As [0] depicted, the spatial distribution has valuable information to support accurate subtraction. Unlike Schindler et al. [0] and Liu, et al [] that exploit MRF (Markov Random Field) to achieve the smoothness value, we extend prominent by involving certain information. As [] explained, a pixel q at coordinates (x,y) has four direct neighbors, denoted by N 4 (q), that are horizontal and vertical neighbors with coordinates (x+, y), (x, y), (x, y+), and (x, y ). Besides horizontal and vertical neighbors, diagonal values can be considered as well. This type of neighbors is called 8- neighbors of q, denoted by N 8 (q), and is illustrated in Fig.. x, y x, y x, y+ x, y x,y x, y+ x+, y x+, y x+, y+ Fig.. Illustration of N 8 (q). By considering those neighbors information (n by n) for q and each block mean m N measurement is depicted as follows, n n (8) mn = xij, n n i = j = Once we have m N, the probability of q when comparing with a model is shown as follows, K Pq ( ) ω * η( qm,, ), = (9) j= jt, N( jt, ) jt, If q matches to one of the Gaussians, the binary result of q is determined by the dominant binary value of N 8. If N 8 has a foreground dominant value, which means the number of is more than that of 0 in N 8, the binary result of q will be regarded as foreground, or vice versa. With this approach, information can give strong evidence to determine if the observed pixel is a background or foreground pixel. However, if none of the Gaussian models match the observed pixel, the existing binary value of q from the pixel-based is maintained. The update process for the model is shown as follows, m = ( ρ) m + ( ρ) m, (0) Nt () Nt ( ) Nt T t t q mnt ( ) q mnt ( ) σ = ( ρ) σ + ρ( ) ( ), ()

3 The remainder of equation would be the same as conven- q of the image, tional. It should be noticed that for a border pixel some neighbors of q will lie outside. In such a case, we only take pixel q that has complete neighbors into account; in other words, the border pixels will use only the pixel-based to determine its binary results. The entire flowchart of our proposed method is shown in Fig.. Incoming Pixel x Smoothing using Gaussian Filter Pixel -based Process Neighborhoodbased Process If x match to Neighborhood BG Model No Using Pixel-based Result for x Yes Count the numbers of and 0 of N 8(x) from pixel-based result If count(0) > count() No Set x to Fig.. Flowchart of proposed method. Set x to 0 One of the common problems in background modeling is incorrect labeling when a shadow is classified as foreground. This problem takes place since the RGB (Red, Green, Blue) color space contains chromaticity and luminance together []. Based on this reason, separation of chromaticity and lumin- independently. ance is imperative in order to model luminance Furthermore, a shadow will have a certain portion of light and thus a certain range can be yielded by deciding a pre-defined threshold for luminance changing. Thereby, instead of using RGB color space, a new color space (r, g, I) will be applied to remove shadows. When the color and intensity have been separated from the (r, g, I) color space, the shadow will only alter intensity without changing r and g channel [0]. Yes CAVIAR public dataset [6] is used to measure qualitative results. Furthermore, a similar dataset is used to calculate quantitative results that will be represented by these following criteria [7] Precision = TP TP + FP Recall = TP P P F-measure = (/ precisi ion + / recall) It should be noticed that TP, FP, FN, TN, P and N are True Positive, False Positive, False Negative, True Negative, Total of Positive and Total of Negative value respectively. TABLE I The parameter values used in the experiments Parameters α TH B Stauffer and Grimson s method Proposed method Type/Video set Frame TP Similarity = TP + FN + FP K N β γ () (3) (4) (5) 3 3 x Frame 74 IV. EXPERIMENTAL RESULTS The performance of the proposed method is compared with Stauffer and Grimson s method [4], using several video se- from outdoor quences. The video sequences were acquired environments and a public dataset of common surveillance cases. The simulated environment for the experiments was equipped with a.93 GHz Core Duo Intel processor and GB of memory. The image resolution was set to pixels. All algorithms were implemented in C++. For the sake of labeling and segmenting the foreground pixels, the connected components algorithm [4, 5] was ap- The parameters plied to each background modeling method. used in the experiments are listed in Table, where α is the learning rate, TH B is used in (3), K denotes the number of Gaussians, N represents neighbor size, β and γ are used to define the range of shadow intensity. Fig. 3. Comparison using outdoor scene dataset.

4 Type/Video set Frame 65 Frame Type/Video set Frame Frame 60 information, using RGI color space Fig. 4. Other comparison using outdoor scene dataset. From the above outdoor case (Figs. 3 and 4), it can be seen that the proposed method can yield a more delicate foreaperture problem. ground and at the same time alleviate the By combining pixel and information, incorrect labeling can be relatively reduced. Type/Video set Frame 86 Frame & Ground Truth information and RGI color space Frame Frame 6 information and RGI color space Frame & Ground Truth Frame 344 Fig.6. Qualitative result of CAVIAR public dataset; OneShopOne- Waitfront. Two different types of CAVIAR public datasets (Figs.5 and 6) show how the proposed method can significantly discard the shadow and enhance the shape of detected objects. As mentioned in the previous section, the r, g, I color space treats color and intensity independently. Two pre-defined thresholds are used to range the common intensity of shadow. From a qualitative perspective, we can see that the proposed method can obtain results that are almost similar to the ground truth of the observed frame. TABLE II st CAVIARDATA public dataset [6] quantitative result Dataset type: One Leave Shop Reenter front Fig.5. Qualitative result of CAVIAR public dataset; OneLeaveShopReenterfront. Method/Measurement Unit Stauffer & Grimson Method Proposed Method using Frame Number: 86 Recall Precision Similarity F- (TP Rate) Measure

5 TABLE III nd CAVIARDATA public dataset [6] quantitative result Dataset type: One Leave Shop Reenter front Frame Number: 344 Method/Measurement Unit Recall (TP Rate) Precision Similarity F- Measure Stauffer & Grimson Method Proposed Method using Precision can be seen as a measure of exactness, whilst recall is a measure of completeness. The F-measure considers both the precision and the recall in computing the score, which can be interpreted as a harmonic mean of precision and recall. Tables and 3 depict the outperformed results when comparing the proposed method with the conventional. V. CONCLUSIONS A collaboration of pixel and based model has been presented. Based on qualitative and quantitative results, we can deduce that the information, even at low level vision, is imperative to enhance object shapes and to alleviate aperture problems. By using a new color space (r, g, I) of conventional, the results demonstrate that the shadow problem has been successfully relieved. ACKNOWLEDGMENT This work was supported by National Science Council, Taiwan, under Grant NSC 0--E REFERENCES [] Lee B., Hedley M. Background Estimation for Video Surveillance, IVCNZ 00, pages 35-30, 00. [] McFarlane N., Schofield C. Segmentation and tracking of piglets in images, BMVA 995, pages 87-93, 995. [3] Zheng J., Wang Y., Nihan N., Hallenbeck, E. Extracting Roadway Background Image: A mode based approach, Journal of Transportation Research Report, No 944, pages 8-88, March 006. [4] Wren C., Azarbayejani A., Darrell T., Pentland A. Pfinder: Real-Time Tracking of the Human Body, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 9, No. 7, pages , July 997. [5] Stauffer C., Grimson W. Adaptive background mixture models for real-time tracking, CVPR 999, pages 46-5, 999. [6] Elgammal A., Harwood D., Davis L. Non-parametric Model for Background Subtraction, ECCV 000, pages , Dublin, Ireland, June 000. [7] Sigari M., Mozayani N., Pourreza H. Fuzzy Running Average and Fuzzy Background Subtraction: Concepts and Application, International Journal of Computer Science and Network Security, Volume 8, No., pages 38-43, 008. [8] El Baf F., Bouwmans T., Vachon B., Type- fuzzy mixture of Gaussians model: Application to background modeling, ISVC 008, pages 77-78, Las Vegas, USA, December 008. [9] Zhang H., Xu D Fusing Color and Texture Features for Background Model, International Conference on Fuzzy Systems and Knowledge Discovery, Volume 43, No. 7, pages , September 006. [0] El Baf F., Bouwmans T., Vachon B., Fuzzy Integral for Moving Object Detection, FUZZ-IEEE 008, pages , Hong-Kong, China, June 008. [] Sivabalakrishnan M., Manjula D., Adaptive Background subtraction in Dynamic Environments Using Fuzzy Logic, International Journal on Computer Science and Engineering, Volume 0, No., pages 70-73, 00. [] Butler D., Sridharan S. Real-Time Adaptive Background Segmentation, ICASSP 003, 003. [3] Kim K., Chalidabhongse T., Harwood D., Davis L. Real-time Foreground-Background Segmentation using Codebook Model, Real-Time Imaging, 005. [4] Culbrik D., Marques O., Socek D., Kalva H., Furht B. Neural network approach to background modeling for video object segmentation, IEEE Transaction on Neural Networks, Volume 8, No. 6, pages 64 67, 007. [5] Maddalena L., Petrosino A. A self organizing approach to background subtraction for visual surveillance applications, IEEE Transactions on Image Processing, Volume 7, No. 7, pages , 008. [6] Biswas S., Sil J., Sengupta N. Background Modeling and Implementation using Discrete Wavelet Transform: a Review, JICGST-GVIP, Volume, Issue, pages 9-4, March 0. [7] Toyama K., Krumm J. Brumitt B., Meyers B. Wallflower: Principles and Practice of Background Maintenance, International Conference on Computer Vision, pages 55-6, Corfu, Greece, September 999. [8] Messelodi S., Modena C., Segata N., Zanin M. A Kalman filter based background updating algorithm robust to sharp illumination changes, ICIAP 005, Volume 367, pages 63-70, Cagliari, Italy, September 005. [9] Chang R., Ghandi T., Trivedi M., Vision modules for a multi sensory bridge monitoring approach, ITSC 004, pages , October 004. [0] Schindler K, Wang H. Smooth Foreground-Background Segmentation for Video Processing, ACCV 006, Hyderabad, India, Volume 385, pages , January 006. [] Liu Z, Huang K, Tan T. Cast Shadow Removal in Hierarchical Manner Using MRF, IEEE Transactions on Circuits and Systems for Video Technology, Volume, No., January 0. [] Digital Image Processing, Gonzalez and Woods, nd Edition, 00. [3] Ramesh Jain, Rangachar Kasturi and Brain G. Schunck, Machine Vision, McGraw-Hill Book Co., Chapter 0, 995. [4] F. Chang, C. J. Chen, and C. J. Lu, A Linear-Time Component-Labeling Algorithm Using Contour Tracing Technique, Computer Vision and Image. Understanding, vol. 93, no., pp. 06-0, 004. [5] R. Lumia, L. Shapiro, and O. Zuniga, A New Connected Components Algorithm for Virtual Memory Computers, Computer Vision, Graphics, and Image Pmcessing, vol., no., pp , 983. [6] EC Funded CAVIAR project/ist , found at URL: [7] T. Fawcett, An introduction to ROC analysis, Pattern Recognition Letters, vol.7, pp , 006.