ARTICLE IN PRESS. Signal Processing

Transcription

1 Signal Processing 89 (2009) Contents lists available at ScienceDirect Signal Processing journal homepage: Fast communication The analysis of the color similarity problem in moving object detection Xiang Zhang, Jie Yang University of Shanghai Jiaotong, Dochuan Road 800, Shanghai, China article info Article history: Received 3 March 2008 Received in revised form 27 September 2008 Accepted 27 October 2008 Available online 5 November 2008 Keywords: Moving object detection Foreground segmentation Color similarity Confusion point Model accuracy abstract The color similarity between the background and foreground causes serious misdetections in moving object detection from video sequences. In this paper, we point out that the existence of a confusion point and the model inaccuracy are the reasons for the misdetections due to the color similarity. Accordingly, the solutions of the color similarity are to shift the confusion point and to improve the model accuracy. Based on this conclusion, a simple algorithm by combining a weighting technique and a new foreground model is presented, and improved results are generated. More accurate weighting techniques and foreground models are expected to be developed in the future based on the solutions. & 2008 Elsevier B.V. All rights reserved. 1. Introduction Serious misdetections are caused in moving object detection when the foreground and background have similar color distributions. An example is shown in Fig. 1a, where the skirt and hair of the pedestrian exhibit similar color to the background. Many foreground pixels are misclassified in Fig. 1b by Sheikh s algorithm [1], which is one of the most excellent detection algorithms nowadays. The work of this paper aims to explore the reasons for the misdetections caused by the color similarity, and to present solutions of the color similarity problem. The moving object detection plays an important role in a wide range of computer vision applications. For example, some works [2,3] regarding sport video analysis and some other works [4,5] regarding traffic monitoring are reported in recent years. The environment is wellconstrained in sports video analysis. Algorithms used in intelligent transportation are often designed to detect Corresponding author. Tel./fax: addresses: hover_chang@sjtu.edu.cn (X. Zhang), jieyang@sjtu.edu.cn (J. Yang). specific objects, such as the pedestrian [4] and shadow [5]. More algorithms are designed to deal with general scenes, such as the dynamic background and illumination changes. The background modeling is the early criterion exploited for moving object detection. A survey of the background modeling can be found in [6]. Many background models have been developed in recent years, such as the Gaussian mixture model [7], the nonparametric statistical model [8,9], the predictive models [10,11], and et al. The foreground modeling is exploited for more accurate detection in recent years. The foreground model can be constructed in a consistent fashion with the background model [12]. The nonparametric statistical model is the most widely used model now, for it is capable of modeling arbitrary probability distributions. In order to accelerate the computation of the kernel density estimator (KDE) used in nonparametric modeling, fast KDE [13,14] are developed. The thresholding is formerly used to classify new observations into foreground and background, whereas the energy minimization tools [15,16] have become the standard classification tools now. In this paper, further analysis of the example in Fig. 1 reveals that the confusion point and the model inaccuracy /$ - see front matter & 2008 Elsevier B.V. All rights reserved. doi: /j.sigpro

2 686 X. Zhang, J. Yang / Signal Processing 89 (2009) independent of each other and have the same variance. Considering the same pixel I n at time instant t, before which all pixels labeled foreground in a K*K neighborhood of position n form the foreground model j fn ¼ (y 1,y,y m,y,y M ). The foreground probability can be computed as the background probability. In fact the foreground probability is smaller than the background probability most of the time, even if I n is a foreground pixel. In order to enhance the foreground probability, the foreground probability is modeled as a mixture of the KDE and a uniform likelihood in [1] as Fig. 1. (a) Is a frame from a video sequence with serious color similarity problem and (b) is segmentation by Sheikh s algorithm. are the reasons for the misdetections due to the color similarity problem. Accordingly, the color similarity can be solved by shifting the confusion point and improving the model accuracy. Based on this conclusion, a simple but effective algorithm is presented. More elaborate algorithms are expected to be developed in the future. This paper is organized as follows. The confusion point and the model accuracy are discussed in Sections 2 and 3, respectively. Experimental results are given in Section 4, followed by the conclusion in Section Confusion point As stated in introduction, the nonparametric statistical model is the most popular model nowadays. In order to improve the detection, multiple features have been used for statistical analysis in many algorithms. A combination of color and optical flow is used [17], a combination of color and spatial-temporal derivatives is used [18], and another combination of spectral, spatial and temporal features is used in [19]. The use of multiple features leads to distinct performance improvement, but the color similarity is still a difficult problem. For example, both the color and spatial features are used in Sheikh s algorithm, but many foreground pixels are still misclassified in Fig. 1b. No matter how many features are used, finally a probability model is computed. Since we are interested in how to use the probability model for classification but not how to construct the probability model, only a simple model with the color feature is used in this paper. Let I n be a pixel of an image in the RGB color space, where n is the index of the image lattice. Moving object detection aims to assign each pixel I n a label from the set (background, foreground). Considering pixel I n at time instant t, before which all pixels labeled background in a K*K neighborhood of position n form the background model j bn ¼ (y 1,y,y m,y,y M ), where M is the total number of pixels in j bn. The background probability, the probability of pixel I n belonging to the background, can be computed with the KDE as ^pði n jj bn Þ¼K X 2 f H ði n y m Þ (1) M where f is a kernel function with a variance matrix H. In this paper, we assume that all color components are ^pði n jj fn Þ¼wtþð1 wþk X 2 f H ði n y m Þ (2) M where w is a weight coefficient, t is a random variable with uniform probability. In order to adapt to the background and foreground changes, sliding windows of length r b frames and r f frames are maintained for the background and foreground models, respectively. More discussions about nonparametric modeling can be found in [1] and [8]. Given the membership probabilities computed with Eqs. (1) and (2), traditionally a likelihood ratio classifier is used to classify new observations into background and foreground. The likelihood ratio Z of I n is defined as ZðI n Þ¼lnð^pðI n jj fn ÞÞ= lnð^pði n jj bn ÞÞ (3) The likelihood ratio classifier w is defined as ( wði n Þ¼ background if ZðI nþ4k foreground otherwise Because segmentations obtained by the likelihood ratio classifier are often noisy, energy minimization tools are used to classify observations instead of the likelihood ratio classifier by introducing prior knowledge into the decision framework. Although much better results are generated by energy minimization from visual observation, in fact the decision for most pixels by energy minimization is the same as that by the likelihood classifier, except a few pixels that are classified as noise points by the likelihood classifier. In this paper, a popular energy minimization tool, the graph cut, is used to classify observations as in [1]. Theoretically the sum of ^pði n jj fn Þ and ^pði n jj bn Þ is 1, but it is not equal to 1 most of the time. Keeping the ratio between ^pði n jj fn Þ and ^pði n jj bn Þ unchanged and normalizing the sum to 1, the normalized probabilities are denoted as p(i n j fn ) and p(i n j bn ), respectively. According to Eqs. (3) and (4), the normalization will not change the final classification of pixels. The negative log-probability ln(p(i n j fn )) can be denoted as lnðpði n jj fn ÞÞ ¼ lnð1 pði n jj bn ÞÞ (5) The definition area of p(i n j bn ) is (0,1). Let p(i n j bn ) be the x axis, the two functions, ln(p(i n j fn )) and ln(p(i n j bn )), are shown in Fig. 2a. The confusion point d, as shown in Fig. 2a, is the cross-point of the two functions. When a pixel is at the confusion point, the probability of the pixel belonging to the background is equal to the probability of the pixel belonging to the (4)

3 X. Zhang, J. Yang / Signal Processing 89 (2009) Fig. 2. (a) Shows the confusion point and the learned Gaussian, (b) shows the c f and c b and (c) shows weighted functions by c f and c b. foreground. As a result, this pixel is very easy to be misclassified. Further, more close to the confusion point a pixel is, more easy to be misclassified it is. When the foreground exhibits similar color to the background, the foreground model will exhibits similar color to the background model too. Consequently, the probability of the foreground belonging to the foreground model should be near to the probability of it belonging to the background model. According to the conclusion in the last paragraph, pixels of the foreground should be near to the confusion point, and therefore are easy to be misclassified. As shown in Fig. 1b, really many foreground pixels showing similar color to the background are misclassified as background. In a word, the existence of the confusion point is one reason for the misdetections due to the color similarity problem. Let us compute the histogram of Z of all misclassified foreground pixels in Fig. 1b, which is shown in Fig. 3. Fig. 3 shows all misclassified foreground pixels cluster in a compact interval but not with Z(d) ¼ 1 as its center point. If the histogram is computed with all foreground pixels showing similar color to the background but not only these misclassified foreground pixels, Fig. 3 should be a compact interval with Z(d) ¼ 1 as its center point. Let us normalize Fig. 3 and approximate the normalized histogram with a Gaussian curve. The learned Gaussian is depicted in Fig. 2a. Fig. 2a also shows that pixels on the left and right of the confusion point tend to be classified as foreground and background, respectively. Considering that all similarly colored foreground pixels are in a compact interval with the confusion point as the center point, many similarly colored pixels can be rightly classified if we can move the confusion point toward the right side slightly. This purpose can be arrived by weighting ln(p(i n j fn )) with a coefficient c f, where c f A(0,1), or weighting ln(p(i n j bn )) with another coefficient c b, where c b A(1,+N), or weighting the two functions simultaneously. Examples of the two weight coefficients and weighted function are shown in Fig. 2b and c, respectively. Experiments show that more foreground pixels can be rightly classified with a small c f and a big c b, but the false alarms increase at the same time. Thus we use a learning sequence to learn the best c f and c b by visual observation. In this paper, ten pairs of c f and c b are predefined, that is (0.9,1.1), (0.91,1.09), Fig. 3. The histogram of Z of all misclassified foreground pixels in Fig. 1b. Fig. 4. (a) Is segmentation by the weighting technique, (b) is segmentation by the weighting technique with nonoptimal parameters and (c) is segmentation by the proposed algorithm. (0.92,1.08),y, and (0.99,1.01). The selected pair is (0.95,1.05). Segmentation of Fig. 1a with the learned parameters is shown in Fig. 4a. For the purpose of comparison, segmentation with a smaller c f and a larger c b compared with the selected c f and c b are shown in Fig. 4b. This example shows the increase of the false alarms constrains the further performance improvement of the weighting technique. The best c f and c b are selected by visual observation. We may be able to develop a method to select the best c f

4 688 X. Zhang, J. Yang / Signal Processing 89 (2009) Fig. 5. (a) Shows c f (I n ) and c b (I n ) and (b) shows weighted functions by c f (I n ) and c b (I n ). and c b automatically by setting a threshold on the upper limit of the false alarms. In order to prevent the increase of the false alarms, we can weight ln(p(i n j fn )) and ln(p(i n j bn )) only in a local interval but not in the whole definition interval. The local interval can be decided from the histogram shown in Fig. 3. Further, better results can be expected by weighting ln(p(i n j fn )) and ln(p(i n j bn )) with two weight functions but not two constants. For example, the Gaussian function learned from Fig. 3 records the probability of each pixel being misdetected. Then we can parameterize c f and c b as two functions c f (I n ) and c b (I n ) with the learned Gaussian. Finally ln(p(i n j fn )) and ln(p(i n j bn )) are weighted by c f (I n ) and c b (I n ), respectively. Because each pixel is weighted according to the probability of the pixel being misdetected, we think that the increase of the false alarms can be prevented to the fullest extent. A conceptualization of c f (I n ), c b (I n ) and weighted functions is shown in Fig. 5. We leave all the above questions for the future work. The famous fuzzy mathematics has been successfully applied in several aspects of computer vision, such as the motion detection and shadow elimination [20], and image denoising [21]. Kerre book [22] gives a thorough description of many fuzzy techniques used in image processing. The confusion point introduced in this section can also be understood by the fuzzy mathematics. All pixels in current frame form the universe of fuzzy sets. ln(p(i n j fn )) is the membership function of fuzzy set {foreground pixels} and ln(p(i n j bn )) is the membership function of fuzzy set {background pixels}. When the membership of pixel I n belonging to both {foreground pixels} and {background pixels} are 0.5, which means the confusion point just is the transition point of fuzzy sets {foreground pixels} and {background pixels}, I n is very easy to be misclassified. From the perspective of fuzzy mathematics, the classification of new observations is turned into another problem of deblurring. We leave the question of using the fuzzy theory to deal with the color similarity for the future work. 3. Model inaccuracy Besides the confusion point, the model inaccuracy is another reason for the misdetections due to the color similarity. For a similarly colored foreground pixel I n in current frame, more misdetections of corresponding pixels in previous frames will lead to decrease in the amount of pixels in j fn. Consequently, the foreground probability of I n will become small, and therefore I n is likely to be situated on the right of the confusion point. In fact corresponding pixels of I n are often misdetected in previous frames due to the color similarity. As a result, the misdetections are accumulated, and finally result in misdetections again in current frame, as shown in Fig. 1b. Contrarily, if more corresponding pixels are rightly classified in previous frames, the foreground probability of I n will become large and I n is more likely to be rightly situated on the left of the confusion point. In a word, more foreground pixels are rightly classified in previous frames, more accurate the foreground model is. All elements in j fn are obtained without the weighting technique. Let us define a new foreground model c fn as C fn ¼ðy 1 ;...; y m ;...; y Q Þ (6) where Q is the total number of pixels in c fn. c fn is constructed in the same manner as j fn but each element in c fn is obtained by the weighting technique. Because the weighting technique is carried out on previous frames, more foreground pixels should be rightly classified before current frame, and Q should be larger than M. Consequently, c fn should be more accurate than j fn. In this paper, a simple algorithm is presented by combing the weighting technique and c fn. Each frame is segmented by the weighting technique. Then segmentations in the last r f frames by the weighting technique are all used for foreground modeling. Final segmentation of Fig. 1a by the proposed algorithm is shown in Fig. 4c. Another example is shown in Fig. 6. Fig. 6a is a frame with color similarity problem. Fig. 6b is segmentation by Sheikh s algorithm. Fig. 6c is segmentation by the weighting technique without c fn. Fig. 6d is segmentation by the proposed algorithm. This experiment shows better results can be obtained by c fn. For current frame, the foreground models of all pixels just make up the segmentations of the last r f frames. Segmentations of the last r f frames of Fig. 6a are shown in Fig. 7. The first and second rows of Fig. 7 are composed of j fn and c fn, respectively. We choose r f ¼ 2 in this paper.

5 X. Zhang, J. Yang / Signal Processing 89 (2009) Fig. 6. (a) Is a frame from a typical sequence, (b) is segmentation by Sheikh s algorithm, (c) is segmentation by the weighting technique without c fn and (d) is segmentation by the proposed algorithm. mean-shift tracker [23] and the particle filter [24], can be used to locate the moving object in current frame. A survey of object tracking can be found in [25]. The results from tracker may be better than that from predictor, because the information in current frame is used by the tracker but not by the predictor. In addition to the location feature, the shape and scale features can also be used for foreground modeling. For example, for a nonrigid moving object, we can select such historical segmentations, in which the moving object shows similar shape to the shape of the moving object in current frame, to create a new foreground model based on shape similarity. The selection of the shape descriptor is a key problem in this idea. Many shape descriptors, such as the Zernike moment descriptors and the generic Fourier descriptor [26], can be used to describe a nonrigid object. A survey of shape representation can be found in [27]. Further, in order to deal with multi-object detection, individual foreground model can be created for each moving object in the scene to take advantage of the location or shape features. The model accuracy is only a subjective criterion in this paper. We need a rigid definition of model accuracy from a mathematical viewpoint to compare various foreground models. We leave all these questions for our future work. 4. Experimental results Fig. 7. Segmentations of the last r f frames of Fig. 6a. See text for details. Evidently the second row is more suitable for foreground modeling than the first row, since more foreground pixels are rightly classified. Both j fn and c fn are based on the color feature. Better results are expected by the use of multiple features for statistical analysis. Multiple features can also be used to improve the model accuracy in a way different from statistical analysis. For example, based on the position of the moving object in previous frames, the position of the moving object in current frame can be predicted by many prediction theories. Then a more accurate foreground model can be created by shifting j fn or c fn to the predicted position. Alternatively, the moving object can be tracked from one frame to the next, and then a new foreground model can be created by shifting j fn or c fn to the tracked position. Many tracking algorithm, such as the Three sequences are used to test the proposed algorithm. The first column of Fig. 8 shows two frames of a typical sequence. The second and third columns are segmentations by Sheikh s algorithm and the proposed algorithm, respectively. This experiment shows better results can be obtained by the proposed algorithm. However, there is some difference between the upper and the lower images in the third column. Almost all foreground pixels are rightly classified in the upper image, but some foreground pixels still cannot be detected in the lower image. This phenomenon is caused by the model inaccuracy due to the movement of the pedestrian. The horizontal displacement of the pedestrian in the bottomleft image is more than 15 pixels, while the displacement is less than 5 pixels in the top-left image. As stated in the previous section, a predictor or tracker should be helpful in improving the model inaccuracy due to the rapid movement of the object.

6 690 X. Zhang, J. Yang / Signal Processing 89 (2009) Fig. 8. The first test sequence. See text for details. The first row of Fig. 9 shows two typical frames of a video sequence from the PETS2001 dataset. A white van is moving in front of a parked white car in this sequence. The second row shows segmentations by Sheikh s algorithm. Detected foreground by the proposed algorithm is given in the third row, showing that almost all pixels on the left half part of the van can be rightly classified, but many pixels on the right half part still cannot be detected. This experiment proves it again that the motion information can be used to improve the model accuracy. Further, the moving object in this experiment is a small object, and small object detection is a difficult problem. Another example of small object detection is shown in Fig. 10. The first column of Fig. 10 shows two typical images of the third sequence, which is available at A pedestrian dressed in a white T-shirt is moving in front of a similarly colored background in this sequence. Detected foreground by Sheikh s algorithm and the proposed algorithm is shown in the second and third columns, respectively. Although better results can be obtained by the proposed algorithm, many foreground pixels still cannot be detected, especially in the bottom-right image. In this example, the shape of the moving object changes quickly and periodically. Improved results are expected by the use of such historical segmentations for foreground modeling that the pedestrian in these segmentations shows similar posture to the posture of the pedestrian in current frame. 5. Conclusions We have established that the confusion point and the model inaccuracy are the reasons for the misdetections Fig. 9. The second test sequence. See text for details. due to the color similarity. Based on this conclusion, a weighting technique and a new foreground model are presented to deal with the color similarity problem through shifting the confusion point and improving the model accuracy. The proposed algorithm is effective but still faulty. Better weighting techniques and foreground models are expected to be developed in the future based on these useful conclusions listed in this paper.

7 X. Zhang, J. Yang / Signal Processing 89 (2009) Fig. 10. The third test sequence. See text for details. Acknowledgments The authors are grateful to the anonymous reviewers for their comments, which have helped us to greatly improve this article. This study is supported by the China 863 High Tech. Plan (no. 2007AA01Z164), and supported by the National Natural Science Foundation of China (no , no and no ). References [1] Y. Sheikh, M. Shah, Bayesian modeling of dynamic scenes for object detection, IEEE Trans. Pattern Anal. Mach. Intell. 27 (11) (2005) [2] A. Ekin, A.M. Tekalp, R. Mehrotra, Automatic soccer video analysis and summarization, IEEE Trans. Image Process. 12 (7) (2003) [3] O. Utsumi, K. Miura, I. Ide, S. Sakai, H. Tanaka, An object detection method for describing soccer games from video, in: IEEE Conference on Multimedia Expo, 2002, pp [4] L. Zhao, C.E. Thorpe, Stereo and neural network-based pedestrian detection, IEEE Trans. Intell. Transportation Syst. 1 (3) (2000) [5] J. Kato, T. Watanabe, S. Joga, J. Rittscher, A. Blake, An HMM-based segmentation method for traffic monitoring movies, IEEE Trans. Pattern Anal. Mach. Intell. 24 (9) (2002) [6] M. Piccardi, Background subtraction techniques: a review, IEEE International Conference Systems, Man and Cybernetics, 2004, pp [7] C. Stauffer, W.E.L. Grimson, Learning patterns of activity using realtime tracking, IEEE Trans. Pattern Anal. Mach. Intell. 22 (8) (2000) [8] A. Elgammal, R. Duraiswami, D. Harwood, L. Davis, Background and foreground modeling using non-parametric kernel density estimation for visual surveillance, Proc. IEEE (2002) [9] L. Lu, G.D. Hager, A nonparametric treatment for location/segmentation based visual tracking, in: IEEE Conference on Computer Vision and Pattern Recognition, 2007, pp [10] J. Zhong, S. Sclaroff, Segmenting foreground objects from a dynamic textured background via a robust Kalman filter, in: IEEE International Conference on Computer Vision, 2003, pp [11] A. Monnet, A. Mittal, N. Paragios, V. Ramesh, Background modeling and subtraction of dynamic scenes, in: IEEE International Conference on Computer Vision, 2003, pp [12] S. Mahamud, Comparing belief propagation and graph cuts for novelty detection, in: IEEE Conference on Computer Vision and Pattern Recognition, 2006, pp [13] A. Elgammal, R. Duraiswami, L. Davis, Efficient kernel density estimation using the fast gauss transform with applications to color modeling and tracking, IEEE Trans. Pattern Anal. Mach. Intell. 25 (11) (2003) [14] C. Yang, R. Duraiswami, N. Gumerov, L. Davis, Improved fast gauss transform and efficient kernel density estimation, in: IEEE Conference on Computer Vision and Pattern Recognition, 2003, pp [15] V. Kolmogorov, R. Zabih, What energy functions can be minimized via graph cuts, IEEE Trans. Pattern Anal. Mach. Intell. 26 (2) (2004) [16] Y. Boykov, O. Veksler, R. Zabih, Fast approximate energy minimization via graph cuts, IEEE Trans. Pattern Anal. Mach. Intell. 23 (11) (2001) [17] A. Mittal, N. Paragios, Motion-based background subtraction using adaptive kernel density estimation, in: IEEE Conference on Computer Vision and Pattern Recognition, 2004, pp [18] A. Criminisi, G. Cross, A. Blake, V. Kolmogorov, Bilayer segmentation of live video, in: IEEE Conference on Computer Vision and Pattern Recognition, 2006, pp [19] L. Li, W. Huang, I. Gu, Q. Tian, Statistical modeling of complex backgrounds for foreground object detection, IEEE Trans. Image Process. 13 (11) (2004) [20] J. Shen, Motion detection in color image sequence and shadow elimination, Proc. SPIE (2004) [21] V. Zlokolica, A. Pizurica, W. Philips, Fuzzy logic recursive motion detection and denoising of video sequences, J. Electron. Imaging 15 (2) (2006) [22] Kerre, Fuzzy techniques in image processing (studies in fuzziness and soft computing vol. 52), June 2000, Lavoisier Editions. [23] M. Park, Y. Liu, R.T. Collins, Efficient mean shift belief propagation for vision tracking, in: IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp [24] A.C. Sankaranarayanan, A. Srivastava, R. Chellappa, Algorithmic and architectural optimizations for computationally efficient particle filtering, IEEE Trans. Image Processing 17 (5) (2008) [25] A. Yilmaz, O. Javed, M. Shah, Object tracking: a survey, ACM J. Comput. Surv. 38 (4) (2006) [26] D.S. Zhang, G. Lu, Generic Fourier descriptor for shape-based image retrieval, in: IEEE International Conference on Multimedia and Expo, 2000, pp [27] D.S. Zhang, G. Lu, Review of shape representation and description techniques, Pattern Recognition 37 (1) (2004) 1 19.