Edge-segment-based Background Modeling: Non-parametric Online Background Update

Transcription

1 th IEEE International Conference on Advanced Video and Signal Based Surveillance Edge-segment-ased Background Modeling: Non-parametric Online Background Update Jaemyun Kim, Adin Ramirez Rivera, Gihun Song, Byungyong Ryu, Oksam Chae Kyung Hee University Gyoenggi-do, South Korea sense21c, adin, gihunsong, read100nm, Astract For ackground-sutraction-ased moving oject detection, reliale ackground modeling is the most important component. Pixel-ased methods are sensitive to illumination change, and edge-ased methods can solve illumination-related prolems, ut have shape distortion prolems. In this paper, we propose an edge-segmentased statistical ackground modeling algorithm and an online update mechanism to detect moving ojects from consecutive frames, which creates a alance etween the pixel- and edge-ased methods. Our ackground modeling method uses a statistical map to model the frequency of the ackground-edges, as distriutions that comprise support regions approximated with a quadratic function and enhanced with color and gradient information, to overcome the edge-distortion prolem y matching the edge-segments to the modeled distriutions. To adjust the changing ackground in the scenes, we propose an online-ackground update step for every incoming frame that updates the statistical map and enhances the information held y the distriutions support regions. Furthermore, our experiments show that the proposed method otains etter results and detects moving edges efficiently. 1. Introduction Moving oject detection has een studied widely due to its several applications, e.g., surveillance [7], video-ased traffic systems [4], video segmentation [8], or human ehavior analysis [1]. The most common approach is to detect moving ojects from the difference of the current frame with the ackground model. Thus, the ackground model must include all the different ackground s characteristics, such as illumination changes, ackground movements, and even dynamic situations (e.g., a car parking or leaving the scene, new ojects left in the scene, ojects take out from the scene, among others). However, current methods have prolems on dynamic ackgrounds. In the following we discuss the drawacks with the pixel- and edge-ased ackground modeling methods. Pixel-ased ackground modeling methods model each pixel in the scene to represent the changing ackground. These methods generate a ackground model faster than the edge-ased methods. However, they have difficulties if the moving ojects have a similar color with the ackground [2, 11]. Also, color changes occur often to certain location y repetitive oject motion, shadows, light reflectance, or noise. To solve this type of prolem, the ackground model should e updated every frame. Furthermore, pixel-ased methods [13, 14, 21] cannot handle the multi-modal distriutions in dynamic environments, and are sensitive to illumination changes and noise. Despite that previous research has explored the multi-modal ackground modeling [19], these methods still detect some ackground regions as moving ojects (the ghost effect) when the moving oject moves faster than the ackground asorption mechanism or moves suddenly [22]. On the other hand, edge-ased methods that model the ackground using edges as features are less sensitive than pixel-ased methods to intensity changes. The former can overcome the prolems of pixel-ased methods mentioned aove, even the ghost effect [4, 7, 8]. However, edge-ased methods have a critical prolem related to edge-distortion deficiencies, e.g., edges from static ojects have position, shape, and orientation variations in consecutive frames. Consequently, edges extracted from sequential frames are not consistent. Therefore, we cannot use a naive framewise sutraction scheme to remove the ackground edges. Moreover, existing edge-ased methods [4, 8] have many false alarms, ecause they use a simple edge differencing method. In this paper, we propose a statistical edge segmentased ackground modeling method that does not require prior motion information. In our method, the ehavior of the ackground is accumulated into a statistical distriutions for each frame, and the ehavior of foreground is elim /13/$ IEEE 214

2 Figure 1: An astraction of the proposed method. inated from these statistical distriutions y adaptive thresholding. Additionally, we refine the ackground statistical distriutions through a quadratic approximation to increase the matching accuracy. Moreover, we use an array of color and gradient information from each ackground distriution region to restore over eliminated edge-segments y matching color and gradient information etween the incoming frames and model. By using these statistical distriutions, we gain tolerance to illumination, position, shape, and orientation variation, and even camera jitter. Furthermore, our method requires lower computational resources for the oject detection step, ecause our method does not need to search for each pixel individually. Thus, our method is more roust and more reliale than traditional pixel- and edgeased methods. 2. Proposed Method Our edge-segment-ased statistical ackground model asors characteristic of the edge s ehavior, such as position, shape, and orientation variation into a statistical distriution map [12, 15, 16]. Therefore, each statistical distriution comprises the edges ehavior from the training frames. Further, our adaptive threshold mechanisms refine the statistical distriution to the optimal ackground distriution to perform the oject detection. Moreover, the update method maintains the ackground model y using the changes of ackground and foreground. Fig. 1 shows the five steps of our ackground method. (1) First, we extract edge from training frame for accumulating the edge s ehavior frame-wise. (2) Then, the edge distriutions from each frame are accumulated. (3) The accumulated distriutions are adaptively thresholded y accumulation to remove moving ojects distriutions. (4) And again, the distriutions are adaptively thresholded y motion to generate an optimal ackground model. (5) Additionally, we create a set of frames (i.e., color and gradient magnitude) to distinguishing etween ackground and moving edges Statistical Modeling To generate the reliale ackground model, we accumulate the ehavior of the edges over time. Thus, we extract the edges from each frame, using a Canny edge detector [3], and create a inary edge map Et for the training frame t, and then we accumulate all the training frames in an accumulator, ACC, y ACC(x, y) = tf X Et (x, y), (1) t=t0 where (x, y) is a pixel position, t0 is the initial training frame and tf is the final training frame (we used N = tf t0 = 100 frames). Then we create the statistical map, SM, that is the set of all the ackground distriutions from all the training frames y smoothing the accumulation to avoid noise and get a etter approximation of the edge ehavior y SM = ACC G, (2) where G is a normalized Gaussian kernel function that uses the numer of edges in the neighorhood to smooth the final accumulation per distriution Adaptive Threshold Given that the ackground and the foreground have different values of accumulation in the SM, we can perform threshold operations to remove spurious foreground distriutions from our ackground model. Hence, we threshold our SM from two points of view: accumulation and motion. The ackground distriutions have narrow shape and higher accumulated values in the SM, ecause the edges have similar shape and location due to the constant nature of the ackground. While the foreground, e.g., moving ojects, has wider shape and lower-accumulated-value distriution in the SM, as the moving edges present complex variations, such as appearance, movement, or disappearance during the consecutive frames. Therefore, the proposed method removes the distriutions of the moving ojects in the SM using an accumulation threshold [16]. Moreover, we apply an adaptive motion 215

3 Figure 2: Illustration of orthogonal cross-sections from a mean segment of a distriution. threshold to define accurate search regions for each distriution to perform the edge-matching process. This motion threshold uses a Quadratic approximation [18] of the distriutions to prune its orders accurately Accumulation Threshold The proposed method is tolerant to moving ojects in the training process, due to our thresholding mechanism that removes the moving oject s distriutions from the SM. To decide the accumulation threshold T, we assume an average minimum speed v for moving ojects in pixels per frames. Thus, we define T = max(acc), (3) v where max(acc) is the maximum value in ACC. The accumulation thresholding generates a statistical distriution map, SDM, with the property that no moving distriutions are on it. Thus, we remove the distriutions with accumulation values elow the threshold T y ACC(x, y) if ACC(x, y) > T SDM(x, y) =. (4) In our experiments, we assume a moving oject minimum speed of five pixel per frame, v = 5, to remove the distriutions of moving ojects Motion Threshold We generate the optimal ackground model y pruning the distriutions underlying regions to their significant values. Thus, we create a support region, SR, for each distriution that covers the variation of the ackground edges without the outliers or infrequent edges occurrence. Therefore, we approximate the distriution created y the ackground edges movement with a quadratic function. Hence, for each distriution in the SDM we find its quadratic approximation [18] and use it to find a optimal cutting point to remove the outliers of the distriution. This process is descrie in the following. First, we apply the Multi-Directional Non-Maximum Suppression [20], to extract the middle segment from each distriution in the SDM. These segments mark the position of the maximum peak in each distriution and can e considered as the mean of each distriution. Second, we extract several points from the orthogonal cross-section to the mean segment of the distriution (as shown in Fig. 2) to compute the quadratic curve that will approximate the distriutions shape. Further, we apply a polynomial regression method [18] using a quadratic model of the form y = a 0 + a 1 x + a 2 x 2 + ɛ, (5) where x = x 1, x 2,..., x n } are the position values within the cross-section, y = y 1, y 2,..., y n } are the respective accumulation values, and ɛ is an unoserved random error with mean zero conditional on a scalar variale x. Consequently, we find the coefficients a 0, a 1, and a 2 with the minimum error ɛ using a least-squares approach [18]. Further, we define the cutting point to prune the distriution as the intercept with the x axis (y = 0) y p cut = a1± a 2 1 4a0a2 2a 2 if a 2 1 4a 0 a 2 0. (6) Finally, we threshold the distriution y using the cutting point as a distance from the mean point. Thus, if the distance of a pixel from the mean of the distriution is larger than the cutting point, we remove that pixel from the distriution, y SDM(x, y) (x x)2 + (y ȳ) SR(x, y) = 2 p cut, (7) where SR is the optimal support region, and (x, y) is a pixel position from the distriution with cut point p cut and mean position ( x, ȳ) Color and Gradient information When the foreground detection, the SR has a prolem that cannot recover moving edges if they lie in the ackground distriution. Inspired y the PBAS [6] method, we store the color and gradient magnitude information from the last N frames (as shown in Fig. 1) into F (x, y) = F 1 (x, y),..., F N (x, y)}, (8) where (x, y) is the location of pixel, and F k = R, G, B, GM} comprises of the frame color and its gradient magnitude in the array F. We use this array to avoid overelimination of moving edge y SR at the detection step. Thus, during foreground detection, over-eliminated edges are decided to e ackground or foreground y a function 216

4 D(x i ), such that 1 #dist(i(x, y), F k (x, y)) < t dist } > # min D(x, y) =, (9) where I = R, G, B, GM} consists of the color and gradient magnitude of am incoming frame, and the distance function is dist(i(x, y), F k (x, y)) = α I GM IGM (x, y) F GM k + I R,G,B} (x, y) F R,G,B} k, (10) where α/i GM weights the importance of pixel values against the previous frame gradient average value I GM. In our experiments, we used N = 20, α = 10, t dist = 20, and # min = 0.1 N Background Update We need to update the ackground model, SR, to asor the ackground changes that appear every incoming frame, such as appearance or disappearance of ackground edges. The ACC may include unveiled ackground edges if a ackground component ecomes foreground, thus revealing new ackground into the scene. On the other hand, the ACC needs to include stopped moving ojects that should ecome ackground. Hence, the updated accumulation ACC update is defined y ACC update = βe new + (1 β)acc, (11) where E new is a inary edge map from incoming frame, and β is the update constant (we used β = 0.05). After updating ACC, the remaining procedure to generate the new SR follows the ackground modeling steps introduced efore. Also, we update the color and gradient magnitude information y deleting the oldest frame in the array F and the new frame is added instead Foreground Detection We use the optimized ackground distriution, SR, and the recent N frames color and gradient magnitude, F, to detect the moving edges in the scene. In the foreground detection, we extract a inary edge map E new from the incoming frame using the Canny edge detector [3]. Then y using E new and SR, we classify the foreground and the ackground, such that, if an edge in E new does not lie within the ackground distriution in the SR then that edge is classified as moving edge, otherwise it is considered a candidate ackground edge. Further, we recover the over-eliminated moving edges from the candidate ackground edges y using the decision D, as shown in Eq. (9). Consequently, we Dataset Precision Recall F-Measure Baseline Intermittent Shadow Average Tale 1: Evaluation of the proposed method in three categories of the ChangeDetection.net dataase [5]. Methods Precision Recall F-Measure PBAS [6] PSP-MRF [17] SOBS [9] SC-SOBS [10] Proposed Tale 2: Comparison of the proposed method to other methods using average performance measures. consider the detected moving edges the result of oth the foreground detection and the recovered edges from the candidate ackground. 3. Experiments and Results We test our proposed method using several datasets from the ChangeDetection.net [5] dataase. This dataase support six categories, for a total of 31 video datasets. Among the dataase scenarios we chose 16 datasets (from 3 categories, namely, Baseline, Intermittent oject motion, and Shadow) that have dynamic environment scenarios, such as unveiled ackground, incorporated ackground y stopped foreground in consequent frames, or ackground illumination change y shadows from foreground. Further, our proposed method shows stale moving oject detection in these sequences Quantitative Evaluation We evaluate the results of the proposed method using a quantitative evaluation according to ChangeDetection.net enchmark [5]. Specifically, we use precision, recall, and F-measure, defined y T P Recall = T P + F N T P Precision = T P + F P Precision Recall F-measure = 2 Precision + Recall (12) (13) (14) where T P is the total numer of true positives, T N is the total numer of true negatives, F N is the total numer of false negative, and F P is the total numer of false positive. 217

5 Original ROI Ground Truth Regenerated GT Proposed method Tramstop Streetlight Figure 3: Regenerated foreground ground truth of two datasets in the intermittent moving oject category. We generate the edge-ased ground truth y using an AND operator etween ChangeDetection.net s region-ased foreground ground truth (with their ROI, as shown in Fig. 3) and extracted edges (using Canny edge detector) from each frame. For two datasets (Streetlight and Tramstop) in the category of the Intermittent oject motion, moving edges should e classified as ackground, as we assume that edges from foreground that stay in the same position consecutively should e consider part of the ackground. Therefore, we eliminate that edges from our edge-ased ground truth (as shown in Fig. 3) Results Tale 1 shows the average evaluation (precision, recall, and F-measure) for our proposed method at each category. And Tale 2 shows the comparison of the proposed method against several other methods using average performance measures. The proposed method has higher precision than the other methods, ut recall is not as high as our method may miss some small edges inside the moving ojects not crucial for its detection. In spite of that evaluation, our F-measure shows a etter result over the previous methods. Furthermore, as shown in Fig. 4, the proposed method detect moving edges closer to the ground truth. In the case when the ackground start moving to ecome foreground, new ackground will reveal where was covered region y the car previously. However, our method can learn and adapt to the new scene quickly enough, as shown in Fig. 4(). Also, the illumination change from shadows of foreground [Fig. 4(c)] presents no challenge to the proposed method that detects foreground edges in a stale way. 4. Conclusion We proposed an edge-segment-ased statistical ackground modeling technique and its update method to detect moving oject in dynamic environments. The proposed method refines the ackground model y using two thresholds (accumulation and motion). Further, the recent color and gradient magnitude information from the incoming frames is used to support the recovering of overeliminated edges and to incorporate that edges to the foreground. Moreover, we update the ackground model to otain a stale foreground detection in the sequence. Our results show that we can overcome the edge-ased prolems (shape and position distortion), as well as prolems of pixelased oject detection (sensitivity to illumination). Furthermore, the proposed method adjusts to scene changes, such as new ackground, due to ackground and foreground ecoming another, and vice versa. Consequently, we found promising results that can e used in several applications, including surveillance in dynamic ackgrounds and content-ased video encoding. References [1] P. Banerjee and S. Sengupta. human motion detectionandtracking for video surveillance. In In National Conference on Communication, IITBomay, India, Fe [2] E. Candès, X. Li, Y. Ma, and J. Wright. Roust principal component analysis? Journal of the ACM, May [3] J. Canny. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell., [4] D. Dailey, F. Cathey, and S. Pumrin. An algorithm to estimate mean traffic speed using uncalirated cameras. IEEE Trans. Intell. Transp. Syst., [5] N. Goyette, P.-M. Jodoin, F. Porikli, J. Konrad, and P. Ishwar. changedetection.net: A new change detection enchmark dataset. In CVPRW, [6] M. Hofmann. Background segmentation with feedack: The pixel-ased adaptive segmenter. In CVPRW, [7] M. Hossain, M. Dewan, and O. Chae. Moving oject detection for real time video surveillance: An edge ased approach. IEICE Transactions, [8] C. Kim and J. Hwang. Fast and automatic video oject segmentation and tracking for content-ased applications. IEEE Trans. Circuits Syst. Video Technol.,

6 Baseline Intermittent Shadow Proposed method Ground Truth Original (a) frame 657 () frame 1927 (c) frame 1626 Figure 4: The result of proposed method in the ChangeDetection.net dataase. [9] L. Maddalena and A. Petrosino. A self-organizing approach to ack-ground sutraction for visual surveillance applications. IEEE Trans. Image Process., [10] L. Maddalena and A. Petrosino. The sos algorithm: what are the limits? In CVPRW, [11] J. McHugh, J. Konrad, V. Saligrama, and P. Jodoin. Foreground-adaptive ackground sutraction. IEEE Signal Process. Lett., May [12] M. Murshed, A. Ramirez Rivera, J. Kim, and O. Chae. Statistical inary edge frequency accumulation model for moving oject detection. International Journal of Innovative Computing, Information and Control, [13] M. Piccardi. Background sutraction techniques: A review. In ICSMC, Oct [14] R. Radke, S. Andra, O. Al-kofahi, and B. Roysam. Image change detection algorithms: A systematic survey. IEEE Trans. Image Process., [15] A. Ramirez Rivera, M. Murshed, and O. Chae. Oject detection through edge ehavior modeling. In AVSS, [16] A. Ramirez Rivera, M. Murshed, J. Kim, and O. Chae. Background modeling through statistical edge-segment distriutions. IEEE Trans. Circuits Syst. Video Technol., [17] A. Schick, M. Bauml, and R. Stiefelhagen. Improving foreground segmentations with proailistic superpixel markov random fields. In CVPRW, [18] K. Smith. On the standard deviations of adjusted and interpolated values of an oserved polynomial function and its constants and the guidance they give towards a proper choice of the distriution of the oservations. Biometrika, [19] C. Stauffer and W. E. L. Grimson. Adaptive ackground mixture models for real-time tracking. In Computer Vision and Pattern Recognition, IEEE Computer Society Conference on., volume 2, pages Vol. 2, [20] C. Sun and P. Vallotton. Fast linear feature detection using multiple directional non-maximum suppression. In ICPR, [21] A. Yilmaz, O. Javed, and M. Shah. Oject tracking: A survey. ACM Comput. Surv., [22] W. Zhang, Q. Wu, and H. Yin. Moving vehicles detection ased on adaptive motion histogram. Digital Signal Processing,