Visual Saliency Based Object Tracking

Transcription

1 Visual Saliency Based Object Tracking Geng Zhang 1,ZejianYuan 1, Nanning Zheng 1, Xingdong Sheng 1,andTieLiu 2 1 Institution of Artificial Intelligence and Robotics, Xi an Jiaotong University, China {gzhang, zjyuan, nnzheng, xdsheng}@aiar.xjtu.edu.cn 2 IBM China Research Lab liultie@cn.ibm.com Abstract. This paper presents a novel method of on-line object tracking with the static and motion saliency features extracted from the video frames locally, regionally and globally. When detecting the salient object, the saliency features are effectively combined in Conditional Random Field (CRF). Then Particle Filter is used when tracking the detected object. Like the attention shifting mechanism of human vision, when the object being tracked disappears, our tracking algorithm can change its target to other object automatically even without re-detection. And different from many other existing tracking methods, our algorithm has little dependence on the surface appearance of the object, so it can detect any category of objects as long as they are salient, and the tracking is robust to the change of global illumination and object shape. Experiments on video clips of various objects show the reliable results of our algorithm. 1 Introduction Object detection and tracking is an essential technology used in computer vision system to actively sense the environment. As the robotic and unmanned technology develops, automatically detecting and tracking interesting objects in unknown environment with little prior knowledge becomes more and more important. The main challenge of object tracking is the unpredictable of the environment which always makes it hard to estimate the state of the object. The changing of illumination, clutter background and the occlusion also badly affects the tracking robust. In order to overcome these difficulties, a variety of tracking algorithms have been proposed and implemented. The representative ones include condensation [3], meanshift [4], and probabilistic data association filter [5] and so on. Generally speaking, most of the tracking algorithm has two major components: the representation model of the object and the algorithm framework. The existing frameworks can be classified into two categories: deterministic methods and stochastic methods. Deterministic methods iteratively search for the optimistic solution of a similarity cost function between the template and the current image. The cost functions widely used are the sum of squared differences (SSD) between the template and the current image [6] and kernel based

2 cost functions [4]. In contrast, the stochastic methods use the state space to model the underlying dynamics of the tracking system and view tracking as a Bayesian inference problem. Among them, the sequential Monte Carlo method, also known as particle filter [7] is the most popular approach. There are many models to represent the target including image patch [6], color histogram [4] and so on. However, color based models are too sensitive to the illumination changes and always confused with background colors. The contour based features [8][9] are more robust to illumination variation but they are sensitive to the background clutter and are restricted to simple shape models. When human sense the environment, they mostly pay attention to the objects which are visually salient. Saliency values the difference between object and background, they are not depend on the objects intrinsic property and is robust to illumination and shape changes. One of the representative visual attention approaches is visual surprising analysis [10] which proves that static and motion features are both important to video attention detection. Itti [11] has proposed a set of static features in his saliency model. More static features are proposed these years [1]. For video series, [2] introduces an method to detect salient object in video series, which combines static and motion features in Dynamic Conditional Random Field (DCRF) under the constraint of global topic model. This approach achieves good results on many challenging videos, but it needs the whole video series to compute the global topic model, which makes it can only be used offtime. In this paper, we elaborate static and motion saliency features into the framework of particle filter to formulate an online salient object tracking method. When computing the color spatial-distribution feature, we use a graph-based segmentation algorithm [12] as the color clustering method instead of Gaussian Mixture Model (GMM) which is used in [1]. Sparse optical flow [13] is used to get motion field for motion saliency feature computing. All these features are adaptively selected and combined. The main contributions of our approach are summarized as follows. First, we propose a novel method to tracking salient object online, which is robust to the illumination and shape changes, it can also automatically rebuild attention to the object being tracked disappears. Second, a segmentation based feature is proposed as the global static feature which is more effective than the feature basedongmm. This paper is organized as follows. We introduce the framework of our algorithm in section 2. The detail of saliency feature computing and combination appears in section 3. In section 3 we also introduce the spatial and temporal coherence used. Section 4 is the collaborative particle filter. Experiment results are shown in section 5. 2 Problem Description and Modeling Object tracking is an important procedure for human to sense and understand the environment. For human, this procedure can be roughly divided into three

3 Compute Constraints Spatial and Temporal Coherences Image frames Feature Extraction Saliency Maps Feature Selection and Combination Object Description Particle Filter Tracking State at time t N Salient Object Exists Y State at time t-1 Salient Object Detection : process : data Fig. 1. The flow chart of salient object tracking parts: attention establishing, attention following and attention shifting. To apply these parts to computer vision, for the input video series I 1,,I t,, detection algorithm is used to find the interesting object and build a description model for it. Usually, when there is no high level knowledge, the objects we are interested in are those who are visual salient. After the attention is established, the object s appearance description X 1 is built and the initial object state is gotten in a short time. The state can be the shape, position or scale of the object. Tracking is to estimate the state of the object at time t (X t ) given the initial state X 1 and the observation up to time t (I t =(I 1,,I t ) ). This process is also called filtering. The flow chart of our method is shown in Fig.1. The tracking model is usually built under the probabilistic framework of Hidden Markov Model (HMM): { Xt = G(X t 1 )+v t 1 I t = H(X t )+n t, (1) where G( ) andh( ) are the system transition function and observation function while v t 1 and n t are the system noise and observation noise. When tracking a single object, we formulate the problem as computing the maximum a posteriori (MAP) estimation of X t. We predict the posterior at time t as P (X t I t ) P (I t X t ) P (X t X t 1 )P (X t 1 I t 1 )dx t 1, (2) According to (2), we can use the method of statistical filtering to solve the problem. But the state space is extremely huge, computation of the integration in (2) is NP hard. So we choose to use the sequential Monte Carlo method [7]. When filtering, the state X t can be defined in various forms. Some people try to track the contour of the object [8]. But contour tracking can be easily disturbed by the background clutters and is time-consuming. So people always simplify the state to a rectangle surround the object: X t =(x t,y t,w t,h t ) X t R 4,wherex t,y t,w t,h t are the position and size of the rectangle at time t. In our method, the observation at time t is the image frame I t. The observation model includes the saliency features and the spatial and temporal coherence

4 constraints. So the observation likelihood P (I t X t ) can be formulated as P (I t X t ) exp { ( Ft (X t,i t )+Sc t (X t,i t )+Tc t (X t 1,t,I t 1,t ) )}, (3) where F t is the description of the object which comes from the final saliency feature in our method. Sc t and Tc t are the spatial and temporal coherence constraints. The detail of these features and constrains will be described in the following section. 3 The Saliency Features and Constraints 3.1 Static Saliency Features Visual saliency can be seen as a binary labeling problem which separates the object from the background. Each kind of saliency feature provides a normalized feature map f(p, I) [0, 1] which indicates the probability that the pixel p belongs to the salient object. We compute the local and regional features using themethodin[1]. Multi-scale Contrast: Contrast is the most common local feature used for attention detection. Without knowing the size of the object, we compute contrast feature f sc (p, I) under a Gaussian image pyramid as f sc (p, I) = L l=1 p N(p) I l (p) I l (p ) 2, (4) where I l is the image in the l-th level of the pyramid and N(p) is the 8- neighborhood of pixel p. Center-Surround Histogram: The salient object can always be distinguished by the difference of it and its context. Suppose the object is enclosed by a rectangle. The regional center-surround histogram feature f sh (p, I) is defined as f sh (p, I) w pp χ 2 (R (p ),RS (p )), (5) {p p R (p ),R S (p )} where R is the most distinct rectangle centered pixel p and containing the pixel p. RS is the surrounding contour of R* and have the same area of it. And the weight w pp =exp( 0.5σ 2 p p p 2 ) is a Gaussian falloff weight with variance σp 2. Color Spatial distribution: The salient object usually has distinguishing color with the background. So the wider a color is distributed in the image, the less possible a salient object contains this color. The global spatial distribution of a specific color can be used to describe the saliencyofanobject.weproposea novel and more effective method to compute this feature.

5 Fig. 2. The left one is the original image. The middle one is the map of segmentation and the right one is the map of the color spatial distribution feature. The first step of computing this feature is color clustering. We use a fast image segmentation algorithm instead of Gaussian Mixture Model to improve the speed and robustness to noise. This algorithm fuses the pixels with similar property, for example color, in a graph-based way[12]. Having the segmentation result, we unify the RGB value in the i-th image segment seg i to its average color. Then we convert the image to index color representation and compute the distribution variance of every color. So the color spatial distribution feature f sd (p, I) is defined as f sd (p, I) = ind(x,y)=ind(p) xy x x y ȳ 1, (6) where ind(x, y) andind(p) are the indexing color of point (x, y) andpointp. The segmentation result and the feature map are shown in Fig Motion Saliency Features Compared to static object, human s attention is more sensitive to moving objects. The static saliency features can be extended to motion field. In this paper, we use the Lucas/Kanade s motion estimation under a pyramidal implementation [14]. The computed motion field is a 2-D map M with the displacement of every pixel in X and Y directions. In order to compute features from the motion map using the method of computing static featureswe do the lighting operation on M to make the moving area connective. The lighting operation is a Gaussian weighting of the spot areas centered at every sparse points in M. Themotion saliency features are computed on the motion field as follows. Multi-scale Motion Contrast: This local feature f Mc shows the difference of motion. It is computed from the Gaussian pyramid of motion field map: f Mc (p, M) = L l=1 p N(p) M l (p) M l (p ) 2, (7) where M l is the motion map in the l-th level of the pyramid.

6 Center-Surround Motion Histogram: This is the regional feature which represents the motion of a block. f Mh is defined as f Mh (p, M) w pp χ 2 (R M (p ),RMS (p )), (8) {x x R (p ),R S (p )} where the weight w pp has the similar definition as that of the regional static feature. Motion Spatial Distribution: The global static feature is also extended to the motion field. Motion is first represented using GMM. Then we compute the distribution variance V M (m) of each component m. So the spatial distribution of motion f Md is defined as f Md (p, M) m P (m M p )(1 V M (m)), (9) where P (m M p ) represents the probability that pixel p belongs to component m. See details of the motion saliency features in [2]. 3.3 Feature Selection And Combination During the process of tracking, the feature space that best distinguishes between object and background is the best feature space to use for tracking [15]. To achieve best performance using the features mentioned above, we adaptively select and combine them to get the final saliency map F t. We notice that when the background of the video is nearly still and the object is moving, the motion features are decisive. In contrast, when the object and background have the similar form of movement, still or moving, we can hardly verify them in the motion field. At this time, static features are more distinctive. Frame difference is used to decide whether the object has the similar motion form as the background. First, we smooth the adjacent frames I t 1,I t by Gaussian filtering. Then, the frame difference of margin and the whole image are computed to judge the movement of background and the whole scene. If both the background and the whole scene are moving or still, we use static features, otherwise, motion features are selected. The final saliency map F t is defined as a linear combination of the selected features: F t = w tk f tk, (10) k where f tk is the k-th selected feature at time t which could be any of the features mentioned above. The weight w tk represents the distinguishing ability of f tk, which is measured by the Saliency Ratio (SR) of the feature: w tk SR tk = / f tk (p) f tk (p), (11) p X t p/ X t

7 where Xt is a simple estimation of X t by extending the area of X t 1. p Xt represents that pixel p is in the corresponding rectangle of Xt. We normalize SR tk to get w tk. Given the final saliency map, the object s description F t (I t,x t ) is defined by the sum square of F t as F t (X t,i t )= 1 (F(I t,p)) 2, (12) w t h t p X t where F(I t,p)=f t (p), represents the process of feature computing. 3.4 Coherence Constraints For the tracking problem, the rectangle should fit the object. In our method, that is to say, the border of the rectangle should be close to the edge of the salient object. So we define the spatial coherence as the sum of edge values near the border of the rectangle as Sc t (X t,i t )=λ S E(I t,p), (13) p N(X t) where N(X t ) is the area near the corresponding rectangle of X t,ande(i t,p)is the edge value of I t at pixel p. λ S = α/w L h L is the normalizing factor. Temporal coherence models similarity between two consecutive salient objects. We use the coherence mentioned in [2]: Tc t (X t 1,t,I t 1,t )=β 1 X t X t 1 + β 2 χ 2 (h(x t ),h(x t 1 )), (14) where χ 2 (h(x t ),h(x t 1 )) is the χ 2 distance between the histogram of two adjacent state. β 1 and β 2 are normalizing factors. 4 Particle Filter Tracking The particle filter [7] tracker consists of an initialization of the template model and a sequential Monte Carlo implementation of a Bayesian filtering for the stochastic tracking system. We use the method mentioned in [1] to initialize the system. In order to track moving object, the saliency features we organize in CRF are not only static but can also be motion. The initial state we get from detection is X 1 = (x 1,y 1,h 1,w 1 ). In the prediction stage, the samples in the state space are propagated through a dynamic model. In our algorithm, we use a first-order regressive process model: X t = X t 1 + v t 1, (15) where v t 1 is a multivariate Gaussian random variable.

8 Fig. 3. The results of color spatial distribution feature computed with our approach and approach mentioned in [1]. The top row are the original images. The middle row are the results of our approach. The bottom row are the comparing results. In the update stage, the particles importance weight is defined by the object description and coherence constraints. The weight of the i-th particle at time t is: wt i = F t (Xt,F i t ) Sc t (Xt,I i t ) Tc t (Xt 1,t,I i t 1,t), i (16) where Xt i is the corresponding system state of the i-th particle at time t. During update, a direct version of Monte Carlo importance sampling technology [7] is applied to avoid the degeneracy. 5 Experiments We show here the saliency map of color spatial distribution feature computed with our method and the results of salient object tracking under a variety of situations, including multifold objects tracking, tracking with object appearance changes, and automatically attention rebuilding. 5.1 Color Spatial Distribution When computing the feature of color spatial distribution. A graph-based segmentation algorithm is used to cluster the adjacent pixels with similar colors. The segmentation is done to every channel of the RGB image and the results are merged to get the final segmentation map. In Fig.3, We compare our feature maps with the maps computed using the approach mentioned in [1]. The lighter area has higher probability to be saliency. As we can see, the results of our approach shows the salient area more clearly than the comparing results. We use images from the Berkeley segmentation dataset [16] for comparing convenience.

9 5.2 Tracking Results Our approach is implemented and experiments are performed on video series of various topics. We have collected a video dataset of different object topics, including people, bicycles, cars, animals and so on. Most of our test videos are real-life data collected with a hand-held camera while others are downloaded from the internet. The objects of interest in our experiments are initiated using the detection approach mentioned in [1]. Different from the original detection process, we manually set them to be 0.3, 0.45 and 0.25 for local, regional and global features. During tracking with sequential Monte Carlo method, we sampled 100 particles for each filter. Fig.4 shows some tracking results of multifold objects, including bicycle, car, people, and bird. Instead of object s intrinsic property, the saliency based detection and tracking method depends only on the distinction between object and background. So we can track any object as long as it is salient. Fig.5 shows the tracking results of our approach when the appearance color feature of the object changes. In this experiment, we manually alter the global illumination. Besides, the red car in the video is occluded by tree leaves in some frames which also causes the changing of its appearance feature. We track the red car using our approach and meanshift [4] for comparison. For the meanshift tracker, we manually set the initial position of object. From Fig.5 we can see, our approach gives good results inspite of illumination changes and partial occlusion while meanshift fails when the appearance of the object is changed. Fig.6 shows the results when the shape of the object changes. In this experiment, the girl comes nearer and near to the camera while making different gestures, which causes obvious changes of the object shape. As we can see, our approach achieve good results under this condition. In Fig.7 we show that our tracking method can automatically rebuild attention when the object being tracked is out of sight. In this experiment, the detection algorithm set attention on the white car as the initial state. When this car goes out of the scene, attention is rebuilt on the bicycle. Finally, when the bicycle disappears and another car comes, this car becomes the salient object and draws the attention. 6 Conclusion This paper presents a novel approach of online salient object tracking. In this method, object s appearance is described by its difference to the background which is compute from the static and motion saliency features locally, regionally and globally. A new segmentation based color spatial distribution feature is proposed which is more distinctive between the object and the background. Features are adaptively selected and combined and the sequential Monte Carlo technology is used to track the saliency object. Our approach can track any salient object and is robust to illumination changes and partially occlusions. Moreover, attention can be automatically rebuilt without re-detection in our approach. We are

10 now preparing to extend this approach to multi-object tracking, which involves the modeling of objects interactions. Acknowledgment This research is supported in part by the National Basic Research Program of China under Grant No.2007CB and the National Natural Science Foundation of China under Grant No References 1. T. Liu, J. Sun, NN. Zheng, X. Tang, HY. Shum: Learning to Detect A Salient Object. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition, T. Liu, NN. Zheng, W. Ding, ZJ. Yuan: Video Attention: Learning to Detect A Salient Object Sequence. In: 19th International Conference on Pattern Recognition, (2008) 3. M. Isard, A. Blake: Condensation: conditional density propagation for visual tracking. International Journal of Computer Vision, 29(1), 5-28 (1998) 4. D. Comaniciu, V. Ramesh, and P. Meer: Kernel-based object tracking. IEEE Trans. Pattern Anal. Mach. Intell., 25(5), (2003) 5. C. Rasmussen, G.D. Hager: Probabilistic Data Association Methods for Tracking Complex Visual Objects. IEEE Trans. Pattern Analysis Machine Intell., 23(6), (2001) 6. G.D. Hager, P.N. Hager: Efficient region tracking with parametric models of geometry and illumination. IEEE Trans. Pattern Analysis Machine Intell., 20(10), (1998) 7. A. Doucet, N. de Freitas, and N. Gordon, editors: Sequential Monte Carlo Methods in Practice. Springer-Verlag, New York (2001) 8. M.Isard, A.Blake: Contour tracking by stochastic propagation of conditional density. In: Proc. European Conf. on Computer Vision, 1, (1996) 9. F. Leymarie, M. Levine: Tracking deformable objects in the plane using an active contour model. IEEE Trans. Pattern Analysis Machine Intell., 15(6), (1993) 10. R. Carmi, L. Itti: Visual causes versus correlates of attentional selection in dynamic scenes. Vision Research, 46(26), (2006) 11. L. Itti, C. Koch, E. Niebur: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Analysis Machine Intell., 20(11): (1998) 12. P. F. Felzenszwalb, D. F. Huttenlocher: Efficient Graph-Based Image Segmentation. International Journal of Computer Vision, 59(2), (2004) 13. S.M. Smith, J.M.Brady: ASSET-2: Real-Time Motion Segmentation and Shape Tracking. IEEE Trans. Pattern Analysis Machine Intell., 17(8), (1995) 14. J. Y. Bouguet: Pyramidal Implementation of the Lucas-Kanade Feature Tracker. Tech. Rep., Intel Corporation, Microprocessor Research Labs (1999) 15. Robert T. Collins and Yanxi Liu: On-Line Selection of Discriminative Tracking Features. In: Proc. IEEE Conf. on Computer Vision (2003) 16. D.Martin,C.Fowlkes,D.Tal,J.Malik:ADatabaseofHumanSegmentedNatural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. In: Proc. IEEE Conf. on Computer Vision (2001)

11 Fig. 4. The results of multifold objects tracking. Fig. 5. The results of tracking under illumination changes and occlusion. The top row are the results of our approach. The bottom row are the results of meanshift Fig. 6. The results of tracking while the shape of object changes. Fig. 7. The results of attention rebuilding.