Combining Multiple Tracking Modalities for Vehicle Tracking in Traffic Intersections

Transcription

1 Combining Multiple Tracking Modalities for Vehicle Tracking in Traffic Intersections Harini Veeraraghavan Nikolaos Papanikolopoulos Artificial Intelligence, Vision and Robotics Lab Department of Computer Science and Engineering University of Minnesota Abstract This paper presents a camera-based system for tracking vehicles in outdoor scenes such as traffic intersections. Two different tracking systems, namely, a blob tracker and a Mean Shift tracker provide the position of each target. These results are then fused sequentially using an Extended Kalman filter. The tracking reliability of the blob tracker is improved by using oriented bounding boxes (which provide a much tighter fit than axis aligned boxes) to represent the blobs and a Joint Probabilistic Data Association filter for dealing with data association ambiguity. The Mean Shift tracker is as proposed by Comaniciu et al. [3]. We show that the above tracking formulation can provide reasonable tracking despite the stop-and-go motion of vehicles and clutter in traffic intersections. Keywords Blob tracking, Mean Shift tracking, Joint Probabilistic Data Association filter, stop-and-go traffic. 1 Introduction The main requirement for any vision-based outdoor tracking system is robustness to adapt to the variabilities in the visual data presented by the environment. Changes in viewpoint, occlusions, illumination changes, different postures of target constantly modify the information presented to the sensing system. In the context of robotics, consider the example of landmark based navigation in outdoors. In such cases, reliance only on color or edges or shape of landmark for example, would not be sufficient for localization. The goal of our project is to monitor activities at traffic intersections. The tracking system has to deal with the usual problems of uncontrolled outdoor scenes such as varying illumination, shadows, clutter etc., but also with the nonfree flowing traffic and increased congestion at the intersections. Most adaptive background learning schemes use the author to whom all correspondence should be sent. assumption that background pixels generally are non varying. However, one problem with this is that, stopped foreground targets will be modeled into the background. In the context of intersection monitoring, this presents a problem as vehicles and pedestrians stop regularly at intersections. Ignoring such vehicles not only results in tracking failure but also affects the performance of the incident monitoring system. Further, increased congestion at intersections results in increased target-data association complexity. Input Image Image Segmentation Blob Tracking Mean Shift Tracking Incident Detection and Visualization Connected Region Extraction Figure 1: Tracking approach. Tracking systems that make use of the image content such as target features templates and color distributions are not affected by ineffective segmentation. However, changes in pose of target or change in viewpoint, might alter the color distribution. Similarly, template matching methods are generally affected by target rotation. Focus of attention or identifying regions of interest is important in several robotic applications. Most attention systems work by looking for specific predefined features such as templates, dark spots in images depending on the application. In our case, targets of interest are vehicles and pedestrians. These generally appear in the scene and are in constant motion (other than stopping at intersections). Hence, they can be easily segmented out using a segmentation algorithm. In this paper, we address the problem of target tracking 1

2 in outdoor environments by making use of the following cues: foreground regions (obtained from image segmentation) and the individual target s color distribution. Foreground region segmentation is achieved by an adaptive background segmentation scheme based on the mixture of Gaussian models proposed by Stauffer and Grimson [11]. Target color distributions (represented by color histograms) are initialized automatically on the detected foreground target regions. Initialized targets are tracked across frames using extended Kalman filter. Predicted target positions are used as starting guesses for searching for targets in the next frame. The results from the tracking module are presented to the incident detection module. The workflow of the system is presented in Figure 1. Multiple cue based tracking has been used frequently in outdoor tracking. Hong et al. [4] combined Ladar with color camera mounted on unmanned vehicle for navigating in an outdoor environment (avoiding obstacles such as puddles, finding road etc.,). In the context of vehicle tracking, Malik et al. [5] used a tracker based on two linear Kalman filters, one for estimating the position and the other for estimating the shape of the vehicles moving in highway scenes. Similar to this approach, Meyer et al. [7] combined a motion filter for estimating the affine parameters of an object for position estimation with a geometric Kalman filter for shape estimation. Cues such as motion derived from optical flow, shading and edges have been recently used by Lu et al. [6] for tracking hands. The cues are integrated using a model-based approach. Sidenbladh and Black [9] used cues such as edges, ridges, and motion cues to learn probabilistic models of people s motion. Rasmussen and Hager [8] use different cues obtained from homogeneous regions, snakes, and textures for tracking targets under occlusion using constrained Joint Probabilistic Data Association filters. Soto and Khosla et al. [10] combined cues from multiple visual cues dynamically in an agent framework for human tracking. The novelty of our approach is that we use two different tracking modalities that essentially provide the same information, namely, the target s position. Instead of using the most dominant or promising cue at a time, we make use of the individual position information obtained from both the cues. Further, instead of representing the individual information as a full measurement vector, the position measurements are treated as arising from two independant sensing modalities and hence they are fused sequentially in a Kalman filter. The blob tracking is made more robust by using a Joint Probabilistic Data Association filter. This paper is arranged as follows. Section 2 discusses the theory behind cue fusion. Section 3 discusses target detection and tracking initialization methodology. The blob tracking method is discussed in Section 4 while the Mean Shift Tracking method is briefly discussed in Section 5. Results and their discussion are in Section 7 and conclusions are drawn in Section 8. 2 Background At time t, when n measurements are available from n different sensors, the standard method for updating the state in a Kalman filter is to represent the entire measurement set as the measurement vector and update them all simultaneously. However, when the measurements are uncorrelated, that is their individual noises w i (t), i = 1,..., n are uncorrelated, R(t) = E[w 1 (t)...w n (t)] = diag[r 1 (t),..., r n (t)] (1) then one can carry out the update step sequentially, incorporating the measurements one after the other as explained by BarShalom et al.in [2]. This is because, the measurement noises are still white (due to the uncorrelatedness) for use in the Kalman filter. Further, note that, uncorrelatedness in measurement errors means that the sequence of application of the individual measurements does not matter. In other words, the measurements from the individual sensors can be applied in any order. In our case, position measurement for each target is obtained from two different tracking modalities, namely, the region or blob tracker and the Mean Shift Tracker. The two different trackers operate independently of each other and the noise in the target localization from the two trackers is independent. In other words, the target localization accuracy in the region tracker depends on the accuracy of the segmentation algorithm and the presence or absence of multiple measurements arising from target or background occlusions. The noise in the target localization in the mean shift tracker arises primarily due to the target getting occluded, and some changes in its color distribution that might occur due to pose change. Hence, the position measurements from the two trackers can be applied sequentially to the Kalman filter. Targets are modeled as first-order translational models in 2D. In other words, the targets are assumed to move with constant velocities. This doesn t hold true always as vehicles stop at intersections and decelerate before turning. However, in our case as the vehicles are generally slow moving at the intersections, the acceleration is not very large and hence a simpler model can still be used. The vehicles are tracked in scene coordinates. Hence, extended Kalman filters are used for tracking due to the nonlinear mapping of the measurements from the image space to the target state space. 2

3 3 Target Detection An adaptive region segmentation method is used for detecting targets of interest. Our region segmentation method is based on the mixture of Gaussian models approach as proposed by Stauffer and Grimson [11]. Each pixel has a mixture of (in our case 6) Gaussian distributions associated with it and is represented by its sufficient statistics and a weight ω indicative of its frequency of occurrence. This method classifies a pixel on the basis that non-varying pixels (those having a lower variance and a larger weight) are background while others are foreground. Foreground regions are extracted using connected components extraction. In order to prevent false detection, only regions that can be tracked successively in the first few frames are classified as targets. Each blob is initialized as a Moving Object (MO) which is then tracked in subsequent frames using an extended Kalman filter as described in Section 4 and Section 5. A MO is just an abstraction of the blobs and can have more than one blob associated with it in each frame. This is to account for the fact that, blobs of vehicles and pedestrians can split and merge due to occlusions. Every MO is initialized with the blob s position and velocity (computed from tracking over the past few frames), and its color distribution. The color distribution is computed as a histogram (consisting of 16 bins) of the blob region bounded by its axis aligned bounding box. It is necessary to include only regions belonging to the target in the histogram computation for good localization. As it is difficult to delineate the target correctly with axis aligned bounding boxes, only half the bounding box size around the center is used for computing the histogram. Furthermore, the histograms are weighted relatively to the surrounding background which increases the relevance of features distinct from the background. The histogram weighting is based on the method described by Comaniciu et al. [3] and the weights are computed as w i = min( ˆB ˆB i, 1), i = 1...m (2) where, ˆBi, i = 1...m, is the discrete representation of the surrounding background s histogram and ˆB is the minimum of the histogram. The term w i represents the weighting of each bin and its effect is to reduce the importance of portions of foreground model similar to the background model. 4 Blob Tracking The blobs obtained from region segmentation are represented as oriented bounding boxes (obtained from principal component analysis of the blobs) which are then tracked from frame to frame. The reason for using oriented bounding boxes as opposed to conventional axis aligned boxes is owing to their tighter target representation which makes data association in case of multiple target tracking easier. These are illustrated in Figure 2. The tracked blobs are represented as Moving Objects (MOs) which are tracked using their respective extended Kalman filters. The relation between the blobs and the MOs are represented in a bipartite graph. The details of the graph construction and pruning are explained in our previous work [13]. (a) Figure 2: Oriented bounding boxes vs Axis aligned boxes fit. The oriented bounding boxes provide much closer fit to the vehicles than axis aligned boxes. 4.1 Joint Probabilistic Data Association Presenting reliable measurements to the filter is necessary to ensure reliable tracking of targets. Data association ambiguity in the context of region tracking for multiple targets arises due to the presence of occlusions from target-target and target-background occlusion. In general, data association ambiguity in scenes may arise due to: 1. noise-like visual occurrences, 2. persistent known scene elements (i.e., other tracked targets), and 3. persistent, unknown scene elements (occurring due to uninstantiated targets). In addition, occlusions can also render the target partially or totally unobservable. These problems manifest in our case as the number of blob associations a vehicle has in a frame. The Joint Probabilistic Data Association Filter (JPDAF) is a suboptimal Bayesian approach for tracking a known number of targets in clutter. The problem of tracking can be formulated as a problem of assigning a set of measurements m 1,..., m M to the known set of targets x 1,..., x N at each time instant t. This is evaluated from the sets of joint events (consisting of target measurement association pairs (x i, m j )) at time t. A joint event is formed with the constraint that no two targets can share the same measurments, and no more than one measurement can be associated with a target in one joint event. However, a different measurement can be associated (b) 3

4 with the same target in a different joint event at ths same time t. The target measuremnet association is then evaluated from marginalizing all the joint event pairs at time t. For details, interested readers may refer to [1]. In the traditional JPDAF framework, all measurements are assumed to be related to all targets. However, using a full target measurement association would make the event calculation very computationally expensive. Gating based methods using Mahalanobis distances are common to limit the number of target measurement associations. In our case, the results from blob tracking are used to limit the number of associations. In other words, only those blobs related to a given target through its child blob(s) from previous frame are used. 5 Mean Shift Tracking The method is based on target representation using a nonparametric isotropic kernel. Tracking involves target localization using a gradient descent based search procedure consisting of comparing the target model with the image. Detailed explanation of the method can be found in [3]. The target model is represented using m bin histograms. The model is normalized to eliminate the influence of target dimensions by independently rescaling the row and column dimensions. Thus, the target model is centered at 0. In each subsequent frame, the target candidates are computed and each compared with the target model using a similarity function ˆp = {ρ[ p(y)ˆq]} ˆ where, p(y) ˆ and ˆq are the target candidate and the target model respectively, and y is the target location. The target candidates are also normalized based on the target model. The function ˆp acts as a likelihood function so that its local maxima indicate the target s position in the current frame. The similarity function defines a distance between the target model and candidate as d(y) = 1 ρ[p(y), q] (3) where, ρ[p(y), q] is an estimate of the Bhattacharya coefficient where, ρ[p(y), q] = σ m u=1 pu (y)q u (4) The goal of the tracking algorithm is the minimization of the distance as a function of y which is equivalent to maximizing the Bhattacharya coefficient. In each frame, the model is searched starting from the predicted target position. In order to account for scale changes of the target as it translates in the image, its scaling factors along x and y are computed to adapt the kernel bandwidth in each frame. 6 Measurement Error The measurement error covariance R k for the filter is given by R ( k) = [ ] σk x σ k y 2 which represent the error in the target position measurement in x and y coordinates in the image. The measurement error standard deviations σ k x 2 and σ k y 2 for the blob tracker are obtained based on the variance in the percentage difference in the measured and previously measured size (area). In the case of the Mean Shift Tracker, the measurement uncertainty is computed as the standard deviation of a scaled Gaussian distribution fitted to three different points on the surface for each coordinate (x and y). The three points along each coordinate direction are chosen as the center of the localized target and two points on either side of the center at a distance equal to the dimension of the bounding box along the particular coordinate axis. 7 Results and Discussion Figure 3 shows tracking in an intersection scene. The vehicle number 34 is tracked consistently although it stops at for 650 frames. The sequence also shows how vehicles 34 and 44 are tracked consistently despite occlusion. Tracking under multiple occlusions is shown in Figure 4 where the vehicle 6 undergoes multiple occlusions. As shown, it is picked up even when it was occluded for several frames behind the truck. Figure 5 shows the tracking performance of just blob tracking and tracking using both region and mean shift tracking. Blob tracking in this case fails because the tracker jumps to the white vehicle from the black vehicle due to occlusion occurring between the two before a track was initialized for the white car. However, adding the Mean Shift Tracker allows to track the black vehicle reliably. Using two cues instead of one clearly improves the tracking performance in cases where one cue fails to provide good measurements as shown in Figure 5. Triesch and Malsburg [12] used five different cues such as color, shape, motion prediction, contrast, and motion detection for tracking faces. Each cue provides a quality metric indicative of the goodness of fit which is then used for integrating the cues. In our approach, the different modalities provide this quality metric in terms of the measurement error (provided to the Kalman filter) computed in each frame for each tracker. The tracking system provides reliable data for most targets but fails when reliable data cannot be obtained from either of the sources. This occurs when the target is occluded behind a structure or another target for example. This clearly suggests, that we need to make use of more information for achieving reliable tracking. The targets are initialized based (5) 4

5 (a) Frame 6531 (b) Frame 7189 (c) Frame 7468 Figure 3: Tracking sequence showing tracking of stop-and-go traffic. The sequence also illustrates occlusion handling between vehicles 34 and 44. (a) Frame 296 (b) Frame 346 (c) Frame 384 Figure 4: Tracking sequence under occlusions. Vehicle 6 is successfully tracked despite occlusions. on the results from the blob tracking. Hence, when two targets enter the scene together, they will be segmented as one blob and hence can be initialized as one target. This is illustrated in Figure 4, where vehicle 6 is actually two cars but gets initialized as one as they enter the scene together. Currently, there is no way to distinguish two targets in such cases. Another problem that must be obvious to the reader from some of the results shown in this paper is the one due to false targets. False targets are those blobs that are produced due to segmentation failure during illumination changes. These blobs are difficult to remove as they will be tracked consistently by the mean shift tracker. One of the ways they are removed currently is by aging the targets not tracked by blob tracking at all and removing them after a certain length of time. Currently, each tracking method is executed serially one after the other which slows down the execution speed. One scope for improvement will be to run the two tracking modalilities in parallel and feed the measurments based on time of arrival to the filter. 8 surement error) to an extended Kalman filter. We demonstrated how such a system can not only improve tracking reliability but can also can be employed successfully for traffic intersection scenes with vehicles exhibiting stop-and-go kind of motion. 9 Acknowledgements This work has been supported in part by the Minnesota Department of Transportation, the ITS institute at the University of Minnesota, and the National Science Foundation through grants #CMS and #IIS We would also like to thank Guillame Gasser for providing the code for mean shift tracking. References [1] Y. Bar-Shalom and T. E. Fortmann. Tracking and data association. Academic Press, Conclusions This paper presented a method for tracking targets in outdoor scenes using two different cues. Each cue is an independent tracking system which provides an estimate of the target s position with uncertainty (which is modeled as mea- [2] Y. Bar-Shalom, X. Rongli, and T. Kirubarajan. Estimation with applications to tracking and navigation. John-Wiley and Sons,

6 (a) Frame 1933 (b) Frame 2063 (c) Frame 2163 Figure 5: Tracking sequence showing performance of blob tracking and tracking using both blob and mean shift tracking. As can be seen, the blob tracker tracking the black car, jumps to track the white (uninitialized) car while it is successfully tracked by the tracker using both blob and Mean Shift tracking. [3] D. Comaniciu, V. Ramesh, and P. Meer. Kernel-based object tracking. In IEEE Trans. Pattern Analysis and Machine Intelligence, volume 25, [4] T. Hong, T. Chang, C. Rasmussen, and M. Shneier. Feature detection and tracking for mobile robots using a combination of ladar and color images. In IEEE International Conf. on Robotics and Automation, [5] D. Koller, J. Weber, and J. Malik. Robust multiple car tracking with occlusion reasoning. In ECCV (1), pages , [6] S. Lu, D. Metaxas, D. Samaras, and J. Oliensis. Using multiple cues for hand tracking and model refinement. In Proc. Computer Vision and Pattern Recognition Conf., pages , [11] C. Stauffer and W. E. L. Grimson. Adaptive background mixture models for real-time tracking. In Proc. Computer Vision and Pattern Recognition Conf. (CVPR 99), June [12] J. Triesch and C.von der Malsburg. Democratic integration: Self-organized integration of adaptive cues. Neural Computation, 13(9): , [13] H. Veeraraghavan, O. Masoud, and N. P. Papanikolopoulos. Vision-based monitoring of intersections. In IEEE Conf. on Intelligent Transportation Systems, pages 7 12, [7] F. G. Meyer and P. Bouthemy. Region-based tracking using affine motion models in long image sequences. In Computer Vision, Graphics and Image Processing: Image Processing, volume 60, pages , September [8] C. Rasmussen and G. Hager. Joint probabilistic techniques for tracking multi-part objects. In Proc. Computer Vision and Pattern Recognition Conf., pages 16 21, [9] H. Sidenbladh and M. Black. Learning image statistics for bayesian tracking. In Int. Conf. on Computer Vision, volume II, pages , [10] A. Soto and P. Khosla. Probabilistic adaptive agent based system for dynamic state estimation using multiple visual cues. In 10th International Symposium of Robotics Research (ISRR 2001). November 9th - 12th, 2001Lorne, Victoria, Australia, November