Multi-Object Tracking Using Color, Texture and Motion

Transcription

1 Multi-Object Tracking Using Color, Texture and Motion Valtteri Takala and Matti Pietikäinen Machine Vision Group Infotech Oulu and Dept. of Electrical and Information Engineering P.O. Box 4500 FIN University of Oulu, Finland Abstract In this paper, we introduce a novel real-time tracker based on color, texture and motion information. RGB color histogram and correlogram (autocorrelogram) are exploited as color cues and texture properties are represented by local binary patterns (LBP). Object s motion is taken into account through location and trajectory. After extraction, these features are used to build a unifying distance measure. The measure is utilized in tracking and in the classification event, in which an object is leaving a group. The initial object detection is done by a texture-based background subtraction algorithm. The experiments on indoor and outdoor surveillance videos show that a unified system works better than the versions based on single features. It also copes well with low illumination conditions and low frame rates which are common in large scale surveillance systems. 1. Introduction One of the most useful and valuable applications for current vision technology is visual surveillance. It is feasible to construct and able to generate direct benefits for its utilizers. It can be used for many purposes in a number of different environments: people and vehicle tracking in traffic scenes [6], proximity detection in battlefields [17] (original study: [14]), suspicious event detection [12] ([5]), and geriatric care, to name but a few. This has been already noticed, and the research is in full flow both in academic and industrial sides. Tracking is the primary part in active visual surveillance where human intervention is to be minimized. It is also a field of numerous methods for numerous different tracking cases: Mean Shift based algorithms [1], [3], [4], [7] can be used in single object problems with both static and moving cameras. Multi-object tracking, however, is a very different and more challenging problem. In addition to the normal frame-to-frame following of a salient area, the system must be able to handle occlusion, splitting, merging and other complicated events related to multiple moving targets. Existing solutions [6], [8], [9], [14] are meant for static cameras and limited types of scenes. Lately, particle filters [13] have gained a great deal of attention [8], [16], [19], [20], [25]. To survive in diverse environments, one should take advantage of multiple image properties, like color, texture, temporal, etc., as none of them alone provides all-around invariance to different imaging conditions. By using a versatile collection of properties the system performance can be enhanced and made more robust against the large variation of data common in surveillance. Still, one has to be careful while choosing multiple features as they may also have a negative effect on each other. Color correlogram [11] (also known as autocorrelogram) and Local Binary Patterns (LBP) [18] are well developed descriptors for general image classification and retrieval. They are fast to extract and provide features with varying size and discrimination power, depending on the used parameters (kernel radius, color channel quantization, sample count, etc.). They offer good histogram-based descriptions for object detection and matching. In low frame rates (< 10 fps), some sort of model-based matching is a worthwhile way to go as the spatial correspondence of objects in adjacent frames is often low. Frame rates like 2 fps are common in large scale surveillance systems where the amount of collected data can be tremendous. One server may have to handle a data stream of tens of video sources. In such situations the load on the system s IO is significant and frame rates like 25 fps per video source are not easily feasible. There is a good likelihood that the situation stays the same in the near future as the camera technology advances rapidly and megapixel-class video frames are already reality. This paper introduces an approach which uses multiple image features for frame-to-frame correspondence matching. RGB color histogram and correlogram are used to /07/$ IEEE

2 describe the object s color properties, LBP is chosen for the texture, and geometric location and the smoothness of trajectory provide the motional support. The merging and splitting of objects are handled using the same set of features. The tracker s performance on low frame rate video is emphasized, as it is the area which has not been considered often enough. 2. Features for Tracking Matching-based tracking requires good feature descriptors to be usable in the diverse conditions of real-world video surveillance. The main sources of descriptors are color, texture, shape, and temporal (motion) properties. Each of them has its pros and cons but the color has gained the most of attention as it is well distinguishable to human eye and seems to contain a good amount of useful information Color Color provides many cues. The most well known color descriptor is the RGB color histogram [23] which has been used for tracking in various occasions [1], [3], [26]. There are also other potential features like the color moments [22], MPEG-7 color descriptors [15], and color correlograms [11], to mention but a few. In this study, the last one was selected together with the RGB color histogram to describe the color properties of objects. Selecting the color correlogram was natural due to its good discrimination power [11]. The main advantage of the correlogram is that it pays attention to the local spatial correlation of color pixels, thus increasing the value of color as the color histogram is purely a global measure Texture It would be shortsightedness to rely on color properties only. For instance, colors are very sensitive to illumination changes. This trouble can be alleviated, to some extent, by using other features that are less responsive to such image transformations. Texture, which has not enjoyed major attention in tracking applications, provides a good option to enhance the power of color descriptors. The list of available texture features is quite a long one but a good survey to different approaches has been made by Tuceryan and Jain [24]. As being one of the most efficient texture descriptors, the LBP texture measure [18] is a logical choice for describing object s textural properties. We selected a modified version of it [10] which is more stable against noise. LBP s main characteristics are invariance to monotonic changes in grayscale and fast computation, and it has proven performance background in texture classification [18]. While operating in gray-scale color space, LBP is also robust to illumination changes common in surveillance videos Motion In addition to visually descriptive features, video provides temporal properties. One can add, of course, a temporal dimension to any feature described above by combining the information extracted at consecutive instants of time in some meaningful way. We decided to use the object s location and trajectory to describe its motional properties. In addition to the close positions in successive frames, physical objects tend to have smooth trajectories, at least when the frame rate is high enough, and this can be exploited. We can calculate the smoothness of direction and speed [21] for each existing object track i: ( ) vi,t 1 v i,t S i,t = w +(1 w) v i,t 1 v i,t ( ) 2 vi,t 1 v i,t, v i,t 1 + v i,t (1) where the first term defines the smoothness of direction and the second one the smoothness of speed. S i,t is the combined smoothness of the track i between the time instants t and t 1. v stands for the difference vector of two points and w is a weight. If m points are extracted from n frames, the total smoothness is defined by Equation (2), which is the sum of smoothnesses of all the interior points of all the m paths. 3. Tracker T s = m n 1 S i,t (2) i=1 t=2 Our tracker (see Figure 1) consists of two main elements: background subtraction (detection) and tracking. The subtraction on the video data, that is first processed with a Gaussian filter to remove noise, is done by an adaptive algorithm which is based on LBP texture distributions [10]. The algorithm was chosen because of its good performance in most environments and the fact that it exploits the same texture properties as the object matching part of the tracker, so the re-use of features is possible. The subtracted foreground is enhanced by filtering the artifacts caused by noise and moving background using standard morphological operations. All the remaining foreground areas are considered as possible object candidates and filtered according to the needs: the size of object depends heavily on the surveillance scene. The tracking is done by matching features extracted from the subtracted foreground shapes. As told in Section 2, three of these are based on histogram distributions: RGB color histogram [23] and correlogram [11], and LBP [18]. The

3 Detection Tracking Frames Texture-based background subtraction Object detection using contours Matching using color, texture and motion features Tracked objects Group handling (merging and splitting) Figure 1. The tracker. Same features (color, texture and motion) are used in initial matching and group handling. reason for choosing two color-based descriptors is the difference in their spatial performance. While the histogram is a global measure and thus invariant to many local attributes like scale, the correlogram takes into account spatial color distributions and has better discrimination performance on coherent data. The spatial properties are usually well preserved in tracking as the objects do not change much between successive frames, depending, of course, on the frame rate. The LBP texture measure was selected due to its qualified performance as stated in Section 2.2. It supports the color features well in natural scenes as those often contain a lot of textural information. It also provides better discrimination capabilities in many situations where a simple color descriptor may fail, for example in low lighting conditions. The other two cues for matching, the geometric distance and the combined smoothness of speed and direction, were included to emphasize the importance of motion on tracking. The tracker uses similar structure as Yang et al. s system [26] in which tracking is based on distance and correspondence matrices and object occlusion is managed through the detection of splitting and merging events. First a distance matrix with new measures as columns and existing tracks as rows is built. This is followed by creating a zero-initialized correspondence matrix in which the best matching track for each measure is marked by incrementing the corresponding matrix element by one. The same is done for each track after which the correspondence matrix elements with value of two are considered as definite matches. Unmatched measures and tracks are considered as possible candidates for merging and splitting. The complete details of the logics are described in the Yang et al. s paper [26]. In our system, a group of diverse features is employed for measure-to-track matching and the events are detected differently. Instead of using the bounding boxes themselves for occlusion detection, we surround the boxes with circles that have the radii of the half diagonals of the boxes as in the Figure 2 and use them for event detection: If the object circles are occluding each other in the previous frame n 1 a merging event in frame n is possible. If the occlusion is true in frame n + 1 and the closest occluding object is Figure 2. Diagonal occlusion. The merging event in frame n is detected by analyzing the situation in the previous frame n 1. The splitting detection is done in frame n + 1. frame rate ffps Radius Weight vs. Frame Rate radius weight w r Figure 3. Weight selection. The weight of event detection circle radius is selected according to the frame rate. The weights of 2 and 25 fps frame rates are displayed with small circles. a group object 1(1&2) then a split might have happened. The circle s radius can be weighted according to the frame rate to cover a smaller or larger detection area. The radius weight w r is inversely related to the frame rate f fps and can be estimated with a hyperbola function: from which we get f fps = α wr 2 + β, (3) α w r = f fps β, (4) where α and β are constants. Figure 3 shows an example curve where α = 15.3 and β = 5.7. The matching itself is carried out by using an overall distance obtained from several descriptors. It is done both in the initial correspondence matching process and in the splitting event, in which one has to recognize the object that is leaving its group. After filtering out the least probable candidates with a separate geometric distance threshold, five descriptors are used: RGB color histogram and correlogram, LBP, geometric distance and smoothness. The first three are applied on the upper and lower half of the object separately. This is done to add more spatial discrimination power to situations where the interesting objects are people wearing two distinctive pieces of clothing, like a shirt

4 and trousers. It should be also noticed that the distributionbased features are extracted from the foreground only. The distance matrix D(i, j) of the initial matching phase is constructed as follows. Each matrix column j corresponding to a measure is filled with a distance vector d j of M elements (tracks). The vector elements are sums of five descriptor measures that have been normalized to the range [0, 1] prior to final summing to the vector: d j = 5 d i,f, i = 1, 2,..., M, (5) f=1 where d i,f is the normalized distance of the feature f of the ith element. The feature-specific normalization is done between the original distance values ˆd i,f as d i,f = ˆd i,f M i=1 ˆd i,f. (6) The smoothness is obtained by adding the center point of the measure to the track s trajectory and then calculating a new smoothness value. Note: if the candidate tracks have long common history (period of occlusion), there is no difference in smoothness as the active trajectories, from which the feature is extracted, have become identical. The actual smoothness value T s, as not being a distance measure converging to zero, is first shifted by its bias, which depends on the length of the trajectory, and then inverted to make it comparable with the other descriptors: d s = 1 T s T s, (7) where d s is the final measure nearing to zero. T s is the bias value which has been rounded down to the closest smaller integer value. The geometric distance is used in the same way as the distribution-based descriptors. After calculating the initial geometric distances between a measure and its candidate tracks, the obtained vector of distance values is normalized to one. Before distance measurement, the track s color and texture descriptors, all consisting of histogram representations, are updated by filtering last N histograms with Gaussian weights: N h updated i = h i,t w G (t), (8) t=1 where h updated i is the updated value of bin i and h i,t the corresponding original bin value, in which t is the index in temporal dimension, 1 referring to the latest bin value and N to the oldest in history. The weights w G are obtained using a standard Gaussian distribution (µ = 0, σ 2 = 1): w init (t) = 1 2π e t2 2, (9) which is then normalized to one to get w G. The histogram distances are compared by an Euclidean distance measure: D eucl (x 1, x 2 ) = N (x 1,i x 2,i ) 2, (10) i=1 where i is the corresponding bin of the histogram x of length N. 4. Experiments Our system consist of a software framework operating on standard Intel Pentium 4 (3GHz) PC hardware with 1 GB of memory. The details of test video sequences are collected in Table 1. The first two sequences (Indoor and Outdoor) are from real surveillance scenes (camera type unknown) and their original frame rate is 2 fps. The Merge & split video has been created using a Sony DFW-VL500 camera. The Near-IR dataset was made with Axis 213 PTZ Network Camera which has a built-in IR source and is capable of sensing near infrared light. The last two video sets belong to the well known CAVIAR [2] and PETS 2001 databases. The CAVIAR sequence had an original frame rate of 25 fps which was decreased to 1 fps for testing purposes. The background subtraction parameters include the radius and sample count of the LBP operator, the radius of a circular region around a pixel over which the LBP histogram is calculated (R region ), the number of LBP histograms per pixel (K), the thresholds of histogram proximity measure (T P ) and background histogram selection (T B ), and the learning rates for model histogram updating (α b, α w ). The selected parameter values are collected in Table 2. Guidance for selecting them properly can be found in the original paper [10]. Both our background subtraction and tracking implementations use the modified version of LBP with the thresholding constant a set to 3, as suggested by the original study. To speed up the subtraction, we apply it to every third pixel in horizontal and vertical directions and use that value as an approximation of the pixel s circular surroundings. Prior to the subtraction, the video frame is processed with a Gaussian 3x3 kernel which has a standard deviation σ of Afterwards, the small artifacts in the subtracted scene are morphologically filtered by doing one close and open operation with a circular kernel of 3 pixels in diameter. The RGB color histogram has 216 bins (6x6x6) and the RGB color correlogram is created from two distances (1 and 3) for 64 colors (4x4x4) making up a 128-bin descriptor in total. The LBP feature is calculated from a circular neighborhood of eight samples with radius of two, thus the length of the LBP histogram becomes 256 bins. In the histogram filtering process, as described by Equations (8) and (9), we

5 Test sequence Size Fps Length (s) Indoor 352x Outdoor 352x Merge & split 320x Near-IR 352x CAVIAR 384x PETS x Table 1. Test videos and their sizes, frame rates and lengths. Examples of tracking in each video are included in the supporting material. Parameter Value LBP radius 2 LBP sample count 6 R region 9 K 3 T P 0.6 T B 0.65 α b 0.01 α w 0.01 Table 2. Background subtraction parameters. use 5 for N. In the smoothness extraction, the active trajectory is taken from the last three seconds and the direction and speed are equally weighted. The other tracker parameters are chosen as follows: The interesting contours for object detection are thresholded by minimum size that depends on the scene. In short distances, when the objects tend to be large, we use a foreground/background ratio of For long distance surveillance data, the size filtering is done by a ratio of The geometric distance threshold for matching, as introduced in Section 3, is selected according to frame size. We use values equaling to times the diagonal of the frame. For smaller frame rates or very large objects even greater values could be used as the locations of the targets change more rapidly. The radius of the merging and splitting detection circle is weighted using Equation (4) with α = 15.3 and β = 5.7. α and β were obtained through experiments with different frame rates (1-30 fps). Figure 4 presents our system s performance in indoor and outdoor scenes at very low frame rate (2 fps). Tracking is successful through the heavy occlusion events of the indoor dataset, and it is not easily disturbed by the moving background elements, like moving trees and reflections from the asphalt. To demonstrate the performance in even lower frame rates, we took a sequence from CAVIAR database and lowered its frame rate from 25 to 1 fps. Figure 5 shows the result where object identities are maintained through a fighting scene containing partial occlusion. Figure 4. Indoor and Outdoor sequences. The tracker survives in situations where the frame rate is very low (2 fps), heavy occlusion exists and the background contains a lot of movement (moving trees, reflections). Figure 5. CAVIAR. A low frame rate do not cause major problems to the tracker. Feature(s) Precision All features 87% Color histogram 73% Color correlogram 73% LBP 60% Geometric distance 60% Smoothness 40% Table 3. Splitting performance. The overall performance of different features in the critical object splitting event is shown in Table 3. Each number is the average percentage of right splitting decisions (15 in total) on six test videos of Table 1. The performance of combined features is clearly better than that of the others. Color histogram and correlogram are equal in average but their performance deviate on different videos. LBP has clearly lower recognition rate which may have been affected by the Gaussian filtering of the preprocessing phase. The filtering process affects the structural patterns that are considered as textures by the LBP operator. The poor performance of smoothness feature may indicate long occlusion times of objects in the test datasets. If the period of occlusion is considerably longer than in normal passing and the objects move in a uniform manner the reliability of the trajectory measure will decrease together with its impact on the final classification decision. Figure 6 contains a few example frames from the multifeature tracking in Merge & split video together with corre-

6 Figure 6. Merge & split. The upper row contains the results of the background subtraction process and the lower shows the actual tracking. Figure 7. PETS The system tracks multiple people through several successive occlusion events. sponding background subtraction steps. The dependency on the texture-based background subtraction is especially high in this sequence as there are substantial color similarities between one of the foreground objects and the background. Figure 7 shows tracking results on a PETS 2001 dataset in which three people enter the scene and become occluded by each other several times. Our system has some problems in the beginning of the sequence, as the persons enter the room in parallel, but it is able to maintain the number of people and their identities most of the time after they have been initially discovered. The system was also tested in low light conditions. In the sequence of Figure 8 the persons were tracked correctly even though no color information was available. In the left image, the person on the left has a turquoise shirt while the other one has white. Both look very similar in noisy grayscale imagery. 5. Conclusions In this study, we have introduced a novel tracker based on the combined use of color, texture and motion features, and texture-based background subtraction. The system is able to track multiple objects in diverse conditions while achieving speeds of fps on a 3 GHz Intel Pentium 4 computer. It is also less sensitive to color due to the use of a versatile collection of cues. The system shows robust performance while most of the parameters are fixed. Future research will concentrate on a different weighting scheme in which the illumination conditions and other ef- Figure 8. Near-IR. The system works also in noisy near-infrared conditions where color information is limited to gray-scale. The background subtraction results are shown in the upper row. fectors are taken into account adaptively. The current way of using static feature weighting prevents the cues from dying but in some cases some of the features become useless and may even decrease the probability of correct classification. For example, after a very long period of occlusion the smoothness of trajectory has no use for classification. The long-term accuracy on people tracking could be improved by confirming the number of people in a group with an additional detector of human shapes or faces. To include cars and other interesting objects of arbitrary shape, a more versatile classifier would be needed. 6. Acknowledgment This research was supported by the Infotech Oulu Graduate School and the Finnish Funding Agency for Technology and Innovation (Tekes). References [1] G. R. Bradski. Computer video face tracking for use in a perceptual user interface. Intel Technology Journal, 2, [2] CAVIAR project: benchmark datasets for video surveillance [3] R. Collins, Y. Liu, and M. Leordeanu. Online selection of discriminative tracking features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27: , [4] D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24: , [5] D. Gibbins, G. N. Newsam, and M. J. Brooks. Detecting suspicious background changes in video surveillance of busy scenes. In Proceedings of the Third IEEE Workshop on Applications of Computer Vision, (WACV 96), Sarasota, Florida, USA, 2-4 December 1996, pages 22 26, [6] A. Hampapur, L. Brown, J. Connell, A. Ekin, N. Haas, M. Lu, H. Merkl, S. Pankanti, A. Senior, C.-F. Shu, and Y. L. Tian. Smart video surveillance: Exploring the concept of

7 multiscale spatiotemporal tracking. IEEE Signal Processing Magazine, 22:38 51, [7] B. Han and L. Davis. Object tracking by adaptive feature extraction. In Proceedings of International Conference on Image Processing, (ICIP 2004), Singapore, 27 June-2 July 2004, pages , [8] B. Han, Y. Zhu, D. Comaniciu, and L. Davis. Kernel-based Bayesian filtering for object tracking. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (CVPR 2005), San Diego, California, USA, June 2005, pages , [9] I. Haritaoglu, D. Hardwood, and L. S. Davis. W4: Real-time surveillance of people and their activities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22: , [10] M. Heikkilä and M. Pietikäinen. A texture-based method for modeling the background and detecting moving objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28: , [11] J. Huang, S. R. Kumar, M. Mitra, W.-J. Zhu, and R. Zabih. Image indexing using color correlograms. In Proceedings of the 1997 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (CVPR 1997), San Juan, Puerto Rico, June 1997, pages , [12] iomniscient [13] M. Isard and A. Blake. CONDENSATION - conditional density propagation for visual tracking. International Journal of Computer Vision, 29:5 28, [14] A. J. Lipton, H. Fujiyoshi, and R. S. Patil. Moving target classification and tracking from real-time video. In Proceedings of the Fourth IEEE Workshop on Applications of Computer Vision, (WACV 98), Princeton, New Jersey, USA, October 1998, pages 8 14, [15] B. S. Manjunath, J.-R. Ohm, V. Vasudevan, and A. Yamada. Color and texture descriptors. IEEE Transactions on Circuits and Systems for Video Technology, 11: , [16] K. Nummiaro, E. Koller-Meier, and L. V. Gool. An adaptive color-based particle filter. Image and Vision Computing, 21:99 110, [17] ObjectVideo [18] T. Ojala, M. Pietikäinen, and T. Mäenpää. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24: , [19] P. Pérez, C. Hue, J. Vermaak, and M. Gangnet. Color-based probabilistic tracking. In Proceedings of the 7th European Conference on Computer Vision (ECCV 2002), Copenhagen, Denmark, May 2002, pages , [20] V. Philomin, R. Duraiswami, and L. Davis. Quasi-random sampling for Condensation. In Proceedings of the 6th European Conference on Computer Vision (ECCV 2000), Dublin, Ireland, 26 June - 1 July 2000, pages , [21] L. G. Shapiro and G. Stockman. Computer Vision, page 267. Prentice Hall, first edition, [22] M. Stricker and M. Orengo. Similarity of color images. In Proceedings of the SPIE Conference on Storage and Retrieval for Image and Video Databases, San Jose, California, USA, 9 February 1995, pages , [23] M. Swain and D. Ballard. Color indexing. In Proceedings of the Third IEEE International Conference on Computer Vision, (ICCV 1990), Osaka, Japan, December 1990, pages 11 32, [24] M. Tuceryan and A. Jain. Texture analysis. In C. Chen, L. Pau, and P. Wang, editors, Handbook of Pattern Recognition and Computer Vision, pages World Scientific, second edition, [25] C. Yang, R. Duraiswami, and L. Davis. Fast multiple object tracking via a hierarchical particle filter. In The Proceedings of the Tenth IEEE International Conference on Computer Vision, (ICCV 2005), Beijing, China, October 2005, pages , [26] T. Yang, S. Z. Li, Q. Pan, and J. Li. Real-time multiple objects tracking with occlusion handling in dynamic scenes. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (CVPR 2005), San Diego, California, USA, June 2005, pages , 2005.