A Scale Adaptive Tracker Using Hybrid Color Histogram Matching Scheme

A Scale Adaptive Tracker Using Hybrid Color Histogram Matching Scheme Nikhil Naik, Sanmay Patil, Madhuri Joshi College of Engineering, Pune-411005, India naiknd06@extc.coep.org.in Abstract In this paper we propose a computationally efficient scale adaptive tracking method using a hybrid color histogram matching scheme. Firstly, we report an important property of the Chi-squared measure- It outperforms Bhattacharyya measure in the task of histogram matching from a few significantly similar multimodal histograms. Also, Bhattacharyya measure performs better while selecting matches from a varied dataset. We employ these results to develop a hybrid histogram matching procedure using the two measures. This method is used for a patch matching algorithm in real time tracking. We first calculate a color histogram of the target which is then compared with histograms of patches in the neighborhood in subsequent frames using this hybrid procedure to obtain the best match. We devise a systematic scale adaptive tracking method which is robust to rapid changes in the target size. It is also robust to partial occlusion of the target. Extensive experimental proof based on real life and test datasets is presented which demonstrates the excellent tracking accuracy achieved by our algorithm at real time. I. INTRODUCTION Human motion tracking is an active area of research in computer vision. It also has a wide range of applications including smart surveillance; perceptual user interface, activity analysis for security and commercial purposes, content based image storage and retrieval, video conferencing, classification and recognition from motion etc. Various challenges in this area include change in appearance, rapid object motion, changes in illumination, occlusion and clutter. The various tracking algorithms can be divided into 4 categories [1] - region based tracking, active contour based tracking, feature based tracking and blob based tracking. In this paper, we present a blob based tracking algorithm. In general, they are much faster than contour based algorithms and are especially useful when it is sufficient to identify the target by a simple rectangle or ellipse rather than with an exact shape. We report a previously unexplored property of Chi squared and Bhattacharyya histogram comparison measures while comparing multimodal histograms. It can have important applications in a number of computer vision areas. Using this property, we develop a hybrid histogram comparison method using both Bhattacharyya and Chisquared measures. We demonstrate its robust performance by designing an efficient scale adaptive tracker. A. Previous Work Color histogram based tracking in its most simple form has proved to be a computationally efficient and robust method of object tracking. Algorithms using color histograms as the main tracking tool include [2] which uses the histogram intersection technique for foreground extraction. In [3] the authors employ color and edge histograms which are compared using Chi-squared and Intersection histogram comparison measures. The Earth Movers Distance (EMD) of color histograms is computed for object tracking in [4]. A weighted combination of more than one feature histograms was employed for tracking in [5]. Euclidian distance based histogram comparison along with region corresponding was developed in [6]. However this blob based tracking model suffers from following problems. The histogram based models used to represent the object are usually vulnerable to clutter and occlusions. Also in real life scenarios, the target undergoes significant variations in scale. The blob based methods fall short in adapting to these changes. For instance, one of the important blob based tracking techniques is the mean shift tracking algorithm [7] proposed by Meer et al. It has become popular due to its simple implementation and speed. It uses the mean shift procedure, a statistical method of finding local extrema in the density distribution of a data set and the Bhattacharyya metric of histogram comparison. However it cannot handle scale variations and clutter. This is also the case in most of the above mentioned algorithms. Tracking methods described in [2, 3, 4, 6] cannot handle target scale changes. In contrast, we develop color histogram based algorithm which is able to track a target under scale changes while retaining the advantages of blob based trackers like speed and efficiency. B. Our contribution We develop an accurate histogram comparison scheme using combination of Bhattacharyya and Chi-squared similarity measures. We develop a systematic method for scale adaption so that the tracking algorithm can efficiently follow a target undergoing significant scale variation. The tracker can also handle partial occlusion. We demonstrate the excellent tracking accuracy of our algorithm using real life scenarios and challenging tracking sequences from the CAVIAR test dataset [8]. The rest of the paper is arranged as follows. In Section II we describe the hybrid histogram comparison method. In Section III, we describe the tracking algorithm in detail. In

Sections IV and V, implementation details and results are given. Finally in Section VI, some conclusions are drawn. II. A HYBRID METHOD FOR HISTOGRAM COMPARISON A. Calculating Color Histograms Histograms can be simply defined as collected counts of the underlying data organized into a set of predefined bins. They can be populated by counts of features computed from the data. In our case the feature is color. The task of calculating the color density function of an image patch is formulated as follows. Let feature v represent the color of the object. The probability of color of a template patch is modeled by its histogram. Let the candidate patches centered at y (y=1,2,,n) be modeled by their color histograms. Now the task is to find the discrete location y whose histogram, is the most similar to the template histogram. B. Histogram Similarity Measures There are various statistical measures to calculate the similarity between histograms. An excellent taxonomy can be found here [9]. Two of the widely used measures are the Chi-squared measure ( ) and Bhattacharyya measure. The Chi-squared measure, denoted here by is given by - (y)= (1) The Chi-squared measure is an unbounded measure in that its maxima (which represents maximum mismatch) does not have a fixed value. Maximum match yields a value of 0. The discrete form of Bhattacharyya coefficient is - P[, ]= (2) Its metric form (proposed in [7]), denoted by is given by (y)= 1, (3) In this case, complete mismatch yields a value of 1 while maximum match yields 0. C. Relation between Bhattacharyya and Measures A similarity measure mathematically determines the shortest distance between two point observations in a high dimensional space. The Chi-squared statistic assumes that the Euclidian distance between the two points is the shortest distance. However for two distant observation points, the shortest distance is not a straight line but a curved path [10]. In contrast, the Bhattacharyya metric measures similarity in a domain where all errors are forced to constant, ensuring that the shortest distance between two points is indeed Euclidian. However due to this property, Bhattacharyya metric approximates the Chi-squared for small distances. The mathematical proof of some of above properties is presented in the appendix. A detailed mathematical analysis can be found in [10]. For a general purpose, Bhattacharyya measure has been found to be more accurate than Chisquared in [10] and a number of other works after that. However here we compare these measures while working with multimodal histograms, which has not been analyzed before. We demonstrate that this general behavior does not hold true in this specialized case due to the properties explained above. D. Effect on Multimodal Histogram Comparison We analyzed how these properties affect the performance of the two measures while comparing different types of histograms. We produce following major observations 1) We compared a large number of candidate patches with template patch with single dominant color. In this case, the Bhattacharyya metric slightly outperforms. 2) While comparing candidate patches with combination of multiple colors i.e. a complex color profile with template patch with a complex color profile, the Bhattacharyya measure loses its accuracy. The actual patch with most similar histogram (for example a patch created by adding some pixels with slightly different color shade) usually has the second or third best score rather than first. The candidate patch with best score is not the actual match. 3) Chi-squared measure cannot accurately select the best match for a multimodal histogram from a large dataset with all types of histograms. However it can select the best match more accurately than Bhattacharyya measure while obtaining the correct match from the best three histograms according to Bhattacharyya measure. These phenomena can be explained as follows. In case of 2), a patch with combination of multiple colors has a multimodal histogram. Bhattacharyya metric approximates the Chi-squared metric for small distances, as in multimodal histograms. So while differentiating among very similar histograms, in this case the best three candidates, it loses its accuracy. On the other hand while selecting a match over large distances as in case of unimodal histograms in 1), Chisquared loses its accuracy due to the Euclidian distance assumption explained in Subsection C. In contrast, it performs significantly better while comparing a small set of very similar multimodal histograms as no approximation is performed over short distances. E. Hybrid Histogram Comparison Method So following inferences can be drawn from these observations 1) Bhattacharyya metric is better suited to shortlist a few very candidates similar to the original template from a wide variety of histograms when template has a complex color profile. 2) measure can be employed to pick the best matching candidate amongst those shortlisted candidates. 3) So even though for a generalized case Bhattacharyya metric performs better Chi squared (as reported in [10]), the combined approach gives better results in case of multimodal histograms. Based on these inferences, we devised a hybrid histogram matching procedure in which Bhattacharyya measure is used to select three histograms with lowest three scores as shortlisted candidates. These candidate histograms are now compared with template using Chi-squared measure to select the best match. We hereafter refer to this procedure as 'BC matching'.

Initialization Frame 149 Frame 319 f Frame 351 Frame 533 Frame 656 Figure 1. Comparison between BC matching and individual Bhattacharyya and Chi-squared measures while comparing multimodal histograms. Black square in 'Initialization' frame is the template. Other squares Red- BC matching, Green- Chi-squared measure, Blue- Bhattacharyya measure. We demonstrate some results in Figure 1. In this experiment, we calculate the color histogram of template patch at initialization. This patch, indicated by a black border has a complex color profile resulting in a multimodal histogram. From then onwards, it is compared with histogram of every patch in the grid (drawn in white) to select the best match using only Bhattacharyya measure, only Chi-squared measure and BC matching. As it can be seen, both Bhattacharyya and Chi- squared measures individually fail to select the best match while BC matching displays the highest accuracy. For this image sequence we manually marked the ground truth every seventh frame and analyzed the matching results using the three methods for about 700 frames. The position error was calculated for the three methods in the form of Euclidian distance between the center of the matching patch and the original patch. The graph of error with respect to the ground truth is plotted in Figure 2. It clearly shows a significant deviation in case of both Bhattacharyya and Chi- Squared methods used individually as compared to BC matching, confirming our hypothesis. This hybrid matching procedure can have applications in various areas of computer vision where histogram matching is a widely used tool. These areas include object tracking, feature selection, image indexing and retrieval, pattern classification and clustering etc. In this paper we demonstrate its application in moving object tracking. III. MOVING OBJECT TRACKING We develop a robust scale adaptive tracker using this hybrid histogram matching method. In the following subsections we describe the algorithm in detail A. Initialization We assume that the target is initialized in the first frame manually or using some detection method and a rectangular patch is identified as the target. We then calculate and store a color histogram of target as template. 0 100 200 300 400 500 600 Figure.2 Error with respect to manually marked ground truth for 700 frames. BC Matching- red, Minimum Chi-squaredgreen, Minimum Bhattacharyya- blue

Figure 3. Target localization process with I n-1 shown in solid red and a few nominees shown in dotted yellow. B. Target Localization After initialization, let O n-1(x,y) be the estimated location of the center of target rectangle I n-1 in the previous frame. Now, in the current frame, a number of rectangular candidate patches of the same size as I n-1 are defined. A candidate patch is defined with its centre at every alternate pixel on or inside the boundary of I n-1 and a color histogram of each patch is calculated and stored. Figure 3 depicts I n-1 with a few representative candidate patches, hereafter referred as 'nominees'. Now each nominee histogram is compared with the template histogram using Bhattacharyya measure and three nominees with the best comparison score as shortlisted for further comparison. These three are now compared with the template using Chi squared measure and a nominee with the least comparison score is selected as the best match R n. Now a scale adaption criterion is applied to R n for selection of best match I n. A. Scale Adjustment R n is the location of target selected from localization in step B with histogram comparison score b using Bhattacharyya measure. We find an 'immediate' average m of the histogram comparison scores of the best matches in previous four frames asm = (b n +b n-1 +b n-2 +b n-3 )/4 (4) We also obtain an average value M of all the previous values of b given by- M = (b n-1 +b n-2 + +b 1 )/(n-1) (5) Now the following relation is used to decide the size of the final rectangle I n with center O n (x,y) I n = η.r n if m (M+ε) (6) I n = R n else where η is the reduction factor in area and ε is a small positive constant between 0 and 1. The reason behind using Bhattacharyya measure for averages is that it always gives a bounded value between 0 and 1. So the comparison of average values becomes easier in contrast to use of Chisquared measure which gives unbounded values. In this way, in every fourth frame, the size of the target rectangle is adjusted using the above relation. When the Initialization Frame 160 Frame 242 Frame 300 Frame 343 Frame 401 Figure 4. The performance of our tracker from a real life outdoor sequence. Tracking accuracy is maintained as the target undergoes rapid changes in scale as it can be seen in from frame 368 onwards. Target has a multimodal histogram with dominant colors being blue and gray.

Initialization Frame 70 Frame 140 Frame 343 Figure 5. Demonstration of robust performance of tracker using the CAVIAR dataset test sequence ThreePastShop2cor.mpeg. The Frame 1 is offset from the CAVIAR video by 105 frames. The target has a multimodal color histogram with dominant colors being red and blue. Initialization Frame 98 Frame 126 Frame 175 Figure 6. Example of robustness of the algorithm to partial occlusion using the CAVIAR dataset test sequence OneShopOneWait2cor.mpg. It can be seen in Frame 98 and Frame 126 that tracking accuracy is maintained even when target is partially blocked. The frame 1 is offset from the CAVIAR video by 205 frames. ` actual size of target becomes smaller than the tracker rectangle, the value of immediate average m increases due to inclusion of background information. However the value of M remains almost constant. Therefore the inequality in Equation (6) is satisfied and the area is reduced by a predefined factor of η. In Figure 4, we demonstrate the performance of our tracker from a real life outdoor sequence in which the target undergoes significant scale changes. Note the systematic reduction in size of the tracker as the target moves away. IV. IMPLEMENTATION DETAILS We now present the implementation details. We use the hue component from the HSV color space for color histogram. 15 bins were used for representing all histograms. For scale reduction, the value of ε was used as 0.05 and the reduction factor η = 0.98 was employed. The algorithm was implemented in C++ on a machine with Intel Core2Duo 1.73GHz Processor and 2GB RAM. Tracking for all CAVIAR sequences was carried out at 25 fps. Note that scale adaptive equation is not applied in first few frames to let M acquire a stable value. V. RESULTS AND DISCUSSION As explained earlier, our tracker is robust to target scale reduction. In Figure 5, the performance of our algorithm is evaluated using a tracking sequence from the CAVIAR test dataset. From the robust tracking in Figure 5 it can be seen that the hybrid histogram comparison method provides very accurate matching. Also, one of the major advantages of our algorithm is the small computational time. Due to the extremely simple implementation, it works easily at 30 fps on our hardware. Our algorithm can also handle partial occlusion of the target. Figure 6 from the CAVIAR dataset demonstrates the excellent performance of our tracker in case of partial occlusion. However we note here that the tracker sometimes tends to lose the target when it is partially occluded by an object with the same global histogram. However the main focus of this work is to demonstrate the performance of the hybrid histogram matching procedure. We intend to implement a fragment based tracker as proposed in [11] which will improve its performance. And considering the small computational cost of the algorithm, various methods can be incorporated to further improve the tracking performance in complex scenarios.

VI. CONCLUSION AND FUTURE WORK In this paper, we first report that Chi squared histogram comparison measure outperforms Bhattacharyya measure while comparing multimodal histograms and provide the theoretical explanation for it. Then we propose an effective hybrid histogram comparison method using these measures. It can be implemented in many applications where histogram comparison is a major tool. To demonstrate its performance, we present a computationally efficient algorithm for real time object tracking. We propose a systematic scheme for scale adjustment in contrast to many previous blob based algorithms. The algorithm can also handle partial occlusion. The performance of tracker is evaluated using real life and test sequences. This paper represents work in progress and we intend to use this histogram comparison scheme in other applications and prove its robustness. We also want to improve the tracking performance in case of complex cluttered scenes. APPENDIX Here we show how Bhattacharyya metric approximates the Chi-squared measure for small distances. For an arbitrary function f applied on a histogram of unknown distribution the chi-squared measure can be approximated as Due to error propagation, the denominator of the right-hand side can be written as: REFERENCES [1] W.M. Hu, T.N. Tan and L. Wang, "A survey on visual surveillance of object motion and behaviors," IEEE Transactions on Systems, Man, and Cybernetics Part C, vol. 34, pp. 334-512, March 2004. [2] Lu, W., Tan, Y.-P.. "A color histogram based people tracking system". In: Proceedings of 2001 IEEE. International Symposium on Circuits Systems, Vol. 2, pp. 137-140, 2001 [3] M. Mason, Z. Duric, "Using Histograms to Detect and Track Objects in Color Video", Proceedings of the Applied Imagery Pattern Recognition Workshop, p.154, October 10-12, 2001 [4] D. Wojtaszek, R. Laganiere, "Using Color Histograms to Recognize People in Real Time Visual Surveillance" International Conference on Multimedia, Internet and Video Technologies, pp. 261-264, 2002 [5] F. Bajramovic, B. Deutsch, C. Grabl and J. Denzler "Efficient Adaptive Combination of Histograms for Real-Time Tracking" EURASIP Journal on Image and Video Processing Volume 2008, Article ID 528297, 2008 [6] Ying Fang, Huiyuan Wang, Shuang Mao, Xiaojuan Wu " Multi-object Tracking Based on Region Corresponding and Improved Color-Histogram Matching", IEEE International Symposium on Signal Processing and Information Technology, pp 1-4, 15-18 Dec. 2007 [7] D. Comaniciu, V. Ramesh, P. Meer "Real-Time Tracking of Non-Rigid Objects using Mean Shift", IEEE Conf. Computer Vision and Pattern Recognition (CVPR'00), Vol. 2, 142-149, 2000 [8] EC Funded CAVIAR project/ist 2001 37540, found at URL: http://homepages.inf.ed.ac.uk/rbf/caviar/ [9]S. Cha, "Taxonomy of Nominal Type Histogram Distance Measures", American Conference on Applied Mathematics (Math '08), Harvard, Massachusetts, USA, March 24-26, 2008. [10] Aherne, F., Thacker, N., Rockett, P. "The Bhattacharyya metric as an absolute similarity measure for frequency coded data". Kybernetika, 32(4), pp 1 7, 1997 [11] Amit Adam, Ehud Rivlin and Ilan Shimshoni, "Robust Fragments-based Tracking using the Integral Histogram" IEEE Conference on Computer Vision and Pattern Recognition (CVPR '06), pp.798-805, June 17-22, 2006 where is any constant. Since variance of a constant term is zero, substituting in (A.1) gives This proves that Bhattacharyya measure approximates the Chi-squared similarity measure. A detailed mathematical analysis is not presented due to space constraints. Please refer to [10] for comprehensive mathematical treatment on relation between Bhattacharyya and Chi-squared measures. ACKNOWLEDGMENTS We gratefully acknowledge the valuable comments provided by Prof. A. S. Abhyankar, VIIT, Pune and Dr. Sumantra Dutta Roy, Dept. of EE, IIT Delhi