TRACKING OF MULTIPLE SOCCER PLAYERS USING A 3D PARTICLE FILTER BASED ON DETECTOR CONFIDENCE

Transcription

1 Advances in Computer Science and Engineering Volume 6, Number 1, 2011, Pages Published Online: February 22, 2011 This paper is available online at Pushpa Publishing House TRACKING OF MULTIPLE SOCCER PLAYERS USING A 3D PARTICLE FILTER BASED ON DETECTOR CONFIDENCE Graduate School of Engineering Kobe University fantakuro@me.cs.scitec.kobe-u.ac.jp takigu@kobe-u.ac.jp ariki@kobe-u.ac.jp Abstract In this paper, a novel method for tracking multiple soccer players based on a support vector machine (SVM) and a particle filter is proposed for a monocular fixed soccer video. This paper considers three main topics. First, the state of a particle filter presented in a 3D world coordinate space, instead of a 2D image coordinate space, provides a more realistic model tracking because depth considerations are taken into account. Second, by integrating SVM into the particle filter framework, the initial positions of the players can be detected or rediscovered when the tracker has lost the players. Third, the feature histogram and correlation are combined when the likelihood for a particle is computed. The effectiveness of this method has been confirmed by experiments that actually tracked multiple soccer players. Keywords and phrases: particle filter, soccer, tracking, earth mover s distance. Received October 11, Introduction When professional cameramen video or edit a sports game, they are required to make it interesting for the TV or video audience. In order to address this requirement by computer, automatic production systems including digital camera work have been studied (ex. [1, 2]). A typical video production technology involves shooting the

2 94 sports game using a fixed camera, and then clipping and connecting the video frames. This technology allows videos to be edited according to individual audience preferences. An automatic production system is composed of several image recognition techniques and requires the semantic analysis of the process and strategy of the game. The position and trajectory of players in the game can provide useful information to facilitate the semantic analysis. In order to obtain such information, detection and tracking of objects (players, the ball, and the lines) are required. Many tracking methods have been proposed to date, and one of the popular methods of tracking-by-detection has been reported, where a detection framework is integrated into a tracking algorithm [3, 4, 6]. It makes it easy to get the initial positions of targets or rediscover them when the tracker has lost the targets. For object detection, there are two types of detection algorithms; SVM [3, 4] and AdaBoost [5, 6]. Both methods can detect their targets correctly if an abundance of training data is used. In the tracking algorithm, a particle filter is often employed because of its robustness with respect to occlusion. Therefore, our tracking algorithm also employs tracking-by-detection with a particle filter for robust tracking despite occlusion. In our system, at first, players are detected on the first frame. Next, tracking using a particle filter is carried out, where the particle filter approximates the probabilistic model discretely using a large number of particles. Each particle is translated by a state transition model and is weighted by an observation model, and then the state is propagated. Due to the detection results, the particles are weighted more accurately, where the proposed method computes the likelihood for a particle using a linear combination of the feature histogram and correlation. 2. Player Detection For each video frame, player detection is first performed. The classifier that detects players is trained in advance using the training data. The number of training data is 3,000 positive samples and 1,000 negative samples. Examples of positive and negative samples are shown in the upper part and the lower part of Fig. 1, respectively. The gray value at each pixel on a gray scale image was used as a feature. In the detection step, the multiple detectors search the video frame by sliding the window and changing the size, and the classifier outputs the detected regions

3 TRACKING OF MULTIPLE SOCCER PLAYERS USING A 3D 95 with an SVM score c(det) for a detector det. The SVM score is defined as the distance between the window image and the decision boundary of the classifier. Figure 1. Training samples. Figure 2 shows the detection results. Player regions are correctly detected, but in the case where multiple players are crossing in front of each other, an individual player cannot be detected D particle filter Figure 2. Detection results. 3. Tracking Algorithm A tracking algorithm is based on estimating the distribution of each state of a particle filter. The world coordinate is fixed to the soccer stadium, and the state x at time t is defined as Eq. (1) using the 3D coordinate. T y x = [ p, p, v, v, a, a ]. (1) x y x The state x consists of the 3D world position ( p x, py ), the velocity ( v x, v y ) and the acceleration ( a x, a y ). The state presented in the 3D world coordinate space, rather than the 2D image coordinate space, makes tracking more robust owing to depth considerations when occlusion occurs. Here, the height of the players is not considered, and ( p x, p y ) represents the position of a player s centroid. y x

4 96 Then, we assumed that the state transition model from time accelerated motion model as follows: t τ to t is an x () t = Cx( t τ) + ϒ C 2 I22τI22 ( τ 2) I22 = O22I22 τi22 O22O22 I22 T p 12, v 12 a 12 ϒ = [ ε I ε I, ε I ]. The process noises ε p εv, εa, for state variables are independently drawn from zero-mean normal distributions. The variance σ 2 p for the position noise changes with the size of the tracking target, whereas the variances σ 2 v and σ 2 a for the velocity and acceleration noise are inversely proportional to the number of successfully tracked frames. Hence, we can expect that longer a target is tracked successfully, the less the particles are spread Assigning detection to tracker In order to integrate the detector into the tracking framework using SVM, a detector/tracker pair must be decided. For this pairing decision, the algorithm described in [3] is utilized. The matching algorithm works as follows. First, a matching score s ( tr, det) for each pair ( tr, det) of the tracker tr and the detector det is computed as shown in Eq. (2): N s ( tr, det) = g( tr, det) ( ) ( ) c det + α pn det p, (2) p tr size (, ) = ( ) = tr size g tr det p size det det tr pn, (3) size 2 where pn ( det p) ~ N( det p; 0, σdet ) denotes the normal distribution evaluated for the distance between the detector det and each particle p of tracker tr, and g ( tr, det) is a gating function which additionally assesses each detector. size tr denotes the size of tracker located at p y, and size det denotes the window size of the tr

5 TRACKING OF MULTIPLE SOCCER PLAYERS USING A 3D 97 2 detector det. The values of α and σ det are set experimentally. The meaning of Eq. (2) is that a pair consisting of tracker tr and detector det gets a higher matching score s ( tr, det), if their average distance and window size difference are smaller. Finally, the matching matrix function S (the number of trackers the number of detectors), whose element is s ( tr, det), is computed. Then, the pair ( tr, det ) with the maximum score in S is iteratively selected, tr and the detector and the rows and columns belonging to the tracker deleted. This is repeated until no further valid pair is available Observation model det in S are To compute the weight ω tr, p for a particle p of the tracker tr, our algorithm estimates the conditional likelihood of the new observation y t of the propagated particle, where the state is x t. For this purpose, we combine different sources of information, namely the likelihood based on the detector det, the histogram distance and the correlation as follows: ω tr, p = p( y x ) t t The first term computed as follows: D = p + β p + ( β) p. (4) D H 1 C p denotes the likelihood based on the detector det, which is pd = I( tr)( φ c( p) + ψ p ( p det )), N 1 s( tr, det ) > threshold I ( tr) =, (5) 0 other where c ( p) denotes the SVM score at the particle position, and p N ( p det ) denotes the distance between the particle p and the associated detector det, evaluated under a normal distribution p N. I ( tr) is the indicator function that returns 1 if a detector is associated with the tracker, namely the matching score s ( tr, det ) is larger than threshold. Otherwise, 0 is returned. When a matching

6 98 detection is found, this term robustly guides the particles. The parameters φ, ψ and threshold are set experimentally. The second and third terms ( p H and p C ) denote the likelihood based on the histogram distance and the likelihood based on the correlation, respectively. The former is robust to occlusion or pose change of players, whereas the variance of particle grows more than the necessity. On the other hand, the latter is weak to occlusion but it has an advantage in that the particle variance is small. Therefore, the weight for the former increases, when the distance between players is small (when occlusion of players tends to occur). In contrast, when the distance is long, the weight for the latter increases. Due to this method, the players can be tracked accurately for a long time. In this paper, we employ the Earth Mover s Distance (EMD) [7] as the histogram distance and Normalized Cross-Correlation as the correlation. They are defined as follows: where p H = ph ( distemd ), (6) {( I i, j) I} { T( i, j) T } i, j p C =, (7) 2 2 {( I i, j) I} { I( i, j) T } i, j i, j β = p N ( min dist ), (8) i P dist EMD in Eq. (6) denotes EMD. The normalized cross-correlation i p is defined by Eq. (7). Here, I ( x, y) and C T ( x, y) are gray values at point ( x, y) in the search area and the template image, respectively. I and T are averaged gray value of the search area and the template image, respectively. Eq. (8) denotes the weight of p H and p C. It is defined as the distance between the target particle and the closest player, evaluated under a normal distribution p N. dist i denotes the distance between the target player and player i, and P denotes a set of players Calibration The perspective projection of a pinhole camera model is needed to project world

7 TRACKING OF MULTIPLE SOCCER PLAYERS USING A 3D 99 coordinates ( X, Y, Z ) of state x to image coordinates ( x, ) as shown in w w w f y f Eq. (9). A is an intrinsic parameter, R is the rotation matrix, and t is the translation vector. R and t are extrinsic parameters. X w x f Y = [ ] w y f A R t (9) Zw 1 1 A soccer field has known details, such as the length of lines in the penalty area and the goal area, etc. Therefore, several points on an image are manually associated with the world coordinates of the soccer field, and then the camera parameters are calibrated using the Tsai-method [8]. Fig. 3 shows the points used for calibration Experiment environment Figure 3. The points used for calibration (red points). 4. Experiment We videoed a film of a soccer game played during the 38th National High School Soccer Championship in Japan. The image size was 1, pixels with 24-bit color. The image size of the player template was pixels. The number of particles was 300, and the frame rate was 30 fps. As the experiment data, the video sequence as shown in Fig. 4 was used, where the left half of the field located in the lower half of the image. We extracted 12 video clips of 330 frames (on average), and evaluated the results in terms of tracking accuracy as defined by Eq. (10). P 1 Number of correctly tracked frames for player i Tracking Accuracy = (10) P Number of frames including player i 1 i =

8 100 Figure 4. Frame shot used in experiment. If the distance between the correct 2D image coordinate obtained manually and the coordinate given to a 2D image from a tracking result obtained by Eq. (9) was five pixels or less, it was defined as correct tracking. P denotes the number of players. The camera parameters estimated by calibration are shown in Table 1. Table 1. Estimated camera parameters Extrinsic parameters Intrinsic parameters R x [deg] f 3.16 [m] R R y 5.39 [deg] k 3.61e-02 R y 0.30 [deg] s x 1.0 t x [m] s y 1.0 t t y [m] C x t z [m] C y 640 [pixel] 360 [pixel] 4.2. Experiment results Fig. 5 shows the tracking results obtained using the proposed 3D particle filter compared with the traditional 2D particle filter [1], where we used the normalized cross-correlation as the likelihood evaluation. The average tracking accuracy was improved significantly from 63.1% to 70.3%, so it can be said that modeling based

9 TRACKING OF MULTIPLE SOCCER PLAYERS USING A 3D 101 on the 3D world coordinates is effective. But the tracking accuracy decreased for sample 13, sample 12, sample 09 and sample 05, where the players hardly move or simply move linearly. It appears that the accuracy decreased in the proposed method, even though the motion was quite simple, because complex 3D modeling is being carried out. Figure 5. Tracking accuracy of the proposed 3D tracking compared with 2D tracking. We compared the performance of p D, p H and p C in Eq. (4). The results are shown in Table 2, where the circle indicates the utilization of the corresponding likelihood. In p H, the Bhattacharyya distance and EMD were compared. Comparing the second line with the third line in Table 2, EMD showed better performance than the Bhattacharyya distance. It can be said that EMD is more effective than the Bhattacharyya distance with respect to occlusion. Comparing the second line with the fourth line in Table 2, p D improved the tracking accuracy significantly. Table 2. Tracking accuracy in terms of observation model p D p H pc Tracking Accuracy [%] Bhattacharyya EMD Bhattacharyya

10 102 Fig. 6 shows the tracking results for video clips with narrow windows viewed close-up. A number of occlusion incidents occurred during a short time, but the Figure 6. Tracking results (close-up view). proposed method was able to track the players without problems. Fig. 7 shows the tracking results for video clips with wide windows view from further away. In both the cases, the players can be tracked accurately even when occlusion occurs. Figure 7. Tracking results (distant view).

11 TRACKING OF MULTIPLE SOCCER PLAYERS USING A 3D Conclusion In this paper, we proposed a novel method for detecting and tracking multiple objects soccer players in a fixed monocular soccer video. By making use of the state in the 3D world coordinate space in a particle filter, we were able to achieve a more realistic model. In addition, the observation model incorporating the different information sources makes tracking more robust when occlusion or pose changes occur. In future, we are planning to carry out work focusing on the detection of more complex soccer events (e.g., offsides and fouls) using the 3D positional information of players acquired by the proposed method. References [1] Y. Ariki, T. Takiguchi and K. Yano, Digital camera work for soccer video production with event recognition and accurate ball tracking by switching search method, IEEE International Conference on Multimedia & Expo, pp , Hannover, Germany, June [2] T. Nishino, T. Takiguchi and Y. Ariki, Situation recognition using 3D positional information of ball from monocular soccer image sequence, International Conference on Multimedia, Information Technology and its Applications (MITA 2009) (2009) [3] M. Breitenstein, F. Reichin, B. Leibe, E. Koller-Meier and L. V. Gool, Robust tracking-by-detection using a detector confidence particle filter, The 12th IEEE International Conference on Computer Vision (ICCV), pp , Kyoto, Japan, Sept [4] G. Zhu, C. Xu, Q. Huang and W. Gao, Automatic multi player detection and tracking in broadcast sports video using support vector machine and particle filter, IEEE International Conference on Multimedia & Expo (ICME), pp , Toronto, Canada, July [5] Y. Wu, X. Tong, Y. Zhang and H. Lu, Boosted interactively distributed particle filter for automatic multi-object tracking, IEEE International Conference on Image Processing (ICIP), pp , San Diego, U. S. A., Oct [6] K. Okuma, A. Taleghani, N. D. Freitas, J. J. Littele and D. G. Lowe, A boosted particle filter: multi target detection and tracking, The 8th European Conference on Computer Vision (ECCV), pp , Prague, Czech, May 2004.

12 104 [7] Y. Rubner, C. Tomasi and L. J. Guibas, The earth mover s distance a metric for image retrieval, International Journal of Computer Vision (IJCV) 40(2) (2000), [8] R. Y. Tsai, An efficient and accurate camera calibration technique for 3D machine vision, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp , Miami, U. S. A., June [9] Xinguo Yu, Xin Yan and Yiqun Li, Tool-aided semantics acquisition for live soccer video, IEEE International Conference on Multimedia & Expo, pp , Hannover, Germany, June 2008.