Multi-Object Tracking in Airborne Video Imagery based on Compressive Tracking Detection Responses Chen, Ting; Sahli, Hichem; Zhang, Yanning; Yang, Tao

Transcription

1 Vrije Universiteit Brussel Multi-Object Tracking in Airborne Video Imagery based on Compressive Tracking Detection Responses Chen, Ting; Sahli, Hichem; Zhang, Yanning; Yang, Tao Published in: International Conference on Advances in Mobile Computing and Multimedia Publication date: 0 Document Version Publisher's PDF, also known as Version of record Link to publication Citation for published version (APA): Chen, T., Sahli, H., Zhang, Y., & Yang, T. (0). Multi-Object Tracking in Airborne Video Imagery based on Compressive Tracking Detection Responses. In International Conference on Advances in Mobile Computing and Multimedia (pp. 9-9). [00] ACM. General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. Users may download and print one copy of any publication from the public portal for the purpose of private study or research. You may not further distribute the material or use it for any profit-making activity or commercial gain You may freely distribute the URL identifying the publication in the public portal Take down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim. Download date: 0. Dec. 0

2 Multi-Object Tracking in Airborne Video Imagery based on Compressive Tracking Detection Responses Ting Chen.School of Computer Science and Engineering, Northwestern Polytechnical University, Xi an, PR China.Dept. of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Brussels, Belgium Hichem Sahli.Dept. of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Brussels, Belgium.Interuniveristy Microelectronics Center (IMEC), Leuven, Belgium Yanning Zhang, Tao Yang.School of Computer Science and Engineering, Northwestern Polytechnical University, Xi an, PR China ABSTRACT Multi-object tracking (MOT) in airborne video is a challenging problem due to the uncertain airborne vehicle motion as well as mounted camera vibrations. Most approaches addressing tracking in such type of scenario, use data association based on motion detection responses. Such approaches fail tracking objects with low speed or static ones. To alleviate the motion detection failures, in this paper we propose a multi-object tracking system based on combining motiondetection and Compressive Tracking detection responses. In this work, as in [], the multi-object tracking problem is solved by associating tracklets according to their confidence values. For reliable association between tracklets and detections, we propose using Compressive Tracking (CT) as a mean to detect objects when motion-detection fails. By exploiting the compressive tracking, which allows discriminating the appearances of objects, tracklet association can be successfully achieved even when objects undertake stopand-go motion as well as when they are partially occluded. Experiments with challenging airborne video datasets show significant tracking improvement compared to existing stateof-art methods. Categories and Subject Descriptors I.. [Computing Methodologies]: Image Processing and Computer Vision-Scene Analysis-Tracking General Terms Algorithms Keywords Tracklet Confidence, Compressive model, Tracking Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. MoMM 0, December -, 0, Brussels, Belgium. Copyright 0 ACM //...$ INTRODUCTION The goal of multi-object tracking (MOT) in airborne video is to estimate the states of multiple objects while conserving their identifications under appearance and motion variations with time. This is challenging since one should consider the uncertain motion of the airborne vehicle as well as the vibration of the camera []. In airborne videos and aerial surveillance, due to the targets small size and their various appearance, most of the proposed MOT approaches, adopted motion-based detection and association to recover the trajectories of all targets [, ]. Motion-based detection rely on a background model [,, ], where objects with different motion model than the background can be detected. Such motion-based detection approaches fails detecting static objects or objects with low speed and hence not useful for long-term tracking. In the other hand, approaches using appearance models for object detection [,, ], are not suitable for airborne videos. This is mainly due changes in appearances of the tracked objects, the dynamic change of the background, the uncertain motion of the airborne vehicle as well as camera vibrations. In this work, the multi-object tracking problem is solved by associating tracklets according to their confidence values. For reliable association between tracklets and detections, we propose using Compressive Tracking (CT) as a mean to detect objects when motion-detection fails. Compressive tracking [9, 0, ] achieved favorably high efficiency and accuracy in several object tracking scenarios, which make it a good candidate for multi-object tracking in airborne videos. We follow [, ] to formulate the MOT problem by using locally and globally tracklet association based on the tracklet confidence. The tracklets with the high confidence are locally associated to the motion-detection responses, whereas tracklets with low confidence are globally associated to other tracklets using CT based tracking results. Different from [, ], we adopt a compressive tracker to obtain extra observations to link the fragmented low confidence tracklets in a global fashion. Our primary contribution in this work, is a multi-object tracker for airborne videos, that uses CT to alleviate the problems due to failures in motion-detection. We adopt the two phases tracking approach introduced in [] and incorporate the information from CT for the tracklets association.

3 . TRACKING FRAMEWORK. System Overview We denote an object i appears at frame t by using a binary function as ϕ i t =, otherwise, ϕ i t = 0. Let, x i t = ( p i t, s i t, v i t), the state of the object i at frame t, where p i t, s i t, v i t are the location, size, velocity, respectively. Then, denote the tracklet T i of the object i as a set of states up to frame t as T i = { x i k ϕ i k =, t i s k t i e t }, where t i s and t i e are the start and end frame number of the tracklet. In addition, denote all the object states appeared at frame t as T t = {x t,, x n t t }, where n t is the total number of objects, and let T :t = {T,, T n t } be a set of tracklets of all objects up to frame t. As in [], for object detection, we adopt a motion-based detection approach using the motion history, applied on frames. Let, d j t the j-th detected observation at frame t, we denote by D t = {d j t j n d } all the observations collected at frame t, n d being the the number of observations, and D :t = {d :t,, d n d :t } the set of all observation up to frame t. All the observations associated to object i up to frame t are referred to as d i :t = {d i,, d i t}. We follow [, ] to formulate the MOT problem as a two phases optimization problem. The main steps of our algorithm are summarized in Algorithm. During the first phase, referred to as Local Phase, the tracklets from frame t are first locally associated with the motion-based detections of frame t using the affinity model, A(T i, d j t) of [], and then their confidence [], conf(t i ), is updated as follows: conf(t i ) = ( k [tis,tie],ϕik= ( L A T i, dk) ) i max (( + β log ((L w t) /L)), 0) where L is the length of the tracklet T i, defined as L = T i, A(T i, d i k) is the affinity score calculated using the affinity model of [], β is a control parameter relying on the performance of the motion-based detection and β = in our experiments, and w t = (t i e t i s + L). The association procedure is summarized in Section.. Since the tracklet confidence lies in [0, ], we consider a tracklet as a reliable tracklet with high confidence, when conf(t i ) > 0.; otherwise it is considered as a fragmented tracklet with low confidence. We denote by T i(h) the tracklets with high confidence, and by T i(l) the tracklets with low confidence. The local association between the tracklets and motion detections allows us to progressively grow locally optimal tracklets. However, it happens that the objects being tracked are not detected due to unreliable motion detection, small objects, static objects, objects evolving at low speed, or occluded objects. When a detection of the object is not available, the confidence of a tracklet is decreased. To solve this problem, we associate to each tracked object, x i, a Compressive Tracker (CT) [9], denoted as CT i (x i ). For each CT tracker we maintain a Bayes classifier H(z i ), and the current tracker location, l i. During the second phase, i.e the global association, tracklets with low confidence, T k(l) are globally associated with other tracklets and detections. Different from [, ], we make use of the Compressive Tracking (CT) results (l i ) into the global association to solve the motion-based detection () Algorithm The two-phases tracking framework. Input: The tracked objects T t = {x t,, x n t t }, The tracked objects locations obtained by compressive tracking C t = {lt,, l nc t }, and compressive tracking classifiers H t = {H(zt ),, H(z n c t )} at frame t, Motion-based detections D t = {d t,, d n d t } at frame t. Local Phase: For each tracked object : if x i t can be observed by d i t then : Update the x i t and conf(t i ) by associated d i t. : Get the compressive tracking result lt i by Eq.(). : Initialize/Update the classifier H(zt). i : else : buffer the object for the global phase. : end if Global Phase: : Retrieve the reliable tracklets from the local phase. : Get the compressive tracking result l i t by Eq.(). : Calculate the pairwise association cost matrix G defined in Eq.() and apply Hungarian algorithm. : Manage the object tracklet according to the conf(t i ). : Update the x i t and conf(t i ) according to the optimal association result from G. : Update/Terminate the classifier H(z i t). Output:The object set T t = {x t,, x n t t } with their compressive tracker attributes C t = {lt,, lñc t } and H t = {H(zt ),, H(zñc t )}. failure problem. In this phase we consider the following associations: Event A: T k(l) is associated with T i(h), Event B: T k(l) is terminated, Event C: T k(l) is associated to an optimal compressive tracker location l j t, Event D: T k(l) is associated with a motion detection d j t not associated to a high confidence tracklet. This phase is described in Section.. For both association phases (local and global) we determine the tracklet-detection pairs and the tracklet-tracklet pairs using the Hungarian algorithm [] such that a total association cost is maximized. This will be explained in the following sections.. Local Association of Tracklets During the local association phase pairwise association is performed to associate motion detection responses d j t with tracklets high confidence T i(h) []. A score matrix S n t n d defined as: S nh n d = [s ij ] nh n d, s ij = ln(a(t i(h), d j t)), d j t D t, where n h and n d is the number of the high confidence tracklets and motion-based detections at frame t, respectively. ()

4 The Hungarian algorithm [] is used to determine trackletdetection pairs with the maximized total affinity in S nh n d. The detection d j t is associated with T i(h) when the association cost of a tracklet-detection pair is less than a predefiend threshold, ln (θ), the position and velocity of a tracklet T i(h) is updated with the associated d j t, and the size of the object is updated by averaging the associated detections of recent past frames []. The confidence of the tracklet conf(t i(h) ) is updated by using d j t in Eq().. Global Association of Tracklets As mentioned above, this phase allows associating globally low confidence tracklets either to high confidence tracklets, or to motion detection responses, d j t, not associated to a high confidence. Moreover, to alleviate the problem of motion detection-failure, and hence missing associations, we make use of the CT trackers to associate low confidence tracklets to other high confidence tracklets. Let, {q j t } n q j= Dt the motion-based detection responses that have not been associated to any high confidence tracklets during the the previous local association. We denote by C t = {lt,, l n c t } the current locations of the objects as determined by the n c CT trackers. The following association events are considered: Event A: T k(l) is associated with T i(h), Event B: T k(l) is terminated, Event C: T k(l) is associated to an optimal compressive tracker location l j t, Event D: T k(l) is associated with a motion detection d j t not associated to a high confidence tracklet. The cost matrix for the above events is defined as follows: A nl n h G (nl +n c+n q) (n h +n l ) = log (θ) nc n h log (θ) nq n h B nl n l C nh n l D nq nl () where A = [a ij] represents T i(l) is associated with T i(h) and a ij= ln(a(t i(l), T j(h) )) is the association cost calculated by the affinity model of Bae et al. []. B = diag[b,, b nl ] models T i(l) is terminated and b i = ln( conf(t i(l) )) is the cost to terminate T i(l). C = [c ij ] represents T i(l) is associated with l j t, where c ij = ln(a(t i(l), l j t )). D = [d ij ] represents T i(l) is associated with q j t, where d ij = ln(a(t i(l), q j t )). The optimal association pairs, which minimize the global association cost in G, are determined using the Hungarian algorithm [], and the tracklets and their confidence values are updated with the results.. Compressive Tracking Model In this section, we give a short overview of Compressive Tracking (CT) and how it is used in our framework to estimate the current tracker location, the interested reader is referred to [9, 0, ] for more details. In CT a highdimensional feature being a series of rectangle features which have different locations and sizes within the region of interest are used. This large set of features is then compressively sensed using a very sparse measurement matrix which satisfies the restricted isometry property. This can then be efficiently computed for real-time tracking. By taking the tracking as the online classification task of the object and background, the naive Bayes classifier is used to model the real-time tracking process. For each frame, some positive samples near the target and some negative samples farther away from the object are cropped to update the classifier parameter. To predict the next target location, samples are drawn around the current target location and sent to the classifier. The sample with the maximal classification score are selected as the most probable target location l t [9]. In our framework, the most probable target location of object i in frame t is given by: ( lt i = arg max H(zt k ) w k) () zt k z t,k=,,n z where z t = {zt, zt,, z nz t }, are the samples around location lt i (candidate samples), and the weights w k is set to w k = in case object i is not associated to any motion detection reponse, and defined as in Eq( ) if object i is associated to the motion detection response d j t. w k = l(z k t ) d j t k [,n z] l(zk t ) d j t () with l(z k t ) being the location of the k-th sample. In order ro account for a high ratio false-alarms a CT tracker is terminated after L c frames that the object is not associated to a motion detection response.. EXPERIMENT We evaluate our approach on DARPA VIVID dataset []. The popular evaluation metrics defined in [] are used for performance comparison: the number of objects in groundtruth (GT), the ratio of mostly tracked trajectories (MT), the ratio of mostly lost trajectories (ML), the number of fragments (Frag) and ID switches (IDS). We compared our approach to [] and []. All of the approaches adopted the same motion-based detection configuration, and a window size of frames was defined to remove the unreliable shorter tracklet. For both [] and [] we used the publicly available MATLAB codes provided by the authors and we selected the p system in [] in our experiment. We set one random matrix for each high confidence tracklet, and the dimension of each random matrix have been randomly (uniform) set between 0 and 00. Similar to [9, 0, ], the search radius to detect the object location is set to γ = 0 and about 00 samples are generated. The algorithm has been implemented in MATLAB on PC with an Intel Core.0 GHz CPU with GB RAM. As shown in Figure (a), the green boxes are the motionbased detection, and the red boxes in #0 and # are the compressive appearance detection for each high confidence tracklets when motion-based detection failed detecting the targets. With the help of the compressive appearance detection for data association, our algorithm increase the MT ratio and reduce the IDs switches ratio. A quantitative comparison between our approach and other algorithm is given in Table. Both [] and [] perform well with the good motion-based detections, while poor performs with failed detection. As expected, our algorithm improves performance for most metrics, such as the noticeably reduced ML and Frags rates and increased MT rate against the [] and []. The tracking results are shown in Figure (b). The main limitation of the proposed approach its possible high computation time which is proportional to the number

5 Table : Performance comparison. Method MT(%) ML(%) Frag IDS []. 0. [].. Our 0.. #9 #0 of high confidence tracklets, and hence number of compressive tracking classifier. For the DARPA VIVID dataset [], where the number of objects is less than 0, the average run-time is 0. sec/frame.. CONCLUSIONS # (a) Compressive appearance detection results. # # # We proposed a multi-object tracking method based on tracklet confidence with compressive appearance model to handle the motion-based detection failure. The online multiobject tracking problem is formulated based on the tracklet confidence: the tracklets with the high confidence are associated with motion-detection responses, whereas tracklets with low confidence are linked to the compressive tracking responses and other tracklets. Experiments with challenging public datasets show the improved tracking performance compared to existing state-of-art methods. #0 # #0 # # # 0. ACKNOWLEDGMENTS This work is supported by the CSC-VUB scholarship (No. 0090), the National Natural Science Foundation of China (No., No. 0, No. 0, No. 0), Northwestern Polytechnical University (NPU) New AoXiang Star (No.G0KY00), Fundamental Research Funds for the Central Universities (No.00AX00), NPU New People and Direction (No.GH00) and NPU AoXiang Star (No.GH0).. REFERENCES [] S.-H. Bae and K.-J. Yoon, Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning, in CVPR, 0, pp.. [] S. Dey, V. Reilly, I. Saleemi, and M. Shah, Detection of independently moving objects in non-planar scenes via multi-frame monocular epipolar constraint, in ECCV, 0, pp. 0. [] J. Prokaj, M. Duchaineau, and G. Medioni, Inferring tracklets for multi-object tracking, in CVPRW, 0, pp.. [] V. Reilly, H. Idrees, and M. Shah, Detection and tracking of large number of targets in wide area surveillance, in ECCV, 00, pp. 99. [] J. Xiao, H. Cheng, H. Sawhney, and F. Han, Vehicle detection and tracking in wide field-of-view aerial video, in CVPR, 00, pp. 9. [] J. Prokaj and G. Medioni, Persistent tracking for wide area aerial surveillance, in CVPR, 0, pp. 9. [] J. Prokaj, X. Zhao, and G. Medioni, Tracking many vehicles in wide area aerial surveillance, in CVPRW, 0, pp.. [] T. Pollard and M. Antone, Detecting and tracking all moving objects in wide-area aerial video, in CVPRW, 0, pp.. 9 Low Tracklet Confidence High (b) Multi-Objects tracking results. Figure : Tracking Results. (a)green boxes correspond to motion based detections, and Red boxes to CS-based tracking. (b)the results of DARPA VIVID dataset. [9] K. Zhang, L. Zhang, and M.-H. Yang, Real-time compressive tracking, in ECCV, 0, pp.. [0] K. Zhang, L. Zhang, and M. Yang, Fast compressive tracking, PAMI, vol., no. 0, pp. 00 0, 0. [] T. Chen, Y. Zhang, T. Yang, and H. Sahli, Dynamic compressive tracking, in MOMM, 0, p.. [] J. Xing, H. Ai, and S. Lao, Multi-object tracking through occlusions by local tracklets filtering and global tracklets association with detection responses, in CVPR. IEEE, 009, pp [] Q. Yu and G. Medioni, Motion pattern interpretation and detection for tracking moving vehicles in airborne video, in CVPR, 009, pp.. [] R. K. Ahuja, T. L. Magnanti, and J. B. Orlin, Network flows: Theory, algorithms, and applications, 99. [] R. Collins, X. Zhou, and S. K. Teh, An open source tracking testbed and evaluation web site, in PETS, 00, pp.. [] Y. Li, C. Huang, and R. Nevatia, Learning to associate: Hybridboosted multi-target tracker for crowded scene, in CVPR, 009, pp