USING MAXIMUM CONSISTENCY CONTEXT FOR MULTIPLE TARGET ASSOCIATION IN WIDE AREA TRAFFIC SCENES

Transcription

1 USING MAXIMUM CONSISTENCY CONTEXT FOR MULTIPLE TARGET ASSOCIATION IN WIDE AREA TRAFFIC SCENES Xinchu Shi 1,2, Peiyi Li 2, Haibin Ling 2, Weiming Hu 1, Eri Blasch 3 1 National Laboratory of Pattern Recognition, Institute of Automation, Beijing, China 2 Department of Computer and Information Science, Temple university, Philadelphia, USA 3 Air Force Research Lab, USA ABSTRACT Tracing multiple vehicles in wide area traffic scenes is challenging due to high target density, severe similar target ambiguity, and low frame rate. In this paper, we propose a novel spatio-temporal context model, named maximum consistency context (MCC), to leverage the discriminative power and robustness in the scenario. For a candidate association, its MCC is defined as the most consistent association in its neighborhood. Such a maximum selection pics the reliable neighborhood context information while filtering out noisy distraction. We tested the proposed context modeling on multi-target tracing using three challenging wide area motion sequences. Both quantitative and qualitative results show clearly the effectiveness of MCC, in comparison with algorithms that use no context and standard spatial context respectively. Index Terms Context modeling, multi-target tracing. 1. INTRODUCTION Surveillance over wide area motion imagery (WAMI) has recently been attracting increasing amount of research attention due to its wide range of applications and the advance in acquisition techniques [1 11]. However, tracing and detection of multiple moving vehicles in such scenes are challenging tass, since a wide area traffic scene usually contains much more moving targets than traditional visual tracing scenarios do. These targets are often hard to distinguish from each other due similar appearances and small sizes in images. In addition, the low frame rate, which is typical in WAMI applications, brings extra ambiguity for associating targets over long time periods.. To address such ambiguity, researchers have investigated ways to integrate additional information, such as prior nowledge [1], scene structures [6], and target context [7, 10, 11]. Using target context to improve multiple target tracing in wide area traffic scenes is of special interest because it is relatively domain independent and therefore applicable to various situations. For a target under investigation, existing methods [7] usually model its context by the spatial distribution of other targets within its neighborhood. Such modeling implicitly assumes that the neighborhood context is reliable, which (a) (b) (c) Fig. 1. Spatial context (a b) and maximum consistency context (c). (a) Traffic scene in frame t. Circular-polar histogram centered at object A represents its spatial context. (b) Traffic scene in frame t + 1. Circular-polar diagram denotes the spatial context of object a. (c) Traffics in frame t and t + 1 are overlapped and shown in different color. Association (B,b) is the maximum consistency context of association (A,a). for example is true for a group of targets moving together. However, in traffic scenes it is common to see vehicles with opposite directions are spatially close to each other due to the juxtaposition of two-way lanes. Furthermore, the false positives in target detection often introduce noises into the spatial context. An example is illustrated in Fig. 1(a b). Motivated by the above observation, we propose a new model named maximum consistency context (MCC), which is a spatio-temporal context and is robust to noises in target neighborhood. For a potential association from two consecutive frames, e.g. (A, a) in Fig. 1(c), the idea is to extract context information from only the most consistent association in (A, a) s neighborhood, e.g. (B, b) in Fig. 1(c). With contextual information, MCC effectively reduces the disturbance from neighbor targets which either head in different directions or are false detections. Meanwhile, MCC provides strong discriminative features to guide multi-object association across frames. 2. RELATED WORK Multiple target tracing (MTT) is a widely studied topic in computer vision (e.g. [12, 13]). The focus of this paper is on the context modeling in MTT. Contextual information plays a critical role in visual tracing. Yang et al. [14] proposed to explore the auxiliary objects which have persistent co-occurrence and consistent motion correlation with the target object, to help localize and reacquire targets. However, it is hard to mine the auxiliary objects. In Reilly s wor [7],

2 Fig. 2. The framewor of wide area surveillance. the context is the representation of geometric relationships of objects with their respective neighbors. Grabner et al. [15] proposed to use local features, which have some temporal relations with the target, to predict the target position. However, the method is computationally expensive on finding and matching features, and it is neither suitable for traffic scene. In [11], Ali et al. propose to use similar objects to predict and reacquire targets after occlusion, but they assume object detections are good enough. Compared with aforementioned wor, the main contribution is the novel context model, which fits well the tas of tracing cloudy targets with mixed local motion patterns. 3. MULTIPLE VEHICLE TRACKING 3.1. Framewor Overview Following the general tracing-by-detection framewor [16, 17], we treat multi-vehicle tracing as a target association problem. We roughly divide the tracing framewor into three consecutive stages: preprocessing, detection, and association. The flow chart is summarized in Fig. 2. In this section we will briefly introduce the first and second stages. Then we will elaborate in detail the proposed context model and association algorithms, which are the focus of our wor. In the preprocessing, similar as in [7], we first mae use of registration for image alignment. In particular, Speeded Up Robust Features (SURF) [18] features are used for the efficiency and we then fit affine models for image warping. After the alignment, bacground modeling is achieved through a standard median filter. In the detection, instead of directly using the results from bacground subtraction, a vehicle classifier cascade is applied on the results in order to eliminate noises. The cascade is composed of two SVMs: the first SVM uses simple shape features (dimensions of a candidate target); and the second SVM uses the histogram of oriented gradient (HOG) [19] Multi-Object Association Some notations are given first. We denote the image sequences as {I t : t=1,...,n I }, such that I t is the frame at time t. The detected candidates at time t are denoted as O t = {o t i : i = 1,...,n i} containing n i candidate targets (objects). A target o is defined as a vector o=(x(o),h(o),θ(o),a(o)) or simply (x, h, θ, a), where x, h, θ and a denote the location, appearance histogram, orientation and area of o respectively. Without loss of generality, we assume the two frames to be associated are frames 1 and 2. To handle the missing targets and false detections in O 1 and O 2, we introduce dummy targets into the two sets and then assume they have the same number of targets, i.e., n 1 = n 2 = n. The multi-object association can be defined as to find the assignment = {π i,j } {0, 1} n n to maximize certain total association score, denoted as E(; O 1, O 2 ). The problem is formulated as max E(; O1, O 2 ) = max π ij (s ij + c ij ), (1) s.t. π ij =1; π ij =1; π ij {0, 1}; i, j {1,...,n}, (2) i=1 j=1 where π ij = 1 (or 0) indicates there is an (or no) association between o 1 i, o2 j ; s ij measures the affinity between o 1 i, o2 j ; and c ij represents the context similarity between o 1 i, o2 j. The association without context modeling can be viewed as a special case where c ij = 0, i, j. The association problem turns to a standard integer assignment, where the Hungarian algorithm [20] provides the optimum solution. The item s ij measures the similarity between targets o 1 i = (x i,h i,θ i,a i ) O 1 and o 2 j = (x j,h j,θ j,a j ) O 2, in terms of appearance, area and orientation. Specifically, it is defined as s ij = αs h,ij + βs o,ij + (1 α β)s a,ij, (3) where s h,s o and s a denote respectively for similarities in appearance, orientation and area; and α,β are weight factors Spatial context. At frame t, for a target candidates o t i = (x i, h i, θ i, a i ) O t, its spatial context (SC), denoted by SC t i, measures the spatial distribution of other candidates in O t. Specifically, it divides the neighborhood of o t i into n d n o distance-orientation bins, and SC t i is then defined as a weighted n d n o histogram as SC t i(p, q)= 1 exp ( (v i u pq ) Σ 1 (v i u pq ) ), (4) Z o N t i where Ni t = {ot :ot Ot, x i x 2 r} defines the neighborhood of o t i with radius r; v i=( x i x 2, atan2(x x i )) calculates the relative distance and orientation of o t with respect to o t i ; u pq=(p d, q θ) represents the bin (p, q) such that d and θ are the distance and angle interval respectively; Σ is the estimated covariance matrix; and, finally, Z is the normalization constant. An illustration of spatial context is shown in Fig. 1(a b). With SC t i, c ij in Eq. (1) is represented as c ij =sim(sc 1 i,sc 2 j), where sim(.,.) defines the similarity between two histograms, which is computed using histogram intersection [21] Maximum consistency context. The spatial context captures rich statistics that is powerful when the target s neighborhood remains stable over time.

3 However, in the wide area traffic environment, this can be violated since vehicles moving on opposite directions are often close to each other. Liewise, SC can be vulnerable by inaccurately taing such noises into account. In the following, we present an alternative context modeling, which captures only the most reliable information in a target s neighborhood. First, for two association pairs (o 1 i, o2 j ) and (o1 i, o2 j ), we measure the consistency between them as follows, φ ( (o 1 i,o 2 j),(o 1 i,o2 j )) =γ cos 2 (θ ij θ i j )+(1 γ)2l ijl i j l 2 ij +l2 i j, (5) where θ ij = atan2(x(o 1 i ) x(o2 j )),l ij = x(o 1 i ) x(o2 j ) 2 are the orientation and length of (o 1 i, o2 j ) respectively, θ i j, l i j have similar definitions; and γ is the weight parameter. With this definition, we define the maximum consistency context (MCC) for an association (o 1 i, o2 j ) as MCC((o 1 i,o 2 j),)= arg max φ ( (o 1 (o 1 i,o 2 j ) N (o 1 i,o2 j,) i,o 2 j),(o 1 i,o2 j )), (6) where N (o 1 i, o2 j, )={(o1 i, o2 j ):o1 i N i 1, o2 j N j 2, π i j = 1} defines the spatio-temporal neighborhood of (o 1 i, o2 j ), which depends on the association. The proposed MCC is more flexible as it does not request the majority consistency in a target s neighborhood. In contrast, it extracts context information only from the most reliable neighbor association. Such a scheme maes MCC robust to distractions of inconsistent motions in a target s neighborhood, e.g., the vehicles running in opposite lanes in the highway scenario. It also performs robustly against false detections, as illustrated in Fig. 1. To integrate MCC in trac association, we define c ij in Eq. (1) as c ij = φ ( (o 1 i,o2 j ), MCC(o1 i,o2 j )). As a result, we have the following MCC-based association problem max π ij (s ij + φ((o 1 i, o 2 j), MCC(o 1 i, o 2 j))). (7) Considering the fact that π ij {0, 1}, we can rewrite (7) as max π ij (s ij + max π i (o 1 i,o 2 j ) N (o 1 i,o2 j ) j φ((o1 i,o 2 j),(o 1 i,o2 j ))). (8) i,j=1 The formulation is a quadratic integer optimization with nonsmooth component (i.e., max ). The global optimal solution is unfortunately computationally expensive. In line with the interloced property of this formation, we devise an iterated algorithm to optimize it, which is given in Algorithm 1. Note that there are other approaches that can approximate the global solution for (8), such as simulated annealing and Monte Carlo methods. These solutions are however very time-consuming owing to their random property as well as the high dimensionality in our problem. Our approach, with a small number of iterations, i.e., n it = 10, efficiently generates excellent results as shown in our experiments. Algorithm 1 Multi-object association with MCC 1: Input: two consecutive candidate target sets O 1 and O 2 2: Output: association matrix 3: Calculate affinities S = {s ij } according to Eq. (3) 4: Initialization = 0;E max = 0;C = {c ij } = 0 5: for i = 1, 2,..., n it do 6: Compute the integrated similarities d ij = s ij + c ij 7: Solve Eq. (9) using the Hungarian algorithm max E(; O1, O 2 ) = max π ij d ij. (9) Denote the solution and score by and Ê respectively. 8: Update C using according to Eq. (6) 9: if Ê > Emax then 10: E max = Ê, =. 11: end if 12: end for 3.3. Multiple frame association We treat multi-frame tracing as a tas of associating short reliable traclets into long tracs, in a similar framewor used in [22, 23]. Two procedures, reliable traclets acquisition and traclet-to-traclet association, are performed iteratively. Reliable traclets are obtained by checing the motion smoothness of trajectories. Suppose a basic traclet is M1:t= {o 1, o2,..., ot }, it is reliable only if { l θt 1,t θt,t+1 t 1,t < θ 0 ; min lt,t+1, l } t,t+1 lt 1,t > l 0, (10) where θt 1,t,l t 1,t are the orientation and length of association (o t 1, o t ) respectively; θ 0 and l 0 are corresponding thresholds. In this way, if association (o t 1, o t ) has inconsistency with adjoining associations, trajectory M1:t is divided into two short traclets by breaing the association (o t 1, o t ). Traclet-to-traclet association follows the similar manner as two-frame multi-target association. First, the affinity between two traclets is constituted of appearance, motion and temporal similarities. Later, Hungarian algorithm is used here again to associate the two traclets into the long one. Finally, isolated detections and too short traclets after multi-frame association are discarded as false alarms. 4. EXPERIMENT Our experiments are conducted on the CLIF dataset [24]. CLIF is challenging with following features. 1) Large image format ( ); 2) Large camera and target motion; 3)Tiny target occupancy (4 70 pixels); 4) Similar target appearance; 5) Low frame rate sampling (2 fps); and 6) A large mount of targets (hundreds). Three sequences with 80 frames (40 seconds), 50 frames (25 seconds) and 100 frames (50 seconds) respectively are used to evaluate the proposed approach. To study the effectiveness of the proposed context, we

4 NoCon SC MCC Seq Seq Seq Table 1. Precisions of multi-object association. : appearance features are used; and : no appearance features are used. Fig. 3. Associations with different contexts on Sequence 1. Top: NoCon. Middle: SC. Bottom: MCC. Blac(White) rectangle: detection in the last(current) frame. Red(blue) line: rightward(leftward) association output. evaluate three types of associations: association without context (NoCon), with the spatial context (SC) and with the proposed maximum consistency context (MCC). Furthermore, for each method, we test two different affinity models: with appearance features and without. In all cases, Hungarian algorithm is used as the basic assignment solution. The parameters are set as follows: γ in Eq. (5) is set as 0.5 in the first sequence and 0.8 in the other two sequences. If appearance feature is used, both α and β in Eq. (3) are set as 0.3 in all sequences. Otherwise, α and β are set as 0.5 equally. The maximum iteration number (nit in Algorithm 1) is 10. We use the precision, 100 t c(t)/ t g(t), to measure the association performances. c(t) and g(t) in the precision measure denote the numbers of correct and groundtruth association respectively. Results are summarized in Table 1. From the results, we have the following observations. First, the proposed MCC performs the best in general. Furthermore, if no appearance is used in the association, all methods have degenerated performances, yet our method is affected least. This confirms that the proposed context plays an important role in handling the motion ambiguity. Second, spatial context helps little in performance. This seemingly contradicting phenomenon can be attributed to two factors: (1) the scenes we selected contain mainly two-direction highways, which largely confuses local context as we conjectured; and (2) the vehicle detection is rather noisy, leading to unstable spatial distributions. Third, the method without using context obtains the moderate results, which attributes to inds of well-designed affinity measures. However, when excluding the appearance information, the performance drop of NoCon is much larger than those of SC and MCC. Qualitative results of three methods on sequence 1 are shown in Fig. 3, which is a snapshot of two-frame association. Fig. 4. Multiple object tracing results, Top: sequence 3; Middle: sequence 2; Bottom: sequence 1 Associations with SC are distracted by those cars moving in opposite directions. While NoCon is vulnerable to ambient similar objects, after embedding MCC, the motion ambiguity is greatly alleviated. With the help of neighbor association, confused object succeeds in finding true associated target, it can be seen from Fig. 3. Results of multiple object tracing are shown in Fig. 4. It can be seen that, our approach has two merits. First, there are few wrong two-frame association. Second, our approach associates the trac fragments into long tracs, which is especially useful in the case of occlusion and missing detections. We do not have quantitative evaluation on multiple object tracing, as no groundtruth data is available. The labeling of associations in wide area traffic scenes is a very time consuming wor, which we will consider in the future. 5. CONCLUSION In this paper, we propose a novel spatial-temporal context, maximum consistency context (MCC), to assist multi-object tracing in wide area traffic scenes. By picing the most reliable ingredient in the spatial-temporal neighborhood of an association, MCC leverages the discriminative power and robustness against clutter distractions. Experiments using challenging wide area surveillance videos validate the effectiveness of the proposed approach. Acnowledgement. This wor is partly supported by NSFC (Grant No ), the National 863 High-Tech RD Program of China (Grant No.2012AA012504), the Natural Science Foundation of Beijing (Grant No ), and Guangdong Natural Science Foundation (Grant No.S ). Ling was supported in part by NSF Grant IIS

5 6. REFERENCES [1] S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool, You ll never wal alone: Modeling social behavior for multi-target tracing, IEEE International Conference on Computer Vision, pp , [2] J. Xiao, H. Cheng, F. Han, and H. Sawhney, Geo-spatial aerial video processing for scene understanding and object tracing, IEEE Conference on Computer Vision and Pattern Recognition, pp. 1 8, [3] J. Xiao, H. Cheng, H.S. Sawhney, and F. Han, Vehicle detection and tracing in wide field-of-view aerial video, in CVPR, [4] E. Blasch, G. Seetharaman, K. Palaniappan, H. Ling, and G. Chen, Wide-area motion imagery (wami) exploitation tools for enhanced situation awareness, in Proc. IEEE Applied Imagery Pattern Recognition (AIPR) Worshop: Computer Vision: Time for Change, [5] J. Proaj, X. Zhao, and G.G. Medioni, Tracing many vehicles in wide area aerial surveillance, in CVPR Worshops, 2012, pp [6] J. Proaj and G. Medioni, Using 3d scene structure to improve tracing, IEEE Conference on Computer Vision and Pattern Recognition, pp , [7] V. Reilly, H. Idrees, and M. Shah, Detection and tracing of large number of targets in wide area surveillance, European Conference on Computer Vision, pp , [8] H. Ling, Y. Wu, E. Blasch, G. Chen, and L. Bai, Evaluation of visual tracing in extremely low frame rate wide area motion imagery, in Proc. of the Int s Conf. on Information Fusion (FUSION), [9] K. Palaniappan, F. Bunya, P. Kumar, I. Ersoy, S. Jaeger, K. Ganguli, A. Haridas, J. Fraser, R.M. Rao, and G. Seetharaman, Efficient feature extraction and lielihood fusion for vehicle tracing in low frame rate airborne video, in Proc. of the International Conference on Information Fusion (FU- SION), [10] X. Shi, H. Ling, E. Blasch, and W. Hu, Context-driven moving vehicle detection in wide area motion imagery, in Int l Conf. on Pattern Recognition (ICPR), [11] S. Ali, V. Reilly, and M. Shah, Motion and appearance contexts for tracing and re-acquiring targets in aerial videos, IEEE Conference on Computer Vision and Pattern Recognition, pp. 1 6, [12] Y. Bar-Shalom and T. Fortmann, Tracing and data association, Academic Press., [13] D. Reid, An algorithm for tracing multiple targets, TAC, vol. 24, no. 6, pp , [14] M. Yang, Y. Wu, and G. Hua, Context-aware visual tracing, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 7, pp , [15] H. Grabner, J. Matas, L. Van Gool, and P. Cattin, Tracing the invisible: Learning where the object might be, IEEE Conference on Computer Vision and Pattern Recognition, pp , [16] M. Andrilua, S. Roth, and B. Schiele, People-tracing-bydetection and people-detection-by-tracing, IEEE Conference on Computer Vision and Pattern Recognition, pp. 1 8, [17] M.D. Breitenstein, F. Reichlin, B. Leibe, E. Koller-Meier, and L. Van Gool, Online multiperson tracing-by-detection from a single, uncalibrated camera, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 9, pp , [18] H. Bay, T. Tuytelaars, and L. Van Gool, Surf: Speeded up robust features, European Conference on Computer Vision, pp , [19] N. Dalal and B. Triggs, Histograms of oriented gradients for human detection, IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp , [20] H.W. Kuhn, The hungarian method for the assignment problem, Naval research logistics quarterly, vol. 2, no. 1-2, pp , [21] M.J. Swain and D.H. Ballard, Color indexing, International journal of computer vision, vol. 7, no. 1, pp , [22] C. Huang, B. Wu, and R. Nevatia, Robust object tracing by hierarchical association of detection responses, pp , [23] C.H. Kuo, C. Huang, and R. Nevatia, Multi-target tracing by on-line learned discriminative appearance models, IEEE Conference on Computer Vision and Pattern Recognition, pp , [24] Afrl: Columbus large image format (clif) 2006.,