Outline Data Association Scenarios Track Filtering and Gating Global Nearest Neighbor (GNN) Review: Linear Assignment Problem Murthy s k-best Assignments Algorithm Probabilistic Data Association (PDAF) Joint Probabilistic Data Assoc () Multi-Hypothesis Tracking (MHT) Data Association Scenarios Two-frame Matching (Correspondence Problem) Matching features across frames Data Association Scenarios Two-frame Matching (Correspondence Problem) e.g. corners, Sift keys, image patches Data Association Scenarios Multi-frame Matching (matching in a new frame to a set of tracked trajectories) Match up detected blobs across frames track How to determine which to add to which track Tracking Matching Intuition: predict next position along each track. Tracking Matching Intuition: predict next position along each track. Intuition: match should be close to predicted position. d track How to determine which to add to which track track d d d d5 How to determine which to add to which track
Tracking Matching Intuition: predict next position along each track. Intuition: match should be close to predicted position. Intuition: some matches are highly unlikely. track d d d Gating A method for pruning matches that are geometrically unlikely from the start. Allows us to decompose matching into smaller subproblems. track gating region gating region How to determine which to add to which track How to determine which to add to which track Filtering Framework Filtering, Gating, Association Recall our earlier discussion of state space filtering We want to recursively estimate the current state at every time that a measurement is received. Two step approach: ) prediction: propagate state pdf forward in time, taking process noise into account (translate, deform, and spread the pdf) ) update: use Bayes theorem to modify prediction pdf based on current measurement But which observation should we update with Add Gating and Data Association ) prediction: propagate state pdf forward in time, taking process noise into account (translate, deform, and spread the pdf) ) Gating to determine possible matching ) Data association to determine best match ) update: use Bayes theorem to modify prediction pdf based on current measurement Prediction Kalman Filter Using Kalman filtering as a concrete example. System model and measurement model are linear. Noise is zero-mean Gaussian Pdfs are all Gaussian All pdfs are then Gaussian. (note: all marginals of a Gaussian are Gaussian) ) System model ) Measurement model p(v k ) = N(v k, Q k ) p(n k ) = N(n k, R k )
Kalman Filter Example ellipsoidal gating region Simpler Prediction/Gating Aside: Camera Motion Constant position + bound on maximum interframe motion r r constant position prediction Hypothesis: constant velocity target motion model is adequate provided we first compensate for effects of any background camera motion. Three-frame constant velocity prediction p k- (p k -p k- ) p k typically, gating region can be smaller prediction p k + (p k -p k- ) Camera Motion Estimation Approach: Estimate sparse optic flow using Lucas-Kanade algorithm (KLT) Estimate parameteric model (affine) of scene image motion Note: this offers a low computational cost alternative to image warping and frame differencing approaches. Parameteric Camera Motion Model Apparent motion of stationary scene pixels in the image is a function of camera motion (R,T) and scene structure (depth at each pixel). Assumption: for small field of view aerial camera, viewing a target on the ground, apparent scene motion in a subsequence can be modeled as low-parameter, global image transformation e.g. 6 parameter affine or 8 parameter projective (note: this assumption has been demonstrated to be good based on the success of Sarnoff s image stabilization work. Main difference: they are estimating over each pixel to do explicit image warping. We will estimate from a sparse flow field, and do NOT do any warping). To estimate a set of global flow parameters from possibly noisy flow vectors, we can use a robust sampling estimation method such as RANSAC or least median-of-squares. used for motion prediction, and zoom detection
Samples of Affine Flow Fitting affine flow original flow Samples of Affine Flow Fitting affine flow original flow compensated flow compensated flow Target Motion Estimation Validation Approach: Constant velocity estimate, after compensating for camera motion P T f = target position in frame f f = camera motion from frame f to frame g g P =T t P t t t * [ Pt + ( Pt (T t * Pt ))] T t t t T t t P t P Now back to data association... t Global Nearest Neighbor (GNN) Evaluate each observation in track gating region. Choose best one to incorporate into track. Global Nearest Neighbor (GNN) Evaluate each observation in track gating region. Choose best one to incorporate into track. ai o track o o o o track o o o. 5. 6. 9. aj = score for matching observation j to track ai = score for matching observation i to track Could be based on Euclidean or Mahalanobis distance to predicted location (e.g. exp{-d}). Could be based on similarity of appearance (e.g. appearance template correlation score) Choose best match am = max{a, a,a,a} max
Global Nearest Neighbor (GNN) Recall: Linear Assignment Problem Problem: if do independently for each track, could end up with contention for the same. track o o o o o 5 ai ai. 5. 6.. 9. 8. 5. We have N objects in previous frame and M objects in current frame. We can build a table of match scores m(i,j) for i=...n and j=...m. For now, assume M=N. 5 5 track both try to claim observation o problem: choose a - correspondence that maximizes sum of match scores. Assignment Problem Example: Mathematical definition. Given an NxN array of benefits {X ai }, determine an NxN permutation matrix M ai that maximizes the total score: maximize: E = N N 5x5 matrix of match scores subject to: constraints that say M is a permutation matrix working from left to right, choose one number from each column, making sure you don t choose a number from a row that already has a number chosen in it. The permutation matrix ensures that we can only choose one number from each row and from each column. How many ways can we do this 5 x x x x = (N factorial) Review: SoftAssign SoftAssign We are going to use an efficient approach called SoftAssign, based on the work of J. Kosowsky and A. Yuille. The invisible hand algorithm: Solving the assignment problem with statistical physics. Neural Networks, 7:77-9, 99. [Sinkhorn] Main points: relax, constraint to be <= Mai <= init with Mai = exp(b*score). This ensures positivity and also spreads out scores as (B approaches infinity) perform repeat row and col normalizations to get a doubly stochastic matrix (rows and cols sum to ) Discuss iterative version to avoid numerical issues with large B. 5
X ai (benefits) SoftAssign B= B=5 SoftAssign B= B=5 B= permutation matrix computed by SoftAssign score:.6 In this example, we can exhaustively search all assignments. The global maximum is indeed.6 permutation matrix! Handling Missing Matches Example Typically, there will be a different number of tracks than. Some may not match any track. Some tracks may not have any. Introduce one row and one column of slack variables to absorb any outlier mismatches. 5 Matches A B C D A B C D slack 5 slack Important: when doing Sinkhorn iterations, don t normalize the slack row and column. track. 5. 6.. 9. 8. 5. track Another Strategy 5x pad with array of small random numbers to get a square score matrix. track track permutation matrix computed by SoftAssign K-Best Assignment score:.6 NxN softassign 5 5x ignore whatever happens in here So far we know how to find the best assignment (max sum scores). But what if we also want to know the second best Or maybe the top best assignments 6
Murty s K-Best Assignments st sweep General Idea. Start with best assignment. Start methodically tweaking it by toggling matches in and out of the assignment Maintain a sorted list of best assignments so far During each iterative sweep, toggle the matches in the next best assignment The K best assignments are found in decreasing order, one per sweep constraints ~(,) (5,)(,),(,),(,),(,5) score. (,),~(,) second best solution (,)(5,),(,),(,),(,5) score.88 solution: (,)(,),(,),(5,),(,5) constraints: none (,)(,),~(,) (,)(,),(,),~(5,) (,)(,),(,),(5,),(,5) score.9 (,)(,),(,),(,),(5,5) score.6 nd sweep st scan, different order solution: (5,)(,),(,),(,),(,5) constraints: ~(,) solution: (,)(,),(,),(5,),(,5) constraints: none constraints constraints ~(,),~(5,) (,)(,),(,),(5,),(,5) score.7 ~(,),(5,),~(,) (5,)(,),(,),(,),(,5) score.8 ~(,)(5,),(,),~(,) ~(,)(5,),(,),(,),~(,) (5,)(,),(,),(,),(,5) score.66 (5,)(,),(,),(,),(,5) score.66 ~(,5) (,)(,),(,),(5,),(,5) score.9 (,5),~(5,) (5,)(,),(,),(,),(,5) score. (,5)(5,),~(,) (,5)(5,),(,),~(,) (,)(,),(,),(5,),(,5) score.85 (,)(,),(,),(5,),(,5) score.7 third best solution second best solution is found again. PDAF Recall: Global Nearest Neighbor Probabilistic Data Association Filter Updating single track based on new. General idea: Instead of matching a single best observation to the track, we update based on all (in gating window), weighted by their likelihoods. Evaluate each observation in track gating region. Choose best one to incorporate into track. track o o o o ai. 5. 6. 9. max a i = score for matching observation i to track Choose best match a m = max{a, a,a,a } 7
PDAF Consider all points in gating window. Also consider the additional possibility that no match. track o o o o ai.. 5. 6. 9. PDAF Kalman filter update is based on residual vector (diff between predicted location and observed location) When using single best observation PDAF uses weighted combination of p i = probability of matching observation i to track note: if we weren t consider the possibility of no match, this would exactly be the diff between the weighted center of mass of and the predicted location PDAF Computation of Kalman posterior covariance must change too, to incorporate weighted matches and possibility of no match. Typical computation when single match is used: PDAF computation: no match, no update update if single match Joint Probabilistic Data Association Filter If maintaining multiple tracks, doing PDAF on each one independently is nonoptimal, since in overlapping gate regions will be counted more than once (contribute to more than one track). reasons over possible combinations of matches, in a principled way. correction term to reflect uncertain association (spread of possible matches or no match) Example (from Blackman and Popoli). track: track: + Oo = no match Possible assignments: (i,j) = assign i to track, j to track (, ) (, ) (, ) (, ) (, ) (, ) (, ) (, ) don t assign observation (, ) (, ) (, ) (, ) more than once 8
Each possible (non-conflicting) assignment becomes a hypothesis with an associated probability. Now compute probability p ij that each observation i should be assigned to track j, by adding probabilities of assignments where that is so. Example: p = prob that observation should be assigned to track..86 +.6 +.9.6 Continuing the example: Track probabilities p =.8 p =.6 p =.98 p =.87 Track probabilities p =.69 p =. p =.5 p =.6 PDAF filter for track PDAF filter for Note: running PDAF filters on each track independently is now OK because any inconsistency (double counting) has been removed. 9