Computer Vision II Lecture 4

Transcription

1 Course Outline Computer Vision II Lecture 4 Single-Object Tracking Background modeling Template based tracking Color based Tracking Color based tracking Contour based tracking Tracking by online classification Tracking-by-detection Bayesian Filtering Bastian Leibe RWTH Aachen Multi-Object Tracking Articulated Tracking leibe@vision.rwth-aachen.de 2 Image source: Robert Collins Recap: Estimating Optical Flow Recap: Lucas-Kanade Optical Flow Use all piels in a K K window to get more equations. Least squares problem: I(,y,t 1) I(,y,t) Optical Flow Given two subsequent frames, estimate the apparent motion field u(,y) and v(,y) between them. Key assumptions Brightness constancy: projection of the same point looks the same in every frame. Small motion: points do not move very far. Spatial coherence: points move like their neighbors. Slide credit: Svetlana Lazebnik 3 Minimum least squares solution given by solution of Slide adapted from Svetlana Lazebnik Recall the Harris detector! 4 Recap: Iterative Refinement Recap: Coarse-to-fine Optical Flow Estimation Estimate velocity at each piel using one iteration of LK estimation. Warp one image toward the other using the estimated flow field. Refine estimate by repeating the process. estimate update estimate update estimate update 0 0 Initial guess: Estimate: Initial guess: Estimate: Initial guess: u=1.25 piels u=2.5 piels u=5 piels Estimate: Iterative procedure Results in subpiel accurate localization. Converges for small displacements. 0 Image 1 u=10 piels Image 2 Gaussian pyramid of image 1 Gaussian pyramid of image 2 Slide adapted from Steve Seitz 0 5 Slide credit: Steve Seitz 6 1

2 Recap: Coarse-to-fine Optical Flow Estimation Recap: Shi-Tomasi Feature Tracker ( KLT) Run iterative LK Warp & upsample Run iterative LK... Idea Find good features using eigenvalues of second-moment matri Key idea: good features to track are the ones that can be tracked reliably. Frame-to-frame tracking Track with LK and a pure translation motion model. More robust for small displacements, can be estimated from smaller neighborhoods (e.g., 5 5 piels). Image 1 Image 2 Checking consistency of tracks Affine registration to the first observed feature instance. Affine model is more accurate for larger displacements. Comparing to the first frame helps to minimize drift. Gaussian pyramid of image 1 Gaussian pyramid of image 2 Slide credit: Steve Seitz 7 J. Shi and C. Tomasi. Good Features to Track. CVPR Slide credit: Svetlana Lazebnik 8 Recap: General LK Image Registration Recap: Step-by-Step Derivation Goal Find the warping parameters p that minimize the sum-ofsquares intensity difference between the template image T() and the warped input image I(W(;p)). LK formulation Formulate this as an optimization problem X 2 arg min I(W(; p)) T() p We assume that an initial estimate of p is known and iteratively solve for increments to the parameters p: X 2 arg min I(W(; p + p)) T() p 9 Key to the derivation Taylor epansion around p I(W(; p + p)) ¼ I(W(; p)) p+o( p2 ) = I(W([; y]; p 1 ; : : : ; p n )) h i : : @Wy : : 4 @pn p n Gradient Jacobian Increment parameters to solve for 10 Recap: LK Algorithm Iterate Warp I to obtain I(W([, y]; p)) Compute the error image T([, y]) I(W([, y]; p)) Warp the gradient ri with W([, y]; p) Evaluate at ([, y]; p) (Jacobian) Compute steepest descent images Compute Hessian matri H = P h i T h @p P h i T T([; y]) I(W([; y]; p)) Compute p = H 1 P h i T y]) I(W([; y]; p)) Update the parameters p à p + p Until p magnitude is negligible 11 [S. Baker, I. Matthews, IJCV 04] Recap: LK Algorithm Visualization 12 [S. Baker, I. Matthews, IJCV 04] 2

3 Eample of a More Comple Warping Function Today: Color based Tracking Encode geometric constraints into region tracking Constrained homography transformation model Translation parallel to the ground plane Rotation around the ground plane normal W() = W obj P W t W α Q Input for high-level tracker with car steering model. 13 [E. Horbert, D. Mitzel,, DAGM 10] 14 Image source: Robert Collins Topics of This Lecture Mean-Shift Mean-Shift Mean-shift mode estimation Using mean-shift on color images Mean-Shift with Eplicit Weight Images Histogram backprojection CAMshift approach Mean-Shift with Implicit Weight Images Comaniciu s approach Bhattacharyya distance Gradient ascent Comparison Qualitative intuition 15 Mean-Shift Tracking Efficient approach to tracking objects whose appearance is defined by color. Actually, the approach is not limited to color. Can also use teture, motion, etc. Popular use for object tracking Very simple to implement Non-parametric method, does not make strong assumptions about the shape of the distribution Suitable for non-static distributions (as typical in tracking) Can be combined with dynamic models (Kalman filters, etc.) Good performance in practice 16 3

4 Using Mean-Shift on Color Models Two main approaches 1. Eplicit weight images Create a color likelihood image, with piels weighted by the similarity to the desired color (best for unicolored objects). Use mean-shift to find spatial modes of the likelihood. 2. Implicit weight images Represent color distribution by a histogram. Use mean-shift to find the region that has the most similar color distribution. 24 4

5 Topics of This Lecture Mean-Shift on Weight Images Mean-Shift Mean-shift mode estimation Using mean-shift on color images Ideal case Want an indicator function that returns 1 for piels on the tracked object and 0 for all other piels. Mean-Shift with Eplicit Weight Images Histogram backprojection CAMshift approach Mean-Shift with Implicit Weight Images Comaniciu s approach Bhattacharyya distance Gradient ascent Comparison Qualitative intuition 25 Instead Compute likelihood maps Value at a piel is proportional to the likelihood that the piel comes from the tracked object. Likelihood can be based on Color Teture Shape (boundary) Predicted location 26 Mean-Shift Tracking Mean-Shift Tracking A closer look at the procedure... Kernel weight evaluated at offset (a ) Weight from the likelihood image at piel a Offset of piel a to kernel center Idea Let piels form a uniform grid of data points. Each piel has a weight proportional to the likelihood that the piel is on the object we want to track. Perform standard mean-shift using the weighted set of points. 27 Image source: Robert Collins Sum over all piels a under kernel K Normalization term Mean-shift computes the weighted mean of all shifts (offsets), weighted by the likelihood under the kernel function. 28 Duality Property Eample: Face Tracking using Mean-Shift Duality Running mean-shift with kernel K on weight image w is equivalent to performing gradient ascent in a (virtual) image formed by convolving w by some shadow kernel H. Note: mode we are looking for is mode of location (,y) likelihood, NOT mode of color distribution. 29 Image source: Robert Collins G. Bradski, Computer Vision Face Tracking for use in a Perceptual User Interface, IEEE Workshop On Applications of Computer Vision, Princeton, NJ, 1998, pp Image source: Gary Bradski 5

6 Eplicit Weight Images Side Note: Color Histograms for Recognition Using color histograms for recognition Works surprisingly well In the first paper (1991), 66 objects could be recognized almost without errors Histogram backprojection Histogram is an empirical estimate of p(color object) = p(c o) Bayes rule says: Simplistic approimation: assume p(o)/p(c) is constant. Use histogram h as a lookup table to set piel values in the weight image. If piel maps to histogram bucket i, set weight for piel to h(i). 31 Image source: Gary Bradski 32 [Swain & Ballard, 1991] Localization by Histogram Backprojection Object Localization Results Where in the image are the colors we re looking for? Query: object with histogram M Given: image with histogram I Compute the ratio histogram : R reveals how important an object color is, relative to the current image. Color is frequent on the object large M i Color is frequent in the image large I i This value is projected back into the image (i.e. the image values are replaced by the values of R that they inde). The result image is convolved with a circular mask the size of the target object. Peaks in the convolved image indicate detected objects. Does this sound familiar? 33 Eample result after backprojection Looking for blue pullover 34 [Swain & Ballard, 1991] Bradski s CAMshift Visualization: Bradski s CAMshift in Action Idea Find,y location of mode by mean-shift. Determine z, roll angle µ by fitting an ellipse to the mode found using mean-shift. 35 Image source: Gary Bradski 36 Image source: 6

7 Problem: Scale Changes Visualization: Scale Adaptation in CAMshift Window always has the same size When the object size changes, does not fit anymore Tracking soon diverges Image source: 38 Image source: CAMShift Results Applications: Perceptual User Interfaces Face tracking Using a skin color model in HSV color space 39 Video source: Gary Bradski Head tracking as input modality Controlling a flight simulator by head gestures 40 Video source: Gary Bradski Topics of This Lecture Using Mean-Shift on Color Models Mean-Shift Mean-shift mode estimation Using mean-shift on color images Mean-Shift with Eplicit Weight Images Histogram backprojection CAMshift approach Two main approaches 1. Eplicit weight images Create a color likelihood image, with piels weighted by the similarity to the desired color (best for unicolored objects). Use mean-shift to find spatial modes of the likelihood. Mean-Shift with Implicit Weight Images Comaniciu s approach Bhattacharyya distance Gradient ascent 2. Implicit weight images Represent color distribution by a histogram. Use mean-shift to find the region that has the most similar color distribution. Comparison Qualitative intuition

8 Implicit Weight Images Mean-Shift Object Tracking Sometimes the weight is not eplicitly created Eample: Mean-shift Tracking by Comaniciu et al. Weight is embedded into the matching procedure Comes out as a side effect of matching two pdfs. Main idea: Match the pdf of the target object Interesting consequence Implicit weight image changes between iterations of mean-shift, as compared to iterating to convergence on an eplicit weight image! We ll take a look at their approach and see how this works. D. Comaniciu, V. Ramesh, P. Meer. Kernel-Based Object Tracking, PAMI, Vol. 25(5), pp , Mean-Shift Object Tracking Approach Color histogram representation Measuring distances between histograms Distance as a function of window location y where is the Bhattacharyya coefficient Approach Finding the Object Compute histograms via Parzen estimation Goal: Find the location y that maimizes the Bhattacharyya coefficient Taylor epansion around current values p u (y 0 ) where k( ) is some radially symmetric smoothing kernel profile, i is the piel at location i, and b( i ) is the inde of its bin in the quantized feature space. Consequence of this formulation Gathers a histogram over a neighborhood Also allows interpolation of histograms centered around an off-lattice location. 48 where This does not depend on y Just need to maimize this. Note: It s a KDE!!! 49 8

9 Finding the Object Taylor epansion around current values p u (y 0 ) Finding the Object At each iteration, perform This does not depend on y Just need to maimize this. Note: It s a KDE!!! Find the mode of the second term by mean-shift iterations which is just standard mean-shift on (implicit) weight image w i. Let s look at the weight image more closely. For each piel i If piel i s value maps to histogram bucket B, then This is only 1 once in the summation Finding the Object Results: Mean-Shift Tracking Summary If model histogram is q 1, q 2,..., q m and current data histogram is p 1, p 2,..., p m Form weights q 1 /p 1, q 2 /p 2,..., q m /p m Do histogram backprojection of these values into the image to get the weight image w i. (Note: this is done implicitly) Note In each iteration, p 1, p 2,..., p m change, and therefore so does the weight image w i. Different from applying mean-shift to fied likelihood image! Configuration Feature space: quantized RGB Target manually selected in 1 st frame Average mean-shift iterations per frame: 4 D. Comaniciu, V. Ramesh, P. Meer. Kernel-Based Object Tracking, PAMI, Vol. 25(5), pp , Slide adapted from Ukrainitz & Sarel 53 Video source: Dorin Comaniciu Results: Mean-Shift Tracking Topics of This Lecture Difficulties Mean-Shift Mean-shift mode estimation Using mean-shift on color images Partial occlusion Distraction Motion blur Mean-shift still performs robustly despite those. Slide adapted from Ukrainitz & Sarel 54 Mean-Shift with Eplicit Weight Images Histogram backprojection CAMshift approach Mean-Shift with Implicit Weight Images Comaniciu s approach Bhattacharyya distance Gradient ascent Comparison Qualitative intuition 55 9

10 Qualitative Intuition Qualitative Intuition Bradski s Mean-Shift procedure Comaniciu s approach Assume that an object is 60% red and 40% green. I.e., q 1 = 0.6, q 2 = 0.4, q i = 0 for all other i. Let s assume the data histogram is perfectly located q 1 = 0.6, q 2 = 0.4, q i = 0 for all other i. p 1 = 0.6, p 2 = 0.4, p i = 0 for all other i. w 1 = sqrt(0.6/0.6), w 2 = sqrt(0.4/0.4), w i = 0 for all other i. If we just did histogram backprojection of these likelihood values (a la Bradski), we would get this weight image: Resulting weight image: Mean-shift does a weighted center-of- computation at each iteration. Much better! Window will be biased towards the region of red piels, since they have higher weight! Perfect object indicator function References and Further Reading The original CAMshift paper G. Bradski, Computer Vision Face Tracking for use in a Perceptual User Interface, IEEE Workshop On Applications of Computer Vision, Princeton, NJ, 1998, pp The Mean-Shift Tracking paper by Comaniciu D. Comaniciu, V. Ramesh, P. Meer. Kernel-Based Object Tracking, PAMI, Vol. 25(5), pp ,