OPPA European Social Fund Prague & EU: We invest in your future.

Transcription

1 OPPA European Social Fund Prague & EU: We invest in your future.

2 Patch tracking based on comparing its piels 1 Tomáš Svoboda, svoboda@cmp.felk.cvut.cz Czech Technical University in Prague, Center for Machine Perception Last update: March 22, 2010 Talk Outline comparing patch piels normalized cross-correlation, ssd... KLT - gradient based optimization good features to track 1 Please note that the lecture will be accompanied be several sketches and derivations on the blackboard and few live-interactive demos in Matlab What is the problem? 2/37 : CTU campus, door of G building Tracking of dense sequences camera motion I J 3/37

3 Tracking of dense sequences object motion I J 4/37 Alignment of an image (patch) 5/37 Goal is to align a template image T () to an input image I(). column vector containing image coordinates [, y]. The I() could be also a small subwindow withing an image. How to measure the aligment? What is the best criterial function? 6/37 How to find the best match, in other words how to find etremum of the criterial function? Criterial function What are the desired properties (on a certain domain)? conve (remember the optimization course?) discriminative...

4 Normalized cross-correlation You may know it as correlation coefficients 7/37 r ij = c ij cii c jj where c i,j are elements of the covariance matri Having template T (, y) and image I(, y), r(, y) = k ( ) ( ) l T (k, l) T I( + k, y + l) I(, y) ( I( + k, y + l) I(, y) ( ) 2 k l T (k, l) T k l ) 2 Be careful about coordinate systems (sketch on blackboard) Normalized cross-correlation in picture 8/37 criterial function ncc well, definitely not conve but the discriminability looks promissing very efficient in computation, see [3]. Sum of squared differences 9/37 ssd(, y) = k (T (k, l) I( + k, y + l)) 2 l criterial function ssd

5 Sum of absolute differences sad(, y) = XX k l 10/37 T (k, l) I( + k, y + l) criterial function sad SAD for the door part 11/37 criterial function sad SAD for the door part truncated 12/37 criterial function sad_truncated Differences greater than 20 intensity levels are counted as

6 Normalized cross-correlation: how it works 13/37 live demo for various patches Normalized cross-correlation: tracking 14/37 What went wrong? Why did it failed? Suggestions for improvement? Iterations? Tracking as an optimization problem finding etrema of a criterial function... 15/37... sounds like an optimization problem Kanade Lucas Tomasi (KLT) tracker Iteratively minimizes sum of square differences. It is a Gauss-Newton gradient descent non-linear optimization algorithm.

7 Importance in Computer Vision Firstly published in 1981 as an image registration method [4]. 16/37 Improved many times, most importantly by Carlo Tomasi [5, 6] Free implementation(s) available2. Also part of the OpenCV library3. After more than two decades, a project4 at CMU dedicated to this single algorithm and results published in a premium journal [1]. Part of plethora computer vision algorithms. Our eplanation follows mainly the paper [1]. It is a good reading for those who are also interested in alternative solutions Original Lucas-Kanade algorithm I Goal is to align a template image T () to an input image I(). column vector containing image coordinates [, y]. The I() could be also a small subwindow withing an image. Set of allowable warps W(; p), where p is a vector of parameters. For translations [ ] + p1 W(; p) = y + p 2 W(; p) can be arbitrarily comple The best alignment, p, minimizes image dissimilarity [I(W(; p)) T ()] 2 17/37 Original Lucas-Kanade algorithm II 18/37 [I(W(; p)) T ()] 2 is a nonlinear optimization! The warp W(; p) may be linear but the piels value are, in general, non-linear. In fact, they are essentially unrelated to. It is assumed that some p is known and best increment p is sought. The the modified problem [I(W(; p + p)) T ()] 2 is solved with respect to p. When found then p gets updated p p + p

8 Original Lucas-Kanade algorithm III 19/37 [I(W(; p + p)) T ()] 2 linearized by performing first order Taylor epansion 5 [I(W(; p)) + I W p T ()]2 I = [ I, I y ] is the gradient image6 computed at W(; p). The term W is the Jacobian of the warp. 5 Detailed eplanation on the blackboard. 6 As a vector it should have been a column wise oriented. However, for sake of clarity of equations row vector is eceptionally considered here. Derive Original Lucas-Kanade algorithm IV [I(W(; p)) + I W p T ()]2 with respect to p 2 [ I W ] [ I(W(; p)) + I W ] p T () 20/37 setting equality to zero yields p = H [ 1 I W ] [T () I(W(; p))] where H is (Gauss-Newton) approimation of Hessian matri. H = [ I W ] [ I W ] Iterate: The Lucas-Kanade algorithm Summary 21/37 1. Warp I with W(; p) 2. Warp the gradient I with W(; p) 3. Evaluate the Jacobian W image I W 4. Compute the H = 5. Compute p = H 1 [ at (; p) and compute the steepest descent I W [ ] [ I W 6. Update the parameters p p + p until p ɛ I W ] ] [T () I(W(; p))]

9 Eample of convergence 22/37 Eample of convergence 23/37 Convergence : Initial state is within the basin of attraction Eample of divergence 24/37 Divergence : Initial state is outside the basin of attraction

10 Eample on-line demo 25/37 Let play and see... What are good features (windows) to track? How to select good templates T () for image registration, object tracking. 26/37 p = H 1 [ I W ] [T () I(W(; p))] where H is the matri H = [ I W ] [ I W ] The stability of the iteration is mainly influenced by the inverse of Hessian. We can study its eigenvalues. Consequently, the criterion of a good feature window is min(λ 1, λ 2 ) > λ min (teturedness). What are good features (windows) to track? [ + p1 Consider translation W(; p) = y + p [ ] 2 W 1 0 = 0 1 [ H = = [ = I W ( I I I y ] [ ] [ I I y ) 2 I I y ( ]. The Jacobian is then I W I y ] ) 2 ] [ I, I ] [ ] 27/37 The image windows with varying derivatives in both directions. Homeogeneous areas are clearly not suitable. Teture oriented mostly in one direction only would cause instability for this translation.

11 What are the good points for translations? The matri H= X I 2 I I y I I y 2 I y 28/37 Should have large eigenvalues. We have seen the matri already, where? Harris corner detector [2]! The matri is sometimes called Harris matri. Eperiments - no occlusions 29/37 Eperiments - occlusions 30/37

12 Eperiments - occlusions with dissimilarity 31/37 Eperiments - object motion 32/37 Eperiments door tracking 33/37

13 Eperiments door tracking smoothed 34/37 Comparison of ncc vs KLT tracking 35/37 References 36/37 [1] Simon Baker and Iain Matthews. Lucas-Kanade 20 years on: A unifying framework. International Journal of Computer Vision, 56(3): , 4. [2] C. Harris and M. Stephen. A combined corner and edge detection. In M. M. Matthews, editor, Proceedings of the 4th ALVEY vision conference, pages , University of Manchaster, England, September on-line copies available on the web. [3] J.P. Lewis. Fast template matching. In Vision Interfaces, pages , Etended version published on-line as "Fast Normalized Cross-Correlation" at [4] Bruce D. Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Conference on Artificial Intelligence, pages , August [5] Jianbo Shi and Carlo Tomasi. Good features to track. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages , [6] Carlo Tomasi and Takeo Kanade. Detection and tracking of point features. Technical Report CMU-CS , Carnegie Mellon University, April 1991.

14 End 37/37

15 OPPA European Social Fund Prague & EU: We invest in your future.