The Lucas & Kanade Algorithm

Transcription

1 The Lucas & Kanade Algorithm Instructor - Simon Lucey Designing Computer Vision Apps

2 Today Registration, Registration, Registration. Linearizing Registration. Lucas & Kanade Algorithm.

3 3 Biggest Problems in Computer Vision?

4 3 Biggest Problems in Computer Vision? REGISTRATION REGISTRATION REGISTRATION (Prof. Takeo Kanade - Robotics Institute - CMU.)

5 What is Registration? Template Source Image

6 What is Registration? Template Source Image

7 What is Registration? x Template Source

8 What is Registration? x 0 W(x; p) x Template Source Our goal is to find the warp parameter vector! x = coordinate in template [x, y] T x 0 = corresponding coordinate in source [x 0,y 0 ] T W(x; p) = warping function such that x 0 = W(x; p) p = parameter vector describing warp p

9 Different Warp Functions #$%&'()#*%+, translation rotation aspect affine perspective cylindrical (Szeliski and Fleet) 6

10 Warping is now a pro Defining Warp Functions!"#$%&'()#*%+, Warping is now a p!"#$%&'()#*%+, W(x; p) = x + p p = [p1 p2 ] T Warping is now a pro rotation translation translation!"#$%&'()#*%+, x W(x; p) = M 1 )#*%+, Warping is now a problem lation M= 1 p1 p4 p2 1 p5 p = [p1..oct,. p2007 6] T rotation p3 translation affine p6 affine Oct, 2007 rotation Warping is n!"#$%&'()#*%+, perspective perspective rotation CS143 Intro to Computer Vision CS143 Intro to Computer Vision aspect translation 7 r

11 Naive Approach to Registration If you were a person coming straight from machine learning you might suggest, Images of Object at various warps

12 Naive Approach to Registration If you were a person coming straight from machine learning you might suggest, [255,134,45,...,34,12,124,67] [123,244,12,...,134,122,24,02] [67,13,245,...,112,51,92,181]... [65,09,67,...,78,66,76,215] Images of Object at various warps Vectors of pixel values at each warp position

13 Naive Approach to Registration If you were a person coming straight from machine learning you might suggest, [255,134,45,...,34,12,124,67] [123,244,12,...,134,122,24,02] [67,13,245,...,112,51,92,181]... [65,09,67,...,78,66,76,215] Vectors of pixel values at each warp position f( ) matching function object < background Th

14 Naive Approach to Registration Problem, Do we sample every warp parameter value? Plausible if we are only doing translation? If the image is high resolution, do we need to sample every pixel? What happens if warps are more complicated (e.g., scale)? Becomes prohibitively expensive computationally as warp dimensionality expands.

15 Measures of Image Similarity Although not always perfect, a common measure of image similarity is: Sum of squared differences (SSD) SSD(p) = NX i=1 I{W(x i ; p)} T(x i ) 2 2 W(x 1 ; p) W(x 1 ; 0) I Source Image T Template

16 Measures of Image Similarity Although not always perfect, a common measure of image similarity is: Sum of squared differences (SSD) SSD(p) = NX i=1 I{W(x i ; p)} T(x i ) 2 2 W(x 1 ; p) x 1 I Source Image T Template

17 Measures of Image Similarity Although not always perfect, a common measure of image similarity is: Sum of squared differences (SSD) SSD(p) = I(p) T(0) 2 2 Vector Form 3 W(x 1 ; p) 7. 5 W(x N ; p) 2 3 W(x 1 ; 0) W(x N ; 0) I Source Image T Template

18 Fractional Coordinates The warp function gives nearly always fractional output. But we can only deal in integer pixels. What happens if we need to subsample an image? (Black)

19 Image Interpolation Simply take the Taylor series approximation. I(x 0 + x) I(x 0 )+ I(x 0) x T x IGxH IFxH I(x 0 + x) I(x 0 ) x 0 x 14 (Black)

20 Image Interpolation Simply take the Taylor series approximation. I(x 0 + x) I(x 0 )+ I(x 0) x T x IGxH IFxH I(x 0 + x) I(x 0 ) I(x 0 ) x x 0 x 14 (Black)

21 Images at various warps Exhaustive Search

22 Exhaustive Search [255,134,45,...,34,12,124,67] [123,244,12,...,134,122,24,02] [67,13,245,...,112,51,92,181]... [65,09,67,...,78,66,76,215] Images at various warps Vectors of pixel values at each warp position

23 Exhaustive Search p2 p2 p 1 Possible Template Warps p 1 Possible Source Warps p = {p 1,p 2 }

24 No. of searches Exhaustive Search One can see that as the dimensionality D of p increases, and, assuming the same number of discrete samples n per dimension, the number of searches becomes, number of searches! O(n D ) Dimensionality of p (D)

25 No. of searches Exhaustive Search One can see that as the dimensionality D of p increases, and, assuming the same number of discrete samples n per dimension, the number of searches becomes, number of searches! O(n D ) Impractical for D >= 3 Dimensionality of p (D)

26 Today Registration, Registration, Registration. Linearizing Registration. Lucas & Kanade Algorithm.

27 Slightly Less Naive Approach Instead of sampling through all possible warp positions to find the best match, let us instead learn a regression, Warp Displacement Appearance Displacement 19

28 Slightly Less Naive Approach Instead of sampling through all possible warp positions to find the best match, let us instead learn a regression, Warp Displacement No longer have to make discrete assumptions about warps. Makes warp estimation computationally feasible. Appearance Displacement 19

29 Problems? For us to learn this regression effectively we need to make a couple of assumptions. What is the distribution p of warp displacements? Is there a relationship between appearance displacement and warp displacement? When does this relationship occur, when does it fail? T (0) T ( p) 20

30 Problems? For us to learn this regression effectively we need to make a couple of assumptions. What is the distribution p of warp displacements? Is there a relationship between appearance displacement and warp displacement? When does this relationship occur, when does it fail? T (0) T ( p) 20

31 I I(x, y) I(x +1,y+ 1) Simoncelli & Olshausen 2001

32 I I(x, y) I(x +8,y+ 8) Simoncelli & Olshausen 2001

33 I I(x, y) I(x + 16,y+ 16) Simoncelli & Olshausen 2001

34 I I(x, y) I(x + 50,y+ 50) Simoncelli & Olshausen 2001

35 I I(x, y) I(x + 50,y+ 50) Simoncelli & Olshausen 2001

36 Linearizing Registration What if I want to know x given that I have only the appearance at I(x 0 ) and I(x 0 + x)? IGxH IFxH I(x 0 + x) I(x 0 ) x 0 x (Black) 22

37 Linearizing Registration Again, simply take the Taylor series approximation? I(x 0 + x) I(x 0 T x IGxH IFxH I(x 0 + x) I(x 0 ) x 0 x 23 (Black)

38 Linearizing Registration Again, simply take the Taylor series approximation? I(x 0 + x) I(x 0 T x IGxH IFxH I(x 0 + x) I(x 0 ) I(x 0 ) x x 0 x 23 (Black)

39 Linearizing Registration Again, simply take the Taylor series approximation, I(x 0 + x) I(x 0 T x Therefore, x 0 T i 0 [I(x 0 + x) I(x 0 )] We refer to at. x 0 as the gradient of the image 24

40 Gradients through Filters Traditional method for calculating gradients in vision is through the use of edge filters. (e.g., Sobel, Prewitt). I x Horizontal I Vertical I y where, I(x) x =[ I x(x), I y (x)] T 25

41 Gradients through Filters Traditional method for calculating gradients in vision is through the use of edge filters. (e.g., Sobel, Prewitt). I x Horizontal I where, I(x) Vertical x =[ I x(x), I y (x)] T 25 I y Often have to apply a smoothing filter as well.

42 Gradients through Regression Another strategy is to learn a least-squares regression for every pixel in the source image such that, I(x + x) I(x)+w T x x + b I(x + x) I(x) x 2 N 26

43 Gradients through Regression Another strategy is to learn a least-squares regression for every pixel in the source image such that, I(x + x) I(x)+w T x x + b I(x + x) I(x) w x x 2 N 26

44 Gradients through Regression Another strategy is to learn a least-squares regression for every pixel in the source image such that, I(x + x) I(x)+w T x x + b I(x + x) I(x) w x Why does b have to equal zero? x 2 N 26

45 Gradients through Regression Can be written formally as, arg min w x x N I(x + x) I(x) w T x x 2 Solution is given as, w x = x 1 x T x N x N x[i(x + x) I(x)] 27

46 Gradients through Regression Can solve for gradients using least-squares regression. Has nice properties: expand neighborhood, no heuristics. Learn Regressor for each Pixel I x I I y where, I(x) x =[ I x(x), I y (x)] T =[w x,w y ] T 28

47 Possible Solution? Could we now just solve for individual pixel translation, and then estimate a complex warp from these motions? z = x 1. x N p Motion Field for Rotation 29

48 Spatial Coherence Problem is pixel noise. Individual pixels are too noisy to be reliable estimators of pixel movement (optical flow). Fortunately, neighboring points in the scene typically belong to the same surface and hence typically have similar motions W(x; p). on 30

49 Reminder: Warp Functions To perform alignment we need a formal way of describing how the template relates to the source image. To do this we can employ what is known as a warp function:- x W(x; p) x 0 where, x i = i-th 2D coordinate p = parametric form of warp 31

50 Today Registration, Registration, Registration. Linearizing Registration. Lucas & Kanade Algorithm.

51 Lucas & Kanade Algorithm Lucas & Kanade (1981) realized this and proposed a method for estimating warp displacement using the principles of gradients and spatial coherence. Technique applies Taylor series approximation to any spatially coherent area governed by the warp W(x; p). I(p + T p 33

52 Lucas & Kanade Algorithm Lucas & Kanade (1981) realized this and proposed a method for estimating warp displacement using the principles of gradients and spatial coherence. Technique applies Taylor series approximation to any spatially coherent area governed by the warp W(x; p). I(p + T p 34

53 Lucas & Kanade Algorithm Lucas & Kanade (1981) realized this and proposed a method for estimating warp displacement using the principles of gradients and spatial coherence. Technique applies Taylor series approximation to any spatially coherent area governed by the warp W(x; p). I(p + T p We consider this image to always be static... 34

54 Lucas & Kanade Algorithm Lucas & Kanade (1981) realized this and proposed a method for estimating warp displacement using the principles of gradients and spatial coherence. Technique applies Taylor series approximation to any spatially coherent area governed by the warp W(x; p). I(p + T p 35

55 Lucas & Kanade Algorithm Lucas & Kanade (1981) realized this and proposed a method for estimating warp displacement using the principles of gradients and spatial coherence. Technique applies Taylor series approximation to any spatially coherent area governed by the warp W(x; p). I(p + T = T T N 0 T N T... 7 N T x 0 = W(x; p) 36

56 Lucas & T = T T N 0 T N N T r x I r y I 37

57 Lucas & T = T T N 0 T N N T

58 Lucas & T = T T N 0 T N W(x; p) 1 N T apple x M = apple 1 p1 p 2 p 3 p 4 1 p 5 p 6 p =[p 1...p 6 ] p) Can you 38

59 Lucas & Kanade Algorithm Lucas & Kanade (1981) realized this and proposed a method for estimating warp displacement using the principles of gradients and spatial coherence. Technique applies Taylor series approximation to any spatially coherent area governed by the warp W(x; p). I(p + p N 1 N 1 N P N = number of pixels P = number of warp parameters 39

60 Lucas & Kanade Algorithm Often just refer to, J T Also refer to, as the Jacobian matrix. H I = J T I J I as the pseudo-hessian. Finally, we can refer to, T (0) =I(p + p) as the template. 40

61 Lucas & Kanade Algorithm Actual algorithm is just the application of the following steps, Step 1: Step 2: p = H 1 I JT I [T (0) I(p)] p p + p keep applying steps until p converges. 41

62 Examples of LK Alignment 42

63 Examples of LK Alignment 42

64 More to read Lucas & Kanade, An Iterative Image Registration Technique with Application to Stereo Vision, IJCAI Baker & Matthews, Lucas-Kanade 20 Years On: A Unifying Framework, IJCV Bristow & Lucey, In Defense of Gradient-Based Alignment on Densely Sampled Sparse Features, Springer Book on Dense Registration Methods In Defense of Gradient-Based Alignment on Densely Sampled Sparse Features Hilton Bristow and Simon Lucey Abstract In this chapter, we explore the surprising result that gradientbased continuous optimization methods perform well for the alignment of image/object models when using densely sampled sparse features (HOG, dense SIFT, etc.). Gradient-based approaches for image/object alignment have many desirable properties inference is typically fast and exact, and diverse constraints can be imposed on the motion of points. However, the presumption that gradients predicted on sparse features would be poor estimators of the true descent direction has meant that gradient-based optimization is often overlooked in favour of graph-based optimization. We show that this intuition is only partly true: sparse features are indeed poor predictors of the error surface, but this has no impact on the actual alignment performance. In fact, for general object categories that exhibit large geometric and appearance variation, sparse features are integral to achieving any convergence whatsoever. How the descent directions are predicted becomes an important consideration for these descriptors. We explore a number of strategies for estimating gradients, and show that estimating gradients via regression in a manner that explicitly handles outliers improves alignment performance substantially. To illustrate the general applicability of gradient-based methods to the alignment of challenging object categories, we perform unsupervised ensemble alignment on a series of non-rigid animal classes from ImageNet. Hilton Bristow Queensland University of Technology, Australia. hilton.bristow@gmail.com Simon Lucey The Robotics Institute, Carnegie Mellon University, USA. slucey@cs.cmu.edu 1