Introduction to Computer Vision

Transcription

1 Introduction to Computer Vision Michael J. Black Oct 2009 Motion estimation

2 Goals Motion estimation Affine flow Optimization Large motions Why affine? Monday dense, smooth motion and regularization. Robust statistics Mon or Wed discuss projects.

3 Assignment 3 Part and 2 due Nov 3 (Tuesday) am All due Nov 9 am

4 Affine Flow

5 Affine Transformation Homogeneous coordinates Linear transformation Translation

6 Affine flow u v = a a2 a3 a4 a5 a6 0 0 x y x' y' = x y 0 + a a2 a3 a4 a5 a6 0 0 x y x' y' = a+ a2 a3 a4 a5 + a6 0 0 x y Motion (flow) between frames Transformation of pixels Use when transforming pixels

7 Important Slide! When I say x and y, I mean relative to the center of the patch. The patch may be the whole image. u v = a a2 a3 a4 a5 a6 0 0 x x c y y c [x c y c ] defines the center of the patch. So the affine transformation is wrt (0, 0).

8 What can be represented? What does this do? x' y' = 0 a3 0 a6 0 0 x y

9 What can be represented? What does this do? x' y' = cos sin 0 sin cos x y

10 What can be represented? What does this do? x' y' = x y

11 What can be represented? What does this do? x' y' = x y

12 Composition and inversion Apply two affine transforms successively. Represent affine transformations as matrices x * = A(Bx) = (AB)x Inversion x = (AB) - x *

13 Transforming images Affine transformation is applied to image coordinates x, y I =I(A[x,y,] T ) How do we do this in Matlab? What are the issues?

14 Get the image coordinates: >> [y,x]=meshgrid(:0, :0) y = x =

15 Forwards Warp Image at t A Contributes to 4 pixels.

16 Backwards Warp Image at t Contributions from 4 pixels.

17 Backwards Warp Image at t Image at t+ A - Contributions from 4 pixels bi-linear interpolation Every pixel at time t+ defined.

18 Interpolation Possible interpolation filters: nearest neighbor bilinear bicubic (interpolating) Needed to prevent jaggies When iteratively warping, always compose the warps and warp the original image Szeliski and Fleet

19 Example warps: Warping images translation rotation aspect affine perspective cylindrical Szeliski and Fleet

20 interp2 INTERP2 2-D interpolation (table lookup). ZI = interp2(x,y,z,xi,yi) interpolates to find ZI, the values of the underlying 2-D function Z at the points in matrices XI and YI. Matrices X and Y specify the points at which the data Z is given. Out of range values are returned as NaN.

21 Image warping function warpim= warpimage(image, a) warpim=zeros(size(image)); [y,x]=meshgrid(:size(image,2),:size(image,)); % find the center of the image % compute the new pixel locations x2 and y2 warpim=interp2(y, x, image, y2, x2, 'linear'); % fix NaNs ind=find(~(warpim>0 & warpim<256)); warpim(ind)=0.0;

22 Algorithm (i.e. homework #3) Incremental optimization * Given the images, construct the structure tensor invert it, and solve for the motion parameters. * Warp image towards image 2 * repeat until convergence

23 Optimization E(a) = (I x u + I y v + I t ) 2 x,y R u v = a a2 a3 a4 a5 a6 0 0 x y E(a) = x,y R (I x a x + I x a 2 y + I x a 3 + I y a 4 x + I y a 5 y + I y a 6 + I t ) 2

24 Optimization E(a) = x,y R (I x a x + I x a 2 y + I x a 3 + I y a 4 x + I y a 5 y + I y a 6 + I t Differentiate wrt the a i and set equal to zero. I x 2 x 2 I x 2 xy I x 2 x I x I y x 2 I x I y xy I x I y x I x 2 xy I x 2 y 2 I x 2 y I x I y xy I x I y y 2 I x I y y a a 2 a 3 a 4 a 5 a 6 = ) 2 I x I t x I x I t y I x I t I y I t x I y I t y I y I t

25 We have a problem Taylor approximation assumed small motions. Real motions may be larger than a pixel. Temporal derivative won t make sense. Need a solution.

26 Compute Flow function a=basicflow(im, im2, ainit, iters) % warp im by current flow parameters (start with ainit) % compute image derivatives % build structure tensor % solve for motion parameters (exclude boundary pixels and non-overlapping pixels from the analysis mark them with nan s) % update current flow parameters (compose affine transformations) % repeat

27 Testing your motion code Take an image and warp it by some known affine motion. Solve for the motion You should be able to fairly accurately recover the parameters. Start with only translation.

28 Coarse to Fine (Translation) function a=pyramidflow(im, im2, ainit, iters, levels) % build image pyramids (dividing a3 and a6 by 2 for each level as you go) % starting with coarse level % warp im by current flow % estimate flow % project flow to next level % (ie multiply a3 & a6 by 2) % repeat to finest level

29 Why Affine? Where does this affine approximation come from? All our models are approximations to the world. What are the assumptions in the affine approximation? For this we need some geometry.

30 Pinhole cameras Abstract camera model - box with a small hole in it. Easy to build but needs a lot of light.