Optic Flow and Motion Detection

Transcription

1 Optic Flow and Motion Detection Computer Vision and Imaging Martin Jagersand Readings: Szeliski Ch 8 Ma, Kosecka, Sastry Ch 4

2 Image motion Somehow quantify the frame-to to-frame differences in image sequences. 1. Image intensity difference. 2. Optic flow dim image motion computation

3 Motion is used to: Attention: Detect and direct using eye and head motions Control: Locomotion, manipulation, tools Vision: Segment, depth, trajectory

4 Small camera re-orientation Note: Almost all pixels change!

5 Classes of motion Still camera, single moving object Still camera, several moving objects Moving camera, still background Moving camera, moving objects

6 The optic flow field Vector field over the image: [u,v] = f(x,y), u,v = Vel vector, x,y = Im pos FOE, FOC Focus of Expansion, Contraction

7 MOVING CAMERAS ARE LIKE STEREO The change in spatial location between the two cameras (the motion ) Locations of points on the object (the structure )

8 Image Correspondance problem Given an image point in left image, what is the (corresponding) point in the right image, which is the projection of the same 3-D point

9 Correspondance for a box The change in spatial location between the two cameras (the motion ) Locations of points on the object (the structure )

10 Motion/Optic flow vectors How to compute? Im(x, y, t) Im(x + îx, y + îy, t + ît) Solve pixel correspondence problem given a pixel in Im1, look for same pixels in Im2 Possible assumptions 1. color constancy: a point in H looks the same in I For grayscale images, this is brightness constancy 2. small motion: : points do not move very far This is called the optical flow problem The change in spatial location between the two cameras (the motion )

11 Optic/image flow Assume: 1. Image intensities from object points remain constant over time 2. Image displacement/motion small Im(x + îx, y + îy, t + ît) = Im(x, y, t)

12 Taylor expansion of intensity variation Im(x + îx, y + îy, t + ît) =Im(x, y, t)+ Im x îx + Im y îy + Im t ît + h.o.t. Keep linear terms Use constancy assumption and rewrite: 0= Im x îx + Im y îy + Im t ît Notice: Linear constraint, but no unique solution

13 Aperture problem f f n Rewrite as dot product ð ñ ò ó ò ó à Im t = Im Im ît x, á îx y = Im á îx îy îy Each pixel gives one equation in two unknowns: n*f = k Min length solution: Can only detect vectors normal to gradient direction The motion of a line cannot be recovered using only local information

14 Aperture problem 2

15 The flow continuity constraint Flows of nearby pixels or patches are (nearly) equal Two equations, two unknowns: n 1 * f = k 1 n 2 * f = k 2 Unique solution f exists, provided n 1 and n 2 not parallel f f n f f n

16 Sensitivity to error n 1 and n 2 might be almost parallel Tiny errors in estimates of k s or n s can lead to huge errors in the estimate of f f f n f f n

17 Using several points Typically solve for motion in 2x2, 4x4 or larger image patches. Over determined equation system:. Im à t. =. Im x.. Im y. dim = Mu Can be solved in least squares sense using Matlab u = M\dIm M Can also be solved be solved using normal equations u = (M T M) -1 *M T dim îx îy

18 Conditions for solvability SSD Optimal (u, v) satisfies Optic Flow equation When is this solvable? A T A should be invertible A T A entries should not be too small (noise) A T A should be well-conditioned Study eigenvalues: λ 1 / λ 2 should not be too large (λ 1 = larger eigenvalue)

19 Optic Flow Real Image Challenges: Can we solve for accurate optic flow vectors everywhere using this image sequence?

20 Edge gradients very large or very small large λ 1, small λ 2

21 Low texture region gradients have small magnitude small λ 1, small λ 2

22 High textured region gradients are different, large magnitudes large λ 1, large λ 2

23 Observation This is a two image problem BUT Can measure sensitivity by just looking at one of the images! This tells us which pixels are easy to track, which are hard very useful later on when we do feature tracking...

24 Errors in Optic flow computation What are the potential causes of errors in this procedure? Suppose A T A is easily invertible Suppose there is not much noise in the image When our assumptions are violated Brightness constancy is not satisfied The motion is not small A point does not move like its neighbors window size is too large what is the ideal window size?

25 Iterative Refinement Used in SSD/Lucas-Kanade tracking algorithm 1. Estimate velocity at each pixel by solving Lucas-Kanade equations 2. Warp H towards I using the estimated flow field - use image warping techniques 3. Repeat until convergence

26 Revisiting the small motion assumption Is this motion small enough? Probably not it s much larger than one pixel (2 nd order terms dominate) How might we solve this problem?

27 Reduce the resolution!

28 Coarse-to-fine optical flow estimation u=1.25 pixels u=2.5 pixels u=5 pixels image H u=10 pixels image I Gaussian pyramid of image H Gaussian pyramid of image I

29 Coarse-to-fine optical flow estimation run iterative L-K run iterative L-K warp & upsample... image J image H image I Gaussian pyramid of image H Gaussian pyramid of image I

30 Application: mpeg compression

31 HW accelerated computation of flow vectors Norbert s trick: Use an mpeg-card to speed up motion computation

32 Other applications: Recursive depth recovery: Kostas and Jane Motion control (we will cover) Segmentation Tracking

33 Lab: Assignment1: Purpose: Intro to image capture and processing Hands on optic flow experience See www page for details. Suggestions welcome!

34 Organizing Optic Flow Martin Jagersand

35 Organizing different kinds of motion Two examples: 1. Greg Hager paper: Planar motion 2. Mike Black, et al: Attempt to find a low dimensional subspace for complex motion

36 Remember: The optic flow field Vector field over the image: [u,v] = f(x,y), u,v = Vel vector, x,y = Im pos FOE, FOC Focus of Expansion, Contraction

37 Remember last lecture: Solving for the motion of a patch Over determined equation system:. Im à t. =. Im x. Imt = Mu Can be solved in e.g. least squares sense using matlab u = M\ImM Imt. Im y. îx îy! t t+1

38 3-6D Optic flow Generalize to many freedooms (DOFs) Im = Mu

39 Example: All 6 freedoms Template Difference images M(u) = Im/ u X Y Rotation Scale Aspect Shear

40 Know what type of motion (Greg Hager, Peter Belhumeur) E.g. Planar Object => Affine motion model: u i = A u i + d I t = g(p t, I 0 )

41 Mathematical Formulation Define a warped image g f(p,x) = x (warping function), p warp parameters I(x,t) (image a location x at time t) g(p,i t ) = (I(f(p,x 1 ),t), I(f(p,x 2 ),t), I(f(p,x N ),t)) Define the Jacobian of warping function M(p,t) = Model h I p i I 0 = g(p t, I t ) (image I, variation model g,, parameters p) ΔI I = M(p t, I t ) Δp (local linearization M) Compute motion parameters Δp = (M( T M) - 1 M T ΔI where M = M(p t,i t )

42 Planar 3D motion From geometry we know that the correct plane- to-plane transform is 1. for a perspective camera the projective homography " # u 0 v 0 2. for a linear camera (orthographic, weak-, para- perspective) the affine warp " # u w v w 1 = W h (x h, h) = 1+h7 u+h 8 v = W a (p, a) = " a 3 # a 4 a 5 a 6 p + " h 1 u h 3 v # h 5 h 2 u h 4 v h 6 " # a 1 a 2

43 Planar Texture Variability 1 Affine Variability Affine warp function " # u w v w Corresponding image variability ΔI a = P " # u u â ã 6 i=1 a i I w Δa i = I I a u, 1 ááá Δa1 a 6. v v v. a 1 ááá a 6 Δa 6 " # â ã 1 0 ã u 0 ã v 0 y1. ΔI a = I I u, v ã u 0 ã v. y 6 Discretized for images = W a (p, a) = =[B 1...B 6 ][y 1,...,y 6 ] T = B a y a " a 3 # a 4 a 5 a 6 p + " # a 1 a 2

44 On The Structure of M Planar Object + linear (infinite) camera -> Affine motion model u i = A u i + d M(p) = g/ p X Y Rotation Scale Aspect " a 3 # a 4 a 5 a 6 = sr(θ) " #" # a h 1 Shear

45 Planar motion under perspective projection Perspective plane-plane transforms defined by homographies

46 Planar Texture Variability 2 Projective Variability Homography warp " u 0 # v 0 1 = W h (x h, h) = 1+h7 u+h 8 v " h 1 u h 3 v # h 5 h 2 u h 4 v h 6 Projective variability: 1 ΔI h = c1 â ã u 0 v à c I I 1, u v 0 u 0 v 0 1 à c 1 uc 2 uc 3 vc à 2 c1 vc 3 à c1 Δh. 1. Δh 8 Where, and =[B 1...B 8 ][y 1,...,y 8 ] T = B h y h c 1 =1+h 7 u + h 8 v c 3 = h 2 u + h 4 v + h 6 c 2 = h 1 u + h 3 v + h 5

47 Planar-perspective motion 3 In practice hard to compute 8 parameter model stably from one image, and impossible to find out-of of plane variation Estimate variability basis from several images: Computed Estimated

48 Another idea Black, Fleet) Organizing flow fields Express flow field f in subspace basis m Different mixing coefficients a correspond to different motions

49 Example: Image discontinuities

50 Mathematical formulation Let: Mimimize objective function: Robust error norm Motion basis = Where

51 Experiment Moving camera 4x4 pixel patches Tree in foreground separates well

52 Experiment: Characterizing lip motion Very non-rigid!

53 Questions to think about Readings: Book chapter, Fleet et al. paper. Compare the methods in the paper and lecture 1. Any major differences? 2. How dense flow can be estimated (how many flow vectore/area unit)? 3. How dense in time do we need to sample?

54 Summary Three types of visual motion extraction 1. Optic (image) flow: Find x,y image velocities D motion: Find object pose change in image coordinates based more spatial derivatives (top down) 3. Group flow vectors into global motion patterns (bottom up) Visual motion still not satisfactorily solved problem

55 (Parenthesis) Euclidean world motion -> image Let us assume there is one rigid object moving with velocity T and w = d R / dt For a given point P on the object, we have p = f P/z The apparent velocity of the point is V = -T w x P Therefore, we have v = dp/dt = f (z V V z P)/z 2

56 Component wise: f y w xy w w x f w z f T x T f v f x w xy w y w f w z f T x T f v x y z x y z y y x z y x z x = + + = Motion due to translation: depends on depth Motion due to rotation: independent of depth

57 Sensing and Perceiving Motion Martin Jagersand

58 Counterphase sin grating Spatio-temporal temporal pattern Time t, Spatial x,y s(x, y, t) =A cos(kx cos Θ + Ky sin Θ à Φ) cos(ωt)

59 Counterphase sin grating Spatio-temporal temporal pattern Time t, Spatial x,y s(x, y, t) =A cos(kx cos Θ + Ky sin Θ à Φ) cos(ωt) Rewrite as dot product: 1 2(cos([a, b] ô x y õ ô à ωt) + cos([a, b] x y õ + ωt) = + Result: Standing wave is superposition of two moving waves

60 Analysis: Only one term: Motion left or right Mixture of both: Standing wave Direction can flip between left and right

61 QT movie Reichardt detector

62 Several motion models Gradient: in Computer Vision Correlation: In bio vision Spatiotemporal filters: Unifying model

63 Definition: Spatial response: Gabor function

64 Temporal response: Adelson,, Bergen 85 Note: Terms from taylor of sin(t) Spatio-temporal temporal D=DsDt

65 Receptor response to Counterphase grating Separable convolution

66 Simplified: For our grating: (Theta=0) L s = A àû 2 exp 2 (kàk) 2 cos(þ à Φ) 2 Write as sum of components: = exp( )*(acos acos + bsin )

67 Space-time receptive field

68 Combined cells Spat: Temp: Both: Comb:

69 Result: More directionally specific response

70 Energy model: Sum odd and even phase components Quadrature rectifier

71 Adaption: Motion aftereffect

72 Where is motion processed?

73 Higher effects:

74 Equivalence: Reich and Spat

75 Conclusion Evolutionary motion detection is important Early processing modeled by Reichardt detector or spatio-temporal temporal filters. Higher processing poorly understood