Lecture 19: Motion. Effect of window size 11/20/2007. Sources of error in correspondences. Review Problem set 3. Tuesday, Nov 20

Transcription

1 Lecture 19: Motion Review Problem set 3 Dense stereo matching Sparse stereo matching Indexing scenes Tuesda, Nov 0 Effect of window size W = 3 W = 0 Want window large enough to have sufficient intensit variation, et small enough to contain onl pixels with about the same disparit. Sources of error in correspondences Low-contrast / textureless image regions Occlusions Camera calibration errors Poor image resolution Violations of brightness constanc (specular reflections) Large motions Figures from Li hang Sparse matching Indexing scenes 1

2 So far Features and filters Grouping, segmentation, fitting Multiple views, stereo, matching Recognition and learning So far: Features and filters So far: Grouping Transforming and describing images; textures and colors Clustering, segmentation, fitting; what parts belong together? [fig from Shi et al] So far: Multiple views So far: Recognition and learning Lowe Hartle and isserman Multi-view geometr and matching, stereo Shape matching, recognizing objects and categories, learning techniques Tomasi and Kanade

3 Motion and tracking Tracking objects, video analsis, low level motion Tomas Izo Outline Motion field and parallax Optical flow, brightness constanc Aperture problem Constraints on image motion Uses of motion Analzing motion can be useful for Estimating 3d structure Segmentation of moving objects Tracking objects, features over time Image sequences A digital video is a sequence of images (frames) captured over time. Now we consider image as a function of both position and time. Tpes of motion in video Considering rigid objects the can rotate and translate in the scene. Motion ma be due to Movement in scene Movement of camera (ego motion) Geometricall equivalent, however illumination effects can make one appear different than the other. Figure b Martial Hebert, CMU 3

4 Motion field and apparent motion Motion field equations Point in the scene Velocit vector (Big V) V = V, V, V ] [ x z p = f P Apparent velocit p v Projection of scene point p v (Little v) Take the time derivative of both sides: V V v = f zp Goal: estimate apparent motion, the u and v values at each pixel x,, i.e., u(x,), v(x,) Figure b Martial Hebert, CMU Figure b Martial Hebert, CMU Velocit of scene point described as Motion Translational motion Angular velocit V = T ω P Vx = Tx ω + ωz Y V = T ωz X + ωx V = T ω Y + ω X z Using this and the motion field equation, can give expressions for the components of the image velocit v... z x V = V, V, V ] ω = P = [ x z [ ω x, ω, ωz ] [ X, Y, ] Motion field equations p v (Big V) V = V, V, V ] (Little v) [ x z p = f P Take the time derivative of both sides: V V v = f zp Figure b Martial Hebert, CMU Motion field equations V v f V = z P Translational components Trucco & Verri Section 8..1 Vx = Tx ω + ωzy V = T ωz X + ωx V = T ω Y + ω X T x x zx Tx f ω ω x vx = ω f + ωz + f f Tz T f ω x ωx v = ωx f + ωzx + f f z z Rotational components x Motion field equations Translational part of image motion depends on (unknown) depth of the point Motion parallax: image motion is a function of both motion in space and depth of each point. T x x zx Tx f ω ω x vx = ω f + ωz + f f Tz T f ω x ωx v = ωx f + ωzx + f f Translational components Trucco & Verri Section 8..1 Rotational components 4

5 Motion parallax Translational motion Parallax/MotionParallax.html Radial motion field if T z nonzero. Length of flow vectors inversel proportional to depth of 3d point Figure from Michael Black, Ph.D. Thesis points closer to the camera move more quickl across the image plane Translational motion Translational motion Radial motion field if T z nonzero. Length of flow vectors inversel proportional to depth of 3d point Radial motion field if T z nonzero. Length of flow vectors inversel proportional to depth of 3d point Figure from Michael Black, Ph.D. Thesis Figure from Michael Black, Ph.D. Thesis Motion vs. Stereo: Similarities Both involve solving Correspondence: disparities, motion vectors Reconstruction Motion vs. Stereo: Differences Motion: Uses velocit: consecutive frames must be close to get good approximate time derivative 3d movement between camera and scene not necessaril single 3d rigid transformation Whereas with stereo: Could have an disparit value View pair separated b a single 3d transformation 5

6 Optical flow problem Optical flow problem How to estimate pixel motion from image H to image I? Solve pixel correspondence problem given a pixel in H, look for nearb pixels of the same color in I Goal: estimate apparent motion, the u and v values at each pixel x,, i.e., u(x,), v(x,) Adapted from Steve Seitz, UW Brightness constanc What might make it difficult to estimate apparent motion? Figure b Michael Black Spatial coherence Temporal smoothness Figure b Michael Black Figure b Michael Black 6

7 Motion constraints Brightness constanc equation To recover optical flow, we need some constraints (assumptions) Brightness constanc: in spite of motion, image measurement in small region will remain the same Spatial coherence: assume nearb points belong to the same surface, thus have similar motions, so estimated motion should var smoothl. Temporal smoothness: motion of a surface patch changes graduall over time. di dt spatial image gradients = 0 Total derivative: x and are also functions of time t I dx I d I = + + x dt dt t temporal derivatives, u and v Brightness constanc equation Brightness constanc equation I dx I d I + + = 0 x dt dt t I dx I d I + + = 0 x dt dt t u v u v Rewritten as: Rewritten as: This is exactl true in the limit as u and v go to 0, for infinitesimal motions. Which terms are measurable from images? How man unknowns in this equation? Aperture problem Aperture problem Brightness constanc equation: single equation, two unknowns; infinitel man solutions. According to brightness constanc constraint, motions that satisf the optical flow equation are onl constrained to lie along a line in u,v space. Can onl compute projection of actual flow vector [u,v] in the direction of the image gradient, that is, in the direction normal to the image edge. Flow component in gradient direction determined Flow component parallel to edge unknown. Figure from Michael Black s Ph.D. Thesis 7

8 Aperture problem Aperture problem Aperture problem Solving the aperture problem How to get more equations for a pixel? Basic idea: impose additional constraints most common is to assume that the flow field is smooth locall one method: pretend the pixel s neighbors have the same (u,v)» If we use a 5x5 window, that gives us 5 equations per pixel! _pole.html RGB version How to get more equations for a pixel? Basic idea: impose additional constraints most common is to assume that the flow field is smooth locall one method: pretend the pixel s neighbors have the same (u,v)» If we use a 5x5 window, that gives us 5*3 equations per pixel! Lucas-Kanade flow Prob: we have more equations than unknowns Solution: solve least squares problem minimum least squares solution given b solution (in d) of: The summations are over all pixels in the K x K window This technique was first proposed b Lucas & Kanade (1981) 8

9 Windows and apparent motion Conditions for solvabilit Optimal (u, v) satisfies Lucas-Kanade equation When is this solvable? A T A should be invertible A T A should not be too small eigenvalues λ 1 and λ of A T A should not be too small A T A should be well-conditioned λ 1 / λ should not be too large (λ 1 = larger eigenvalue) Slide from Trevor Darrell, MIT Edge Low texture region gradient strong in one direction large λ 1, small λ gradients have small magnitude smallλ 1, small λ Adapted from Steve Seitz, UW High textured region Good conditions for solving flow Recall Harris corner detection Good feature windows to track in time can be detected independentl in a single frame. gradients are different, large magnitudes large λ 1, large λ 9

10 Revisiting the small motion assumption Reduce the resolution! Is this motion small enough? Probabl not it s much larger than one pixel ( nd order terms dominate) How might we solve this problem? Coarse-to-fine optical flow estimation Coarse-to-fine optical flow estimation u=1.5 pixels u=5pixels u=.5 u=5 pixels run iterative L-K warp & upsample run iterative L-K... image H u=10 pixels image II image HJ image II Gaussian pramid of image H Gaussian pramid of image I Gaussian pramid of image H Gaussian pramid of image I Example use of optical flow: Motion Paint Use optical flow to track brush strokes, in order to animate them to follow underling scene motion. Coming up Problem set 4 due 1/4 More on motion Multiple motions and segmentation Tracking SfM 10