Computer Vision Lecture 2 Motion and Optical Flow Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de 28.1.216 Man slides adapted from K. Grauman, S. Seitz, R. Szeliski, M. Pollefes, S. Lazebnik Course Outline Image Processing Basics Segmentation & Grouping Object Recognition Local Features & Matching Object Categorization 3D Reconstruction Epipolar Geometr and Stereo Basics Camera calibration & Uncalibrated Reconstruction Active Stereo Motion Motion and Optical Flow 3D Reconstruction (Reprise) Structure-from-Motion 2 Recap: Epipolar Geometr Calibrated Case X Recap: Epipolar Geometr Uncalibrated Case X [ t ( R)] T E with E [ t ] R ˆ T E ˆ T F with F K EK T 1 Essential Matri (Longuet-Higgins, 1981) K ˆ Kˆ Fundamental Matri (Faugeras and Luong, 1992) 3 4 Recap: The Eight-Point Algorithm 2 3 = (u, v, 1) T, = (u, v, 1) T F11 F12 F13 F21 [u u; u v; u ; uv ; vv ; v ; u; v; 1] F22 = F23 6F317 4 5 F32 2 3 F33 2 3 2 3 u 1u1 u 1v1 u 1 u1v1 v1v1 v F11 1 u1 v1 1 u 2 u2 u 2 v2 u 2 u2v2 v2v2 v2 u2 v2 1 F12 u 3 u3 u 3 v3 u 3 u3v3 v3v3 v3 u3 v3 1 F13 u 4u4 u 4v4 u 4 u4v4 v4v4 v4 u4 v4 1 F21 u 5u5 u 5v5 u 5 u5v5 v5v5 v5 u5 v5 1 F22 = 6u 6u6 u 6v6 u 6 u6v6 v6v6 v6 u6 v6 1 F23 7 6 7 4u 7 u7 u 7 v7 u 7 u7v7 v7v7 v7 u7 v7 156F317 4 4 5 5 u This minimizes: 8u8 u 8v8 u 8 u8v8 v8v8 v8 F32 u8 v8 1 Solve using? SVD! Slide adapted from Svetlana Lazebnik F33 Af = N T 2 ( F i ) i i1 5 Recap: Normalized Eight-Point Algorithm 1. Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 piels. 2. Use the eight-point algorithm to compute F from the normalized points. 3. Enforce the rank-2 constraint using SVD. SVD F UDV 4. Transform fundamental matri back to original units: if T and T are the normalizing transformations in the two images, than the fundamental matri in original coordinates is T T F T. T d11 v11 v13 U d 22 d v31 v33 33 6 [Hartle, 1995] T Set d 33 to zero and reconstruct F 1
Practical Considerations Topics of This Lecture Introduction to Motion Applications, uses Motion Field Derivation Small Baseline Large Baseline 1. Role of the baseline Small baseline: large depth error Large baseline: difficult search problem Solution Track features between frames until baseline is sufficient. Optical Flow Brightness constanc constraint Aperture problem Lucas-Kanade flow Iterative refinement Global parametric motion Coarse-to-fine estimation Motion segmentation KLT Feature Tracking Slide adapted from Steve Seitz 15 16 Video Motion and Perceptual Organization A video is a sequence of frames captured over time Now our image data is a function of space ( and time (t) Sometimes, motion is the onl cue 17 18 Motion and Perceptual Organization Motion and Perceptual Organization Sometimes, motion is foremost cue Even impoverished motion data can evoke a strong percept Slide credit: Kristen Grauman 19 2 2
Motion and Perceptual Organization Uses of Motion Even impoverished motion data can evoke a strong percept Estimating 3D structure Directl from optic flow Indirectl to create correspondences for SfM Segmenting objects based on motion cues Learning dnamical models Recognizing events and activities Improving video qualit (motion stabilization) 21 Slide adapted from Svetlana Lazebnik 22 Motion Estimation Techniques Topics of This Lecture Direct methods Directl recover image motion at each piel from spatiotemporal image brightness variations Dense motion fields, but sensitive to appearance variations Suitable for video and when image motion is small Feature-based methods Etract visual features (corners, tetured areas) and track them over multiple frames Sparse motion fields, but more robust tracking Suitable when image motion is large (1s of piels) Introduction to Motion Applications, uses Motion Field Derivation Optical Flow Brightness constanc constraint Aperture problem Lucas-Kanade flow Iterative refinement Global parametric motion Coarse-to-fine estimation Motion segmentation KLT Feature Tracking 23 24 Motion Field The motion field is the projection of the 3D scene motion into the image Motion Field and Paralla P(t) is a moving 3D point Velocit of 3D scene point: V = dp/dt p(t) = ((t),(t)) is the projection of P in the image. Apparent velocit v in the image: given b components v = d/dt and v = d/dt These components are known as the motion field of the image. P(t) V p(t) v P(t+dt) p(t+dt) 25 26 3
Motion Field and Paralla P(t) Quotient rule: (f/g) = (g f g f)/g 2 V P(t+dt) Motion Field and Paralla Pure translation: V is constant everwhere To find image velocit v, differentiate p with respect to t (using quotient rule): p(t) v p(t+dt) Image motion is a function of both the 3D motion (V) and the depth of the 3D point (Z). 27 28 Motion Field and Paralla Pure translation: V is constant everwhere Motion Field and Paralla Pure translation: V is constant everwhere V z is nonzero: Ever motion vector points toward (or awa from) v, the vanishing point of the translation direction. V z is nonzero: Ever motion vector points toward (or awa from) v, the vanishing point of the translation direction. V z is zero: Motion is parallel to the image plane, all the motion vectors are parallel. The length of the motion vectors is inversel proportional to the depth Z. 29 3 Topics of This Lecture Introduction to Motion Applications, uses Motion Field Derivation Optical Flow Brightness constanc constraint Aperture problem Lucas-Kanade flow Iterative refinement Global parametric motion Coarse-to-fine estimation Motion segmentation KLT Feature Tracking Optical Flow Definition: optical flow is the apparent motion of brightness patterns in the image. Ideall, optical flow would be the same as the motion field. Have to be careful: apparent motion can be caused b lighting changes without an actual motion. Think of a uniform rotating sphere under fied lighting vs. a stationar sphere under moving illumination. 31 32 4
Apparent Motion Motion Field Estimating Optical Flow I(,t 1) Given two subsequent frames, estimate the apparent motion field u( and v( between them. Ke assumptions I(,t) Brightness constanc: projection of the same point looks the same in ever frame. Small motion: points do not move ver far. Spatial coherence: points move like their neighbors. Slide credit: Kristen Grauman 33 Figure from Horn book 34 The Brightness Constanc Constraint Brightness Constanc Equation: I(, t 1) I( u(, v(, t) Linearizing the right hand side using Talor epansion: Hence, I(,t 1) I(,t) I(, t 1) I(, t) I u( I v( I u I v It Spatial derivatives Temporal derivative 35 The Brightness Constanc Constraint I u I v I t How man equations and unknowns per piel? One equation, two unknowns Intuitivel, what does this constraint mean? The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown I ( u, v) I If (u,v) satisfies the equation, so does (u+u, v+v ) if I ( u', v') t gradient (u,v) (u,v ) (u+u,v+v ) edge 36 The Aperture Problem The Aperture Problem Perceived motion Actual motion 37 38 5
The Barber Pole Illusion The Barber Pole Illusion http://en.wikipedia.org/wiki/barberpole_illusion 39 http://en.wikipedia.org/wiki/barberpole_illusion 4 The Barber Pole Illusion Solving the Aperture Problem How to get more equations for a piel? Spatial coherence constraint: pretend the piel s neighbors have the same (u,v) If we use a 55 window, that gives us 25 equations per piel http://en.wikipedia.org/wiki/barberpole_illusion 41 B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674 679, 1981. 42 Solving the Aperture Problem Least squares problem: Conditions for Solvabilit Optimal (u, v) satisfies Lucas-Kanade equation Minimum least squares solution given b solution of When is this solvable? A T A should be invertible. A T A entries should not be too small (noise). A T A should be well-conditioned. (The summations are over all piels in the K K window) Slide adapted from Svetlana Lazebnik 43 44 6
Eigenvectors of A T A Haven t we seen an equation like this before? Recall the Harris corner detector: M = A T A is the second moment matri. The eigenvectors and eigenvalues of M relate to edge direction and magnitude. The eigenvector associated with the larger eigenvalue points in the direction of fastest intensit change. The other eigenvector is orthogonal to it. Interpreting the Eigenvalues Classification of image points using eigenvalues of the second moment matri: 2 Edge 2 >> 1 Corner 1 and 2 are large, 1 ~ 2 1 and 2 are small Flat region Edge 1 >> 2 45 Slide credit: Kristen Grauman 1 46 Edge Low-Teture Region Gradients ver large or ver small Large 1, small 2 47 Gradients have small magnitude Small 1, small 2 48 High-Teture Region Per-Piel Estimation Procedure Let T II t M I I and b II t Algorithm: At each piel compute U b solving MU b M is singular if all gradient vectors point in the same direction E.g., along an edge Triviall singular if the summation is over a single piel or if there is no teture I.e., onl normal flow is available (aperture problem) Gradients are different, large magnitude Large 1, large 2 49 Corners and tetured areas are OK 5 7
Iterative Refinement Optical Flow: Iterative Refinement 1. Estimate velocit at each piel using one iteration of Lucas and Kanade estimation. estimate update Initial guess: Estimate: 2. Warp one image toward the other using the estimated flow field. (Easier said than done) 3. Refine estimate b repeating the process. (using d for displacement here instead of u) Slide adapted from Steve Seitz 51 52 Optical Flow: Iterative Refinement Optical Flow: Iterative Refinement estimate update Initial guess: Estimate: estimate update Initial guess: Estimate: (using d for displacement here instead of u) (using d for displacement here instead of u) 53 54 Optical Flow: Iterative Refinement Optic Flow: Iterative Refinement Some Implementation Issues: Warping is not eas (ensure that errors in warping are smaller than the estimate refinement). Warp one image, take derivatives of the other so ou don t need to re-compute the gradient after each iteration. Often useful to low-pass filter the images before motion estimation (for better derivative estimation, and linear approimations to image intensit. (using d for displacement here instead of u) 55 56 8
Etension: Global Parametric Motion Models Eample: Affine Motion u( a a a 1 2 v( a4 a5 a6 Substituting into the brightness constanc equation: 3 I u I v I t Translation Affine Perspective 3D rotation 2 unknowns 6 unknowns 8 unknowns 3 unknowns 57 58 Eample: Affine Motion Problem Cases in Lucas-Kanade u( a a a 1 2 v( a4 a5 a6 Substituting into the brightness constanc equation: I 3 ( 6 a1 a2 a3 I ( a4 a5 a It The motion is large (larger than a piel) Iterative refinement, coarse-to-fine estimation A point does not move like its neighbors Motion segmentation Brightness constanc does not hold Do ehaustive neighborhood search with normalized correlation. Each piel provides 1 linear constraint in 6 unknowns. Least squares minimization: Err a) I ( a a a I ( a a a I 2 1 2 3 4 5 t ( 6 59 6 Dealing with Large Motions Temporal Aliasing Temporal aliasing causes ambiguities in optical flow because images can have man piels with the same intensit. I.e., how do we know which correspondence is correct? actual shift estimated shift Nearest match is correct (no aliasing) Nearest match is incorrect (aliasing) To overcome aliasing: coarse-to-fine estimation. 61 62 9
Idea: Reduce the Resolution! Coarse-to-fine Optical Flow Estimation u=1.25 piels u=2.5 piels u=5 piels Image 1 u=1 piels Image 2 63 Gaussian pramid of image 1 Gaussian pramid of image 2 64 Coarse-to-fine Optical Flow Estimation Dense Optical Flow Dense measurements can be obtained b adding smoothness constraints. Color map Run iterative L-K Warp & upsample Run iterative L-K... Image 1 Image 2 Gaussian pramid of image 1 Gaussian pramid of image 2 65 T. Bro C. Bregler, J. Malik, Large displacement optical flow, CVPR 9, Miami, USA, June 29. 83 Summar References and Further Reading Motion field: 3D motions projected to 2D images; dependenc on depth. Solving for motion with Sparse feature matches Dense optical flow Optical flow Brightness constanc assumption Aperture problem Solution with spatial coherence assumption Here is the original paper b Lucas & Kanade B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proc. IJCAI, pp. 674 679, 1981. And the original paper b Shi & Tomasi J. Shi and C. Tomasi. Good Features to Track. CVPR 1994. Slide credit: Kristen Grauman 84 85 1