Optical flow. Cordelia Schmid

Transcription

1 Optical flow Cordelia Schmid

2 Motion field The motion field is the projection of the 3D scene motion into the image

3 Optical flow Definition: optical flow is the apparent motion of brightness patterns in the image deall, optical flow would be the same as the motion field Have to be careful: apparent motion can be caused b lighting changes without an actual motion Think of a uniform rotating sphere under fied lighting vs. a stationar sphere under moving illumination

4 Estimating optical flow (,,t 1) Given two subsequent frames, estimate the apparent motion field u(,) and v(,) between them Ke assumptions (,,t) Brightness constanc: projection of the same point looks the same in ever frame Small motion: points do not move ver far Spatial coherence: points move like their neighbors

5 Brightness Constanc Equation: ), ( 1),, ( ),, ( ), ( t v u t ), ( ), ( ),, ( 1),, ( v u t t Linearizing the right side using Talor epansion: The brightness constanc constraint (,,t 1) (,,t) 0 t v u Hence,

6 The brightness constanc constraint 0 How man equations and unknowns per piel? One equation, two unknowns What does this constraint mean? u v The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown f (u, v) satisfies the equation, so does (u+u, v+v ) if ( u, v) t t 0 ( u', v') 0 gradient edge (u,v) (u,v ) (u+u,v+v )

7 The aperture problem Perceived motion

8 The aperture problem Actual motion

9 Solving the aperture problem How to get more equations for a piel? Spatial coherence constraint: pretend the piel s neighbors have the same (u,v) E.g., if we use a 55 window, that gives us 25 equations per piel B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. n nternational Joint Conference on Artificial ntelligence,1981. ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( n t t t n n v u

10 Lucas-Kanade flow Linear least squares problem The summations are over all piels in the window Solution given b ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( n t t t n n v u n n b A d b A A)d A T T ( t t v u

11 Lucas-Kanade flow Recall the Harris corner detector: M = A T Ais the second moment matri When is the sstem solvable? B looking at the eigenvalues of the second moment matri The eigenvectors and eigenvalues of M relate to edge direction and magnitude The eigenvector associated with the larger eigenvalue points in the direction of fastest intensit change, and the other eigenvector is orthogonal to it t t v u

12 Uniform region gradients have small magnitude small 1, small 2 sstem is ill-conditioned

13 Edge gradients have one dominant direction large 1, small 2 sstem is ill-conditioned

14 High-teture or corner region gradients have different directions, large magnitudes large 1, large 2 sstem is well-conditioned

15 Optical Flow Results

16 Multi-resolution registration

17 Coarse to fine optical flow estimation

18 Optical Flow Results

19 Horn & Schunck algorithm Additional smoothness constraint : nearb point have similar optical flow Addition constraint e s (( u 2 u 2 ) ( v 2 v 2 )) dd, B.K.P. Horn and B.G. Schunck, "Determining optical flow." Artificial ntelligence,1981

20 Horn & Schunck algorithm Additional smoothness constraint : e s (( u 2 u 2 ) ( v 2 v 2 )) dd, besides OF constraint equation term e c ( u v t ) 2 dd, minimize es+ec λ regularization parameter B.K.P. Horn and B.G. Schunck, "Determining optical flow." Artificial ntelligence,1981

21 Horn & Schunck algorithm Coupled PDEs solved using iterative methods and finite differences

22 Horn & Schunck Works well for small displacements For eample Middlebur sequence

23 Large displacement estimation in optical flow Large displacement is still an open problem in optical flow estimation MP Sintel dataset

24 Large displacement optical flow Classical optical flow [Horn and Schunck 1981] energ: color/gradient constanc smoothness constraint minimization using a coarse-to-fine scheme Large displacement approaches: LDOF [Bro and Malik 2011] a matching term, penalizing the difference between flow and HOG matches MDP-Flow2 [Xu et al. 2012] epensive fusion of matches (SFT + PatchMatch) and estimated flow at each level DeepFlow [Weinzaepfel et al. 2013] deep matching + flow refinement with variational approach

25 Deep Matching: main idea First image Second image Each subpatch is allowed to move: independentl in a limited range depending on its size The approach is fast to compute using convolution and ma-pooling The idea is applied recursivel

26 Deep Matching (1) non-overlapping patches of 44 piels Reference image convolution Target image

27 Deep Matching (2) response maps for each 44 patch response maps of 88 patches ma-pooling sub-sampling aggregation ma-pooling (33 filter) sub-sampling (half) aggregation

28 Deep Matching (2) response maps for each 44 patch response maps of 88 patches ma-pooling sub-sampling aggregation response maps of 1616 patches ma-pooling sub-sampling aggregation ma-pooling sub-sampling aggregation response maps of 3232 patches Pipeline similar in spirit to deep convolutional nets [Lecun et al. 1998]

29 Deep Matching (3) Etract scale-space local maima Backtrack quasi-dense correspondences Multi-scale response pramid Bottom-up Top-down

30 Deep Matching (3) local maimum First image Second image

31 Deep Matching: eample results Repetitive tetures First image Second image

32 Deep Matching: eample results Non-rigid deformation First image Second image

33 DeepFlow Classical optical flow [Horn and Schunck 1981] energ ntegration of Deep Matching energ matches guide the flow similar to [Bro and Malik 2011] Minimization using: coarse-to-fine strateg fied point iterations Successive Over Relaation (SOR)

34 Eperimental results: datasets MP-Sintel [Butler et al. 2012] sequences from a realistic animated movie large displacements (>20p for 17.5% of piels) atmospheric effects and motion blur

35 Eperimental results: datasets KTT [Geiger et al. 2013] sequences captured from a driving platform large displacements (>20p for 16% of piels) real-world: lightings, surfaces, materials

36 Eperimental results: sample results Ground-truth LDOF [Bro & Malik 2011] MDP-Flow2 [Xu et al. 2012] DeepFlow

37 Eperimental results: sample results Ground-truth LDOF [Bro & Malik 2011] MDP-Flow2 [Xu et al. 2012] DeepFlow

38 Eperimental results: improvements due to Deep Matching Comparison on MP-Sintel training set AEE: average endpoint error s40+: onl on large displacements HOG matching Deep Matching

39 EpicFlow: Sparse-to-dense interpolation based on Deep Matching accurate quasi dense matches with DeepMatching [Revaud et al., CVPR 15]

40 Approach: Sparse-to-dense interpolation based on Deep Matching nterpolation Does not respect motion boundaries Ground-Truth

41 Approach: Sparse-dense interpolation with motion boundaries image edges often coincide with motion boundaries (recall 95%) state-of-the-art SED detector [structured forest for edge detection, Dollar 13] image SED edges ground-truth flow ground-truth motion boundaries

42 Approach: Sparse-dense interpolation with motion boundaries EpicFlow: Matching [Deep Matching] Sparse-dense interpolation preserving motion boundaries Geodesic distance based on edges [SED] Refinement: One-level energ minimization with variational approach

43 Distance : edge-aware geodesic distance geodesic distance: shortest distance knowing a cost map C q Cost map C: image edges here computed with SED p

44 Sparse-to-dense nterpolation: edge-aware geodesic distance mage edges Matches Geodesic distance 100 closest matches Geodesic distance 100 closest matches

45 Comparing nterpolation/epicflow/deepflow

46 Comparison to the state of the art

47 Comparison to the state of the art (AEE) Method Error on MP-Sintel Error on Kitti Timings EpicFlow s TF+OFM ~500s DeepFlow s NLTGV-SC s (GPU) TF + OFM : Kenned 15. Optical flow with geometric occlusion estimation and fusion of multiple frames NLTGV-SC: Ranftl 14. Non-local total generalized variation for optical flow estimation.

48 Failure cases Missing matches (spear and horns of dragon) Missing contours (arm)

49 CNN to estimate optical flow: FlowNet [A. Dosovitski et al. CCV 15]

50 Architecture FlowNetSimple

51 Architecture FlowNetCorrelation

52 Snthetic dataset for training: Fling chairs A dataset of appro. 23k image pairs

53 Eperimental results S: simple, C: correlation, v: variational refinement, ft:fine-tuning

54 Eperimental results