Optical flow. Cordelia Schmid

Transcription

1 Optical flow Cordelia Schmid

2 Motion field The motion field is the projection of the 3D scene motion into the image

3 Optical flow Definition: optical flow is the apparent motion of brightness patterns in the image deall, optical flow would be the same as the motion field Have to be careful: apparent motion can be caused b lighting changes without an actual motion Think of a uniform rotating sphere under fied lighting vs. a stationar sphere under moving illumination

4 Estimating optical flow,,t 1 Given two subsequent frames, estimate the apparent motion field u, and v, between them Ke assumptions,,t Brightness constanc: projection of the same point looks the same in ever frame Small motion: points do not move ver far Spatial coherence: points move like their neighbors

5 Brightness Constanc Equation:, 1,,,,, t v u t,,,, 1,, v u t t Linearizing the right side using Talor epansion: The brightness constanc constraint,,t 1,,t 0 t v u Hence,

6 The brightness constanc constraint 0 How man equations and unknowns per piel? One equation, two unknowns What does this constraint mean? u v The component of the flow perpendicular to the gradient i.e., parallel to the edge is unknown f u, v satisfies the equation, so does u+u, v+v if u, v t t 0 u', v' 0 gradient edge u,v u,v u+u,v+v

7 The aperture problem Perceived motion

8 The aperture problem Actual motion

9 Solving the aperture problem How to get more equations for a piel? Spatial coherence constraint: pretend the piel s neighbors have the same u,v E.g., if we use a 55 window, that gives us 25 equations per piel B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. n nternational Joint Conference on Artificial ntelligence, n t t t n n v u

10 Lucas-Kanade flow Linear least squares problem The summations are over all piels in the window Solution given b n t t t n n v u n n b A d b A Ad A T T t t v u

11 Lucas-Kanade flow Recall the Harris corner detector: M = A T Ais the second moment matri When is the sstem solvable? B looking at the eigenvalues of the second moment matri The eigenvectors and eigenvalues of M relate to edge direction and magnitude The eigenvector associated with the larger eigenvalue points in the direction of fastest intensit change, and the other eigenvector is orthogonal to it t t v u

12 Uniform region gradients have small magnitude small 1, small 2 sstem is ill-conditioned

13 Edge gradients have one dominant direction large 1, small 2 sstem is ill-conditioned

14 High-teture or corner region gradients have different directions, large magnitudes large 1, large 2 sstem is well-conditioned

15 Optical Flow Results

16 Multi-resolution registration

17 Coarse to fine optical flow estimation

18 Optical Flow Results

19 Horn & Schunck algorithm Additional smoothness constraint : nearb point have similar optical flow Addition constraint e s u 2 u 2 v 2 v 2 dd, B.K.P. Horn and B.G. Schunck, "Determining optical flow." Artificial ntelligence,1981

20 Horn & Schunck algorithm Additional smoothness constraint : e s u 2 u 2 v 2 v 2 dd, besides OF constraint equation term e c u v t 2 dd, minimize es+ec λ regularization parameter B.K.P. Horn and B.G. Schunck, "Determining optical flow." Artificial ntelligence,1981

21 Horn & Schunck algorithm Coupled PDEs solved using iterative methods and finite differences

22 Horn & Schunck Works well for small displacements For eample Middlebur sequence

23 Large displacement estimation in optical flow Large displacement is still an open problem in optical flow estimation MP Sintel dataset

24 Large displacement optical flow Classical optical flow [Horn and Schunck 1981] energ: color/gradient constanc smoothness constraint minimization using a coarse-to-fine scheme Large displacement approaches: LDOF [Bro and Malik 2011] a matching term, penalizing the difference between flow and HOG matches MDP-Flow2 [Xu et al. 2012] epensive fusion of matches SFT + PatchMatch and estimated flow at each level DeepFlow [Weinzaepfel et al. 2013] deep matching + flow refinement with variational approach

25 CNN to estimate optical flow: FlowNet [A. Dosovitski et al. CCV 15]

26 Architecture FlowNetSimple

27 Architecture FlowNetCorrelation

28 Snthetic dataset for training: Fling chairs A dataset of appro. 23k image pairs

29 Eperimental results S: simple, C: correlation, v: variational refinement, ft:fine-tuning

30 Eperimental results

31 FlowNet2.0 [lg et al. CVPR 17]

32 FlingThings3D [Maer et al., CVPR 16]

33 Comparison training data Best: pretraining on a simpler dataset, then fine tuning on a more comple set FlowNetC better than FlowNetS

34 mportance of warping Stacking of networks

35 Comparison to the state of the art

36 Optical flow results on Sintel

37 Video object segmentation Segment the moving object in all the frames of a video DAVS ground-truth [Tokmakov et al., CVPR 2017]

38 Challenges Strong camera or background motion LDOF flow DAVS

39 Network architecture MP-Net Convolutional/deconvolutional network, similar to U-Net

40 Training data FlingThings3D dataset [Maer et al., CVPR 16] 2700 snthetic, 10-frame stereo videos of random object fling in random trajectories 2250/450 training/test split Ground-truth optical flow and camera data available Labels for moving object can be obtained from the data

41 Results on FlingThings3D test set

42 Motion estimation in real videos Flow estimation inaccuracies DAVS LDOF MP-Net Background motion DAVS LDOF MP-Net

43 Addition of an objectness measure Etract 100 object proposals per frame with SharpMask [Pinheiro et al., ECCV 16] Aggregate to obtain piel-level objectness scores o i Combine with the motion predictions m i DAVS LDOF MP-Net Objectness Result

44 FlowNet 2.0 Evaluation Setting LDOF flow FLowNet 2.0 flow MP-Net MP-Net + Obj MP-Net + Obj + CRF Mean ou on DAVS trainval set