Optical Flow CS 637. Fuxin Li. With materials from Kristen Grauman, Richard Szeliski, S. Narasimhan, Deqing Sun

Transcription

1 Optical Flow CS 637 Fuxin Li With materials from Kristen Grauman, Richard Szeliski, S. Narasimhan, Deqing Sun

2 Motion and perceptual organization Sometimes, motion is the only cue

3 Motion and perceptual organization Sometimes, motion is the foremost cue

4 Motion and perceptual organization Even impoverished motion data can evoke a strong percept

5 Motion and perceptual organization Even impoverished motion data can evoke a strong percept

6 Motion estimation: Optical flow Will start by estimating motion of each pixel separately Then will consider motion of entire image

7 Image Warping non-parametric Specify more detailed warp function Examples: splines triangles optical flow (per-pixel motion) 2/1/2005 Motion estimation 7

8 Image Warping non-parametric Move control points to specify spline warp 2/1/2005 Motion estimation 8

9 Local Patch Analysis How certain are the motion estimates? CSE 576, Spring 2008 Motion estimation 9

10 The aperture problem Perceived motion

11 The aperture problem Actual motion

12 The barber pole illusion

13 The barber pole illusion

14 Aperture Problem in Real Life 14

15 Human Motion System Illusory Snakes 15

16 Why estimate motion? Lots of uses Track object behavior Correct for camera jitter (stabilization) Align images (mosaics) 3D shape reconstruction Special effects

17 Problem definition: optical flow How to estimate pixel motion from image H to image I? Solve pixel correspondence problem given a pixel in H, look for nearby pixels of the same color in I Key assumptions color constancy: a point in H looks the same in I For grayscale images, this is brightness constancy small motion: points do not move very far This is called the optical flow problem

18 Local Patch Analysis How certain are the motion estimates? CSE 576, Spring 2008 Motion estimation 18

19 Two Directions of Estimating Optical Flow Consider 2D motion field as projections of 3D scene flow Lucas-Kanade algorithm Scene Flow Cost Volume Processing (CVPR 2017 paper) Will talk about in the second part of the course Don t care about 3D, directly estimate 2D motion Horn-Schunck Classic-NL EpicFlow, FlowFields etc.

20 20 Horn & Schunck algorithm Regularize for ill conditioned areas of the image. ( ) + + = j i ij v ij j u i I j i I E, 2 2 1, ), ( ), ( ), ( min v u v u Initial cost (Color constancy) ( ) ] ) ( ) ( ) ( ) [( ), ( ), ( ), ( 2, 1, 2, 1, 2, 1, 2, 1,, j i j i j i j i j i j i j i j i j i ij ij v v v v u u u u v j u i I j i I E = λ v u Unary + pairwise!

21 Dense Optical Flow ~ Michael Black s method Michael Black took this one step further, starting from the regularized cost: E( u, v) = i, j ( ) 2 I ( i, j) I ( i + u, j + v ) + λρ( u, ) 1 2 ij ij v He replaced the inner distance metric, a quadradic: with a Lorentzian robust regularization: E( u, v) = Ψ i, j ( I ( i, j) I ( i + u, j + v )) + λρ( u, ) 1 2 ij ij v Where Ψ looks something like Basically, one could say that Black s method adds ways to handle occlusion, non-common fate, and temporal dislocation 21

22 Example

23 Optical Flow Result

24 Low Texture Region - Bad gradients have small magnitude

25 Edges so,so (aperture problem) large gradients, all the same

26 High Textured Region - Good gradients are different, large magnitudes

27 Secrets of Optical Flow Estimation Paper Deqing Sun s paper in CVPR 2010 tests many variants of this approach Sun, Roth and Black. Secrets of Optical Flow Estimation and Their Principles. CVPR 2010

28 Coarse-to-Fine In a coarse layer Compute flow Median filter (Wedel et al. 2008) Warp the 2 nd image Upsample Compute flow Median Filter

29 1 st : Loss Function Tested Charbonnier (L1) (Classic-C) L2 (HS) Lorentzian (Classic-L) Results:

30 2 nd : Features for the Loss Color constancy Gradient constancy Gaussian + XY constancy i, j + ψ i, j (8 variants tested in the paper) ( I1 ( i, j) I 2( i + uij, j v ij )) ( di ( i, j) di ( i + u, j ) ψ + i, j ψ 2 + i, j 1 ij v ij ) ( I1( i, j) I 2( i + uij, j + vij )) + ψ ( dx1( i, j) dx2( i + uij, j + vij )) ψ ( dy ( i, j) dy ( i + u, j + v )) 1 2 ij ij i, j

31 3 rd : Interpolation In optical flow one needs to find image intensity/gradient values of non-integral image points Authors found bi-cubic interpolation of both the image and its derivative image is better than bi-linear interpolation or bi-cubic interpolation of the image only Among trials of 7-8 things

32 4 th : GNC Graduated non-convexity A continuation method to solve the non-convex optimization 3 stages (Quadratic loss EE qq, quadratic + L1 loss 1 2 (EE qq + EE), then EE Without this performance is significantly worse Quadratic helps improve the diagonal of the Hessian, hence problem being more likely to be convex

33 5 th : More Loss Functions Generalized Charbonnier ρρ xx = xx 2 + εε 2 αα

34 6 th : Coarse-to-Fine and Median Filtering Coarse-to-Fine estimation (standard approach) Important step in upsampling Median filtering

35 What s going on in Median Filtering? 5x5 median filtering imposes smoothness constraints over 5x5 region A more connected graph than the original 4-connected! Median is related to L1 optimization (Lee and Osher 2009) New Fully-Connected Term

36 Convert to auxiliary flow field Split variables into 2 sets Introduce an auxiliary flow field for the new regularization term Proximal point method Allows approaches such as ADMM Improve convexity from the quadratic term

37 Problem with Median Filtering Oversmoothing in sharp regions Solution: weighted median filtering Represents how similar II(ii, jj) is to II(ii, jjj)

38 After adding weights Weighted median filtering improves results significantly

39 More ablation experiments Reconfirms that LAB is the best color space (0.221 error) Color difference most important in weighted median filtering

40 Qualitative Results

41 More Qualitative Results

42 Take-home Many computer vision common-place tricks Coarse-to-fine Filtering Robust loss function Non-local neighborhoods Weighted (regularization only on the same object) Tricks, tricks and more tricks Edge-aware smoothing important in computer vision