Optical Flow CS 637 Fuxin Li With materials from Kristen Grauman, Richard Szeliski, S. Narasimhan, Deqing Sun
Motion and perceptual organization Sometimes, motion is the only cue
Motion and perceptual organization Sometimes, motion is the foremost cue
Motion and perceptual organization Even impoverished motion data can evoke a strong percept
Motion and perceptual organization Even impoverished motion data can evoke a strong percept
Motion estimation: Optical flow Will start by estimating motion of each pixel separately Then will consider motion of entire image
Image Warping non-parametric Specify more detailed warp function Examples: splines triangles optical flow (per-pixel motion) 2/1/2005 Motion estimation 7
Image Warping non-parametric Move control points to specify spline warp 2/1/2005 Motion estimation 8
Local Patch Analysis How certain are the motion estimates? CSE 576, Spring 2008 Motion estimation 9
The aperture problem Perceived motion
The aperture problem Actual motion
The barber pole illusion http://en.wikipedia.org/wiki/barberpole_illusion
The barber pole illusion http://en.wikipedia.org/wiki/barberpole_illusion
Aperture Problem in Real Life 14
Human Motion System Illusory Snakes 15
Why estimate motion? Lots of uses Track object behavior Correct for camera jitter (stabilization) Align images (mosaics) 3D shape reconstruction Special effects
Problem definition: optical flow How to estimate pixel motion from image H to image I? Solve pixel correspondence problem given a pixel in H, look for nearby pixels of the same color in I Key assumptions color constancy: a point in H looks the same in I For grayscale images, this is brightness constancy small motion: points do not move very far This is called the optical flow problem
Local Patch Analysis How certain are the motion estimates? CSE 576, Spring 2008 Motion estimation 18
Two Directions of Estimating Optical Flow Consider 2D motion field as projections of 3D scene flow Lucas-Kanade algorithm Scene Flow Cost Volume Processing (CVPR 2017 paper) Will talk about in the second part of the course Don t care about 3D, directly estimate 2D motion Horn-Schunck Classic-NL EpicFlow, FlowFields etc.
20 Horn & Schunck algorithm Regularize for ill conditioned areas of the image. ( ) + + = j i ij v ij j u i I j i I E, 2 2 1, ), ( ), ( ), ( min v u v u Initial cost (Color constancy) ( ) ] ) ( ) ( ) ( ) [( ), ( ), ( ), ( 2, 1, 2, 1, 2, 1, 2, 1,, 2 2 1 j i j i j i j i j i j i j i j i j i ij ij v v v v u u u u v j u i I j i I E + + + + + + = + + + + λ v u Unary + pairwise!
Dense Optical Flow ~ Michael Black s method Michael Black took this one step further, starting from the regularized cost: E( u, v) = i, j ( ) 2 I ( i, j) I ( i + u, j + v ) + λρ( u, ) 1 2 ij ij v He replaced the inner distance metric, a quadradic: with a Lorentzian robust regularization: E( u, v) = Ψ i, j ( I ( i, j) I ( i + u, j + v )) + λρ( u, ) 1 2 ij ij v Where Ψ looks something like Basically, one could say that Black s method adds ways to handle occlusion, non-common fate, and temporal dislocation 21
Example
Optical Flow Result
Low Texture Region - Bad gradients have small magnitude
Edges so,so (aperture problem) large gradients, all the same
High Textured Region - Good gradients are different, large magnitudes
Secrets of Optical Flow Estimation Paper Deqing Sun s paper in CVPR 2010 tests many variants of this approach Sun, Roth and Black. Secrets of Optical Flow Estimation and Their Principles. CVPR 2010
Coarse-to-Fine In a coarse layer Compute flow Median filter (Wedel et al. 2008) Warp the 2 nd image Upsample Compute flow Median Filter
1 st : Loss Function Tested Charbonnier (L1) (Classic-C) L2 (HS) Lorentzian (Classic-L) Results:
2 nd : Features for the Loss Color constancy Gradient constancy Gaussian + XY constancy i, j + ψ i, j (8 variants tested in the paper) ( I1 ( i, j) I 2( i + uij, j v ij )) ( di ( i, j) di ( i + u, j ) ψ + i, j ψ 2 + i, j 1 ij v ij ) ( I1( i, j) I 2( i + uij, j + vij )) + ψ ( dx1( i, j) dx2( i + uij, j + vij )) ψ ( dy ( i, j) dy ( i + u, j + v )) 1 2 ij ij i, j
3 rd : Interpolation In optical flow one needs to find image intensity/gradient values of non-integral image points Authors found bi-cubic interpolation of both the image and its derivative image is better than bi-linear interpolation or bi-cubic interpolation of the image only Among trials of 7-8 things
4 th : GNC Graduated non-convexity A continuation method to solve the non-convex optimization 3 stages (Quadratic loss EE qq, quadratic + L1 loss 1 2 (EE qq + EE), then EE Without this performance is significantly worse Quadratic helps improve the diagonal of the Hessian, hence problem being more likely to be convex
5 th : More Loss Functions Generalized Charbonnier ρρ xx = xx 2 + εε 2 αα
6 th : Coarse-to-Fine and Median Filtering Coarse-to-Fine estimation (standard approach) Important step in upsampling Median filtering
What s going on in Median Filtering? 5x5 median filtering imposes smoothness constraints over 5x5 region A more connected graph than the original 4-connected! Median is related to L1 optimization (Lee and Osher 2009) New Fully-Connected Term
Convert to auxiliary flow field Split variables into 2 sets Introduce an auxiliary flow field for the new regularization term Proximal point method Allows approaches such as ADMM Improve convexity from the quadratic term
Problem with Median Filtering Oversmoothing in sharp regions Solution: weighted median filtering Represents how similar II(ii, jj) is to II(ii, jjj)
After adding weights Weighted median filtering improves results significantly
More ablation experiments Reconfirms that LAB is the best color space (0.221 error) Color difference most important in weighted median filtering
Qualitative Results
More Qualitative Results
Take-home Many computer vision common-place tricks Coarse-to-fine Filtering Robust loss function Non-local neighborhoods Weighted (regularization only on the same object) Tricks, tricks and more tricks Edge-aware smoothing important in computer vision