Motion Estimation (II) Ce Liu Microsoft Research New England

Transcription

1 Motion Estimation (II) Ce Liu Microsoft Research New England

2 Last time Motion perception Motion representation Parametric motion: Lucas-Kanade T I x du dv = I x I T x I y I x T I y I y T I y 1 Ix T I t I y T I x Dense optical flow: Horn-Schunck I x 2 + αl I x I y I x I y I 2 y + αl U V = I xi t I y I t

3 Who are they? Berthold K. P. Horn Takeo Kanade

4 Content Robust optical flow estimation Applications Feature matching Discrete optical flow Layer motion analysis Other representations

5 Content Robust optical flow estimation Applications Feature matching Discrete optical flow Layer motion analysis Other representations

6 Spatial regularity Horn-Schunck is a Gaussian Markov random field (GMRF) I x u + I y v + I t 2 + α u 2 + v 2 dxdy Spatial over-smoothness is caused by the quadratic smoothness term Nevertheless, real optical flow fields are sparse! u u x u y v v x v y

7 Data term Horn-Schunck is a Gaussian Markov random field (GMRF) I x u + I y v + I t 2 + α u 2 + v 2 dxdy Quadratic data term implies Gaussian white noise Nevertheless, the difference between two corresponded pixels is caused by Noise (majority) Occlusion Compression error Lighting change The error function needs to account for these factors

8 Explicitly model the noise n Noise model I 2 x + u, y + v = I 1 x, y + n It can be a mixture of two Gaussians, inlier and outlier 2 n ~ λn 0, σ i + 1 λ N(0, σ 2 o ) Inlier Outlier Mixture (λ = 0.9) PDF Potential

9 More components in the mixture Consider a Gaussian mixture model K n~ 1 ξ k N(0, ks 2 ) Z k=1 Varying the decaying rate ξ, we obtain a variety of potential functions ξ = 0.1 ξ =

10 Typical error functions L2 norm ρ z = z L1 norm ρ z = z Truncated L1 norm ρ z = min( z, η) Lorentzian ρ z = log(1 + γz 2 )

11 Robust statistics Traditional L2 norm: only noise, no outlier Example: estimate the average of 0.95, 1.04, 0.91, 1.02, 1.10, Estimate with minimum error z = arg min z L2 norm: z = L1 norm: z = i ρ z z i Truncated L1: z = Lorentzian: z = L2 norm ρ z = z Truncated L1 norm ρ z = min( z, η) L1 norm ρ z = z Lorentzian ρ z = log(1 + γz 2 )

12 The family of robust power functions Can we directly use L1 norm ψ z = z? Derivative is not continuous Alternative forms L1 norm: ψ z 2 = z 2 + ε 2 Sub L1: ψ z 2 ; η = z 2 + ε 2 η, η < z z 2 + ε η = 0.5 η = 0.4 η = 0.3 η =

13 Modification to Horn-Schunck Let x = (x, y, t), and w x = u x, v x, 1 be the flow vector Horn-Schunck (recall) I x u + I y v + I t 2 + α u 2 + v 2 dxdy Robust estimation ψ I x + w I x 2 + αφ u 2 + v 2 dxdy Robust estimation with Lucas-Kanade g ψ I x + w I x 2 + αφ u 2 + v 2 dxdy

14 A unifying framework The robust object function g ψ I x + w I x 2 + αφ u 2 + v 2 dxdy Lucas-Kanade: α = 0, ψ z 2 = z 2 Robust Lucas-Kanade: α = 0, ψ z 2 = z 2 + ε 2 Horn-Schunck: g = 1, ψ z 2 = z 2, φ z 2 = z 2 One can also learn the filters (other than gradients), and robust function ψ, φ( ) [Roth & Black 2005]

15 Derivation strategies Euler-Lagrange Derive in continuous domain, discretize in the end Nonlinear PDE s Outer and inner fixed point iterations Limited to derivative filters; cannot generalize to arbitrary filters Energy minimization Discretize first and derive in matrix form Easy to understand and derive Iteratively reweighted least square (IRLS) Variational optimization Euler-Lagrange = Variational optimization = IRLS

16 Iteratively reweighted least square (IRLS) Let φ z = z 2 + ε 2 η be a robust function We want to minimize the objective function Φ Ax + b = n i=1 φ a i T x + b i 2 where x R d, A = a 1 a 2 a n T R n d, b R n By setting Φ x Φ x = = n = 0, we can derive i=1 n i=1 n i=1 φ a i T x + b i 2 w ii a i T xa i + w ii b i a i a i T w ii xa i + b i w ii a i = A T WAx + A T Wb a i T x + b i a i w ii = φ a i T x + b i 2 W = diag Φ (Ax + b)

17 Iteratively reweighted least square (IRLS) Derivative: Φ x = AT WAx + A T Wb = 0 Iterate between reweighting and least square 1. Initialize x = x 0 2. Compute weight matrix W = diag Φ (Ax + b) 3. Solve the linear system A T WAx = A T Wb 4. If x converges, return; otherwise, go to 2 Convergence is guaranteed (local minima)

18 IRLS for robust optical flow Objective function g ψ I x + w I x 2 + αφ u 2 + v 2 dxdy Discretize, linearize and increment x,y g ψ I t + I x du + I y dv 2 + αφ u + du 2 + v + dv 2 IRLS (initialize du = dv = 0) Reweight: Least square: Ψ xx = diag g ψ I x I x, Ψ xy = diag g ψ I x I y, Ψ yy = diag g ψ I y I y, Ψ xt = diag g ψ I x I t, = diag g ψ I y I t, L = D T x Φ D x + D T y Φ D y Ψ yt Ψ xx + αl Ψ xy + αl Ψ xy Ψ yy du dv = Ψ xt + αlu + αlv Ψ yt

19 What s changed? Optical flow with robust function Ψ xx = diag g ψ I x I x, Ψ xy = diag g ψ I x I y, Ψ yy = diag g ψ I y I y, Ψ xt = diag g ψ I x I t, = diag g ψ I y I t, L = D T x Φ D x + D T y Φ D y Ψ yt Ψ xx + αl Ψ xy + αl Ψ xy Ψ yy du dv = Ψ xt + αlu + αlv Ψ yt Horn-Schunck I x 2 + αl I x I y I x I y I 2 y + αl du dv = I xi t + αlu I y I t + αlv

20 Example Robust optical flow Input two frames Horn-Schunck Flow visualization Coarse-to-fine LK with median filtering

21 Content Robust optical flow estimation Applications Feature matching Discrete optical flow Layer motion analysis Other representations

22 Video stabilization

23 Video denoising

24 Video super resolution

25 Content Robust optical flow estimation Applications Feature matching Discrete optical flow Layer motion analysis Contour motion analysis Obtaining motion ground truth

26 Block matching Both Horn-Schunck and Lucas-Kanade are sub-pixel accuracy algorithms But in practice we may not need sub-pixel accuracy MPEG: block matching using MMSE H264: variable block size and quarter-pixel precision

27 Tracking reliable features Idea: no need to work on ambiguous region pixels (flat regions & line structures) Instead, we can track features and then propagate the tracking to ambiguous pixels Good features to track [Shi & Tomashi 94] T I x du dv = I x I T x I y I x T I y I y T I y Block matching + Lucas-Kanade refinement 1 Ix T I t I y T I t

28 Feature detection & tracking

29 From sparse to dense Interpolation: given values {d i } at { x i, y i }, reconstruct a smooth plane f(x, y) Membrane model (first order smoothness) i w i f x i, y i d i 2 + α f x 2 + f y 2 dxdy Thin plate model (second order smoothness) i w i f x i, y i d i 2 + α f xx 2 + f xy 2 + f yy 2 dxdy

30 Membrane vs. thin plate

31 Dense flow field from sparse tracking

32 Pros and Cons of Feature Matching Pros Efficient (a few feature points vs. all pixels) Reliable (with advanced feature descriptors) Cons Independent tracking (tracking can be unreliable) Not all information is used (may not capture weak features) How to improve Track every pixel with uncertainty Integrate spatial regularity (neighboring pixels go together)

33 Content Robust optical flow estimation Applications Feature matching Discrete optical flow Layer motion analysis Other representations

34 Discrete optical flow The objective function is similar to that of continuous flow x = (x, y) is pixel coordinate, w = (u, v) is flow vector E w = x x min( I 1 x I 2 x + w x, t) + η( u x + v x ) + Data term Small displacement min α u x 1 u x 2, d + min α v x 1 v x 2, d x 1,x 2 ε Spatial regularity Truncated L1 norms: Account for outliers in the data term Encourage piecewise smoothness in the smoothness term

35 Decoupled smoothness Coupled smoothness Smoothness O(L 4 ) Decoupled smoothness Smoothness: O(L) 2 ) Data term: O(L 2 ) Data term: O(L 2 ) Smoothness: O(L) 2 )

36 Combinatorial optimization on graph E w = min( I 1 x I 2 x + w x, t) + x η( u x + v x ) + x min α u x 1 u x 2, d + min α v x 1 v x 2, d x 1,x 2 ε Optimization strategies Belief propagation Graph cuts MCMC (simulated annealing)

37 Dual-layer belief propagation Horizontal flow u Vertical flow v w = (u, v) Data term min( I 1 x I 2 x + w, t) Smoothness term on u min α u x 1 u x 2, d v Smoothness term on v min α v x 1 v x 2, d u [Shekhovtsov et al. CVPR 07] Regularization term on u η u x Regularization term on v η v x

38 Dual-layer belief propagation Message M j k : given all the information at node k, predict the distribution at node j k j v u Update within u plane

39 Dual-layer belief propagation v u Update within v plane

40 Dual-layer belief propagation k j v u Update from u plane to v plane

41 Dual-layer belief propagation v u Update from v plane to u plane

42 Example Discrete optical flow Input two frames Robust optical flow Flow visualization Coarse-to-fine LK with median filtering

43 Content Robust optical flow estimation Applications Feature matching Discrete optical flow Layer motion analysis Other representations

44 Layer representation Optical flow field is able to model complicated motion Different angle: a video sequence can be a composite of several moving layers Layers have been widely used Adobe Photoshop Adobe After Effect Compositing is straightforward, but inference is hard Wang & Adelson, 1994

45 Wang & Adelson, 1994 Strategy Obtaining dense optical flow field Divide a frame into non-overlapping regions and fit affine motion for each region Cluster affine motions by k-means clustering Region assignment by hypothesis testing Region splitter: disconnected regions are separated

46 Results Optical flow field Clustering to affine regions Clustering with error metric Flower garden Three layers with affine motion superimposed Reconstructed background layer

47 Weiss & Adelson, 1996 Chicken & egg problem Good motion good segmentation Good segmentation good motion We don t have either of them, so iterate! Perceptually organized expectation & maximization (POEM) E-step: estimate the motion parameter of each layer M-step: estimate the likelihood that a pixel belongs to each of the layers (segmentation)

48 Liu et. al Reliable layer segmentation for motion magnification Layer segmentation pipeline Feature point tracking Trajectory clustering Dense optical flow interpolation Layer segmentation

49 Normalized Complex Correlation Feature point tracking Trajectory clustering Dense optical flow interpolation Layer segmentation The similarity metric should be independent of phase and magnitude Normalized complex correlation S( C, C 1 2 ) t C ( t) C ( t) 1 t C ( t) C ( t) t C 2 2 ( t) C 2 ( t)

50 Trajectory Spectral Clustering Feature point tracking Trajectory clustering Dense optical flow interpolation Layer segmentation Trajectory Two clusters Affinity matrix Clustering Reordering of affinity matrix

51 Clustering Results Feature point tracking Trajectory clustering Dense optical flow interpolation Layer segmentation

52 From Sparse Feature Points to Dense Optical Flow Field Feature point tracking Trajectory clustering Dense optical flow interpolation Layer segmentation Interpolate dense optical flow field using locally weighted linear regression Dense Flow vectors optical of flow field clustered of cluster sparse 12 (leaves) (swing) feature points Cluster 1: leaves Cluster 2: swing

53 Motion Layer Assignment Feature point tracking Trajectory clustering Dense optical flow interpolation Layer segmentation Assign each pixel to a motion cluster layer, using four cues: Motion likelihood consistency of pixel s intensity if it moves with the motion of a given layer (dense optical flow field) Color likelihood consistency of the color in a layer Spatial connectivity adjacent pixels favored to belong the same group Temporal coherence label assignment stays constant over time Energy minimization using graph cuts

54

55 How good is optical flow? The AAE (average angular error) race on the Yosemite sequence for over 15 years Improvement # Yosemite sequence State-of-the-art optical flow * # I. Austvoll. Lecture Notes in Computer Science, 2005 * Brox et al. ECCV, 2004.

56 Middlebury flow database Baker et. al. A Database and Evaluation Methodology for Optical Flow. ICCV 2007

57 Middlebury flow database

58 Human-assisted motion annotation Ground truth is essential to progress the field A first step in computer vision to obtain ground-truth motion for arbitrary scenes (a different approach from Baker et al. 2007) An interactive system to combine human perception and the state-of-the-art computer vision algorithms to annotate motion Liu et al. Human-assisted motion annotation. CVPR 2008

59 Demo: interactive layer segmentation

60 Demo: interactive motion labeling

61 Motion database of natural scenes Color map Bruhn et al. Lucas/Kanade meets Horn/Schunck: combining local and global optical flow methods. IJCV, 2005

62 Content Robust optical flow estimation Applications Feature matching Discrete optical flow Layer motion analysis Other representations

63 Particle video P. Sand and S. Teller. Particle Video: Long-Range Motion Estimation using Point Trajectories. CVPR 2006

64 Particle video

65 Seemingly Simple Examples Kanizsa square From real video

66 Output from the State-of-the-Art Optical Flow Algorithm Kanizsa square Optical flow field T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004

67 Output from the State-of-the-Art Optical Flow Algorithm Dancer Optical flow field T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004

68 Optical flow representation: aperture problem Corners Lines Flat regions Spurious junctions Boundary ownership Illusory boundaries

69 Optical flow representation: aperture problem We need motion representation beyond pixel level! Corners Lines Flat regions Spurious junctions Boundary ownership Illusory boundaries

70 Challenge: Textureless Objects under Occlusion Corners are not always trustworthy (junctions) Flat regions do not always move smoothly (discontinuous at illusory boundaries) How about boundaries? Easy to detect and track for textureless objects Able to handle junctions with illusory boundaries

71 Frame 1

72 Frame 2

73 Extracted boundary fragments

74 Optical flow from Lucas-Kanade algorithm

75 Estimated motion by our system, after grouping

76 Boundary grouping and illusory boundaries (frame 1)

77 Boundary grouping and illusory boundaries (frame 2)

78 Rotating Chair

79 Frame 1

80 Frame 2

81 Extracted boundary fragments

82 Estimated flow field from Brox et al.

83 Estimated motion by our system, after grouping

84 Boundary grouping and illusory boundaries (frame 1)

85 Boundary grouping and illusory boundaries (frame 2)