Displacement estimation

Transcription

1 Displacement estimation Displacement estimation by block matching" l Search strategies" l Subpixel estimation" Gradient-based displacement estimation ( optical flow )" l Lukas-Kanade" l Multi-scale coarse-to-fine" l Parametric displacement estimation" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 1

2 Where is the defect? Image g[x,y] (no defect)" Image f [x,y] (w/ defect)" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 2

3 Absolute difference between two images f-g w/o alignment" f-g w/ alignment" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 3

4 Displacement estimation by block matching Measurement window is compared with a shifted array of pixels in the other image, to determine the best match" Rectangular array of pixels is selected as a measurement window" Image g[x,y] " Image f [x,y] " Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 4

5 Displacement estimation by block matching Image g[x,y] " Image f [x,y] "... process repeated for another measurement window position." Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 5

6 Integer pixel shifts Measurement window is compared with a shifted array of pixels in the other image, to determine the best match" Rectangular array of pixels is selected as a measurement window" Image g[x,y] " Image f [x,y] " Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 6

7 Integer pixel shifts Rectangular array of pixels is selected as a measurement window" Measurement window is compared with a shifted array of pixels in the other image, to determine the best match" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 7

8 Error metric Sum of Squared Differences! Sum all values in measurement window" SSD Δ x,δ y = f x, y g x + Δ x, y + Δ y x,y msmnt window ( ) 2 Horizontal displacement" Vertical displacement" Alternatives: SAD (Sum of Absolute Differences), cross correlation, mutual information... " Robustness against outliers: sum of saturated squared differences, median of squared differences... " Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 8

9 SSD values resulting from block matching 1.6 SSD" Estimated displacement" 0.2 Integer-pixel accuracy" Vertical shift Δ y" " Horizontal shift Δ x" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 9

10 Block matching: search strategies Full search" Δd y Δd x All possible displacements within the search range are compared." Computationally expensive" Highly regular, parallelizable" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 10

11 Block matching: search strategies Conjugate direction search" Δd y Δd x Alternate search in x and y directions" Stop when there is no further improvement" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 11

12 Block matching: search strategies Coarse-to-fine" Δd y Δd x Start with coarsely spaced candidate displacements" Smaller pattern when best match is in the middle" Stop when desired displacement accuracy is reached" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 12

13 Block matching: search strategies Diamond search [Li, Zeng, Liou, 1994] [Zhu, Ma, 1997]" Δd y Δd x Start with large diamond pattern at [0,0]" If best match lies in the center of large diamond, proceed with small diamond" If best match does not lie in the center of large diamond, center large diamond pattern at new best match" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 13

14 Absolute difference between images w/ integer-pixel alignment" w/o alignment" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 14

15 Interpolation of the SSD Minimum SSD" Sub-pixel" Accurate" Minimum" Fit parabola through" 3 > 3 points" " exactly" approximately" Horizontal shift Δ x" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 15

16 2-d Interpolation of SSD Minimum SSD" Paraboloid" Perfect fit through 6 points" Approximate fit through > 6 points" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 16

17 Sub-pixel accuracy Interpolate pixel raster of the reference image to desired sub-pixel accuracy (e.g., by bi-linear or bi-cubic interpolation)" Straightforward extension of displacement vector search to fractional accuracy" Example: half-pixel accurate displacements" Δx 4.5 = Δ y 4.5 Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 17

18 Gradient-based displacement estimation Consider a space-time continuous signal f (x,y,t) ( Optical flow: direction x, y,t along which f (x,y,t) is constant" Δ x ( ) Δ ( y x, y,t) 1 )T Derivative in direction of optical flow # ( Δ ( x x, y,t) Δ ( y x, y,t) ) 1 $ % f ( x, y,t) x f ( x, y,t) y f ( x, y,t) t & ' ( T = 0 f f f Δ x + Δ y = x y t Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 18

19 Illustration of optical flow equation f Δ t t (, Δ ) f x t t Δ x f x f ( x, t) f x f x f Δ + Δ = x y y f t Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 19

20 Aperture problem f f f Δ x + Δ y = x y t One equation, two unknowns!" Can only determine component of Δ x, Δ y in direction of brightness gradient (normal flow)" Must combine at least two observations with brightness gradients of different direction" Consider Barber Pole Illusion" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 20

21 Lukas-Kanade Algorithm Assume single displacement within a measurement window W Minimize mean-squared error cost function" Solution" ( ) = ( f x "# x, y$ % Δ x + f "# y x, y $ % Δ + f "# x, y $ y t % ) E Δ x,δ y x,y W ( ) = 2 f x ( f x Δ x + f y Δ y + f t ) Δ x E Δ x,δ y Δ y E Δ x,δ y = 0 x,y W ( ) = 2 f y x,y W ( f x Δ x + f y Δ y + f t ) = 0 2 " f x [x,y] horizontal image gradient f y [x,y] vertical image gradient" f t [x,y] temporal image gradient" # % % $ % x,y W f x 2 x,y W f x f y 2 f x f y f y x,y W x,y W & ( # ( % ' ( $ % Δ x Δ y # & ( ' ( = % % % $ x,y W x,y W f x f t f y f t & ( ( ( ' Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 21

22 Lucas-Kanade Algorithm (cont.) # % % $ % x,y W f x 2 x,y W f x f y 2 f x f y f y x,y W x,y W & ( ( = M Structure matrix! ' ( Structure matrix M must be inverted to obtain Δ x, Δ y M can be singular" l l l If all gradient vectors are zero (flat area of image)" If all gradient vectors point in the same direction (aperture problem)" Measurement window W comprises only one pixel" Not singular for corners, blobs, textured areas " Kanade-Lucas-Tomasi (KLT) tracker" l l Compute eigenvalues λ 1, λ 2 of M Only consider windows, where min(λ 1, λ 2 ) > θ Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 22

23 Sampling issues Image gradients must be approximated by pixel differences, assuming a linear signal in a small spatiotemporal neighborhood" Linearization inaccurate for displacements > 0.5 pixel" Multiple iterations with displacement compensation Multi-scale implementation" Δ x, Δ y " f x Δ + f x y Δ = f y t f x Δ + f x y Δ = f y t f x Δ x + f y Δ y = f t Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 23

24 Lucas-Kanade optical flow Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 24

25 Lucas-Kanade optical flow Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 25

26 Lucas-Kanade optical flow Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 26

27 Parametric displacement estimation Displacement vector field is represented as Δ x (x, y) = a 1 Δ x1 (x, y) + a 2 Δ x2 (x, y) + a 3 Δ x3 (x, y) Δ y (x, y) = b 1 Δ y1 (x, y) + b 2 Δ y2 (x, y) + b 3 Δ y3 (x, y) Optical flow constraint f x Δ + f x y Δ = f y t Combination f x (a Δ + a Δ + a Δ ) + f 1 x1 2 x2 3 x3 y (b Δ + b Δ + b Δ ) = f 1 y1 2 y2 3 y3 t yields a system of linear equations: f x Δ, f x1 x Δ, f x2 x Δ, f x3 y Δ, f y1 y Δ, f y2 y Δ, y2 Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 27 a 1 a 2 a 3 b 1 b 2 b 3 = f t

28 Parametric model of human head Rigid motion: 6 d.o.f." Facial expression: 18 d.o.f." Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 28

29 Facial expression tracking Input video Avatar" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 29

30 Facial expression tracking Input video Avatar" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 30

31 Facial Expression Tracking Input video Avatar" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 31