Displacement estimation Displacement estimation by block matching" l Search strategies" l Subpixel estimation" Gradient-based displacement estimation ( optical flow )" l Lukas-Kanade" l Multi-scale coarse-to-fine" l Parametric displacement estimation" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 1
Where is the defect? Image g[x,y] (no defect)" Image f [x,y] (w/ defect)" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 2
Absolute difference between two images f-g w/o alignment" f-g w/ alignment" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 3
Displacement estimation by block matching Measurement window is compared with a shifted array of pixels in the other image, to determine the best match" Rectangular array of pixels is selected as a measurement window" Image g[x,y] " Image f [x,y] " Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 4
Displacement estimation by block matching Image g[x,y] " Image f [x,y] "... process repeated for another measurement window position." Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 5
Integer pixel shifts Measurement window is compared with a shifted array of pixels in the other image, to determine the best match" Rectangular array of pixels is selected as a measurement window" Image g[x,y] " Image f [x,y] " Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 6
Integer pixel shifts 28 42 42 43 44 40 32 20 29 32 22 54 53 52 49 31 21 30 44 45 45 45 42 30 21 26 27 18 62 63 59 60 44 33 35 54 54 54 53 54 53 52 54 53 52 49 53 52 49 31 52 49 31 21 49 31 21 28 31 21 24 21 15 120 114 112 111 80 32 40 63 62 62 63 62 63 59 62 63 59 60 63 59 60 44 59 60 44 33 60 44 33 35 44 33 31 33 26 130 128 124 125 88 24 74 121 120 120 114 120 114 112 120 114 112 111 114 112 111 80 112 111 80 32 111 80 32 23 80 32 22 32 17 131 124 127 127 96 42 79 127 130 130 128 130 128 124 130 128 124 125 128 124 125 88 124 125 88 125 88 24 17 88 24 20 24 13 80 129 131 131 124 131 124 127 131 124 127 124 127 96 127 96 42 127 96 42 29 96 42 28 42 19 50 78 77 77 71 77 71 73 77 71 73 75 71 73 75 63 73 75 63 52 75 63 52 47 63 52 46 52 29 77 71 73 75 63 52 Rectangular array of pixels is selected as a measurement window" 22 37 37 37 39 40 40 41 41 38 25 Measurement window is compared with a shifted array of pixels in the other image, to determine the best match" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 7
Error metric Sum of Squared Differences! Sum all values in measurement window" SSD Δ x,δ y = f x, y g x + Δ x, y + Δ y x,y msmnt window ( ) 2 Horizontal displacement" Vertical displacement" Alternatives: SAD (Sum of Absolute Differences), cross correlation, mutual information... " Robustness against outliers: sum of saturated squared differences, median of squared differences... " Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 8
SSD values resulting from block matching 1.6 SSD" 1.4 1.2 1 0.8 0.6 0.4 Estimated displacement" 0.2 Integer-pixel accuracy" 0 0 5 Vertical shift Δ y" " 10 15 2 4 8 6 14 12 10 Horizontal shift Δ x" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 9
Block matching: search strategies Full search" Δd y Δd x All possible displacements within the search range are compared." Computationally expensive" Highly regular, parallelizable" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 10
Block matching: search strategies Conjugate direction search" Δd y Δd x Alternate search in x and y directions" Stop when there is no further improvement" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 11
Block matching: search strategies Coarse-to-fine" Δd y Δd x Start with coarsely spaced candidate displacements" Smaller pattern when best match is in the middle" Stop when desired displacement accuracy is reached" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 12
Block matching: search strategies Diamond search [Li, Zeng, Liou, 1994] [Zhu, Ma, 1997]" Δd y Δd x Start with large diamond pattern at [0,0]" If best match lies in the center of large diamond, proceed with small diamond" If best match does not lie in the center of large diamond, center large diamond pattern at new best match" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 13
Absolute difference between images w/ integer-pixel alignment" w/o alignment" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 14
Interpolation of the SSD Minimum SSD" Sub-pixel" Accurate" Minimum" Fit parabola through" 3 > 3 points" " exactly" approximately" Horizontal shift Δ x" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 15
2-d Interpolation of SSD Minimum SSD" Paraboloid" Perfect fit through 6 points" Approximate fit through > 6 points" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 16
Sub-pixel accuracy Interpolate pixel raster of the reference image to desired sub-pixel accuracy (e.g., by bi-linear or bi-cubic interpolation)" Straightforward extension of displacement vector search to fractional accuracy" Example: half-pixel accurate displacements" Δx 4.5 = Δ y 4.5 Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 17
Gradient-based displacement estimation Consider a space-time continuous signal f (x,y,t) ( Optical flow: direction x, y,t along which f (x,y,t) is constant" Δ x ( ) Δ ( y x, y,t) 1 )T Derivative in direction of optical flow # ( Δ ( x x, y,t) Δ ( y x, y,t) ) 1 $ % f ( x, y,t) x f ( x, y,t) y f ( x, y,t) t & ' ( T = 0 f f f Δ x + Δ y = x y t Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 18
Illustration of optical flow equation f Δ t t (, Δ ) f x t t Δ x f x f ( x, t) f x f x f Δ + Δ = x y y f t Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 19
Aperture problem f f f Δ x + Δ y = x y t One equation, two unknowns!" Can only determine component of Δ x, Δ y in direction of brightness gradient (normal flow)" Must combine at least two observations with brightness gradients of different direction" Consider Barber Pole Illusion" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 20
Lukas-Kanade Algorithm Assume single displacement within a measurement window W Minimize mean-squared error cost function" Solution" ( ) = ( f x "# x, y$ % Δ x + f "# y x, y $ % Δ + f "# x, y $ y t % ) E Δ x,δ y x,y W ( ) = 2 f x ( f x Δ x + f y Δ y + f t ) Δ x E Δ x,δ y Δ y E Δ x,δ y = 0 x,y W ( ) = 2 f y x,y W ( f x Δ x + f y Δ y + f t ) = 0 2 " f x [x,y] horizontal image gradient f y [x,y] vertical image gradient" f t [x,y] temporal image gradient" # % % $ % x,y W f x 2 x,y W f x f y 2 f x f y f y x,y W x,y W & ( # ( % ' ( $ % Δ x Δ y # & ( ' ( = % % % $ x,y W x,y W f x f t f y f t & ( ( ( ' Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 21
Lucas-Kanade Algorithm (cont.) # % % $ % x,y W f x 2 x,y W f x f y 2 f x f y f y x,y W x,y W & ( ( = M Structure matrix! ' ( Structure matrix M must be inverted to obtain Δ x, Δ y M can be singular" l l l If all gradient vectors are zero (flat area of image)" If all gradient vectors point in the same direction (aperture problem)" Measurement window W comprises only one pixel" Not singular for corners, blobs, textured areas " Kanade-Lucas-Tomasi (KLT) tracker" l l Compute eigenvalues λ 1, λ 2 of M Only consider windows, where min(λ 1, λ 2 ) > θ Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 22
Sampling issues Image gradients must be approximated by pixel differences, assuming a linear signal in a small spatiotemporal neighborhood" Linearization inaccurate for displacements > 0.5 pixel" Multiple iterations with displacement compensation Multi-scale implementation" Δ x, Δ y " f x Δ + f x y Δ = f y t f x Δ + f x y Δ = f y t f x Δ x + f y Δ y = f t Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 23
Lucas-Kanade optical flow Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 24
Lucas-Kanade optical flow Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 25
Lucas-Kanade optical flow Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 26
Parametric displacement estimation Displacement vector field is represented as Δ x (x, y) = a 1 Δ x1 (x, y) + a 2 Δ x2 (x, y) + a 3 Δ x3 (x, y) Δ y (x, y) = b 1 Δ y1 (x, y) + b 2 Δ y2 (x, y) + b 3 Δ y3 (x, y) Optical flow constraint f x Δ + f x y Δ = f y t Combination f x (a Δ + a Δ + a Δ ) + f 1 x1 2 x2 3 x3 y (b Δ + b Δ + b Δ ) = f 1 y1 2 y2 3 y3 t yields a system of linear equations: f x Δ, f x1 x Δ, f x2 x Δ, f x3 y Δ, f y1 y Δ, f y2 y Δ, y2 Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 27 a 1 a 2 a 3 b 1 b 2 b 3 = f t
Parametric model of human head Rigid motion: 6 d.o.f." Facial expression: 18 d.o.f." Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 28
Facial expression tracking Input video Avatar" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 29
Facial expression tracking Input video Avatar" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 30
Facial Expression Tracking Input video Avatar" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Displacement Estimation 31