Image preprocessing in spatial domain Sampling theorem, aliasing, interpolation, geometrical transformations Revision:.4, dated: May 25, 26 Tomáš Svoboda Czech Technical University, Faculty of Electrical Engineering Center for Machine Perception, Prague, Czech Republic svoboda@cmp.felk.cvut.cz http://cmp.felk.cvut.cz/~svoboda Reminder Convolution theorem The Fourier transform of a convolution is the product of the Fourier transforms. F{f(x, y) h(x, y)} = F (u, v)h(u, v) The Fourier transform of a product is the convolution of the Fourier transforms. F{f(x, y)h(x, y)} = F (u, v) H(u, v) 2/9 Impulse (delta) function 3/9 δ(x) = H(x) x where H(x) is the Heaviside step function Sifting property scaling property f(x)δ(x a)dx = f(a) δ(ax) = a δ(x) Eric W. Weisstein. Heaviside Step Function. From MathWorld A Wolfram Web Resource. http://mathworld.wolfram.com/heavisidestepfunction.html
Replication of delta function Shah function 4/9 III(x) = δ(x k) k= 2 2 Eric W. Weisstein. Shah Function. From MathWorld A Wolfram Web Resource. http://mathworld.wolfram.com/shahfunction.html 2D Shah function bed and nails 5/9 y x III(x, y) = δ(x k, y l) k= l= For unit intervals 2D Shah function properties 6/9 III(x, y) = δ(x k, y l) k= l= If we need samples spaced X and Y ( x III X, y ) = XY Y k= l= because of the scaling property of the delta function δ(ax) = a δ(x) δ(x kx, y ly )
2D Shah function Fourier pair 7/9 ( x F{III X, y ) } = XY III (Xu, Y v) Y The Shah function is its own transform but the frequency inverses! This will have important consequences! Sampling by using the Shah function Product of a function with the Shah function. 8/9 s(x, y) = III(x, y)f(x, y) We know from convolution theorem that S(u, v) = III(u, v) F (u, v) Sampling theorem 9/9 3 3 Image taken from []
Sampling theorem Aliasing /9 Sampling theorem Aliasing /9 Sampling theorem Aliasing 2/9
Sampling theorem Aliasing 3/9 From continuous to discrete and back again 4/9 2 4 6 8 2.5.5 2 4 6 8 2 x 4 2 4 6 8 2 x 4 From continuous to discrete and back again 5/9
From continuous to discrete and back again 6/9 From continuous to discrete and back again 7/9 From continuous to discrete and back again 8/9
From continuous to discrete and back again 9/9 From continuous to discrete and back again 2/9 From continuous to discrete and back again 2/9
From continuous to discrete and back again 22/9 From continuous to discrete and back again 23/9 Sum the sinc functions up. From continuous to discrete and back again 24/9 Compare the reconstructed result with the original function.
D sampling T vz = T max 8 25/9.5.5.5.5 2 4 6 8 2.5.5 2 4 6 8 2 x 4.5.5.5.5 2 4 6 8 2 x 4 D sampling T vz = T max 2 26/9.5.5.5.5 2 4 6 8 2.5.5 2 4 6 8 2 x 4.5.5.5.5 2 4 6 8 2 x 4 D sampling T vz = T max 27/9.5.5.5.5 2 4 6 8 2.5.5 2 4 6 8 2 x 4.5.5.5.5 2 4 6 8 2 x 4
D sampling T vz = 2. T max 28/9.5.5.5.5 2 4 6 8 2.5.5 2 4 6 8 2 x 4.5.5.5.5 2 4 6 8 2 x 4 Image aliasing example % 29/9 Aliasing example 9% 3/9
Aliasing example 8% 3/9 Aliasing example 7% 32/9 Aliasing example 6% 33/9
Aliasing example 5% 34/9 Aliasing example 4% 35/9 Aliasing example 3% 36/9
Aliasing example 2% 37/9 Aliasing example % 38/9 Aliasing problem in digital photography 39/9
Aliasing example 9% 4/9 Aliasing example 8% 4/9 Aliasing example 7% 42/9
Aliasing example 6% 43/9 Aliasing example 5% 44/9 Aliasing example 4% 45/9
Aliasing example 3% 46/9 Aliasing example 2% 47/9 Aliasing example % 48/9
What can we do against aliasing? Increase sampling frequency number of pixels (but not only, optics is also important) Decrease the frequency blurring The same snapshots but the input image blurred (convolved) with 5 5 Gaussian with σ = 2: 49/9 Suppressed Aliasing 5/9 Suppressed Aliasing example 9% 5/9
Suppressed Aliasing example 8% 52/9 Suppressed Aliasing example 7% 53/9 Suppressed Aliasing example 6% 54/9
Suppressed Aliasing example 5% 55/9 Suppressed Aliasing example 4% 56/9 Suppressed Aliasing example 3% 57/9
Suppressed Aliasing example 2% 58/9 Suppressed Aliasing example % 59/9 Sampling, Aliasing Revisited Sampling is an important issue in images. 6/9 Aliasing is typically not wanted. Aliasing is ubiquitous. Toy www example4 Defeating Aliasing Low-pass filter before subsampling Subsampling by using image interpolation (will come to that later) 4 http://cmp.felk.cvut.cz/cmp/courses/ezs/demos/aliasing/
Geometrical Transformation Transformation of spatial coordinates 6/9 Geometrical Transformation What for? intentional image transformations (you know where to go) 62/9 resizing rotation shift warping, texture mapping correction of distortions (you know how it should look) projective skew non-linear distortion (fish-eyes) Techniques shared by Image processing, Computer graphics, even Robotics or Mechanics. Example of texture mapping 63/9
Realization Rotation and shift 64/9 More elegant and efficient in a matrix form x y x = cos(α)x + sin(α)y + t x y = sin(α)x + cos(α)y + t y = cos(α) sin(α) t x sin(α) cos(α) t y x = T x where x and x are homogeneous coordinates: x = [λx, λy, λ] T, λ. x y Identity 65/9 T = 5 identity 5 2 25 5 5 2 25 Rotation 66/9 T = for α = T = cos(α) sin(α) sin(α) cos(α).9848.736.736.9848 5 5 2 25 rotation 5 5 2 25
Rotation + translation 67/9 T = cos(α) sin(α) t x sin(α) cos(α) t y for α = and t = [ 5, 3] T T =.9848.736 5.736.9848 3 5 5 2 25 3 Euclidean 5 5 5 2 Affine 68/9 T = in general T =.9.3 6 degrees of freedom a b c d e f 5 5 5 2 affine 25 5 5 2 25 Projective 69/9 T =.445.47 98.4.8.99 5... 55 6 projective in general T = a b c d e f g h 65 7 75 8 85 8 degrees of freedom 9 95 5 5 5 2 25
Correction of converging lines I 7/9 Correction of converging lines II 7/9 Correction of converging lines III 72/9
x = T x; Forward mapping 73/9 I(x,y) I (x,y ) Forward mapping problems Maps outside the pixel locations. 74/9 I(x,y) I (x,y ) Forward mapping problems Solution: Spread out the effect of each pixel 75/9 I(x,y) I (x,y )
Forward mapping problems May produce holes in the output 76/9 I(x,y) I (x,y ) Forward mapping problems May produce holes in the output 77/9 I(x,y) I (x,y ) Forward mapping problems May produce holes in the output 78/9 I(x,y) I (x,y )
Forward mapping problems May produce holes in the output 79/9 I(x,y)? I (x,y ) x = T x Backward mapping 8/9 I(x,y) I (x,y ) Backward mapping problems Does not always map from a pixel 8/9 I(x,y) I (x,y )
Backward mapping problems Solution: Interpolate between pixels 82/9 I(x,y) I (x,y ) Interpolation as 2D convolution We have sampled (possibly outside the regular grid) function s(x, y) instead of the wanted f(x, y). We want to find a good approximation by convolution ˆf(x, y) = k= l= s(k, l)h(x k, y l) 83/9 Interpolation nearest neighbour 84/9
Interpolation bilinear 85/9 ˆf(x, y) = ( a)( b) s(l, k) + a( b) s(l +, k) +b( a) s(l, k + ) + ab s(l +, k + ), where l = round (x), a = x l, k = round (y), b = y k. Just D, for clarity Interpolation bicubic 86/9 2 x 2 + x 3 pro x <, ˆf = 4 8 x + 5 x 2 x 3 pro x < 2, elsewhere. Does not remind you the shape something? sinc(x)! The ideal reconstructor. Interpolation nearest vs. bilinear 87/9 rotation rotation 5 5 5 5 2 2 25 25 5 5 2 25 nearest 5 5 2 25 bilinear
Interpolation nearest vs. bilinear close up 88/9 affine affine 55 55 6 6 65 65 7 7 95 5 nearest 75 95 5 bilinear Geometrical Transformation Summary Closed form solution, all pixels undergo the same transformation 89/9 rotation, scaling, translation affine, perspective General, deformation meshes. Transformation depends on position (warping, morphing) References 9/9 [] Ronald N. Bracewell. Fourier analysis and imaging. Kluwer Academic/Plenum Publishers, New York, USA, 23.