Image Processing, Warping, and Sampling

Transcription

1 Image Processing, Warping, and Sampling Michael Kazhdan ( /657) HB Ch. 4.8 FvDFH Ch

2 Outline Image Processing Image Warping Image Sampling

3 Image Processing What about the case when the modification that we would like to make to a pixel depends on the pixels around it? Blurring Edge Detection Etc.

4 Multi-Pixel Operations Stationary/Local Filtering In the simplest case, we define a mask of weights telling us how values at adjacent pixels should be combined to generate the new value.

5 Blurring To blur across pixels, define a mask: Whose values are non-negative Whose value is largest at the center pixel Whose entries sum to one. Filter = Original Blur

6 Blurring Pixel(x,y): red = 36 green = 36 blue = 0 Filter = Original

7 Blurring Pixel(x,y): red = 36 green = 36 blue = 0 X - 1 X X + 1 Y - 1 Y Filter = Original Y Pixel(x,y).red and its red neighbors

8 Blurring New value for Pixel(x,y).red = (36 * 1/16) + (109 * 2/16) + (146 * 1/16) (32 * 2/16) + (36 * 4/16) + (109 * 2/16) (32 * 1/16) + (36 * 2/16) + (73 * 1/16) X - 1 X X + 1 Y - 1 Y Filter = Original Y Pixel(x,y).red and its red neighbors

9 Blurring New value for Pixel(x,y).red = X - 1 X X + 1 Y - 1 Y Filter = Original Y Pixel(x,y).red and its red neighbors

10 Blurring New value for Pixel(x,y).red = Original Blur

11 Blurring Repeat for each pixel and each color channel Keep source and destination separate to avoid drift. For boundary pixels, not all neighbors are used, and you need to normalize the mask so that the sum of the values is correct.

12 In general, the mask can have arbitrary size: We can express a smaller mask as a bigger one by padding with zeros. Blurring / / Original Narrow Blur

13 In general, the mask can have arbitrary size: We can have more non-zero entries to give rise to a wider blur. Blurring Original Narrow Blur 48 / Wide Blur /

14 Blurring A general way for defining the entries of an n n mask is to use the values of a Gaussian: GaussianMask[i][j] =e di2 +dj 2 2σ 2 σ equals the mask radius (n/2 for an n n mask) di is i s horizontal distance from the center pixel dj is j s vertical distance from the center pixel Don t forget to normalize!

15 Edge Detection An edge is a point in the image where the image is far from constant.

16 Edge Detection To find the edges in an image, define a mask: Whose value is largest at the center pixel Whose entries sum to zero. Edge pixels are those whose value is larger (on average) than those of its neighbors. Filter = Original Detected Edges

17 Edge Detection Pixel(x,y): red = 36 green = 36 blue = 0 Original Filter =

18 Edge Detection Pixel(x,y): red = 36 green = 36 blue = 0 X - 1 X X + 1 Y Original Y Y Pixel(x,y).red and its red neighbors Filter =

19 Edge Detection New value for Pixel(x,y).red = (36 * -1/8) + (109 * -1/8) + (146 * -1/8) (32 * -1/8) + (36 * 1 ) + (109 * -1/8) (32 * -1/8) + (36 * -1/8) + (73 * -1/8) X - 1 X X + 1 Y Original Y Y Pixel(x,y).red and its red neighbors Filter =

20 Edge Detection New value for Pixel(x,y).red = -285/8 X - 1 X X + 1 Y Original Y Y Pixel(x,y).red and its red neighbors Filter =

21 Edge Detection New value for Pixel(x,y).red = 0 X - 1 X X + 1 Y Original Y Y Pixel(x,y).red and its red neighbors Filter =

22 Edge Detection New value for Pixel(x,y).red = 0 Original Detected Edges Filter = Note: Edge values are not colors, so we have to rescale/remap for visualization.

23 Outline Image Processing Image Warping Image Sampling

24 Image Warping Move pixels of image Mapping Resampling Warp Source image Destination image

25 Overview Mapping Forward Inverse Resampling Point sampling Triangle filter Gaussian filter

26 Mapping Define transformation Describe the destination x, y = Φ(u, v) for every location (u, v) in the source v y u x

27 Example Mappings Scale by σ: Φ(u, v) = (σu, σv) y v Scale σ = 0.8 u x

28 Example Mappings Rotate by θ degrees: Φ(u, v) = (u cos θ v sin θ, u sin θ + v cos θ) y v Rotate θ = 30 u x

29 Example Mappings Shear in x by σ x : Φ u, v = u + σ x v, v v y Shear in y by σ y : u Shear x σ x = 1.3 x Φ(u, v) = (u, v + σ y u ) v y u Shear y σ y = 1.3 x

30 Other Mappings Any function of u and v: Φ u, v = Swirl Fish-eye Rain

31 Image Warping Implementation I Forward mapping: for( int v=0 ; v<vmax ; v++ ) for( int u=0 ; u<umax ; u++ ) float (x,y) = Φ(u,v); dst(x,y) = src(u,v); (u, v) Φ (x, y) Source image Destination image

32 Forward Mapping Iterate over source image v y u Rotate + Translate x

33 Forward Mapping BAD! Iterate over source image Multiple source pixels can map to same destination pixel v y u Rotate + Translate x

34 Forward Mapping BAD! Iterate over source image Multiple source pixels can map to same destination pixel Some destination pixels may not be covered v y u Rotate + Translate x

35 Image Warping Implementation II Inverse mapping: for( int y=0 ; y<ymax ; y++ ) for( int x=0 ; x<xmax ; x++ ) float (u,v) = Φ -1 (x,y); dst(x,y) = src(u,v); (u, v) Φ (x, y) Source image Destination image

36 Reverse Mapping GOOD! Iterate over destination image Must resample source May oversample, but much simpler! v y u Rotate Translate x

37 Resampling Evaluate source image at arbitrary (u, v) (u, v) does not usually have integer coordinates (u, v) Source image (x, y) Destination image

38 Overview Mapping Forward Inverse Resampling Nearest Point Sampling Bilinear Sampling Gaussian Sampling

39 Nearest Point Sampling Take value at closest pixel: int iu = floor(u+0.5); int iv = floor(v+0.5); dst(x,y) = src(iu,iv); v y Rotate + Translate u x

40 Bilinear Sampling Bilinearly interpolate four closest source pixels dst(x, y) = Weighted average of source at (u 1, v 1 ), (u 2, v 1 ), (u 1, v 2 ), and (u 2, v 2 ) (u 1, v 2 ) (u 1, v 1 ) (u, v) (u 2, v 2 ) (u 2, v 1 ) v y Rotate + Translate u x

41 Linear Sampling Linearly interpolate two closest source pixels dst(x) = linear interpolation of u 1 and u 2 u u 1 u 2 du u1 = floor( u ); u2 = u1 + 1; du = u u1; dst(x) = src(u1)*(1-du) + src(u2)*du;

42 Bilinear Sampling Bilinearly interpolate four closest source pixels a = linear interpolation of src(u 1, v 1 ) and src(u 2, v 1 ) b = linear interpolation of src(u 1, v 2 ) and src(u 2, v 2 ) dst(x, y) = linear interpolation of a and b (u 1, v 2 ) b u1 = floor( u ), u2 = u1 + 1; v1 = floor( v ), v2 = v1 + 1; du = u u1; (u, v) (u 2, v 2 ) a = src(u1,v1)*(1-du) + src(u2,v1)*( du); b = src(u1,v2)*(1-du) + src(u2,v2)*du; (u 1, v 1 ) a (u 2, v 1 ) dv = v v1; dst(x,y) = a*(1-dv) + b*dv;

43 Bilinear Sampling Bilinearly interpolate four closest source pixels a = linear interpolation of src(u 1, v 1 ) and src(u 2, v 1 ) b = linear interpolation of src(u 1, v 2 ) and src(u 2, v 2 ) dst(x, y) = linear interpolation of a and b (u 1, v 2 ) b (u 2, v 2 ) u1 = floor( u ), u2 = u1 + 1; v1 = floor( v ), v2 = v1 + 1; Make sure to test that the pixels (u, v) (u 1, v 1 ), (u 2, v 2 ), (u 1, v 2 ), and (u 2, v 1 ) are within the image. du = u u1; a = src(u1,v1)*(1-du) + src(u2,v1)*( du); b = src(u1,v2)*(1-du) + src(u2,v2)*du; dv = v v1; dst(x,y) = a*(1-dv) + b*dv; (u 1, v 1 ) a (u 2, v 1 )

44 Gaussian Sampling Compute weighted sum of pixel neighborhood: The blending weights are the normalized values of a Gaussian function. (u, v)

45 Filtering Methods Comparison Trade-offs Jagged edges versus blurring Computation speed Nearest Bilinear Gaussian

46 Image Warping Implementation Inverse mapping: for( int y=0 ; y<ymax ; y++ ) for( int x=0 ; x<xmax ; x++ ) float (u,v) = Φ -1 (x,y); dst(x,y) = resample_src(u,v,w); (u, v) Φ (x, y) Source image Destination image

47 Image Warping Implementation Inverse mapping: for( int y=0 ; y<ymax ; y++ ) for( int x=0 ; x<xmax ; x++ ) float (u,v) = Φ -1 (x,y); dst(x,y) = resample_src(u,v,w); (u, v) w Φ (x, y) Source image Destination image

48 Example: Scale Scale( src, dst, σ ): float w?; for( int y=0 ; y<ymax ; y++ ) for( int x=0 ; x<xmax ; x++ ) float (u,v) = (x,y) / σ; dst(x,y) = resample_src(u,v,w); v (u, v) y w = 1 σ u Scale σ = 0.5 (x, y) x

49 Example: Rotate Rotate( src, dst, θ ): float w?; for( int y=0 ; y<ymax ; y++ ) for( int x=0 ; x<xmax ; x++ ) float (u,v) = ( x*cos(-θ) - y*sin(-θ), x*sin(-θ) + y*cos(-θ) ); dst(x,y) = resample_src(u,v,w); y v (u, v) (x, y) w = 1 x = u cos θ v sin θ y = u sin θ + v cos θ u Rotate θ = 30 x

50 Example: Fun Swirl( src, dst,θ ): float w?; for( int y=0 ; y<ymax ; y++ ) for( int x=0 ; x<xmax ; x++ ) float (u,v) = rot( (xc,yc),(x,y), dist((x,y)-(xc,yc))*theta); dst(x,y) = resample_src(u,v,w); v (u,v) y f (x,y) Swirl u x

51 Outline Image Processing Image Warping Image Sampling

52 Sampling Questions How should we sample an image: Nearest Point Sampling? Bilinear Sampling? Gaussian Sampling? Something Else?

53 Image Representation What is an image? An image is a discrete collection of pixels, each representing a sample of a continuous function. Continuous image Digital image

54 Sampling Let s look at a 1D example: Continuous Function Discrete Samples

55 Sampling At in-between positions, values are undefined. How do we determine the value of a sample? We need to reconstruct a continuous function, turning a collection of discrete samples into a 1D function that we can sample at arbitrary locations.? Discrete Samples

56 Nearest Point Sampling The value at a point is the value of the closest discrete sample.? Reconstructed Function Discrete Samples

57 Nearest Point Sampling The value at a point is the value of the closest discrete sample. The reconstruction: Interpolates the samples Is not continuous? Reconstructed Function Discrete Samples

58 Bilinear Sampling The value at a point is the (bi)linear interpolation of the two surrounding samples.? Reconstructed Function Discrete Samples

59 Bilinear Sampling The value at a point is the (bi)linear interpolation of the two surrounding samples. The reconstruction: Interpolates the samples Is not smooth? Reconstructed Function Discrete Samples

60 Gaussian Sampling The value at a point is the Gaussian average of the surrounding samples.? Reconstructed Function Discrete Samples

61 Gaussian Sampling The value at a point is the Gaussian average of the surrounding samples. The reconstruction: Does not interpolate Is smooth? Reconstructed Function Discrete Samples

62 Image Sampling Typically this is done in two steps: 1. Reconstruct a continuous function from input samples. 2. Sample a continuous function at a fixed resolution. (1) (2) Challenge: Reconstruction is an under-constrained problem. To make headway, we need to define what makes a reconstruction good.

63 Image Sampling Typically this is done in two steps: 1. Reconstruct a continuous function from input samples. 2. Sample a continuous function at a fixed resolution. (1) (2) Challenge: Reconstruction Key Idea: is an under-constrained problem. Of all possible reconstructions, we want the one that is smoothest To make (has headway, lowest frequencies). we need to define what makes a reconstruction good. Signal processing helps us formulate this precisely.

64 Fourier Analysis Fourier analysis provides a way for expressing (or approximating) any signal as a sum of scaled and shifted cosine functions. cos(0) cos(1) cos(2) cos(3) cos(4) cos(5) cos(6) The Building Blocks for the Fourier Decomposition

65 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 0 th Order Approximation f(θ) f 0 θ = a 0 cos 0 θ + φ 0 0 th Order Component

66 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 1 st Order Approximation f(θ) 0 th Order Approximation + f 1 θ = a 1 cos 1 θ + φ 1 1 st Order Component

67 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 2 nd Order Approximation f(θ) 1 st Order Approximation + f 2 θ = a 2 cos 2 θ + φ 2 2 nd Order Component

68 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 3 rd Order Approximation f(θ) 2 nd Order Approximation + f 3 θ = a 3 cos 3 θ + φ 3 3 rd Order Component

69 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 4 th Order Approximation f(θ) 3 rd Order Approximation + f 4 θ = a 4 cos 4 θ + φ 4 4 th Order Component

70 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 5 th Order Approximation f(θ) 4 th Order Approximation + f 5 θ = a 5 cos 5 θ + φ 5 5 th Order Component

71 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 6 th Order Approximation f(θ) 5 th Order Approximation + f 6 θ = a 6 cos 6 θ + φ 6 6 th Order Component

72 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 7 th Order Approximation f(θ) 6 th Order Approximation + f 7 θ = a 7 cos 7 θ + φ 8 7 th Order Component

73 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 8 th Order Approximation f(θ) 7 th Order Approximation + f 8 θ = a 8 cos 8 θ + φ 8 8 th Order Component

74 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 9 th Order Approximation f(θ) 8 th Order Approximation + f 9 θ = a 9 cos 9 θ + φ 9 9 th Order Component

75 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 10 th Order Approximation f(θ) 9 th Order Approximation + f 10 θ = a 10 cos 10 θ + φ th Order Component

76 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 11 th Order Approximation f(θ) 10 th Order Approximation + f 11 θ = a 11 cos 11 θ + φ th Order Component

77 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 12 th Order Approximation f(θ) 11 th Order Approximation + f 12 θ = a 12 cos 12 θ + φ th Order Component

78 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 13 th Order Approximation f(θ) 12 th Order Approximation + f 13 θ = a 13 cos 13 θ + φ th Order Component

79 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 14 th Order Approximation f(θ) 13 th Order Approximation + f 14 θ = a 14 cos 14 θ + φ th Order Component

80 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 15 th Order Approximation f(θ) 14 th Order Approximation + f 15 θ = a 15 cos 15 θ + φ th Order Component

81 Fourier Analysis As higher frequency components are added to the approximation, finer details are captured. Initial Function 16 th Order Approximation f(θ) 15 th Order Approximation + f 16 θ = a 16 cos 16 θ + φ th Order Component

82 Fourier Analysis Combining all of the frequency components together, we get the initial function: f θ = k=0 f k θ = k=0 a k cos k θ + φ k a k : amplitude of the k th frequency component φ k : shift of the k th frequency component Initial Function f(θ) f 0 () f 1 () f 2 () f 3 () f 4 () = f 5 () f 6 () f 7 () f 8 ()

83 Question As higher frequency components are added to the approximation, finer details are captured. If we have n samples, what is the highest frequency that can be represented? Initial Function f(θ) f 0 () f 1 () f 2 () f 3 () f 4 () = f 5 () f 6 () f 7 () f 8 ()

84 Question As higher frequency components are added to the approximation, finer details are captured. If we have n samples, what is the highest frequency that can be represented? Each frequency component has two degrees of freedom: Amplitude Shift f( ) With n samples we can + represent + the first + n/2 frequency + components = Initial Function f 0 () f 1 () f 2 () f 3 () f 4 () f 5 () f 6 () f 7 () f 8 ()

85 Sampling Theorem Shannon s Theorem: A signal can be reconstructed from its samples, if the original signal has no frequencies above 1/2 the sampling rate -- a.k.a. the Nyquist Frequency. Definition: A signal is band-limited if its highest non-zero frequency is bounded. The frequency is called the bandwidth. The minimum sampling rate for band-limited function is called the Nyquist rate (twice the bandwidth).

86 Image Sampling 1. To reconstruct the continuous function from m samples, we can find the unique function of frequency m/2 that interpolates the values. 2. Why don t we just evaluate this function at the n sample positions? If n < m we sample below the Nyquist rate! (1) (2) m samples n samples

87 Aliasing When a high-frequency signal is sampled with insufficiently many samples, it can be perceived as a lower-frequency signal. This masking of higher frequencies as lower ones is referred to as aliasing. -

88 Aliasing When a high-frequency signal is sampled with insufficiently many samples, it can be perceived as a lower-frequency signal. This masking of higher frequencies as lower ones is referred to as aliasing. -

89 Aliasing When a high-frequency signal is sampled with insufficiently many samples, it can be perceived as a lower-frequency signal. This masking of higher frequencies as lower ones is referred to as aliasing. -

90 Aliasing When a high-frequency signal is sampled with insufficiently many samples, it can be perceived as a lower-frequency signal. This masking of higher frequencies as lower ones is referred to as aliasing. -

91 Temporal Aliasing Artifacts due to limited temporal resolution 10 fps

92 Temporal Aliasing Artifacts due to limited temporal resolution 10 fps 25 fps

93 Temporal Aliasing Artifacts due to limited temporal resolution

94 Temporal Aliasing Artifacts due to limited temporal resolution

95 Temporal Aliasing Artifacts due to limited temporal resolution

96 Temporal Aliasing Artifacts due to limited temporal resolution

97 Sampling There are two problems: You don t have enough samples to correctly reconstruct your high-frequency information You corrupt the low-frequency information because the high-frequencies mask themselves as lower ones.