2D Image Processing INFORMATIK. Kaiserlautern University. DFKI Deutsches Forschungszentrum für Künstliche Intelligenz

Transcription

1 2D Image Processing - Filtering Prof. Didier Stricker Kaiserlautern University DFKI Deutsches Forschungszentrum für Künstliche Intelligenz 1

2 What is image filtering? f(x,y) g(x,y) filtering filtering 2

3 Linear filtering 3

4 Motivation: Image denoising How can we reduce noise in a photograph? 4

5 Image Filtering Methods Spatial Domain Frequency Domain (i.e., uses Fourier Transform) 5

6 Spatial Domain Methods f(x,y) f(x,y) g(x,y) g(x,y) 6

7 Point Processing Methods Convert a given pixel value to a new pixel value based on some predefined function. 7

8 Point Processing Methods - Examples Negative Contrast stretching Thresholding Histogram Equalization 8

9 Area Processing Methods Need to define: (1) Area shape and size (2) Operation output image 9

10 Area Shape and Size Area shape is typically defined using a rectangular mask. Area size is determined by mask size, e.g., 3x3 or 5x5 Mask size is an important parameter! 10

11 Operation Typically linear combinations of pixel values. e.g., weight pixel values and add them together. Different results can be obtained using different weights. e.g., smoothing, sharpening, edge detection). mask 11

12 Practice with linear filters Original Filtered (no change) 12 Source: D. Lowe

13 Practice with linear filters ? Original 13 Source: D. Lowe

14 Practice with linear filters Original Shifted left By 1 pixel 14 Source: D. Lowe

15 Practice with linear filters ? Original 15 Source: D. Lowe

16 Practice with linear filters Original Blur (with a box filter) 16 Source: D. Lowe

17 Practice with linear filters ? Original 17 Source: D. Lowe

18 Practice with linear filters Original Sharpening filter - Accentuates differences with local average 18 Source: D. Lowe

19 Example Local image neighborhood mask 8 Modified image data 19

20 Common Linear Operations Cross-Correlation Cross correlation is a measure of similarity of two series as a function of the displacement of one relative to the other. Wikipedia 20

21 Cross-correlation A filtered image is generated as the center of the mask visits every pixel in the input image. n x n mask h(i,j) g(i,j) f(i,j) filtered image 21

22 Handling Pixels Close to Boundaries wrap around pad with zeroes or 22

23 Correlation Example 23

24 Geometric Interpretation of Correlation Suppose x and y are two n-dimensional vectors: x = (x1, x2,..., xn ) y = ( y1, y2,..., yn ) The dot product of x with y is defined as: x. y = x1 y1 + x2 y xn yn using vector notation: x. y = x y cos(q ) x θ y Correlation generalizes the notion of dot product 24

25 Geometric Interpretation of Correlation (cont d) cos(θ) measures the similarity between x and y xy x. y = x y cos(q ) or cos(q ) = x y Normalized correlation (i.e., divide by lengths) n 2 n 2 å å h(k, l ) f (i + k, j + l ) N (i, j ) = k =n 2 [å k =- n 2 åh n n l =2 2 2 n n l =2 2 n 2 (k, l )]1/ 2 [ å k =- n 2 å n n l =2 2 f 2 (i + k, j + l )]1/ 2 25

26 Zero-mean normalized crosscorrelation The cross-correlation of a template t(x,y), with a subimage f(x,y) is: where n is the number of pixels in t(x,y) and f(x,y), is the average of f and is standard deviation of t. 26

27 Normalized Correlation Measure the similarity between images or parts of images (template matching). mask = 27

28 Application: TV Remote Control 28

29 Application: TV Remote Control (cont d) 29

30 Application : TV Remote Control (cont d) 30

31 Application : TV Remote Control (cont d) 31

32 Application: TV Remote Control (cont d) 32

33 Application : TV Remote Control (cont d) 33

34 Normalized Correlation (cont d) Traditional correlation cannot handle changes due to: size orientation shape (e.g., deformable objects).? 34

35 Common Linear Operations Convolution 35

36 Convolution Same as correlation except that the mask is flipped, both horizontally and vertically g(i, j) = n 2 H V n 2 n 2 n 2 h(k,l) f (i k, j l) = h(i k, j l) f (k,l) k= n n l= 2 2 k= For symmetric masks (i.e., h(i,j)=h(-i,-j)), convolution is equivalent to correlation! n n l= 2 2 Notation: h*f=f*h 36

37 Properties in more detail Commutative: a * b = b * a Conceptually no difference between filter and signal Associative: a * (b * c) = (a * b) * c Often apply several filters one after another: (((a * b1) * b2) * b3) This is equivalent to applying one filter: a * (b1 * b2 * b3) Distributes over addition: a * (b + c) = (a * b) + (a * c) Scalars factor out: ka * b = a * kb = k (a * b) Identity: unit impulse e = [, 0, 0, 1, 0, 0, ], a*e=a 37

38 How do we choose the mask weights? Depends on the application. Usually by sampling certain functions and their derivatives. Gaussian 1st derivative of Gaussian Good for image smoothing 2nd derivative of Gaussian Good for image sharpening 38

39 Normalization of Mask Weights Sum of weights affects overall intensity of output image. Positive weights Normalize them such that they sum to one. Both positive and negative weights are possible 1/9 1/16 39

40 Smoothing Using Averaging Idea: replace each pixel by the average of its neighbors. Useful for reducing noise and unimportant details. The size of the mask controls the amount of smoothing. 40

41 Smoothing Using Averaging (cont d) Trade-off: noise vs. blurring and loss of detail. original 3x3 15x15 5x5 7x7 25x25 41

42 Smoothing Filters: Averaging (cont d) Example: extract largest, brightest objects image thresholding 15 x 15 averaging 42

43 Smoothing with box filter revisited What s wrong with this picture? What s the solution? 43 Source: D. Forsyth

44 Smoothing with box filter revisited What s wrong with this picture? What s the solution? To eliminate edge effects, weight contribution of neighborhood pixels according to their closeness to the center fuzzy blob 44

45 Gaussian Smoothing Idea: replace each pixel by a weighted average of its neighbors Mask weights are computed by sampling a Gaussian function Note: weight values decrease with distance from mask center! 45

46 Gaussian Kernel σ = 2 with 30 x 30 kernel σ = 5 with 30 x 30 kernel Standard deviation s: determines extent of smoothing 46 Source: K. Grauman

47 Choosing kernel width The Gaussian function has infinite support, but discrete filters use finite kernels 47 Source: K. Grauman

48 Gaussian Smoothing (cont d) mask size depends on σ : σ determines the degree of smoothing! σ=3 48

49 Gaussian Smoothing (cont d) Gaussian(sigma, hsize, h) float sigma, *h; int hsize; { int i; float cst, tssq, x, sum; p ( x) = 1 e s 2p ( x-µ )2 2s 2 halfsize=(int)(2.5*sigma); hsize=2*halfsize; if (hsize % 2 == 0) ++hsize; // odd size cst = 1./(sigma*sqrt(2.0*PI)) ; tssq = 1./(2*sigma*sigma) ; for(i=0; i<hsize; i++) { x=(float)(i-hsize/2); h[i]=(cst*exp(-(x*x*tssq))) ; } // normalize 49

50 Gaussian Smoothing - Example = 1 pixel = 5 pixels = 10 pixels = 30 pixels 50

51 Averaging vs. Gaussian Smoothing Averaging Gaussian 51

52 Gaussian vs. box filtering 52

53 Properties of Gaussian Convolution with self is another Gaussian Special case: convolving two times with Gaussian kernel of width is equivalent to convolving once with kernel of width * = 53

54 Properties of Gaussian (cont d) Separable kernel: a 2D Gaussian can be expressed as the product of two 1D Gaussians. 54

55 Properties of Gaussian (cont d) 2D Gaussian convolution can be implemented more efficiently using 1D convolutions: 55

56 Properties of Gaussian (cont d) row get a new image Ir Convolve each column of Ir with g 56

57 Example 2D convolution (center location only) O(n2) The filter factors into a product of 1D filters: Perform convolution along rows: Followed by convolution along the remaining column: * = * = O(2n)=O(n) 57

58 Noise Salt and pepper noise: contains random occurrences of black and white pixels Impulse noise: contains random occurrences of white pixels Gaussian noise: variations in intensity drawn from a Gaussian normal distribution 58 Source: S. Seitz

59 Gaussian noise Mathematical model: sum of many independent factors Good for small standard deviations Assumption: independent, zero-mean noise 59 Source: M. Hebert

60 Reducing Gaussian noise Smoothing with larger standard deviations suppresses noise, but also blurs the image 60

61 Reducing salt-and-pepper noise 3x3 5x5 7x7 What s wrong with the results? 61

62 Alternative idea: Median filtering A median filter operates over a window by selecting the median intensity in the window Is median filtering linear? -> try filter([0 0 0; 0 1 0; 0 0 0] + [1 1 1; 1 1 2; 2 2 2]) 62 Source: K. Grauman

63 Median filter What advantage does median filtering have over Gaussian or average filtering? Robustness to outliers 63 Source: K. Grauman

64 Median filter Salt-and-pepper noise Median filtered MATLAB: medfilt2(image, [h w]) 64 Source: M. Hebert

65 Gaussian vs. median filtering 3x3 5x5 7x7 Gaussian Median 65

66 Smoothing Filters: Median Filtering Very effective for removing salt and pepper noise (i.e., random occurrences of black and white pixels). averaging median filtering 66

67 Review: so far Correlation Convolution Image smoothing Gaussian filter Nonlinear filtering 67

68 Sharpening revisited What does blurring take away? = detail smoothed (5x5) original Let s add it back: +α original = detail 68 sharpened

69 Sharpening Filters: High Boost (cont d) α=2,5 α=1,1 69

70 Unsharp mask filter f + a ( f - f * g ) = (1 + a ) f - a f * g = f * ((1 + a )e - g ) image unit impulse unit impulse (identity) blurred image Gaussian Laplacian of Gaussian 70

71 Image Sharpening Idea: compute intensity differences in local image regions. Useful for emphasizing transitions in intensity (e.g., in edge detection). 1st derivative of Gaussian 71

72 Sharpening Filters (high-pass) Useful for highlighting fine details. The elements of the mask contain both positive and negative weights. Sum of mask elements is 0. 1st derivative of Gaussian 2nd derivative of Gaussian 72

73 Sharpening Filters - Example Note that the response of sharpening might be negative. Values must be re-mapped to [0, 255] input image sharpened image 73

74 Sharpening revisited 74 Source: D. Lowe

75 An application Finding lines of text for OCR book analysis 75

76 Original image Binarization 76

77 Cleaned and inverted image 2D Gaussian filter 77 with Sigma 9.88 and for the two axes

78 2D uniform filter filter size 1.0 and for the two axes 78

79 79

80 Apply the masks obtained in the previous images Each textline is separated from the next textline with a different color (some seem to have the same color, but they are actually not the same, but are separate textlines). 80

81 Thank You! Next time: Edge and point detection! 81