Point and Spatial Processing

Transcription

1 Filtering 1

2 Point and Spatial Processing

3 Spatial Domain g(x,y) = T[ f(x,y) ] f(x,y) input image g(x,y) output image T is an operator on f Defined over some neighborhood of (x,y) can operate on a set of Images

4 Gray-level transformation Simplest form of T Pixel neighborhood is 1 x 1 Notation: s = T(r) r, s denote gray level of f(x,y) and g(x,y) for any point (x,y)

5 Plot of s=t(r) 255 Lighter s T(r) Image using pixels r 0 r 255 r = input pixel intensity s = output pixel intensity Image using pixels s

6 Example of T(r) Lighter s Lighter s T(r) Darker r Lighter r Lighter

7 Image Histogram Image Histogram digital image with gray level [0, L-1] p(r k ) = n k /N r k is the k th gray level n k number of pixels with k th gray level N total number of pixels k=0,1,2,3,4,5..., L-1

8 Image Histogram p(r k ) is the probability of the occurrence of gray-level r k p(r k ) r k

9 Image Histogram P(r k ) Bright Image P(r k ) Dark Image r k r k

10 Image Histogram P(r k ) Low Contrast P(r k ) High Contrast r k r k Contrast is ratio of L_max to L_min (i.e. max intensity and min intensity)

11 Image Histogram Keep in mind that histograms are not unique

12 Point-processing Lighter s T(r) Lighter s T(r) Darker r Lighter r s=t(r) manipulates an image s histogram

13 Contrast Stretching High Contrast P(r k ) L-1 s T(r) r r L-1?

14 Contrast Stretching High Contrast P(r k ) L-1 s T(r) r r L-1 P(s k ) s

15 Contrast Compressing Low Contrast P(r k ) r? P(s k ) s

16 Contrast Compressing Low Contrast P(r k ) r L-1 s T(r) P(s k ) r L-1 s

17 Properties of T(r) Non-Monotonic (increasing or decreasing) Monotonic s r T(r) No-inverse Doesn t preserve gray level ordering Looks very unnatural s T(r) r Inverse Preserves gray level ordering (monotonic increasing) Looks natural

18 Photoshop T(r) manipulation Look under Image->adjustments->curves The curve is the function T, i.e: s=t(r) or output=t(input)

19 Histogram Equalization Say we want an image with equally many pixels at every gray level This makes the image look nice Also maximizes pixel resources We would say this has an equal histogram So, we want a flat histogram, where each gray level, r k, appears (N/r m ) times where r m is the maximum gray level N is number of pixels in the image

20 Histogram Equalization Following transform derived from the inputs histogram itself is the T needed to equalize the histogram (at least the best approximation) It is discrete approximation of the cumulative distribution function (CDF) s k = T ( r k ) = k j= 0 n j / N = k j= 0 p r ( r j ) s k is output intensity r k is input intensity n j is number of pixels with jth gray level k = 0, 1, 2, 3,.. L-1 (gray levels)

21 Example n k Image Histogram (Notice, this is not normalized, y axis is n k. To normalize, let y axis = p(r k )= n k /N r k

22 Example k j= 0 n j n k Histogram (Notice, this is not normalized, y axis is n k. To normalize, let y axis = p(r k )= n k /N r k r k

23 Example n k Np( r k ) = n j k j= 0 n k r k T(r) s k

24 Histogram Equalization Can significantly improve image appearance Automatic Derived completely from the image input Nice pre-processing step before image comparison Account for different lighting conditions Account for different camera/device properties

25 Spatial Domain Spatial Filtering Processing a pixel using neighborhood information Two main types Linear filters Non-linear filters Linear filters Foundation based on the convolution theorem g(x,y) = h(x,y)*f(x,y) <- spatial operator G(u,v) = H(u,v)F(u,v) <- frequency operator Goal is typically to either remove, or isolate frequencies in the image Non-linear filters Typically based on image statistics Goal is to remove noise from image

26 Spatial Filtering using a Mask Neighborhood operators z1 z2 z3 z4 z5 z6 z7 z8 z9 Pixels w1 w2 w3 w4 w5 w6 w7 w8 w9 Mask Response of a linear mask, R, R = ( w1z1 + w2z2 + w3z w9z9) = w i z i

27 Applying a Spatial mask Filter* h Image f Image g Apply mask about pixel f(x,y) to get value g(x,y) *this sub-image of coefficients has various terms: filter, mask, kernel, etc.

28 Convolution g = f h Apply the filter for each pixel in the image. We often call this process convolution. That is, we are convolving a mask, or convolution kernel against the image. f input image h is convolution kernel (i.e. mask) g output image

29 Smoothing Filters Smoothing Filters blurring pre-processing removal of small details before object extraction noise reduction removal of noise in an image Often called Low-Pass Filters Filter lets low-frequencies pass Stops high-frequencies

30 Low-pass spatial filtering One requirement for a low-pass filter is that w i be positive w1 w2 w3 w4 w5 w6 w7 w8 w9 Note, that the result can be larger than the valid output range(l-1) We typically pre-scale the filter scale_factor = i 1 wi

31 Example original n=5 (nxn mask) n=15 (nxn mask) n=25 (nxn mask)

32 Other arrangements Weighted Filter Filter should be normalized by sum of the mask coefficients.

33 Isotropic Gaussian Filter G x + y 1 ( x, y) = e 2 2σ 2 2πσ 2 2 2D Gaussian distribution with σ=1 *Isotropic means the same in all directions.

34 Discrete Gaussian Filter Center the mask at (0,0). So, a 5x5 mask would compute values by at: G x + y 1 2 2σ ( x, y) = e 2 2πσ Use G(x,y) to compute values for mask. 2 2

35 Example G x + y 1 2 2σ ( x, y) = e 2 2πσ 2 2 Normalize by the sum of the filter.

36 Linear vs. Non-linear Convolution /Linear Filters Linear operation Have corresponding frequency domain filter Non-linear Filters Mask used to determine the proper substitution of a good pixel value Examine neighbors using various orderings Often use Rank or Order Statistics Harder to interpret effect in frequency domain

37 Ordered Statistic Filters Also called rank filters Consider a neighborhood about a pixel. Rank (sort) the pixels. {2, 2, 3, 3, 4, 4, 8, 9, 10}

38 Rank Filters: Median Filter One of the most popular non-linear filter Find the median of the window Preserves edges Removes impulse noise, avoids excessive smoothing pixel values about (x,y) window 3x3 neighbor sort = {2,2,3,3,4,4,8,9,10} f(x,y) = median

39 Comparison Input image Low-pass liner filter Median Filter

40 Rank Filters: Min/Max Filter Find the min or max of the neighborhood Not as mainstream as median filter Has various uses, will talk about these more later neighbor sort = {2,2,3,3,4,4,8,9,10} pixel values about (x,y) window 3x3 f(x,y) = min f(x,y) = max

41 Examples Original Image Median Min Max

42 Derivatives Derivative = measure of local change Requirements for 1 st derivative (1) Must be 0 for flat regions (2) Must be non-zero at the onset of a step or ramp (3) Must be non-zero along ramps Consider 1D case (along x axis) f x = f ( x + 1) f ( x)

43 Derivatives Requirements for 2 nd derivative (1) Must be 0 for flat regions (2) Must be non-zero at the onset of a step or ramp (3) Must be zero along constant slopes Consider 1D case (along x axis) 2 2 f x = f ( x + 1) + f ( x 1) 2 f ( x)

44 Derivative Filters Gray-level Profile of this scan-line First derivative Second

45 First Order Derivatives Called the Image Gradient Function of 2 variables x, y f = f x f y y x

46 First Order Derivatives For each (x,y) you are storing two values: Often have two images to represent this X-Gradient and Y-Gradient Computed independently = y f x f f

47 Examples gx gy

48 Gradient Gradient Magnitude Gradient Angle 2 1/ 2 2 y f x f f) mag( + = = f This is considered the strength of the gradient. = Ψ x f/ y f/ tan ) ( 1 f Note: the angle is with respect to the x image axis

49 1 st Derivate Masks Prewitt Masks Sobel Masks Variations for 45 degrees.

50 2 nd Derivative Laplacian operator provides the 2 nd order derivative ), ( 2 ) 1, ( ) 1, ( 2 2 y x f y x f y x f x f + + = ), ( 2 1), ( 1), ( 2 2 y x f y x f y x f y f + + = ), ( 4 )] 1, ( ) 1, ( 1), ( 1), ( [ 2 y x f y x f y x f y x f y x f f = X-direction y-direction Combined (just sum them together)

51 Derivation )] 1, ( ), ( [ )], ( ) 1, ( [ y x f y x f y x f y x f + ), ( 2 ) 1, ( ) 1, ( ) 1, ( ), ( ), ( ) 1, ( y x f y x f y x f y x f y x f y x f y x f X-1 x X+1 1 st derivative 1 st derivative 2 nd derivative

52 2 nd Derivative Laplacian Operator 2 f = [ f ( x, y + 1) + f ( x, y 1) + f ( x + 1, y) + f ( x 1, y)] 4 f ( x, y) Common Variations on Laplacian Operator Expand to include diagonals Positive center coefficients

53 Example 2 nd Derivative using [ ; ; ]

54 Comparison 1 st Derivative Input 2 nd Derivative

55 Comparison of 1 st and 2 nd Derivative 1 st Derivative Generally produces thicker edges Stronger response to gray level steps 2 nd Derivative Better response to fine detail Produces double response at step changes Stronger response to a line than a gray-level ramp

56 Derivative Filters Sometimes call high-pass filters Let the high-frequencies pass through the filter Critical in finding features, such as edges and lines The results may be negative Sign can tell you transition Dark to Light Light to Dark For visualization, you ll need to scale and/or clip so that the gray levels of the result span [0, L-1]

57 Image Sharpening using Laplacian Derivative operator finds changes (detail) Derivative operation doesn t response in static regions (no-detail) So, if we add the derivative to the original Boost detail, i.e. sharpening

58 Enhancement using Laplacian g = f 2 + f This enhancement depends on the kernel used. If center coefficient is negative, then: g = f 2 f

59 High Pass filter Another way to think about the high pass filter: Highpass = Original Lowpass =

60 Comments on Spatial Filters Spatial Convolution We are convolving a function about each (x,y) approximation of filters in the frequency domain (at least for the linear filters.. non-linear is hard to analyze) types Blurring, Smoothing, Sharpening, Derivatives and the common non-linear (Median, Min, Max) Input gray-levels may be different than output levels very common May need to scale your image for visualization Filter coefficients do not have to be integers Results are non-integer use a float image

61 Point/Spatial Summary Most common routines in image processing Point processing only manipulates intensity Does not consider neighborhood information Tends to modify the visual appearance (contrast, etc) Spatial Looks at a neighborhood Tends to modify the visual information (blurring, sharpening) Relationship Computational Photography All papers will assume you have this background Some CP techniques, like High-Dynamic-Range Imaging, are completely related to manipulating intensity values

62 Frequency Domain Processing

63 Idea of Frequency Decomposition Frequencies Frequencies Original function Decomposition to 10 frequencies Result adding up the frequencies Result adding up the frequencies A function can be expressed as a sum of Sine waves with different frequencies.

64 2D Discrete Fourier Transform Converts an image into a set of 2D sinusoidal patterns The DFT returns a set of complex coefficients that control both amplitude and phase of the basis Some examples of sinusoidal basis patterns 64

65 Inversing from Frequency Domain v Inversing is just a matter of summing up the basis weighted by their F(u,v) contribution. u = Result after summing only the first 16 basis* 32 basis 64 basis 128 basis 256 basis 512 basis original * f 4 4 j2π( ux N + vy M ) ( x, y) = F( u, v) e v= 0 u= 0

66 Filtering We can filter the DFT coefficients. This means we throw away or suppress some of the basis in the frequency domain. The filtered image is obtained by the inverse DFT. We often refer to the various filters based on the type of information they allow to pass through: Lowpass filter Low-order basis coefficients are kept Highpass filter High-order basis coefficients are kept Bandpass filter Selected bands of coefficients are kept Also can be considered band reject Etc.. 66

67 Filtering and Image Content Consider image noise: Original Noise 67

68 Typical Filtering Approach From: Digital Image Processing, Gonzalez and Woods. 68

69 Example F(u,v) H(u,v) G = H(u,v) F(u,v) F = fft2(i); H = yourownfilter.m G = H *. F; Note that G(u,v) = H(u,v) F(u,v) is not matrix multiplication. It is a element-wise multiple. f(x,y) I = imload( saturn.tif ); g = ifft2(g); g(x,y) * Examples here have shifted the F,H, and G matrices for visualization. F,G log-magnitude are shown.

70 Equivalence in Spatial Domain (, ) (, ) (, ) (, ) f x y h x y F u v H u v Recall convolution theorem (In spatial domain we call h a point spread function) (In frequency domain we often call H a optical transfer function) Spatial Filtering (, ) = (, ) (, ) G( u, v) = F ( u, v) H ( u, v) g ( x, y) = I G( u, v) g x y f x y h x y The frequency domain filter H, should be inversed to obtain h(x,y): h( x, y) = I 1 { H ( u, v)} Frequency Domain Filtering 1 { } 70

71 Ideal Lowpass Filter From: Digital Image Processing, Gonzalez and Woods 71

72 Example from DIP Book From: Digital Image Processing, Gonzalez and Woods. 72

73 Original Do=5 Do=15 Do=30 Do is the filter radius cut-off. That is, all basis outside Do are thrown away. Note the ringing artifacts in Do=15,30. Do=80 Do=230 73

74 Ringing Why ringing? This is best demonstrated when looking at the inverse of the ideal filter back in the spatial domain. H(u,v) h(x,y) = F -1 (H(u,v)) Imagine the effect of performing spatial convolution with this filter. Probably look like ringing... 74

75 Making Smoother Filters The sharp cut-off of the ideal filter results in a sinc function in the spatial domain which leads to ringing in spatial convolution. Instead, we prefer to use smoother filters that have better properties. Some common ones are: Butterworth and Gaussian 75

76 Butterworth Lowpass Filter (BLPF) This filter does not have a sharp discontinuity Instead it has a smooth transition A Butterworth filter of order n and cutoff frequency locus at a distance D 0 has the form 1 1+ [ D( u, v) / H ( u, v) = 2n D0] where D(u,v) is the distance from the center of the frequency plane. 76

77 1. The BLPF transfer function does not have a sharp discontinuity that sets up a clear cutoff between passed and filtered frequencies. 2. No ringing artifact visible when n = 1. Very little artifact appears when n <= 2 (hardly visible). 3. Ringing does start to appear when n gets larger. From: Digital Image Processing, Gonzalez and Woods. 77

78 BLPF Profile Butterworth low-pass filters in spatial domain of order 1, 2, 5, and 20. Note ringing increasing with filters order. Also, from previous slide we see the filter begins to approach the ideal filter as the order increases. From: Digital Image Processing, Gonzalez and Woods. 78

79 Butterworth Example Filters are as follows: n=2, radii = 5, 15, 30, 80, 230. Note no (or little) visible ringing artifacts with n=2. 79

80 Gaussian Lowpass Filter The Gaussian lowpass filter in the frequency domain is * expressed as: H u + v 2 2σ ( u, v) = e where sigma is the variance and used to control the cut-off. The inverse DFT for this function is: 2 2 h( x, y) = 2πσe 2π 2 σ 2 ( x 2 + y 2 ) Note that the GLPF has a Gaussian form in both the frequency and spatial domain. Variance in the frequency domain is inversely proportional to variance in spatial domain *The u,v in the equation are assumed to be centered of the original of the FT.

81 Gaussian Lowpass Filter (GLPF) Here the Gaussian is expressed in a slightly different form, more similar to the BLPF. Note the D 0 is equal to the variance. ( ) 2, / H ( u, v) = e D u v D

82 GLPF applied Variance set to 5, 15, 30, 80, and 230. No ringing artifacts are present. 82

83 Restoration

84 What is Image Restoration? Image restoration attempts to restore images that have been degraded Identify the degradation process and attempt to reverse it Often distinguished from enhancement because it is undoing some clear degradation process, not just making the image look good for perception Degraded Image Restored

85 Degradation Model + ), ( ), ( ), ( ), ( v u N v u F v u H v u G + = ), ( y x f ), ( y x h Degradation function Noise ), ( y x η ), ( y x g ), ( ), ( ), ( ), ( y x y x f y x h y x g η + = Ideal Image Degraded Image Spatial and Frequency Domain Description 85

86 Image Noise The sources of noise in digital images arise during image acquisition (digitization) and transmission Imaging sensors can be affected by ambient conditions Interference can be added to an image during transmission 86

87 Noise Only Model We can consider a noisy image to be modeled as follows: g( x, y) = f ( x, y) + η( x, y) We have various ways to model noise. Lets assume now, the only problem with the image is noise. Note that noise is additive, not convolution.

88 Noise Models There are many different models for the image noise term η(x, y): Gaussian Most common model - easy Rayleigh Erlang Exponential Uniform Impulse Salt and pepper noise Erlang Gaussian Uniform Rayleigh Exponential Impulse

89 Noise Example The test pattern to the right is ideal for demonstrating the addition of noise The following slides will show the result of adding noise based on various models to this image Image Histogram to go here Histogram

90 Noise Example (cont ) Gaussian Rayleigh Erlang

91 Noise Example (cont ) Histogram to go here Exponential Uniform Impulse

92 Remove Noise Spatial Filtering We can use different spatial filters to remove different kinds of noise S xy is a window about each point size mxn The arithmetic mean filter is a simple approach: fˆ( x, y) = 1 mn ( s, t) g( s, t) S xy 1/ 9 1/ 9 1/ 9 1/ 9 1/ 9 1/ 9 Simple average blur kernel introduced in spatial filtering. 1/ 9 1/ 9 1/ 9

93 Median Filter Median Filter: fˆ( x, y) = median{ g( s, t)} ( s, t) S xy Excellent at noise removal, without the smoothing effects that can occur with other smoothing filters Particularly good when salt and pepper noise is present

94

95 Periodic Noise Typically arises due to electrical or electromagnetic interference Gives rise to regular noise patterns in an image Frequency domain techniques in the Fourier domain are most effective at removing periodic noise

96 Band Reject Filters Removing periodic noise form an image involves removing a particular range of frequencies from that image Band reject filters can be used for this purpose An ideal band reject filter is given as follows: + > + < = 2 ), ( 1 2 ), ( ), ( 1 ), ( W D v D u if W D v D u W D if W D v D u if v u H

97 Band Reject Filters (cont ) The ideal band reject filter is shown below, along with Butterworth and Gaussian versions of the filter Ideal Band Reject Filter Butterworth Band Reject Filter (of order 1) Gaussian Band Reject Filter

98 Band Reject Filter Example Image corrupted by sinusoidal noise Fourier spectrum of corrupted image Butterworth band reject filter Filtered image

99 Image Degradation Degradation function h, and H We think of the degradation function as a convolution. 99

100 Degradation Models Image degradation can occur for many reasons, some typical degradation models are: h( i, j) 1 ai + bj = 0 = 0 otherwise Motion Blur: due to camera panning or subject moving quickly. [a b] is direction of the motion. h( i, j) = Ke h i j = 1 i + j 2σ (, ) L h( i, j) = π R 0 2 L i, j otherwise L i + j R otherwise Atmospheric Blur: long exposure Uniform 2D Blur Out-of-Focus Blur 100

101 Restoration Via Deconvolution Image f Goal: restored image fˆ ), ( ), ( ), ( ), ( y x y x f y x h y x g η + = Observed image g Observation (input) is image g. Our goal is to recover the original image f.

102 Inverse Filtering Assume we know or can estimate the filter h or H ( u, v) = F( u, v) H( u v) F( u, v) G, Inverse filter ˆ = (, v) ( u, v) G u H F -1 { F( u, v)} F -1 ( u, v) H( u, )} { F v F -1 F ( u, v) H( u, v) H ( u, v)

103 Inverse Filter in Practice ( u, v) = F( u, v) H( u v) F( u, v) G, Problems with Inverse Filtering ˆ = (, v) ( u, v) G u H H often has zeros (typically at the high-frequencies)! This makes division ill-posed We can often just ignore 0 values and focus on the low-frequencies where H is defined Uncertainties of H have a significant impact on 103

104 Ringing Some filters just inverse poorly E.g. A box filter completely removes frequencies; inverse filter is therefore ill-posed This can results in what often looks like ringing or interference 104

105 What about noise? ( u, v) = F( u, v) H( u, v) + N( u v) G, ( ) F ( u v) Fˆ u, v =, + (, ) (, ) N u v H u v Noise often is attributed to the device.. Blurred image + noise That means noise happens after PSF or OTF is applied. Inverse filtering can make noise more noticeable. Inverse Filtering 105

106

107 Restoration Summary Image Restoration Broad topic in Image Processing Could be a course in itself Many more procedures not discussed Blind Deconvolution, Lucy-Richardson Algorithm, Regularization,.. Relationship to Computational Photography Researchers will assume you have some basic understanding of restoration Often will give an overview of this topic to many readers Many researchers have been modifying camera optics/exposure to facilitate subsequent restoration. This means they assume some degradation will occur, but want to reduce its effects

108