Filters Advanced and Special Topics: Filters Dr. Edmund Lam Department of Electrical and Electronic Engineering The University of Hong Kong ELEC4245: Digital Image Processing (Second Semester, 2016 17) http://www.eee.hku.hk/ elec4245 We have studied various kinds of filters: Linear averaging filters (e.g., mean), differencing filters (e.g., Sobel) Nonlinear filters (e.g., median) Adaptive filters Spectral domain view (multiplication in the frequency domain) Filtering in image restoration (e.g., inverse filter, pseudoinverse filter, Wiener filter) E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 1 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 2 / 48 Filters Further generalizations: Filters with values that depend on the underlying image content bilateral filter Filters that take into account information outside the immediate neighborhood nonlocal means filter Filters that form a sequence of successive influence to the neighborhood iterative filters (including anisotropic diffusion) Filters that operate on digital logic rather than arithmetic morphological filters 1 2 Non-local means filter 3 4 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 3 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 4 / 48
Mean and median filters Alpha-trimmed mean filter Use an N N filter, aimed at reducing noise in an image. (Let K = N 2, and that K is odd.) (Simple) mean filter: Average all K numbers within a window. Median filter: Order the K numbers within a window: x (1) x (2)... x (K) where x (i) is called the order statistic of the set of numbers {x 1, x 2,..., x K }. Median filter returns the value at the center of the order: x ((K+1)/2). Is it possible to combine averaging and ordering? The alpha-trimmed mean filter works as follows: Define a parameter 0 α 0.5. Let a denote the greatest integer less than or equal to a (also called the floor function). Reject: x (1),..., x ( αk ) and x (K αk +1),..., x (K) and average the remaining K 2 αk values. Example: K = 9, α = 0.3. Thus, αk = 2, and we reject x (1), x (2), x (8), and x (9), while averaging the remaining 5 values. α = 0: nothing gets rejected = mean filter α = 0.5: everything except the middle is rejected = median filter E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 5 / 48 Further extension: Instead of rejecting a fixed percentage given by α, a soft reject of the pixels whose intensity values differ too much from the central pixel value. This gives the (popular and powerful) bilateral filter. Wide applications in computer vision and computational photography. Tradeoff: Filter weight changes with every window, so often slow implementation. E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 6 / 48 For each pixel at (x, y), assume the set of {(i, j)} are within the window: i,j w(x, y, i, j)i in (i, j) I out (x, y) = (1) i,j w(x, y, i, j) If w(x, y, i, j) = 1, we get back the mean filter. If w(x, y, i, j) depends only on (x i) and (y j), we get other linear filters with different weights. In bilateral filtering, w(x, y, i, j) also depends on I in (i, j). C. Tomasi and R. Manduchi, ing for gray and color images, International Conference on Computer Vision 839 846 (1998). E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 7 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 8 / 48
has two components: 1 Domain kernel (adjustable parameter σd ): 2 + (y j)2 (x i) d(x, y, i, j) = exp 2 2σd 2 Range kernel (adjustable parameter σr ): Fig 1: noisy step edge. Assume (x, y) is at the high value. 2 I (i, j) Iin (x, y) in r(x, y, i, j) = exp 2 2σ r Fig 2: domain kernel (Gaussian) Fig 3: range kernel (small for big intensity difference) Fig 4: w(x, y, i, j) = d(x, y, i, j)r(x, y, i, j) Fig 5: after bilateral filtering Combining the effects of the two, in bilateral filtering, (2) w(x, y, i, j) = d(x, y, i, j)r(x, y, i, j) E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 9 / 48 F. Durand and R. Szeliski, Fast bilateral filtering for the display of high-dynamic-range images, ACM Trans. on Graphics, 21, 257 266 (2002). E. Lam (The University of Hong Kong) Non-local means filter ELEC4245 Jan Apr, 2017 10 / 48 Non-local means filter Non-local means 1 2 For image denoising / filtering: Non-local means filter Local means: (traditional filtering) Compute the mean value of a local neighborhood of an input image to generate an output pixel 3 4 Non-local means: How about taking the mean value of places that are not in the neighborhood? Fundamental assumption: An image contains many similar patches We can reduce noise if we can average across these similar patches, which may not necessarily be nearby A. Buades, B. Coll, and J.M. Morel, A non-local algorithm for image denoising, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2, 60 65 (2005). E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 11 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 12 / 48
Non-local means filter Non-local means filter Non-local means Non-local means Implementation: where I out (x, y) = 1 C(x, y) I in (i, j)w(x, y, i, j) (3) C(x, y) is the normalization, C(x, y) = i,j w(x, y, i, j) w(x, y, i, j) is a weighting function, designed to determine how closely related is the image at (i, j) and at (x, y). Note the similarity with the bilateral filter formula! Here, however, (i, j) can be quite far from (x, y) i,j The weighting function w(x, y, i, j) can take various forms, e.g., Gaussian weighting. } B(i, j) B(x, y) 2 w(x, y, i, j) = exp {, B(i, j) is the local mean value surrounding (i, j), e.g., a 3 3 window. h is a filtering parameter, i.e., the standard deviation. Check out http://www.ipol.im/pub/art/2011/bcm_nlm/ for an online demo! h 2 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 13 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 14 / 48 Diffusion 1 Normally we think of filtering as signal influenced by the neighbors. But we can equally well think of it as diffusion to the neighbors. 2 Non-local means filter image filter 3 4 diffusion E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 15 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 16 / 48
Diffusion Diffusion Isotropic diffusion: uniform diffusion in all directions. Example: Gaussian filter. Idea: We start with an original image f 0 (x, y). We have a family of derived images: f (x, y, t) = f 0 (x, y) G(x, y; t) (4) where G(x, y; t) is a Gaussian kernel with variance t, which grows with time. Initial condition is f (x, y, 0) = f 0 (x, y). We often discretize in actual implementation, so f (n) (x, y) = f (0) (x, y) G (n) (x, y), (5) Often, the iteration is based on the previous step: where f (n+1) (x, y) = f (n) (x, y) + λ [ R ( f (n) (x, y) )], (6) R( ) is a (diffusion) function on the nth iteration of f (x, y) λ is a weight suggesting how much we want to add (or subtract, if R( ) is negative) to the nth iteration to generate the (n + 1)th iteration where n denotes the step or iteration number, discretizing the time t. E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 17 / 48 Diffusion Anisotropic diffusion: rate of diffusion depends on local image content. This is also called Perona-Malik diffusion, who proposed two functions for the diffusion coefficient: and g( f ) = exp g( f ) = ( f K 1 + ) 2 (7) 1 ( ) 2 (8) f f is the gradient of image f (x, y), and K is a parameter. The gradient is large across an edge. R ( f (x, y) ) = g ( f ) f. K E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 18 / 48 Iteration Iterative approach can also be applied for more general filters. Advantage: can tolerate inaccurate parameters. Can let user (human) decide when to stop. Algorithm can change from one iteration to another, which may equate rather complex algorithm if done with one step. Disadvantage: may involve more computation. P. Perona and J. Malik, Scale-space and edge detection using anisotropic diffusion, IEEE Trans. Pattern Analysis and Machine Intelligence, 12(7), 629 639 (1990). E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 19 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 20 / 48
Iteration There are many possible iterative schemes. Here is a (classic) example: Consider g(x, y) = f (x, y) h(x, y) + n(x, y). We can treat (g f h) as the residual. If, at each iteration, we estimate f and denote it as f (n), then if f (n) is too big, ( g f (n) h ) would be negative ; if f (n) is too small, ( g f (n) h ) would be positive ; if f (n) is accurate, ( g f (n) h ) would be closest to zero. So, it makes sense to estimate f (n) by f (n+1) (x, y) = f (n) (x, y) + β [ g(x, y) f (n) (x, y) h(x, y) ]. (9) Iteration There is an alternative interpretation to the previous equation. If we rearrange the terms, f (n+1) (x, y) = f (n) (x, y) + β [ g(x, y) f (n) (x, y) h(x, y) ] (10) = βg(x, y) + [ δ(x, y) βh(x, y) ] f (n) (x, y). The current estimate is therefore a weighted sum of the observed image and the last estimate (being filtered). (for more rigorous mathematical derivation, you have to show if this equation converges, whether it converges to the desired solution, and what the convergence rate is, etc. ) This works only for a limited range of (small) β. (Proof omitted) E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 21 / 48 Iteration E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 22 / 48 Iteration Equivalent forms in Fourier domain: F (n+1) ( f x, f y ) = F (n) ( f x, f y ) + β [ G( f x, f y ) F (n) ( f x, f y )H( f x, f y ) ] = βg( f x, f y ) + [ 1 βh( f x, f y ) ] F (n) ( f x, f y ). (11) Equivalent interpretation: We pick the low frequency from G( f x, f y ), and the high frequency from the current estimate (since 1 βh( f x, f y ) is a high pass filter). All iterative solutions should state: 1 Initial point: you can take f (1) (x, y) = g(x, y). 2 Termination point: when the image looks good enough! (or quantitative criterion, such as when f (n+1) (x, y) f (n) (x, y) < ɛ). What happens for n? The parameter β controls the rate with which we systematically add the high frequency component. E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 23 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 24 / 48
Iteration In steady state, we should expect the next estimate to be not so different from the current one. (Mathematically, convergence of a Cauchy sequence.) Therefore, F (n+1) ( f x, f y ) F (n) ( f x, f y ). Put in Eq. (11), F (n) ( f x, f y ) F (n) ( f x, f y ) + β [ G( f x, f y ) F (n) ( f x, f y )H( f x, f y ) ] G( f x, f y ) F (n) ( f x, f y )H( f x, f y ) ( ) F (n) 1 G( f x, f y ). H( f x, f y ) 1 2 Non-local means filter 3 4 In other words, we begin with the noisy, blurred observation, and the steady-state of the iterative restoration is the inverse-filtered image! E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 25 / 48 Mathematical morphology We limit ourselves to binary images, which are informative on the shape of the image content. Morph = form Morphology = the study of form and structure Mathematical morphology = a tool for extracting image components to describe region shape and structure The language of mathematical morphology is set theory (we ll have a quick review here) Some filtering operations have the same purpose as these morphological operations, but for binary images, the reasons above usually justify their use All functions are built by logics: that s how our computers are built! E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 26 / 48 Review of set theory We have a set A (usually denoted in capital). We also have an element called a (usually in lower case). If a is an element of A, we denote a A If a is not an element of A, we denote a A If every element in A is also an element in B, then A is a subset of B, i.e. A B E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 27 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 28 / 48
Review of set theory Morphological operations Morphological operators (Ω denotes the whole space): Intersection: A B = { p p A and p B } Union: A B = { p p A or p B } Complement: A c = { p p Ω and p A } Difference: A B = A B c Reflection: Â = { p p = a for a A } Translation by a point z: A z = { p p = a + z for a A } Conceptually, we maneuver the shape of the image for some desired purposes. In binary images, the shape is directly evident. They are usually non-linear, local operations. They can be effectively implemented = fast operations. We have a choice of using 4-neighbor or 8-neighbor in the following definitions. Operations often involve duality. E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 29 / 48 Expanding E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 30 / 48 Shrinking Expanding: Change a pixel from 0 to 1 if any neighbor is 1. Example: (4-connectivity) Shrinking: Change a pixel from 1 to 0 if any neighbor is 0. Example: (4-connectivity) E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 31 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 32 / 48
Use of expanding and shrinking When used successively, Expanding: an image blows up to fill the entire image. Shrinking: an image is eaten up until nothing is left behind. Dilation Expansion is, in a sense, an omni-directional growth. Is there a way to control its extent and direction? Key: a structuring element. The operation is called dilation and is represented mathematically by the symbol. = structuring element E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 33 / 48 Dilation Mathematically, dilation computes the set of the vector sum of pixels from two sets. A B = c Ω }{{} c = a + b for some a A and b B (12) whole space = A b (13) E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 34 / 48 Erosion Shrinking is, in a sense, an omni-directional death. Is there a way to control its extent and direction? Key: a structuring element. The operation is called erosion and is represented mathematically by the symbol. OR operation The above two equations imply different implementations. is similar to + with the following two properties: A B = B A (commutative law) (A B) C = A (B C) (associative law) = structuring element E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 35 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 36 / 48
Erosion Duality of dilation and erosion Mathematically, erosion computes the set of the points such that a set B, when translated by one such point, is contained in another set A. A B = c AND operation Ω }{{} whole space c + b A for every b B A pattern matching interpretation: identify locations where the structuring element is found in an image. (14) Dilation and erosion are called duals because of the following identities: (remember ˆB is the reflection of set B) (A B) c = A c ˆB (15) (A B) c = A c ˆB (16) Interpretation: Dilation of the foreground (1 s) is equivalent to the erosion of the (reflection of) the background (0 s). E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 37 / 48 Opening and closing E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 38 / 48 Opening and closing Dilation and erosion form the elementary building block of all morphological operations. We will now see some more sophisticated operations involving dilations and erosions. These operations will have their graphical effects and mathematical representations. Although and are similar to + and, they are not identical. Be very careful in their mathematical manipulations! Opening: Erosion followed by dilation A B = (A B) B (17) Closing: Dilation followed by erosion A B = (A B) B (18) E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 39 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 40 / 48
Opening example 0 1 0 Open with B = 1 1 1 0 1 0 Closing example Close with B = [ 1 1 1 ] 1 1 1 1 1 It smoothes the contour of an object, breaks narrow bridges, and eliminates thin protrusions. It fuses narrow breaks and long thin gulfs, eliminates small holes, and fills gaps in the contour. E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 41 / 48 Opening and closing E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 42 / 48 Hit-or-Miss transform Opening and closing are duals: Repeated operations: (A B) c = A c ˆB (19) (A B) c = A c ˆB (20) (A B) B = A B (21) (A B) B = A B (22) Intention: shape detection Erosion can help us find the pattern. But any shapes that are supersets of our pattern are also found. A more precise specification: we should not only match the pattern but also the background. So, we define two patterns: one for the image, and one for the boundary of the image (background) This shows that multiple operations of opening and closing have no additional effects. E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 43 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 44 / 48
Hit-or-Miss transform Hit-or-Miss transform In other words, we want Definition with B = (B 1, B 2 ): foreground to match pattern B 1 background to match a pattern B 2 Note that B 1 B 2 = φ. Example: Example: A B = (A B 1 ) } {{ } foreground (A c B 2 ). (23) } {{ } background B 1 B 2 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 45 / 48 Other morphological operations E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 46 / 48 Conclusions Further combinations can lead to additional morphological operations. Examples: thinning, thickening, skeletonization, and pruning... Thickening: A B = A (A B) Thinning: A B = A (A B) c Boundary extraction: β(a) = A (A B) More advanced filters are continuously being developed. There are also many specialized filters, e.g., creating a specific photo effect. E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 47 / 48 E. Lam (The University of Hong Kong) ELEC4245 Jan Apr, 2017 48 / 48