Local Image preprocessing (cont d)

Local Image preprocessing (cont d) 1

Outline - Edge detectors - Corner detectors - Reading: textbook 5.3.1-5.3.5 and 5.3.10 2

What are edges? Edges correspond to relevant features in the image. An edge corresponds to a change in intensity, surface orientation, shadow etc. Object recognition and detection use often times edge information (in both CV and biological vision) 3

Why study edges? Collecting information about edges will help us to identify contours in the image and to retrieve regions enclosed by those contours. An edge image represents a higher level of abstraction (i.e. less information to process) Edges are features invariant to absolute illumination (as opposed to colour information) 4

Linear filters for edge detection We need linear filters for: Edge enhancement image sharpening Removal of false edges (noise generated) Edge localization Needs other operations (such as thresholding) 5

Types of 1D edge profiles From Forsyth and Ponce 6

Edge representation From Forsyth and Ponce 7

Edge enhancement Computing the image gradient : vector composed of first-order partial derivatives The gradient magnitude gives the amount of the difference between pixels in the neighborhood (the strength of the edge). The gradient orientation gives the direction of the greatest change, which presumably is the direction across the edge (the edge normal). Derivatives are linear and shift invariant, thus the gradient can be computed with convolution 8

First order differences of an image Estimates of derivatives produced by 1 st order differences 9

Convolution kernels: Roberts Idea: Approximate the gradient using finite differences Forward differences - The Roberts kernels correspond to derivatives with respect to the two diagonal directions Pros: Only four pixels are needed for the gradient computation Cons: Masks are too small to obtain reliable results in the presence of noise 10

Convolution kernels: Prewitt Based on idea of central difference: very sensitive to noise Averaging for reducing noise sensitivity: In y when computing In x when computing 11

Convolution kernels: Sobel Also rely on central differences, but give greater weight to central pixels when averaging: 12

From gradient magnitude to edges The gradient magnitude gives a measure at every pixel of the edginess of each pixel: We still have to find and connect the best edge points: Local maxima of gradient magnitude Connected maxima (ridges) Maximum-magnitude contours/paths Adapted from Brian Morse, http://morse.cs.byu.edu/450/home/index.php 13

From gradient magnitude to edges The gradient magnitude gives a measure at every pixel of the edginess of each pixel: We still have to find and connect the best edge points: Local maxima of gradient magnitude Connected maxima (ridges) Maximum-magnitude contours/paths Adapted from Brian Morse, http://morse.cs.byu.edu/450/home/index.php 14

From Tomas Svoboda: Image preprocessing in the spatial domain 15

Marr-Hildreth Edge Detection approximate finding maxima of gradient magnitude (edges) by finding pixels where the Laplacian is zero valued. Problem: we cannot always find discrete pixels where the Laplacian is exactly zero. Zero-crossings instead 16

Combining first and second order derivatives Laplacian zero crossings: Problem: Tells us the gradient magnitude is at a maximum, not how strong it is lots of spurious edges. Idea: Combine two measures From Brian Morse, http://morse.cs.byu.edu/450/home/index.php 17

Edge detection and noise Partial derivatives are very sensitive to noise image noise results in pixels that look very different from their neighbours Solution: Gaussian smoothing before filtering Response of finite differences to noise σ=0.03 σ=0.09 18

19

Original image Gradient magnitudes using the derivatives of a Gaussian with σ=1 pixel σ=2 pixels Major issues: 1) The gradient magnitude at different values of σ is different; which one should we choose? 2) Large values of the gradient magnitude form thick trails; how do we extract 1 pixel-wide boundaries? 20

Laplacian of Gaussian To include a smoothing Gaussian filter, combine the Laplacian and Gaussian functions to obtain a single equation: A discrete kernel of the LoG with σ=1.4 21

Laplacian of Gaussian - example From Efford 22

Limits of LoG Boundaries of sharp corners are inaccurate; Trihedral vertices are not detected loops. Problem with LOG: adds the principal curvatures together; does not determine a gradient of maximum amplitude in any one direction The Canny Edge Detector defines edges as zero-crossings of second derivatives in the direction of greatest first derivative. 23

The Canny edge detector Also known as the optimal edge detector Published in IEEE Trans. on Pattern Analysis and Machine Intelligence, 1986 Intended to enhance the already existent edge detectors Formulated a list of criteria to enhance the current methods of edge detection 24

Canny edge detector: criteria detection: edges occurring in images should not be missed; there should be no responses to non-edges. edge points have to be well localized. The distance between the edge pixels as found by the detector and the actual edge is to be minimal. Only one response at a single edge is allowed. Eliminates multiple detections of the same edge. Note: There is a trade-off between good detection and good localization. Why? 25

Canny edge detector: main steps 1. Filter out noise in the original image before trying to locate and detect any edges. 2. Compute the image gradient to highlight regions with high spatial derivatives. 3. Non-maximum suppression 4. Hysteresis 26

Canny edge detector: Step 1+2 We can implement both steps by filtering with a Difference of Gaussian (DoG) filter or: Step 1: convolution with a Gaussian Step 2: computation of spatial derivatives with Sobel masks. Approximate edge strength with: G = Gx + Gy 27

Canny edge detector: Non maxima suppression The gradient magnitude tends to be large along thick trails in an image. - To replace this thick trail with a representative thin curve, we have to find a cut direction and to extract the intensity maxima along this direction. - The gradient direction is a good estimate of the cut direction 28

Canny edge detector: non maximum suppression Nonmaximum suppression obtains points where the gradient magnitude is at a maximum along the direction of the gradient. - The gradient direction (forward and backward) does not pass through any pixels on the grid. - Linear interpolation for computing the values of gradient magnitude at p and r (considering the pixels to the left and right of p and r) - if G(q)>max(G int (p), G int (r)) then q is retained as an edge point; else q is suppressed. 29

Canny edge detector: hysteresis In spite of the smoothing step, the non maxima suppressed magnitude image will contain many false edge fragments caused by noise and fine texture. Solution: thresholding One threshold is not enough, since the gradient magnitude is variable Thresholding with hysteresis: - retain all edge candidates with gradient magnitude>t H - Edge following by linking edge candidates with gradient magnitude>t L 30

31

Strong edges reinforce weak edges 32

Matlab implementation of Canny BW = edge(i,'canny') specifies the Canny method. BW = edge(i,'canny',thresh) specifies sensitivity thresholds for the Canny method. thresh is a two-element vector in which the first element is the low threshold, and the second element is the high threshold. If you specify a scalar for thresh, this value is used for the high threshold and 0.4*thresh is used for the low threshold. If you do not specify thresh, or if thresh is empty ([]), edge chooses low and high values automatically. BW = edge(i,'canny',thresh,sigma) specifies the Canny method, using sigma as the standard deviation of the Gaussian filter. The default sigma is 1; the size of the filter is chosen automatically, based on sigma. [BW,thresh] = edge(i,'canny',...) returns the threshold values as a two-element vector. 33

Edge detectors: examples Edge detectors by gradient operators (Prewitt, Roberts, Sobel) 34

Edge detectors: examples (2) Edge detection with DOG and LOG 35

Edge detectors: examples Edge detection with Canny: σ=1,2,3 τ 1 =0.3 τ 2 =0.7 36

What to remember Edge = Rapid intensity (color) change Edge information is one of the most important in CV and Human Vision Three steps in edge detection: Noise reduction (image smoothing) Edge enhancement Edge localisation Three types were presented: Based on first-order derivative Roberts, Prewitt and Sobel Based on second-order derivative Laplacian of Gaussian Based on linking edges Canny 37

Corner detection Corners are important features in the image Useful for matching two images of the same static scene acquired from different viewpoints (stereo vision, building mosaics) Useful for object recognition as well 38

The Moravec corner detector Developed by Hans Moravec in 1977 in his research about autonomous navigation and obstacle avoidance http://www.frc.ri.cmu.edu/~hpm/project.archive/ robot.papers/1977/aips.txt - defined the concept of "points of interest - these interest points could be used to find matching regions in consecutive image frames - interest points=points where there is a large intensity variation in every direction Slides about Moravec detector use materials from - http://www.cim.mcgill.ca/~dparks/cornerdetector/moravec.htm 39

The Moravec corner detector: measuring the intensity variation 40

The Moravec corner detector: why does it detect corners? 41

42

Cornerness map 43

Moravec-results 44

Moravec-limitations Anisotropic response; large response to diagonal edges 45

The Harris corner detector Improves upon Moravec Isotropic More computationally intensive 46

Ideal vertical edge No Change Change 47

Ideal horizontal edge Change No Change 48

Harris corner detector C.Harris, M.Stephens. A Combined Corner and Edge Detector. 1988 49

The Basic Idea We should easily recognize the point by looking through a small window Shifting a window in any direction should change significantly the intensity distribution inside the window From Darya Frolova, Denis Simakov The Weizmann Institute of Science Lecture notes on Matching with Invariant features 50

Harris Detector: Basic Idea flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions From Darya Frolova, Denis Simakov The Weizmann Institute of Science Lecture notes on Matching with Invariant features 51

Harris Detector: Mathematics Change of intensity for the shift [Δx, Δy]: S(x, y) = (x i,y i ) W ( ) f ( x i Δx, y i Δy) [ ] f x i, y i 2 Intensity Shifted intensity Window function w(x,y) = or 1 in window, 0 outside Gaussian 52

Harris Detector: Mathematics We will show that S(x, y) = [ Δx,Δy]A W (x, y) Δx Δy Where A captures the intensity structure of the local neighborhood. Its main modes of variation are reflected in its eigenvalues λ 1 and λ 2 53

Harris Detector: Mathematics Classification of image points using eigenvalues of A: λ 2 Edge λ 2 >> λ 1 Corner λ 1 and λ 2 are large, λ 1 ~ λ 2 ; E increases in all directions λ 1 and λ 2 are small; E is almost constant in all directions Flat region Edge λ 1 >> λ 2 From Darya Frolova & Denis Simakov λ 1 54

Harris Detector: Mathematics (cont d) Measure of corner response: R(A) = det(a) k trace 2 (A) (k tunable empirical constant, k = 0.04-0.15) Det(A)=λ 1 λ 2 Trace(A)=λ 1 + λ 2 But: we do not need to compute λ 1 and λ 2 for the computation of R. 55

Harris Detector: Mathematics R depends only on eigenvalues of M λ 2 Edge R < 0 Corner R is large for a corner R is negative with large magnitude for an edge R > 0 R is small for a flat region Flat R small Edge R < 0 From Darya Frolova, Denis Simakov Lecture notes on Matching with Invariant features λ 1 56

Harris Detector: the algorithm Filter the image with a Gaussian Compute the horizontal and vertical gradient images with Sobel masks or a double 1D convolution Define the neighborhood W (preferably gaussian, thus isotropic) Find points with large corner response function R(A) (R > threshold) Take the points of local maxima of R 57

Harris Detector: Workflow Compute corner response R 58

Harris Detector: Workflow Find points with large corner response: R>threshold 59

Harris Detector: Workflow Take only the points of local maxima of R 60

Harris Detector: Workflow 61

Harris detector - examples 62

Harris Detector: Some Properties Rotation invariance Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation From Darya Frolova, Denis Simakov Lecture notes on Matching with Invariant features 63

Harris Detector: Some Properties Partial invariance to affine intensity change Only derivatives are used => invariance to intensity shift I I + b Intensity scale: I a I R threshold R x (image coordinate) x (image coordinate) 64

Harris Detector: Some Properties But: non-invariant to image scale! All points will be classified as edges Corner! From Darya Frolova, Denis Simakov The Weizmann Institute of Science Lecture notes on Matching with Invariant features 65

Invariance to scale: SIFT D. Lowe, 2004: Distinctive image features from scale-invariant keypoints Paper is mandatory reading for ELEC 536; optional for CENG 421. 66