Corner Detection Harvey Rhody Chester F. Carlson Center for Imaging Science Rochester Institute of Technology rhody@cis.rit.edu April 11, 2006 Abstract Corners and edges are two of the most important geometrical features in image processing. This talk discusses corner detection approaches using SUSAN and compares the results with gradient approaches Harris and KLT. Corner Detection
Corners and Edges Edges are locations where the brightness f(x, y) varies rapidly in a certain direction. Corners are locations where f(x, y) varies rapidly in two (nearly orthogonal) directions. Corner Edge Corner Detection 1
Change Measures The variation in brightness can be characterized by directional derivatives. f x = f x f Mx f y = f y f My where M x and M y are directional masks such as those shown below. M x = 1 0-1 1 0-1 1 0-1 M y = 1 1 1 0 0 0-1 -1-1 We want to use the directional derivatives to find edges and corners and to determine their orientation. Because derivatives are involved, the measures are vulnerable to noise. Corner Detection 2
Change Measures To reduce noise effects it is customary to smooth the directional derivatives. Let w(x, y) be a smoothing filter mask. Form smoothed products of derivatives, denoted by f 2 x = w f 2 x f 2 y = w f 2 y f x f y = w (f x f y ) Define the matrix C = [ ] f 2 x fx f y f x f y f 2 y The matrix C is real and symmetric. Therefore, it is positive-definite and has positive real eigenvalues. Corner Detection 3
Eigenvalues and Eigenvectors The eigenvalues and eigenvectors of a symmetric matrix are [ ] a b M = b c λ 1 = a + c 2 λ 2 = a + c 2 + 1 2 (a c2 + 4b 2 1 2 (a c2 + 4b 2 v 1 = [ 1 2 (a c) + 1 (a c)2 + 4b 2 2, 1] T v 2 = [ 1 2 (a c) 1 (a c)2 + 4b 2 2, 1] T The values and directions can be checked by setting, for example, a b c or a b c. In the both cases, λ 1 λ 2 > 0. However, the dominant eigenvector changes to correspond to the greatest gradient. Corner Detection 4
Corner Detection Original Image KLT Corner Detection Contour Plot for λ 1 with λ 2 extremes marked in red Surface Plot for λ 2 Corner Detection 5
Edge Classification - Monochrome Images We can classify the region near a particular pixel (x, y) as: Nearly uniform if λ 1 and λ 2 are small. Step edge if λ 1 is large and λ 2 is small. The eigenvector v 1 is orthogonal to the edge. A corner if λ 1 and λ 2 are both large. The eigenvectors v 1 and v 2 are orthogonal to the edges at the corner. Some observations: Since λ 1 λ 2 0 the above tests may be simplified. The eigenvalues correspond to edge magnitudes and eigenvectors to directions. Corner Detection 6
Harris Corner Detector The Harris algorithm implements a way compute a parameter based on the eigenvalues that captures cornerness. However, it does not require that the eigenvalues be explicitly calculated. We can make use of the following results from matrix algebra: det C = λ 1 λ 2 trace C = λ 1 + λ 2 Define the Harris detection parameter as H = λ 1 λ 2 α(λ 1 + λ 2 ) 2 = det C + α(trace C) 2 Introduce λ 1 = λ and λ 2 = κλ, 0 κ 1. H = λ 2 (κ α(1 + κ) 2 ) Corner Detection 7
Harris Corner Detector (cont) The parameter α controls the sensitivity of the detector. To have H > 0 requires κ α < (1 + κ) 2 0.25 α plays the role of a sensitivity parameter. Larger α Smaller H less sensitive detector and fewer corners detected. Smaller α Larger H more sensitive detector and more corners detected. For a given α a corner is detected if H > H t, a threshold. Usually H t is a fixed value near zero and α is a variable parameter. H = det C α(trace C) 2 > H t Corner Detection 8
Kanade-Lucas-Tomasi (KLT) Corner Detector Also based on matrix C, but uses explicit calculation of eigenvalue λ 2. The detector has two parameters: a threshold λ t on λ 2 and the edge dimension D of a square window of size D D. KLT Corner Detector Algorithm 1. Compute C at each point (x, y) of the image. 2. For each image point p = (x, y) (a) Find the smallest value of λ 2 in D-neighborhood of p. (b) If that λ 2 > λ t then put p into a list L. 3. Sort L in decreasing order of λ 2 4. Scan the sorted list from top to bottom. For each current point p, delete all lower points in the list in the D-neighborhood of p. Corner Detection 9
KLT Algorithm (cont) The KLT algorithm produces a list of feature points such that: For each point, λ 2 > λ t. The D-neighborhoods of the points do not overlap. Parameter Selection The threshold λ t may be estimated from a histogram of λ 2. Try to select λ t to a valley of the histogram. Window size D is usually between 2 and 10. Select by trial and error. For large D the corners may move away from the actual position and some close neighbor corners may be lost. Corner Detection 10
Comparison of Harris and KLT KLT and Harris are both based on eigenvalues of C KLT tends to be used in America and Harris in Europe. KLT used for motion tracking in KLT Tracker algorithm. Harris provides good repeatability under rotation and changing illumination. Harris often used in image matching and image database retrieval. Both may detect interest points other than corners. Reference: http://ssip2003.info.uvt.ro/lectures/chetverikov/corner_detection.pdf Corner Detection 11
SUSAN Feature Detector SUSAN is derived from the phrase Univalue Segment Assimilating Nucleus Algorithm For each pixel place a mask at the pixel find the pixels that match the brightness of the center pixel compare the area of the USAN to a threshold Corner Detection 12
eus (USAN). He argues that the USAN corresponding to a corner (case (a) in Figure 3) has an USAN area ss than half the total mask area. It is clear from Figure 3 that a local minimum in USAN area will find the t point of the corner. SUSAN Mask Approximation to the circular mask Nucleus Figure 3: The Smith USAN (SUSAN) corner finding mask. The typical SUSAN mask employs an approximately circular region of 37 pixels. actice, the circular mask is approximated using a 5 x 5 pixel square with 3 pixels added on to the center of edge (Figure 3). The intensity of the nucleus is then compared with the intensity of every other pixel within ask using the following comparison function: Corner Detection 13
SUSAN Principle The concept of each image point having associated with it a local area of similar brightness is the basis for the SUSAN principle. The local area or USAN contains much information about the structure of the image. It is effectively region finding on a small scale. From the size, centroid and second moments of the USAN two dimensional features and edges can be detected. This approach to feature detection has many differences to the gradient methods, the most obvious being that no image derivatives are used and that no noise reduction is needed. SUSAN Smallest Univalue Segment Assimilating Nucleus Stephen M. Smith SUSAN site: http://www.fmrib.ox.ac.uk/~steve/susan/ Corner Detection 14
USAN Information The area of an USAN conveys the most important information about the structure of the image in the region around any point in question. The USAN area is maximum when the nucleus lies in a flat region of the image surface falls to half of this maximum very near a straight edge and falls even further when inside a corner This property of the USAN s area is used as the main determinant of the presence of edges and two dimensional features. Corner Detection 15
USAN Information (cont) Surface plot of the USAN as the mask is moved over the test area. Note that the scale is inverted. Flat areas have the largest USAN, edges have an intermediate USAN, and corners have a small USAN. Corner Detection 16
USAN Information (cont) USAN for a small part of a noisy image. The smoothing effect of the mask suppresses the noise, which is on the order of 15 counts out of 256 levels. The edge and corner features are evident as the smallest values on the inverted scale. Corner Detection 17
SUSAN Algorithm The mask is placed over each pixel and the following calculation is made: 1. Let r 0 be the current pixel location and r any other within the mask. Calculate C(r, r 0 ) = e I(r) I(r0 ) 6 t where I is the image gray level and t is a threshold setting. 2. Calculate the sum (USAN) for pixel r 0 : n(r 0 ) = r C(r, r 0 ) 3. Compare n(r 0 ) to a threshold g. For corner detection it is safe to set g = n max /2. Corner Detection 18
Corner Detection with SUSAN Corner Detection 19
Edge Detection with SUSAN Corner Detection 20
Corner Detection with SUSAN House Test Image Corner Detection 21
Corner Detection with SUSAN Points found by SUSAN corner detection with t = 20. Corner Detection 22
Corner Detection with SUSAN Points found by SUSAN corner detection with t = 80. Corner Detection 23
Corner Detection with SUSAN Original Test Image Corner Detection 24
Corner Detection with SUSAN Points found by SUSAN corner detection (t = 20) Corner Detection 25
Comparison of Corner Detector Performance In the paper Assessing the Performance of Corner Detectors for Point Feature Tracking Applications the authors considered four detectors: 1. The Kitchen-Rosenfeld detector (historically one of the first to use a cornerness measure.) 2. The Harris corner detector 3. The Kanade-Lucas-Tomasi corner detector 4. The SUSAN corner detector. The Kitchen-Rosenfeld method uses second and first order derivatives, the Harris and KLT methods use only first order derivatives and SUSAN uses a geometrical criteria in calculating the cornerness value witn no derivatives. Corner Detection 26
Kitchen-Rosenfeld Corner Detector The Kitchen-Rosenfeld algorithm is one of the earliest corner detectors reported in the literature, hence it has been used as a bench mark. This algorithm calculates the cornerness value C as the produce of the local gradient magnitude and the rate of change of gradient direction. The quantity C is given by, C = f xxf 2 y + f yy f 2 x 2f xy f x f y f 2 x + f 2 y where f(x, y) is the image brightness. Corner Detection 27
Harris Corner Detector The Harris cornerness measure used in the test was C = f 2 x + f 2 y f 2 x f 2 y f x f y 2 = trace C det C This is closely related to the Harris cornerness measure H discussed earlier and is often referred to as the Harris detector. It is actually a simplification of a form due to Noble [4]: where ɛ is a small positive value. C = trace C det C + ɛ Corner Detection 28
Performance Requirements Good temporal stability - corners should appear in every frame of a sequence (from the time they are first detected), and should not flicker (turn on and off) between frames. Accurate localization - the calculated image-plane position of a corner, given by the detector, should be as close to the actual position of the corner as possible. Robust with respect to noise Computationally efficient Corner Detection 29
Test Setup The test employed sequences of 30 image with no motion so that the appearance and disappearance of corners in each frame is purely due to image plane noise and illumination conditions. The internal parameters of each detector were adjusted to give its best performance. Four image sequences were used: 1. indoor image sequence of a toy dog is used with only artificial light interference 2. outdoor sequence of a building illuminated with only natural lighting 3. indoor lab sequence with plenty of identifiable corners using direct light sources 4. computer image sequence with many light reflections and curved objects Corner Detection 30
Tests 1. Corner stability result: The corner stability result reveals the number of stable corners identified throughout the sequence. 2. First frame corner matches result: This quantity indicates the number of corners found in the initial frame that appeared in the subsequent frames. 3. Corner displacement result: The corner displacement result reveals the displacement of a corner in the nth frame from its position in the initial frame (assuming that the corner considered appeared in the n-th frame). The above measures were collected for each detector without added noise. In a second experiment the performance was evaluated in the presence of additive gaussian noise of various levels. Corner Detection 31
Measures of Matching Gradient Vector Matcher (GVM): To compare two corners for a possible match, form vectors v and w at each corner from the quantities [f, f x, f y ] T. The GVM measure is the scalar m(v, w) = v w v w Product Moment Coefficient Matcher (PMCM): Let t i and p i be intensity values in two images located at the corners to be tested for a match. The PMCM value is c = n i=1 (t i t)(p i p) n i=1 (t i t) 2 n i=1 (p 1 c 1 i p) 2 c and m are both insensitive to lighting changes and describe the structure of regions around potential corners. Corner Detection 32
Dog Sequence Results (a) Using KLT (b) Using Harris (c) Using Kitchen-Rosenfeld (d) Using SUSAN Figure 5: The best 100 corners extracted from the indoor static dog sequence. (a) KLT corner detector. (b) Harris corner detector. (c) Kitchen-Rosenfeld corner detector. (d) SUSAN corner detector. The best 100 corners extracted from the indoor static dog sequence. (a) KLT corner detector. (b) Harris corner detector. (c) Kitchen-Rosenfeld corner detector. (d) SUSAN corner detector. Corner Detection 33
(c) Using Kitchen-Rosenfeld (d) Using SUSAN Figure 5: The best 100 corners extracted from the indoor static dog sequence. (a) KLT corner detector. (b) Harris corner detector. (c) Kitchen-Rosenfeld corner detector. (d) SUSAN corner detector. Dog Sequence Results (no noise) (a) (b) (c) Using GVM (Thresh=0.009) (d) (e) (f) Using PMCM (Thresh = 0.7) Figure 6: Corner detector performance-test for the 'static dog' sequence (the best 100 corners as seen by each Cornerdetector are performance-test extracted from for the each static frame). dog (a) sequence: Percentage (a) Percentage stable corners stable corners (using (using GVM). GVM). (b) Corner (b) Corner displacement displacement (GVM). (c) Number of first-frame matches (GVM). (d) Percentage stable corners (using PMCM). (e) Corner (GVM). displacement (c) Number of (PMCM). first-frame (f) matches Number (GVM). of first-frame (d) Percentage matches stable (PMCM). corners (using PMCM). (e) Corner displacement (PMCM). (f) Number of first-frame matches (PMCM). Corner Detection 34
Dog Sequence Results (with noise) Dog Sequence (with noise) (a) (b) (c) Using GVM (Thresh=0.004) (d) (e) (f) Using PMCM (Thresh = 0.8) Figure 7: Performance of the corner detectors when applied to the static dog sequence at varied noise levels Performance (noise of variance the corner ranging detectors from when 0 applied 25). (a) to the Percentage static dogstable sequence corners at varied (using noisegvm). levels (noise (b) Corner variancedisplacement ranging from 0 (GVM). (c) Number of first-frame matches (GVM). (d) Percentage stable corners (using PMCM). (e) Corner 25). (a) displacement Percentage stable (PMCM). corners (f)(using Number GVM). of first-frame (b) Cornermatches displacement (PMCM). (GVM). (c) Number of first-frame matches (GVM). (d) Percentage stable corners (using PMCM). (e) Corner displacement (PMCM). (f) Number of first-frame matches (PMCM). Building Sequence Results Corner Detection 35
(noise variance ranging from 0 25). (a) Percentage stable corners (using GVM). (b) Corner displacement (GVM). (c) Number of first-frame matches (GVM). (d) Percentage stable corners (using PMCM). (e) Corner displacement (PMCM). (f) Number of first-frame matches (PMCM). Building Sequence Results (a) Using KLT (b) Using Harris (c) Using Kitchen-Rosenfeld (d) Using SUSAN Figure 8: The best 150 corners extracted from the outdoor static building sequence. (a) Using KLT corner detector. (b) Using Harris corner detector. (c) Using Kitchen-Rosenfeld corner detector. (d) Using SUSAN The best 150 corners extracted from the outdoor static building sequence. (a) Using KLT corner detector. (b) Using Harris corner corner detector. detector. (c) Using Kitchen-Rosenfeld corner detector. (d) Using SUSAN corner detector. 14 Corner Detection 36
Building Sequence Results (no noise) (a) (b) (c) Using GVM (Thresh=0.009) (d) (e) (f) Using PMCM (Thresh = 0.7) Figure 9: Performance-test for the static building sequence (the best 150 corners as seen by each detector are Performance-test extracted from for the each static frame). building (a) sequence Percentage (the best stable 150corners corners(using as seengvm). by each(b) detector Corner aredisplacement extracted from(gvm). each frame). (c) Number of first-frame matches (GVM). (d) Percentage stable corners (using PMCM). (e) Corner displacement (a) Percentage (PMCM). stable (f) Number corners of (using first-frame GVM). matches (b) Corner (PMCM). displacement (GVM). (c) Number of first-frame matches (GVM). (d) Percentage stable corners (using PMCM). (e) Corner displacement (PMCM). (f) Number of first-frame matches (PMCM). Building Sequence Results (with noise) Corner Detection 37
extracted from each frame). (a) Percentage stable corners (using GVM). (b) Corner displacement (GVM). (c) Number of first-frame matches (GVM). (d) Percentage stable corners (using PMCM). (e) Corner displacement (PMCM). (f) Number of first-frame matches (PMCM). Building Sequence Results (with noise) Building (with noise) (a) (b) (c) Using GVM (Thresh=0.004) (d) (e) (f) Using PMCM (Thresh = 0.8) Figure 10: Performance for the static building sequence at varied noise levels (noise variance ranging from 0 25). (a) Percentage stable corners using GVM. (b) Corner displacement using the GVM. (c) Number of first Performance for the static building sequence (noise variance ranging from 0 25). (a) Percentage stable corners using GVM. (b) frame corner matches using GVM. (d) Percentage stable corners using PMCM matcher. (e) Corner Corner displacement using thepmcm. GVM. (c) (f) Number of of first first frame corner matches matches using using GVM. PMCM. (d) Percentage stable corners using PMCM matcher. (e) Corner displacement using the PMCM. (f) Number of first frame corner matches using PMCM. 15 Corner Detection 38
Lab Sequence Results (a) Using KLT (b) Using Harris (c) Using Kitchen-Rosenfeld (b) Using SUSAN Figure 11: The best 200 corners extracted from the static lab sequence. (a) Using KLT corner detector. (b) Using Harris corner detector. (c) Using Kitchen-Rosenfeld corner detector. (d) Using SUSAN corner detector. The best 200 corners extracted from the static lab sequence. (a) Using KLT corner detector. (b) Using Harris corner detector. (c) Using Kitchen-Rosenfeld corner detector. (d) Using SUSAN corner detector. Corner Detection 39
Figure 11: The best 200 corners extracted from the static lab sequence. (a) Using KLT corner detector. (b) Using Harris corner detector. (c) Using Kitchen-Rosenfeld corner detector. (d) Using SUSAN corner detector. Lab Sequence Results (no noise) (a) (b) (c) Using GVM (Thresh=0.004) (d) (e) (f) Using PMCM (Thresh = 0.8) Figure 12: Corner detector performance-test for the 'static lab' sequence (the best 200 corners as seen by each Corner detector are performance-test extracted from foreach the static frame). lab (a) sequence. Percentage (a) Percentage stable corners stable corners (using (using GVM). GVM). (b) Corner (b) Corner displacement (GVM). (c) Number of first-frame matches (GVM). (d) Percentage stable corners (using PMCM). (e) Corner (GVM). (c) Number of first-frame matches (GVM). (d) Percentage stable corners (using PMCM). (e) Corner displacement (PMCM). displacement (PMCM). (f) Number of first-frame matches (PMCM). (f) Number of first-frame matches (PMCM). Corner Detection 40
Computer Sequence Sequence Results (a) Using KLT (b) Using Harris (c) Using Kitchen-Rosenfeld (d) Using SUSAN Figure 13: The best 250 corners extracted from the static computer sequence. (a) Using KLT corner detector. The best (b) 250 Using corners Harris extracted corner from detector. the static (c) Using computer Kitchen-Rosenfeld sequence. (a) Using corner KLTdetector. corner detector. (d) Using (b) SUSAN Using Harris corner corner detector. detector. (c) Using Kitchen-Rosenfeld corner detector. (d) Using SUSAN corner detector. Corner Detection 41
(c) Using Kitchen-Rosenfeld (d) Using SUSAN Figure 13: The best 250 corners extracted from the static computer sequence. (a) Using KLT corner detector. (b) Using Harris corner detector. (c) Using Kitchen-Rosenfeld corner detector. (d) Using SUSAN corner detector. Computer Sequence Results (no noise) (a) (b) (c) Using GVM (Thresh=0.004) (d) (e) (f) Using PMCM (Thresh = 0.8) Figure 14: Corner detector performance-test for the 'static computer' sequence (the best 250 corners as seen by Corner each detector detector performance-test are extracted for the from static each computer frame). sequence. (a) Percentage (a) Percentage stable corners stable corners (using (using GVM). GVM). (b) (b) Corner Corner displacement (GVM). (c) Number of first-frame matches (GVM). (d) Percentage stable corners (using PMCM). displacement (e) Corner (GVM). displacement (c) Number (PMCM). of first-frame (f) Number matches of first-frame (GVM). (d) matches Percentage (PMCM). stable corners (using PMCM). (e) Corner displacement (PMCM). (f) Number of first-frame matches (PMCM). Corner Detection 42
References 1. An Empirical Study of Corner Detection to Extract Buildings from VHR Satellite Images, L. Martinez-Fonte, S. Gautama and W. Philips, IEEE-ProRisc, 25-26 November 2004, Veldhoven, The Netherlands. Proceedings of ProRisc 2004, pp. 288 293. 2. P. Tissainayagam and D. Suter. Assessing the performance of corner detectors for point feature tracking applications. Image and Vision Computing, 22(8):663-679, August 2004. 3. L. Kitchen and A. Rosenfeld, Gray level corner detection, Pattern Recognition Letters, pp.95-102, 1982 4. A. Noble, Finding Corners, Image and Vision Computing Journal 6(2): 121-128, 1988. Corner Detection 43