Robotica Anno accademico 6/7 Davide Migliore migliore@elet.polimi.it
Today What is a feature? Some useful information The world of features: Detectors Edges detection Corners/Points detection Descriptors?!?!? 7//6 Corso di Robotica 6/7
What is a feature? 7//6 Corso di Robotica 6/7
Images An image is a matrix of pixels Resolution Digital cameras: 6 X at a minimum Video cameras: ~64 X 48 Grayscale: generally 8 bits per pixel Intensities in range [ 55] RGB color: 8-bit color planes 7//6 Corso di Robotica 6/7 4
Image conversion RGB Grayscale: Mean color value, or weight by perceptual importance Grayscale Binary: Choose threshold based on histogram of image intensities 7//6 Corso di Robotica 6/7 5
Color representation RGB, HSV (hue, saturation, value), YUV, etc. Luminance: Perceived intensity Chrominance: Perceived color HS(V), (Y)UV, etc. Normalized RGB removes some illumination dependence: 7//6 Corso di Robotica 6/7 6
Binary Operations Dilation, erosion Connected components Dilation: All s next to a (Enlarge foreground) Erosion: All s next to a (Enlarge background) Uniquely label each n-connected region in binary image 4- and 8-connectedness Moments: Region statistics Zeroth-order: Size First-order: Position (centroid) Second-order: Orientation 7//6 Corso di Robotica 6/7 7
Bilinear Interpolation Idea: Blend four pixel values surrounding source, weighted by nearness Vertical blend 7//6 Horizontal blend Corso di Robotica 6/7 8
Image Comparison: SSD Given a template image and an image, how to quantify the similarity between them for a given alignment? Sum of squared differences (SSD) 7//6 Corso di Robotica 6/7 9
Cross-Correlation for Template Matching Note that SSD formula can be written: First two terms fixed last term measures mismatch the cross-correlation: In practice, normalize by image magnitude when shifting template to search for matches 7//6 Corso di Robotica 6/7
Filtering Idea: Analyze neighborhood around some point in image with filter function ; put result in new image at corresponding location System properties Shift invariance: Same inputs give same outputs, regardless of location Superposition: Output on sum of images = Sum of outputs on separate images Scaling: Output on scaled image = Scaled output on image Linear shift invariance Convolution 7//6 Corso di Robotica 6/7
Discrete Filtering - - Linear filter: Weighted sum of pixels over - rectangular neighborhood kernel defines weights Think of kernel as template being matched by correlation Convolution: Correlation with kernel rotated 8 Dealing with image edges Zero-padding Border replication 7//6 Corso di Robotica 6/7
Convolution Definition: Shorter notation: Properties Commutative Associative Fourier theorem: Convolution in spatial domain = Multiplication in frequency domain 7//6 Corso di Robotica 6/7
Filtering Example : - - - Rotate - - - 7//6 Corso di Robotica 6/7 4
- - - 7//6 Step - 4 - - 5 Corso di Robotica 6/7 5
- Step - - 7//6 4 - - - 5 Corso di Robotica 6/7 4 6
Step - - - - 4 5 4 4 - - 7//6 Corso di Robotica 6/7 7
Step 4 - - - 7//6-6 - - 5 Corso di Robotica 6/7 4 4-8
- - - 7//6 Step 5 5-4 9 - - Corso di Robotica 6/7 4 4-9
- - - 7//6 Step 6 5 4-9 6 - - Corso di Robotica 6/7 4 -
Final Result 5 4 9 6 4 5 7 7//6 4-6 5 9 8 5 Corso di Robotica 6/7
Separability Definition: -D kernel can be written as convolution of two -D kernels Advantage: Efficiency Convolving image with kernel requires multiplies vs. for nonseparable kernel 7//6 Corso di Robotica 6/7
Smoothing (Low-Pass) Filters Replace each pixel with average of neighbors Benefits: Suppress noise, aliasing Disadvantage: Sharp features blurred Types Mean filter (box) Median (nonlinear) Gaussian x box filter 7//6 Corso di Robotica 6/7
Box Filter: Smoothing Original image 7//6 7 x 7 kernel Corso di Robotica 6/7 4
Gaussian Kernel Idea: Weight contributions of neighboring pixels by nearness 7//6 Corso di Robotica 6/7 5
Gaussian: Benefits Rotational symmetry treats features of all orientations equally (isotropy) Smooth roll-off reduces ringing Efficient: Rule of thumb is kernel width 5σ Separable Cascadable: Approach to large σ comes from identity 7//6 Corso di Robotica 6/7 6
Gaussian: Smoothing Original image 7x7 kernel σ= 7//6 σ= Corso di Robotica 6/7 7
The Features world 7//6 Corso di Robotica 6/7 8
Edge Causes Depth discontinuity Surface orientation discontinuity Reflectance discontinuity (i.e., change in surface material properties) Illumination discontinuity (e.g., shadow) surface normal discontinuity depth discontinuity surface color discontinuity illumination discontinuity 7//6 Corso di Robotica 6/7 9
Gradient Think of image intensities as a function. Gradient of image is a vector field as for a normal -D height function: Edge: Place where gradient magnitude is high; orthogonal to gradient direction 7//6 Corso di Robotica 6/7
What is the gradient? No Change I I, = (k,) x y Change 7//6 Corso di Robotica 6/7
What is the gradient? Change I I, = (, k ) x y No Change 7//6 Corso di Robotica 6/7
What is the gradient? I I, = (k, k ) x y Gradient direction is perpendicular to edge. Gradient Magnitude measures edge strength. 7//6 Less Change Much Change Corso di Robotica 6/7
Edge Detection Edge Types Step edge (ramp) Line edge (roof) Searching for Edges: Filter: Smooth image Enhance: Apply numerical derivative approximation Detect: Threshold to find strong edges Localize/analyze: Reject spurious edges, include weak but justified edges 7//6 Corso di Robotica 6/7 4
Effects of noise Consider a single row or column of the image Plotting intensity as a function of position gives a signal 7//6 Corso di Robotica 6/7 5
Effects of noise: solution 7//6 Corso di Robotica 6/7 6
Step edge detection First derivative edge detectors: Look extrema - Sobel operator - Prewitt, Roberts cross - Derivative of Gaussian Sobel x for - - - Sobel y Second derivative: Look for zero-crossings Laplacian Second directional derivative Laplacian of Gaussian/Difference of Gaussians 7//6 : Isotropic Corso di Robotica 6/7 7
Prewitt, Roberts cross (a): Roberts cross operator (b): x Prewitt operator (c): Sobel operator (d) 4x4 Prewitt operator 7//6 Corso di Robotica 6/7 8
Derivative of Gaussian 7//6 Corso di Robotica 6/7 9
Derivative of Gaussian Derivative theorem of convolution 7//6 Corso di Robotica 6/7 4
Laplacian of Gaussian 7//6 Corso di Robotica 6/7 4
Laplacian of Gaussian Consider Laplacian of Gaussian operator 7//6 Corso di Robotica 6/7 4
Edge Filtering Example - - - Rotate 7//6 - - - Corso di Robotica 6/7 4
- - - Step - - - 7//6 Corso di Robotica 6/7 44
- - - 7//6 Step - 4 Corso di Robotica 6/7 6 45
- - - 7//6 Step - 4 Corso di Robotica 6/7 6 6 46
- - - 7//6 Step 4 - -4 - Corso di Robotica 6/7 6 6-6 edge effect 47
Sobel Edge Detection: Gradient Approximation Horizontal 7//6 Vertical Corso di Robotica 6/7 48
Sobel vs. LoG Edge Detection Sobel 7//6 LoG Corso di Robotica 6/7 49
Canny Edge Detection Derivative of Gaussian Non-maximum suppression Thin multi-pixel wide ridges down to single pixel Thresholding Low, high edge-strength thresholds Accept all edges over low threshold that are connected to edge over high threshold 7//6 Corso di Robotica 6/7 5
Non-maximum Suppression We wish to mark points along the curve where the magnitude is biggest. 7//6 Corso di Robotica 6/7 5
Non-maximum Suppression At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values. 7//6 Corso di Robotica 6/7 5
Non-maximum Suppression Assume the marked point is an edge point. Then we construct the tangent to the edge curve (which is normal to the gradient at that point) and use this to predict the next points (here either r or s). 7//6 Corso di Robotica 6/7 5
Non-maximum Suppression courtesy of G. Loy Original image 7//6 Gradient magnitude Corso di Robotica 6/7 Non-maxima suppressed 54
Edge Hysteresis Hysteresis: A lag or momentum factor Idea: Maintain two thresholds khigh and klow Use khigh to find strong edges to start edge chain Use klow to find weak edges which continue edge chain Typical ratio of thresholds is roughly khigh / klow = 7//6 Corso di Robotica 6/7 55
Edge Hysteresis gap is gone Strong + connected weak edges Original image Strong edges only Weak edges courtesy of G. Loy 7//6 Corso di Robotica 6/7 56
Canny Edge Detection: Example 7//6 Corso di Robotica 6/7 57
Edge detection by subtraction original 7//6 Corso di Robotica 6/7 58
Edge detection by subtraction smoothed (5x5 Gaussian) 7//6 Corso di Robotica 6/7 59
Edge detection by subtraction Why does this work? smoothed original (scaled by 4, offset +8) 7//6 Corso di Robotica 6/7 6
Gaussian - image filter Laplacian of Gaussian Gaussian 7//6 delta function Corso di Robotica 6/7 6
An edge is not a line... How can we detect lines? 7//6 Corso di Robotica 6/7 6
Finding lines in an image Option : Search for the line at every possible position/orientation What is the cost of this operation? Option : 7//6 Use a voting scheme: Hough transform Corso di Robotica 6/7 6
Finding lines in an image y b b m x image space m Hough space Connection between image (x,y) and Hough (m,b) spaces A line in the image corresponds to a point in Hough space To go from image space to Hough space: given a set of points (x,y), find all (m,b) such that y = mx + b 7//6 Corso di Robotica 6/7 64
Finding lines in an image y b y x image space x Hough space m Connection between image (x,y) and Hough (m,b) spaces A line in the image corresponds to a point in Hough space To go from image space to Hough space: given a set of points (x,y), find all (m,b) such that y = mx +b What does a point (x, y) in the image space map to? A: the solutions of b = -xm + y this is a line in Hough space 7//6 Corso di Robotica 6/7 65
Hough transform algorithm Typically use a different parameterization d is the perpendicular distance from the line to the origin θ is the angle this perpendicular makes with the x axis Why? 7//6 Corso di Robotica 6/7 66
Hough transform algorithm Typically use a different parameterization d is the perpendicular distance from the line to the origin θ is the angle this perpendicular makes with the x axis Why? Basic Hough transform algorithm. Initialize H[d, θ]=. for each edge point I[x,y] in the image for θ = to 8 H[d, θ] +=. Find the value(s) of (d, θ) where H[d, θ] is maximum 4. The detected line in the image is given by What s the running time (measured in # votes)? 7//6 Corso di Robotica 6/7 67
Hough transform algorithm tokens Votes Horizontal axis is θ, vertical is d. 7//6 Corso di Robotica 6/7 68
Hough transform algorithm tokens 7//6 votes Corso di Robotica 6/7 69
Hough transform algorithm 7//6 Corso di Robotica 6/7 7
Extensions Extension : Use the image gradient. same. for each edge point I[x,y] in the image compute unique (d, θ) based on image gradient at (x,y) H[d, θ] +=. same 4. same What s the running time measured in votes? 7//6 Corso di Robotica 6/7 7
Extensions Extension : Use the image gradient. same. for each edge point I[x,y] in the image compute unique (d, θ) based on image gradient at (x,y) H[d, θ] +=. same 4. same What s the running time measured in votes? Extension give more votes for stronger edges Extension change the sampling of (d, θ) to give more/less resolution Extension 4 7//6 The same procedure can be used with circles, squares, or any other shape Corso di Robotica 6/7 7
Finding Corners Edge detectors perform poorly at corners. Corners provide repeatable points for matching, so are worth detecting. Idea: Exactly at a corner, gradient is ill defined. However, in the region around a corner, gradient has two or more different values. 7//6 Corso di Robotica 6/7 7
Corner Detection Basic idea: Find points where two edges meet i.e., high gradient in orthogonal directions Examine gradient over window (Shi & Tomasi, 994) Edge strength encoded by eigenvalues corner is where over threshold Harris corners (Harris & Stephens, 988), Susan corners (Smith & Brady, 997) 7//6 Corso di Robotica 6/7 ; 74
The Harris corner detector Form the second-moment matrix: Gradient with respect to x, times gradient with respect to y Sum over a small region around the hypothetical corner I C= I x I y x I I I x y y Matrix is symmetric 7//6 Corso di Robotica 6/7 Slide credit: David Jacobs 75
Simple Case First, consider case where: I C= I x I y x I I I x y y λ = λ This means dominant gradient directions align with x or y axis If either λ is close to, then this is not a corner, so look for locations where both are large. 7//6 Corso di Robotica 6/7 76
General Case It can be shown that since C is symmetric: λ C=R R λ So every case is like a rotated version of the one on last slide. 7//6 Corso di Robotica 6/7 77
So, to detect corners Filter image with Gaussian to reduce noise Compute magnitude of the x and y gradients at each pixel Construct C in a window around each pixel (Harris uses a Gaussian window just blur) Solve for product of λs (determinant of C) If λs are both big (product reaches local maximum and is above threshold), we have a corner (Harris also checks that ratio of λs is not too high) 7//6 Corso di Robotica 6/7 78
Gradient Orientations 7//6 Corso di Robotica 6/7 79
Step : Gradient Orientations 7//6 Corso di Robotica 6/7 8
Eigenvalues Corners are detected where the product of the ellipse axis lengths reaches a local maximum. 7//6 Corso di Robotica 6/7 8
Example: Corner Detection courtesy of S. Smith SUSAN corners 7//6 Corso di Robotica 6/7 8