Local Descriptors CS 510 Lecture #21 April 6 rd 2015
A Bit of Context, Transition David G. Lowe, "Three- dimensional object recogni5on from single two- dimensional images," Ar#ficial Intelligence, 31, 3 (March 1987) David G. Lowe, "Object recogni5on from local scale- invariant features," Interna#onal Conference on Computer Vision, Corfu, Greece (September 1999) CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 2
Focus of Attention Mark image features that: 1. Specifies a location and a scale 2. Can be repeatedly identified 3. Contain information But how do we use them? CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 3
Recall DoG Example Earlier We did the following in Photoshop Now we will push this idea further CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 4
Create a whole Sequence Build an image pyramid An octave of the pyramid is where σ = 2 Lowe recommends 3 images per octave Estimate DoG responses by subtracting layers Highly efficient Extract local max/min positions and scales CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 5
Pyramid Example Google finds over 200 instances of this image on the web, alas I ve not sa1sfied myself that I have found the original author/creator. CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 6
Corners & DoG extrema Lines and bars in images tend to create high DoG responses The positions of extrema along these features are determined by noise, and therefore not stable CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 7
Therefore What is a corner? Checkerboard, Wikipedia Leads us to the Structure Tensor. Checkerboard Mesa Morning by Don Lee Brown CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 8
The Structure Tensor The structure tensor is the outer product of the partial derivatives with themselves: # 2 dx i dx i dy & % i ( i i C= % 2 ( dx i dy i dy i $ % ' ( i i Consider the Eigenvalues Both near zero => no edge (image is locally flat) One large, one near zero => edge Both large => a strong corner CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 9
Structure Tensor Eigenvalues The structure tensor summarizes the edge information in an image window If both eigenvalues are small, the window is a roughly flat surface Not good for matching, good for grouping If one eigenvalue is small, the window contains an edge Orientation is reliable Position of match is not (aperture effect) If both eigenvalues are large, the window contains a corner Orientation is one of two Position matches are reliable (good points to match) CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 10
DoG Extrema should be corners! Analyze the structure tensor of the window around the interest points (DoG extrema) Compute edge directions (dx & dy) in window around extremum Compute the eigenvectors of the derivative covariance matrix Exclude extrema whose smaller eigenvalue is below a threshold CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 11
Interest Point Examples hyp://computervisionblog.wordpress.com/tag/si[- feature- point/ hyp://www.scholarpedia.org/ar1cle/scale_invariant_feature_transform CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 12
Matching With Interest Points One approach: Extract multiple mini templates Corresponding to interest points on your object Correlate template IPs to target IPs Template IPs should have a match Positioning doesn t have to be exact Less sensitive to pose, other variations But we will pursue another approach CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 13
Local Descriptors Goal: describe image patches around interest points As high-dimensional points in feature space But not too high Edge and/or color information Invariant to small affine transformations Keep spatial relations CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 14
SIFT features (Lowe 2004) Extract edges Divide image patch into quarters Create a histogram of edge orienta1ons in each quarter Concatenate the histograms CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 15
SURF Features (Bay et al 2006) Speeded-Up Robust Features (avoid the SIFT patent) Same idea as SIFT localized edge histograms Edges estimated using box-filter orientation approximations Much faster to compute Less accurate Available in OpenCV (SIFT is not) CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 16
SURF: Fast Approximation SIFT analyzes structure tensor to determine local orientation Dx, Dy, DxDy are non-integer values SURF speeds up directional computations Images from OpenCV documenta#on CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 17
Histogram of Gradients (HoG) Generalize SIFT detector to bigger grids Typically 4x4 Histogram of 8 edge directions Normalized to make max value 1 hyp://www.emeraldinsight.com/content_images/fig/0870330310029.png CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 18
Local Binary Patterns (LBP) Method: reduce local texture around a pixel to a code Histogram codes Radius = scale hyp://www.scholarpedia.org/ar1cle/local_binary_payerns CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 19
LBP in Practice 256 codes are too many Most codes never occur (under-sampling) Some applications prefer smaller feature spaces Ignore codes with more than two transitions i.e. ignore speckle patterns ~ 58 codes (there are variations ) CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 20
GIST Another approach is to ignore interest points Create a feature vector from the whole image Detect edge orientations & color patterns Histogram the result CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 21
hyp://ilab.usc.edu/siagian/research/gist/gist.html GIST Divide image into cells 16 cells are shown Convolve cells with feature masks Produce 7 vectorized images per cell Use PCA to reduce dimensionality CS 510, Image Computa1on, Ross Beveridge & Bruce Draper 22