521466S Machine Vision Assignment #3 Image Features

Transcription

1 521466S Machine Vision Assignment #3 Image Features Spring 2018 This assignment explores feature detection, extraction, and matching. We will implement the well-known Harris corner detector and use Matlab s computer vision toolbox to match the detected corners between images. We will demonstrate the usefulness of image features by stitching multiple images together to produce a panorama image. Figure 1: Results of feature matching. Theory Finding correspondences The process of automatically finding correspondences between two images usually consists of four steps: feature detection, feature extraction, feature matching, and outlier rejection. Feature detection selects the points of interest in the image. These points should have a high likelihood of being accurately matched in another image. Since areas of little texture and straight edges cannot be accurately matched, this step is also known as corner detection. In this assignment we will use the Harris corner detector. Feature extraction creates a description of the image texture around a detected feature. The descriptor can be as simple as the intensity values of a block around the point, or it can be a complex descriptor such as SIFT or SURF.

2 Feature matching tries to find correspondences between the extracted features of two images. The details of the matching depend on the type of descriptor extracted. Even the most robust matching algorithms produce incorrect matches. These incorrect matches (outliers) must be detected and removed. Harris corner detector Here is a brief summary of the formulation of the Harris corner detector. Refer to the lecture notes for more details. The Harris corner detector computes a corner response function R for each pixel based on the image gradients (δi/δx and δi/δy). For each pixel (x, y) we define a measure of change as a matrix M of the form: M xy = ( ) 2 δixy δi xy δi xy δx δx δy ( δi xy δi xy δixy δx δy δy ) 2 (1) We perform a weighted sum over an area to obtain a more realistic measure of change: M xy = W (i x, j y)m ij (2) (i,j) N xy ( ) 2 δixy δixy δi xy δx δx δy = ( ) 2 δixy δi xy δixy (3) δx δy δy where N xy represents a neighbourhood around (x, y) (usually a rectangular block centered at (x, y)) and W is a weighted function (usually a Gaussian kernel). The final corner response function R is an approximation of the product of eigenvalues of M: R xy = det(m xy ) α trace(m xy ) 2 (4) To extract corner points from this corner response function we apply non-maxima suppression, select a suitable threshold τ R, and select all pixels (x, y) where R xy > τ R. Image stitching Image stitching is the process of combining multiple images with overlapping fields of view to a single panorama or high-resolution image. The panorama image in Figure 4 has been produced by combining 5 images. We will first detect and extract features from each image. Then we will match the features in consecutive image pairs and estimate the geometric transformations between them. The middle image is used as a reference and the panorama image is produced by registering all other images to the reference image. Note: you will learn more about transformations later in the course, for now we will let the Computer Vision toolbox compute them for us and warp the images to align them with the reference image.

3 Instructions 1. Download the code and images This assignment comes with Matlab functions and test images. They can be downloaded from: The assignment also has a main script that should guide you through the steps: main.m. It is useful to use Ctrl+Enter in the editor window to execute only a single section of the main script. The numbering of the instructions here corresponds to the numbering in the main script. 2. Read reference and target image Select an image that we will use as reference and one that we will use as a target. Use the provided settings, that is, we will use the first and second image. 3. Detect harris corners Complete the provided function detectharris(). The function already does nonmaxima suppression and thresholding, but it doesn t compute the corner response function R. Hints: Use imgradientxy() to compute δi/δx and δi/δy Compute 3 matrices of the same size as the image that hold (δi xy /δx) 2, (δi xy /δy) 2, and (δi xy δi xy /δxδy) Use the provided gaussiankernel as the window kernel for the weighted sum. Use imfilter() to compute the weighted sum for each component of M. Try to use Matlab s per-element operators (A.*B instead of A(i)*B(i)) to compute things faster. Once the function is completed, execute step 3 to view the detected corners for the images. They should look similar to Figure Extract feature descriptors We will now extract descriptors for all detected corners. The Computer Vision toolbox provides several types of descriptors that we can use. For now we will use the simplest (and fastest) type: BLOCK, which simply copies the pixel values from a rectangular block around the image. 5. Match features Execute step 5 to match the extracted descriptors using the Computer Vision toolbox. The resulting matches should look like in Figure 3. The reference and target images are overlayed on different color channels. Notice how many matches are correct but there are some wrong matches as well. 6. Aligning the images Once matches are obtained, we can fit a transformation that describes the movement between the reference and target image. We will learn more about transformations

4 Figure 2: Results of feature detection. Figure 3: Results of feature matching. later in the course. For now, execute step 6 to fit the transformation and warp the target image. This process detects and rejects the outliers as shown in Figure Image stitching Execute step 7 to stitch together 5 images. Notice that we can only stitch together the first 3 images. The image (4) causes problems since it has been rotated with respect to the reference image. (Q1) Why does the matching fail when we use the square neighbourhood (BLOCK) as a descriptor? 8. Using SURF descriptors Change the featuremethod in step 4 to use SURF features instead of BLOCK. Repeat the step 7. Observe how the matching now handles rotations in 2D. The resulting image should look similar to Figure 4. (Q2) Why does matching a rotated image work when using a more complicated feature descriptor (like SIFT or SURF)? Choose one of these feature descriptors and explain how it deals with this rotation.

5 Figure 4: Panorama image obtained with SURF features. Deliverables Send an to with the following. The subject line should be MV-A3-[student number] (e.g. MV-A ). A single attached zip file named MV-A3-[student number].zip which should include: 1. detectharris.m (1.0 point) 2. answers.txt or answers.pdf with (a) Answer to the question Q1. (b) Answer to the question Q2. (0.5 points) (0.5 points) Deadline of the assignment