EE368/CS232 Digital Image Processing Winter Homework #7 Solutions

Transcription

1 1. Robustness of the SIFT Descriptor EE368/CS232 Digital Image Processing Winter Homework #7 Solutions Part A: Below, we show the SIFT keypoints for the original test image, before any additional distortions are introduced. We neglect the effect that the four types of distortions would have on the keypoint detector and focus exclusively on their effect on the feature descriptor in this problem, although in practice both the detector and the descriptor would be affected. Part B: The following plot shows how repeatability varies as the brightness offset changes. For small brightness offsets, the SIFT descriptor does not change, because the descriptor is formed from image gradients which are unaffected by a brightness offset. The reason why some of the descriptors change for larger brightness offsets is that the gray values are saturated when they reach 0 or 255 in an 8-bit image representation. 1

2 Repeatability Brightness Offset Part C: The following plot shows how repeatability varies as the g value in the contrast adjustment is changed. Like for small brightness offsets, the SIFT descriptor only changes very slightly when the contrast is adjusted. The SIFT descriptor is normalized to have unit L2 norm, so most effects of contrast changes are removed during the normalization Repeatability Contrast Gamma Part D: The following plot shows how repeatability varies as the noise standard deviation is increased. There is a slow but steady drop in the repeatability for larger noise levels. Since the image is low-pass filtered with a Gaussian kernel in generating the scale-space, the noise is reduced before the image gradients and feature descriptors are computed. 2

3 Repeatability Noise Standard Deviation Part E: The following plot shows how repeatability varies as the standard deviation of the Gaussian kernel used for blurring is increased. Since large amounts of blurring significantly alter the gradients and consequently the histograms of gradients, the SIFT descriptor changes noticeably as the amount of blurring increases. Of the four types of distortions tested, blurring seems to have the most serious effect on the SIFT descriptor Repeatability MATLAB Code: Blur Kernel Standard Deviation % EE368/CS232 % Homework 7 % Problem: SIFT Robustness % Script by David Chen, Huizhong Chen clc; clear all; run('../../../vlfeat /toolbox/vl_setup.m'); imagefiles = {'building.jpg'}; for nimage = 1:length(imageFiles) % Load image img = imread(imagefiles{nimage}); imggray = rgb2gray(img); peakthresh = 2; edgethresh = 10; 3

4 [f,d] = vl_sift(single(imggray)); disp(sprintf('image %d: %d features', nimage, size(f,2))); [fo,do] = vl_sift(single(imggray), 'Frames', f); [f,d] = reorder_sift_features(f, d, fo, do); figure(1); clf; set(gcf, 'Color', 'w'); imshow(imggray); hold on; h1 = vl_plotframe(f); h2 = vl_plotframe(f); set(h1, 'Color', 'k', 'linewidth', 3); set(h2, 'Color', 'y', 'linewidth', 2); % Study effect of brightness offset featurematches = []; offsetvec = -100 : 20 : 100; for offset = offsetvec % Add brightness offset if offset == 0 imgoffset = imggray; else imgoffset = uint8(double(imggray) + offset); [fo,do] = vl_sift(single(imgoffset), 'Frames', f); % Compare SIFT descriptors matches = vl_ubcmatch(do, d); featurematches(+1) = size(matches,2); % offset figure(2); clf; set(gcf, 'Color', 'w'); maxmatches = size(f,2); h = plot(offsetvec, featurematches/maxmatches, 'b-o'); grid on; set(h, 'LineWidth', 2); set(gca, 'FontSize', 14); xlabel('brightness Offset'); ylabel('repeatability'); axis([min(offsetvec) max(offsetvec) 0 1]); % Study effect of contrast change featurematches = []; gammavec = 0.5 : 0.25 : 2.0; for gamma = gammavec % Adjust contrast if gamma == 0 imggamma = imggray; else imgdouble = im2double(imggray); imggamma = uint8(255*imgdouble.^gamma); [fo,do] = vl_sift(single(imggamma), 'Frames', f); % Compare SIFT descriptors matches = vl_ubcmatch(do, d); featurematches(+1) = size(matches,2); % offset figure(3); clf; set(gcf, 'Color', 'w'); h = plot(gammavec, featurematches/maxmatches, 'b-o'); grid on; set(h, 'LineWidth', 2); set(gca, 'FontSize', 14); xlabel('contrast Gamma'); ylabel('repeatability'); axis([min(gammavec) max(gammavec) 0 1]); % Study effect of adding noise featurematches = []; gaussvarvec = (0:5:30).^2; for gaussvar = gaussvarvec % Add Gaussian noise gaussstd = sqrt(gaussvar); if gaussvar == 0 imgnoise = imggray; 4

5 else noise = gaussstd * (randn(size(imggray)) - 0.5); imgnoise = uint8(double(imggray) + noise); [fo,do] = vl_sift(single(imgnoise), 'Frames', f); % Compare SIFT descriptors matches = vl_ubcmatch(do, d); featurematches(+1) = size(matches,2); % gaussvar figure(4); clf; set(gcf, 'Color', 'w'); h = plot(sqrt(gaussvarvec), featurematches/maxmatches, 'b-o'); grid on; set(h, 'LineWidth', 2); set(gca, 'FontSize', 14); xlabel('noise Standard Deviation'); ylabel('repeatability'); axis([min(sqrt(gaussvarvec)) max(sqrt(gaussvarvec)) 0 1]); % Study effect of blurring featurematches = []; gaussvarvec = (1:10).^2; for gaussvar = gaussvarvec % Filter with Gaussian kernel if gamma == 0 imgblur = imggray; else sigma = sqrt(gaussvar); width = ceil(10*sigma + 1); if mod(width,2) == 0 width = width+1; kernel = fspecial('gaussian', [width width], sigma); imgblur = imfilter(imggray, kernel, 'symmetric'); [fo,do] = vl_sift(single(imgblur), 'Frames', f); % Compare SIFT descriptors matches = vl_ubcmatch(do, d); featurematches(+1) = size(matches,2); % offset figure(5); clf; set(gcf, 'Color', 'w'); h = plot(sqrt(gaussvarvec), featurematches/maxmatches, 'b-o'); grid on; set(h, 'LineWidth', 2); set(gca, 'FontSize', 14); xlabel('blur Kernel Standard Deviation'); ylabel('repeatability'); axis([min(sqrt(gaussvarvec)) max(sqrt(gaussvarvec)) 0 1]); if nimage < length(imagefiles) pause % nimage 5

6 2. Recognition of Posters with Local Image Features Below, we show (1) the query image and the best matching database image, (2) the SIFT features extracted from both images, (3) the feature matches after nearest neighbor search with a distance ratio test, and (4) the feature matches after RANSAC with a homography. Query Image (Left) and Best Matching Database Image (Right) SIFT Features Feature Matches After Distance Ratio Test Feature Matches after RANSAC 6

7 Query Image (Left) and Best Matching Database Image (Right) SIFT Features Feature Matches After Distance Ratio Test Feature Matches After RANSAC 7

8 Query Image (Left) and Best Matching Database Image (Right) SIFT Features Feature Matches After Distance Ratio Test Feature Matches After RANSAC MATLAB Code: % % % % EE368/CS232 Homework 7 Problem: Poster Matching Script by David Chen, Huizhong Chen clc; clear all; run('../../../vlfeat /toolbox/vl_setup'); 8

9 %% Process database dbimagefiles = dir('database/*.jpg'); peakthresh = 1; edgethresh = 10; for nimage = 1:length(dbImageFiles) fprintf('%d: %s \n', nimage, dbimagefiles(nimage).name); % Load and resize image img = imread(['database/' dbimagefiles(nimage).name]); [height,width,channels] = size(img); targetwidth = 800; resizefactor = targetwidth/width; img = imresize(img, resizefactor, 'bicubic'); dbimages{nimage} = img; imggray = rgb2gray(img); [f,d] = vl_sift(single(imggray),... 'PeakThresh', peakthresh, 'EdgeThresh', edgethresh); dbfeatures{nimage}.f = f; dbfeatures{nimage}.d = d; % nimage %% Recognize queries queryimagefiles = {'hw7_poster_1.jpg', 'hw7_poster_2.jpg', 'hw7_poster_3.jpg'}; for nquery = 1:length(queryImageFiles) fprintf('%d: %s \n', nquery, queryimagefiles{nquery}); % Load image img = imread(queryimagefiles{nquery}); imggray = rgb2gray(img); [f,d] = vl_sift(single(imggray),... 'PeakThresh', peakthresh, 'EdgeThresh', edgethresh); qfeatures.f = f; qfeatures.d = d; % Match to database SIFT features maxransacinliers = 0; maxransacinliersimage = -1; for ndatabase = 1:length(dbImageFiles) result = sift_match(img, dbimages{ndatabase},... qfeatures, dbfeatures{ndatabase}); if result.ransac > maxransacinliers maxransacinliers = result.ransac; maxransacinliersimage = ndatabase; % ndatabase sift_match(img, dbimages{maxransacinliersimage},... qfeatures, dbfeatures{maxransacinliersimage}); if nquery < length(queryimagefiles) pause % nquery 9

10 3. Recognition of Paintings with a Vocabulary Tree Below, we show next to each query image, the 10 most similar database images in terms of vocabulary tree scoring. Note that your database ranked lists may be slightly different, due to randomness in the hierarchical k-means clustering. The scores shown are the L1 distances between the query tree histogram and the database tree histograms. For all 5 query images, the top-ranked database image is the correct result. The feature extraction latency is around 0.81 seconds per query, and the retrieval latency is around 0.19 seconds per query. Query Image Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Elapsed time for feature extraction: 0.63 seconds Elapsed time for image retrieval: 0.20 seconds Query Image Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Elapsed time for feature extraction: 0.62 seconds Elapsed time for image retrieval: 0.19 seconds 10

11 Query Image Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Elapsed time for feature extraction: 0.99 seconds Elapsed time for image retrieval: 0.19 seconds Query Image Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Elapsed time for feature extraction: 0.88 seconds Elapsed time for image retrieval: 0.19 seconds 11

12 Query Image Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Database Score: Elapsed time for feature extraction: 0.94 seconds Elapsed time for image retrieval: 0.19 seconds MATLAB Code: % EE368/CS232 % Homework 7 % Problem: Image Retrieval of Paintings % Script by David Chen, Huizhong Chen clc; clear all; run('../../../vlfeat /toolbox/vl_setup.m'); %% Train vocabulary tree k = 10; numleaves = 10000; treefile = 'vocabulary_tree.mat'; if ~exist(treefile, 'file'); load('training_descriptors.mat'); [tree, assign] = vl_hikmeans(uint8(trainingdescriptors), k, numleaves); save(treefile, 'tree'); else load(treefile); %% Extract features for database images peakthresh = 1; edgethresh = 10; databasefeatures = {}; databaseimages = {}; for nimage = 1:91 filename = sprintf('database/%03d.jpg', nimage); img = imread(filename); imggray = rgb2gray(img); databaseimages{+1} = img; [f,d] = vl_sift(single(imggray), 'PeakThresh', peakthresh, 'EdgeThresh', edgethresh); features.f = f; features.d = d; 12

13 databasefeatures{+1} = features; disp(sprintf('database Image %d: %d features', nimage, size(f,2))); % nimage %% Extract features for query images queryfeatures = {}; queryimages = {}; for nimage = 1:5 tic; filename = sprintf('query/%03d.jpg', nimage); img = imread(filename); imggray = rgb2gray(img); queryimages{+1} = img; [f,d] = vl_sift(single(imggray), 'PeakThresh', peakthresh, 'EdgeThresh', edgethresh); features.f = f; features.d = d; queryfeatures{+1} = features; disp(sprintf('query Image %d: %d features', nimage, size(f,2))); toc; % nimage %% Extract tree histograms for database images databasetreehists = {}; for nimage = 1:length(databaseImages) % Quantize descriptors through vocabulary tree treepaths = vl_hikmeanspush(tree, databasefeatures{nimage}.d); % Compute histogram over bins treehist = vl_hikmeanshist(tree, treepaths); databasetreehists{+1} = treehist(-numleaves+1:) /... sum(treehist(-numleaves+1:)); % nimage %% Match query to database by vocabulary tree scoring for nquery = 1:length(queryImages) tic; % Quantize descriptors through vocabulary tree treepaths = vl_hikmeanspush(tree, queryfeatures{nquery}.d); % Compute histogram over bins treehist = vl_hikmeanshist(tree, treepaths); treehist = treehist(-numleaves+1:) / sum(treehist(-numleaves+1:)); % Compare against database histograms scores = zeros(1, length(databaseimages)); for ndatabase = 1:length(databaseImages) scores(ndatabase) = sum(abs(treehist - databasetreehists{ndatabase})); % ndatabase [scores, sortidx] = sort(scores, 'asc'); figure(1); clf; subplot(2,6,1); imshow(queryimages{nquery}); title('query Image'); for ntop = 1:10 subplot(2,6,ntop+1); imshow(databaseimages{sortidx(ntop)}); title(sprintf('database Score: %.3f', scores(ntop))); % ntop toc; if nquery < length(queryimages) pause % nquery 13