CS221: Final Project Report - Object Recognition and Tracking

Size: px

Start display at page:

Download "CS221: Final Project Report - Object Recognition and Tracking"

Juliana Price
5 years ago
Views:

1 CS221: Final Project Report - Object Recognition and Tracking Shaojie Deng Ya Xu March 20th, 2009 We call a sub-window positive if it is one of the objects:,,, or. Otherwise it is called negative or background image. The number of positive training samples in each category is listed in Table 1. All images (training or testing) are grayscaled and histogram-equalized (using cvequalizehist()). xxx Table 1: Number of positive training samples 1 Components of Our Classifier Our program is mainly comprised of four components, as described in detail below. 1.1 Cascade of boosted trees with Haar features The cascade 1 is built based on Haar features. On top of the seven simple Haar feature templates used in the milestone, we use more complicated templates as shown in Figure 1. We believe some of the added templates can help distinguish the objects from the background or one object from another. For instance, templates 1(c) and (d) may help separate from, whereas template 2(c) can help identify. The cascade is then formed by stages of boosted trees, using the Haar features as the basic input and Adaboost as the basic learning algorithm. Because of the large number of features, the Viola and Jones implementation has a few tricks to speed up the classification: (1) Use integral image to compute Haar features. The integral image can be computed in one pass over the original image. Any Haar feature in Figure 1 can be computed within a few array references. (2) Adaboost is used to both select important features and boost the performance of weak leaners. This is particularly important because the total number of features considered is very large. (3) The cascade is built by stages. At each stage, sub-windows that are identified as background are thrown away and only the ones identified as positive objects are passed over to the next stage. This largely reduces computation spent on negative windows because earlier stages are built using smaller number of features. Since most sub-windows are background, the total computation cost is significantly reduced. 1 This component of our program is based on Paul Viola and Michael Jones, Rapid object detection using a boosted cascade of simple features, International Journal of Computer Vision,

2 CS221 Shaojie Deng and Ya Xu 2 Figure 1: Haar feature templates. Taken from OpenCV reference manual. Because the cascade is designed for binary classification, we build five cascades, one for each category, to adapt to our problem. When training for one object, images of other objects, together with images of background, are used as negative samples. This turns out to be a little better than using only the background images as negative samples when we combine all the components together (see section 2). 1.2 Nearest neighbor with SURF SURF 2 is a scale and rotation invariant interest point detector and descriptor. The interest points are selected based on the Hessian matrix. The descriptor describes a distribution of Haar-wavelet responses within the interest point neighborhood. The following tricks are used to speed up computation of SURF features of a given image: (1) Use integral image to compute Haar-wavelet. (2) Use box filters, instead of Gaussian filters, for the Hessian matrix. (3) Reduce the descriptor s dimension and complexity. This not only reduces cost to compute the descriptors, but also reduces the cost to match two interest points. For any pair of images, we can then compute the SURF similarity score between them. The higher the score, the more similar the two images are. (xxxhow to define the scores?) Each test image is compared against all the positive training images. We then classify the test image to be in the same category as the positive training image that is the most similar to it based on the similarity score. Notice that this nearest neighbor method is used only to distinguish among different object categories, i.e. it does distinguish objects from background. To avoid repeatedly computing SURF features for the same training images, we save the SURF features in an XML file and load them before testing. 1.3 Random forest with various features Random forest is an ensumble method that puts together a large number of trees (e.g. 500) to achieve good classification results. A random set of the training samples is used to build a single 2 This component of our program is based on Herbert Bay, Tinne Tuytelaars and Luc Van Gool, SURF: Speeded Up Robust Features..

3 CS221 Shaojie Deng and Ya Xu 3 tree and the rest are used as validation set for that tree. Each node is also chosen from a random set of variables. See Breiman s paper 3 for more information. The features used to build the random forest include (all input images are resized to 64 by 64 before extracting the features): (1) Number of circles: the circles are detected and counted using cvhoughcircles(). The input images are smoothed with cvsmooth(). This is a feature that is particularly helpful identifying s, as shown in Table 2. No circle is found in the training images of objects other than. Out of the 49 training images, at least one circle is found in 40 of them. detected circle 0/295 0/393 40/49 0/307 0/235 Table 2: Number of training images with at least one circle detected. (2) Standard deviation: standard deviation of the canny edged image. The distributions of the standard deviation of images from different categories are plotted in Figure 2 left. As we can see, in general has large standard deviation and the values are concentrated between 0.45 and 0.5, whereas is concentrated between 0.4 and (3) Maximum eigenvalue: singular value decomposition is done using cvsvd() on canny edged image. The distributions of the maximum singular value of the images from different categories are plotted in Figure 2 right. Similar to the standard deviation distributions, stands out the most. distribution of std distribution of maxsingular Density Density standard deviation maxium singular value Figure 2: Left: Distribution of standard deviation of images of different categories. Right: Distribution of maximum sigular value of images of different categories. (4) Corner-related features: corners are found using cvgoodfeaturestotrack(). The first corner feature is the number of corners found. The distribution is plotted in Figure 3. The second 3 Leo Breiman, Random Forests Machine Learning 45 (1), 5-32.

4 CS221 Shaojie Deng and Ya Xu 4 group of features is the location distribution of the corners in terms of polar coordinates ρ and θ. In more detail, the location of a corner relative to the center of the image is recorded in ρ and θ. The distribution of ρ (and θ) from one image is then binned into histograms of fixed bins. The percentages of points fall in each bin forms the second group of features (i.e. the distribution of ρ and θ). Figure 4 includes the distribution of both ρ and θ averaged within each category. We notice that, and all have similar ρ distribution, but really stands out as having a flat ρ distribution. On the other hand, has a almost uniform θ distribution that really stands out from the rest. To compute the third group of features, we need to store the signiture distribution for each category, which is computed as the averaging histogram of all the positive training images of that category (for both ρ and θ). The third group of features is then the Earth Moving Distance between the ρ (or θ) distribution of any image and the signiture distributions as included in Table 3. Notice that the diagonals are in general smaller than the off-diagonals as we would expect. distribution of numcorners Density number of corners found Figure 3: Distribution of number of corners found in images of different categories. ρ signiture θ signiture Table 3: Average Earth Moving Distance of an image in the row category to the signiture of object in the column category. (5) SURF-related features: all positive training images are treated as signitures. A test image is compared against all positive images of all categories. The corresponding SURF similarity scores and the number of matched interest points are recorded. The top 5 scores (or number of interest points), together with the mean scores, against training images in each category

5 CS221 Shaojie Deng and Ya Xu 5 average rho distribution average theta distribution rhosig[1, ] thetasig[1, ] rho distribution bin theta distribution bin Figure 4: Distribution of standard deviation of images of different categories. are used as features for each image. In Table 5, we include the average of the mean scores. Except for the category, the diagonals are much larger than the off-diagonals, indicating that these features have strong classification power. Average SURF similarity scores Average number of interest points Table 4: Finally, after we put all the features together in the random forest, we show the important features in Figure 5. The SURF similarity scores turn out to be very important, together with the ρ EMD to and signitures and number of corners. The Out of Bag error rate (with 1000 negative images) is 4.8% and the confusion matrix is included in Table?? background error rate background Table 5: Confusion matrix of random forest

6 CS221 Shaojie Deng and Ya Xu 6 Random Forest variable importance scoretop1 scoretop1 scoretop1 scoretop2 scoretop2 rhoemd numcorners rhoemd scoreave6 scoretop2 scoretop1 scoretop1 scoretop1 scoretop2 numkeytop1 scoretop2 scoretop2 numkeytop1 numkeytop1 scoretop MeanDecreaseAccuracy MeanDecreaseGini Figure 5: The important features used in random forest. 1.4 Tracking SURF can not only be used for object recognition, it can also be used for tracking. This is because it implicitely tracks an interest point in frame one by matching it with the same point in frame two. Because two consecutive frames in a video stream are almost identical, we can track the positive sub-windows found in an earlier frame by averaging the location shifts of the well matched interest points between the eariler frame and the current frame. Such SURF tracking works quite well for our purpose here. This is because the objects tracked are not moving themselves, it is the entire frame that is moving. Therefore, most of the interest points detected have similar movements and the average of them all gives a de-noised measurement of the movement of the entire frame (in x and y direction). Notice that SURF tracking does not work well with rotation or zoom in/out. However, this does not cause much problem for our purpose here, since we only track objects up to 10 frames and the movement induced by rotation or zoom in/out is negligible. xxxplease feel free to add more 2 Combining Components order of combination; why do we make such a decision how to deal with duplications; any rules; any assumptions speed concern and choices made accordingly

7 CS221 Shaojie Deng and Ya Xu 7 3 Experiments and Results 4 Comments Need more positive samples

SURF: Speeded Up Robust Features. CRV Tutorial Day 2010 David Chi Chung Tam Ryerson University

SURF: Speeded Up Robust Features CRV Tutorial Day 2010 David Chi Chung Tam Ryerson University Goals of SURF A fast interest point detector and descriptor Maintaining comparable performance with other detectors