CS231A Section 6: Problem Set 3 Kevin Wong Review 6 -! 1 11/09/2012
Announcements PS3 Due 2:15pm Tuesday, Nov 13 Extra Office Hours: Friday 6 8pm Huang Common Area, Basement Level. Review 6 -! 2
Topics Review Optical Flow PS3 Keypoints and features SIFT Descriptors SIFT Matching Projects OpenCV with Matlab Review 6 -! 3 11/09/2012
Review: Optical Flow Overview Review of Theory Applications Review 6 -! 4 11/09/2012
Optical Flow Learning OpenCV, page 322 Review 6 -! 5 11/09/2012
Optical Flow A low level motion tracking algorithm that tracks apparent motion from frame to frame of a video. Uses brightness patterns as features, not corners. Key Assumptions Brightness consistency: I(x, y, t 1) = I (x + u(x, y),y+ v(x, y),t) Review 6 -! 6 11/09/2012
Optical Flow Key Assumptions Cont. Spatial Coherence Neighboring points are from the same surface and will have similar motion. Small Motion The points do not move very far between frames * Image from Michael Black, CS143 2003 Review 6 -! 7 11/09/2012
Optical Flow I (x + u, y + v, t) I(x, y, t 1) + I x u + I y v + I t I (x + u, y + v, t) I(x, y, t 1) = I x +I y v + I t I x u + I y v + I t 0 5I [u v] T + I t =0 Taylor Expansion -> Brightness consistency We have one equation and two unknowns, motion perpendicular to the gradient, parallel to an edge is ambiguously, Aperture problem/barber pole illusion. Review 6 -! 8 11/09/2012
Review 6 -! 9 11/09/2012
Lucas-Kanade Flow Solvable when A T A is invertible Eigenvectors of A T A describe the patch Good patches turn out to be Harris corners. Improvements? Spatial consistency doesn t really hold across the scene. Solution? Motion segmentation: Divide the region into blocks, cluster pixel movement, reassign pixels to closets clusters and repeat. Review 6 -! 10 11/09/2012
Optical flow applications Object tracking via motion segmentation Traffic analysis Crowd flow/behavior analysis. Review 6 -! 11 11/09/2012
SIFT Motivations What makes a point interesting What to do with Keypoints SIFT detector SIFT descriptor Object Recognition with SIFT Review 6 -! 12
Where are we? CV Timeline Scene Object Local Patches Pixels Camera Models Review 6 -! 13
What to identify? (Keypoints) Salient points that would be present across a set of (reasonably) related images Repeatable Distinctive Review 6 -! 14
What information? (Descriptors) Review 6 -! 15
SIFT Ubiquitous: 10,000+ Citations David Lowe, 2001 and 2004, from UBC Both a detector and a descriptor Invariant to: Illumination Scale Rotation Affine Perspective Review 6 -! 16
SIFT: Keypoint Detector Difference of Gaussians using Scale Space Pyramid Section 3 and 4 of Lowe, 2004 = Review 6 -! 17
Extract and Prune max(dog) A point is extreme if it larger/smaller than its 26 neighboring points Prune for: Low Contrast Edge Points Review 6 -! 18
What can we do with keypoints? Review 6 -! 19
SIFT Descriptor Sections 5 and 6 in Lowe, 2004. Inspired by neurological research and models Keypoint is center of square patch of pixels, blurred at the scale of the keypoint Construction of Orientation Histogram for each 4x4 set of pixels All pixels are rotated by the orientation of the keypoint Review 6 -! 20
Steps of finding descriptor (for PS3) Given a keypoint P 1. Find gradient: 1. Apply Gaussian filter to the image using the scale of P as sigma (standard deviation) 2. Find gradient in x and y directions (down - up, right - left) 3. Magnitude: sqrt(ix 2 +Iy 2 ); Theta: atan2(iy, Ix) 4. Account for keypoint orientation. 2. Find the patch associated with P 3. For each bin b of the 4 x 4 bins in the patch 1. Initialize the histogram of the 8 angle bins 2. For each pixel p in b 1. Find the angle bin it falls into 2. In the histogram, increase the value of this angle bin by the gradient magnitude of p 3. Append this histogram to the end of the descriptor Review 6 -! 21
Many little details Many, many experimentally determined parameters for descriptor. The standard patch size is 16 Feature Vector = 16 patches * 8 bins = 128 elements Normalize each feature vector Avoid large (and potentially erroneous) gradients by capping each element at 0.2 Add extra blur for camera sensor [More steps ] We try to make the process as linear as possible Review 6 -! 22
The Matching Problem Locate arbitrary objects despite environmental difficulties Review 6 -! 23
Object Recognition with SIFT (Sec 7, Lowe 2004) Step 1: Feature matching Step 2: Hough transform in pose space Step 3: Geometric verification via affine transformation Review 6 -! 24
Step 1: Feature matching A match is determined by distance between closest neighbor and second closest neighbor Euclidean distance between descriptor vectors Nearest Neighbor problem k-d tree is inefficient Best-Bin-First (BBF) (Beis and Lowe, 1997), modified from k-d tree, giving approximated result Review 6 -! 25
Step 2: Hough transform in pose space Goal: given test image and training image, find object pose Input: Descriptor matches (p i -> p i ) of an object, many are false matches Output: estimated object pose A match is specified by 4 parameters <x, y, scale, orientation> Review 6 -! 26
Step 2: Hough transform in pose space 1. Discrete bins in 4D space; 2. Assign each match to the bin whose object pose is consistent with it; 3. Find bins with > 3 votes. Review 6 -! 27
Step 2: Hough transform in pose space Have to use broad bins since there are only 4 parameters but 6 dof bin size of 30 degrees for orientation, a factor of 2 for scale, etc. For the problem, you can use a uniform bins, which isn t optimal, good enough for the problem. Review 6 -! 28
Step 3: Geometric verification via affine transformation Verify each bin with at least 3 entries 3 matches determine an affine transformation (An approximation of finding the fundamental matrix which requires more matches) Review 6 -! 29
Parameters of Affine Transformation Affine transformation: Least-squares solution for the best affine projection parameters Review 6 -! 30
Results Review 6 -! 31
Object Matching in PS3 PS3 uses a simplified version of object matching Match on keypoints descriptors Use Hough to determine bounding box parameters. http://www.cs.washington.edu/homes/ankit/research_projects/sura/sura-website/sift.htm Review 6 -! 32
Projects OpenCV with Matlab http://xanthippi.ceid.upatras.gr/people/evangelidis/ matlab_opencv/ Guide to building OpenCV Mex files for windows that can be called in Matlab. Note that the Guide is for OpenCV 2.1. Macs: Tested using OSX 10.8, Xcode 4.5, OpenCV 2.4.2. Other options: Use files for I/O to standalone OpenCV applications Use OpenCV exclusively, with CGAL for potential matrix operations. Review 6 -! 33 11/09/2012