16720 Computer Vision: Homework 3 Template Tracking and Layered Motion.

Transcription

1 16720 Computer Vision: Homework 3 Template Tracking and Layered Motion. Instructor: Martial Hebert TAs: Varun Ramakrishna and Tomas Simon Due Date: October 24 th, Instructions You should submit your completed homework by placing all your files in the directory named after your andrew id and upload it to your submission directory on Blackboard. The submission location is the same section where you downloaded your assignment. Please include a line in your write-up indicating how many hours you spent on this homework. Do not include the handout materials in your submission. All questions marked with a Q require a submission. For the implementation part of the homework please stick to the function headers described. Final reminder: Start early! Start early! Start early!. This homework contains mostly implementation questions which can get tricky and could take a while to debug. 2 Lucas-Kanade Tracking The first groundbreaking work on template tracking was the Lucas-Kanade tracker 1. It basically assumes that the template undergoes constant motion in a small region. The Lucas-Kanade Tracker works on two frames at a time, and does not assume any statistical motion model throughout the sequence. The algorithm estimates the deformations between two image frames under the assumption that the intensity of the objects has not changed significantly between the two frames. A good reference for the Lucas-Kanade tracker is given below. See Figure 2 in the above paper for a good overview of the algorithm. Lucas-Kanade 20 Years On - Baker, S. and Matthews, I. view.html?pub_id= Developed here in CMU! 1

2 Starting with a rectangle R t on frame I t, the Lucas-Kanade Tracker aims to move it by an offset (u, v) to obtain another rectangle R t+1 on frame I t+1, so that the pixel squared difference in the two rectangles is minimized: minj(u, v) = u,v (x,y) R t (I t+1 (x + u, y + v) I t (x, y)) 2 (1) Starting with a initial guess of (u, v) (usually (0, 0)), we can compute the optimal (u, v ) iteratively. In each iteration, the objective function is locally linearized by first-order Taylor expansion and optimized by solving a linear system. Q1. (15 points) Upload the following function in your submission directory: [u,v] = LucasKanade(It,It1,rect) that computes the optimal local motion from frame I t to frame I t+1 that minimizes Eqn. 1. Here It is the image frame I t, It1 is the image frame I t+1, and rect is the 4-by-1 vector representing a rectangle on the image frame I t. The four components of the rectangle are [x1, y1, x2, y2], where (x1, y1) is the top-left corner and (x2, y2) is the bottom-right corner. The rectangle is inclusive, i.e., it includes all the four corners. To deal with fractional movement of the template, you will need to interpolate the image using the MATLAB command interp2. You will also need to iterate the estimation until the change in (u, v) is below a threshold. Q2. (15pts) Load the video sequence (carsequence.mat) and run the Lucas-Kanade Tracker to track the speeding car. The rectangle in the first frame is [x1, y1, x2, y2] = [328, 213, 419, 265]. In otherwords, the rectangle starts from (328, 213) (row 213 and column 328 in the image) and ends at (419, 265). In your answer sheet, print the tracking result (image + rectangle) at frames 20, frame 50 and frame 100. What are the possible reasons the tracker could fail? How could you possibly fix some of these problems? 3 Tracking with Dominant Motion Estimation In this section, you will implement a tracker for estimating dominant affine motion in a sequence of images and subsequently identify pixels corresponding to moving objects in the scene. You will be using the images in the file Sequence1.tar.gz consisting of aerial views of moving vehicles from a non-stationary camera. 3.1 Dominant Motion Estimation You will start by implementing a tracker for affine motion using the equations for affine flow described in class and in the Motion slides from class. As in the previous section, the tracker will only work on two frames at a time. To estimate dominant motion, the entire image I(t) will serve as the template to be tracked in image I(t+1), i.e. I(t+1) is assumed to be approximately an affine warped version of I(t). This approach is reasonable under the assumption that a majority of the pixels correspond to stationary objects in the scene whose depth variation is small 2

3 relative to their distance from the camera. Using the equations for the affine model of flow, you can recover the 6-vector [ a b c d e f ]T of affine flow parameters. They will be related to the equivalent affine transformation matrix as: M = 1 + a b c d 1 + e f (2) relating the homogenous image coordinates of I(t) to I(t + 1) according to x t+1 = Mx t t (where x = [ x y 1 ] T ). For the next pair of consecutive images in the sequence, image I(t + 1) will serve as thetemplate to be tracked in image I(t + 2), and so on through the rest of the sequence. Note that M will differ between successive image pairs. Each update of the affine parameter vector, p is computed via a least squares method using the pseudo- inverse as described in class: p = (A T A) 1 A T I t where the matrix A of image derivatives is defined as A = [ xi x yi x I x xi y yi y I y ] Note: Unlike previous examples where the template to be tracked is usually small in comparison with the size of the image, image I(t) will almost always not be contained fully in the warped version of I(t + 1). Hence the matrix of image derivatives, A, and the temporal derivatives, I t, must be computed only on the pixels lying in the region common to I(t) and the warped version of I(t + 1) to be meaningful. 3.2 Moving Object Detection Once you are able to compute the transformation matrix M relating an image pair I(t) and I(t + 1), a naive way of determining pixels lying on moving objects is as follows. Warp the image I(t) using M so that it is registered to I(t + 1), and subtract it from I(t + 1). The locations where the absolute difference exceeds a threshold can then be declared as corresponding to locations of moving objects. For better results, hysteresis thresholding may be used. We have provided MATLAB code for hysteresis thresholding (hysthresh). Q3 (30 Points) Using the above method for dominant motion estimation, write a MATLAB function [moving image] = SubtractDominantMotion(image1, image2) where image1 and image2 (in uint8 format) form the input image pair and moving image is a binary image of the same size as the input images, in which the non-zero pixels correspond to locations of moving objects. The script test motion.m that we will use for grading this section has been provided to you for testing your function. This 3

4 script simply makes repeated calls to SubtractDominantMotion on every consecutive image pair in the file Sequence1.tar.gz, makes an AVI movie out of the moving image returned for every image pair processed, and saves it as a file motion.avi for your offline viewing pleasure. An example of such a movie is also provided consisting of results using the methods described above. You will notice that the estimates of moving pixels are a bit noisy. 2 We do not expect perfect results this is, after all, real world data! You are free to implement any algorithm of your choice for estimating the moving image following the affine registration step 3. We will be awarding Extra Credit(10 Pts) to the best looking results as judged by the instructors. 4 Motion Layers (a) (b) Figure 1: (a) Optical flow between the first two frames. (b) Computed motion segmentation Optical flow is the deformation undergone by each pixel in the current frame from the previous frame. Optical flow has found varied application in computer vision and in this question we will explore the task of extracting motion layers. In many situations, motion in the image is caused by the movements of objects at different depths in the scene. In such situations, the image sequence can be represented compactly if pixels are grouped into appropriate objects or layers. Segmentation of a scene into foreground objects and background is crucial for scene understanding and recognition as we will see later in the course. To extract motion layers we will mainly be following the method outlined in the following paper: Representing Moving Images with Layers - Wang, J. Y. A., and Adelson, E. H., IEEE Transactions on Image Processing, 3(5): , wang_tr279.pdf 2 For de-noising your output, you may also want to look at the imerode and imdilate morphological operators in MATLAB. 3 A basic version of this would involve computing the difference between the warped version of the first image and the second image and thresholding. 4

5 You will need to read sections 4 and 5 of the above paper. The method proceeds by generating hypotheses by fitting affine motion models in overlapping square regions to the optical flow. Regions are then merged using a clustering algorithm and each pixel is assigned a region satsifying a minimum distortion criterion. Regions that are spatially disconnected are split and small regions are filtered out. The process is iterated until the pixel memberships do not change. We have provided you with precomputed flow in the folder /flow/ in the form of.mat files. Each.mat file contains two variables vx and vy which contain the x-component and the y-component respectively of the flow for each pixel location. We have also provided you with a function viewflow to visualize the optical flow. 4.1 Generating hypotheses Q4. (20 Pts) Read section 4 of the paper and upload the following function in your submission directory [A] = generateaffinehypotheses(vx, vy, I) that generates affine parameter hypotheses A (a Nx6 matrix where N is the number of hypotheses) for the regions specified in the matrix I which is a mask of indices which is the same size as the image. In the first iteration of the algorithm, I will be empty, during this iteration you will generate hypotheses by dividing the image into small square regions and compute the affine warp parameters for each of the regions. IMPORTANT: This can be very slow if you do not use a vectorized implementation and convolutions. In subsequent iterations you will proceed by computing the affine warps for each of the regions indexed by I. 4.2 Clustering hypotheses To merge regions, we cluster the hypothesized affine warps as computed in 4.1. Q5. (10 Pts) Write and upload the function [C] = mergeregions(a,i,distancethreshold) This function clusters the hypotheses and returns the cluster centres in the matrix C (NumClusters x 6) and the assigned indices for the hypotheses in I. To perform the adaptive k-means as described in the paper, we provide you with a simple version which performs k-means and as a post-processing step, merges clusters if the distance between the clusters is less than distancethreshold. As an alternative, you could also use MATLAB s clusterdata function. 4.3 Assigning Regions We now associate each pixel location with a cluster centre. Follow the method outlined in the paper in section 5.3, and assign each pixel to the affine model which minimizes the distortion at that pixel location. Q6. (20 Pts) Create and upload the function 5

6 4.4 Splitting and filtering [I]= assignregions(vx,vy,c) As described in the paper, regions that are spatially disconnected can get assigned to the same region during the clustering step. We also want to get rid of small regions which can throw off the estimation of the affine warps. We post process the segmentation by separating the regions that are disconnected and filtering out small regions. Q7. (10 Pts) Create and upload the function: [I,C] = splitandfilter(i,c) where I and C are the index mask and cluster centres respectively. The function should recompute the index mask and return the new index mask and new set of cluster centres. For this function you might want to use some of MATLAB s image processing tools such as bwconncomp and regionprops. The goal here is to familiarize you with some of the post-processing that is required in many computer vision applications but is usually not taught in detail. 4.5 Iterative Motion Segmentation Now apply the above procedure iteratively in a loop until the pixel-region memberships do not change. 4 Q8. (15 Pts) Create and upload a function [I, C] = motionsegment(vx,vy) where vx and vy are the optical flow pre-computed for a pair of frames and I is the matrix of pixel-region memberships. Compute the motion segmentation for the flow corresponding to the first two frames of the sequence given in the folder and visualize your I matrix using the imagesc function. 4.6 Extra Credit: Video Segmentation So far we estimated the motion segmentation for a pair of frames. To obtain a consistent segmentation of an entire video sequence would require you to associate the generated segments across the sequence. Estimate the motion segmentation for each pair of frames in the video. Associate the segments across the frames in the sequence by assigning the segments to the same id in the first frame by comparing the respective motion models 5, warping and checking overlap consistency etc. QX1. (20 Pts) Write and upload the script motionlayers.m which reads in the flow for each frame sequentially, performs motion segmentation and associates segments across frames. Upload a video of your motion segmentation. 4 In practice 20 iterations were enough for convergence. 5 Euclidean distance betwen the affine warp parameters 6

7 5 Implementation hints 5.1 Lucas-Kanade Tracking Please make a.mat file which has top-left and bottom-right corners of your tracked object for us to evaluate your tracking result. The.MAT file should contain a variable called TrackedObject which is a nx4 matrix. n is number of frames in CarSequence and each row in the matrix contains 4 numbers: x1 y1 x2 y2, where indicates the coordinates of top-left and bottom-right corners to the object you tracked. Please name the.mat file as track coordinates.mat. Recall that determining the motion vector (u, v) for each frame is an iterative procedure and should be repeated until convergence which means u and v are close to zero. You may want to set a maximum number of iterations to try in case your template is lost between frames. Because the car will not be moving by integer amounts, you will need to interpolate values of the image or its gradient within the moving template windows. Using interp2 for this can be helpful. The meshgrid function will likely be useful for creating arrays of (x, y) coordinates for all the pixels within a car s window. Because of the necessity of iterating over image frames, and iterating until convergence, efficiency will be important to prevent your code from running painfully slowly. Please try to vectorizing your image operations. 5.2 Dominant Motion Estimation Scaling image coordinates can be crucial for this type of problem. You should scale coordinates to be between 0 and 1 when computing the A matrix. But be sure to undo that scaling whenever you need to use coordinates to actually reference pixels in an image. Also note that Ix and Iy will vary depending on your scaling. Note that transforming by an affine matrix M1 and then by M2 is the same as transforming by M12 = M2 M1. It may be easier to keep track of the total accumulated warp (e.g. M12 here) as you iterate towards convergence, and warp the original I(t + 1) using that cumulative warp at each iteration. As before, note that the spatial derivatives remain the same between iterations when converging to the correct affine parameters, p. The only thing that changes will be It as you warp I(t + 1) more and more to better match I(t). Related to the previous point, the matrix A contains one row for every pixel in the image. But as discussed above, when we compute the temporal derivative, It, we should only consider those pixels of the warped version of I(t + 1) that overlap with I(t). Thus, at a given iteration, we need to ignore some of the rows 7

8 of A specifically those that correspond to pixels that do not overlap with the current warped version of I(t + 1). Note that this still does not require completely rebuilding A. Instead, you only need to keep track of which of its rows are currently active and index those accordingly when computing p. 5.3 Motion Layers As in the previous problem you will need to scale your co-ordinates to be between 0 and 1. You do not need to undo your scaling as long as you are consistent with scaling when you perform the pixel assignment step. The MATLAB representation NaN and the function isnan are useful for filtering out pixel locations you do not want to consider. While estimating affine models in your first iteration, some of the linear systems that you solve might be singular or have a large condition number. You will have to filter out these hypotheses. You might need to experiment with different values of distancethreshold in order to correctly merge hypotheses. 8