User-assisted Segmentation and 3D Motion Tracking. Michael Fleder Sudeep Pillai Jeremy Scott

Transcription

1 User-assisted Segmentation and 3D Motion Tracking Michael Fleder Sudeep Pillai Jeremy Scott

2 3D Object Tracking Virtual reality and animation Imitation in robotics Autonomous driving Augmented reality

3 Motivation Want low-cost tracking with no instrumentation

4 Method Overview User-assisted Segmentation

5 Method Overview Iterative Closest Point (ICP) for 3D tracking

6 Method Overview Color Histogram 2D tracking to seed ICP

7 DEMO

8 Preliminary Experiments / Results FAST features for saliency detection ORB / SURF / SIFT feature descriptor extractor SURF/ SIFT Robust Lacking real-time capabilities ORB (Oriented FAST and Rotated BRIEF) Runs in real-time Produces a lot of false positives

9 Preliminary Experiments / Results DEMO

10 Preliminary Experiments / Results Limitations Kinect RGB Sensor Low resolution Signal-to-noise ratio not particularly good Feature detection Doesn't cope well with motion blur Actively learn new features of the object Correspondence requires rigid object tracking

11 Appearance-based tracking Appearance model + Temporal coherence Histogram appearance models Base color representative over multiple views Robust to perspective views Don't have to actively learn new features Sufficient to provide object localization belief Relatively faster to compute Mean-Shift tracking Robust to motion blur Eliminates false positives that have same color distribution

12 Two approaches Create a likelihood image, with pixels weighted by similarity to the desired color Best for uni-colored objects Fails when there is a color distribution Represent color distribution with a histogram Use mean-shift to find region that has most similar distribution of colors

13 Comparing color distributions Bhattacharya Distance/Coefficient Measure of the amount of overlap between two statistical samples i.e. between histograms Imposes a metric structure Invariant to object size (number of pixels) Valid for arbitrary distributions

14 Mean-Shift Algorithm Finding modes in a set of data samples, manifesting an underlying PDF in R N Feature space can be color, scale or saliency

15 Mean-Shift Algorithm

16 Mean-Shift Algorithm

17 Mean-Shift Algorithm

18 Mean-Shift Object Tracking Target Representation

19 Mean-Shift Object Tracking PDF Representation

20 First Attempt: Tracking using 3D Only (No RGB) Iterative Closest Point Given: Two point-sets A (object to track), B (new point cloud) R 3 Goal: Find a one-to-one matching function μ : A B that minimizes the root mean squared distance (RMSD) between A and B Incorporating rotation R and translation t:

21 First Attempt: Tracking using 3D Only (No RGB) Iterative Closest Point ICP(A,B) : //A,B point sets. R = rotation matrix. t = translation vector. //Find μ one-to-one matching function. 1. Initialize R = I (the identity matrix), t = Matching Step: Given R and t, compute optimal μ by finding min μ RMSD(A, B, μ). 3. Transformation Step: Given μ, compute optimal R and t by finding min R,t RMSD(RA t, B, μ). 4. Go to step 2 unless μ is unchanged.

22 First Attempt: Tracking using 3D Only (No RGB) Iterative Closest Point Problems: (1) Can t handle large displacement between frames (2) Computationally intense (3) Ambiguous for geometrically-symmetric objects Benefits: (1) Resolve 3D orientation (2) Accurate for small-displacements

23 Third Pass: Tracking using 2D and Tracking Benefits: 3D 2D (Histogram): (1) Handles large displacements between frames (2) Accurate small-patch tracking (3) Fast 3D (ICP): (1) Accurate for small-displacements (2) Resolves 3D orientation Combined: Seed 3D ICP with result of 2D tracking

24 Third Pass: Tracking using 2D and Initialize_Tracker(Segmented_Object, out trackedobject): (1) 2D_Segment_Points 3dTo2D(Segmented_Object) (2) Init_Histogram_Tracker(2D_Segment_Points) (3) trackedobject Segmented_Object 3D Run_Tracker(lastKnownObject3D, latest_image, latest_cloud): (1) 2D_Points = run2dtracker(latest_image) (2) 3D_Search_Cloud = 2dTo3d(2D_Points) (3) lastknownobject3d ICP(lastKnownObject3D, 3D_Search_Cloud)

25 Summary Enabling low-cost 3D object tracking on the Kinect via: 1. User-assisted segmentation 2. Combining color histogram tracker in 2D and ICP in 3D 3. Modifications to enable real-time tracking