Feature Trajectory Retrieval with Application to Accurate Structure and Motion Recovery

Transcription

1 Feature Trajectory Retrieval with Application to Accurate Structure and Motion Recovery Kai Cordes, Oliver M uller, Bodo Rosenhahn, J orn Ostermann Institut f ur Informationsverarbeitung Leibniz Universit at Hannover 7th International Symposium on Visual Computing

2 camera motion Motivation Structure and Motion Estimation I k structure of rigid objects line of sight corresponding feature points p j,k p j,k+1 I k+1 3D object points P j I k+2 I k+3 Feature detection Correspondence analysis Outlier detection Bundle adjustment 2

3 Motivation Structure and Motion Estimation 3D object points Pj structure of rigid objects foreground occlusion ht e o lin I k I k +3 corresponding feature points Ik+1 I k+2 Error-prone object points Drift pj,k pj,k+1 came Occlusion Trajectories lost ig fs ra m otion 2

4 Motivation Reconstruction Accuracy High-Accuracy needed for Structure and Motion Estimation Reprojection Errors of 1/4 pel are visible and disturbing a Need wide baseline correspondence analysis SIFT a Hillman et al., ICIP 2010 Limited Localization Accuracy SIFT designed for Object Recognition Due to more invariance, the detector looses Accuracy Increase localization accuracy of SIFT detector a a Cordes et al., ISVC 2010, LNCS

5 Motivation Objectives Solve occlusion problem Store discontinued trajectories Use SIFT for robust assignment of newly appearing features Increase localization accuracy of SIFT Use gradient signal approximation for subpel localization 4

6 Outline Bivariate Feature Localization Assuming a Gaussian Shape Feature Trajectory Retrieval Experimental Results Conclusion 5

7 Bivariate Feature Localization Assuming a Gaussian Shape Scale Invariant Feature Transform Image y x Detection of Scale-Space Extrema Ui wfit Localization of Features 6 4 Orientation Assignment n ue Descriptor Calculation Image Features 6

8 Bivariate Feature Localization Assuming a Gaussian Shape f p (x) = v Σ e 1 2 ((x x 0) Σ 1 (x x 0 )) (1) f p New Localization Strategy: Exchange parabolic approximation of DoG by Gaussian function ( ) a 2 b Covariance matrix Σ = b c 2 rotated, scaled ellipse x = (x 0, y 0 ) subpel position 7

9 Feature Trajectory Retrieval Outline Bivariate Feature Localization Assuming a Gaussian Shape Feature Trajectory Retrieval Experimental Results Conclusion 8

10 Feature Trajectory Retrieval Feature tracking situation. p j,k t j I K L 1 I K L I K 1 I K Objective: Additional constraints in bundle adjustment: Memory for discontinued trajectories use SIFT descriptor for assignment use RANSAC and epipolar geometry for outlier elimination 9

11 Feature Trajectory Retrieval Bundle Adjustment Bundle Adjustment Equation ɛ = J K d(p j,k, A k P j ) 2 MIN (2) j=1 k=1 RMSE ɛ = ɛ 2JK reprojection error (RootMeanSquareError). 10

12 Feature Trajectory Retrieval Bundle Adjustment Bundle Adjustment Equation ɛ = J K d(p j,k, A k P j ) 2 MIN (2) j=1 k=1 RMSE ɛ = ɛ 2JK Extend Bundle Adjustment Equation reprojection error (RootMeanSquareError). use correspondences from non-consecutive s assign detected features to already reconstructed object points RMSE increases for FTR because of additional constraints in the BA 10

13 Experimental Results Outline Bivariate Feature Localization Assuming a Gaussian Shape Feature Trajectory Retrieval Experimental Results Conclusion 11

14 Experimental Results object points object points per RMSE t 1 t 2 t 1 t t 1 t t 1 t Bellevue Sequence temporary occlusions 12

15 Experimental Results object points object points per RMSE t 1 t 2 t 1 t t 1 t t 1 t FTR FTR Bellevue Sequence temporary occlusions less object pts for FTR more object pts for Gauss SIFT 12

16 Experimental Results object points object points per RMSE t 1 t 2 t 1 t t 1 t t 1 t FTR ref Bellevue Sequence temporary occlusions less object pts for FTR more object pts for Gauss SIFT RMSE(FTR) > RMSE(ref) RMSE(Gauss SIFT) < RMSE(SIFT) 12

17 Experimental Results object points object points per RMSE t Lift Sequence temporary Occlusion reference fails 13

18 object points object points per RMSE Experimental Results t Lift Sequence temporary Occlusion reference fails more object pts for Gauss SIFT 13

19 Experimental Results object points object points per RMSE t Lift Sequence temporary Occlusion reference fails more object pts for Gauss SIFT RMSE(Gauss SIFT) < RMSE(SIFT) 13

20 Experimental Results object points object points per RMSE t Lift Sequence temporary Occlusion reference fails more object pts for Gauss SIFT RMSE(Gauss SIFT) < RMSE(SIFT) RMSE(FTR) > RMSE(ref) 13

21 Experimental Results Example: Lift sequence 1 1. Input sequence (15 fps) 2. Result: 3. Result: 4. Result: Gaussian

22 Conclusion Summary: Extended Structure and Motion Recovery Increase localization accuracy of SIFT features Use feature correspondences in non-consecutive s (FTR) 15

23 Conclusion Summary: Extended Structure and Motion Recovery Increase localization accuracy of SIFT features Use feature correspondences in non-consecutive s (FTR) Detailed analysis of results: Longer trajectories, more constraints RMSE increases, but reconstruction improves Validated by integrating virtual objects to scene 15

24 Conclusion Summary: Extended Structure and Motion Recovery Increase localization accuracy of SIFT features Use feature correspondences in non-consecutive s (FTR) Detailed analysis of results: Longer trajectories, more constraints RMSE increases, but reconstruction improves Validated by integrating virtual objects to scene Future Work: Exploit occlusion image locations for further scene understanding