Revisiting 3D Geometric Models for Accurate Object Shape and Pose

Transcription

1 Revisiting 3D Geometric Models for Accurate Object Shape and Pose M. 1 Michael Stark 2,3 Bernt Schiele 3 Konrad Schindler 1 1 Photogrammetry and Remote Sensing Laboratory Swiss Federal Institute of Technology (ETH), Zurich 2 Artificial Intelligence Lab Stanford University, USA 3 Max-Planck-Institute for Informatics Saarbrücken, Germany

2 Current object models: coarse grained estimates 1

3 Our goal: finer-grained models to aid scene-level reasoning 2

4 Revival of 3D geometric representations 1970 [Marr, Nishihara 78] [Brooks 81] [Pentland 86] [Lowe 87] [Koller, Daniilidis, Nagel 93] [Sullivan, Worrall, Ferryman 95] [Haag, Nagel 99]

5 Revival of 3D geometric representations 1970 [Marr, Nishihara 78] [Brooks 81] [Pentland 86] [Lowe 87] [Koller, Daniilidis, Nagel 93] [Sullivan, Worrall, Ferryman 95] [Haag, Nagel 99] 2000 [Hoiem, Efros, Hebert 08] [Ess, Leibe, Schindler, Van Gool 09] [Wang, Gould, Koller 10] [Hedau, Hoiem, Forsyth 10] [Barinova, Lempitsky, Tretyak, Kohli 10] [Gupta, Efros, Hebert 10] [Wojek, Roth, Schindler, Schiele 10]

6 Related work in viewpoint invariant detection Multiple, viewpoint dependent representations (connected in different ways) [Thomas et al., 06] [Yan, Khan, Shah 07] [Ozuysal, Lepetit, Fua 09] [Nachimson, Basri 09] [Su, Sun, Fei-Fei, Savarese 09] [Gu, Ren 10] [Stark, Goesele, Schiele 10] 1) 1) 2) Explicit 3D geometry representation [Liebelt, Schmid 10] 2) [Sun, Xu, Bradski, Savarese 10] [Gupta, Efros, Hebert 10] [Chen, Kim, Cipolla 10] [Gupta, Satkin, Efros, Hebert 11] 4

7 Overview Simplify 3D Active Shape Model PCA 3D CAD Models 5

8 Overview Simplify 3D Active Shape Model PCA 3D CAD Models Render Positive examples (per part) 5

9 Overview Simplify 3D Active Shape Model PCA 3D CAD Models Render Positive examples (per part) AdaBoost Negative examples (background) 5

10 Overview Simplify 3D Active Shape Model PCA 3D CAD Models Render Positive examples (per part) AdaBoost Negative examples (background) Detection maps Test image 5

11 Overview Simplify 3D Active Shape Model PCA Inference 3D CAD Models Render Positive examples (per part) AdaBoost Negative examples (background) Detection maps Test image 5

12 Representation: 3D geometry Simplified 3D wireframes : fixed number of vertices 6

13 Learning: 3D geometry Eigen-Cars Principal Components Analysis (PCA) Tightly constrained global geometry 7

14 Representation: Local appearance Accurate foreground shape Very cheap training data, dense sampling of viewpoints! 8

15 Learning: Local appearance Dense Shape Context features [Belongie, Malik. 00] AdaBoost classifiers (per part-viewpoint) + - Annotated vertices are our parts. Related work: [Andriluka, Roth, Schiele 09] 9

16 Inference Test Image 10

17 Inference Test Image Detection maps 10

18 Inference Test Image Detection maps Sample 3D wireframes, project, compute image likelihood 10

19 Inference Detection maps Sample 3D cars, project, compute image likelihood image evidence shape of wireframe camera focal length recognition hypothesis viewpoint parameters, azimuth and elevation image space translation and scaling Projection matrix local part scale part likelihood self-occlusion indicator 11

20 Experimental evaluation Test Dataset Evaluations on 3D Object Classes dataset [Savarese et al., 2007] Car class (8 azimuth angles, 2 elevation angles, 3 distances, varying backgrounds) 240 images, 5 cars 12

21 Experimental evaluation - Training 38 3D CAD models 36 vertices as model points, 20 annotations per model (due to symmetry). Separate local part shape detectors trained from: - 72 different azimuth angles, - 2 different elevation angles (7.5, 15 from ground plane) 13

22 Experimental evaluation - Initialization Two initializations : 20 Stark et al., 2010 (full system) True initial value (tight bounding box, rough azimuth) 14

23 Experimental evaluation - Inference

24 Example wireframe fits Parts correctly localized Full system: 74.2% True initial value: 83.4% 15

25 Fine-grained 3D geometry estimation Accurate estimation of closest 3D CAD model, camera parameters, and ground plane 16

26 Ultra-wide baseline matching UW-Baseline matching using only model fits (corresponding part locations) Impossible using interest point matching Related work: [Bao, Savarese 11] 17

27 Ultra-wide baseline matching UW-Baseline matching using only model fits (corresponding part locations) Impossible using interest point matching Related work: [Bao, Savarese 11] 18

28 Ultra-wide baseline matching Azimuth Difference No. of Image Pairs True initial value Full system SIFT Part detections only % 55% 2% 27% % 60% 0% 27% % 52% 0% 10% % 41% 0% 24% Correct fit = Sampson error < E max on ground truth correspondences 3D Geometric model improves significantly over part detections only 19

29 Multiview recognition Rescored hypotheses Good 2D localization 20

30 Continuous viewpoint estimation Total Images True Positives % correct azimuth Average error azimuth Average error elevation Stark et al., % Full system % True initial value* % Comparison against ground truth pose, manually labeled. Full system improves 6% over Stark et al., * Approximate pose initialization quantized to 45 steps 21

31 Conclusion 3D deformable object class model have potential for accurate geometric reasoning on scene level. - accurate object localization - geometric parts in 2D - 3D pose estimation Novel application examples - fine-grained object categorization - ultra-wide baseline matching Future extensions - efficient multi-class methods for part likelihoods - analyze importance of geometric model vs. local appearance - occlusion invariance 22

32 OLD SLIDES

33 Learning: 3D Geometry any wireframe mean wireframe weight of k th principal component standard deviation of j th principal component direction of j th principal Eigen-Cars component residual (if r < m)

34 Part localization correct localization ~ localized within 4% of car length from ground truth

35 Experimental evaluation - Inference

36 Experimental evaluation - Inference