Joint Vanishing Point Extraction and Tracking. 9. June 2015 CVPR 2015 Till Kroeger, Dengxin Dai, Luc Van Gool, Computer Vision ETH Zürich

Transcription

1 Joint Vanishing Point Extraction and Tracking 9. June 2015 CVPR 2015 Till Kroeger, Dengxin Dai, Luc Van Gool, Computer Vision ETH Zürich

2 Definition: Vanishing Point = Intersection of 2D line segments, imaged from scene-parallel 3D line segments 2D Vanishing Point + known internal camera calibration = 3D unit bearing vector ( Vanishing Direction ) Applications: 3D reconstruction, autonomous navigation, camera calibration, camera pose estimation 2

3 Problem Find multiple Vanishing Points in an image sequence. (Unknown camera pose, known camera calibration) Why jointly? 1) Better Accuracy, 2) Temporal Regularization 3

4 Related Work Single View Multi-View / Multi-Frame Xu, et al. A Minimum Error Vanishing Point Detection Approach for uncalibrated Monocular Images of Manmade Environments. CVPR, Bazin, et al. 3-line RANSAC for Orthogonal Vanishing Point Detection. IROS, Lezama, et al. Finding Vanishing Points via Point Alignments in Image Primal and Dual Domains. CVPR, Tretyak, et al. Geometric Image Parsing in Man-Made Environments. IJCV, Antunes, et al. A Global Approach for the Detection of Vanishing Points and Mutually Orthogonal Vanishing Directions. CVPR Straub, et al. A Mixture of Manhattan Frames : Beyond the Manhattan World. CVPR, Tardif. Non-Iterative Approach for Fast and Accurate Vanishing Point Detection. ICCV, Using pose information Antone, et al. Automatic Recovery of Relative Camera Rotations for Urban Scenes. CVPR, Hornacek, et al. Extracting Vanishing Points across Multiple Views. CVPR, Kim, et al. Planar Structures from Line Correspondences in a Manhattan World. ACCV, Separate detection & association Elloumi, et al. Tracking Orthogonal Vanishing Points in Video Sequences for a Reliable Camera Orientation in Manhattan World. CISP, Moghadam, et al. Road Direction Detection Based on Vanishing-Point Tracking. IROS, Rasmussen. RoadCompass: following rural roads with vision + ladar using vanishing point tracking. Autonomous Robots, Lee, Yoon, CVPR We are the first to introduce joint multi-frame VP detection and tracking. 4

5 Vanishing Point Extraction on Gaussian Sphere (Barnard, 1982) Each 2D line segment = Plane through camera center ( Interpretation plane ) Intersection of interpretation planes = Vanishing Direction Noisy line segments : Use Least-Squares Solution for Vanishing Direction 5

6 Probabilistic Occupancy Sphere Discretization into 5120 bins, subdivisions of icosahedron. EXAMPLE: Each line (= interpretation plane) votes for ALL intersected bins. 6

7 Line Intersection Ambiguity High density of line intersection, but no VP present Due to line intersection ambiguities, we keep line-bin associations as free variables! 7

8 Solution using Linear Programming 1) Compute line segments in each frame (LSD lines: Von Gioi, Jakubowicz, Morel, G. Randall. LSD: a Line Segment Detector. IPOL, 2012.) 2) Compute a) Line-VP association probabilities b) VP Transition probabilities 3) Model problem as directed acyclic graph ( min-cost flow network ) 4) Extract VP tracks in graph by Linear Programming (LP) Inspired by network-flow tracking problem: Berclaz, et al. Multiple Object Tracking using K-Shortest Paths Optimization. PAMI,

9 Solution using Linear Programming cont d Synthetic example: 4 line segments, 1 VP in three frames. We model the problem as graph of all possible trajectories. 9

10 Solution using Linear Programming cont d VP Transition cost: Cost based on enclosing angle. 10

11 Solution using Linear Programming cont d Source : link to all possible VP locations. Uniform cost. Terminal : link from all possible VP locations. Uniform cost. 11

12 Solution using Linear Programming cont d From each VP location: Line-VP consistency score. Only a subset of all graph edges is shown. 12

13 Solution using Linear Programming cont d Following Berclaz2011: Find non-overlapping paths with minimum cost! Use (boolean) Integer Linear Programming 13

14 Solution using Linear Programming cont d LP: maximize linear objective function: Sum of : Line-VP consistency scores, [-inf, +inf] Start/Terminal Costs [-inf, 0] Transition Costs [-inf, 0] Constraints: Flow Conservation: maximally one track per VP bin. Line-VP association: A line can at most be linked to one (active) VP. Non-Maximum Suppression. Angle preservation: For each two VP tracks, enforce angular constancy. Orthogonality (optional). 14

15 Experiments New dataset for Joint VP detection and Tracking 48 sequences * 301 frames, 10 fps, street-view, 1-3 VPs. City scenes, vegetation, street furniture, street traffic Street view dataset augmented with Vanishing Point tracks Comparison 1) Proposed method 2) Our Method, frame-wise VP extraction, greedy tracking 3) Tardif2009, frame-wise VP extraction, greedy tracking (Tardif. Non-Iterative Approach for Fast and Accurate Vanishing Point Detection. ICCV, 2009.) 15

16 Experiment (1) Unknown Pose Metrics: MOTA (Multi Object Tracking Accuracy) IDS (Identity Switches) Precision/Recall (-> Suppl. Mat) Results: Best result: LPMF (ours) (Low IDS, highest MOTA score) Best baselines: LPSF, TNO.4 f (Tardif, Non-orthogonal, 40% downscaling, remove short tracks) Greedy tracking often fails (high IDS) Strongest negative influence on all methods: weak line support. 16

17 Experiment (2) Known Pose Vanishing Directions are static. Given camera orientations, VPs should be constant over time. We evaluate how MOTA changes when including pose knowledge. Removing pose knowledge leads to: Baseline degrades significantly LPMF (our method) remains unchanged Reason: smooth transition scores -> temporal regularization LPMF still outperforms the baseline 17

18 Summary First method for joint VP extraction and tracking in image sequences with unknown pose. New dataset for VP extraction and tracking (available online) Evaluation : 1. Comparison against sequential VP extraction + greedy tracking 2. Inclusion of known camera pose 3. (-> Paper) Single-view orthogonal VP extraction experiment Future work: Inclusion of priors (e.g. known gravity direction) More discriminative VP consistency measures Extensions to non-spherical VP parametrizations (if non-parametric prob. density over VP locations is available) 18

19 Thank you for your attention! Exemplary good results: Exemplary failure cases: 19

20 BACKUP: Implementation details Pruning, Implementation details VP location pruning: only half the sphere, Remove bins with low probability Hough grouping of line segments Batch processing of 30 frames using CPLEX Runtime: 2-3 seconds per frame. (~ 1 sec. for line detection + grouping) 20

21 BACKUP: MTT on Probabilistic Occupancy Grid 1. Gather occupancy evidence on Grid. 2. Model set of possible trajectories in graph. 3. Solve for trajectories through grid cells. Figure from: Berclaz, Fleuret, Turetken, Fua. Multiple Object Tracking using K-Shortest PathsOptimization. PAMI,

22 BACKUP: Combining two fields for VP extraction Vanishing Point extraction on Gaussian sphere Multi-Target Tracking with Probabilistic Occupancy Model + Barnard. Interpreting Perspective Images. Artificial Intelligence, Berclaz, Fleuret, Turetken, Fua. Multiple Object Tracking using K-Shortest Paths Optimization. PAMI,

23 BACKUP: Precision / Recall VP Detection 9. June