Probabilis)c Temporal Inference on Reconstructed 3D Scenes

Transcription

1 Probabilis)c Temporal Inference on Reconstructed 3D Scenes Grant Schindler Frank Dellaert Georgia Ins)tute of Technology

2 The World Changes Over Time How can we reason about )me in structure from mo)on problems?

3 Temporal Inference Problem 1. When was each photograph taken? 2. When did each building first appear? 3. When was each building removed? Set of Photographs: Set of Objects: Buildings 3

4 Temporal Inference Problem Temporal Inference Problem Given observa)ons Z and scene geometry X, what are the temporal parameters T? Structure (a, b) (x,y,z) Time Interval Posi)on Images Correspondences 3D Cameras t (x,y,z) (y,p,r) f Time Transla)on Rota)on Focal Length Structure from Mo?on 4

5 Overview Images Correspondences 3D Buildings Images + Time Structure + Time Intervals 4D 5

6 Overview Images Correspondences 3D Buildings Images + Time Structure + Time Intervals 4D 6

7 Overview Images Correspondences 3D Buildings Images + Time Structure + Time Intervals 4D 7

8 Overview Images Correspondences 3D Buildings Images + Time Structure + Time Intervals 4D

9 3D Reconstruc)on Images 3D Point Cloud Structure from Mo?on Bundler So\ware by Noah Snavely (Snavely et al SIGGRAPH 2006)

10 Grouping Points into Buildings 3D Point Cloud 3D Buildings Point Grouping Group points according to both: a. Distance < threshold in 3D b. Observed simultaneously in 1+ images Building Geometry 1. Convex hull of each group 2. Fit ground plane to camera centers 3. Extend convex hulls to ground

11 3D Reconstruc)on: Points vs. Objects Lower Manhaban 454 images 83,860 points 960 Buildings

12 Reasoning About Time: From Constraint Sa)sfac)on Schindler et al. CVPR Inferring Temporal Order of Images from 3D Structure. Key Idea: Visibility of 3D points constrains image ordering Disadvantage: Only rela)ve image ordering Disadvantage: Perfect observa)ons required rules out fully automa)c reconstruc)on, inherently noisy

13 to Probabilis)c Temporal Inference Maximize T = )me parameters Z = observa)ons X = scene geometry Observa?on Probability Visibility Reasoning Image Date Prior Noisy Building Observa)ons Circa 1910 Undated Uncertain Dates 13

14 P(T) Image Date Prior Date Sources: EXIF tags database annota)ons Undated: Otherwise: Uniform Historical dates o\en uncertain Es)mated Date Uncertainty May 1970 Undated Circa

15 Observa)on Probability Model Time Parameters Hypothesized Geometry Known Binary variable: Was object i observed in image j? X only Viewability: Is object i within the field of view of camera j? T only a t b Existence: Did object i exist at the )me image j was captured? X & T Occlusion: Is object i occluded by some other object(s) in image j? 15

16 Op)mizing Temporal Parameters Markov Chain Monte Carlo (MCMC) Sampling Propose to move an image date Analy)cally solve building date intervals Evaluate P(T Z,X) Accept or reject move 1941 Image # Time

17 Op)mizing Temporal Parameters Markov Chain Monte Carlo (MCMC) Sampling Propose to move an image date Analy)cally solve building date intervals Evaluate P(T Z,X) Accept or reject move 1941 Image # Time

18 Op)mizing Temporal Parameters Markov Chain Monte Carlo (MCMC) Sampling Propose to move an image date Analy)cally solve building date intervals Evaluate P(T Z,X) Accept or reject move 1941 Image # Time

19 Op)mizing Temporal Parameters Markov Chain Monte Carlo (MCMC) Sampling Propose to move an image date Analy)cally solve building date intervals Evaluate P(T Z,X) Accept or reject move 80,000 samples = 1.3 hours for scene with 100 images 1941 Image # Time

20 Results: Full Temporal Op)miza)on Synthe)c Scene 100 images, 30 buildings, 80 years 1/3 known date, 1/3 circa, 1/3 unknown Ini)al RMS Error: years RMS Error of Our Solu)on: 2.87 years Image # Ground Truth MCMC Samples 1930 Time

21 Results: Full Temporal Op)miza)on Downtown Atlanta (102 Images) Image # Given Date Estimates MCMC Samples Building Date Intervals 1956 Time 1975 Image Dates 21

22 Results: Full Temporal Op)miza)on Downtown Atlanta Original Date Label : Undated Image # Solu)on: Time

23 Results: Leave One Out Image Da)ng Lower Manhaban Error < 5 years for 48% images P(T Z,X) 23

24 Results: Building Date Intervals 24

25 Conclusions and Future Work Automa)c, probabilis)c temporal inference method Results for synthe)c scene and 2 challenging real data sets Future Work: Feature correspondence across )me challenging Find more data, densely sampled in space and )me Design )me invariant features 25

26 Thank you!