Large Scale 3D Reconstruction (50 mins) Yasutaka Washington University in St. Louis

Transcription

1 Large Scale 3D Reconstruction (50 mins) Yasutaka Washington University in St. Louis

2 Outline 1. Large scale MVS for organized photos (Aerial photos) 2. Large scale MVS for unorganized photos (Internet community photos) 3. Large scale indoor modeling

3 Common framework Divide Reconstruct Merge

4 Common framework Divide (the problem domain) Reconstruct (small problems in parallel) Merge (independent reconstructions)

5 Large Scale MVS for Organized Photos (Aerial photos) [ Google Maps ]

6 Drones are getting hot [ images.politico.com ] [ jubileehotel.com.au ] [ dronewars.net ]

7 Drones are getting hot [ Pix4d ]

8 Input images [ diydrones.com ]

9 Input images [ diydrones.com ]

10 Divide and Reconstruct [ diydrones.com ]

11 Divide and Reconstruct [ diydrones.com ]

12 Simplest strategy for divide and reconstruct Given N images Divide into N depthmap-reconstruction problems Detailed Real-Time Urban 3D Reconstruction From Video, Pollefeys et al., IJCV 2008.

13 Merge The most difficult step Depends on the application (visualization, analysis, )

14 Merge for visualization Hide gaps between reconstructions See through! Ideal model Model before merge

15 Merge for visualization Hide gaps between reconstructions See through! Ideal model Model before merge

16 Merge for visualization Hide gaps between reconstructions See through! More consistent and fatter Ideal model Model before merge Model after merge

17 Merge to a single consistent model Open problem for a polygonal mesh Multilevel Streaming for Out-of-Core Surface Reconstruction [Bolitho et al., 2007] Companies in industry might have something [ Google, Microsoft, Apple, Pix4d, Accute3D, ] Point cloud representation is easy Towards Internet-scale Multi-View Stereo [Furukawa et al., 2010]

18 Merged 3D Point-cloud

19 Example (Google Maps)

20 Outline 1. Large scale MVS for organized photos (Aerial photos) 2. Large scale MVS for unorganized photos (Internet community photos) 3. Large scale indoor modeling

21 Large-Scale MVS for Unorganized Photos (Internet Photos) Building Rome in a Day [ Agarwal, Snavely, et al., 2009 ]

22 Divide is a challenge

23 Divide is a challenge Divide in the image space?

24 Divide is a challenge Divide in the model space?

25 Image-clustering based on a model Towards Internet-scale Multi-View Stereo [furukawa et al., 2010 ] Image cluster

26 Image-clustering based on a model Towards Internet-scale Multi-View Stereo [furukawa et al., 2010 ] Image cluster

27 One way to look at the clustering problem Image Common features

28 One way to look at the clustering problem Normalized Cuts Image Common features

29 Formulation P1 is reconstructed well in C1 based on (image resolutions/distribution) P1 is covered by C1. P4 is covered by C2.

30 Formulation We want to 1. remove redundant images 2. keep each cluster small (memory limit) 3. cover many points

31 Formulation

32 Image Clustering Algorithm

33 Image Clustering Algorithm

34 Image Clustering Algorithm 1. Start with all the images in a cluster C1

35 Image Clustering Algorithm 1. Start with all the images in a cluster 2. Optimize compactness while keeping coverage C1

36 Image Clustering Algorithm 1. Start with all the images in a cluster 2. Optimize compactness while keeping coverage 3. If size constraint is broken, split a cluster C1 C2

37 Image Clustering Algorithm 1. Start with all the images in a cluster 2. Remove redundant images while keeping coverage 3. If size constraint is broken, split a cluster 4. Add an image to each cluster to satisfy coverage C1 C2

38 Image Clustering Algorithm 1. Start with all the images in a cluster 2. Remove redundant images while keeping coverage 3. If size constraint is broken, split a cluster 4. Add an image to each cluster to satisfy coverage C1 C2 5. Repeat (3,4) until size and coverage are satisfied?

39 Merge to a single consistent model Open problem for a polygonal mesh Multilevel Streaming for Out-of-Core Surface Reconstruction [Bolitho et al., 2007] Fusion of Depth Maps with Multiple Scales [Fuhrmann et al., 2011] Companies in industry might have something [ Google, Microsoft, Apple, Pix4d, Accute3D, ] Point cloud representation is easy Towards Internet-scale Multi-View Stereo [Furukawa et al., 2010]

40 View clustering

41 St. Peter s Basilica 1275 images 4 clusters 6M 3D points

42 Reconstruct

43 Merge Statistics

44 Outline 1. Large scale MVS for organized photos (Aerial photos) 2. Large scale MVS for unorganized photos (Internet community photos) 3. Large scale indoor modeling

45 Large-Scale Indoor Modeling Reconstructing the World s Museums [Xiao and Furukawa, 2012] Best Student Paper Award at ECCV

46 Technical Contributions Architectural shape priors Single consistent 3D model (real merging)

47 Technical Contributions Architectural shape priors Single consistent 3D model (real merging) Inverse Constructive Solid Geometry (CSG)

48 Technical Contributions Architectural shape priors! Single consistent 3D model (real merging) Inverse Constructive Solid Geometry (CSG)

49 + - + Technical Contributions Architectural shape priors Single consistent 3D model (real merging) Inverse Constructive Solid Geometry (CSG)

50 From CSG to a mesh CGAL

51 Algorithm

52 Cut into Slices side view gravity

53 Cut into Slices point count

54 Cut into Slices

55 2D CSG Reconstruction 1. Generate primitives 2. Build 2D CSG

56 Primitive Generation and Fitting (bottom-up) 2D rectangle 2D line point

57 2D CSG Reconstruction 1. Generate primitives point à line

58 2D CSG Reconstruction 1. Generate primitives From 4 line segments point à line line à rectangle

59 2D CSG Reconstruction 1. Generate primitives 2. Build 2D CSG

60 2D CSG Reconstruction Explain the data Free space Laser points

61 2D CSG Reconstruction 1. Generate primitives 2. Build 2D CSG Repeat in each slice

62 3D CSG Reconstruction 1. Generate primitives (cuboids) 2. Build 3D CSG (out of primitive candidates)

63 3D CSG Reconstruction 1. Generate primitives (cuboids)

64 3D CSG Reconstruction 1. Generate primitives (cuboids) Rectangle primitive 2D CSG

65 3D CSG Reconstruction 1. Generate primitives (cuboids) Rectangle primitive 2D CSG

66 3D CSG Reconstruction 1. Generate primitives (cuboids) Rectangle primitive 2D CSG

67 3D CSG Reconstruction 1. Generate primitives (cuboids) 2. Build a 3D CSG

68 Algorithm on Run

69 Last Step 1. Remove Ceiling 2. Texture Mapping

70

71 Challenging Navigation Goal Start Frick Collection Gallery (New York City)

72

73

74 # of sub problems Statistics

75 Ground vs. Aerial à Ground+Aerial Ground Aerial Ground+Aerial Google Streetview Google/Bing/NASA Google MapsGL Furukawa et al. This paper This paper Outdoor Indoor

76 Summary 1. Large scale MVS for organized photos (Aerial photos) 2. Large scale MVS for unorganized photos (Internet community photos) 3. Large scale indoor modeling Divide -> Reconstruct -> Merge

77 Summary 1. Large scale MVS for organized photos (Aerial photos) 2. Large scale MVS for unorganized photos (Internet community photos) 3. Large scale indoor modeling Divide -> Reconstruct -> Merge Divide is a challenge for unorganized data

78 Summary 1. Large scale MVS for organized photos (Aerial photos) 2. Large scale MVS for unorganized photos (Internet community photos) 3. Large scale indoor modeling Divide -> Reconstruct -> Merge Divide is a challenge for unorganized data Merge is always a challenge (a real solution for indoors)

79 Thank You! For slides and materials, check the tutorial website through our home pages.