Data Acquisition, Leica Scan Station 2, Park Avenue and 70 th Street, NY

Transcription

1 Automated registration of 3D-range with 2D-color images: an overview 44 th Annual Conference on Information Sciences and Systems Invited Session: 3D Data Acquisition and Analysis March 19 th 2010 Ioannis Stamos City University it of New York Hunter College & Graduate Center Data Acquisition, Leica Scan Station 2, Park Avenue and 70 th Street, NY

2 Park Avenue: Six registered scans (top view) Brightness Images System Overview Range Images 2D feature extraction Segmentation 3D feature extraction Partial Model Range-Intensity Registration Range-Range Registration Complete Model FINAL PHOTOREALISTIC MODEL FINAL MODEL

3 Range-Intensity Registration Fixed relative positioning - limitations 2D sensor is limited by 3D scanner 3D scanning slow/laborious process 2D image capture fast/flexible 2D sensor cannot dynamically adjust parameters on site Cannot handle historical photographs

4 Automated methods One 2D image Untextured 3D model Ikeuchi 03 (edges reflectance image) Troccoli & Allen 04 (shadows) Stamos & Allen 01 (rectangles) Liu, Stamos 05 (parallelepipeds) Liu, Stamos 07(lines) One 2D image - Textured 3D model Yang, Stewart etal. 07 Schindler (regularities) Multiple 2D images (SfM/stereo) 3D model Liu, Stamos 06, Zhao etal. 05 Aerial 2D images 3D model Ding 08, Fisher 09, Neumann 09, Kaminsky 09. Input: Range Images System Outline (Liu, Stamos) Input: 2D Images 3D Line Extraction Range Registration 3D Line Clustering 3D Feature Extraction 2D Line Extraction Vanishing Point Extraction Camera Calibration and Rectification 2D Feature Extraction Automatic Features Matching and Translation Computation Final Transformation Optimization

5 3D feature extraction, Thomas Hunter Building 3D Line Extraction Clustered 3D Lines (3 major directions) 3D Face Extraction Range-Range Registration 3 Faces Extracted (Front, Left and Right) 3D Feature Extraction Line Clustering All 3D Lines Extracted Rectangular Parallelepipeds Clustering lines into vertical/horizontal parallelepipeds

6 System Outline (Liu, Stamos) Input: Range Images Input: 2D Images 3D Line Extraction Range Registration 3D Line Clustering 3D Feature Extraction 2D Line Extraction Vanishing Points Extraction Camera Calibration and Rectification 2D Feature Extraction Automatic Features Matching and Translation Computation Final Transformation Optimization 2D FEATURE EXTRACTION 2D Line Extraction 2D Feature Extraction VPs Extraction Rectified 2D Lines Camera Calibration and Rectification Matching between VPs and 3D scene major directions Extracted Rectangles Rotation between camera and 3D scene

7 Vanishing Points (2D line clustering) Computing Vanishing Points

8 Solving for the rotation Clustering 2D lines from 2D image into horizontal/vertical rectangles

9 3D Features 2D Features

10 Translation computation from a matched 3D-2D pair Matching 3D parallelepipeds with 2D rectangles 3D parallelepipeds 2D rectangles Matches shown in blue

11 The performance on Thomas Hunter Building FP CM G OP (%) RDT (%) TIME (sec) 117 x x x x x x x x x x x Recovered camera locations wrt texture-mapped point model

12 Recovered camera locations wrt texture-mapped point model Virtual (synthetic) scene views

13 Virtual (synthetic) scene views Virtual (synthetic) scene views

14 Discussion Advantages Accurate Limited user interaction 2D plane to 3D plane matching Limitations Requires major facades Higher-order features decrease generality Non-interactive speed System II Outline (lines) Input: Range Images Input: 2D Images 3D Line Extraction Range Registration 3D Line Clustering 3D Feature Extraction 2D Line Extraction Vanishing Points Extraction Camera Calibration and Rotation Computation 2D Feature Extraction Automated Feature Matching and Translation Computation Final Transformation Optimization

15 Feature Detection User Interaction Z w Y w O w X w OpenGL Frame Y i Z c P Y c f X i C X c C o

16 Matching VPs and 3D dirs Image Plane D 3D a α D 3D b V a V b Z c Y c P β f O w Y w C Z w X w X c Initial List of Candidate Matches D 3D a D 3D b V a V b D 3D c D 3D d V a D 3D a V b D 3D c D 3D b Refined List of Candidate Matches D 3D d

17 Focal length computation D 3D a α D 3D b Automatic Feature Matching and Translation Computation 3D Line Feature Get 2 pairs (l 3D a, l 2D a) and (l 3D b, l 2D b) 2D Line Feature Compute Translation C(l 3D a,l 2D a) Compute overlap O(l 3D b, l 2D b) by applying C(l 3D a, l 2D a) O(l 3D b, l 2D b) > Oth Compute Translation C(l 3D b, l 2D b) O(l 3D b, l 2D b) < Oth Compute overlap O(l 3D O(l 3D a, l 2D a) < Oth a, l 2D a) by applying C(l 3D b, l 2D b) No more pairs Grade each Ci in T Gi > Gth Optimization of Ci O(l 3D a, l 2D a) > Oth Candidate Trans C is the weighted average of C(l 3D a, l 2D a) and C(l 3D b, l 2D b). Store C in list T Final Translation

18 Translation computation: 3D-2D line match Image Plane A l 3D a S Y c l 2D a B Z c T C O w Y w X c R X w Z w Metric to be minimized for transform computation Kumar & Hanson

19 GCT 3D scene 2D image GCT User interaction 2D image

20 GCT Text. mapped 3D model (automated) 2D image GCT Text. mapped 3D model (automated) 2D image Corresponding 2D/proj. 3D lines

21 GCT 3D scene 2D image GCT Texture mapped 3D model Corresponding 2D/proj. 3D lines

22 Evaluation Selected 3D lines from 3D cloud Selected 2D lines Projection of 3D lines on 2D img Calculate error

23 3D lines 2D lines focal (init) focal (comp) Matches Error (pixels) Thomas Hunter Astor Place Grand Central Terminal

24 2D-image to 3D-range Registration Structure From Motion Range Images Many images CVPR, IJCV Unregistered Point Clouds 3D-range to 3D-SFM Registration Texture Mapping SFM Point Cloud Two Registered Point Clouds Final Output MULTIVIEW EXAMPLE

25 POSE AND STRUCTURE RECOVERY 3D Structure X j Common 2D-2D.. Frame 0 K[I 0] Frame 1 K[R 1 T 1 ] New Structure Frame 2 K[R 2 T 2 ] Frame N K[R N T N ] Initial pose and structure Compute [Ri Ti] using 6-point RANSAC Sequential updating to recover pose and structure 84 STRUCTURE AND MOTION REFINEMENT Bundle adjustment Minimize min reprojection D m error:,pˆ Mˆ m n Pˆ,Mˆ k i k 1 i 1 ki k i 2 Maximum Likelihood Estimation (assume error is zero-mean Gaussian noise) Huge problem but can be solved efficiently (use sparse bundle adjustment) 85

26 ALIGNING RANGE AND SFM POINT CLOUDS Range and SFM models Range model Range with overlaid SFM model (red)

27 3D-RANGE TO 3D-SFM REGISTRATION 3D-SFM points projected onto image 3D-Range points projected on same image Finding corresponding points between 3D-range and 3D-SFM models is possible on the 2D-image space 3D-RANGE TO 3D-SFM REGISTRATION 3D-SFM points projected onto image 3D-Range points projected on same image x :Projection of SFM point X x: Projection of SFM point X y : Projection of range point Y y: Projection of range point Y (X,Y) and (X,Y ) are candidate matches: y is the closest point to x and y the closest point to x in 2D-image space

28 3D-RANGE TO 3D-SFM REGISTRATION Candidate matches (X,Y) and (X,Y ) are compatible iff the following scale factors are equal: 1. s 0 = X-C sfm / Y-C range 2. s 1 = X -C sfm / Y -C range 3. s 2 = X -X / Y -Y where C sfm is the COP wrt the SFM model and C range is the COP wrt the range model Confidence of (X,Y) := # compatible matches (X,Y ). Match (X,Y) with largest confidence among all 2D images provides the optimal scale factor s opt List C of robust matches := {(X,Y) such that s 0 s opt }U{(C sfm, C range ) of images} Scale, rotation, and translation computed by minimizing QUANTITATIVE RESULTS 91

29 Single Image to 3D range Structure-from-motion approach GRAND CENTRAL (1)

30 GRAND CENTRAL (2) CONCLUSIONS Integration of multiview geometry with range registration 2D-to-3D registration is used for a subset of images that contain a sufficient number of linear features Brings these images into alignment with the dense 3D-range model Multiview geometry exploits 2D-point correspondences Brings all images into alignment and produces sparse SFM model 3D-range to 3D-SFM registration Aligns all images with the dense 3D-range model Accurate texture mapping onto dense 3D-range data Limitations: Need accurate vanishing points (3D-2D registration) Need accurate 3D features Need to handle symmetries

31 Credits Collaborators: Dr. Cecilia Chao Chen (Google, Inc.) Dr. Lingyun Liu (Google, Inc.) Dr. Siavash Zokai Dr. Gene Yu Prof. George Wolberg (CCNY) Funding: NSF CAREER,MRI,RI,MSC Awards PSC-CUNY Grants CUNY Institute of Software Design & Development Thank you

32 Automated registration of 3D-range with 2D-color images: an overview 44 th Annual Conference on Information Sciences and Systems Invited Session: 3D Data Acquisition and Analysis March 19 th 2010 Ioannis Stamos City University it of New York Hunter College & Graduate Center