Real-time Image-based Reconstruction of Pipes Using Omnidirectional Cameras

Transcription

1 Real-time Image-based Reconstruction of Pipes Using Omnidirectional Cameras Dipl. Inf. Sandro Esquivel Prof. Dr.-Ing. Reinhard Koch Multimedia Information Processing Christian-Albrechts-University of Kiel In cooperation with IBAK Helmut Hunger GmbH & Co. KG, Kiel

2 Motivation Quelle: Quelle: 2

3 Sewer inspection systems Inspection of sewer shafts and pipes with mobile camera systems Manual evaluation of videos by expert Task: Create 3d reconstruction for visualization and measuring 3

4 Contents Image Acquisition and Structure from Motion Our reconstruction approach Cylinder mapping Global geometric correction Profile fitting and model creation Tests and applications Conclusion 4

5 Image acquisition for sewer pipe inspection pull cameras out on cable with 35 cm / sec shaft cable vertical profiles mobile system with cameras capture images every 5 cm 5

6 Image acquisition for sewer shaft inspection cable pull camera up on cable with 35 cm / sec entry capture image every 5 cm shaft camera with rotation sensor horizontal profiles 6

7 Spherical camera model X max. angle radius x principal point 7

8 Structure from Motion Initialization X x0 C0 x1 2d/2d correspondence C1 camera distance 5 cm 8

9 Structure from Motion Tracking X 2d/3d correspondence x2 x0 C0 x1 2d/2d correspondence C1 C2 2d/2d correspondence camera distance 5 cm 9

10 3d reconstruction Problems and challenges Challenging lighting conditions and depth of view Strong image distortions Brief feature point visibility, large frame-to-frame translation Inaccurate camera calibration Demands for real-time processing: min. 7 Hz 10

11 Our approach Image Acquisition and Structure from Motion Our reconstruction approach Cylinder mapping Global geometric correction Profile fitting and model creation Tests and applications Conclusion 11

12 Overview of our approach camera image rotation sensor Image preprocessing rotation compensation and cylinder mapping 1 Feature point detection and tracking apply KLT tracker 2 Structure from Motion compute camera poses and 3d geometry 3 4 Global optimization correct camera path and 3d geometry camera poses 5 Measuring classify profiles, create 3d model profiles 12 3d point cloud 3d model

13 1. Image preprocessing: Roll rotation compensation roll rotation sensor 45 fisheye camera image aligned fisheye image principal point 13 5

14 1. Image preprocessing: Cylinder mapping fisheye camera image aligned cylinder image principal point 14

15 2. Feature point tracking Initialization: Detect KLT feature points 1 Find correspondences along image rows Create row-dependent horizontal translation function search window image row [pel] frame 0 feature points frame 1 15 translation [pel]

16 2. Feature point tracking Tracking: Predict new position with translation function 1 Use small search window around target position Update horizontal translation function search window image row [pel] frame k frame k+1 16 translation [pel]

17 3. Structure from Motion Output: 1 Computed 3d point cloud Computed camera poses for each frame (position + rotation) 2 Robust estimation using RANSAC

18 4. Global optimization Error accumulation 1 Inaccurate camera calibration Bent and conical 3d reconstruction 2 Global optimization needed Bundle Adjustment not useful for real-time application

19 4. Global optimization Correct 3d reconstruction using geometric constraints: 1 Average camera path is known 2 (vector of gravity = z-axis) Camera baseline between frames 3 is known (approx. 5 cm)

20 4. Global optimization Fit polynomial P(t) to average camera path Transform 3d points and camera poses via mapping of P(t) onto corrected path P*(t) = 5 cm t ez average camera path P(t) camera baseline = 5 cm z-axis ez (vector of gravity) distorted reconstruction corrected reconstruction 20 5

21 5. Measuring profile shapes Classification of horizontal shaft / vertical pipe profiles Robust shape estimation from 3d points within slice 1 2 circular profile 3 4 rectangular profile 21 5

22 5. Creating 3d models for visualization Classify profile shapes in 3d points Connect subsequent contours of the same shape class Brightness alignment of images Optional 3d geometry fitting Output: 4 Profile shapes Wire-frame model 5 Textured model 22

23 Output of our approach camera image rotation sensor Image preprocessing rotation compensation and cylinder mapping 1 Feature point detection and tracking apply KLT tracker 2 Structure from Motion compute camera poses and 3d geometry 3 4 Global optimization correct camera path and 3d geometry camera poses 5 Measuring classify profiles, create 3d model profiles 23 3d point cloud 3d model

24 Tests and applications Image Acquisition and Structure from Motion Our reconstruction approach Cylinder mapping Global geometric correction Profile fitting and model creation Tests and applications Conclusion 24

25 Example test sequence for shaft reconstruction 60 cm 100 cm 3m depth 100x100 cm 25

26 Evaluation of profile measurements Average diameter estimation error for 57 sequences 8% with correction no correction (mean + std.dev.) 6% Circular profile Rectangular profile 4% 2% 0% No. of sequence Success rate of diameter estimation 100% 90% 80% 70% No. of sequence 26

27 Evaluation of global geometric correction Variate intrinsic camera parameters (here: max. angle ± 2 ) Systematic reconstruction error (here: up to 15 cm) Compensation by global geometric correction 120 cm Estimated diameter 110 cm Parameter error without global geometric correction 100 cm 90 cm 104 cm 102 cm ground truth with global geometric corrrection cm 98 cm 96 cm ground truth Frame no. in sequence 27

28 Runtime evaluation (offline) Original application (offline) 2 3 frames per second Computation on CPU RANSAC with many iterations needed for robustness Initialization: ~800 ms / frame Image preprocessing Tracking: Image preprocessing Structure from Motion KLT tracking 1s ~380 ms / frame KLT tracking Profile measuring SfM 0.5 s 0.75 s 1s Global correction 28

29 Runtime evaluation (realtime) Realtime application (on site) ~7 frames per second Image preprocessing and KLT tracking on GPU Use PreemptiveRANSAC instead of RANSAC Initialization: ~240 ms / frame IP KLT SfM 0.25 s Tracking: 0.5 s 0.75 s 1s 0.5 s 0.75 s 1s ~140 ms / frame Profile measuring IP SfM 0.25 s KLT Global correction 29

30 Use camera poses for manual measuring Manual selection of measuring points in camera image Automatic correspondence search in subsequent image Triangulation of 3d point using known camera poses Compute measurements in 3d space (e.g. distance) computed circle selected image points 25 cm 30

31 Visualization of 3d models Kiel Elmschenhagen #3 Kiel Hürth #1 31 Kiel Nordmarksportfeld #1

32 Conclusion Image Acquisition and Structure from Motion Our reconstruction approach Cylinder mapping Global geometric correction Profile fitting and model creation Tests and applications Conclusion 32

33 Conclusion Robust automatic approach for image-based 3d reconstruction of sewer pipes and shafts with IBAK Panoramo / SI Simple geometric correction replaces Bundle Adjustment Creation of 3d models for visualization Manual measuring in camera images using computed camera poses Real-time capability to use on-site Successful application to practise 33

34 Thank you for your attention! 34