Towards a visual perception system for LNG pipe inspection

Transcription

1 Towards a visual perception system for LNG pipe inspection LPV Project Team: Brett Browning (PI), Peter Rander (co PI), Peter Hansen Hatem Alismail, Mohamed Mustafa, Joey Gannon Qri8 Lab

2 A Brief Overview of Previous Work Thesis work: wide-angle keypoint detection and matching for vision-based localization. Supervisors: Peter Corke and Wageeh Boles. Visual odometry - integrate incremental estimates of camera egomotion. Compute motion from the (sparse) optical flow Detect and describe keypoints Match keypoints 2

3 Using Wide-Angle Images Wide-angle images can improve egomotion estimates. Most keypoint detection/matching algorithms designed for perspective images. Scale and affine invariant. Do not account for image distortion and certain camera rotations. SIFT: perspective SIFT: fisheye ssift: fisheye 3

4 How to Process Wide-Angle Images? Consider an image as a 2D representation of a 3D function. Perspective 4

5 How to Process Wide-Angle Images? Consider an image as a 2D representation of a 3D function. Formulate image processing algorithms as scale & rotationally shift invariant 1 operations. 2 u v R 2 I, 2 I S θ,φ S Daniilidis, Makadia and Bulow. Image Processing in Catadioptric Planes, OMNIVIS workshop,

6 SIFT on the Sphere (ssift, psift) Reformulate SIFT as an image processing algorithm on the sphere. e.g. Perform scale-space filtering on the sphere (spherical Fourier domain, conformal projections) SIFT: convolution with 2D Gaussian ssift & psift: convolution on the sphere image solution of heat equation on the sphere 6

7 Applications Relevant to Localization Equiangular catadioptric Fisheye 7

8 Liquid Natural Gas (LNG) Processing Plants Key industry relevant to Qatar. Inspection is critical for safety. RasGas 10-12m ~15m m ~ 4 layers of pipe racks 8

9 Monitor corrosion rates using: Current Industry Practice Magnetic Flux Leakage (MFL), Radiography (X-ray), Ultrasonic, Sacrificial samples. Typically applied external to the pipes. Makes full coverage difficult to impossible. Use predictive models based on metallurgical properties to predict corrosion rates. 9

10 LPV Concept Use visual perception system for corrosion detection. Map pipe structure and appearance. Register maps across different runs. Overlay other sensor data. Automate corrosion detection via surface changes. Need to estimate pose of robot (camera) We have explored monocular visual odometry algorithms (dense and sparse) 10

11 Datasets collected from 2 pipes Datasets Pipe 1: NREC, 6 meters long, 152mm diameter *~20 long and 6 diameter+ Pipe 2: Qatar, 4 meters long, 406mm diameter *~13 long and 16 diameter+ 11

12 Dense Monocular: estimating pose Each image pair is related by a translation motion model (image space). Assumes that the camera is viewing a planar scene. I t 1 I t I x, t I x x, t 1 x x, y Image shift Image shift is related to change in camera pose (pipe coordinate frame). X y h x is a reference scale (meters / pixel) x (pixels) h (meters) y (pixels) X (meters) 12

13 Dense Monocular: pixels to meters The scale factor : Convert pixels to meters. Use reference pattern with know geometry (distance between circle centers). Take image of reference pattern affixed to interior surface of pipe. Find pixel distance between circle centers in undistorted image. Scale factor is the mean ratio (meters / pixel). Original image Undistorted image 13

14 Dense Monocular: gain correction Divide original image with a gain mask (image of white card in pipe) Dense monocular algorithm performs poorly without gain correction Gain Image Original Gain Corrected 14

15 Dense Monocular: estimating pose Estimate integer pixel shift x,y using full-search. Minimize Sum of Absolute Difference (SAD). I x, t I x x, t 1 Subpixel iterative refinement. Employs the same intensity constancy assumption. I x, t I x x, t 1 Minimize the Sum of Squared Distance (SSD). Solve iteratively using Gauss-Newton (gradient based). Can converge on local minima: this is why we first use full search. 15

16 Dense Monocular: results Results: distance travelled (h) versus ground truth. Pipe 1: ground truth = mm Pipe 2: ground truth = mm Dataset Metric Dense (full-search) Dense (model) Pipe 1a Error (mm) 46.7 (0.799%) -9.0 (0.154%) Pipe 1b Error (mm) 52.0 (0.890%) 42.4 (0.725%) Fwd. Error (mm) (0.246%) 8.7 (0.149%) Pipe 1c Rev. Error (mm) (0.837%) (0.361%) Total Error (mm) 34.5 (0.295%) 29.8 (0.255%) Fwd. Error (mm) 9.0 (0.270%) 28.4 (0.840%) Pipe 2 Rev. Error (mm) (3.690%) (2.920%) Total Error (mm) (1.980%) (1.879%) 16

17 Dense Monocular: stitched images h X Pittsburgh (NREC): 6 meters long, 6 diameter Qatar: 4 meters long, 16 diameter 17

18 Sparse Monocular: correspondences Use sparse correspondences between key frames to estimate 6 DOF motion. Minimal scale change between images. I n In 1 I n 2 Small motion Region-based Harris (sub-pixel accuracy) SIFT descriptor Mutual ambiguity matching (SIFT descriptors) Outlier rejection using RANSAC and Nister s 5 point algorithm (Only small subsets of correspondences shown) 18

19 Sparse Monocular: estimating pose Estimate current camera pose P by minimizing the error n We need to use the measured radius r to map image points to the pipe. I n 2 I n 1 I n P n 2 (known) P n 1 (known) P n (unknown) X X ~ I n X known scene point positions(previous frames) ~ X estimated scene point positions(for estimate of P T R R C poseof camera with respect topipe: R 3x3 rotation, C,C Y,C C X h 19 P n ) i ~ X i X i 2 T

20 Sparse Monocular: estimating pose Estimate current camera pose P by minimizing the error n We need to use the measured radius r to map image points to the pipe. P T R R C poseof camera with respect topipe: R 3x3 rotation, C,C Y,C C X h T Incremental constraint removal (simple to complex approach). Estimate initial pose using 1 degree of freedom translation (C h ). Initial batch optimization (10 key frames) of camera center (C x, C y ), while R remains fixed. Periodic refinement (every 50 th key frame) of previous 100 key frames using full 6D model. 20

21 Sparse Monocular: why incremental? Ambiguity in decoupling rotational and translational (6DOF) motion from flow field. Minimal depth discontinuities, small changes in camera pose (R, C). focus of expansion is outside of camera FOV. Incremental gives more robust results, yet retains accuracy. 21

22 Sparse Monocular: results Distance travelled (h) versus ground truth. Pipe 1: ground truth = mm Pipe 2: ground truth = mm Dataset Metric Dense (full-search) Dense (model) Sparse Pipe 1a Error (mm) 46.7 (0.799%) -9.0 (0.154%) 17.0 (0.306%) Pipe 1b Error (mm) 52.0 (0.890%) 42.4 (0.725%) 16.9 (0.289%) Fwd. Error (mm) (0.246%) 8.7 (0.149%) 7.5 (0.128%) Pipe 1c Rev. Error (mm) (0.837%) (0.361%) 3.5 (0.060%) Total Error (mm) 34.5 (0.295%) 29.8 (0.255%) (0.034%) Fwd. Error (mm) 9.0 (0.270%) 28.4 (0.840%) 7.1 (0.210%) Pipe 2 Rev. Error (mm) (3.690%) (2.920%) 20.7 (0.610%) Total Error (mm) (1.980%) (1.879%) (0.201%) 22

23 Sparse Monocular: pose estimates Results shown for small segment of pipe. 23

24 Sparse Monocular: stitched images h τ=rφ Pittsburgh (NREC): 6 meters long, 6 diameter Qatar: 4 meters long, 16 diameter 24

25 Sparse Monocular: stitched images 25

26 Current Work 26

27 Limitations of Monocular VO Need reference scale measurement. Can only recover weak 3D structure. Pose estimates assumed fixed radius pipe. 27

28 We can use stereo: Current Work: Stereo High resolution/accuracy 3D structure if internal pipe surface. Visual odometry using stereo data (no scale ambiguity). Commercial stereo cameras are not suitable for our application. No (or minimal) image overlap in small pipes. Minimum Operating Distance (MOD) of lens is too large (i.e. images are blurry). 28

29 Current Work: stereo Verged stereo pair (6mm focal length S-mount lens, 1024x768 images). 29

30 Current Work: stereo Accurate stereo reconstruction using a large baseline 30

31 Exploring dense stereo winner takes all. Dynamic programming. Graph cuts.. Current Work: stereo Use sparse/dense stereo for visual odometry Do not assume fixed radius pipe. 31

32 Current Work: stereo Exploring omni-directional stereo pair configurations (accuracy). 32

33 Current Work: pipe-crawling robot 33

34 Questions? 34