AIT Inline Computational Imaging: Geometric calibration and image rectification B. Blaschitz, S. Štolc and S. Breuss AIT Austrian Institute of Technology GmbH Center for Vision, Automation & Control Vienna, Austria www.ait.ac.at/hpv
INLINE COMPUTATIONAL IMAGING: WORKING PRINCIPLE Multi-line scan camera Camera IlluminationConstant illumination Inspected object Inspected object Transport stage Transport stage 2
INLINE COMPUTATIONAL IMAGING: WORKING PRINCIPLE Multi-line scan camera Constant illumination Inspected object Transport stage 3
Blaschitz et al. Geometric calibration and image rectification of Inline Computational Imaging AIT ICI LIGHT FIELD: MULTIPLE VIEWING & ILLUMINATION ANGLES y x pin grid array
AIT INLINE COMPUTATIONAL IMAGING: INDUSTRIAL USE CASES Electronic parts Metal parts Product packaging Coins Printed circuit boards Measurement Security print / OVD 2381μm 1064μm 5
ICI SOFTWARE MODULES FOR 2D/3D TASKS Acquisition Stereo matching Coarse refinement Light field data acquisition Camera, illumination, and transport control High speed multi-view / light field stereo matching Fast discrete depth denoising / regularization preserving discontinuities ICI Sensor System Camera Illumination Transport stage Rectification Camera lens undistortion & rectification Transport misalignment correction Feature extraction Extraction of robust image features Extraction of local surface features Fine refinement Fast continuous depth / regularization Fusion with additional depth cues Refined 2D/3D data Fine depth model (depth map, point cloud) Depth measurement confidence Refined 2D texture images (all-in-focus, gloss / shadow suppression) Fine 2.5D surface map 6
MODELLING SKEW TRANSPORT Linescan image n (x 3,y 3 ) Line 1 v Sensor Line n (x 2,y 2 ) u Lens (x 1,y 1 ) u v time Input Image Stack Linescan image 1 Corresponding points in different linescan images have different u-coordinate Goal: Rectified Image Stack (for easier matching) where transport appears aligned Transport direction 7
GOAL: RECTIFIED IMAGE STACK (FOR EASIER MATCHING) WHERE TRANSPORT APPEARS ALIGNED Corresponding points in different linescan images should have same u-coordinate Rectification Camera lens undistortion & rectification Transport misalignment correction uncalibrated calibrated 8
CALIBRATION PRINCIPAL Transport direction t=(t 1,t 2,t 3 ) x v u I v u c Focal length f Sensor plane Camera center 9
VIRTUAL IMAGE PLANE I t=(t 1,t 2,t 3 ) x x v u H Coordinate system of new image plane I I v u Parallel to transport direction c Minimal rotation f Same distance to camera center 10
VIRTUAL IMAGE PLANE Project sensor lines t=(t 1,t 2,t 3 ) x x v u H v u v I Homography from I to I u c f 11
VIRTUAL IMAGE PLANE Resample in I x x Equal u -coordinates H R x x v u Equidistant spacing 12
VIRTUAL IMAGE PLANE Pull back to I t=(t 1,t 2,t 3 ) I x x x v u H R H -1 v v I u c u c f f 13
VIRTUAL IMAGE PLANE Interpolate color values x v H R H -1 u x x M(u,v) is the color value at position (u,v) (u-1,v) (x,v) x=(u,v) 1 - s s v u 14
CALIBRATION IMPROVES STEREO MATCHING Acquisition Stereo matching Coarse refinement Light field data acquisition Camera, illumination, and transport control High speed multi-view / light field stereo matching Fast discrete depth denoising / regularization preserving discontinuities ICI Sensor System Camera Illumination Transport stage Rectification Camera lens undistortion & rectification Transport misalignment correction Feature extraction Extraction of robust image features Extraction of local surface features Fine refinement Fast continuous depth / regularization Fusion with additional depth cues Refined 2D/3D data Fine depth model (depth map, point cloud) Depth measurement confidence Refined 2D texture images (all-in-focus, gloss / shadow suppression) Fine 2.5D surface map 15
IMPROVED CORRESPONDENCE ANALYSIS Stereo matching High speed multi-view / light field stereo matching Rectification Camera lens undistortion & rectification Transport misalignment correction Feature extraction Extraction of robust image features Extraction of local surface features Refined 2D/3D data Fine depth model (depth map, point cloud) Depth measurement confidence Refined 2D texture images (all-in-focus, gloss / shadow suppression) Fine 2.5D surface map 16
SCENE COMPARISON IMAGE GRADIENT Uncalibrated Calibrated Maximal pixel intensity Reconstructed z value after depth denoising Maximal pixel intensity Reconstructed z value after depth denoising 17
SCENE COMPARISON 3D RECONSTRUCTION Uncalibrated Calibrated 18
EXAMPLE: STAIRCASE INPUT LIGHTFIELD STACK CAD model of the 3d printed model staircase (print tolerance approx. 0.0615mm) 19
EXAMPLE: STAIRCASE 3D RECONSTRUCTION Distance from camera center Height difference of two steps 20
EXAMPLE: EURO CENT COINS IMAGE STACK 10 Euro Cent coin Diameter 19.75mm Thickness 1,93mm 2 Euro Cent coin Diameter 18.75mm Thickness 1,67mm 21
EXAMPLE: EURO CENT COINS - UNCALIBRATED 22
EXAMPLE: EURO CENT COINS - CALIBRATED 23
EXAMPLE: EURO CENT COINS ERROR MEASURE 24
TAKE-HOME MESSAGE New calibration method turns AIT s Inline Computational Imaging system, which uses a single multi-line scan camera to generate 3d light field stacks into a measurement device in µm-scale at industrial inline production speed For industrial applications, scientific work and our patents visit www.ait.ac.at/hpv High speed transport 25
THANK YOU FOR YOUR ATTENTION! Bernhard Blaschitz bernhard.blaschitz@ait.ac.at 26