Structured Light II Johannes Köhler Johannes.koehler@dfki.de Thanks to Ronen Gvili, Szymon Rusinkiewicz and Maks Ovsjanikov
Introduction Previous lecture: Structured Light I Active Scanning Camera/emitter systems Geometric calibration Today: Structured Light II Gamma calibration Mesh registration 1/7/2015 Lecture 3D Computer Vision 2
Gamma function 1/7/2015 Lecture 3D Computer Vision 3
Gamma function Phase shifting heavily relies on the different gray values being transmitted correctly Information A Information B Intensity encoding Intensity decoding Physical transmission 4 1/7/2015 Lecture 3D Computer Vision 4
Gamma function Problem: Usually projector and camera distort the values being sent / received Result is a non-linear gamma curve of the system x Axis: Intensity values sent by the projector [0 255] y Axis: Intensity values captured by the camera [0 255] 1/7/2015 Lecture 3D Computer Vision 5
Gamma function Non-linearity causes large errors in the reconstructions 1/7/2015 Lecture 3D Computer Vision 6
Gamma function Solution: Distort the values being sent by the projector in a way, that once they are received, this happens in a linear manner Find function f best describing the systems response curve Exponential function Sigmoid function Use inverse f^{-1} of f and send f^{-1}(x) instead of x Received value: f(f^{-1}(x)) = x 1/7/2015 Lecture 3D Computer Vision 7
Gamma function By linearizing the gamma response, the quality of the reconstructions is highly improved 1/7/2015 Lecture 3D Computer Vision 8
Matching data + 1/7/2015 Lecture 3D Computer Vision 9
Motivation Usually whole objects should be reconstructed from all sides A structured light system can only reconstruct the parts of the objects which are seen by both the camera and the projector Problem: Complex objects 1/7/2015 Lecture 3D Computer Vision 10
Recall: Calibration Extrinsics Rel Intrinsics C Intrinsics P 1/7/2015 Lecture 3D Computer Vision 11
Motivation Not all details can be captured in one scan Problem: Each scan has its own coordinate system Question: How to align scans? 1/7/2015 Lecture 3D Computer Vision 12
Aligning 3D data If the correct correspondences are known, we can find correct relative rotation/translation 1/7/2015 Lecture 3D Computer Vision 13
Aligning 3D data How to find correspondences: User input? Feature detection? Signatures? Alternative: Assume closest point 1/7/2015 Lecture 3D Computer Vision 14
Aligning 3D data Iterate to find alignment => Algorithm: Iterative Closest Points (ICP) [Besl & McKay 92] Problem: Converges only if starting position is close enough Closest point is often a bad approximation for the corresponding point 1/7/2015 Lecture 3D Computer Vision 15
ICP method Extended formulation of the ICP method 1. Selecting source points (from one or both meshes) 2. Matching to points in the other mesh 3. Weighting the correspondences 4. Rejecting outlier point pairs 5. Assigning an error metric to the current transform 6. Minimizing the error metric w.r.t. transformation 1/7/2015 Lecture 3D Computer Vision 16
ICP method 1. Step: Selecting source points Use all points Uniform sampling Random sampling Normal-space sampling Ensure that samples have normals distributed as uniformly as possible Uniform sampling Normal-space sampling 1/7/2015 Lecture 3D Computer Vision 17
Random vs. normal-space sampling Random sampling Normal-space sampling Normal-space sampling better for mostly-smooth areas with sparse features 1/7/2015 Lecture 3D Computer Vision 18
ICP method 2. Step: Matching Closest point => Expensive For range images we can simply project point [Blais 95] Slightly worse performance per iteration 1/7/2015 Lecture 3D Computer Vision 19
ICP method 3. Step: Assigning weights Constant weight Assigning lower weights to pairs with greater point-to-point distance : Weight = 1 Dist( p Dist 1, p2) max Weighting based on compatibility of normals Weight = n 1 n 2 Scanner uncertainty 1/7/2015 Lecture 3D Computer Vision 20
ICP method 4. Step: Rejecting outliers Point to point distance higher than a given threshold Pairs containing points on end vertices 1/7/2015 Lecture 3D Computer Vision 21
ICP method 4. Step: Rejecting outliers Rejection of pairs that are not consistent with their neighboring pairs [Dorai 98] (p1,q1), (p2,q2) are inconsistent <=> Dist( p1, p2) Dist( q1, q2) > τ q1 q2 p2 p1 1/7/2015 Lecture 3D Computer Vision 22
ICP method 5. & 6. Step: Error metric and minimization Minimize e.g. f ([ R t]) = 1 N S N S i= 1 qi [ R t] 1 p i 2 Sequential minimization Minimize 1 and 2, then 2 and 3, => Accumulated error (loop closing problem) 1/7/2015 Lecture 3D Computer Vision 23
ICP method 5. & 6. Step: Error metric and minimization Simultaneous minimization Simultaneously minimize the various transformations [R i t i ] Diffusively distribute the alignment error over all overlaps of each range images Large computational cost Difficult convergence, if correspondences are updated 1/7/2015 Lecture 3D Computer Vision 24
ICP method 6. Step: Minimization by Pulli s method Assumption: Pair-wise alignment yields best result for each pair Align pairs with each other using standard ICP After pair-wise alignment, correspondences can be kept fixed Subsequent global pose minimization converges quickly and distributes error diffusely 1/7/2015 Lecture 3D Computer Vision 25
ICP method How to find good initial position? Try to find and match features in both meshes Closely related to image features and matching but in 3D Therefore: Use neighborhood geometric information Descriptors should be Invariant under transform Local Cheap Improves correspondence searching and matching 1/7/2015 Lecture 3D Computer Vision 26
ICP method Features: 0.20 Inherent smoothing Related to mean curvature Robust to noise 1/7/2015 Lecture 3D Computer Vision 27
ICP method Feature identification Pick as features points with rare descriptor values Results in few but relatively robust correspondences 100 90 80 70 # occurences 60 50 40 30 20 Features 10 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Descriptor value 1/7/2015 Lecture 3D Computer Vision 28
ICP method Feature matching Match features with similar descriptor values Use e.g. RANSAC (see lecture 5: Parameter estimation) to filter out wrong correspondences P Q 1/7/2015 Lecture 3D Computer Vision 29
ICP method Use features to roughly align the various reconstructions Then use classical ICP for refinement using the good starting approximation Input: 10 scans Alignment Refined by ICP 1/7/2015 Lecture 3D Computer Vision 30
SL reconstruction example 31
Is ICP a good choice? The previous approach (pre-calibration + subsequent alignment) is by far the most common one in active scanning For alignment, ICP is heavily dominating Does it always yield good results? We will evaluate the global alignment approach of Pulli: Input: N partial scans Pairwise alignment of scans with largest mutual overlapping Global optimization with fixed correspondences 1/7/2015 Lecture 3D Computer Vision 32
Synthetic Data Generation 10 meshes are rendered from 8 views each in order to generate a point cloud This data serves as ground truth 1 out of 8 poses is the world, i.e. 70 poses must be estimated To each point cloud besides the world, we apply a small error E = [R t]. This yields fair starting values within ICPs convergence range 1/7/2015 Lecture 3D Computer Vision 33
Error computation We apply Pulli s global alignment method to the transformed point clouds; It estimates E E -1 E E = I I is the estimation error. We apply it to every vertex v of the respective pose to obtain a one-dimensional error e = v-i v. 1/7/2015 Lecture 3D Computer Vision 34
Error evaluation When is a registration good? Scanner sampling density: Zoom x10 Zoom x100 1/7/2015 Lecture 3D Computer Vision 35
Error evaluation Sampling density: Distance between vertices of neighboring pixels is the maximal theoretical precision (for given depth) Note, that this does not regard depth error We consider a registration as good, if the error it introduces is <= the scanner s sampling density, i.e. the registration error is not larger than the theoretical precision at a given depth 1/7/2015 Lecture 3D Computer Vision 36
ICP Error distribution 70 estimated poses are evaluated Estimation error propagated to vertices Histogram with 509101060 error entries (x: Error, y: amount of points; red line is sampling density, note the peak at the right border): 1/7/2015 Lecture 3D Computer Vision 37
ICP Error distribution Why are so many points below the sampling density threshold? Peak (failure) for (partially) symmetric objects Weak precision for smooth objects Good precision for objects with complex geometry 1/7/2015 38
ICP Drawbacks Failure for symmetric objects Weak precision for symmetric objects Strong dependence on sampling strategy 1/7/2015 Lecture 3D Computer Vision 39
Registration from Optical Features Structured light scanner is equipped with camera Use of multiple cameras possible and reasonable (no gamma calibration, if equal camera types) Regular camera images allow feature matching 1/7/2015 Lecture 3D Computer Vision 40
Registration from Optical Features Regular camera images are perfectly registered to the point cloud/depth map 2D-2D Matches between cameras of different scanner positions thus induce 3D-3D matches Pose estimation from 3D-3D matches: http://www.kwon3d.com/theory/jkinem/rotmat.html 1/7/2015 Lecture 3D Computer Vision 41
Discussion Constant correspondences fast and accurate Independent of geometry Decreased precision for sparse features Failure for symmetric or repetitive features In the literature usually used for initialization of ICP Some (few) authors point out better robustness than ICP Further (potential) drawback: pre-calibration Parameters are kept constant, but might change due to e.g. heat generation, cheap hardware, scanner motion or external influences Distorted geometry, bad fit (see Furuakwa) 1/7/2015 Lecture 3D Computer Vision 42
Registration Using Structure from Motion Structure from Motion (SfM) is widely used for estimating camera and structure parameters (see lectures 6/7) Choose camera pair Initialize Intrinsics and Extrinsics Triangulate points Optimize Find most 2D3D-correspondences Compute new Intrinsics and Extrinsics K 0 [ R t] 0 K 1 [ R t] 1 K 2 [ R t] 2 1/7/2015 Lecture 3D Computer Vision 43
Registration Using Structure from Motion SfM input (lecture 6/7): 2D correspondences, intrinsics; poses and structure remain to be computed SfM approach (lecture 6/7): Initial pose from 2D-2D correspondences (E-matrix), subsequent poses from 2D-3D correspondences (see lectures 2,6 and 7) During the whole SfM process, it is possible to compute the intrinsics as well (autocalibration). This is beyond the scope of this lecture With autocalibration, only 2D-2D correspondences are required easy projector calibration Consequence: Global scanner calibration by SfM Difference: 2 types of correspondences: camera to projector (phase), camera to camera (e.g. SIFT) 1/7/2015 Lecture 3D Computer Vision 44
Registration Using Structure from Motion Instead of [I 0] and relative projector pose [R t], each camera/projector gets a global pose before reconstruction Subsequent reconstruction in common frame 1/7/2015 45
Relationship to ICP Assume scanner frame given by projector frame (wlog), scanner pose is P SfM computes scanner poses P i in global frame, P i maps from world to local frame Pre-calibration: Geometry computed in local scanner frames, ICP computes mapping to common world. ICP thus computes P -1 1/7/2015 Lecture 3D Computer Vision 46
Discussion SfM enables self-calibrating scanner Metric calibration requires special attention More degrees of freedom: Individual parameter set for each device Global optimization easy Use of multiple cameras easily possible, e.g. scanning unit with 2 cameras, 1 projector Projector calibration not necessary 1/7/2015 Lecture 3D Computer Vision 47
Evaluation Evaluation with the same dataset Correspondence generation between cameras and projector: Artificial (noisy) phase for camera-projector, texture rendering for camera-camera Task for SfM: Estimate original camera poses used for rendering 1/7/2015 Lecture 3D Computer Vision 48
Error distribution For full evaluation, refer to our paper Structure from Motion in the Context of Active Scanning ICP SfM (sharp texture) SfM (blurred texture) SfM (artificial texture) 1/7/2015 Lecture 3D Computer Vision 49
Independence from Natural Features 1/7/2015 Lecture 3D Computer Vision 50
Conclusion ICP can always be used and must be used if no imagery is available ICP in general performs worse than optical features ICP failure cases: (partially) symmetric geometry Decreased performance for smooth geometry Optical feature failure cases Low quality/ sparse/ symmetric features 1/7/2015 Lecture 3D Computer Vision 51
References Besl, McKay A Method for Registration of 3D Shapes; in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, 1992 Rusinkiewicz, Levoy Efficient Variants of the ICP Algorithm, In Proceedings of the Third Intl. Conf. on 3D Digital Imaging and Modeling (2001), pp. 145-152. Dorai, Weng, Jain Registration and Integration of Multiple Object Views for 3D Model Construction, Trans. PAMI, Vol. 20, No. 1, 1998 Gelfand, Mitra, Guibas, Pottmann Robust global registration, in Proceedings of 3 rd Eurographics Symposium on Geometry Processing, 2005 1/7/2015 Lecture 3D Computer Vision 52
References Pulli Multiview registration for large data sets, 3-D Digital Imaging and Modeling, 1999 Furuakwa Multi-view reconstruction for projector camera systems based on bundle adjustment, CVPR workshops, 2009 1/7/2015 Lecture 3D Computer Vision 53