Advances in 3D data processing and 3D cameras Miguel Cazorla Grupo de Robótica y Visión Tridimensional Universidad de Alicante
Contents Cameras and 3D images 3D data compression 3D registration 3D feature extraction Egomotion and reconstruction
Cameras and 3D images A sad story: Lot of work with stereo cameras. Stereo cameras are good enough for 3D, but they suffer from the lack of texture. Lot of work with 3D laser: long distance, but heavy and a bit slow and no color!. 2009: grant from Ministry for working with 3D data. 2010: search for a good 3D camera. SR4000 infrarred camera: 7.000 November of 2010: Kinect is launched. We are still crying.
Different 3D sensors
Comparison Name Indoor/outdoor Range Resolution Pixel inf. 3D sweeping unit Both 20m-50m100m 360 x 360 Stereo camera Both 5m-10m- 640 x 480 Color/Gray SR4000 Indoor (Caution!) 5m-10m 176 x 144 Infrared Kinect Indoor 7m 640 x 480 Color No
Data
3D data compression
3D data compression Kinect: resolution 640x480 ~ 300.000 3D points [X Y Z R G B] Kinect can provide 30fps HUGE AMOUNT OF DATA!!! Data compression is needed. There are several methods: Lossless/lossy methods (ussually using octrees) Progressive/non progressive methods
Proposed method Different from the ones in the literature Vicente Morell, Sergio Orts, Miguel Cazorla, Jose GarciaRodriguez Geometric 3D Point Cloud Compression. Pattern Recognition Letters (in review) Idea: if you know something about the environment, use it!!! Designed for man-made environments, mainly indoors It allows to define the level of compression
Compression step Parameter: compression level Plane extraction using Ransac For each plane extracted: Color segmentation: Transform points from 3D to 2D (using the plane) Use your favorite segmentation method (we used Kmeans) Provides some set of 3D points with similar color (and in a plane) For each 3D point set: Calculate the concavehull Apply a Delaunay triangulation Store the plane and triangles (3 vertex, statistics) Store the rest of points
Reconstruction Read the stored points For each plane P A is the area of P npoints is the points on P For each triangle t At is the area of t pointstogenerate=(at/a)*npoints Randomly generate points inside t Add color information to points
Results Compression obtained: 0.5 Distance error: 0.0074 Color error: 59.7
Experimentation
Experimentation With K=1 Compression level: 0.2, 0.5 and 0.8
Experimentation
Results
3D feature extraction Extracting features directly from 3D data Application: egomotion and reconstruction
3D feature extraction Again, 3D data is huge. We need a method to extract/model data without losing information. We'll see two combined approaches: Method for reducing data, preserving their topology Method to extract features, like small planes Using the two approches combined will improve the whole process
3D registration Registration: process to find the transformation between two objects, mainly used in 3D matching With 3D point sets, you can follow two approaches Find the transformation using raw 3D data point Extract features and apply Ransac (or similar) Ransac is fast and eliminate outliers, but sometimes it does not provide best results Raw methods are slooooow and also suffers from outliers Some methods first apply Ransac then raw methods It is directly related with the egomotion of a robot
3D registration: Ransac Find features (2D, 3D or 2½D) Find correspondences between features at consecutive frames Some of those correspondences are incorrect
3D registration: ransac Ransac: Select a number of correspondences Find the transformation which best describe the selected correspondences Calculate the number of correspondences which are correct and their error Repeat a number of iterations and/or until an error is reached
3D registration: ICP Iterative Closest Point (ICP) 1982!! We have two point sets: model M and scene S An initial transformation T Apply T to M = Mt Find closest points between Mt and S (correspondences) Find newt from the correspondences Return to 1. Until a number of iterations or error are reached
Extracting planar patches from 3D data
Planar patches Our main goal is to process 3D data from any kind of 3D sensor Trying to reduce 3D points to chuncks of planar surfaces Using a seeding approach, several points are selected and checked if they belong to a plane
Planar patches extraction From one point, a certain neighbourhood is selected That neighbourhood will be different depending on The distance Z to points The error inside the plane Calculate the covarianze matrix M Robust estimator: Singular Value Decomposition of M SVD provides 3 singular values and 3 singular vectors If the minimum singular value is much less than the other two, the point set is a plane and the two singular vectors define the plane.
Planar patches extraction This process takes less than 1 second (it depends on the sensor)
Using planar patches to get registration Now we have normals instead of points!! We want to find the transformation: We apply an ICP-like method (find correspondences then find transformation) In order to find correspondences we have to define a distance planar patches
Distance between planar patches A normal is defined by a point and a vector. where da is an angular distance and ξ is a factor to convert euclidean distances to angular Now we are able to find the closest patch to one given. Applying an ICP algorithm was not satisfactory, so we decided to disconnect rotation and traslation
Registration using planar patches: rotation Finding the rotation to align both point sets. With the normals, find the cross covarianze where wi is the weight for a given correspondence. It is calculated using the angular distance This allows to reject outliers. Find the SVD decomposition of and from it, calculate R
Registration using planar patches: translation Apply R to the model Now, the correspondences between both set make a force system, with three main directions Each correspondence is changed by a vector Using again SVD we can obtain the main directions D1, D2, D3 ti are the translation components
Complete method
Planar patches registration in progress
Processing time
Comparison with ICP
Results
Results Data: Riegl VZ-400 and a Optris PI IR camera. 11 poses, separated 40 meters, more than 7 million points each Error obtained: 0.89ms, 1.27 degrees Processing time: 1.6s patch extraction, 54.8s registration
Results
Improving the results
Some errors With some data (mainly SR4000), our method was not good Remember: the greater the error the greater neighbourhood is
Improvement of the method using GNG The error can be smoothed using Bilateral filtering or similar But we decided to try a different approach: a Growing Neural Gas (GNG) GNG is a neural network forming a map (neurons and edges between them) which expands to map the complete input data and preserves the topology
Results applying GNG to 3D data
Improvement when applying the planar patches algorithm
Results using GNG ICP Planar patches Planar patches plus GNG
Speeding up
Processing time FAIL!!!! Slooooooow GNG depends of the number of neurons used From seconds (5-20 s.) to minuts. Idea: apply GPU programming (CUDA) to speed up CUDA architecture reflects a SIMT model: Single Instruction, Multiple Threads Not all the method could be parallelize
Parallel algorithm
Results Trade-off
Conclusions Working with 3D data is amazing Always process the information in order to reduce data Sometimes, merging different technologies provides better results Do not buy SR4000!
Advances in 3D data processing and 3D cameras Thanks for your attention!!! Miguel Cazorla Grupo de Robótica y Visión Tridimensional Universidad de Alicante