Advances in 3D data processing and 3D cameras

Transcription

1 Advances in 3D data processing and 3D cameras Miguel Cazorla Grupo de Robótica y Visión Tridimensional Universidad de Alicante

2 Contents Cameras and 3D images 3D data compression 3D registration 3D feature extraction Egomotion and reconstruction

3 Cameras and 3D images A sad story: Lot of work with stereo cameras. Stereo cameras are good enough for 3D, but they suffer from the lack of texture. Lot of work with 3D laser: long distance, but heavy and a bit slow and no color!. 2009: grant from Ministry for working with 3D data. 2010: search for a good 3D camera. SR4000 infrarred camera: November of 2010: Kinect is launched. We are still crying.

4 Different 3D sensors

5 Comparison Name Indoor/outdoor Range Resolution Pixel inf. 3D sweeping unit Both 20m-50m100m 360 x 360 Stereo camera Both 5m-10m- 640 x 480 Color/Gray SR4000 Indoor (Caution!) 5m-10m 176 x 144 Infrared Kinect Indoor 7m 640 x 480 Color No

6 Data

7 3D data compression

8 3D data compression Kinect: resolution 640x480 ~ D points [X Y Z R G B] Kinect can provide 30fps HUGE AMOUNT OF DATA!!! Data compression is needed. There are several methods: Lossless/lossy methods (ussually using octrees) Progressive/non progressive methods

9 Proposed method Different from the ones in the literature Vicente Morell, Sergio Orts, Miguel Cazorla, Jose GarciaRodriguez Geometric 3D Point Cloud Compression. Pattern Recognition Letters (in review) Idea: if you know something about the environment, use it!!! Designed for man-made environments, mainly indoors It allows to define the level of compression

10 Compression step Parameter: compression level Plane extraction using Ransac For each plane extracted: Color segmentation: Transform points from 3D to 2D (using the plane) Use your favorite segmentation method (we used Kmeans) Provides some set of 3D points with similar color (and in a plane) For each 3D point set: Calculate the concavehull Apply a Delaunay triangulation Store the plane and triangles (3 vertex, statistics) Store the rest of points

11 Reconstruction Read the stored points For each plane P A is the area of P npoints is the points on P For each triangle t At is the area of t pointstogenerate=(at/a)*npoints Randomly generate points inside t Add color information to points

12 Results Compression obtained: 0.5 Distance error: Color error: 59.7

13 Experimentation

14 Experimentation With K=1 Compression level: 0.2, 0.5 and 0.8

15 Experimentation

16 Results

17 3D feature extraction Extracting features directly from 3D data Application: egomotion and reconstruction

18 3D feature extraction Again, 3D data is huge. We need a method to extract/model data without losing information. We'll see two combined approaches: Method for reducing data, preserving their topology Method to extract features, like small planes Using the two approches combined will improve the whole process

19 3D registration Registration: process to find the transformation between two objects, mainly used in 3D matching With 3D point sets, you can follow two approaches Find the transformation using raw 3D data point Extract features and apply Ransac (or similar) Ransac is fast and eliminate outliers, but sometimes it does not provide best results Raw methods are slooooow and also suffers from outliers Some methods first apply Ransac then raw methods It is directly related with the egomotion of a robot

20 3D registration: Ransac Find features (2D, 3D or 2½D) Find correspondences between features at consecutive frames Some of those correspondences are incorrect

21 3D registration: ransac Ransac: Select a number of correspondences Find the transformation which best describe the selected correspondences Calculate the number of correspondences which are correct and their error Repeat a number of iterations and/or until an error is reached

22 3D registration: ICP Iterative Closest Point (ICP) 1982!! We have two point sets: model M and scene S An initial transformation T Apply T to M = Mt Find closest points between Mt and S (correspondences) Find newt from the correspondences Return to 1. Until a number of iterations or error are reached

23 Extracting planar patches from 3D data

24 Planar patches Our main goal is to process 3D data from any kind of 3D sensor Trying to reduce 3D points to chuncks of planar surfaces Using a seeding approach, several points are selected and checked if they belong to a plane

25 Planar patches extraction From one point, a certain neighbourhood is selected That neighbourhood will be different depending on The distance Z to points The error inside the plane Calculate the covarianze matrix M Robust estimator: Singular Value Decomposition of M SVD provides 3 singular values and 3 singular vectors If the minimum singular value is much less than the other two, the point set is a plane and the two singular vectors define the plane.

26 Planar patches extraction This process takes less than 1 second (it depends on the sensor)

27 Using planar patches to get registration Now we have normals instead of points!! We want to find the transformation: We apply an ICP-like method (find correspondences then find transformation) In order to find correspondences we have to define a distance planar patches

28 Distance between planar patches A normal is defined by a point and a vector. where da is an angular distance and ξ is a factor to convert euclidean distances to angular Now we are able to find the closest patch to one given. Applying an ICP algorithm was not satisfactory, so we decided to disconnect rotation and traslation

29 Registration using planar patches: rotation Finding the rotation to align both point sets. With the normals, find the cross covarianze where wi is the weight for a given correspondence. It is calculated using the angular distance This allows to reject outliers. Find the SVD decomposition of and from it, calculate R

30 Registration using planar patches: translation Apply R to the model Now, the correspondences between both set make a force system, with three main directions Each correspondence is changed by a vector Using again SVD we can obtain the main directions D1, D2, D3 ti are the translation components

31 Complete method

32 Planar patches registration in progress

33 Processing time

34 Comparison with ICP

35 Results

36 Results Data: Riegl VZ-400 and a Optris PI IR camera. 11 poses, separated 40 meters, more than 7 million points each Error obtained: 0.89ms, 1.27 degrees Processing time: 1.6s patch extraction, 54.8s registration

37 Results

38 Improving the results

39 Some errors With some data (mainly SR4000), our method was not good Remember: the greater the error the greater neighbourhood is

40 Improvement of the method using GNG The error can be smoothed using Bilateral filtering or similar But we decided to try a different approach: a Growing Neural Gas (GNG) GNG is a neural network forming a map (neurons and edges between them) which expands to map the complete input data and preserves the topology

41 Results applying GNG to 3D data

42 Improvement when applying the planar patches algorithm

43 Results using GNG ICP Planar patches Planar patches plus GNG

44 Speeding up

45 Processing time FAIL!!!! Slooooooow GNG depends of the number of neurons used From seconds (5-20 s.) to minuts. Idea: apply GPU programming (CUDA) to speed up CUDA architecture reflects a SIMT model: Single Instruction, Multiple Threads Not all the method could be parallelize

46 Parallel algorithm

47 Results Trade-off

48 Conclusions Working with 3D data is amazing Always process the information in order to reduce data Sometimes, merging different technologies provides better results Do not buy SR4000!

49 Advances in 3D data processing and 3D cameras Thanks for your attention!!! Miguel Cazorla Grupo de Robótica y Visión Tridimensional Universidad de Alicante