Depth Sensors Kinect V2 A. Fornaser

Transcription

1 Depth Sensors Kinect V2 A. Fornaser

2 Vision

3 Depth data It is not a 3D data, It is a map of distances

4 Not a 3D, not a 2D it is a 2.5D or Perspective 3D

5 Complete 3D - Tomography

6 2.5D 3D?... Multiplication of the devices

7 2.5D 3D?... Multiplication of the samples Rotating table

8 2.5D 3D?... Multiplication of the views, hand moved scanner (Kinect Fusion)

9 Kinect Fusion

10 Depth Sensing/Sensors

11 Depth Sensing/Sensors Cheap Good in reconstructing 3D points, not surfaces Single/multi shot/s WAS expensive Good accuracy (millimeters) Fast Single shot Old technique Cheap Required usually a structured environment Most diffused on Usually expensive Industrial level Very high accuracy Incremental reconstruction

12 Depth Sensing/Sensors

13 Oldest technology High accuracy High costs!! Most diffused as industrial level Quality control

14 3D scan of environment 3D scan of an engine: Quality control Reverse engineering

15 LIDAR - Light Detection and Ranging

16 For driving

17

18 3D from laser scanner

19 Depth Sensing/Sensors

20 Depth from Stereo Triangulation

21 Epipolar geometry The same Point /feature seen by both cameras Easy to be said, but difficult to be done

22 Epipolar constraint EPIPOLAR GEOMETRY

23

24 For driving For safety

25

26 3D from dense stereo reconstruction

27 Depth Sensing/Sensors

28

29

30 Trinocular & Structured light (Projector) 3 1 2

31 A different kind of structured light

32 Structured Light MICROSOFT KINECT V1

33

34 Region-growing Random Dot Matching 1. Detect dots ( speckles ) and label them unknown 2. Randomly select a region anchor, a dot with unknown depth a) Windowed search via normalized cross correlation along scanline 1. Check that best match score is greater than threshold; if not, mark as invalid and go to 2 b) Region growing 1. Neighboring pixels are added to a queue 2. For each pixel in queue, initialize by anchor s shift; then search small local neighborhood; if matched, add neighbors to queue 3. Stop when no pixels are left in the queue 3. Stop when all dots have known depth or are marked invalid

35 Depth Sensing/Sensors

36 Time-of-Flight (ToF) Imaging It refers to the process of measuring the depth of a scene by quantifying the changes that an emitted light signal encounters when it bounces back from objects in a scene. Two common principals: Pulsed Modulation Continuous Wave Modulation

37 Pulsed Modulation Measure distance to a 3D object by measuring the absolute time a light pulse needs to travel from a source into the 3D scene and back, after reflection Speed of light is constant and known, c = 3 10^8m/s Advantages: Direct measurement of time-of-flight High-energy light pulses limit influence of background illumination Illumination and observation directions are collinear Disadvantages: High-accuracy time measurement required Measurement of light pulse return is inexact, due to light scattering Difficulty to generate short light pulses with fast rise and fall times Usable light sources (e.g. lasers) suffer low repetition rates for pulses

38 Continuous Wave Modulation Continuous light waves instead of short light pulses Modulation in terms of frequency of sinusoidal waves Detected wave after reflection has shifted phase cross correlation Phase shift proportional to distance from reflecting surface Advantages: Variety of light sources available as no short/strong pulses required Applicable to different modulation techniques (other than frequency) Simultaneous range and amplitude images Disadvantages: In practice, integration over time required to reduce noise Frame rates limited by integration time Motion blur caused by long integration time

39 Retrieve phase shift by demodulation of received signal Demodulation by cross-correlation of received signal with emitted signal Emitted sinusoidal signal: ω modulation frequency Received signal after reflection from 3D surface: φ phase shift Cross-correlation of both signals: Distance d is equal to:

40 Kinect depth data

41 Example of usage of 3D data object recognition and localization

42 and maybe grasping

43 Hololens?

44 From prototype to an engineered device

45 Depth cameras? YES!

46 Depth for spatial mapping

47 Kinect V2

48 Kinect V2 material

49 Application examples/areas

50 V1 vs V2

51 Uncertainty

52 Uncertainty

53 Software 6 APIs overview

54 Hardware configuration

55 Special hardware configuration

56 Available data from Kinect

57 Reference systems and spaces 2D 2D 3D

58 Change in space

59 Code analysis Device Mapper Readers 3D points 2D image coordinates

60 Code analysis Search for the device Start the device Initialization of the sources

61 Grab data Data grab Copy the data into a buffer Depth to 3D points Depth to RGB img Release buffers

62 Body/Skeleton Kinect tracks up to 6 bodies Move along joint structure Projection of the body data into different spaces

63 Skeleton

64 Unity

65 Unity and Kinect DLL

66 Unity interface in action 3D colored point cloud Luca Skeleton

67 Unity wrapper to DLL calls

68 From C++ to Unity 3D points + RGB + IR Body

69 Unity wrapper to DLL calls

70 Unity wrapper to DLL calls

71 Unity wrapper to DLL calls

72 Unity structure