3D Computer Vision 1

Transcription

1 3D Computer Vision 1

2 Multiview Stereo

3 Multiview Stereo

4 Multiview Stereo

5 Shape from silhouette

6 Shape from silhouette

7 Shape from silhouette

8 Shape from silhouette

9 Structured light

10 Kinect: Structured infrared light

11 Photometry stereo Helmholtz Stereopsis

12 3D Imaging with ToF Camera

13 Time-of-Flight Principle Time-of-flight of Light Distance : It is not simple to measure the flight time directly at each pixel of any existing image sensor Reflected IR shows phase delay proportional to the distance from the camera.

14 Phase Delay Measurement Q1 through Q4 are the amount of electrons measured at each corresponding time. t(d In real situations, it is difficult to sense electric charge at certain time instance

15 Phase Delay Measurement Distance arctan 2 ( 2 Q Q Q Q c d t c arctan 2 arctan 2 c c Assumption: Single reflected IR signal In principle, amplitude of the reflected IR does not affect the depth calculation.

16 Multiple IR Signals - Large Sensor Pixel - Scattering - Multipath - Motion Blur - Transparent Object In real situations, multiple reflected IR signals with different phase delays & amplitudes can be superposed. ( ( ( ( arctan 2 ( c d t We do not know how many IR signals will be superposed.

17 Large Sensor Pixel In order to increase sensitivity, - large pixel size or pixel binning IR signal #1 IR signal #2 ( ( ( ( arctan 2 ( c d t

18 Light Scattering Multiple light reflections between the lens and the sensor Light scattering [1] [1] Real-time scattering compensation for time-of-flight camera, CVS07

19 Multipath Errors IR LED Sensor Multipath Interference Depth error in concave objects ( ( ( ( arctan 2 ( c d t

20 Motion Blur Moving camera/object within single integration time make wrong depth calculation Moving Object Moving Object Image sensor

21 Motion Blur The characteristic of Tof motion blur is different from color Overshoot Blur Undershoot Blur Overshoot Blur

22 Transparent Object 2-Layer approximation of transparent object (1 ( (1 ( (1 ( (1 ( arctan 2 ( c d t - Sometimes 2-Layer is not enough - Multiple reflection between objects (when they are close - In most cases, they have specular surface

23 Integration time-related Error Due to the variation of the number of collected electrons during the integration time the repeatability of each depth point varies Integration Time: 30(ms Integration Time: 80(ms

24 y(pixel Error(m Amplitude-related Errors Due to the non-uniformity of IR illumination and reflectivity variation of objects use a polynomial fitting model x(pixel Amplitude image of a planar object with a ramp image. Parts of the ramp are selected for calibration (blue rectangle Amplitude The depth samples (blue and the fitted model (green to the error 1

25 Kinect Principle (1/3 Basically, it is based on structured light principle IR Speckle Pattern

26 Kinect Principle (2/3 0. Calibrate source and detector 1. Known IR pattern is projected from the source 2. Detector identify each dot (or set of dots 3. Triangulate to calculate depth

27 Kinect Principle (3/3 - Random speckles identify x,y locations - Orientation and shape of the speckles change along distance identify z location

28 Depth/Point Cloud Processing 3D Features 3D Filtering Registration Surface Processing 28

29 Depth Distortion Upon Materials Conventional approaches assume the Lambertian materials. Various surface materials exhibit the complex light interaction, causing the non-linear distortion on light transport. Depth cameras suffer from the depth distortion upon material properties. The type of distortion varies upon the sensing principle of depth cameras. 29

30 Depth Cameras We provide the distortion analysis based on two sensor types: A Time-of-Flight and a structured light sensor [Swissranger] [Kinect] 30

31 Depth Distortion Lambertian Material affects the sensing performance (Lambertian ToF depth camera Structured light depth camera IR LED Projector Sensor All existing 3D sensing techniues are limited to Lambertian object. Sensor 31

32 Depth Distortion Specularity Non-Lambertian materials causes the failure in sensing reflected signal (Specularity ToF depth camera Structured light depth camera IR LED Projector Sensor Sensor 32

33 Depth Distortion Translucency Non-Lambertian materials causes the failure in sensing reflected signal (Translucency ToF depth camera Structured light depth camera IR LED IR LED Projector Sensor Sensor 33

34 Depth Distortion Global Illumination Complex illumination affects the sensing performance (Global Illumination ToF depth camera Structured light depth camera IR LED IR LED Projector Sensor Sensor 34

35 Color-Depth Calibration

36 Given a calibrated TOF-Stereo system Each TOF point P T defines a correspondence between P L and P R

37 Correspondences (samples obtained by using the calibration parameters each correspondence comes from a TOF point different color -> different depth

38 Correspondences (samples obtained by using the calibration parameters each correspondence comes from a TOF point different color -> different depth

39 TOF-to-Left Mapping We use the left image as reference

40 TOF-to-Left Mapping Resolution mismatch

41 TOF-to-Left Mapping Left-to-Tof Occlusions Left-to-Tof Occlusions: the depth decreases from left to right

42 TOF-to-Left Mapping Tof-to-Left Occlusions Tof-to-Left Occlusions: the depth increases from left to right

43 Paper List 1 [Depth from Stereo] DeMoN: Depth and Motion Network for Learning Monocular Stereo Ummenhofer et al. CVPR [RGBD for object recog.] Multimodal Deep Learning for Robust RGB-D Object Recognition Eitel et al. IROS [Multiview 3D] RayNet : Learning Volumetric 3D Reconstruction with Ray Potential Paschalidou et al. CVPR [Multiview 3D] 3D-R2N2: A Unifed Approach for Single and Multi-view 3D Object Reconstruction Choy et al. ECCV