Depth Camera for Mobile Devices Instructor - Simon Lucey 16-423 - Designing Computer Vision Apps
Today Stereo Cameras Structured Light Cameras Time of Flight (ToF) Camera
Inferring 3D Points Given we have prior knowledge of the, Intrinsics parameters, { j } J j=1 Extrinsic parameters, { j, j } J j=1 Corresponding points, {x j } J j=1 Question is how to estimate the 3D point w?
Inferring 3D Points ŵ =min w JX j=1 {x j pinhole[w, j, j, j ]} e.g. {x} = x 2 2
Inferring 3D Points Optimization problem is inherently non-linear due to the pinhole camera function. Can be made linear using homogeneous coordinates.
Inferring 3D Points Write j-th out the pinhole camera in homogenous coordinates, Pre-multiply with inverse of the intrinsics matrix,
Inferring 3D Points Last equation gives, Substituting back into the other two equations, Re-arranging gives the following system of equations,
Inferring 3D Points Last equation gives, Substituting back into the other two equations, Re-arranging gives the following system of equations, What is the minimum number of cameras (J)?
Stereo Camera
Stereo Camera
Stereo Camera 6.35 cm
Stereo Camera 6.35 cm What is better wide or narrow baseline?
Stereo Camera
Stereo Camera
Amazon Fire Phone Examples in Mobile
Examples in Mobile Amazon Fire Phone Why 4 cameras?
Limitations - Texture Approach only works if an image patch has texture!! X A( x) = I(x k ) I(x k + x) 2 x k 2N (x) 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 A( x) 12
Limitations - Texture Approach only works if an image patch has texture!! X A( x) = I(x k ) I(x k + x) 2 x k 2N (x) 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 A( x) 12
Today Stereo Cameras Structured Light Cameras Time of Flight (ToF) Camera
Projector vs.camera 14
Projector vs.camera Camera 14
Projector vs. Camera 15
Projector vs. Camera Projector 15
Depth from Structured Light 16
Depth from Structured Light How can we get away with one camera? 16
Depth from Structure Light 17
Depth from Structured Light 18
Prime Sense - Kinect 1.0 Camera How pattern looks like? First Region: Allows to obtain a high accurate depth surface for near objects aprox. (0.8 1.2 m) Second Region: Allows to obtain medium accurate depth surface aprox. (1.2 2.0 m). Third Region: Allows to obtain a low accurate depth surface in far objects aprox. (2.0 3.5 m). 19
Examples in Mobile 20
ItSeez - App
ItSeez - App
Limitations - Range 22
Limitations - DeFocus November 6, 2015 DRAFT (a) Scene (b) Disparity Map Figure 1.2: Illumination Defocus: In (a) a checkerboard pattern is projected onto a scene using a DMD projector. The scene consists of three planar targets at distances of 60cm, 80cm and 120cm from the projector-camera system. The pattern is focused on the nearest plane. Due to the shallow depth of field, the pattern is poorly focused at other distances. Running structured light on this scene yields poor results (b) because of illumination defocus. The foreground of the scene is recontructed accurately because it is well focused, but the rest of the scene is reconstructed 23 poorly.
Limitations - Ambient Light A sunny day on Earth can reach up to 1120Wm -2 Tabletop projector releases on average 10W of light. Spectral Irradiance (in Wm 2 nm 1 ) 2.5 2 1.5 1 0.5 Extraterrestrial Radiation Direct + Circumsolar Irradiance 0 0 500 1000 1500 2000 2500 3000 3500 4000 Wavelength (in nm) 24
Today Stereo Cameras Structured Light Cameras Time of Flight (ToF) Camera
Time of Flight Cameras Light travels at approximately a constant speed c = 3x10 8 ms -1. Measuring the time it takes for light to travel over a distance once can infer distance. Can be categorized into two types:- 1. Direct TOF - switch laser on and off rapidly. 2. Indirect TOF - send out modulated light, then measure phase difference to infer depth.
Direct - TOF Light Detection And Ranging (LiDAR) probably best example in computer vision and robotics. High-energy light pulses limit influence of background illumination. However, difficulty to generate short light pulses with fast rise and fall times. High-accuracy time measurement required. Prone to motion blur. Expensive.
Direct - TOF Light Detection And Ranging (LiDAR) probably best example in computer vision and robotics. High-energy light pulses limit influence of background illumination. However, difficulty to generate short light pulses with fast rise and fall times. High-accuracy time measurement required. Prone to motion blur. Expensive.
Direct TOF - Zebedee CSIRO
Direct TOF - Zebedee CSIRO
Indirect - TOF Continuous light waves instead of short light pulses. Modulation in terms of frequency of sinusoidal waves. Detected wave after reflection has shifted phase. Phase shift proportional to distance from reflecting surface.... continuous wave... 20 MHz Emitter Detector Phase Meter...... phase shift 3D Surface
Indirect - TOF
Indirect TOF
Examples - Mobile
REAL3 TM Image Sensor
The Future
The Future