3D Shape and Indirect Appearance By Structured Light Transport

Transcription

1 3D Shape and Indirect Appearance By Structured Light Transport CVPR Best paper honorable mention Matthew O Toole, John Mather, Kiriakos N. Kutulakos Department of Computer Science University of Toronto By FU, Kangping Credits: K. Kutulakos, TI TM, J. Geng

2 Structured-light 3D surface imaging Projecting a narrow band of light onto a 3D shaped surface produces a line of illumination that appears distorted from other perspectives than that of the projector, and can be used for an exact geometric reconstruction of the surface shape.

3 Light Transport - Indirect light Common assumption in CV: Light travels along direct paths; Lights in natural scenes Light reflects and refracts; Undergoes specular and diffuse inter-reflections;

4 Light Transport - Indirect light Two families of transport paths dominate image formation in a projector-camera systems Epipolar paths; Non-epipolar paths; Contributions of these paths are hard to separate in images; Easy to untangle in the optical domain before acquisition takes place;

5 Experimental Results

6 Contributions A novel technique Structured Light Transport Define and address four imaging problems: One-shot indirect-only imaging; One-shot indirect-invariant imaging; Given any desired illumination, capture an image where light appears to have been transported by direct paths only; Two-shot direct-only imaging; One-shot multi-pattern imaging; Any N desired illuminations;

7 Contributions A physical device that just outputs live video A novel combination of existing off-the-shelf components; A conventional 28Hz video camera; A pair of synchronized digital micro-mirror devices (DMDs) operating at 2.7kHz to 24kHz; Optics for coupling them;

8 Contributions The first demonstration of an indirect-only video camera ; Shows how to capture views of a scene that are invariant to indirect light with just one SLT shot; Shows that any ensemble of structured-light patterns can be made robust to indirect light; Shows that SLT imaging can turn any multipattern 3D structured-light method into a oneshot technique for dynamic shape capture;

9 The Stereo Transport Matrix Light transport equation The camera and projector respond linearly to light; i: image, column vector; p: the P-pixel projected pattern, column vector T: I P instantaneous light transport matrix; T[i,p]: total radiance transported from projector pixel p to image pixel I over all possible paths; NG, R., etc. All-frequency shadows using non-linear wavelet lighting approximation. SIGGRAPH 2003

10 The Stereo Transport Matrix A projector and a camera define a stereo pair Classify elements of T into: Epipolar elements Non-epipolar elements Direct elements

11 The Stereo Transport Matrix

12 Dominance of Non-Epipolar Transport Non-epipolar dominance assumption: The non-epipolar component is very largely relative to the epipolar indirect for a broad range of scenes; Theoretically and experimentally supported.

13 Imaging by SLT Primal-dual coding Primal-dual coding primal-dual coding acquires photos governed by the above transport probing equation; l: a column vector of all ones; denotes the element-wise multiplication of two equal-sized matrices; Conventional photography: M DoF for controlling the output photo; The probing matrix: N M. M O Toole, etc. Primal-Dual Coding to Probe Light Transport. SIGGRAPH 2012

14 Imaging by SLT Primal-dual coding How to set π? How to design an imaging system that implements the equation? The key ingredient is an ability to simultaneously modulate light at its source (the projector) and its destination (the sensor). M O Toole, etc. Primal-Dual Coding to Probe Light Transport. SIGGRAPH 2012

15 Imaging by SLT Primal-dual coding How to determine m and q for a given π? Stochastic estimators for general probing; Optical probing: Opening the camera s shutter; Project pattern q(t) onto the scene; Use a semi-transparent pixel mask m(t) to modulate the light arriving at individual camera pixels; Close the shutter. M O Toole, etc. Primal-Dual Coding to Probe Light Transport. SIGGRAPH 2012

16 Imaging by SLT Primal-dual coding Prototype: To acquire a primal-dual coded photo, we project a sequence of illumination patterns onto the scene and simultaneously display a sequence of modulation patterns on the LCD panel; The camera s shutter remains open throughout this process. M O Toole, etc. Primal-Dual Coding to Probe Light Transport. SIGGRAPH 2012

17 Dominance of Non-Epipolar Transport Classify elements of T into: Epipolar elements Non-epipolar elements Direct elements Non-epipolar dominance assumption

18 Imaging by Structured Light Transport Conventional structured-light imaging - π 1 Non-epipolar blocks are dependent on patterns; Indirect-invariant imaging - π 2 Indirect-only imaging - π 3 Epipolar-only imaging - π 4 One-show, multi-pattern, indirectinvariant imaging - π 5 Partition the I image pixels into S sets and let b(i) be vinary vectors of size I indicating the pixel membership of each set. S sub-images, each of which is a view of the scene under a specific structured-light pattern in the sequence.

19 Live Structured-Light-Transport Imaging - DMD Digital micro-mirror device Microscopic mirrors, corresponding to pixels; Mirrors rotatable ±10-12 ; Toggled on: bright; Toggled off: dark; Toggled on and off quickly: grayscale;

20 Live Structured-Light-Transport Imaging LCD vs DMD LCDs are not suitable for video-rate probing: they refresh at Hz; DMD: kHz; Problem: DMDs are binary Turn the derivation of masks and projection patterns into a combinatorial optimization problem. Approach: Derive randomized decompositions of π that approximate Eq.(right) in expectation.

21 Experimental Results

22 Concluding Remarks SLT imaging offers a powerful new way to analyze the appearance of complex scenes and to boost the abilities of existing reconstruction algorithms.