Computational Optical Imaging - Optique Numerique. -- Active Light 3D--

Transcription

1 Computational Optical Imaging - Optique Numerique -- Active Light 3D-- Winter 2013 Ivo Ihrke with slides by Thorsten Thormaehlen

2 Overview 3D scanning techniques Laser triangulation Structured light Photometric stereo Calibration Time-of-Flight Transient Imaging

3 Active Light - 3D Scanning Laser Scanning

4 Range scanning Generate range from stereo information Scales from desktop to whole body scanners

5 Range scanning systems Passive: n 2 cameras; match images

6 Optical Triangulation passive setup: match corresponding images left sensor object right sensor

7 Active Techniques project some additional signal into the scene requires signal source, and additional calibration often more or less direct measurements typically higher precision Examples: range scanning matting applications measurement of reflection properties

8 Range scanning systems Active: Laser light source Sweep & track reflection (triangulation) Project a line instead of a single point

9 Optical triangulation active setup: track light spot laser source z b tan( a) tan( b ) a object base length b sensor b (x,z) ray intersection z

10 Laser triangulation Source: The Digital Michelangelo Project, Stanford University

11 Laser triangulation light plane object camera laser line of sight Depth from ray-plane triangulation: Intersect line of sight with light plane (canonical camera)

12 Laser triangulation Advantages simple approach works quite reliable accuracy proportional to working volume (typical is ~1000:1) scales down to small working volume (e.g. 5 cm. working volume, 50 m. accuracy) Disadvantages long scanning time (no real-time capture of moving objects) does not scale up (baseline too large ) two-lines-of-sight problem (shadowing from either camera or laser) triangulation angle: lower accuracy if too small, shadowing if too big (useful range: 15-30) needs stable reliable calibration with good mechanics, thus often expensive to buy

13 Digital Michelangelo Project, motorized axes extensions for tall statues cyberware scanner (laser triangulation)

14 Digital Michelangelo Project, height of gantry: weight of gantry: 7.5 meters 800 kilograms Source: The Digital Michelangelo Project, Stanford University

15 Digital Michelangelo Project, working in the museum scanning geometry scanning color Source: The Digital Michelangelo Project, Stanford University

16 Digital Michelangelo Project, individually aimed scans 2 billion polygons 7,000 color images 32 gigabytes 30 nights of scanning 22 people Source: The Digital Michelangelo Project, Stanford University

17 Digital Michelangelo Project, photograph 1.0 mm computer model Source: The Digital Michelangelo Project, Stanford University

18 Digital Michelangelo Project, St. Matthew David Forma Urbis Romae Source: The Digital Michelangelo Project, Stanford University

19 Range Scanners Optical Considerations Laser Range Scanners focal plane selection Scheimpflug principle tilt-shift lenses focal plane Scheimpflug principle application in range scanning extend depth of field

20 Tilt-Shift Lens Photographic Examples

21 Range Scanning Problems - Shadowing sensor light source sensor can t see lit point point in shadow longer baseline: more shadowing shorter baseline: less precision

22 Range Scanning Problems Acquired Range Image a single range image is acquired for each view Artifacts: ribbed lots of holes due to surface properties due to occlusion Solution: Merging of multiple scans -> (ICP, Gaël Guennebaud)

23 Range Scanning Problems - Beam location error Assumption: Reflection has Gaussian shape ideal reflection partial occlusion partial illumination Corrupt due to occlusion or texture Biased depth estimate

24 Beam location error Possible solution: Spacetime analysis [Curless, Levoy 95] Estimate when beam centered on spot

25 DIY Range Scanners Laser Range Scanning Cheapo Version [Winkelbach06] hand-held line laser known background geometry need two planes that are not co-planar known camera calibration compute laser plane from lines on the background planes triangulate by ray-plane intersection

26 Active Light - 3D Scanning Structured Light

27 Structured Light Idea: increase speed by projecting multiple stripes But which stripe is which? Answer: Coded stripe patterns Binary code Gray code Color code

28 Range scanning systems Active: Structured light source Analyze image of stripe pattern

29 Structured Lighting: Swept-Planes Revisited Point Grey Flea2 ( x 768) Mitsubishi XD300U ( x 768) Swept-plane scanning recovers 3D depth using ray-plane intersection Use a data projector to replace manually-swept laser/shadow planes How to assign correspondene from projector planes to camera pixels? Solution: Project a spatially- and temporally-encoded image sequence What is the optimal image sequence to project?

30 Structured Lighting: Binary Codes Point Grey Flea2 ( x 768) Mitsubishi XD300U ( x 768) Binary Image Sequence [Posdamer and Altschuler 1982] Each image is a bit-plane of the binary code for projector row/column Minimum of 10 images to encode 1024 columns or 768 rows In practice, 20 images are used to encode 1024 columns or 768 rows Projector and camera(s) must be synchronized

31 Structured Light - Binary Coding Example:3 binary-encoded patterns which allows the measuring surface to be divided in 8 sub-regions projected over time pattern 3 pattern 2 pattern 1

32 Structured Light - Binary Coding Assign each stripe a unique illumination code over time [Posdamer 82] Time Space

33 Structured Light - Binary Coding projected over time pattern 3 Codeword of this píxel: identifies the corresponding pattern stripe pattern 2 pattern 1

34 Structured Lighting: Gray Codes Point Grey Flea2 ( x 768) Mitsubishi XD300U ( x 768) Gray Code Image Sequence Each image is a bit-plane of the Gray code for each projector row/column Requires same number of images as a binary image sequence, but has better performance in practice [Inokuchi 1984] Bin2Gray(B,G) 1 G B 2 for i n-1 downto 0 3 G[i] B[i+1] xor B[i]

35 Binary vs Gray Codes Decimal Binary Gray Code

36 Binary vs Gray Codes Binary Gray Code pattern 3 pattern 2 pattern 1

37 Gray Codes: Decoding Performance Recovered Rows Recovered Columns 3D Reconstruction using Structured Light [Inokuchi 1984] Implemented using a total of 42 images (2 to measure dynamic range, 20 to encode rows, 20 to encode columns) Individual bits assigned by detecting if bit-plane (or its inverse) is brighter Decoding algorithm: Gray code binary code integer row/column index

38 Calibration of a Projector-Camera System Camera calibration (matlab and OpenCV) Extention for projector calibration (matlab) Idea: Consider the projector as an inverse camera, which maps 2D image points into 3D rays, thus making the calibration of a projector the same as that of a camera

39 Calibration of a Projector-Camera System Calibrate the camera using [Zhang 2000] Recover calibration plane in camera coordinate system Project a checkerboard on the calibration board and detect corners Apply ray-plane intersection to recover 3D position for each projected corner Calibrate the projector using the correspondences between the 2D points of the projector image and the recovered 3D position of projected points (Zhang s method).

40 Dynamic Scanning Single-shot patterns (N-arrays, grids, random, etc.) Spatial encoding strategies [Chen et al. 2007] De Bruijn sequences [Zhang et al. 2002] Pseudorandom and M-arrays [Griffin 1992] J. Salvi, J. Pagès, and J. Batlle. Pattern Codification Strategies in Structured Light Systems. Pattern Recognition, 2004 Phase-shifting [Zhang Ivoet Ihrke al. 2004] / Winter 2013

41 Popular example: Kinect I [Matthew Fisher] [Andres Reza]

42 Continuum of Triangulation Methods Multi-stripe Multi-frame Single-stripe Single-frame Slow, robust Fast, fragile

43 Active Light Photometric Stereo The Basics and Some Hints Towards More Complex Issues

44 Photometric Stereo Idea: One camera, multiple (known) light sources s 1 s 2 n s 3 v in matrix form Lambertian reflection

45 The Reflectance Map Reflectance Map: image intensity of a pixel as a function of surface orientation of the corresponding scene point p, q,1 p s, qs,1 s p, q,1 p q,1 s, s Surface normal Light source direction cos s p, q,1 p, q,1 1 p 1 p 2 2 q q 2 1 p s 2 p q s s 1 p q 1 p 2 s 2 s s q q 2 s 2 s Angle between surface normal and direction to point light source

46 The Reflectance Map II Reflectance map of Lambertian surface for p s 0.2, q 0. 4 s R(p,q): Reflectance for all surface orientations (p,q) for a given light source distribution and a given surface material Lambertian surface R p, q cos s 1 p 1 p 2 q s 2 p q s q 1 p 2 s q 2 s

47 Photometric Stereo Point light source: lines of constant reflectance in reflectance map defined by second-order polynomial For each image point (x,y), the possible surface orientation (p,q) is restricted to lie on the curve defined by the polynomial 3 different point sources give 3 second-order curves

48 Photometric Stereo II Light source from left Light source from top Light source from right

49 Photometric Stereo III p,q: surface normal direction For each pixel, the 3 measured intensities corresponding to the 3 different illumination directions lie on 3 different curves in the reflectance map that intersect in one point

50 Photometric Stereo IV ), ( ), ( q p R y x E 2 2 1,1), ( ˆ q p q p n 2 2 1,1), ( ˆ i i i i i q p q p s i i i E n s ˆ ˆ z y x z y x z y x s s s s s s s s s 3, 3, 3, 2, 2, 2, 1, 1, 1, ˆ ˆ ˆ s s s S E E E E n S E ˆ E S n ˆ 1 Surface normal Point light source direction Lambertian reflection Image irradiance 3 illumination directions/ substitution : surface albedo Linear set of 3 equations per pixel

51 Photometric Stereo Advantages A normal value for each pixel with very few images Approach captures details, like pores, wrinkles, or spots Disadvantages Normals must be integrated to get a 3D model (errors will accumulate) Needs controlled lighting or robust estimation of light direction

52 Photometric Stereo V Real Geometry Normal Map

53 Integrating the normals Normals of a height field surface: Tangent space is spanned by Laplacian operator discretize

54 Integrating the normals Solve system Compute right-hand side numerically Include Boundary conditions: e.g. von Neumann = 0 Modify system above it corresponds to boundary on the

55 Photometric Stereo VI Reconstruct normal directions Integrate normal directions to obtain depth map Triangulate depth map

56 Photometric Stereo Assumptions Assumptions: Surface has Lambertian reflectance characteristics Illumination directions are known Illumination sources are distant from object Secondary illumination is insignificant There are no cast shadows Distant camera (orthographic projection) Extensions Non-local illumination or views Non-Lambertian BRDF

57 Active Light Time-of-Flight Imaging

58 Time-of-Flight Pulse-based time-of-flight scanners [Gvili03] NOT triangulation based short infrared laser pulse is sent from camera reflection is recorded in a very short time frame (ps) results in depth profile (intensity image)

59 Time-of-Flight Pulse-based time-of-flight scanner examples accuracy 1-2 cm in a range of 4 7 m applications: "depth keying" replaces chroma keying 3D interaction large scale 3D scanning (LIDAR light detection and ranging)

60 Canesta/3DV Kinect 2 (?)

61 PMDs Photonic Mixer Devices Also called multi-bucket sensors Chip layout Schematic view Electric symbol Fast on-chip modulation [Luan 01]

62 PMDs Photonic Mixer Devices Working principle: Measure phase difference of emitted, modulated signal and received one [Metrilus] Current hardware: ~20MHz modulation frequency in practice square wave

63 [slide by Felix Heide] Photonic Mixer Device (PMD) sensor: P PMD Sensor dt

64 [slide by Felix Heide] Single-path Time-of-Flight depth imaging: P PMD Sensor dt

65 [slide by Felix Heide] Single-path Time-of-Flight depth imaging: P PMD Sensor dt distance-dependent correlation

66 Time-of-Flight Cameras LIDAR University of Washington NASA PMD Panasonic Fotonic ARTTS MESA See [Kolb et al. 10], EG STAR for more details

67 Active Light Transient Imaging

68 [slide by Felix Heide] What is Transient Imaging? a.k.a. Light-in-Flight Imaging Ultrafast imaging of nonstationary light distribution in scenes Tracking of wavefronts of light as they propagate in the scene Equivalent to imaging ultrashort pulses of light t =

69 [slide by Felix Heide] Applications of Transient Imaging Many applications by analyzing a transient image! Understanding light transport: Surface reflectance capture: Decomposing light transport: Reconstructing hidden object geometry (and motion): Wu et al 2012 Velten et al 2013 Naik et al 2011 Velten et al 2012 Pandharkar et al 2011

70 Visualizing Light in Motion Repetitive Event, 1012 FPS, 672 x 1000 pixel resolution, 1 ns+ capture time Light moves 0.6 mm per frame Ability to see light transport [Velten et al. 13]

71 Streak Cameras Picosecond time resolution 1D: 1x672 pixels Result as 2D image ( Streak Photo ) Hamamatsu [Velten et al. 13]

72 Experimental Setup Ti:Sapphire Laser [Velten et al. 13] Synchronization Camera Object Scanning Unit

73 Actual Setup [Velten et al. 13]

74 Streak Camera Picture Time [Velten et al. 13] Space

75 [Velten et al. 13]

76 [Velten et al. 13]

77 Example Result [Velten et al. 13]

78 [Velten et al. 13]

79 Scattering in Real Scenes [Velten et al. 13]

80 [Velten et al. 13]

81 Transient Imaging With PMD Devices linked to the multipath or mixed pixel problem PMD Panasonic Fotonic ARTTS MESA

82 [slides by Felix Heide] Multi-path contributions: P2 PMD Sensor P1 dt

83 [slides by Felix Heide] Recovering multi-path contributions: P2 PMD Sensor P1 dt

84 [slides by Felix Heide] Recovering multi-path contributions: P2 PMD Sensor P1 dt

85 [slides by Felix Heide] Recovering multi-path contributions: P2 PMD Sensor P1 dt

86 [slides by Felix Heide] Image formation model: Relation of Transient Image to PMD measurement: Linear system is ill-posed: assume model for temporal response (signal sparsity) Mixture Model: Gaussian + Exponential and spatial smoothness solve non-linear system multiple measurements Spatial coherence Temporal model

87 [slides by Felix Heide] PMDTechnologies CamBoard nano 2 Too slow LED unit Fixed frequency PLL 25MHz Control FPGA PMD sensor ADC

88 [slides by Felix Heide] Modifications to the hardware Prototype: Max mod. frequency 180 MHz, stable up to 110 MHz Novatech DDS9m Laser unit ic-hg + LPC826 2-ch. function generator MHz Control FPGA Control MOD PMD sensor TRIG ADC

89 Prototype setup MOD in Used for results in paper Camera Power TRIG out Laser unit Function generator

90 [slides by Felix Heide] Result Discoball : Color coding of strongest component:

91 [slides by Felix Heide] Results Discoball : Different time-steps:

92 [slides by Felix Heide] Results Discoball : Different time-steps:

93 [slides by Felix Heide] Results Discoball : Different time-steps:

94 [slides by Felix Heide] Results Discoball : Different time-steps:

95 [slides by Felix Heide] Results Discoball : Different time-steps:

96 [slides by Felix Heide] Result Bottles : Color coding of strongest component:

97 [slides by Felix Heide] Emerging Technologies prototype Camera unit Light source AD9958 DDS (function generator) MHz Control FPGA Control MOD PMD sensor TRIG ADC

98 [slides by Felix Heide] Camera Light source

99 [slides by Felix Heide] Results: People

100 References Curless, Levoy, Better optical triangulation through spacetime analysis, CVPR 1995 Levoy et al., The digital Michelangelo project: 3D scanning of large statues, SIGGRAPH 2000 [Gvili03] R. Gvili, A. Kaplan, E. Ofek, G. Yahav, "Depth Keying", SPIE ISOE 5006, 2003, pp Luan, Experimental Investigation of Photonic Mixer Device and Development of TOF 3D Ranging Systems Based on PMD Technology, PhD Thesis, Siegen University, 2001 Velten et al. Femto-photography: capturing and visualizing the propagation of light, SIGGRAPH 2013 Heide et al. Low-budget Transient Imaging using Photonic Mixer Devices., SIGGRAPH 2013