Lecture 8 Active stereo & Volumetric stereo

Transcription

1 Lecture 8 Active stereo & Volumetric stereo Active stereo Structured lighting Depth sensing Volumetric stereo: Space carving Shadow carving Voxel coloring Reading: [Szelisky] Chapter 11 Multi-view stereo S. Savarese, M. Andreetto, H. Rushmeier, F. Bernardini and P. Perona, 3D Reconstruction by Shadow Carving: Theory and Practical Evaluation, International Journal of Computer Vision (IJCV), 71(3), , 2006 Seitz, S. M., & Dyer, C. R. (1999). Photorealistic scene reconstruction by voxel coloring.international Journal of Computer Vision, 35(2), Silvio Savarese Lecture 7-12-Feb-18

2 Traditional stereo P p p O 1 Camera O 2 Camera What s the main problem in traditional stereo? We need to find correspondences!

3 Active stereo (point) P image projector virtual image p p Projector (e.g. DLP) O 2 Camera Replace one of the two cameras by a projector - Projector geometry calibrated - What s the advantage of having the projector? Any limitation?? Correspondence problem solved!

4 Active stereo (stripe) P projector virtual image p S l p image s s Projector (e.g. DLP) - Projector and camera are parallel - Correspondence problem is solved! O Camera

5 Calibrating the system projector virtual image image Projector (e.g. DLP) - Use calibration rig to calibrate camera and localize rig in 3D - Project patterns on rig and calibrate projector O Camera

6 Laser scanning Digital Michelangelo Project (1990) Optical triangulation Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning Source: S. Seitz

7 Laser scanning The Digital Michelangelo Project, Levoy et al. Source: S. Seitz

8 Active stereo (shadows) J. Bouguet & P. Perona, 99 Light source O - 1 camera,1 light source - very cheap setup - calibrated the light source

9 Active stereo (shadows)

10 Limitations of Laser scanning Slow Cannot capture deformations in time Source: S. Seitz

11 Active stereo (color-coded stripes) L. Zhang, B. Curless, and S. M. Seitz 2002 S. Rusinkiewicz & Levoy 2002 projector LCD or DLP displays - Dense reconstruction - Correspondence problem again - Get around it by using color codes O

12 L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multipass Dynamic Programming. 3DPVT 2002 Active stereo (color-coded stripes) Rapid shape acquisition: Projector + stereo cameras

13 Active stereo the kinect sensor Infrared laser projector combined with a CMOS sensor Captures video data in 3D under any ambient light conditions. Pattern of projected infrared points to generate a dense 3D image Depth map Source: wikipedia

14 Lecture 8 Active stereo & Volumetric stereo Active stereo Structured lighting Depth sensing Volumetric stereo: Space carving Shadow carving Voxel coloring Silvio Savarese Lecture 7-12-Feb-18

15 Traditional Stereo P p p O 1 O 2 Goal: estimate the position of P given the observation of P from two view points Assumptions: known camera parameters and position (K, R, T)

16 Traditional Stereo P p p O 1 O 2 Subgoals: 1. Solve the correspondence problem 2. Use corresponding observations to triangulate

17 Volumetric stereo Working volume P O O Hypothesis: pick up a point within the volume 2. Project this point into 2 (or more) images 3. Validation: are the observations consistent? Assumptions: known camera parameters and position (K, R, T)

18 Consistency based on cues such as: - Contours/silhouettes à Space carving - Shadows à Shadow carving - Colors à Voxel coloring

19 Lecture 8 Active stereo & Volumetric stereo Active stereo Structured lighting Depth sensing Volumetric stereo: Space carving Shadow carving Voxel coloring Reading: [Szelisky] Chapter 11 Multi-view stereo Silvio Savarese Lecture 7-12-Feb-18

20 Contours/silhouettes Contours are a rich source of geometric information

21 Apparent Contours Projection of the locus of points on the object surface which separate the visible and occluded parts on the surface [sato & cipolla] apparent contour Camera Image

22 Silhouettes A silhouette is defined as the area enclosed by the apparent contours silhouette Camera Image

23 Detecting silhouettes Object Easy contour segmentation Image Camera

24 Detecting silhouettes

25 How can we use contours? Silhouette Object Visual cone Camera Camera Center Image Object apparent contour

26 How can we use contours? Silhouette Object Visual cone Camera Image Object apparent contour Object s silhouette Visual cone 2D slice: Object Camera Image line

27 How can we use contours? View point 1 Object Object Camera intrinsics are known We know location and pose of the camera as it moves around the object Object estimate (visual hull) View point 2

28 How to perform visual cones intersection? Decompose visualcone inpolygonalsurfaces (amongothers: Reed andallen 99)

29 Space carving [ Martin and Aggarwal (1983) ] Using contours/silhouettes in volumetric stereo, also called space carving Voxel à empty or full

30 Computing Visual Hull in 2D

31 Computing Visual Hull in 2D

32 Computing Visual Hull in 2D

33 Computing Visual Hull in 2D Visual hull: an upper bound estimate Consistency: A voxel must be projected into a silhouette in each image

34 Space carving has complexity O(N 3 ) N Any suggestion for speeding this process up? Octrees! (Szeliski 93) N N

35 Complexity Reduction: Octrees 1 1 1

36 Complexity Reduction: Octrees Subdiving volume in sub-volumes of progressive smaller size 2 2 2

37 Complexity Reduction: Octrees 2 2 2

38 Complexity Reduction: Octrees

39 Complexity reduction: 2D example 4 elements analyzed in this step

40 Complexity reduction: 2D example 16 elements analyzed in this step

41 Complexity reduction: 2D example 52 elements are analyzed in this step

42 Complexity reduction: 2D example 16x34=544 voxels are analyzed in this step = 617 voxels have been analyzed in total (rather than 32x32 = 1024)

43 Advantages of Space carving Robust and simple No need to solve for correspondences

44 Limitations of Space carving Accuracy function of number of views Not a good estimate What else?

45 Limitations of Space carving Concavities are notmodeled Concavity For 3D objects: Are all types of concavities problematic?

46 Limitations of Space carving Concavities are notmodeled Concavity Visual hull (hyperbolic regions are ok) -- see Laurentini (1995)

47 Space carving: A Classic Setup Camera Object Courtesy of Seitz & Dyer Turntable

48 Space carving: A Classic Setup

49 Space carving: Experiments 30 cm 24 poses (15 O ) voxel size = 1mm

50 Space carving: Experiments 24 poses (15 O ) voxel size = 2mm 6cm 10cm

51 Space carving: Conclusions Robust Produce conservative estimates Concavities can be a problem Low-end commercial 3D scanners blue backdrop turntable

52 Lecture 8 Active stereo & Volumetric stereo Active stereo Structured lighting Depth sensing Volumetric stereo: Space carving Shadow carving Voxel coloring Silvio Savarese Lecture 7-12-Feb-18

53 Shape from Shadows Self-shadows are visual cues for shape recovery Self-shadows indicate concavities (no modeled by contours)

54 Shadow carving: The Setup Camera Object Array of lights Turntable

55 Shadow carving: The Setup Camera Object Array of lights Turntable

56 Shadow carving: The Setup [Savarese et al 01] Camera Object Array of lights Turntable

57 Shadow carving [Savarese et al. 2001] Self-shadows + Object s upper bound

58 Algorithm: Step k Camera Image line Light source Image Image shadow virtual light image Upper-bound from step k-1 Object

59 Algorithm: Step k Camera projection of the shadow points in the image to the upper bound estimate of the object Image line virtual image shadow Light source Image? Image shadow virtual light image Upper-bound from step k-1 Theorem: A voxel that projects into an image shadow AND an virtual Object image shadow cannot belong to the object.

60 Algorithm: Step k Camera Image line Virtual image shadow Light source Image Properties: No further volume can be removed Carving process always conservative Image shadow virtual light image Upper-bound from step k-1 Proof of correctness Consistency: In order for a voxel to be removed it must project Object into both image shadow and virtual image shadow

61 Algorithm: Step k Camera Image line Virtual image shadow Light source Image Image shadow virtual light image Upper-bound from step k-1 Complexity? O(2N 3 ) Object

62 Simulating the System - 24 positions - 4 lights - 72 positions - 8 lights

63 Results - 16 positions - 4 lights Space carving Shadow carving

64 Results Space carving Shadow carving

65 Shadow carving: Summary Produces a conservative volume estimate Accuracy dependingon viewpointand lightsource number Limitations with reflective & lowalbedoregions

66 Lecture 8 Active stereo & Volumetric stereo Active stereo Structured lighting Depth sensing Volumetric stereo: Space carving Shadow carving Voxel coloring Silvio Savarese Lecture 7-12-Feb-18

67 Voxel Coloring [Seitz & Dyer ( 97)] [R. Collins (Space Sweep, 96)] Color/photo-consistency Jointly model structure and appearance

68 Basic Idea Is this voxel in or out? View1 View2 View3

69 Basic Idea View1 View2 View3

70 Uniqueness

71 Uniqueness Multiple consistent scenes How to fix this? Need to use a visibility constraint

72 Algorithm for enforcing visibility constraints C

73 The previous ambiguity is removed!

74 The previous ambiguity is removed!

75 Algorithm Complexity Voxel coloring visits each N 3 voxels only once Project each voxel into L images à O(L N 3 ) NOTE: not function of the number of colors

76 A Critical Assumption: Lambertian Surfaces

77 Non Lambertian Surfaces

78 Photoconsistency Test C = corr (, ) w w C = If C > l= threshold è voxel consistent (w w)( w " w ") (w w) ( w" w ") w = mean(w) w! = mean(w')

79 Experimental Results Dinosaur 72 k voxels colored 7.6 M voxels tested 7 min to compute on a 250MHz Image source:

80 Experimental Results Flower 70 k voxels colored 7.6 M voxels tested 7 min to compute on a 250MHz Image source:

81 Experimental Results Room + weird people Image source:

82 Voxel Coloring: Conclusions Good things Model intrinsic scene colors and texture No assumptions on scene topology Limitations: Constrained camera positions Lambertian assumption

83 Further Contributions A Theory of Space carving Voxel coloring in more general framework No restrictions on camera position [Kutulakos & Seitz 99] Probabilistic Space carving [Broadhurst & Cipolla, ICCV 2001] [Bhotika, Kutulakos et. al, ECCV 2002]

84 Next lecture Intro to visual recognition