Structured Light. Tobias Nöll Thanks to Marc Pollefeys, David Nister and David Lowe

Transcription

1 Structured Light Tobias Nöll Thanks to Marc Pollefeys, David Nister and David Lowe

2 Introduction Previous lecture: Dense reconstruction Dense matching of non-feature pixels Patch-based multiple view reconstruction Today: Structured Light Usage of active devices (lasers/ projectors) Correspondence generation Mesh alignment 1/10/2012 Lecture 3D Computer Vision 2

3 Motivation Last lecture: Patch-based dense matching of pixels with NCC This works well only for objects with distinctive texture features High NCC within quad Low NCC within quad It is thus impossible to reconstruct objects of constant color or monotonous texture Object multiple view reconstruction 1/10/2012 Lecture 3D Computer Vision 3

4 Motivation While the vase yielded some geometry in the colored parts, unicolor objects cannot be reconstructed multiple view reconstruction 1/10/2012 Lecture 3D Computer Vision 4

5 Motivation Solution: If objects don t have a texture, give them one with e.g. Point/line lasers Video projectors 1/10/2012 Lecture 3D Computer Vision 5

6 Active Reconstruction: Concept 1/10/2012 Lecture 3D Computer Vision 6

7 Traditional stereo Concept: Active Reconstruction I J 1/10/2012 Lecture 3D Computer Vision 7

8 Concept: Active Reconstruction Active stereo I J 1/10/2012 Lecture 3D Computer Vision 8

9 Structured Light Concept: Active Reconstruction I J 1/10/2012 Lecture 3D Computer Vision 9

10 Calibration: Extrinsics and Intrinsics 1/10/2012 Lecture 3D Computer Vision 10

11 Calibration Extrinsics Rel Intrinsics C Intrinsics P 3D position (Midpoint triangulation) 1/10/2012 Lecture 3D Computer Vision 11

12 Calibration Intrinsics camera: Usual procedure: Capture calibration sequence with known calibration pattern (here: chessboard) Find chessboard in the images Use captured correspondences in order to calibrate K c => See lecture 3: Calibration We assume that the camera moves around the board and the board stays static 1/10/2012 Lecture 3D Computer Vision 12

13 Calibration Intrinsics projector How to model the projector? => Use same pinhole model as for the camera (see lecture 1: Camera) But: How to generate correspondences between the calibration board and the projector? Problem: Projector can t see like the camera 1/10/2012 Lecture 3D Computer Vision 13

14 Calibration Recall: Stereo camera case (2 cameras) The same chessboard is seen by both camera Thus correspondences to both cameras can be established using only one chessboard 3D points of the chessboard corners stay the same, what changes are their 2D projections in the images We virtually move the cameras around the calibration board 1/10/2012 Lecture 3D Computer Vision 14

15 Calibration Now: Projector / Camera system The projector can not see anything, thus no correspondences between the calibration board and the projector can be established We need 2 chessboards, one to see for the camera and one to be projected by the projector 1/10/2012 Lecture 3D Computer Vision 15

16 Calibration Now: Projector / Camera system The image plane of the projector (i.e. seen image ) is the image to be projected and is static independent of the board position => 2D points stay the same 3D positions where the chessboard corners are projected change with the board position Printed chessboard can be seen by the camera and thereby the camera pose w.r.t. to the board can be calculated since K C is known => See lecture 4: Camera pose estimation 1/10/2012 Lecture 3D Computer Vision 16

17 Calibration Summary Calibration Camera Projector 3D chessboard corners 2D chessboard corners Fixed Variable Variable Fixed So how to get the variable values? Camera s 2D points: Use chessboard detection algorithm Projector s 3D points: Use calibrated camera to compute the positions 1/10/2012 Lecture 3D Computer Vision 17

18 Calibration Projector s 3D point obtained by intersecting camera rays with calibration plane Extrinsics Printed chessboard R t 1/10/2012 Lecture 3D Computer Vision 18

19 Calibration With the 3D <-> 2D correspondences of the projector and the camera compute independently K C and K P [R Ci t Ci ] and [R Pi t Pi ] with i = 1 n for n the number of board positions However we need one relative extrinsics [R t] Simple method: Choose some i and compute [R t] = [R i t i ] = [R Ci (R Pi ) T t Ci - R Ci (R Pi ) T t Pi ] which maps projector coordinates to camera coordinates Better method: Formulate as an optimization problem Use simple method s output as initial guess Optimize in parallel K C, K P and [R t] 1/10/2012 Lecture 3D Computer Vision 19

20 Calibration Initial setup: Static calibration board (for calibration we assumed that we moved the camera / projector around the board) P 2 C 2 C 1 P 1 P 3 C 3 1/10/2012 Lecture 3D Computer Vision 20

21 Equivalent setup: Calibration In reality however we moved the board while the camera / projector remained static It doesn t matter whether we moved the camera / projector or the world P P 1 C 1 P 2 C 2 P 3 C 3 C 1/10/2012 Lecture 3D Computer Vision 21

22 Calibration We search for one optimal pose which mostly approximates the input correspondences This is done by minimizing the reprojection error using Levenberg- Marquardt => see lecture 5: Parameter estimation P C 1/10/2012 Lecture 3D Computer Vision 22

23 Calibration Camera Non-uniform coverage => Additional camera constraints Projector Iteratively optimize in parallel K C, K P and [R t] => Effect can be seen as the red lines in the projector image 1/10/2012 Lecture 3D Computer Vision 23

24 Correspondences generation 1/10/2012 Lecture 3D Computer Vision 24

25 Correspondence generation Correspondence generation is essential for 3d reconstruction Correspondences => triangulation => 3d information We focus now on correspondence generation between cameras and light emitting devices Remark: There exist several classes of laser scanners. We only consider those based on triangulation 1/10/2012 Lecture 3D Computer Vision 25

26 Structured light principle Light source projects a known pattern onto the measuring scene Captured and projected patterns are related to each other Establish correspondences 3D scene object j i i pattern projection detail imaging pattern detail j Image Sensor Pattern Projecting System 1/10/2012 Lecture 3D Computer Vision Slide from UdG 26

27 Correspondence methods Single dot: No correspondence problem. Scanning both axis Stripe patterns: Correspondence problem among slits No scanning Single stripe: Correspondence problem among points of the same slit Scanning the axis orthogonal to the stripe Grid, multiple dots: Correspondence problem among all the imaged segments No scanning Slide from UdG 1/10/2012 Lecture 3D Computer Vision 27

28 Correspondence methods Multi-stripe Multi-frame Single-stripe Single-frame Slow, robust Fast, fragile 1/10/2012 Lecture 3D Computer Vision 28

29 Single dot The correspondence in this case is unique Laser Camera P. Hurbain, 1/10/2012 Lecture 3D Computer Vision 29

30 Single stripe Because the lines span planes in 3D, it is possible to intersect camera rays with the plane spanned by the emitted light rays Object Light Plane Ax By Cz D 0 Laser/projector Image Point ( x', y') Camera 1/10/2012 Lecture 3D Computer Vision 30

31 Multiple stripes / Grid When multiple stripes are to be used at the same time, they must be encoded in order to be identifiable (Similar: Grid) Stripes have to be made distinguishable Solution: 1D or 2D encoding Encoded axis Encoded axes Camera Projector Camera Projector J. Salvi 1/10/2012 Lecture 3D Computer Vision 31

32 Encoding A pattern is called encoded, when after projecting it onto a surface, a set of regions of the observed projection can be easily matched with the original pattern. Example: encoding by color Decoding a projected pattern allows a large set of correspondences to be easily found thanks to the a prior knowledge of the pattern Jung, Computer Vision (EEE6503) Fall 2009, 1/10/2012 Lecture 3D Computer Vision Yonsei Univ. 32

33 Multiple stripes - Color The simplest way: Unique color for each stripe (direct codification) Problem: Colors are altered during the light transport Colors interfere with surface colors Lower robustness for larger amount of stripes 1/10/2012 Lecture 3D Computer Vision 33

34 Multiple stripes - Color Solution: Encode the color assignment itself, such that colors can repeat themselves Each point is encoded by its surrounding intensities (spatial codification) 1/10/2012 Lecture 3D Computer Vision 34

35 Multiple stripes Binary coding A more robust way for distinguishing a large amount of stripes are binary codes n stripe patterns can encode stripes n 2 1 Projected over time Example: 3 binary-encoded patterns which allows the measuring surface to be divided in 8 sub-regions Jung, Computer Vision (EEE6503) Fall 2009, Yonsei Univ. 1/10/2012 Lecture 3D Computer Vision 35

36 Multiple stripes Binary coding Assign each stripe a unique illumination code over time [Posdamer 82] Time 0110 Space 1/10/2012 Lecture 3D Computer Vision 36

37 Multiple stripes Binary coding Example: 7 binary patterns proposed by Posdamer & Altschuler Projected over time Pattern 3 Pattern 2 Codeword of this pixel: identifies the corresponding pattern stripe Pattern 1 Jung, Computer Vision (EEE6503) Fall 2009, 1/10/2012 Lecture 3D Computer Vision Yonsei Univ. 37

38 Multiple stripes Binary coding More robust but requires a lot of images one image for each bit Instead of binary codes, gray codes are often used in practice Adjacent code words differ only in one bit This allows to correct some errors 1/10/2012 Lecture 3D Computer Vision 38

39 Multiple stripes Binary coding Problem: Large resolution requires a lot of images to be projected i.e. 1024x768 => 10 images (2^10 = 1024) In practice not possible to distinguish projected stripes with a width of only one pixel Consequently the full resolution of the projector cannot be exploited 1/10/2012 Lecture 3D Computer Vision 39

40 Conclusion so far Scanning with a single dot yields a 1:1 correspondence The correspondence for a scanline were implicitly included in a rayplane intersection Ok for a laser (no lens) Camera/projector: How should lens distortion of the projector s model be regarded? Stripe encoding cannot exploit the full resolution of the projector Is it possible to go down to pixel level? 1/10/2012 Lecture 3D Computer Vision 40

41 Phase shifting A widely used method to achieve these goals is phase-shifted structured light Consider a function φ ref (x,y) = x Encoding this function into grayscales and projecting it could be used as direct codification Problem again: The camera cannot precisely distinguish between the grayscales Camera Projector (φ ref (x,y)) 1/10/2012 Lecture 3D Computer Vision 41

42 Phase shifting Solution: Phase shifted structured light can be used to encode the function φ ref (x,y) in a more efficient way. Let x [0..2nπ] Then g(x,y) = cos(φ ref (x,y)) = cos(x) yields an image with n vertical fringes in horizontal direction Camera Projector (g(x,y)) 1/10/2012 Lecture 3D Computer Vision 42

43 Phase shifting The captured fringe images can be described as I(x,y) = A(x,y) + B(x,y)cos(φ obj (x,y)) A(x,y): Background or ambient light intensity B(x,y): Cosine amplitude Φ obj (x,y): Object phase This is what we want to compute The 2D reference phase on the object as seen by the camera 1/10/2012 Lecture 3D Computer Vision 43

44 Phase shifting To compute φ obj (x,y), the initial phase φ ref (x,y) is shifted. We present the 3 step phase shifting algorithm by Zhang et al. We project the following shifted fringe images 2 g x, y) cos( ( x, y) ), i( ref i , /10/2012 Lecture 3D Computer Vision 44

45 Phase shifting The 3 captured images are described by We thus have 3 equations with 3 unknowns (A, B, Φ obj ) for each pixel Solving it with the known shifts yields 1/10/2012 Lecture 3D Computer Vision 45 ) 2 3 ( tan ), ( I I I I I y x obj ) ), ( )cos(, ( ), ( ), ( i obj i y x y x B y x A y x I

46 Phase shifting Problem: Due to tan^{-1} the resulting phase is wrapped for more than 1 stripe in 2π steps Φ obj using only stripe (x in [0..2π]) Φ obj using multiple stripes 1 stripe: No wrapping problem (=> unique correspondences) but imprecise 3D reconstruction Multiple stripes: Wrapping problem (=> ambiguous correspondences) but precise reconstruction 1/10/2012 Lecture 3D Computer Vision 46

47 Phase shifting The phase thus must be unwrapped (elimination of the 2πdiscontinuities Challenging Unrobust, if using only a single wrapped phase especially at discontinuities 1/10/2012 Lecture 3D Computer Vision 47

48 Phase shifting A robust solution is to use the level based unwrapping algorithm by Wang et al. Capture multiple fringe levels, e.g. Level 0 => shift 1 stripe => no wrapping Level 1 => shift e.g. 5 stripes => wrapping => use information from level previous for unwrapping Level 2 => shift e.g. 20 stripes => 1/10/2012 Lecture 3D Computer Vision 48

49 Phase shifting 1/10/2012 Lecture 3D Computer Vision 49

50 Phase shifting Once the phase has been computed, every pixel in the image creates a point to stripe correspondence (either horizontal or vertical) Camera Projector 1/10/2012 Lecture 3D Computer Vision 50

51 Phase shifting Finding the pixel correspondences is easy using epipolar geometry However no distortion is included 1/10/2012 Lecture 3D Computer Vision 51

52 Phase Shifting Correspondence generation including lens distortion 1/10/2012 Lecture 3D Computer Vision 52

53 Phase Shifting Correspondence generation including lens distortion 1. Undistort 2. Compute epipolar curve 1/10/2012 Lecture 3D Computer Vision 53

54 Phase Shifting - Summary 1/10/2012 Lecture 3D Computer Vision 54