Optical Imaging Techniques and Applications

Transcription

1 Optical Imaging Techniques and Applications Jason Geng, Ph.D. Vice President IEEE Intelligent Transportation Systems Society

2 Outline Structured light 3D surface imaging concept Classification framework of structured light 3D surface imaging techniques Temporal/multi-shot projection techniques Sequential Projection Spatial/single-shot projection techniques Continuously varying pattern projection Strip indexing Grid indexing Hybrid Application examples Conclusions

3 Structured Light 3D Surface Imaging Triangulation R = B Sin(q ) Sin(a + q) B and a are known, How to get the value of q? Structured Light Projection Structured Light Projector B Encode q information into projection pattern design, P such that the q value corresponding to each pixel in the acquired image can be derived and a full frame of 3D surface image can be obtained. q a R 3D Object in the Scene Camera

4 Structured Light 3D Surface Imaging 3D surface imaging results: point cloud {P i =(x i, y i, z i, f i ), i=1,2,, N}, where f i represents the value at the i th surface point in the data set SPIE AAPM 2011 Photonics Annual Meeting, West, San Vancouver, Francisco, 7/31/2011 CA,01/26/2011

5 Structured Light Projection (SLP) Classification Framework Structured Light 3D Surface Imaging Techniques Sequential Projections (Multi-Shots) Binary Code Gray Code Phase Shift Hybrid: Gray code + Phase Shift Continuous Varying Pattern (Single Shot) Rainbow 3D Camera Continuously Varying Color Code Stripe Indexing (Single Shot) Color Coded Stripes Segmented Stripes Gray Scale Coded Stripes De Bruijn Sequence Grid Indexing (Single Shot) Pseudo Random Binary-Dots Mini-Patterns as Codewords Color Coded Grid 2D Color Coded Dot Array Hybrid Methods

6 Sequential Projections - Binary Patterns N patterns -> 2 N stripes Sequence of Projection LSB Horizontal Spatial Distribution MSB Triangulation R = B Sin(q ) Sin(a + q) Encode q information into projection pattern design, such that the q value corresponding to each pixel in the acquired image can be derived. Pros: Very robust to surface texture and environmental illumination Cons: Slow, unable to acquire 3D image of moving object Ishii, I., Yamamoto, K., Doi, K. and Tsuji, T., High-speed 3D image acquisition using coded structured light projection. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp (2007) K. Sato and S. Inokuchi, Range-imaging system utilizing nematic liquid crystal mask, Proc. Int. Conf. on Computer vision, pp (1987). R. J. Valkenburg and A. M. McIvor, Accurate 3d measurement using a structured light system. Image and Vision Computing, 16(2), pp , (1998). J. L. Posdamer and M. D. Altschuler, Surface measurement by space-encoded projected beam systems. Computer Graphics &Image Processing, 18(1), pp. 1, 1982

7 Sequential Projections - Gray Coding N patterns -> M N stripes M = 4 Pros: use M distinct levels of intensity (instead of only two in the binary code), to produce unique coding, reduced number of required projection patterns, faster N patterns -> M N stripes (v.s 2 N stripes in binary) Cons: Unable to acquire 3D image of moving object J. L. Posdamer and M. D. Altschuler, Surface measurement by space-encoded projected beam systems. Computer Graphics d Image Processing, 18(1), pp. 1, S. Inokuchi, K. Sato and F. Matsuda, Range-imaging for 3-D object recognition, Proc. Int. Conf. on Pattern Recognition, pp (1984). D. Caspi, N. Kiryati, and J. Shamir, Range imaging with adaptive color structured light. Pattern Analysis and Machine Intelligence, 20(5), pp , (1998). W. Krattenthaler, K. Mayer, and H. Duwe, 3D-surface measurement with coded light approach, In Proceedings O esterr. Arbeitsgem. MustererKennung, volume 12, pp , (1993).

8 Sequential Projections - Phase Shift Reference Plane d Z x L P x x x Image Sensor B Pattern Projector A set of sinusoidal patterns are projected. The variation form of these projected fringe patterns are similar, except the phase of the fringe patterns are shifted a constant angle with respect to each other. The minimum number of projections is three x Pros: fairly robust, efficient algorithms, sub-pixel accuracy Cons: Phase ambiguity, need multiple projections, unable to acquire 3D image of fast moving object E. Horn, N. Kiryati, Toward optimal structured light patterns, Image and Vision Computing volume 17 (2), pp , (1999).

9 Sequential Projections - Phase Shift + Gray Coding Eliminate ambiguity Pros: Accurate, very robust to surface texture and environmental illumination Cons: Slow, unable to acquire 3D image of moving object P. S. Huang and S. Zhang, A Fast Three-Step Phase Shifting Algorithm, Appl. Opt., (2006). Zhang, S., Yau, S.T. High-resolution, real-time 3d absolute coordinate measurement based on a phase-shifting method, Opt. Eng 14, pp , (2006).

10 Sequential Projections - Photometric Stereo Camera I 1, I 2, I 3, and I 4. L 2 (x 2,y 2,z 2 ) L 3 (x 3,y 3,z 3 ) L1 (x 1,y 1,z 1 ) q 1 Normal s(x,y,z) Surface Patch Intensity Images L 4 (x 4,y 4,z 4) Photometric stereo a variant approach to Shape from Shading (SfS). estimates local surface orientation by using several images of the same surface taken from the same viewpoint but under illumination from different directions. solves the ill-posed problems in SfS by using multiple images. Shape from Shading (SfS) by using multiple images All light sources be point light Only estimates the local surface orientation (gradients p,q). Assumes continuities of 3D surface needs a starting point (a point on object surface whose (x,y,z) coordinate is known) R. Woodham, Photometric method for determining surface orientation from multiple images, Optical Engineering, 19:1, pp , (1980).

11 Single Shot - Rainbow 3D Camera Elegant encoding scheme High speed 3D imaging Infinite resolution Single shot simple implementation Low-cost Triangulation R = B Sin(q ) Sin(a + q) Encode q information into projection pattern design, such that the q value corresponding to each pixel in the acquired image can be derived. Z. J. Geng, Rainbow three-dimensional camera: new concept of high-speed three-dimensional vision systems, Opt. Eng. 35(2), pp (1996)

12 Single Shot - Spatially Varying Color Coding Red Channel Intensity Variation Pattern Composite Three Color Saw-Tooth Pattern Green Channel Intensity Variation Pattern Blue Channel Intensity Variation Pattern Multiple cycles of variation Improved sensitivity and accuracy Z. J. Geng, Rainbow three-dimensional camera: new concept of high-speed three-dimensional vision systems, Opt. Eng. 35(2), pp (1996)

13 Stripe Indexing (Single Shot) - Stripe Indexing Using Color Structured Light Projector Camera Stripe Indexing pattern variation is in horizontal direction 3D Object in the Scene Stripe Indexing using color a set of distinct colors Projection angles can be derived from one-to-one color-angle correspondence K. L. Boyer, A. C. Kak, Color-encoded structured light for rapid active ranging, IEEE trans. Pattern Anal.Mach. Intell. 9(1), pp (1987).

14 Stripe Indexing (Single Shot) - Segment Pattern with Random Cuts Stripe Indexing using Segment Pattern with Random Cuts Each stripe has unique segment cut pattern Projection angles can be derived from one-to-one cut pattern-angle correspondence Pros: Simple one shot scheme Cons: Require continuous surface, ambiguity occurs when surface texture disturbs the projected pattern M. Maruyama and S. Abe, Range Sensing by Projecting Multiple Slits with Random Cuts, IEEE Transaction Patten Analysis a d Machine Intelligence 15(6), pp (1993).

15 Stripe Indexing (Single Shot) - Repeated Grey Scale Pattern Stripe Indexing using Repeated Gray Scale Pattern Each stripe has unique grey scale level Projection angles can be derived from one-to-one grey scale level -angle correspondence Pros: Simple, one-shot scheme Cons: ambiguity occurs when surface texture disturbs the projected pattern N. G. Durdle, J. Thayyoor, V. J. Raso, An improved structured light technique for surface reconstruction of the human trunk, IEEE Canadian Conference on Electrical and Computer Engineering, Vol. 2, pp (1998).

16 Stripe Indexing (Single Shot) - Stripe Indexing Based on De Bruijn Sequence De Bruijn sequence of rank n on an alphabet of size k is a cyclic word in which each of the k n words of length n appears exactly once as we travel around the cycle. n = 3, k = 2 As we travel around the cycle, we encounter each of the 2 3 = 8 three-digit patterns 000, 001, 010, 011, 100, 101, 110, 111 exactly once. R G B Stripe Indexing using pseudo-random color sequence any sub-sequence is not correlated to any other in the De Bruijn sequence This unique feature is used in constructing stripe pattern sequence that has unique local variation patterns without repeating themselves. Uniqueness simplifies the pattern decoding task. F. J. MacWilliams, N. J. A. Sloane, Pseudorandom sequences and arrays, Proceedings of the IEEE 64 (12), pp ((1976). H. Fredricksen, A survey of full length nonlinear shift register cycle algorithms, Society of Industrial and Applied Mathematics Review, 24 (2), pp (1982). H. Hügli, G. Maïtre, Generation and use of color pseudo random sequences for coding structured light in active ranging, Proc Industrial Inspection, Vol.1010, pp.75 (1989) T. Monks, J. Carter, Improved stripe matching for colour encoded structured light, 5th Int Conference on Computer Analysis of Images and Patterns, pp (1993). L. Zhang, B. Curless, S. M. Seitz, Rapid shape acquisition using color structured light and multi-pass dynamic programming, Int. Symp 3D Data Processing Visualization Transmission, Padova, Italy (2002). n=3, k=5

17 2D Spatial Grid Patterns (Single Shot) - Pseudo Random Binary Array (PRBA) 2D Grid Indexing to uniquely label every sub-window in the projected 2D pattern, such that the pattern in any sub-window is unique and fully identifiable with respect to its 2D position in the pattern pattern variation is in both horizontal and vertical directions Grid Indexing using PRBA to produce grid locations that can be marked by dots (or other patterns), such that the coded pattern of any sub-window in unique. A PRBA is defined by a n1 by n2 array encoded using a pseudo-random sequence, such that any k1 by k2 sub-window sliding over the entire array is unique and fully defines the sub-window s absolute coordinate (i,j) within the array. J. Le Moigne and A.M. Waxman, Structured Light Patterns for Robot Mobility, IEEE Journal of Robotics and Automation 4(5), (1988)

18 2D Spatial Grid Patterns (Single Shot) - Color Coded Grids Grid Indexing using colors to color-code both vertical and horizontal stripes. Encoding schemes in two directions can be same of different There is no guarantee on the uniqueness of sub-windows, colored stripes in both directions can help the decoding in most situations to establish the correspondence. The thin grid lines may not be as reliable in pattern extraction as other patterns (dots, squares, etc). Petriu, E.M., Sakr, Z., Spoelder, H.J.W. and Moica, A., Object recognition using pseudo-random color encoded structured light. IEEE Instrumentation and Measurement. v (2000). J. Pagès, J. Salvi and C. Matabosch. Robust segmentation and decoding of a grid pattern for structured light. 1st Iberian Conf Pattern Recognition/ Image Analysis, IbPRIA pp , (2003)

19 2D Spatial Patterns (Single Shot) - Mini-Patterns as Codewords Instead of using a pseudo random binary array, a multi-valued pseudo random array can be used. One can correspond each value with a mini-pattern as special codeword, thus forming a grid indexed projection pattern. An example of a three-valued pseudo random array and a set of mini-patterns codewords. Using the specially defined codewords, a multi-valued pseudo random array can be converted into a projection pattern with unique sub-windows P. M. Gri n, L. S. Narasimhan and S. R. Yee, Generation of uniquely encoded light patterns for range data acquisition, Pattern Recognition 25(6), (1992).

20 2D Spatial Patterns (Single Shot) - 2D Array of Color-Coded Dots 6 x 6 array with sub-window size of 3 x 3 using three codeword (R,G,B). Fill the upper left 3x3 corner with a randomly chosen pattern. add a 3-element column on the right with random codeword. The uniqueness of the subwindow is verified before adding such a column. Keep adding columns until all are filled with random codeword and uniqueness are verified. Similarly, add random rows in the downward direction from the initial sub-window position. Afterwards, add new random codeword along the diagonal direction. Repeat these procedures until all dots are filled with colors. Payeur, P. and Desjardins, D., Structured Light Stereoscopic Imaging with Dynamic Pseudo-random Patterns. 6th Int Conf Image Analysis and Recognition (2009). Desjardins, D. and Payeur, P Dense Stereo Range Sensing with Marching Pseudo-Random Patterns. Proceedings of the Fourth Canadian Conference on Computer and Robot Vision (2007). Alternative methods of generating pseudo random array. Brute force algorithm to generate an array that preserve the uniqueness of sub-windows, May not exhaust all possible subwindow patterns. Intuitive to algorithm implementation.

21 2D Spatial Patterns (Single Shot) - Hybrid Methods There are many opportunities to improve specific aspect(s) of 3D surface imaging system performance by combining more than one encoding schemes. Here is just one example

22 Summary of Typical SLP patterns Sequential Projections (Multi-Shots) Continuously Varying Pattern (Single Shot) Stripe Indexing (Single Shot) Grid Indexing (Single Shot) Hybrid Methods

23 Performance Evaluation of 3D Surface Imaging Systems Primary performance indexes Resolution Accuracy Primary Performance Space of 3D Imaging Systems Speed Field of view (FOV) Depth of field (DOF) Stand-off distance Cost etc

24 Thank You!