Three-dimensional scene reconstruction using digital holograms

Transcription

1 Three-dimensional scene reconstruction using digital holograms Conor P. Mc Elhinney, a Jonathan Maycock, a John B. McDonald, a Thomas J. Naughton, a and Bahram Javidi b a Department of Computer Science, National University of Ireland, Maynooth, County Kildare, Ireland b Electrical and Computer Engineering, University of Connecticut, 371 Fairfield Road, Unit 1157 Storrs, CT 06269, USA ABSTRACT One of the principal successes of computer vision over the past thirty years has been the development of robust techniques for the estimation of the structure of a 3D scene given multiple views of that scene. Since holograms permit reconstruction of arbitrary views of the scene they provide a novel avenue of extension to these traditional computer vision techniques. In traditional stereo or multi-view vision systems, the quality of the 3D reconstruction is dictated by the camera configuration. Hence, if a part of the scene is occluded to each view, then reconstruction of that part is impossible. Active vision systems aim to increase the accuracy of the final reconstruction by dynamically changing their camera configuration. However, this must be performed online and is only possible if the scene is static. This online requirement comes from the fact that the algorithms follow a closed-loop strategy. Therefore the optimal views are dependent on the particular scene being reconstructed. We capture digital holograms of 3D scenes using phase-shift digital interferometry. Holographic systems permit the synthesis of arbitrary views of the scene offline. These can still be considered as closed loop approaches where the loop is closed between the image reconstruction and the hologram (i.e. not the sensor). Since all views are captured simultaneously, arbitrary views can be reconstructed after the hologram is recorded. We have built an active vision system that uses a digital hologram of a scene. The system responds to requests for particular 3D information (such as views of particular portions of the scene from particular perspectives). It keeps track of previously reconstructed views, and determines when and where it needs further reconstruction information to satisfy subsequent requests. We believe this would be particularly useful in scenarios where exhaustive reconstruction of the scene or every perspective of a scene would not be possible, such as with near-real-time Internet-based digital hologram applications. Keywords: three-dimensional image processing, digital holography, scene reconstruction 1. INTRODUCTION One of the principal successes of computer vision over the past thirty years has been the development of robust techniques for the estimation of the structure of a three-dimensional (3D) scene given multiple views of that scene. 1 3 In traditional stereo or multiple-view vision systems, the quality of the 3D reconstruction is dictated, in part, by the camera configuration. Hence, if a part of the scene is occluded from each view, then reconstruction of that part is impossible. Active vision systems 4 aim to increase the accuracy of the final reconstruction by dynamically changing their camera configuration. However, this must be performed online and is only possible if the scene is static. This online requirement comes from the fact that the algorithms follow a closed-loop strategy. Therefore the optimal views are dependent on the particular scene being reconstructed [see Fig. 1 (top)]. Holography 5 is an established technique for recording and reconstructing real-world 3D objects. Digital holography 6 11 has recently become feasible due to recent advances in megapixel CCD sensors with high spatial Further author information: JMcD: johnmcd@cs.may.ie; TN: tom.naughton@may.ie; BJ: bahram@engr.uconn.edu

2 Scene Acquisition 3D Reconstruction Evaluation of Reconstruction 3D Scene Representation Hologram Acquisition 2D View Reconstruction 3D Reconstruction Evaluation of Reconstruction 3D Scene Representation Figure 1. Traditional mutliple-view computer vision approach to 3D reconstruction (top) vs. digital holography approach (bottom). resolution and high dynamic range. A technique known as phase-shift interferometry 8, 10 (PSI) was used to 11, 12 create our in-line digital holograms. The resulting digital holograms are in an appropriate form for data transmission and digital image processing (noise removal, object recognition, and so on). Since holograms permit reconstruction of arbitrary views of the scene they provide a novel avenue of extension to these traditional computer vision techniques. A single hologram of a scene encodes multiple perspectives of the scene simultaneously. Since reconstructions of these views can be performed offline, holographic systems permit the synthesis of arbitrary views without the need for repeated sensing of the scene. Hence, if a 3D scene reconstruction is performed from views that are synthesised from a hologram, then further refinement of this reconstruction is possible by selectively incorporating information from additional views of the scene. This can still be considered as a closed loop approach, however here the loop is closed between the image reconstruction and the hologram [see Fig. 1 (top)]. In this paper, we explore the potential use of digital holograms in 3D scene reconstruction where particular regions of interest are occluded under particular views. Specifically, we demonstrate the ability inherent in digital holograms to overcome the problems associated with standard camera configurations outlined above. In our demonstrations we employ both synthetic holograms of artificial scenes, and optically-captured digital holograms of real-world objects. For each hologram we simulate one or multiple occlusions such that parts of the object of interest are only visible from particular points-of-view. Our results show how, even with these occlusions, the original scene can be reconstructed by combining information from a set of perspectives. It should be pointed out that digital holography has yet to be demonstrated in general environments. This is due to the sophisticated optical apparatus required to capture digital holograms. However many 3D applications domains exist which due to their constraints make digital holography a viable approach. One such mature area is non-destructive measurement and inspection. In Sect. 2, we describe how 3D objects are captured using phase-shift digital holography. In Sect. 3 we explain our experimental procedure. The results of experiments with single and multiple occlusions for 2D objects are presented Sect. 4. Experiments with 3D objects are presented in Sect. 5, and we conclude in Sect PHASE-SHIFT DIGITAL HOLOGRAPHY We record whole Fresnel fields with an optical system 11, 12 based on a Mach-Zehnder interferometer (see Fig. 1). A linearly polarized Argon ion (514.5 nm) laser beam is expanded and collimated, and divided into object and reference beams. The object beam illuminates a reference object placed at a distance of approximately d = 350 mm from a 10-bit pixel Kodak Megaplus CCD camera. Let U 0 (x, y) be the complex amplitude distribution immediately in front of the 3D object. The linearly polarized reference beam passes through half-wave plate RP 1 and quarter-wave plate RP 2. By selectively removing the plates we can achieve

3 BE BS RP 1 RP 2 M λ/2 λ/4 Ar laser M M BS CCD Figure 2. Experimental setup for PSI: BE, beam expander; BS, beam splitter; RP, retardation plate; M, mirror. d four phase shift permutations of 0, π/2, π, and 3π/2. The reference beam combines with the light diffracted from the object and forms an interference pattern in the plane of the camera. At each of the four phase shifts we record an interferogram. We use these four real-valued images to compute the camera-plane complex field 8, 10 H 0 (x, y) by PSI. We call this computed field a digital hologram. A digital hologram H 0 (x, y) contains sufficient amplitude and phase information to reconstruct the complex 10, 11, 13 field U(x, y, z) in a plane in the object beam at any distance z from the camera. This can be calculated from the Fresnel approximation 14 as U(x, y, z) = i (i λz exp 2πλ ) [ ( x 2 z + y 2) ] H 0 (x, y) exp iπ, (1) λz where λ is the wavelength of the illumination and denotes a convolution operation. At z = d, and ignoring errors in digital propagation due to discrete space (pixelation) and rounding, the discrete reconstruction U(x, y, z) closely approximates the physical continuous field U 0 (x, y). As with conventional holography, a hologram encodes different views of a 3D object from a small range of 14, 15 angles. In order to reconstruct a particular 2D perspective of the object, the appropriate windowed subset 10, 11, 13 of pixels must be extracted from the hologram and subjected to simulated Fresnel propagation. As the window explores the field a different angle of view of the object can be reconstructed. The range of viewing angles is determined by the ratio of the window size to the full CCD sensor dimensions. Our CCD sensor has approximate dimensions of mm and so a pixel window has a maximum lateral shift of 9 mm across the face of the sensor. With an object positioned d = 350 mm from the camera, viewing angles in the range ±0.74 are permitted. Smaller windows will permit a larger range of viewing angles at the expense of image quality at each viewpoint. 3. EXPERIMENTAL OVERVIEW Our simulated experimental set-up is shown in Fig. 3, which depicts the part of the optical apparatus from Fig. 2 between the object and camera. The object plane contains a 2D complex-valued signal representing a coherent wavefront, in some plane, that was incident on, and reflected from, a diffuse 3D object. By applying Eq. (1) to this complex wavefront we can generate the whole Fresnel field in any plane. Simulated Fresnel propagation by a distance z = d to the camera plane gives us the whole Fresnel field that would be generated through PSI. By inserting opaque regions into the path of the propagating field we can generate occlusions inherent in 3D scenes. Figure 3 depicts a situation with four occlusions that, in the direction parallel to the optical axis, are nonoverlapping and that appear in different planes orthogonal to the optical axis. Figure 3 depicts a situation where the four nonoverlapping occlusions appear in one plane orthogonal to the optical axis to form a single contiguous occlusion.

4 50mm 10mm 10mm 10mm 40mm 3D Object Hologram Plane 80mm 40mm 3D Object Hologram Plane Figure 3. Simulated experimental set-ups with four opaque occlusions at distances of 50 mm, 60 mm, 70 mm and 80 mm from the object, and a single opaque occlusion at a distance of 80 mm from the object. (c) Figure 4. Reconstructions without occlusion showing the set of objects used in these experiments: a 2D binary image, a 3D die object, and (c) a 3D bolt object.

5 g a b e c d f Figure 5. Illustration of the pixel hologram plane with the pixel windows used to reconstruct different views of the objects. One 2D object and two 3D objects were used in our experiments. The 2D object is an image with complex-valued pixels. Each pixel has an amplitude in {0, 1}, and a phase value chosen with uniform probability from the range [0, 2π), so that f appeared as a diffuse reflective object. The 3D objects used in our experiments were reconstructed whole Fresnel fields from digital holograms of a die and bolt object captured using the phaseshift digital holography apparatus described in Sect , 12 Each Fresnel field is an image with complex-valued pixels. A reconstruction of the amplitudes of each object as it would appear in the object plane is shown in Fig. 4. The occlusion in Fig. 3 is a pixel opaque object centred on the optical axis. It is simply partitioned into four equal quadrants for the situation depicted in Fig. 3. For the multiple occlusion experiments, the quadrants of the occlusion were positioned at distances of 50 mm, 60 mm, 70 mm and 80 mm from the object. For the single occlusion experiments, the whole occlusion was positioned at a distance of 80 mm from the object. The distance d from object to camera (hologram plane) was 120 mm, 323 mm, and 390 mm for the 2D object, die object, and bolt object, respectively. In our experiments we reconstruct different pixel windows of the digital holograms to generate particular perspectives of the objects. Figure 5 illustrates the positions of these windows in the pixel hologram plane. Windows a through d have been shifted 512 pixels vertically and horizontally from the optical axis. Window e is centred on the optical axis. Windows f and g are used in Sect. 5 to reconstruct specific object information that is not apparent from perspectives corresponding to windows a through e. 4. 2D OBJECTS Experiment Summary: The reconstruction plane is a distance of 120mm from the hologram plane. The first occlusion was added at 50mm covering the character N, the second occlusion was added at 60mm covering the character M, the third occlusion was added at 70mm covering the character U and the fourth occlusion was added at 80mm covering the character I. The total size of the added occlusions is a 512x512 opaque surface. Each occlusion is 305x305 in size and overlaps with its neighbour by 49 pixels. The object being used is the characters NUIM. Fig. 6 figure a) is the snapshot of the wavefront at 80mm with the occlusions added at resolution 2048x2048. figure b) is the 512x512 reconstruction of a centred 512x512 window centred around the optical axis of the hologram plane. Fig. 7 A 512x512 window is taken from the hologram plane. Each reconstruction is 512x512 in size. figs through (d) correspond to windows through (d) in Fig. 5. Experiment Summary: The distance to the hologram plane from the 3D object is 120mm The occlusion is a 512x512 centerd opaque surface placed at a distance of 80mm from the 3D object. The object being used is the character NUIM.

6 Figure 6. Four occlusions added to wavefront at 50mm, 60mm, 70mm and 80mm from 3D object, shows the wavefront with the added occlusions, shows reconstruction on optical axis. (c) Figure 7. Reconstructions of the characters NUIM from four different views with four separate occlusions added. (d) Figure 8. Occlusion added to wavefront at 80mm from 3D object, shows the wavefront with the added occlusion, shows reconstruction on optical axis.

7 (c) Figure 9. Reconstructions of the characters NUIM from four different views with a single centred occlusion. (d) Figure 10. Reconstructions of the object using multiple occlusions, and a single occlusion. Fig. 8 figure a) is the snapshot of the wavefront at 80mm with the occlusion added at resolution 2048x2048. figure b) is the 512x512 reconstruction of a centred 512x512 window centred around the optical axis of the hologram plane. Fig. 9 A 512x512 window is taken from the hologram plane. Each reconstruction is 512x512 in size. Figs. through (d) correspond to windows through (d) in Fig. 5. Fig. 10 The reconstruction plane is a distance of 120mm from the hologram plane. The object being used is the characters NUIM. figure a) Object reconstruction using the a-d windows from figure 5 and the setup of 3 figure b) Object reconstruction using the a-d windows from figure 5 and the setup of D OBJECTS Experiment: the reconstruction plane is a distance of 390 mm from the hologram plane. The occlusion is a centred opaque surface placed at a distance of 80 mm from the 3D object. The object being used is the bolt.

8 Figure 11. Occlusion added to wavefront at 80 mm from 3D object, shows the wavefront with the added occlusion, shows reconstruction on optical axis. Figure 12. Occlusion added to wavefront at 80 mm from 3D object, shows the wavefront with the added occlusion, shows reconstruction on optical axis. Experiment: The same experiments were repeated for the die. Identical to that for the bolt, except the reconstruction plane is a distance of 323 mm from the hologram plane. The same experiments were performed with the die object. The results from Figs. 12, 14 and 16 are repeated in Figs. 11, 13, and 15, respectively. In Fig. 12 is the snapshot of the wavefront at 80 mm with the occlusion added at resolution figure b) is the reconstruction of a centred window padded to be centred around the optical axis of the hologram plane. In Fig. 14 Each reconstruction is 1024x1024 in size. A 512x512 window is taken from the hologram plane and padded to 1024x1024 to reconstruct the full object since it is bigger than 512x512 in size. figure through (d) correspond to windows through (d) in Fig. 5. In Fig. 16 figure a) Object reconstruction using the a-d windows from Fig. 5. figure b) Object reconstruction using the same four windows and a fifth window to get higher detail. 6. CONCLUSION In active vision systems new information is integrated into the 3D world model as the camera configuration is altered. This is due to the fact that certaing parts of the scene are occluded from particular camera configurations. It is only as parts of the scene become visible that their 3D structure can be estimated. In this paper we have demonstrated that it is possible to select and synthesise arbitrary view points of a scene that has been encoded in a digital hologram. Furthermore, we have shown that the set of sub-windows of the hologram (i.e. the set of scene view points) for 3D reconstruction is determined by the scene itself. This can be seen in our experiments by noting that for each hologram we have used a different set of windows to perform our reconstructions.

9 (c) (d) Figure 13. Reconstructions of the die object from four different views with a single centred occlusion. (c) (d) Figure 14. Reconstructions of the bolt object from four different views with a single centred occlusion.

10 Figure 15. Reconstructions of the object using four windows, five windows. Figure 16. Reconstructions of the object using four windows, five windows. Our work is part of a larger project to apply and adapt image processing algorithms and techniques to digital holographic data. Other aspects include compression,16 network transmission,17 encryption,18 and real-time optical reconstruction.19 Acknowledgements The authors wish to thank Enrique Tajahuerce and Yann Frauel for use of their digital hologram data. The authors wish to acknowledge support from Enterprise Ireland, Science Foundation Ireland, and the Embark Initiative of the Irish Research Council for Science, Engineering, and Technology. REFERENCES 1. O. Faugeras, Three-dimensional Computer Vision: A Geometric Viewpoint, Artificial Intelligence Series, MIT Press, R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, O. Faugeras and Q.-T. Luong, The Geometry of Multiple Images: The Laws That Govern the Formation of Multiple Images of a Scene and Some of Their Applications, MIT Press, T. Vieville, A Few Steps Towards 3D Active Vision, Springer Series in Information Sciences, Springer, D. Gabor, A new microscopic principle, Nature 161, pp , May J. W. Goodman and R. W. Lawrence, Digital image formation from electronically detected holograms, Applied Physics Letters 11(2), pp , L. P. Yaroslavskii and N. S. Merzlyakov, Methods of Digital Holography, Consultants Bureau, Plenum, New York, Translated from Russian by Dave Parsons. 8. J. H. Bruning, D. R. Herriott, J. E. Gallagher, D. P. Rosenfeld, A. D. White, and D. J. Brangaccio, Digital wavefront measuring interferometer for testing optical surfaces and lenses, Applied Optics 13, pp , Nov

11 9. U. Schnars and W. P. O. Jüptner, Direct recording of holograms by a CCD target and numerical reconstruction, Applied Optics 33, pp , Jan I. Yamaguchi and T. Zhang, Phase-shifting digital holography, Optics Letters 22, pp , Aug B. Javidi and E. Tajahuerce, Three-dimensional object recognition by use of digital holography, Optics Letters 25, pp , May Y. Frauel, E. Tajahuerce, M.-A. Castro, and B. Javidi, Distortion-tolerant three-dimensional object recognition with digital holography, Applied Optics 40, pp , Aug L. Onural and P. D. Scott, Digital decoding of in-line holograms, Optical Engineering 26, pp , Nov J. W. Goodman, Introduction to Fourier Optics, Roberts & Company, Englewood, Colorado, third ed., H. J. Caulfield, ed., Handbook of Optical Holography, Academic Press, New York, T. J. Naughton, Y. Frauel, B. Javidi, and E. Tajahuerce, Compression of digital holograms for threedimensional object reconstruction and recognition, Applied Optics 41, pp , July T. J. Naughton, J. B. McDonald, and B. Javidi, Efficient compression of Fresnel fields for Internet transmission of three-dimensional images, Applied Optics 42, pp , Aug T. J. Naughton and B. Javidi, Compression of encrypted three-dimensional objects using digital holography, Optical Engineering 43, pp , Oct O. Matoba, T. J. Naughton, Y. Frauel, N. Bertaux, and B. Javidi, Real-time three-dimensional object reconstruction by use of a phase-encoded digital hologram, Applied Optics 41, pp , Oct