PRE-PROCESSING OF HOLOSCOPIC 3D IMAGE FOR AUTOSTEREOSCOPIC 3D DISPLAYS M.R Swash, A. Aggoun, O. Abdulfatah, B. Li, J. C. Fernández, E. Alazawi and E. Tsekleves School of Engineering and Design, Brunel University London, UK ABSTRACT Holoscopic 3D imaging also known as Integral imaging is an attractive technique for creating full color 3D optical models that exist in space independently of the viewer. The constructed 3D scene exhibits continuous parallax throughout the viewing zone. In order to achieve depth control, robust and real-time, a single aperture holoscopic 3D imaging camera is used for recording holoscopic 3D image using a regularly spaced array of microlens arrays, which view the scene at a slightly different angle to its neighbor. However, the main problem is that the microlens array introduces a dark borders in the recorded image and this causes errors at playback on the holoscopic 3D Display. This paper proposes a reference based pre-processing of holoscopic 3D image for autostereoscopic holoscopic 3D displays. The proposed method takes advantages of microlens as reference point to detect amount of introduced dark borders and reduce/remove them from the holoscopic 3D image. representation of the original object space, to scale and in full color. A flat panel display for example one using LCD technology is used to reproduce the captured intensity modulated image and a microlens array re-integrates the captured rays to replay the original scene in full color and with continuous parallax in all directions. The constructed 3D scene can be viewed by more than one person and independently of the viewer s position. Index Terms Holoscopic image, Integral image, 3D, Lens array, Microlens array, 3D Display, Viewpoint, autostereoscopic 1. INTRODUCTION Holoscopic three dimensional (H3D) imaging [1] also known as Integral imaging [2] is attractive for the scientific community, entertainment and display industry to open a new market. 3D images can be applied in broadcasting, communications and many other areas [2][3]. There are many technologies developed for the 3D imaging system such as stereoscopic, multiview 3D however holoscopic 3D imaging as a spatial imaging method is a strong candidate for next generation 3D visualization systems [3] because it is a simpler approach for recording and replay true 3D images. Holoscopic 3D imaging is first proposed by Lippmann [4] in 1908 as a very promising method for capturing and reproducing three-dimensional image [5]-[8]. This technique uses the principle of Fly s eye and hence allows natural viewing of objects. Unlike the Stereo [9][10] or multiview [11] imaging, holoscopic 3D imaging technique creates physical duplicates of light field, so it s a true 3D imaging technique. Compared with Holographic Imaging [12], it uses incoherent radiation and forms an image that is a sampled Fig 1. (a) Recording and (b) replaying in holoscopic 3D imaging system 978-1-4799-3203-0/13/$31.00 c 2013 Crown
Holoscopic 3D imaging is based on concurrent capture of many different views of 3D scene by using a Microlens array shown in Fig 1(a). Under each micro-lens, there are certain pixels to depict different view point. On the reconstruction stage shown in Fig 1(b), an appropriate microlens array is placed on top of the LCD display equipment to reproduce the 3D scene [13]. As camera sensor s RGB pixel pitch is extremely small and microlens border introduce large dark borders in the recorded holoscopic 3D image if it is counted in pixel and this dark borders introduce errors to the constructed 3D scene in space. In 2006, A. Aggoun [14] proposed a pre-processing filter based on Hough Transform to correct geometric distortions caused during the capture of 3D image and in very recently, S. Lee et al [15] proposed a portable 3D camera based on integral imaging and they have proposed and applied Fourierholographic stereogram on viewpoint sub-image to generate elemental images which are later stitched to form the final image. In this paper, we propose a new method for pre-processing of holoscopic 3D image that eliminates the dark borders introduced during the capture of 3D image caused by the microlens array. The proposed method uses the microlens array information for detecting amount of dark borders in the holoscopic 3D image and then it uses the microlens array information grid as a reference point to eliminate unwanted dark borders without altering or removing the important visual information in elemental images. It is successfully demonstrated on a single aperture holoscopic 3D camera developed at Brunel University and The principle can also be transferred or applied to other holoscopic 3D cameras or light field 3D camera as it uses the microlens information which is generated by capturing a white background to eliminated dark borders in the holoscopic 3D image. 2. THE PROPOSED PRE-PROCESSING METHOD A holoscopic 3D image is represented entirely by a planar intensity distribution and each microlens views the scene at a slightly different angle to its neighbour however the problem is that the microlens arrays introduce dark borders in recorded holoscopic 3D images. Such distortions are scaling errors which cause noises dark moiré effect on the playback. The lens correction algorithm is applied to correct nonlinear distortions to ensure the 3D images are distortion free and in this paper, lens correction algorithm is not going to be discussed because the paper s main focus is on pre-processing filter, which filters out the unwanted dark borders of holoscopic 3D image before replaying on the 3D display. The block diagram of proposed method is presented in Fig 2. The process is rather simple and fast compare to any other visual data processing because once the camera is setup, a simple histogram filtering algorithm generates microlens grid from a holoscopic 3D image of white background as shown in Fig 3 and then on the capture of holoscopic 3D image, the proposed method uses the calibrated microlens grid information to eliminate the dark borders of the 3D image. Fig 2. Block diagram of proposed method (a) Holoscopic 3D image of White background (b) MLA Grid Fig 3. Generated calibrated grid holoscopic 3D image of a white background (Zoomed in) Once the calibration grid is generated (Fig 4.a), dark borders of the input holoscopic 3D image (Fig 4.b) are eliminated based on the calibrated grid information and a new holoscopic 3D image is formed from stitched elemental images. The calibration grid enables one to accurately
eliminate dark borders without effecting visual information. Fig 5 (b) shows eliminated dark borders of holoscopic 3D image. The proposed method does not remove the lens array border completely instead it refines enough the dark borders to reduce dark moiré and also this is because; removing the borders completely will also eliminate 3D visual details of elemental images, which should be avoided. (a) The resulting holoscopic 3D image after applying the proposed method (a) Generated calibrated microlens grid (b) The eliminated dark borders Fig 5. The resulting images after applying the proposed method (b) Original image before applying the proposed method Fig 4. Generated calibrated grid and original holoscopic 3D image As shown in Fig 4, The first step is generating the calibration grid from a white background of holoscopic 3D image (Fig 4.a) and then this information is used to detect and eliminate dark borders introduced by the microlens array in the holoscopic 3D image (Fig 4.b). Fig.5 (b) shows the resulting holoscopic 3D image after the proposed method is applied. Fig 5 (a) shows he eliminated dark borders from the input holoscopic 3D image shown in Fig.4 (b). The proposed method pursues a dynamic calibration approach and the calibration must be fulfilled for every holoscopic 3D camera setup because the dark border noise could various as the camera setup changes. The process is a self-tuning method, which is valid for all type of holoscopic 3D cameras. (a) Original image with thicker (b) resulting image with finer microlens border microlens border Fig 6. Direction comparison of original image with the resulting image after applying the processed image
As seen in Fig 6, the resulting holoscopic 3D image after applying the proposed method has refined microlens array borders. In order to test the proposed method, viewpoints of the original image [Fig.4 (b)] and the resulting image [Fig. 5(a)] are extracted and compared in Fig 7. Viewpoints of the original image have running dark line in the first and last viewpoints [Fig.7 (a)] whereas this dark moiré is removed in the image s viewpoints [Fig. 7(b)] of the resulting image. Viewpoint 1 Viewpoint 3 (a) Super resolution image of original image [Fig.4 (b)] Viewpoint 5 Viewpoint 7 Viewpoint 9 Viewpoint 11 Viewpoint 13 (a) Viewpoints of Original image [Fig 4.(b)] (b) Viewpoints of new resulting image [Fig 5.(a)] Fig 7. Viewpoints of the original and the resulting image The test image has low 3D resolution (69 46 microlensimages) so the viewpoint images are very low resolution and hard to see detailed information except running dark line. Further experiment is carried out on the original and resulting image using super resolution algorithm [16], which is applied on both original and resulting and its results are compared in Fig 8. As seen in the Fig 8 (b), super resolution of the proposed method is very clear compared to super resolution of the original image [Fig 8. (a)]. This proves the proposed method eliminate the dark borders without effecting spatial information. (b) super resolution image of resulting image [Fig. 5 (a)] Fig 8. Super resolution of both original and resulting image 3. CONCLUSION In this paper, we have proposed and implemented a preprocessing filter, which eliminates dark borders introduced during the capture of holoscopic 3D image by the microlens. The proposed method has a single step calibration process that is done by capturing a white background which is used to generate a calibration grid and to learn the level of the dark borders introduced by the microlens array. Later the information is used for eliminating the noise from holoscopic 3D images. Such noise should be removed otherwise it introduces dark moiré in the constructed 3D scene. The experimental result is promising as it does not remove any visual information and it measure and eliminate dark borders based on the3d camera s microlens information. As a result the method is valid for any holoscopic 3D cameras or light field cameras. In addition, it is computationally efficient and fast therefore it can be implemented on camera in real time for filtering out dark borders of final image(s). 4. ACKNOWLEDGEMENT This work has been supported by European Commission under Grant FP7-ICT-2009-4 (3DVIVANT). The authors wish to express their gratitude and thanks for the support given throughout the project.
5. REFERENCES [1] A. Aggoun, E. Tsekleves, M. R. Swash, D. Zarpalas, A. Dimou, P. Daras, P. Nunes, and L. D. Soares, Immersive 3D Holoscopic Video System, MultiMedia, IEEE, vol. 20, no. 1, pp. 28 37, 2013. [2] F. Okano, 3D TV with integral imaging, Proc. of SPIE, 6983, 69830N, (2008). [3] L. Onural, Television in 3-D: What are the Prospects, Proc. IEEE, 95, (2007). [4] G. Lippmann, La photographie integrale, Acad. Sci., Paris, C. R. 146, 446 451 (1908). [5] A. Aggoun, 3D Holoscopic Video Content Capture, Manipulation and Display Technologies, Information Optics (WIO), (2010) [6] A. Aggoun, 3D Holoscopic Imaging Technology for Real-Time Volume Processing and Display, High- Quality Visual Experience Signals and Communication Technology, 411-428, DOI: 10.1007/978-3-642-12802- 8_18, (2010). [7] N. Davies, et. al., Three-dimensional imaging systems: A new development, Applied Optics, 27, 4520, (1988). [8] M. Martínez-Corral and B. Javidi, Formation of real, orthoscopic integral images by smart pixel mapping, Opt. Express, 13, 9175-9180, (2005). [9] H. Jorke and M. Fritz, Stereo projection using interference filters, Proc. of SPIE, 6055, (2006). [10] L. Zhang and W. J. Tam, Stereoscopic Image Generation Based on Depth Images for 3D TV, IEEE Transaction on Broadcasting, 51, 191-199, (2005) [11] J.-Y. Son and B. Javidi, Three-Dimensional Imaging Methods Based on Multiview Images, J. Disp. Technol., vol. 1, no. 1, pp. 125 140, Sep. 2005. [12] C. Slinger, C. Cameron, and M. Stanley, Computergenerated holography as a generic display technology, IEEE Computer, 38, 46 53 (2005) [13] P. Papageorgas, S. Athineos, N. Sgouros, N. Theofanous, 3D capturing devices based on the Principles of Integral Photography, T.E.I of Piraeus & University of Paisley, 1st International Scientific Conf. era, Sept. 16-17,Tripoli, 2006. [14] A. Aggoun, Pre-Processing of Integral Images for 3-D Displays, J. Disp. Technol., vol. 2, no. 4, pp. 393 400, Dec. 2006. [15] S. Lee, S. Hong, Y. Kim, and H. Lim, Hologram synthesis of three-dimensional real objects using portable integral imaging camera, Opt. Express, vol. 21, no. 20, pp. 23662 23670, 2013. [16] O. Abdul Fatah, P. M. P. Lanigan, a. Aggoun, M. R. Swash, E. Alazawi, B. Li, J. C. Fernandez, D. Chen, and E. Tsekleves, Three-dimensional integral image reconstruction based on viewpoint interpolation, 2013 IEEE Int. Symp. Broadband Multimed. Syst. Broadcast., pp. 1 4, Jun. 2013.