Depth extraction from unidirectional integral image using a modified multi-baseline technique

Transcription

1 Depth extraction from unidirectional integral image using a modified multi-baseline technique ChunHong Wu a, Amar Aggoun a, Malcolm McCormick a, S.Y. Kung b a Faculty of Computing Sciences and Engineering, De Montfort University, U.K. b Electrical Engineering, Princeton University, USA ABSTRACT Integral imaging is a technique capable of displaying images with continuous parallax in full natural colour. This paper presents a modified multi-baseline method for extracting depth information from unidirectional integral images. The method involves first extracting sub-images from the integral image. A sub-image is constructed by extracting one pixel from each micro-lens rather than a macro-block of pixels corresponding to a micro-lens unit. A new mathematical expression giving the relationship between object depth and the corresponding sub-image pair displacement is derived by geometrically analyzing the three-dimensional image recording process. A correlation-based matching technique is used to find the disparity between two sub-images. In order to improve the disparity analysis, a modified multi-baseline technique where the baseline is defined as the distance between two corresponding pixels in different sub-images is adopted. The effectiveness of this modified multi-baseline technique in removing the mismatching caused by similar patterns in object scenes has been proven by analysis and experiment results. The developed depth extraction method is validated and applied to both photographic and computer generated unidirectional integral images. The depth estimation solution gives a precise description of object thickness with an error of less than.0% from the photographic image in the example. Keywords: Integral Image, Depth Extraction, Multi-baseline Technique. INTRODUCTION The development of three-dimensional (3-D) imaging systems is a constant pursuit of the scientific community and entertainment industries. Many applications exist for fully three dimensional video communication systems. One much discussed application is 3-D television. Integral imaging is a 3-D display technique capable of encoding a true volume spatial optical model of the object scene in the form of a planar intensity distribution by using unique optical capture apparatus. The recorded planar intensity distribution can be stored or transmitted as a conventional two dimensional pixel array. It is akin to holography in that 3-D information is recorded on a two dimensional medium, but does not require coherent light sources. This allows continuous parallax, wide viewing zone and very good live capture and display practicality. All integral imaging can be traced from the work of Gabriel Lippmann, 908 [], where a micro-lenses sheet was used to record the optical model of an object scene. A full natural colour scene with continuous parallax can be replayed when another micro-lenses sheet with appropriate parameters is used to overlay the original image, see figure. A modification to the system was proposed by Ives [] where a two-stage photograph is used to overcome the problem imposed by the pseudoscopic (spatially depth invert) nature of the image. A two-tier network as a combination of macro-lens arrays and micro-lens arrays designed by Davies and McCormick [3][4] further overcomes the image degradation caused by the two-stage recording. The two-tier network works as an optical transmission inversion screen, which allows direct spatially correct 3-D image capture. With progress in micro-lens manufacturing, integral imaging is becoming a practical and prospective 3-D display technique and hence is attracting much interest.

2 Figure. The principle of integral image This paper is concerned with the extraction of depth information and the reconstruction of a 3-D scene from the recorded integral image data. One particular usage of depth extraction is to enable content-based interactive manipulation, which allows flexible operations on visual objects to be carried out. Hence, real and computer generated 3-D object space can be combined in a virtual studio. It can also be used for content-base image coding. Recently, a method for extracting depth based the point spread function (PSF) of the optical recording process has been reported [5] and [6]. The object space is conceived as a discrete set of points endowed with intensity. A correspondence matrix is associated to the PSF function, which transforms the object space into the pixel defined integral image. Depth estimation from the 3-D integral image data is formulated as an inverse problem. Although this technique works well on synthetic numerical experimental data, produced using the PSF matrix assumed, further research is needed towards its possible application to the real integral image [5] and [6]. This paper presents an alternative method for extracting depth information from an integral image. The method involves first extracting sub-images from the integral image. A sub-image is generated by taking one pixel out from each micro-lens unit rather than the macro block corresponding to the micro-lens unit. Each sub-image contains pixels at the same position under different micro-lens hence each sub-image record the object scene from and only from one particular direction. A mathematical expression giving the relationship between the depth of the object and the corresponding sub-images displacement is then derived by geometrically analyzing the integral image recording process. Correlation-based disparity analysis methods are used and a multi-baseline technique is adopted in order to remove the mismatch arising from ambiguity in object space. Application and validation of this method is presented for both photographic and computer generated images. Although current work is applied to unidirectional integral images, extension of the technique to omni-direction integral images (parallax in all direction) is straightforward.. EXTRACTING SUB-IMAGES FROM UNIDIRECTIONAL INTEGRAL IMAGE The key point of the integral image involves using a micro-lenses sheet in recording. A micro-lenses sheet is made up of many micro-lenses having the same parameters and lying in the same focal plane, each micro-lens works as an individual small low-resolution camera. A recording film is placed behind the micro-lenses sheet and coincident with the focal plane. As all parallel incident rays pass through the same point in the focal plane after refraction in an ideal lens, the parallel incident rays will be recorded at the same position under each micro-lens. The different recording position only depends on which micro-lens surface it reaches, as shown in figure. As an example, all rays along direction θ will be recorded on position numbered n while all rays along θ will be recorded on position numbered n, in other words, all pixels at the same position under different micro-lenses record the object scene from the same direction. Therefore a new image representing a particular viewpoint can be composed by simply sampling the pixels at the same position under different micro-lenses. This new image (termed here sub-image) records the object scene from and only from one particular view direction. Changing the positions of the pixels selected, other sub-images can be constructed. Figure 3 illustrates how the sub-image is extracted from the unidirectional integral image. For simplicity,

3 only four pixels are assumed under each micro-lens. Pixels in the same position under different micro-lenses, which record rays from one particular direction and represented by the same color, are employed to form one sub-image. Figure. The direction selectivity of integral recording procedure Figure 3. Illustration of sub-image extraction Figure 4a is an example of a unidirectional integral image. The original object scene contains a flat background with Chinese characters and a small box attached to the front of it. The pitch size (%), focal length (F) of the micro-lenses and the radius of curvature (r) of the micro-lens surface are 600nm,.37mm and 0.88mm, respectively. The optical system used to take the image is explained in Davies and McCormick [4]. The 3-D scene can be replayed by scaling the image back to the original size (0.0cm 9.065cm) and overlaying with micro-lens sheet having the same parameters. Figures 4b and 4c are two sub-images extracted from the unidirectional integral image in figure 4a. Each sub-image presents a two-dimensional (-D) recording of the object space from a particular direction. These sub-images are used in the following section for the disparity analysis. (a) (b) (c) Figure 4. An example of an unidirectional integral image example and two corresponding sub-images

4 3. MATHEMATICAL RELATIONSHIP BETWEEN OBJECT DEPTH AND SUB-IMAGES DISPLACEMENT Further sub-images are extracted from the original integral image and a correlation-based matching technique is used to find the disparity between sub-images. The extracted sub-image pairs are different to those used in stereoscopic calibration since they are generated from a different source. Geometric analysis on the optical recording procedure is necessary to find the relationship between the object depth and the disparity information of sub-images. Figure 5 depicts the Cartesian coordinate system used in the analysis. Only one dimensional disparity is considered here since the unidirectional integral image is being discussed. The z-axis denotes the depth and the x-axis represents the lateral position. The z-axis starts from the plane coinciding with the micro-lenses surface, while the x-axis starts from the center of the first micro-lens. Consider an object point P (x 0, D) at distance D from the micro-lens surface plane. Suppose the first sub-image is obtained by choosing pixels at distance ds from the micro-lenses center, which records rays from the direction. The ray from P along the direction intersects the plane of the micro-lenses sheet surface (x-axis) at the N th micro-lens. The intersection is at P (x,0). Following lens refraction, the ray is recorded on film at point Q (x, -t) at distance ds from the micro-lens center, see figure 5b. The following geometric relationship can be easily obtained from figure 5: ( D + d r ) ds 0.5) ψ < x0 + < ( 0.5) ψ F ( N N A similar relationship can be obtained for the second sub-image: ( D + d r ) ds 0.5) ψ < x0 + < ( 0.5) ψ F ( N N In this paper, the baseline is defined DV û GV -ds, which represents the sampling distance between two sub-images. The name baseline is inherited from stereoscopic. Since only one pixel is sampled out from each micro-lens in each sub-image, the disparity between two sub-images, d, corresponds to the micro-lenses numbers differences between the position Q and Q, d= N N. Substituting d and û and manipulating equation (), () yields the following: () ()

5 ( d ± ) ψ F D = d r (3) Here d ± means the expected value is d but it may vary from the range d- to d+. In most cases dr<<d, the depth equation can be simplified as or ( d ± ) ψ F D = D d = ± (5) ψ F For an object at a particular depth, equation (5) describes the relationship of the disparity between two sub-images and the object depth, given the baseline between the two sub-images and micro-lenses parameters. It can be seen from the equation that the disparity is proportional to the object depth and also increases with distance between two subimages (baseline) increase. The equation also reveal that the accuracy of the depth estimate obtained by this method is OLPLWHG E\ % ) 7KLV GHSWK HVWLPDWH DPELJXLW\ LV FDXVHG E\ WKH DPELJuity existing in the recording process. The coordinate of P in the X direction can be solved from equation () using the achieved depth information: D ds x 0 = N ± ψ (6) F The original object scene can be reconstructed by mapping the intensity information on the reconstructed depth map. (4) 4. DISPARITY ANALYSIS AND OBJECT DEPTH ESTIMATION Having derived the mathematical relationship between object depth and disparity information of the sub-images, efforts are made to find the correct disparity information from sub-images. Obviously depth estimation relies upon the precision and the correctness of the disparity analysis. In this section, correlation-based disparity analysis is carried out. 4. Correlation-based Disparity Analysis Method Three popular correlation-based disparity analysis methods are used to find out the displacement between two subimages, namely, sum of the square difference (SSD), mean absolute error (MAE) and cross correlation (CC). Only one directional disparity is considered since a unidirectional image is used here. Assuming two sub-images, I and I, (x,y) are the coordinates of the point being analyzed. I(x,y) is the intensity of the point (x,y). w+ is the width of the correlation window used in matching and R is the search scope in the second image associate with the first image. The three correlation-based disparity methods can be described as: SSD (d)= arg min d R w w i= w j= w [ I ( x + i, y + j) I ( x + i + d, y + j)] (7) MAE (d)= arg min I( x + i, y + j) I ( x + i + d, y + d R w w i= w j= w j) (8) CC (d) = arg max w w I ( x + i, y + j) I i= w j= w d R σ ( x, y) σ ( x + ( x + i + d, y + d, y) j) (9)

6 w w Where I ( x, y) = I( x + i, y + j) is the average intensity of the chosen window, N i= w j= w I ( x + i, y + j) = I ( x + i, y + j) I ( x, y) is the intensity after local adjustment, σ = w w N I i= w j= w ( x + i, y + j) is the mean square error of the chosen window, N = ( w + ) (w + ) is the window size of the window. arg min, arg max means for all d R d R d R maximum value of the function respectively., find d for which the equation obtains the minimum value or Disparity analysis with all the above three correlation algorithms was performed on the three selected windows of the sub-image shown in Figure 6, which represent areas containing either object or background only. Figures 7, 8 and 9 show the disparity analysis results for each window for the three methods. The plot is normalized such that the maximum SSD(d) and MAE(d) are equalized to respectively. The searching scope R is from 0 to 0 pixels for object (window ) and 0 to 0 pixels for background (window and 3). The disparity of window of object is identified as 8 pixels as shown in figure 7 and the disparity of window of background can be recognized at pixels as shown in figure 8. The difference in the results among the three correlation-based methods is insignificant. However, when the disparity between image pairs is examined in window 3, see figure 9, a disparity of -5 is obtained from the SSD and CC method and 9 from the MAE method respectively. All of the three curves tends to periodical and have extrema around 5, and 9. This is caused by the similar background pattern that exists. Mismatching around the extrema positions occurs due to noise and other intensity distortion exist in recording process, such as the unbalanced illumination, projection distortion, etc. It can be noticed that even when a large window is chosen from the background, window, figure 8, there still exist local extrema at d=-5 and d=9 that are competitive to the global extrema. In this situation, correct disparity is obtained from the global extrema because a larger window size is chosen, which provides a higher probability of obtaining a higher signal-to-noise ratio. A multi-baseline technique is adopted in order to improve the disparity analysis result. Window3 Window Window Figure 6. Region choosing in depth estimation

7 0.8 MAE SSD CC disparity (d) Figure 7. Disparity analysis on window (object region) 0.9 SSD MAE CC disparity (d) Figure 8. Disparity analysis on window (background region) 0.8 MAE 0.6 SSD CC disparity (d) Figure 9. Disparity analysis on window 3 (background region)

8 4. Removing Mismatching with a Modified Multi-baseline Technique In the previous sections, a way of obtaining depth information from unidirectional integral images by extracting subimages from integral image and analyzing disparity between sub-images has been described. It is obvious from the equation (4) that the precision and the correctness of the depth estimate depend upon the performance of disparity analysis obtained. Consequently, a correct disparity analysis is of great importance in order to have correct depth estimation. As discussed in the last section, mismatching tends to occur when periodic or ambiguous information exists in the object scene. In order to tackle this problem, multi-baseline technique is adopted which uses several sub-image pairs simultaneously in the disparity analysis. Theory and experiments prove that the technique works effectively in remove mismatching. The multi-baseline technique was proposed by Okutomi and Kanadi [7] as a stereo matching method using multiple stereo pairs with various baselines in obtaining precise distance estimates without suffering from ambiguity. Unlike the general fusing technique, which computes the displacement for each pair first and then calculates the final displacement from the intermediate results, this technique accumulates the matching evaluation function from all the sub-image pairs into a single evaluation function and makes one single decision at the end. The definition of the baseline in this work is different from that used in the stereoscopic case and consequently the disparity and depth equation is different from that for stereoscopy. Instead of computing the SSSD-in-inverse distance function, SSSD(/D), a direct distance function,-scc(d), is used here. Using SCC(D) instead of SSSD(/D) makes the calculation and illustration easier since the disparity is proportional to the depth here and the CC function is a normalized function itself. In addition it is much easier to find the minimal than the maximal of a function. Figure 0 illustrate the disparity analysis result for window 3 obtained by using this modified multi-baseline technique. In order to compare, horizontal axis is normalized to the longest baseline searching scope. Compared to the evaluation function obtained by direct disparity analysis from a pair, i, -CC i (d), only one clear and sharp minimum exists in -SCC (d) curve. The error caused by mismatching in the situation has been effectively removed by the new approach. Figure 0. Disparity analysis on window 3 by using multi-baseline based method (-SCC(d)) 4.3 Precise Depth estimate and Depth Mapping of 3-D Object Space Choosing five points around the minimum and using a polynomial curve fitting method allows the precise disparity estimate of the background to be obtained as.475 pixel. The corresponding depth can be calculated as.43mm by using equation 5. Applying the same method to window, the disparity is founded to be pixels indicating the precise object depth is at 8.3mm, as shown in figure. The object thickness can be identified as 5.7mm, which is very close to the actual value of 5.6mm manually measured with Vernier caliper.

9 dispaity (d) Figure. Polynomial curve fitting and precise solution finding Applying the same technique on every pixel of the sub-image generates the dense depth map of the object scene, shown in a. The window size used in this example is *. With this depth map, the reconstructing of threedimension object space becomes straightforward by mapping the original intensity image on the construct depth map. The reconstructed object scene is shown in figure b. The depth of the object is correctly estimated for all but the shadow region below and to the right edge of the object. Figure. Depth map and object space reconstruction from unidirectional integral image using multi-baseline technique A computer generated unidirectional integral image was also produced and analyzed to compare the result obtained from the photographic integral image. Figure 3a shows the computer-generated image using modified PoV-Ray software. It contains a background plane with Chinese characters, the same as that in photographic image and a thin box in front of it. The parameters of the recording micro-lenses sheet are the same as in photographic image. Figure3b and 3c are two sub-images extracted from the computer generated integral image. Figure4 is the corresponding depth map and object space reconstruction from the image. A 7*7 window size is used in this disparity analysis. Better resolution can be obtained from the computer-generated integral image since balanced illumination is used and also it is noise free. The notable error in the computer-generated image is along the right edge of the object, which is mainly caused by occlusion.

10 a) b) c) Figure 3. Computer generated integral image a) b) Figure 4. Depth map and object space reconstruction from computer generated image 5.CONCLUSION In this paper, a method of depth extraction from a unidirectional integral image using disparity analysis and a modified multi-baseline technique has been described. This method involves first extracting sub-images from the integral image. The relationship between object depth and sub-image displacement is derived through geometrical analysis of the optical recording procedure. A modified multi-baseline technique is used to improve the disparity analysis results. The developed depth extraction method is applied to both captured and computer-generated images. Results show that the proposed method works effectively in both applications. Although only unidirectional integral images are discussed, the method can be easily applied to omni-direction integral images by processing continuous parallax in all direction with minor adjustments. Further work is ongoing to improve the disparity analysis accuracy. REFERENCES [] Lippmann G: La Photographie integrale, comtes Rendus, Academie des Sciences, Vol.46, pp ,908. [] Ives H E: Optical Properties of a Lippmann Lenticulated Sheet, J. Opt. Soc. Amer., Vol., pp7-76,93. [3] Davis N and McCormick M: Holoscopic imaging with true 3D-content in full natural colour, J. Phot. Science, Vol. 40, pp46-49, 99. [4] Davies N, McCormick M and Brewin M: Design and analysis of an image transfer system using microlens arrays, Opt. Eng., Vol. 33, No., pp , 994.

11 [5] Manolache S, Aggoun A, McCormick M and Davies N: A mathematical model of 3D-unidirectional integral recording system, Proceedings of Vision, Modeling, and Visualisation '99', pp.5-58, Erlangen, Germany, Nov [6] Manolache S, McCormick M and Kung SY: Hirearchical adaptive regularization method for depth extraction from planar recording of 3D-integral images, Proceedings of ICASSP, May, 00. [7] Okutomi M and Kanade T: A Multiple-Baseline Stereo, IEEE Transactions on Pattern Analysis and Machine Intelligence,