A Fast Image Multiplexing Method Robust to Viewer s Position and Lens Misalignment in Lenticular 3D Displays

Transcription

1 A Fast Image Multiplexing Method Robust to Viewer s Position and Lens Misalignment in Lenticular D Displays Yun-Gu Lee and Jong Beom Ra Department of Electrical Engineering and Computer Science Korea Advanced Institute of Science and Technology (KAIST) Daejeon, Republic of Korea ABSTRACT Among various autostereoscopic display systems, the lenticular display is one of the most popular systems due to its easy manufacturability. For N-view lenticular display, N view images are to be regularly sub-sampled and interleaved to produce a D image. A lenticular system provides the best quality only when a viewer locates at a pre-determined optimal viewing distance and the lenticular sheet is precisely aligned on the LCD pixel array. In our previous work, we have proposed an algorithm to compensate the viewer s position change and the lenticular misalignment. However, since the previous algorithm requires a considerable computational burden, we propose a new fast multiplexing algorithm. To improve the processing speed, we introduce a mapping table instead of directly using complex equations. In contrary to the previous algorithm, the proposed one can make real time compensation possible without degrading image quality. Keywords: Lenticular display, autostereoscopic display, D display, misalignment. INTRODUCTION In order to visualize D images, various display technologies such as anaglyph, volumetric display, lenticular display, parallax barrier display, and integral photography, and hologram display, have been developed []-[]. Among them, the lenticular display is one of the most popular, because it can be easily manufactured, provide high brightness, and generate multiple views [], []. In a lenticular display system, the LCD pixel array is located at the focal plane of the lenticular sheet that is a cylinder-shape lens array. Since the lenticular sheet refracts the light from the LCD pixel array, images observed through the lenticular sheet vary depending on a viewer s position. Thus, the images viewed from the left and right eyes are different, so that the viewer may perceive the depth information or the system may generate stereoscopic information []. In order to generate a multi-view image for an N view lenticular display system, N view images are first prepared and multiplexed [-]. Then, the generated multi-view image is allocated to LCD pixels. Thereby, the lenticular display provides N different views depending on the viewer s eye position. However, the resolution of an observed view image is inversely proportional to N. If the lenticular display provides multiple views by sacrificing the horizontal resolution only, the horizontal and vertical resolutions of a D image will be mismatched. Hence, a slanted lenticular display system has been introduced to distribute the resolution degradation into both x and y directions []. In a practical lenticular display system, D display quality is degraded due to intrinsic and extrinsic problems [0]. Intrinsically, the optimal viewing distance from a viewer to LCD display is fixed, once design parameters of a system are determined. So, if a viewer is not located at the optimal viewing distance, the system produces undesirable display jbra@ee.kaist.ac.kr; phone +---; fax +---0; Dept. of EECS, KAIST, -, Guseong-dong Yuseong-gu, Daejeon, 0-0, South Korea Stereoscopic Displays and Virtual Reality Systems XIII, edited by Andrew J. Woods, Neil A. Dodgson, John O. Merritt, Mark T. Bolas, Ian E. McDowall, Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 0, 00M, 00 SPIE-IS&T 0-X/0/$ SPIE-IS&T/ Vol. 0 00M-

2 A L H B L D d 0 LCD sub-pixel array T O T I T F Fig.. Relationship between a viewer s eye position and a lenticular display system distortion. Extrinsically, a lenticular sheet may not be precisely aligned on the LCD panel, especially when the lenticular sheet is slanted to solve the imbalance problem between horizontal and vertical resolutions. This alignment error causes considerable distortion in D display. In order to alleviate these two major problems, we had proposed a method to generate a more appropriate multi-view image for the lenticular image display with less distortion [0]. The method first finds the amount of alignment error by using a pattern image. Then, it analyzes and compensates the system to reduce D image distortion. However, since the algorithm is quite complex and time-consuming in generating a multi-view image, it is not adequate for a real time system. In this paper, we propose a new fast multiplexing algorithm. Instead of detecting the alignment error of lenticular sheet, the proposed method finds the contribution of LCD sub-pixels to a D image observed from a viewing zone by using N pattern images. Note here that the contribution can be directly obtained from pattern images without analyzing or modeling the system. Then, a mapping table between the LCD sub-pixels and original view images is generated based on the measured contribution. Since the compensation procedure using a mapping table is fairly simple, it can be easily combined with a procedure to compensate the other extrinsic problem, namely, the inhomogeneity of lenticular sheet. A very homogeneous lenticular sheet is generally expensive, especially when the pitch of lens is narrow. Therefore, in a LCD panel for high resolution display, distortion correction due to the lenticular inhomogeneity is meaningful. The remainder of this paper is organized as follows. Section describes several problems in the lenticular display in detail. The fast scheme to generate a multi-view image is proposed in Sections. Section provides experimental results. Finally, conclusions are given in Section. SPIE-IS&T/ Vol. 0 00M-

3 . D IMAGE DISTORTION An N-view lenticular display system provides N different view images or generates N different viewing zones. Thereby, a viewer s eye locating at the kth viewing zone ideally observes only the kth view image. However, due to the three problems mentioned above, a viewer observes unwanted pixel intensities coming from several neighboring view images. Even if a viewer s eye moves within the same viewing zone, an observed image changes depending on the distance from a viewer s eye to the lenticular display. Fig. illustrates this intrinsic problem or the change of observed sub-pixel depending on a viewer s eye position. Let us assume in the figure that the homogeneous lenticular sheet is precisely aligned on the LCD pixel array and the optimal viewing distance of the system is located at the infinite. We also assume that the viewer s eye always locates in the th viewing zone. If the distance from the LCD display to the viewer is infinite, he/she will see sub-pixel through the nd lens. Meanwhile, if a viewer s eye locates at positions A and B, he/she will see sub-pixels and, respectively. Hence, a multi-view image multiplexed by using a fixed mapping table cannot provide the best D image for the change of viewing distance. A similar phenomenon can be observed for a system having a finite optimal viewing distance.. Therefore, a viewer s eye location should be considered in generating a mapping table. Now we consider the two extrinsic problems, namely, the misalignment and inhomogeneity ones of the lenticular sheet. In attaching a lenticular sheet on the LCD panel, there may exist alignment errors such as translational and rotational ones. Although these errors are very small, they can noticeably affect the display quality [0]. Also, the inhomogeneity of lenticular sheet causes the D image distortion. The inhomogeneity problem is mainly due to the nonuniform pitch values on the LCD panel and provides the mismatch between the lenticular sheet and LCD sub-pixels. Thereby, an observed image can contain sub-pixels corresponding to wrong view images. While the pitch error due to lenticular misalignment is consistent over the whole LCD panel, the pitch error due to the inhomogeneous lenticular sheet varies depending on a geometric location on the panel. An example given in Fig. demonstrates how the extrinsic and intrinsic problems distort a D image. Ideally, the indices of view image in Fig. (b) should be for all the sub-pixels. However, due to the problems mentioned above, the index may increase or decrease along the horizontal and vertical directions. Therefore, when a viewer watches the st view image nd view image th view image (a) (b) (c) Original line Observed line Distorted part Abrupt change Fig.. An example of D image distortion. (a) Original view images. (b) View indices at sub-pixels in the th observed view image and (c) the corresponding distorted image. Ideally all view indices are to be in (b). SPIE-IS&T/ Vol. 0 00M-

4 lenticular display, the distorted black line is observed at the th viewing zone instead of the desired straight gray line (see Fig. (c).). Note here that the abrupt change occurs when the view number changes from the th view to the st one.. IMAGE MULTIPLEXING FOR D DISPLAY In this section, we will propose a multiplexing algorithm for distortionless D image display in the lenticular system. Although the algorithm considers only a view lenticular display system, it can be easily extended into the general N view lenticular system.. Mapping table Fig. illustrates the cross section of the lenticular system and viewer s eye position. As shown in the figure, a lenticular sheet is misaligned to the LCD pixel array, and a viewer is not located at the optimal viewing distance. In this case, the value at a sub-pixel in the original mapping table is different from the index of the viewing zone where a light emitted from the sub-pixel arrives. Namely, a viewer s eye locating at the th viewing zone sees a sub-pixel having a value of in the mapping table. If we assume that T O denotes the original mapping table, the final mapping table T F that compensates all the intrinsic and extrinsic problems mentioned above can be given as T F = T O + T E + T I. () Here, T E and T I denote compensating terms for the extrinsic and intrinsic problems, respectively. And T F, T E, T I, and T O are matrices with a size of M N, and M and N are the vertical and horizontal resolutions of the LCD panel, respectively. T E is constant for a given product and T I varies according to a viewer s eye position. In the following subsections, we will explain how to predict T E and T I. Note here that all the values in the table have floating point accuracy. For example, if the view number of a sub-pixel in T F is., it means that the light emitted from the center of a sub-pixel on the LCD panel arrives in the middle of the rd and th viewing zones. Located at the th viewing zone Practical lenticular sheet Ideal lenticular sheet Fig.. Cross section of a lenticular display system and the relationship among tables T O, T E, and T F T O T E T F SPIE-IS&T/ Vol. 0 00M-

5 . Prediction of T I As mentioned in Subsection., a viewer locating at a non-optimal viewing distance sees a wrong sub-pixel. So we introduce T I to compensate the mismatch between the view index in the original mapping table and a viewing zone where a viewer s eye locates. In order to predict T I, we calculate the displacement, d, between the locations observed at the optimal viewing distance and a practical viewing distance as shown in Fig.. The value of d can be calculated as [0] d L = f H. () nr ( LD + LH ) LH Here, f is the focal length of the lenticular sheet, L D denotes the vertical distance from the lenticular sheet to the eye, L H denotes the horizontal displacement from the center viewing zone (or the th viewing zone in Fig. ) to the observed lenticular center, and n r is the refractive index of the lenticular sheet. In practice, it is not easy to know the exact value of L H due to the misalignment and inhomogeneity of the lenticular sheet. And it makes difficult to obtain the accurate value of d. However, since the width of a sub-pixel is much smaller than L D, the value of d can be well estimated from Eq. () even though L H is not very accurate. In order words, the value of d slowly increase or decrease along the horizontal direction, and those at neighboring sub-pixels are almost the same. Therefore, even if we calculate the value of d by assuming a precisely aligned lenticular sheet, the error will be negligible. The value of d is interpreted as a shift of view index in the mapping table. For example, the position observed from eye A is shifted by sub-pixel to the left at the nd lens. In order to compensate this intrinsic problem, the mapping table should be modified by considering this value. Since the view index difference between successive sub-pixels in the mapping table is for a view slanted lenticular system, sub-pixel shift corresponds to a shift of two viewing zones. Hence, the value at the corresponding sub-pixel is set to in T I as in Fig.. In general, the value of each sub-pixel in T I is predicted by using d(m, n),.namely, d( m, n) T ( m, n) (mod ), () I p L where (m, n) denotes the position of sub-pixel on the LCD pixel array and p L is the horizontal length of a sub-pixel. Note here that values of only one row are needed to obtain all the values in T I. According to Eq. (), we can change the optimal viewing distance by simply modifying the mapping table. We should also note that any viewing zone can become a center viewing zone by redefining L H in Fig.. Therefore, if a system can track or detect a viewer s eye position, the scheme can provide the best D image even though the viewer locates at any position... Prediction of T E In our previous work, we have assumed that a lenticular sheet is homogeneous [0]. Hence, only the misalignment problem of the lenticular sheet was considered. Since we deal with an inhomogeneous lenticular sheet in this paper, the lenticular pitch may be different depending on the position. And it causes an additional mismatch problem between the LCD pixel array and lenticular sheet. Thus, the error of lenticular pitch should be described for the two extrinsic problems and the corresponding compensation scheme has to be suggested. In this sub-section, we will introduce table T E to compensate the mismatch coming from those extrinsic problems. Fig. represents the relationship among T O, T E, and T F in a misaligned lenticular display system. In the figure, a viewer s eye locating at the th viewing zone sees a sub-pixel that is sub-sampled from the th view image. Hence, the value of T E at the sub-pixel should be assigned to - to provide a proper D image. By using Eq. () and ignoring the intrinsic problem, the value in the final mapping table T F becomes. Thus, if we know the geometric relationship between the LCD sub-pixels and the lenticular sheet, T E can be predicted. SPIE-IS&T/ Vol. 0 00M-

6 For detecting the alignment error and inhomogeneity of a lenticular sheet, we use the nine multi-view pattern images. In generating the ith multi-view pattern image, the ith view image is set to be white and the others to be black. Then, they are multiplexed by using the original mapping table. Then, each pattern image is observed at the th or the center viewing zone. It should be noted that the images should be observed at a distance sufficiently far from the lenticular display, in order to avoid the image distortion due to the intrinsic problem. If there are no extrinsic problems, the th pattern image is to be observed as the pure white image, and the others as the pure black one. In this case, all the values of T E are zero. Otherwise, T E should be calculated from the captured pattern images. If point (n, m) in the ith captured pattern image is white, a viewer s eye locating at the th viewing zone sees a sub-pixel sampled at point (n, m) in the ith view image. So, the value of T E at (n, m) should be set to (-i) as shown in Fig.. Meanwhile, if a viewer s eye locating at the th viewing zone watches the display, the th and th view images as well as the th view image are concurrently observed in a slanted lenticular display system. So if point (n, m) in the ith captured pattern image is white, the same points in the (i-)th and (i+)th captured pattern images may not be black but gray. A corresponding example is demonstrated in Fig.. As shown in the figure, their values of the th, th, and th observed pattern images are not black. Hence, the value of T E can be calculated by estimating the peak position. Note here that values of T E have floating-point accuracy. Intensity. The kth observed view image k Fig.. Peak detection at a sub-pixel in the nine observed view images.. EXPERIMENTAL RESULTS To verify the proposed algorithm, we set up a lenticular display system by attaching a lenticular sheet to a LCD display panel and perform experiments using several D images. The resolution of the LCD panel is 0 0 and its dot pitch is 0.mm. Fig. represents the pictures observed from the th pattern image. The pictures are taken at the distances with 00mm and 000mm, respectively. We can note in the figure that, since the lenticular sheet is not homogeneous, intensity variation in observed pattern images is not regular or symmetric. Also, two observed pictures from the same pattern can be different even if they are taken in the same viewing zone. It means that the distorted aspect of observed D images varies according to the distance from the display due to the intrinsic problem. Performance of the proposed algorithm is demonstrated in Fig.. While Figs. (a) and (b) show D images obtained by multiplexing the nine view images with the original mapping table, Figs. (c) and (d) shows D images obtained by using the proposed final mapping table to compensate the intrinsic and extrinsic distortions. Here, Figs. (a) and (c) are taken from a distance of 00mm and Figs. (b) and (d) are from a distance of 000mm, respectively. Note in Figs. (a) and (b) that D images include unwanted distortion and vary depending on the camera position. Meanwhile, the proposed algorithm provides the best D images regardless of the camera position. SPIE-IS&T/ Vol. 0 00M-

7 (a) (b) Fig.. Pictures of a pattern image, which are practically taken from the th viewing zone with distances of (a) 00mm and (b) 000mm, respectively. 'If''' (a) (b) qiiii (c) (d) Fig.. Pictures of a synthesized D image which are taken from the th viewing zone with distances of (a), (c) 00mm and (b), (d) 000mm, respectively. The synthesized image consists of vertically allocated parallel rods. Results (a), (b) without compensation; and (c), (d) with compensation SPIE-IS&T/ Vol. 0 00M-

8 . CONCLUSION In this paper, we propose a new fast multiplexing algorithm to solve the intrinsic and extrinsic problems in the lenticular display system. While our previous work requires a considerable computational burden, the newly proposed scheme significantly reduces the computation complexity by using a mapping table. The table includes two compensation terms related with a viewer s position and the alignment error. In addition, by using the proposed algorithm, we can easily deal with the lenticular inhomogeneity, problem which also degrades the D display quality. A homogeneous lenticular sheet is generally expensive, especially when the pitch of lens is very narrow. Moreover, lenticular misalignment and inhomogeneity problems become more critical as the LCD resolution becomes higher. Therefore, the proposed software approach seems to be very useful in obtaining a good quality of D images regardless of the system imperfectness or a viewer s position. ACKNOWLEDGEMENT This work was supported in the part by the university IT research center program of the government of Korea. The authors would like to thank Samsung SDI Co., Ltd for providing a lenticular monitor. REFERENCES [] P. Soltan, M. Lasher, et al., Laser-projected D volumetric displays, Proc. SPIE, Projection Displays II, vol. 0, pp. -,. [] L. Lipton and M. Feldman, A new autostereoscopic display technology: The SynthaGram TM, Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems, vol. 0, pp. -, 00. []. F. Okano, H. Hoshino, J. Arai, and I. Yuyama, Real-time pickup method for a three-dimensional image based on integral photography, Appl. Opt., vol., pp. -0,. [] A. Schmidt and A. Grasnick, Multi-viewpoint autostereoscopic displays from D-Vision, Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems, vol. 0, pp. -, 00. [] C. V. Berkel, A. R. Franklin, and J. R. Mansell, Design and applications of multiview D-LCD, Proc. SID Euro- Display, pp. 0-,. [] J. Konrad and P. Agniel, Artifact reduction in lenticular multiscopic -D displays by means of anti-alias filtering, Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems, vol. 00A, pp. -, 00. [] J. Konrad and P. Agniel, Non-orthogonal sub-sampling and anti-alias filtering for multiscopic -D displays, Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems, vol. A, 00. [] K. Mashitani, G. Hamagishi, M. Higashino, T. Ando, and S. Takemoto, Step barrier system multi-view glass-less - D display, Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems, vol., pp. -, 00. [] C. V. Berkel, Image preparation for D-LCD, Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems, vol., pp. -,. [0] Y. G. Lee and J. B. Ra, Reduction of the distortion due to non-ideal lens alignment in lenticular D displays, Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems, vol., pp. 0-, 00. SPIE-IS&T/ Vol. 0 00M-