Crosstalk in multiview 3-D images

Transcription

1 Invited Paper Crosstalk in multiview 3-D images * Jung-Young Son, 1 Beom-Ryeol Lee, 2 Min-Chul Park, and 2 Thibault Leportier Dept. of Biomedical Engineering, Konyang University, Nonsan, Chungnam, , Korea 1 Next generation visual computing research Section, ETRI, Daejeon, , Korea 2 Sensor Systems Research center, Korea Institute of Science and Technology, Seoul, , Korea * jyson@konyang.ac.kr Abstract Crosstalk in the contact-type multiview 3-D images is not an effective parameter of defining the quality of 3-D images. This is because the viewing zone in the contact-type multiview 3-D displays allows viewing the images which are composed of an image piece from each view image in a predefined set of consecutive view images, except the part along the viewing zone cross section. However, this part cannot guarantee to view individual view images separately because the viewing region of each view image is contacted to its neighboring viewing regions through a point for each neighbor due to its diamond like shape. Furthermore, the size of each view region can be smaller than the viewers pupil sizes as the pixel size decreases and/or the number of view images increases as in super-multiview imaging. The crosstalk has no meaning. Keywords: Crosstalk, contact-type multiview 3-D image, viewing region, image pieces, patched image 1. Introduction In the 3-D image, the crosstalk is defined as the interference between neighboring view images [1]. The term has been used to quantify the quality of 3-D images but it is not appropriate for the contact-type multiview 3-D displays [2] because the viewing zones of this type displays are divided into a number of diamond shaped viewing regions, which is much more than that of the multiview images displayed on the display panel and each region provides an image different from those in other regions. The crosstalk is based on the assumption that different view images are separately viewed for viewers in the proper positions of the viewing zone. However, many factors including imperfectness in system and component parameters and characteristics, misalignment viewers posture and so on, force the complete separation impossible. The crosstalk in the stereoscopic images is caused by a small intensity portion of an eye image added to the other eye image. This added portion will get into the other eye simultaneously, and makes the other eye image blurred and the depth sense reduced. In the stereoscopic imaging systems, two images in a stereoscopic image pair cannot be completely separated from each other due to the imperfect optics used to deliver each view image to its corresponding eye and the viewers postures do not allow the optics to work at their full capacities. Hence there is always a small portion of an eye image in the other eye image. This means that crosstalk exists always in the stereoscopic image. The issue is how to reduce the crosstalk to the level at which viewers do not aware of it by making the intensity of the corresponding eye image much higher than that of the other eye image. In the contact-type multiview 3-D displays, the crosstalk between different view images can also present because the different view images displayed on the panel appear only at the viewing regions along the viewing Three-Dimensional Imaging, Visualization, and Display 2015, edited by Bahram Javidi, Jung-Young Son, Proc. of SPIE Vol. 9495, 94950P 2015 SPIE CCC code: X/15/$18 doi: / Proc. of SPIE Vol P-1

2 zone cross section (VZCS) [3]. Since the numbers of different view images and the regions are the same, each view image appears at its corresponding viewing region. In the other than these viewing regions, the image at each viewing region is a mixture of the images from its surrounding viewing regions [4]. Since each viewing region which has a diamond shape is aligned along the VZCS by sharing its side tips with its neighbors, an intensity of a viewing region can smear into its neighbors, its neighbors neighbors and so on. Hence crosstalk is not between two closest neighbor viewing regions but all viewing regions. However, since the viewing regions are contacted only at their side tips, it is difficult to tell that the crosstalk is caused only by the interference between neighboring view images. Hence it is necessary to redefine the crosstalk in the contact-type multiview 3-D displays. For the redefining, it is necessary to understand the viewing zone structure of the displays because it has a unique structure which is different from other 3-D displays based on projections. The viewing zone is a light field formed by lights from each pixel in the display panel. These lights propagate along a specific direction defined by the elemental lenses in front of the panel and expand continuously. The expanding angle is defined by the focal length of the elemental lenses and the pixel size. The plate where the elemental lenses are inscribed is named as viewing zone forming optics (VZFO). In the displays, the propagation directions of the lights from pixels consisting of a view image are designed to converge to, i.e., crossed each other at a point on the parallel plane to the panel. Since the lights are expanding, the converged lights have a certain area. This area is geometrically determined by the relationship between the pixel cell and VZFO parameters, and works as a common viewing area of the pixels. The area is the viewing region of the view image represented by the pixels. The parallel plane is the only plane where all view images are separately appearing. After passing this plane, the light from each pixel is separately propagating. The plane is named as viewing zone cross section (VZCS). The distance from the panel to VZCS is typically defined as the viewing distance. The crosstalk can be induced because the lights from pixels which are consisting of a view image do not confined completely to the area in VZCS. Hence lights from different view images can be interfere each other to induce the crosstalk. The separately propagating lights will be mixed with lights from pixels of other view images. More different view images will be mixed as the distance from the VZCS increases. This mixing induces the image perceived by a viewer located at a place other than VZCS to be composed of image strips from different view images. The perceived image is a patched image of image pieces from different view images. The crosstalk will be meaningful if the viewing regions are confined along the VZCS. However, if these patched images can also give depth sense, the viewing zone of contact-type multiview 3-D images will be extended more from the VZCS and the crosstalk will lose its meaning in quantifying the quality of 3-D images. In fact, the crosstalk will have no meaning if a supermultiview condition in the 3-D images is met because at least two different view images will be projected to the pupil of viewer s each eye in the super-multiview image. In this paper, the viewing zone structure in contact-type multiview 3-D displays is analyzed to show the inappropriateness of using the crosstalk to quantify the quality of 3-D images. For this purpose, the compositions of images to be projected to viewer s eyes in the viewing zone are analyzed. Proc. of SPIE Vol P-2

3 2. Viewing Zones in a contact-type 3-D imaging system Since the viewing zone is a light field formed by lights from pixels in display panel, it will be easily specified in the optical geometry formed by a display panel and a VZFO [5]. The optical geometry of the contact-type multiview 3-D imaging systems has two different configurations: One is parallel and the other radial [6]. These configurations are induced by the difference in the dimensions of pixel cell (elemental image) in the panel and elemental lens in VZFO. When no difference, it is parallel and when the dimension of the elemental lens is smaller than the pixel cell, it is radial. The dimension difference between the elemental lens and the pixel cell is usually less than 1/100 of the dimension, but the resulted viewing zone shapes of the radial and parallel have a large difference. The viewing zone of the parallel is barely defined to VZCS in that of the radial. Figs. 1 and 2 show the viewing zone forming geometry of radial and parallel configurations, respectively. P C Display Panel Line/Point Image Viewing Zone (36 Regions) /2 6/3 4/2 2/1 1/1 3/3 4/4 6/6 1/2 4/5 2/4 1/4 2/5 3/6 1/5 2/6 1/6 Expanding Image Expanding Pixel Image Viewing Zone Cross Section VZFO Fig. 1. Viewing zone forming geometry of a radial-type multiview 3-D Display Proc. of SPIE Vol P-3

4 Line/Point Image Array Line/Point Image Display Panel Expanding Image Field of View 7/2 Viewing Region Viewing Zone 26 Sub-Regions Fig. 2. Viewing zone forming geometry of a parallel-type multiview 3-D Display These figures are drawn with the assumption that each pixel cell ( P C in Fig. 1) image is crossed at the center of the elemental lens assigned to it and expands with the same angle as the crossing angle to form the viewing zone. The line and point images in these figures represent the centers of their corresponding elemental lenses. The numbers of pixel cells and pixels in each pixel cell are given as 10 and 6 for Fig. 1, and 16 and 8 for Fig. 2, respectively. The number of pixels in each view represents the number of different view images and the number of pixel cells the number of pixels composing each view image perceived at the viewing zone [4]. Hence there are 6 different view images with 10 pixels for each view for Fig. 1 and 8 different view images with 16 pixels for each view for Fig. 2. The 6 diamond shape areas along VZCS are viewing regions for the view images specified by the number in each viewing region. Since the viewing zone is formed in the space where the ever expanding images of left and right most pixel cells are crossed and these images are equally expanded, it has a shape of a diamond for the radial because they are completely crossed but a circular cone for the parallel because they are partly crossed. VZCS is defined as the cross sections of two images when they are completely matched. Hence it becomes a parallel plane to the panel. In the parallel configuration, since the expanding image of each pixel cell is propagating in parallel, VZCS is theoretically at infinite distance from the panel, i.e., it doesn t exist. As shown in Figs 1 and 2, not even the viewing regions along VZCS, other part of the viewing zone is also divided into many diamond shape regions. Each of these regions is formed by crossing between the images of pixels in left and right most pixels cells. The boundary lines represent the boundary lines between pixels in each pixel cell. This contains several meanings: 1) The number of the divided regions in viewing zone 2 is represented by m, where m is the number of pixels in a pixel cell. This informs that the number of viewing regions in the parallel configuration is mm+ ( 1)/2. However, 2m 1viewing regions along VZCS will be partly appeared. Hence there are 36 viewing regions for Fig. 1 and 36 for Fig. 2. 2) These divided regions can be identified by the pixels forming them. For example, when the pixels in each pixel cell is Proc. of SPIE Vol P-4

5 numbered by 1 to m from right to left, the regions in Fig. 1 is identified as 1/2, 1/5, 1/6, 4/2, 2/5, 5/2 and so on, where the first and second numbers in each number set indicate the numbered pixels in left and right most pixel cells, respectively. And 3) the images viewed at these regions have the first and last pixels from first and second numbers, respectively, of number sets identifying their regions. This is obvious that the regions are defined by the two pixel images. Furthermore, all these regions are divided into sub-regions because they are also crossed partly or whole by pixel images from pixel cells other than the left and right most pixel cells 1. For the viewing regions along the VZCS, a numbered pixel from each pixel is completely crossed with the same numbered pixels from other pixel cells. Hence it is possible to find the image composition at each sub-region by identifying the pixels in each pixel cell involved in forming the sub-region. In Fig. 2, the magnified view of region 7/2 is shown and Fig. 3 shows the magnified view of the viewing zone in the radial. This figure clearly indicates that each viewing region is divided into many sub-regions. 5/2 4/2 R -1 R 0 R 1 R 2 1/1 2/2 3/3 4/4 5/5 6/6 VZCS Viewing zone R 3 R 4 1/5 2/5 R 5 1/6 Normal line of the display panel Fig. 3. Magnified view of the viewing zone in the radial-type in Fig Image compositions Fig. 4 shows the individual views of the sub-regions 1/5, 1/6, 4/2, 2/5, 5/2 specified by the number sets in Figs. 1 and 3. The region 7/2 in the parallel contains 26 sub-regions, regions 1/5 and 1/6 12, regions 4/2 17, and regions 2/5 and 5/2 13. Fig. 4 also shows that regions 1/5 and 1/6 have the same dividing structure, and that the regions 2/5 and 5/2 have the same structure. Proc. of SPIE Vol P-5

6 4/ /5 1 5/2 AND 2/ / Fig. 4. Sub-regions in several viewing regions Tables 1, 2 and 3 show the compositions of the images viewed at these sub-regions. Each of the compositions is consisted of 10 pixels, i.e., a pixel from each pixel cell. There are 10 pixel cells in the panel. In table 1, the compositions of region 4/2 consist of pixels from 3 different view images of 2, 3 and 4, since each pixel cell consists of the same numbered pixels from different view images and they are aligned in the pixel cell as the camera order in the 180 rotated multiview camera array for the different view images. Hence 2, 3 and 4 in the compositions represent pixels from view images 2, 3 and 4, respectively. In these compositions, the consecutively appearing same numbers represent an image strip from the numbered view image. This means that the images in these sub-regions are composed of 3 image pieces, 1 st piece from view image 4, 2 nd piece from view image 3 and the 3 rd piece from view image 2. For example, the image composition of sub-region 7, indicates that its 1 st, 2 nd and 3 rd pixels from left are coming from the same order pixels in view image 4, 4 th to 7 th pixels from the same order pixels in view image 3 and 8 th to 10 th pixels the same order pixels in view image 2. The relative positions of the image pieces in the compositions are the same as their positions in their original view images. In the region 4/2, the total number of numbers 2, 3 and 4 in 17 compositions is 42, 86 and 42, respectively. These numbers imply that region 4/2 is dominated by view image 3, i.e., the view image between 2 and 4. It is also seen in Fig. 3 that regions 3/1, 5/3 and 6/4 are also consisted of 17 viewing subregions. The image compositions in these regions are consisted of 1, 2 and 3 for region 3/1, 3, 4 and 5 for region 5/3 and 4, 5 and 6 for region 6/4. Hence the image compositions will be the same as the region 4/2 if the numbers 4, 3 and 2 are replaced by 3(5, 6), 2(4, 5) and 1(3, 4) for region 3/1(5/3, 6/4). The total number of numbers in these regions will be the same as the region 4/2. Table 2 is for the image compositions in viewing regions 5/2 and 2/5. The compositions for 2/5 are in the parenthesis. These compositions are composed of 4 image pieces each from view images 2, 3, 4 and 5. The relative positions of the image pieces in each composition are the same as their relative positions in view images 2, 3, 4 and 5. Table 2 shows that the number order of the compositions in region 5/2 and 2/5 are completely reversed at their corresponding sub-regions. Subregions 1, 7 and 13, the numbers in region 5/2 is reversed to those in region 2/5; for example Proc. of SPIE Vol P-6

7 ( (7), (13)) in 5/2 and ( (7), (13)) in 2/5 are completely reversed. The sub-regions 2 and 5, 3 and 6, 4 and 10, 8 and 11, and 9 and 12, the number orders are reversed to each other. This informs that if the image compositions of a region i( = 1 to m)/ j( = 1 to m) are known, those of the region j / i are found by reversing the number orders of the compositions. The total numbers of 5, 4, 3, and 2 in the compositions are 26, 39, 39 and 26, respectively. The numbers are more distributed than those in region 4/2. The image compositions in Table 3 are for regions 5/1 and 1/6. The compositions in parenthesis are for region 1/6. For region 5/1, the compositions consist of 5 different numbers, 1, 2, 3, 4 and 5, i.e., 5 different image pieces view images 1 to 5. For the regions 1/6, the compositions consist of 6 different numbers of 1, 2, 3, 4, 5, and 6. Each of 6 view images contributes an image piece to form the composition. The total numbers of numbers 1, 2, 3, 4 and 5 are 18, 28, 28, 28 and 18, respectively. For region 1/6, the numbers of number 1 to 6 are 19, 20, 21, 21, 20 and 19, respectively. These numbers indicate that as the viewing region involves with more view images, the dominance of an image in the compositions is diminished. In fact for region 5/1, view images 2, 3, and 4 contribute mostly but equally. For region 1/6, the contributions from 6 view images are almost the same. Hence tables 1, 2 and 3 will allows estimating the compositions of images for all viewing sub-regions, except viewing regions between the viewing regions along the VZCS. The image compositions in the viewing regions between the viewing regions along VZCS are divided into 9 viewing sub-regions as shown in Fig. 3. The number is 1 less than the number of pixel cells along the horizontal direction of the multiview display. The image compositions of the sub-regions in region 1/2 are , , , and from left to right sub-regions. The numbers of number 2 increase 1 at a time from right to left. For the region 2/1, the compositions have reversed number order of those in region 1/2. The total numbers of 1 and 2 in these compositions are 45 and 45. They are equal. So it is possible to consider that the viewing regions between the viewing regions along VZCS, such as 1/2(2/1), 2/3(3/2),, 5/6(6/5) are for images composed of two neighboring view images of 1 and 2(2 and 1), 2 and 3(3 and 2),, 5 and 6(6/5) in 0.5:0.5 ratio, i.e., first half of the view image specified by 1 st number in each number set are combined with 2 nd half of the view images specified by the 2 nd number in each number set. These composed images will not be too different from each view image, except some image distortions along the combined line. The distortions can be minimized by minimizing the disparity between different view images. Proc. of SPIE Vol P-7

8 No. PC1 PC2 PC3 PC4 PC5 PC6 PC7 PC8 PC9 PC Table 1. Image compositions in sub-regions in viewing region 4/2 No. PC1 PC2 PC3 PC4 PC5 PC6 PC7 PC8 PC9 PC10 1 5(2) 4(2) 4(2) 4(3) 3(3) 3(3) 3(4) 2(4) 2(4) 2(5) 2 5(2) 4(2) 4(2) 4(3) 3(3) 3(3) 3(4) 3(4) 2(5) 2(5) 3 5(2) 4(2) 4(2) 4(3) 4(3) 3(4) 3(4) 3(4) 2(5) 2(5) 4 5(2) 4(2) 4(2) 4(3) 4(3) 3(4) 3(4) 3(5) 3(5) 2(5) 5 5(2) 5(2) 4(3) 4(3) 3(3) 3(3) 3(4) 2(4) 2(4) 2(5) 6 5(2) 5(2) 4(3) 4(3) 4(3) 3(4) 3(4) 2(4) 2(4) 2(5) 7 5(2) 5(2) 4(3) 4(3) 4(3) 3(4) 3(4) 3(4) 2(5) 2(5) 8 5(2) 5(2) 4(3) 4(3) 4(3) 3(4) 3(4) 3(5) 3(5) 2(5) 9 5(2) 5(2) 4(3) 4(3) 4(4) 4(4) 3(4) 3(5) 3(5) 2(5) 10 5(2) 5(3) 5(3) 4(3) 4(3) 3(4) 3(4) 2(4) 2(4) 2(5) 11 5(2) 5(3) 5(3) 4(3) 4(3) 3(4) 3(4) 3(4) 2(5) 2(5) 12 5(2) 5(3) 5(3) 4(3) 4(4) 4(4) 3(4) 3(4) 2(5) 2(5) 13 5(2) 5(3) 5(3) 4(3) 4(4) 4(4) 3(4) 3(5) 3(5) 2(5) Table 2. Image compositions in sub-regions in viewing region 5/2(2/5) No. PC1 PC2 PC3 PC4 PC5 PC6 PC7 PC8 PC9 PC10 1 1(1) 1(1) 1(2) 2(2) 2(3) 3(3) 3(4) 4(4) 4(5) 5(6) 2 1(1) 1(1) 2(2) 2(2) 2(3) 3(3) 3(4) 4(5) 4(5) 5(6) 3 1(1) 1(1) 2(2) 2(2) 3(3) 3(4) 3(4) 4(5) 4(5) 5(6) 4 1(1) 1(1) 2(2) 2(3) 3(3) 3(4) 4(4) 4(5) 4(5) 5(6) 5 1(1) 2(1) 2(2) 2(2) 3(3) 3(4) 3(4) 4(5) 4(6) 5(6) 6 1(1) 2(1) 2(2) 2(3) 3(3) 3(4) 4(4) 4(5) 4(6) 5(6) 7 1(1) 2(1) 2(2) 3(3) 3(3) 3(4) 4(5) 4(5) 4(6) 5(6) 8 1(1) 2(2) 2(2) 2(3) 3(3) 3(4) 4(4) 4(5) 5(6) 5(6) 9 1(1) 2(2) 2(2) 3(3) 3(3) 3(4) 4(5) 4(5) 5(6) 5(6) 10 1(1) 2(2) 2(2) 3(3) 3(4) 4(4) 4(5) 4(5) 5(6) 5(6) 11 1(1) 2(2) 2(3) 2(3) 3(4) 3(4) 4(5) 4(5) 5(6) 5(6) 12 1(1) 1(2) 2(2) 2(3) 3(3) 3(4) 4(4) 4(5) 5(5) 5(6) Table 3. Image compositions in sub-regions in viewing region 1/5(1/6) The analyses so far inform that the viewing zone can be divided into viewing regions which show individual view images and patched images of two to m neighboring view images. This is shown in Fig. 3. The regions Proc. of SPIE Vol P-8

9 are specified by R 0 to R 6 ( R 6 ). In the lines between R 1 to R 1, the viewing regions for individual view images and two image patching regions are existing. These viewing regions can be hardly distinguished in real viewing situations because they are in side by side and a viewer s one eye can be in a patched image region but the other eye in an individual view image region. This is the reason why the crosstalk is not effective in the multiview 3-D images. Furthermore, when the size of the viewing regions are reduced to less than viewers pupil sizes by implementing many different view images, several viewing regions can be within the pupil as in the super multiview imaging as shown in Fig. 3. In this case, each viewing region will work as an image cell [7]. The different composition images in the sub-regions within the region can be hardly identified because they are too small, the first and last pixels of their images are the same for all the sub-pixel images and all these images have the same number combinations though the number ratio is different for different sub-regions. Hence the composite images in this region will be mixed together and consequently the representing image of the region will be defined by the total numbers of the numbers composing all the images of the sub-regions as mentioned in regarding region 4/2. The number ratio of 2 : 3 : 4 in this region is 42 : 88 : 42. Number 3, i,e, view image number 3 will dominate the region. 4. A new parameter quantify the image quality in the contact-type multiview 3-D images It is good to have a parameter to quantify the images quality in 3-D images. As mentioned before, the crosstalk is not appropriate to represent the quality in the multiview 3-D images. There was an attempt to quantify the image quality by taking inverse of the number of different view images involved in the patched images [8]. This inverse value revels that the quality is reducing as the involved different view images are increase but it does not reflect the dominant image as in the region 4/2(2/4). To account the image, it will be possible to use the total number of pixels for each composing view images. For example, as shown in region 4/2(2/4), the number ratio of 2 : 3 : 4 is 42 : 88 : 42. This number ratio indicates that the quality of the dominant image is 88/(84+88)= of the view image number 3. For the case of region 5/2(2/5), the pixel number ratio of view images 5 : 4 : 3 : 2 = 26 : 39 : 39 : 26. The dominant images in this combination has the quality value 39/130 = 0.3. For the regions along the VZCS, the viewing regions for individual view images will have the quality value 1, and the regions for two patches images 0.5. In this representation, region 4/2(2/4) has better quality than two patched image viewing regions. Conclusions: Crosstalk is inappropriate parameter of quantifying quality of multiview 3-D images. Patched images of different view images appearing at the viewing zone of the images cannot represent their image qualities by the crosstalk. Since the viewing zone will be divided into smaller regions as the number of different view images increases and/or the pixel size becomes smaller, the regions can be small enough to be smaller than the pupil size. This means that there are no boundaries between different view images. This situation is common for a super-multiview condition. This is another reason of making the crosstalk ineffective in quantifying the quality. It is necessary to find a new parameter for quantifying the quality. Proc. of SPIE Vol P-9

10 Acknowledgements This work was supported by GigaKOREA project, [GK14D0100, Development of Telecommunications Terminal with Digital Holographic Table-top Display], and GK14C0100, Development of Interactive and Realistic Massive Giga-Content Technology] References 1. Lenny Lipton, Foundations of the STEREOSCOPIC CINEMA, A Study in Depth, Van Nostrand Reinhold Company, New York, Jung-Young Son and Bahram Javidi, 3-Dimensional Imaging Systems Based on Multiview Images, IEEE/OSA J. of Display Technology, V1(1), pp , Jung-Young Son, Byung-Gyu Chae, Wook-Ho Son, Jeho Nam and Beom-Ryeol Lee, Comparisons of Viewing Zone Characteristics of MV and IP, IEEE/OSA JDT, V8(8), pp , Beom-Ryeol Lee, Jung-Young Son, Characteristics of composite images in MV and IP, Applied Optics, V51(21), pp , Chun-Hea Lee, Jung-Young Son, Sung-Kyu Kim and Min-Chul Park, Visualization of Viewing Zones formed in a contact type multiview 3-D imaging system, IEEE/OSA JDT, V8(9), pp , Jung-Young Son, Wook-Ho Son, Sung-Kyu Kim, Kwang-Hoon Lee and Bahram Javidi, 3-D imaging for creating real world like environments, Proceedings of THE IEEE (Invited), V101(1), pp , Wook-Ho Son, Jinwoong Kim, Jung-Young Son, Beom-Ryol Lee and Min-Chul Park, The basic image cell in contact-type multiview 3-D Imaging systems, Optical Engineering, 52(10), pp ~ , Oct 10, V. V. Saveljev, Jung-Young Son and Kyung-Hoon Cha, Estimation of Image Quality in Autostereoscopic Display, SPIE V5908, pp (-14), 2005(2005, Annual Meeting) Proc. of SPIE Vol P-10