Stereo-Foveation for Anaglyph Imaging

Transcription

1 Stereo-Foveation for Anaglyph Imaging Arzu Coltekin (Çöltekin) Helsinki University of Technology, Institute of Photogrammetry and Remote Sensing, FIN Espoo, Finland hut.fi ABSTRACT For 1:1 displays and network visualization in stereo imaging, we suggest that foveation is a feasible and efficient compression method which also gives a good basis for Level of Detail (LOD) control. The demand for highest resolution images in the area(s) of interest is a generally wanted feature, but it is particularly important for photogrammetric 3D modelling where the precision might be highly required depending on the project. Particularly for 1:1 stereo-viewing in large screen displays such as panoramic screens or caves, the actual area of interest is much smaller than the whole screen. Instead of loading the whole image pair, we would foveate both images, and project them after the stereo-foveation. This principally gives us the best possible resolution in the area of interest and still a very good overview of the neighbouring areas to navigate and locate other areas of interest as the human eyes do throughout the non-uniform 3D image. We test the idea with an anaglyph pair, and create a hybrid model by combining algorithms that deal with anaglyph imaging; disparity maps, foveation and we create a LOD function for the resolution control along the z axis. Keywords: Foveation, space variant imaging, stereo imaging, anaglyph, visualization, level of detail management, photogrammetry. 1. INTRODUCTION Foveation is a widely used biologically motivated compression technique. It is often utilized for two dimensional images and videos, particularly when transmitting over a network in real time. It also attracts interest from camera research and robotics fields, where often the third dimension also matters. This method relies on the ways the human visual system (HSV) works. In a human visual system, the eyes and the brain process the visual input in a space-variant nature. When the object of interest is located, the eyes accommodate on that object. Once this happens, fovea keeps the object in focus and reconstructs the rest of the scene gradually more blurry towards the edges in all dimensions. When we say all dimensions, we mean also the depth. In other words, (most) people foveate in stereo. Photogrammetry works with stereo vision to recover depth from images. It deals with 3D modeling and measurements. For most of its tasks, high resolution images are the main input into a photogrammetric system. Videos too, can be utilized. The term videogrammetry refers to this kind of videographic photogrammetry. Even though photogrammetry traditionally has been aerial and recently a lot space-born (remote sensing), and mainly to produce maps; the close-range photogrammetry (also referred as non-topographic photogrammetry) has always been there as well. This branch of the field works with terrestrial images. The scale changes from city and building models down to microscopic imaging. In any case, the field has extensive interest in image geometry (less in radiometry though not excluded) and the accuracy of measurements of the 3D coordinates, based on the project requirements. All of the above said, photogrammetry and foveation did not seem to have met yet. Very little literature takes a closer look at the needs of this field and how stereo-foveation would fit in. There is an obvious need to manage the big images and the level of detail in the scene being inspected. Foveation lets humans successfully navigate with the peripheral

2 vision while providing the area of interest in full resolution, so it does on a display, which should be helpful also in photogrammetric tasks. This is particularly true in 1:1 visualizations like the one in Figure 1. Figure 1: An illustration of the display used for full-scale stereo visualizations in Helsinki Technical University s Institute of Photogrammetry and Remote Sensing. It is called The Stereodrome by the institute members. (Illustration by Henrik Haggrén) In this type of displays, panorama rooms and caves where the user is spatially immersed into the system, a big part of the scene is anyway lost because of natural foveation. After all one should take into account that; a stereoscopic display is an optical system whose final component is the human mind (Lipton, 1997). Rendering the naturally foveated parts in full resolution would be a clear waste of resources. In this paper we present a conceptual stereo-foveation model to provide the usual benefits of binocular foveation to photogrammetry and an implementation of it for anaglyph imaging. Anaglyph imaging was chosen for its simplicity, availability, low cost and platform independence. 2. BACKGROUND If we can understand how the perception works, our knowledge can be transferred into rules for displaying information. Following perception based rules, we can present our data in such a way that the important and informative patterns stand out (Ware, 2000). The points in the statement above are well taken by the researchers in several fields. Percepted world is being recorded, modelled and reconstructed based on our perceptions and beyond. The following terms that meet us in the literature for human vision based level of detail (LOD) management may be an indicator (Çöltekin, 2004): - Gaze directed - Gaze Contingent - Perceptually Driven - Eccentricity - Foveated All of these terms refer to a way to capture and simulate the human visual system and are widely met in literature. The research partly presented in this paper also gets its inspiration from human visual system.

3 2D foveation 2D foveation is also called Eccentritcy LOD and takes the visual acuity into account in lateral plane. The caption for figure 2 briefly explains while Figure 2 illustrates the concept. Figure 2: A rough illustration to show the concept of 2D foveation. The image is segmented in several layers and an image pyramid created, the resolution gradually decreases towards the periphery and then image is re-composed into one space variant result. Stereo-foveation (also referred as binocular foveation or 3D foveation) In virtual reality literature, the exact same concept; the imitation of human visual system taking stereo acuity into account is called depth of focus simulation. The robotics literature seems to use binocular foveation more often. There is a very large amount of literature on the topic that we will not cover here, but mention the most popular human visual system inspired theory. The stereo depth perception is said to be possible within the Panum s Fusional area, where the received input from each eye overlaps. Panum's Fusional Area Right eye Left eye α Screen β disparity = α - β Figure 3: Ware, C., Chapter 8 of Information Visualization: Perception for Design. 2000, San Fancisco: Morgan Kaufmann. There are a few different approaches to recover the depth that falls into this area, but the space limitation in this paper restricts us from introducing each of them. Please refer to Ware 2000, Luebke et al and Oshima et al for more on these methods.

4 3. IMPLEMENTATION The conceptual model of the implementation aims at a real time 3D foveation. That means the presence of a precise binocular eye tracking system at hand. In the current conditions we do not have the opportunity to test this with an eye tracking system due to hardware limitations. Therefore the pilot implementation takes a stereo pair as input, foveates the anaglyph image based on user interaction. With a precise binocular eye tracker, the system would exactly know where the user is looking at and that would be considered as the area of interest. Although, unfortunately in present-day computer graphics systems, particularly those that allow for real-time interaction, depth of focus is never simulated (Ware 2000). We try to animate how it would be like if there was a functioning eye tracker. In this implementation, the area of interest is selected interactively by the user by specifying a point by its image coordinates. This point is accepted as the center of foveation and the image is segmented using this as one of the parameters. After that a level of detail aware compression in all x,y,z directions is applied (stereo-foveation). Foveaglyph Since it is a foveation application for anaglyph imaging, it is named as foveaglyph. The program operates in the following order: 1) Camera is calibrated (pre-calculation). 2) The input images are corrected; lens distortion and affinity removed (pre-calculation). 3) Disparity map is calculated (pre-calculation). 4) Based on normal case of stereo geometry, the z values for each pixel are calculated. 5) Anaglyph image is created and foveation applied based on a simple geometric model. Even though the 3D foveation is applied directly to the anaglyph result, program allows the user to foveate a single image in 2D. Algorithms Foveaglyph uses original algorithms for making the anaglyph, 2D and 3D foveations, but uses Depth Discontinuities by Pixel-to-Pixel Stereo (p2p) by Stan Birchfield and Carlo Tomasi for image matching and generating the disparity map. Before the stereo pair is put to foveation process, the camera is calibrated; the lens distortions and affinity are corrected using Petteri Pöntinen s software which was developed for Helsinki Technical University s Institute of Photogrammetry and Remote Sensing. Image Matching and Disparity Map Calculation: Depth Discontinuities by Pixel-to-Pixel Stereo Shortly referred as p2p, this is an algorithm and a piece of code the authors Stan Birchfield and Carlo Tomasi distribute on a web site (Birchfield et. al ). The algorithm explained in Stanford University Technical Report STAN-CS-TR , July 1996 (Birchfield et.al.1996) In brief, as stated Birchfield et.al.1996 the features of the algorithm can be described as follows: Part I. Our stereo algorithm explicitly matches the pixels in the two images, leaving occluded pixels unpaired. Matching is based upon intensity alone without utilizing windows. Since the algorithm prefers piecewise constant disparity maps, it sacrifices depth accuracy for the sake of crisp boundaries, leading to precise localization of the depth discontinuities. Three features of the algorithm are worth noting: (1) unlike most stereo algorithms, it does not require texture throughout the images, making it useful in unmodified indoor settings, (2)

5 it uses a measure of pixel dissimilarity that is provably insensitive to sampling, and (3) it prunes bad nodes during the search, resulting in a running time that is faster than that f standard dynamic programming. Part II. After the scan lines are processed independently, the disparity map is post processed, leading to more accurate disparities and depth discontinuities. Both the algorithm and the postprocessor are fast, producing a dense disparity map in about 1.5 microseconds per pixel per disparity on a workstation. Calculating the Z values Normal Case of Stereo Once we have the disparity map, we have the parallax values for each pixel. Using this information, it is possible to recover the Z values. The geometry is simpler in so-called normal case of stereography. This case sometimes is referred as the ideal case. The normal case of geometry should be met while taking the pictures as in Figure 4: the optical axes of the two cameras are parallel, their image planes are coincident, and there is no vertical parallax. Then several similar triangles can be formed and the following formula can be used to calculate the Z value: Z = B c p x Where; B: Base distance between the two cameras c: Camera constant (focal length after calibration), P x : Parallax (See the appendix for the use of terms parallax and disparity) Figure 4: A bird s view graphic of the normal case of stereography. In this setting the optical axes of the two cameras are parallel and their image plane is coincident. The geometry is simple and suits well to anaglyph imaging, even though it is restricted to a careful image acquisition setting and not likely to be achieved by free hand. Forming the Image Pyramid A simple geometric model is used to calculate the image pyramid. This is being developed. The two most remote points are searched (also taking the max and min disparities for max depth) and the distance between the two is set as the maximum distance. D =

6 D max = B B 2 (( ). xb ( ). xa)) (Point of interest is a, each pixel visited is b.) Px a ) B B 2 + (( ). yb ( ). ya)) b) a ) B. c B. c 2 + (( ) ( )) b) a ) Once the maximum distance is known, the image is segmented to levels taking this and the user specified level of detail into account. Image pyramid is formed like in mip-mapping; first derived image is half the size of the original and the next one is the half of the number 2. Then, every pixel is visited and a decision is made from which element of the pyramid it should take its resolution. This can basically be expressed as: l = d. lmax / D max Where d is the distance between the PoI and the pixel to be determined, l is the the pixel s LOD, l max is maximum number of levels that is possible, D max is the maximum distance. A more detailed description of this algorithm and the developments will follow in future publications. A Nikon D100 is used to capture the images which have the following camera parameters when the focus is set to infinity, and the aperture (f stop) is set to 5.6 after calibration: Calibrated by Petteri Pöntinen and has the following intrinsic parameters in sub pixels:. Camera constant: Principle point: Here are some graphic results from foveaglyph: Figure 5: The image pyramid of 5 levels.

7 Approximate Point of interest 4. FUTURE WORK Figure 6: Stereo-foveated image. We are working on a more perceptually driven geometrical model for our LOD function for stereo foveation. The next step in this work will be to compare the performance of our method to its alternative approaches. Another future work will be to study the cognitive effects, by a user survey. Intention is to determine if the biological foveation done by the human eyes and the interference of the artificial stereo fovetion to the natural one creates a disturbance or strain in the viewer or not. 5. ACKNOWLEDGEMENTS The programming of foveaglyph would not be possible without Çağrı Çöltekin of RIPE NCC. I also owe thanks to Petteri Pöntinen for letting me use his calibration software and giving valuable practical advice. Last but not least, Henrik Haggrén has given inspiration and been in the center of discussion about the development of the ideas discussed in this paper.

8 APPENDIX On the use of the terms disparity and parallax: We often meet these two words used interchangeably. Lipton 1997 suggests that disparity is retinal (biological) and parallax is what happens on the screen. The rest of the people in the filed do not seem to differentiate. In the photogrammetric jargon, the word parallax is more commonly used and might refer to either retinal shift or the shift after the projection on image plane. Here is what Lipton says about the issue: Parallax and disparity are similar entities. Parallax is measured at the display screen, and disparity is measured at the retinae. When wearing our eyewear, parallax becomes retinal disparity. It is parallax which produces retinal disparity, and disparity in turn produces stereopsis. Parallax may also be given in terms of angular measure, which relates it to disparity by taking into account the viewer s distance from the display screen. (Lipton 1997). REFERENCES Birchfiled et al. Depth Discontinuities by Pixel-to-Pixel Stereo : Birchfiled et al. Stanford University Technical Report STAN-CS-TR , July 1996 Coltekin, A., Foveation Support and Current Photogrammetric Software, ISPRS Congress Proceedings, 2004, Istanbul Lipton, Lenny Stereographics Handbook, 1997 Luebke, D., Reddy, M., Cohen, J.D., Varshney A., Watson, B., Huebner, R., Chapter 8 of Level of Detail for 3D Graphics. Textbook. Morgan Kaufmann, series in Computer Graphics and Geometric Modeling. ISBN URL: Oshima, T., Yamamoto, H., Tamura, H. Gaze-Directed Adaptive Rendering for Interacting with Virtual Space Proceedings of VRAIS 96, 1996 Ware, C., Chapter 8 of Information Visualization: Perception for Design. 2000, San Fancisco: Morgan Kaufmann.