Implementation of a panoramic-based walkthrough system

Transcription

1 Implementation of a panoramic-based walkthrough system Abstract A key component in most virtual reality systems is the ability to perform a walkthrough of a virtual environment from different viewing positions and orientations. Recent advances in image-based rendering techniques allow us to have interactive panoramic viewing of real world environment using real photos. With panorama visualization tools like QuickTime VR, we are able to have fast navigation in an image-based environment. However, the navigation control is restricted to panning, zooming and tilting of a single panoramic view. Smooth walkthrough between panoramic nodes is not yet supported. This paper describes implementation of a panoramic-based walkthrough system using real photos. Keywords: virtual reality, walkthrough, view interpolation, environment maps, panoramic images, panorama walkthrough, triangulation, triangle-based image warping, image-based rendering, plenoptic modeling, epipolar geometry. 1. INTRODUCTION Objectives of VR system are as follows [22]: First, the system should playback at interactive speed on most personal computers available today without hardware acceleration. Second, the system should accommodate both real and synthetic scenes. Real-world scenes contain enormously rich details often difficult to model and render with a computer. Third, the system should be able to display high quality images independent of scene complexity. Many virtual reality systems often compromise by displaying low quality images and/or simplified environments in order to meet the real-time display constraint. We wanted our system's display speed to be independent of the rendering quality and scene complexity. Traditionally, virtual reality systems use 3D computer graphics to model and render virtual environments in real-time. This approach usually requires laborious modeling and expensive special purpose rendering hardware. The rendering quality and scene complexity are often limited because of the real-time constraint. Walkthrough requires the synthesis of the virtual environment and the simulation of a virtual camera moving in the environment with up to six degrees of freedom. Systemization and navigation are usually accomplished with one of the following two methods [ 21]: 1.1 3D Modeling and Rendering A virtual environment is synthesized as a collection of 3D geometrical entities. The geometrical entities are rendered in real-time, often with the help of special purpose 3D rendering engines, to provide an interactive walkthrough experience. The 3D modeling and rendering approach has three main problems. First, creating the geometrical entities is a laborious manual process. Second, because the walkthrough needs to be performed in real-time, the rendering engine usually places a limit on scene complexity and rendering quality. Third, the need for a special purpose rendering engine has limited the availability of virtual reality for most people since the necessary hardware is not widely available. Despite the rapid advance of computer graphics software and hardware in the past, most virtual reality systems still face the above problems. The 3D modeling process will continue to be a very human-intensive operation in the near future. The real-time rendering problem [7, 8] will remain since

2 there is really no upper bound on rendering quality or scene complexity. Special-purpose 3D rendering accelerators are still not ubiquitous and are by no means standard equipment among personal computer users. 1.2 Branching Movies Another approach to synthesize and navigate in virtual environments is branching movies. Multiple movie segments depicting spatial navigation paths are connected together at selected branch points. The user is allowed to move on to a different path only at these branching points. This approach usually uses photography or computer rendering to create the movies. A computer-driven analog or digital video player is used for interactive playback. An obvious problem with the branching movie approach is its limited navigability and interaction. It also requires a large amount of storage space for all the possible movies. However, this method solves the problems mentioned in the 3D approach. The movie approach does not require 3D modeling and rendering for existing scenes; it can use photographs or movies instead. Even for computer synthesized scenes, the movie-based approach decouples rendering from interactive playback. The movie-based approach allows rendering to be performed at the highest quality with the greatest complexity without affecting the playback performance. It can also use inexpensive and common video devices for playback. 1.3 Image-based Systems In an image-based system for virtual environment navigation, the system uses real-time image processing to generate 3D perspective viewing effects. The approach presented is similar to the movie-based approach and shares the same advantages. It differs in that the movies are replaced with orientation-independent images and the movie player is replaced with a real-time image processor. The images currently use are cylindrical panoramas. The panoramas are orientation-independent because each of the images contains all the information needed to look around in 360 degrees. A number of these images can be connected to form a walkthrough sequence. The use of orientation-independent images allows a greater degree of freedom in interactive viewing and navigation. These images are also more concise and easier to create than movies. 1.4 QuickTime VR The image-based approach has been implemented in a commercial product called QuickTime VR [21]. The current implementation includes continuous camera panning and zooming, jumping to selected points and object rotation using frame indexing. Currently, QuickTime VR uses cylindrical environment maps or panoramic images to accomplish camera rotation. [21] QuickTime VR presents an approach which uses 360-degree cylindrical panoramic images to compose a virtual environment. The panoramic image is digitally warped on-the-fly to simulate camera panning and zooming. The panoramic images can be created with computer rendering, specialized panoramic cameras or by "stitching" together overlapping photographs taken with a regular camera. Walking in a space is currently accomplished by "hopping" to different panoramic points. QuickTime VR includes an interactive environment which uses a software-based real-time image processing engine for navigating in space and an authoring environment for creating VR movies. The interactive environment comprises two types of players. The panoramic movie player allows the user to pan, zoom and navigate in a scene. It also includes a hot spot picking capability. Hot spots are regions in an image that allow for user interaction. The object movie player allows the user to rotate an

3 object or view the object from different viewing directions. The panoramic authoring environment consists of a suite of tools to perform panoramic image stitching, hot spot marking, linking, dicing and compression. The object movies are created with a motion-controllable camera. 2. RELATED WORK As there is only one center of projection associated with a panoramic image, the warped view is correct only if the viewpoint is fixed there. In order to relax this constraint, Hirose et al. [1, 2] suggested to use morphing to generate intermediate views. On the other hand, by assuming depth information is available, Darsa et al. [3] proposed to apply mesh triangulation and morphing to generate warped views. However, recovering the dense depth information for real photos is still an open problem in the field of computer vision, this approach is not practical for real-world photos at present. Meanwhile, Faugeras and Robert [4,5,6] formulated the epipolar geometry between images and this confines correspondence between images to be on the locus of epipolar lines. Based on epipolar geometry, Bishop and McMillan [7] proposed a warping algorithm. This warping technique is further used in several later image-based rendering systems [8,9,10]. Nevertheless, these approaches need dense depth or correspondence information to figure out how to carry out warping during walkthrough and this is tedious and sometimes impractical for real-world photo. To solve this problem, use a triangle-based image warping algorithm [13, 14] extended from McMillan s warping algorithm [7, 11,12]. It can be regarded as an extension of the original pixel-based warping algorithm to triangles. An image-based walkthrough system can only requires only sparse correspondence information for warping realworld photos. No depth information is needed. Basically, it takes advantage of epipolar geometry between neighboring panoramic nodes. By combining various computer vision techniques [6, 15, 16] in the interactive matcher program, we are able to recover corresponding lines and patches between neighboring panoramic nodes semi-automatically. Triangulation is applied to the patches defined in each panoramic image to generate a resultant triangular mesh. After that, by pre-computing the drawing order [13, 14] of triangles, correct occlusion between objects can be ensured. It is worth to note that it warps triangles instead of pixels. With this approach, dense correspondence information between panoramas is not required. Moreover, there is no gap problem in previous pixel-by-pixel warping approaches. We can also utilize existing graphics hardware to further accelerate the rendering. Furthermore, as the number of triangles to be warped is far smaller than the number of pixels in a panoramic image, rendering performance is significantly improved. In this way, it is able to provide real-time walkthrough between panoramic nodes representing the real-world environment. 3. IMPLMENTATION The following discusses some implementation issues of this system. 3.1 Scene Capture First, take photos (panoramic images) of an environment (such as the university). When taking photos for a certain panoramic node, the actual location is recorded. The actual path is depicted in Figure 1 and the number below each node represents the 2D coordinate on the map.

4 epipole along the path of the 5 nodes node 4 node 3 node 2 node 1 node 0 (x4,y4) (x3,y3) (x2,y2) (x1,y1) (x0,y0) Figure 1 the walkthrough path consists of five panoramic nodes At each panoramic node, take twelve perspective photos of the environment with a camera. The tripod is specially designed so that the optical center of the camera remains fixed at the center of tripod even when the viewing direction of the camera changes. 3.2 Stitching Then, these twelve photos are stitched together to create a panoramic image by QuickTime VR Authoring [21]. It is created in five steps. First, nodes are selected in a space to generate panoramas. Second, the panoramas are created with computer rendering, panoramic photography or stitching a mosaic of overlapping photographs. Third, if there are any hot spots on the panorama, a hot spot image is constructed by marking regions of the panorama with pseudo colors corresponding to the hot spot identifiers. Alternatively, the hot spots can be generated with computer rendering [17], [18]. Fourth, if more than one panoramic node is needed, the panoramas are linked together by manually registering their viewing directions. Finally, the panoramic images and the hot spot images are diced and compressed to create a panoramic movie. The authoring process is illustrated in figure 2.

5 Node selection Still camera Renderer Panoramic Camera Stitch Mark hot sopts Link Dice&Compress QuickTime VR Movies Figure 2 the panoramic movie authoring process 3.3 Determining Correspondence After acquiring the panoramic images, apply the epipolar geometry [4, 5, 6] between cylindrical projection manifolds to find point correspondences between real-world panoramic photos, so as to match the geometric features of corresponding objects. It is implemented by patch construction: Specify an object or region of an object by enclosing a region in polygon, call polygon patch. Each patch is defined by a sequence of line segments. The basic goal is to group objects with similar distances from the camera so that they will move in similar speed when the user navigates through the environment. The correspondence can be defined in a two-way fashion. That is, the user can mark an object in panorama A and then adjust the generated patch in panorama B or vice versa. 3.4 Triangulation Once finishes specification, triangulation and subdivision are applied on these patches to obtain a finer mesh. For the case of synthetic images, the depth information of each pixel is available for the accurate calculation of image correspondence of every image pixel. Hence, we are able to accurately warp the image in a pixel-by-pixel manner [7, 8] (pixel-based image warping). On the other hand, it is

6 significantly different for the case of real-world images. Even with the state-of-the-art computer vision technologies, it is hard to recover the correspondence of every pair of pixels accurately and automatically. The resultant correspondence map is usually noisy. Hence warping image in a pixel-by-pixel manner is not satisfactory. As dense pixel correspondence information is not very reliable, so another warping method, namely triangle-based image warping [13, 14] is used, to warp a group of pixels (triangle) instead of a single pixel. The triangle-based image warping algorithm [13, 14] provides a drawing order of triangles on the mesh such that the visibility problem in the warped result is correctly resolved. With this rendering technique, we are able to warp those triangles on the panoramic surface correctly to create another panoramic image corresponding to another viewing position. The details of the algorithm can be found in [14]. The advantages of warping triangle are as follows. Firstly, warping triangle does not require dense pixel correspondence. Sparse correspondence information is usually sufficient. Secondly, when we employ triangles as the warping entities, gaps that resulted from pixel-based image warping can be avoided automatically. Thirdly, the number of triangles is usually smaller than that of pixels. Consider a panorama with the resolution of 1024*256, there are altogether 262,144 pixels while the number of triangles can be only a few thousands. Hence, the computational cost can be greatly reduced. Lastly, warping triangles allows us to utilize existing graphics hardware to further accelerate the rendering. If warp patches directly to generate the intermediate views, serious distortion of images may occur if patches are too large. Moreover, warping on arbitrarily shaped patches is not efficient. Furthermore, the visibility in the warped image is not guaranteed to be correctly resolved. Instead of warping patches in arbitrary shape, first triangulate the panoramic images. The defined patches provide a good guidance for image segmentation. The generated triangles are not allowed to cross the boundary of any patch. Sometimes triangles generated are still too large that distortion during warping cylindrical panorama may still be significant. Those large triangles can be further subdivided into smaller ones. It is worth to note that during triangle subdivision, extra vertices may be introduced to the mesh. These vertices also require correspondence information to be attached to them. Thus, to find correspondence on the neighboring panorama, bilinear interpolation is employed. Therefore, the subdivision process can be fully automatic. 4. CONCLUSIONS AND FUTURE DIRECTIONS In conclusion, the implementation of a system that allows the user to walkthrough an environment is described above. The environment is modeled by a set of real photos. After pre-processing the data with the interactive correspondence matching, the correspondences between two panoramic images are determined. By warping triangles, we are able to synthesize intermediate views of the same environment at different viewpoints. However, the current player does not require any additional input and output devices other than those commonly available on personal computers. Input devices with more than two degrees of freedom may be useful since the navigation is more than two-dimensional. Similarly, impressive stereo displays combined with 3D sounds may enhance the experience of navigation. One of the ultimate goals of virtual reality will be achieved when one cannot discern what is real from what is virtual. With the ability to use photographs of real scenes for virtual navigation, there may be one step closer.

7 REFERENCES [1] Michitaka Hirose. Image-based virtual world generation. In IEEE Multimedia, 04(01), pages 27 33, [2] Michitaka Hirose, S. Watanabe, and Takaaki Endo. Generation of wide-range virtual spaces using photographic images. In proceedings of VRAIS 98, pages , March [3] Lucia Darsa, Bruno Costa Silva, and Amitabh Varshney. Navigating static environments using imagespace simplification and morphing. In Proceedings of the 1997 Symposium on Interactive 3D Graphics, pages 25 34, April [4] Olivier Faugeras. Three-dimensional Computer Vision: A Geometric Viewpoint. MIT Press, Cambridge, Massachusetts, [5] Olivier Faugeras and Luc Robert. What can two images tell us about a third one? Technical report, INRIA, July [6] Stephane Laveau and Olivier Faugeras. 3-D scene representation as a collection of images. In Proceedings of the Twelfth International Conference on Pattern Recognition (ICPR 94), pages , Jerusalem, Israel, October [7] Leonard McMillan and Gary Bishop. Plenoptic modeling: An image-based rendering system. In Computer Graphics Proceedings, Annual Conference Series, SIGGRAPH 95, pages 39 46, August [8] William R. Mark, Leonard McMillan, and Gary Bishop. Post-rendering 3d warping. In Proceedings of the 1997 Symposium on Interactive 3D Graphics, pages 7 16, April [9] Matthew M. Rafferty, Daniel G. Aliaga, Voicu Popescu, and Anselmo A. Lastra. Images for accelerating architectural walkthroughs. IEEE ComputerGraphics & Applications, pages 38 45, November/December [10] Jonathan Shade, Steven Gortler, Li wei He, and Richard Szeliski. Layered depth images. In Computer Graphics Proceedings, Annual Conference Series (Proc. SIGGRAPH 98), pages , July [11] Leonard McMillan. Computing visbility without depth. Technical report, University of North Carolina, October UNC Computer Science Technical Report TR [12] Leonard McMillan. An Image-Based Approach to Three-Dimensional Computer Graphics. PhD thesis, Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, [13] Chi-Wing Fu, Tien-TsinWong, and Pheng-Ann Heng. Computing visibility for triangulated panorama. In in Proceedings of the 10th Eurographics Rendering Workshop, pages , Granada, Spain, June Eurographics. [14] Chi-Wing Fu, Tien-Tsin Wong, and Pheng-Ann Heng. Triangle-based view interpolation without depth buffering. accepted for publication in ACM Journal of Graphics Tools, [15] K. Prazdny. On the information in optical flows. Computer Vision, Graphics and Image Processing, 22(9): , [16] Gang Xu and Zhengyou Zhang. Epipolar Geometry in Stereo, Motion and Object Recognition - A Unified Approach, volume 6. Kluwer Academic, [17] Weghorst, H., G. Hooper and D. Greenberg. Improved Computational Methods for Ray Tracing. ACM Transactions on Graphics. 3(1): [18] Miller, G., E. Hoffert, S. E. Chen, E. Patterson, D.Blackketter, S. Rubin, S. A. Applin, D. Yim, J.

8 Hanan. The Virtual Museum: Interactive 3D Navigation of a Multimedia Database. The Journal of Visualization and Computer Animation, (3): , [19] Jonathan Shade, Steven Gortler, Li wei He, and Richard Szeliski. Layered depth images. In Computer Graphics Proceedings, Annual Conference Series(Proc. SIGGRAPH 98), pages , July [20] Chen, S. E. and G. S. P. Miller. Cylindrical to planar image mapping using scanline coherence. United States Patent number 5,396,583. Mar. 7, [21] Shenchang Eric Chen. QuickTime VR An Image-Based Approach to Virtual Environment Navigation, Computer Graphics, 1995 [22] Martin Goebel, Michitaka Hirose, Lawrence Rosenblum. Today's VR. IEEE Computer Graphics and Applications, page 22-24, November/December 2001 (Vol. 21, No. 6).