Free Viewpoint Video Synthesis and Presentation of Sporting Events for Mixed Reality Entertainment

Transcription

1 Free Viewpoint Video Synthesis and Presentation of Sporting Events for Mixed Reality Entertainment Naho Inamoto Hideo Saito Department of Information and Computer Science, Keio University {nahotty, ABSTRACT This paper presents a new framework for arbitrary view synthesis and presentation of sporting events for mixed reality entertainment. In accordance with the viewpoint position of an observer, virtual view image of sporting scene is generated by view interpolation among multiple videos captured at real stadium. Then the synthesized sporting scene is overlaid onto a desktop stadium model in the real world via HMD. This makes it possible to watch the event in front of the observer. Projective geometry between cameras is used for virtual view generation of the dynamic scene and geometric registration between the real world and the virtual view image of sporting scene. The proposed method does not need to calibrate multiple video cameras for capturing the event and the HMD camera. Therefore it can be applied even to dynamic events in a large space and enables observation with immersive impression. The proposed approach leads to make a new type of mixed reality entertainment for sporting events. 1. INTRODUCTION Sporting events are the most popular form of remote live entertainment in the world, attracting millions of viewers on television. Recently computer-generated visualization is increasingly used in sports broadcasting to enhance the viewer experience. One example form is that virtual objects, such as virtual offside lines in soccer scene and virtual lines indicating world records in swimming, are overlaid onto the live video [8]. Another example is virtual replay which allows observation from any viewpoint. This includes reconstruction of sports match using CG animation [8, 9] and arbitrary view generation from multiple camera images using computer vision technology [22, 29, 17]. With the ongoing convergence of television and Internet broadcasting, interactive visualization is becoming more important. However, these visualization techniques, typically using standard television screen or computer screens, cannot give enough immersive impression and interactivity for entertainment. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ACE 04, June 3-5, 2004, Singapore Copyright 2004 ACM /04/ $5.00. On the other hand, Virtual Reality (VR) or Mixed Reality (MR) produces stronger impression to be immersed into the virtual world or virtual and real mixed world. Special devices, such as spherical screen, 3D display, and head mounted display, are often used for visualization. This enables viewers to engage and immerse into the action. If sporting events are presented with high sensation as being present at live event, watching the events can be more enjoyable and can emphasis viewer experience. In this paper, we extend arbitrary view generation technique to the field of Augmented Reality (AR) for immersive visualization of real sporting matches. We have already proposed view-synthesis method targeting dynamic events in a large space, and also developed Viewpoint on Demand System which enables viewers to select their favorite viewpoints while observation through standard GUI [14]. This paper introduces a new AR application based on the above system for mixed reality entertainment. The proposed system allows viewers to watch sporting events at any place in the real world via a head mounted display (HMD) with favorite viewpoint. For example, they can observe a soccer match on the table where a soccer stadium model is placed. Virtual view image of the objective scene is synthesized by view interpolation and overlaid on the stadium model in the real world. In the field of AR, typical approach includes overlaying virtual CG objects onto images sequences captured by video camera [2]. In the proposed approach, instead of CG objects, real images of sporting scene, which are captured by multiple cameras at stadium, are overlaid onto the stadium model based on the concepts of IBR [15]. In conventional methods, virtual objects, which have 3D shapes and positions, are inserted into the 3D space of the real world. On the contrary, the proposed method overlays virtual view images of sporting scene onto the stadium model without 3D information. For the geometric registration, a novel approach using projective geometry between cameras is introduced. When calculating player positions on HMD, using centroide of players instead of foot positions presents player motions more stably than previously proposed method [15]. This paper focuses on an application targeting soccer match observation for remote entertainment. The proposed method, however, can be applied to other sporting events or musical events and so on. 2. RELATED WORKS The methods for synthesizing arbitrary view images from a number of real camera images have been studied since

2 Figure 1: Overview of the proposed method. the 1990s in the field of computer vision [16, 22]. These techniques, called Image Based Rendering (IBR), can be categorized into two groups, Model Based Approach and Transfer Based Approach. Model Based Approach constructs a 3D shape model of an object to generate the desired view [7, 24, 28]. Since the quality of the virtual view image depends on accuracy of the 3D model, a large number of video cameras surrounding the object or range scanners are used for construction of an accurate model. Also strong camera calibration [27], which is carried out to relate 2D coordinates in images to 3D coordinates in object space, is usually required. As 3D positions of several points in the object space must be measured, this calibration becomes difficult especially in a large space. For such reasons, the object area is generally limited within a few cubic meters in this approach. On the other hand, Transfer Based Approach synthesizes arbitrary view images without an explicit 3D model [1, 5, 23]. Instead, image warping such as transfer of correspondence is employed for synthesizing new view images. The dense correspondence between the original images, which is required for view-synthesis, is often obtained manually or by the use of optical-flow, so almost all targets are static images or slightly varying images such as facial expression. Thus we have proposed view-synthesis method targeting dynamic events in a large-space such as a soccer match captured at stadium, which is included in Transfer Based Approach [12, 13]. As for arbitrary view presentation, A related approach to our method has been proposed in [21, 18]. Prince et al. introduce a system for live capture of 3-D content and simultaneous presentation in augmented reality [21]. The user can watch the superimposed images of a remote collaborator in the real world, whose action is captured by 15 cameras surround him/her. The difference from our method is that 3D models of the subjects are reconstructed using shapefrom-silhouette in order to render the appropriate view. The subject is captured in the limited area, which is the volume of 3.3m diameters and 2.5m heights, while our method is targeting a large-scale event at stadium. Koyama et al. have proposed a method for augmented virtuality presentation for soccer match [18]. Players are represented with a simplified 3D model, which are reconstructed from multiple videos. Observers can watch the motion of players with a CG stadium from arbitrary viewpoint positions. While this method calculates 3D positions of players and overlay them on the CG stadium, our proposed method superimposes players into any empty stadium in the real world via HMD. Image-based registration technique enables augmented reality presentation of soccer match without 3D information. One advantage of our method is view synthesis and presentation based on projective geometry between multiple cameras instead of reconstructing 3D models with strong camera calibration. Projective geometry between cameras can easily be obtained by just images, while strong calibration of multiple cameras is difficult to obtain, which is imperative in above two methods [21, 18]. The proposed method is new challenge to image based AR presentation for large-scale dynamic events. 3. OVERVIEW Figure 1 shows overview of the proposed method. A video see-through HMD is used for augmented reality entertainment. An observer sees a desktop stadium model in the real world through the HMD, while images of players and ball of the soccer scene are overlaid on the display. Firstly, a soccer match is taken by uncalibrated multiple cameras at stadium and stored as video images. The projective geometry used for view-synthesis is estimated between neighboring cameras. The proposed system employs fundamental matrices between the viewpoints of the cameras, and homographic matrices for the plane, which forms the soccer ground, between neighboring views. Then the features of captured video images, such as the player positions and correspondence map of players among multi-views, are obtained from time-series images. The above process is executed in

3 Figure 2: Process flowchart for virtual view synthesis. advance. The online process consists of three stages, (1)calculation of the viewpoint position, (2)virtual view synthesis for the soccer scene, and (3)overlay the synthesized scene on the stadium model. At the first stage, the viewpoint position of the observer is determined by the position and pose of the HMD camera. At the second stage, neighboring cameras near the viewpoint position, which are reference cameras, synthesize the virtual viewpoint image of the soccer scene by view interpolation. At the final stage, synthesized soccer scene is overlaid on the desktop stadium model through the HMD. The observer can virtually watch the soccer match from favorite viewpoints in the real world. 4. VIRTUAL VIEW SYNTHESIS In this section, we explain the algorithm of arbitrary view synthesis for the dynamic regions of soccer scene that stand for the events. Figure 2 indicates flowchart of the viewsynthesis algorithm. View interpolation between neighboring cameras near the virtual viewpoint position generates virtual view images of soccer players and ball for each frame. Firstly, all dynamic regions are extracted by subtracting the background image from the whole image of soccer scene. If the background image, which includes neither players nor ball, cannot be captured, it can be made by setting mode value of image sequence to each pixel. After the silhouettes have been extracted by binarization, every silhouette region is segmented with different label. The silhouette of a player between neighboring cameras is corresponded by using homography [11] of the ground plane. This is based on the fact that the feet of the players usually contact with the ground. If a player is occluded by other players, however, the above algorithm may not work well. In this case, segmented silhouettes of the previous frame are used for dividing and corresponding the silhouette regions of the current frame. Foot position of the occluded player in the current frame can be also calculated with homography of the ground plane. Then bounding box(surrounding rectangle for each player) is projected from the previous frame, and it segments overlapping players. In this way, silhouette is corresponded even when players occlude each other. Next, each pair of silhouettes is extracted to obtain the pixel-wise correspondence within the silhouette. Epipolar lines are drawn between neighboring views by using a fundamental matrix [11]. On each epipolar line, the edge points are at first corresponded and the pixels inside the silhouette are then corresponded by linear interpolation of the edge points. After a dense correspondence for the whole silhouette is obtained, the pixel positions and values are transferred from the source images of two reference cameras to the destination image by image morphing [5] as described by the following equations, ṕ = (1 α){(p 1 c 1 )z 1 + c 1 } + α{(p 2 c 2 )z 2 + c 2 } (1) I(ṕ) = (1 α)i(p 1 ) + αi(p 2 ) (2) where p 1, p 2 are the coordinates of the matching points in reference camera 1, 2, and c 1, c 2 are the coordinates of the principal points as well. I(p 1 ), I(p 2 ) are the value of p 1, p 2. ṕ is the interpolated coordinates and I(ṕ) is the interpolated value. α defines the interpolating weights to the reference cameras, and z 1, z 2 are the zoom ratios of the virtual camera to the reference camera 1, 2 respectively. All correspondences are used in the transfer to generate a warped image. Here two transfers are required, one from reference camera 1 and the other from reference camera 2. Two generated warped images are then blended to complete the image of the virtual view. If the color of a pixel is different in two images, the corresponding pixel in the virtual view is rendered with the average of the colors; otherwise the rendered color is taken from either actual image. The above algorithm is applied to every pair of silhouettes. Synthesizing them in order of distance from the viewpoint

4 completes view interpolation for dynamic regions of soccer scene. 5. VIDEO-BASED AUGMENTED REALITY SYSTEM One of the most important issues for AR systems is geometric registration between the real and the virtual world. This generates a correct view of a virtual object, and overlays it onto a view of the real world. Many kinds of methods for the geometric registration have been proposed, such as the methods using positioning sensors [3], vision-based method using captured AR images [10, 19, 25], and combining methods using both of them [4, 20]. In the proposed system, the virtual objects overlaid on the real world are the image of players and ball of the synthesized soccer scene from uncalibrated cameras. Therefore, even if the 3D positions and poses of the HMD can be obtained, it is useless for the registration between the virtual soccer scene and the desktop stadium model. Therefore we propose a new image-based registration method using projective geometry between cameras. 5.1 Calculating the position of the observer viewpoint In order to display soccer scene on the desktop stadium model, we need to generate the soccer scene at the same viewpoint of the HMD camera, and overlay the image onto the HMD. Our view-synthesis algorithm is based on view interpolation between neighboring two cameras near the virtual viewpoint as described in section 4. Then the position of the viewpoint of the generated soccer image is specified by three elements, which are (a) neighboring two reference cameras, (b) interpolating weight value between two reference cameras and (c) zoom ratio between the real camera and the virtual camera. We need to determine these elements from the HMD camera image, so that the HMD viewpoint can be same as the generated soccer image viewpoint Figure 3: Examples of edge images (top) and detected natural feature lines (bottom). Detecting feature lines The image of the soccer stadium model captured by the HMD camera contains natural feature lines, which are easy to track such as the lines of the penalty area or the goal area. We employ these natural feature lines instead of using any artificial markers. Thus the efforts for locating artificial markers can be reduced. The Canny operator [6] is first applied for edge detection and all edge points are mapped into the Hough space. The strong peaks that form the lines of penalty area and goal area are then found in the Hough space. The results of line detection are shown in Figure 3 (bottom) with the edge images (top). All elements for specifying virtual viewpoint position are determined based on these natural feature lines, which must be tracked in the HMD camera image at every frame. In the examples of Figure 3, 4 lines are tracked and used for determination of the viewpoint. Figure 4: Location of the vanishing points. sions of parallel lines appear to converge in a perspective projection. In the proposed method, orientations of the user s view are estimated by the position of the vanishing point. Two cameras whose orientations are the closest to the user s view orientation are selected as the reference cameras. In advance, locations of vanishing points in all viewpoint images captured at stadium are measured by extending lines of the goal area and the penalty area. Whenever HMD camera image is captured, the feature lines are detected in the image and the location is measured in the same way (See Figure 4). Just a horizontal component of the location of a vanishing point is used for calculation of viewpoint position because we assume that a user moves the viewpoint almost horizontally from side to side. We also assume that all cameras capturing a soccer match at a stadium are placed at the almost same height. According to such assumptions, we select two stadium images in which the location of the vanishing point is closest to the vanishing point in the HMD camera image as reference camera images. Then relative distance between the vanishing points of reference cameras and that of the HMD camera determines interpolating weight w as the following equation, w= Determination of reference cameras and interpolating weight We apply a vanishing point for selection of reference cameras and determination of interpolating weight between the cameras. Vanishing point is the point to which the exten- xhmd xstl xstr xstl (3) where VstL (xstl, ystl ) and VstR (xstr, ystr ) represent the vanishing points in two reference camera images, and also Vhmd (xhmd, yhmd ) represents the vanishing point in the HMD camera image.

5 Figure 6: Observation of soccer match on the desktop stadium model in the real world with HMD. Figure 5: Determination of the rendering positions on the HMD Determination of zoom ratio Just change of the interpolating weight does not generate the virtual view image at the same viewpoint of the HMD. For the registration between HMD camera image and stadium camera images, we need to determine the zoom ratio between these two cameras. If extrinsic parameters of the HMD camera and stadium cameras can be obtained, we can obtain the geometric registration between the HMD camera and stadium cameras. However, extrinsic and intrinsic parameters of cameras are unknown as the proposed method uses uncalibrated cameras. Therefore an image-based registration technique is required, and so the zoom ratio in addition to the interpolating weight is changed for the registration. Then we assume that the position change for the direction of the side line (right-and-left movement) can be controlled by the interpolating weight between reference stadium cameras, and also the position change for the direction of the goal line (back-and-forth movement) can be controlled by the zoom ratio between stadium cameras and the HMD camera. With this assumption, the focal length of the stadium camera f st and the HMD camera f hmd decide the zoom ratio as z = f hmd /f st. (4) The focal lengths of the HMD camera and stadium cameras are actually fixed, but the zoom ratio can be calculated by changing the focal length of the HMD camera virtually when we consider zooming as the change of the focal length. As uncalibrated cameras are used in our approach, the intrinsic parameters of the cameras are unknown. The focal length is computed with two vanishing points V 1(x v1, y v1) and V 2(x v2, y v2) by the following equation, x v1x v2 + y v1y v2 + f 2 = 0. (5) Here it is supposed the skew of the camera is 0, aspect ratio is 1, and principal point is the center of the image. The detailed explanation is found in [26]. 5.2 Presentation with HMD We use homography transformation for the registration between the HMD camera images and the synthesized soc- Figure 7: Camera configuration at soccer stadium. cer scenes. In order to generate natural views of a soccer match, the dynamic objects need to be rendered correctly onto the stadium model. Homographic matrix determines the positions of the players and ball on the HMD camera images. The homography represents transformation between the soccer ground plane of the real soccer scene and the plane of the stadium model captured in the HMD camera image. The homography is computed from more than 4 corner points of goal area and penalty area. The intersection points of detected feature lines are used as corner points of each area. The position of every player on HMD can be determined by transforming the position of the foot in the real camera images to the HMD image by homography of the ground plane. However, detecting the foot position in the real camera image is not stable and accurate, so the players may vibrate in the HMD image. In order to calculate correct positions of players, we use the centroide of the player region instead of the foot position. The centroide of ball region is used as well for rendering a ball. Centroidal lines of each player and ball (described as red line for ball in Figure 5) are projected onto HMD camera image from two reference camera image with the homography using the following equation, p hmd = H pst (6) where H is the homographic matrix that represents the transformation between the planes, and p st, p hmd are homogenous coordinates of the position in the reference camera image, in the HMD camera image accordingly. The intersection point of projected lines ǵ is the position of player/ball on the stadium model. Then we calculate the distance between the centroide and the ground plane, that is h 1 and h 2, by backprojection of intersection point ǵ to each reference image. The following equation gives the distance between

6 Figure 8: Soccer scenes taken at the real stadium and overlaid scene on the desktop stadium model. the centroide and the plane on the stadium model h in HMD camera image. h = (1 α)h1 z1 + αh2 z2 (7) where z1, z2 are zoom ratios and α is interpolating weight. Thus we can obtain the rendered positions of the dynamic regions on the stadium model even when players are jumping off the ground. Since it is obvious that players and ball exist on/over a soccer field, overlaying them onto the HMD camera images completes the destination image. 6. EXPERIMENTAL RESULTS We have implemented a free viewpoint observation system for actual soccer match. Figure 6 describes the proposed system where the observer see the desktop stadium model on the table through HMD. In preparation for observation, soccer matches were taken by multiple uncalibrated video cameras at two soccer stadiums. One is Oita stadium in Oita city, which is one of the stadiums the 2002 FIFA World Cup was held, and the other is Edogawa athletics stadium in Tokyo, Japan. As Figure 7 shows, a set of 4 fixed cameras was placed to one side of the soccer field to capture the penalty area mainly. The captured videos are converted to BMP format image sequences, which are composed of pixels, 24-bit-RGB color images. Secondly, fundamental matrices between the viewpoints of the cameras and homographic matrices between the planes form the ground in neighboring views are computed from images by manual selection of 50 corresponding feature points in the images. Then the positions of vanishing points are calculated in each image of actual camera positions with the lines of the goal/penalty area. In addition, for each frame of image sequences, silhouettes of the dynamic regions are extracted. After every region is segmented and labeled, the regions of the same player in the neighboring view images are corresponded by using homography of the ground plane between the views. The above process is implemented as preprocessing of arbitrary viewpoint observation and the dataset is stored in a server PC. When watching the soccer match from a remote location, the preprocessed data of soccer scene is transferred via Ethernet. The projective geometry, such as fundamental matrices and homographic matrices, and the vanishing point positions are supplied to a remote PC in advance. While observation, the texture and 2D positions of all player regions in two reference camera images and correspondence maps are sent to the remote PC for each frame according to the observer s viewpoint. At the remote PC, after observation starts, the lines indicate the goal area and the penalty area of stadium model are detected from the HMD camera image in each frame. Then virtual view image of the dynamic regions are synthesized in accordance with the viewpoint position determined by the feature lines. Next, homographic matrices of the field plane between each of reference camera images and the HMD camera image are calculated, and rendered position of the dynamic objects is determined with the homographic matrices. Finally the synthesized soccer scene is overlaid onto the real stadium model through the HMD. Online process is iterated until viewer stops observation of the soccer match. Figure 8 presents the captured soccer scenes at Edogawa stadium and results of overlaid soccer scenes on the stadium model. The first and the second columns are reference camera images used for virtual view generation, and the third column is the synthesized virtual view image. The fourth column is displayed soccer scene on the HMD and interpol-

7 Figure 9: Free viewpoint observation in mixed reality. Figure 10: Close-up view of the inserted soccer players and ball in the real world.

8 ating weight and zoom ratio are indicated as w and z at the bottom of each image. For example, the image on the top of the last columns was generated based on the parameters that interpolating ratio is 0.47 between camera 1 and camera 2, and zoom ratio is We see that the virtual ball, can be inserted naturally in the real world. When we compare the overlaid scene with the original soccer scene, the players are located at almost correct positions on the stadium model. The appearance of players and ball looks different in virtual view images and the overlaid soccer scenes. This is because the locations of players and ball are modified by homography transformation corresponding the appearances of the ground plane in HMD camera images. Thus overlaid soccer scene is comfortably fitted to the stadium model on HMD. Figure 9 shows free viewpoint video images of another soccer scenes taken at Oita stadium. Figure 10 presents some close-up views of the same scene. In Figure 9, from the top on the left to the bottom on the right, the results indicate the motion of players and ball are replayed smoothly. The rendered scenes look so natural that user does not feel any discomfort. However, the way to decide the viewpoint positions and zoom ratios is not stable enough. Therefore appearance of the objects sometimes has a small error. We are currently improving the method of determination of viewpoint position and zoom ratio stably. 7. CONCLUSIONS This paper has presented a method for free viewpoint video synthesis and presentation of sporting events in mixed reality. Arbitrary view synthesis algorithm for a dynamic event in a large space has been extended to the AR application so that the soccer match can be observed on a desktop stadium model in the real world. In order to overlay the virtual view images of the dynamic objects onto the real world environment, a novel registration method, which is based on projective geometry between cameras, is introduced. Only natural feature lines in HMD camera images are used for the geometric registration, without any artificial markers or sensor devices. The strong calibration of the HMD camera and multiple cameras that capture the subject is not necessary. The proposed method can be applied to observations not only on the desktop stadium model, but also at any place where user likes. It can be possible that sporting events, such as Olympic games or World cup games, held in a foreign country can be observed at the domestic stadium with the proposed concepts for mixed reality entertainment. 8. ACKNOWLEDGMENTS This work is supported in part by a Grant in Aid for the 21st century Center of Excellence for Optical and Electronic Device Technology for Access Network from the Ministry of Education, Culture, Sport, Science, and Technology in Japan. The first author is a JSPS Research Fellow and partly supported by JSPS Research Fellowship for Young Scientists. 9. REFERENCES [1] S. Avidan and A. Shashua. Novel view synthesis by cascading trilinear tensors. IEEE Trans. on Visualization and Computer Graphics, 4(4): , [2] R. T. Azuma. A survey of augmented reality. Presence, 6(4): , [3] M. Bajura, H. Fuchs, and R. Ohbuchi. Merging virtual objects with the real world: Seeing utlrasound. Commun of the ACM, 36(7):52 62, [4] M. Bajura and U. Neumann. Dynamic registration correction in video-based augmented reality system. IEEE Computer Graphics and Applications, 15(5):52 60, [5] T. Beier and S. Neely. Feature-based image metamorphosis. Proc. of SIGGRAPH 92, pages 35 42, [6] J. Canny. Computational approach to edge detection. IEEE Trans. on Pattern Analysis and Machine Intelligence, 8(6): , [7] S. E. Chen and L. Williams. View interpolation for image synthesis. Proc. of SIGGRAPH 93, pages , [8] CyberPlay. [9] W. Du, H. Li, and A. Gagalowicz. Video based 3d soccer scene reconstruction. Proc. of Mirage 2003, pages 70 75, Mar [10] V. Ferrari, T. Tuytelaars, and L. V. Bool. Markerless augmented reality with a real-time affine region tracker. Proc. of the IEEE and ACM Intl. Symposium on Augmented Reality, pages 87 96, [11] R. Hartley and A. Zisserman. Multiple view geometry in computer vision. Cambridge University Press, [12] N. Inamoto and H. Saito. Fly through view video generation of soccer scene. International Workshop on Entertainment Computing (IWEC2002) Workshop Note, pages , May [13] N. Inamoto and H. Saito. Intermediate view generation of soccer scene from multiple videos. Proc. of International Conference on Pattern Recognition (ICPR2002), 2: , August [14] N. Inamoto and H. Saito. Fly-through viewpoint video system for multi-view soccer movie using viewpoint interpolation. Proc. of Visual Communications and Image Processing (VCIP2003), SPIE, 5150(122), July [15] N. Inamoto and H. Saito. Immersive observation of virtualized soccer match at real stadium model. The Second International Symposium on Mixed and Augmented Reality (ISMAR03), pages , October [16] T. Kanade, P. J. Narayanan, and P. W. Rander. Virtualised reality: concepts and early results. Proc. of IEEE Workshop on Representation of Visual Scenes, pages 69 76, [17] I. Kitahara, Y. Ohta, H. Saito, S. Akimichi, T. Ono, and T. Kanade. Recording multiple videos in a large-scale space for large-scale virtualized reality. Proc. of International Display Workshops (AD/IDW 01)s, pages , [18] T. Koyama, I. Kitahara, and Y. Ohta. Live mixed-reality 3d video in soccer stadium. The Second International Symposium on Mixed and Augmented Reality (ISMAR03), pages , October [19] K. N. Kutulakos and J. Vallino. Affine object representations for calibration-free augmented reality.

9 Proc. IEEE Virtual Reality Ann. Int. Symp.(VRAIS 96), [20] U. Neumann, S. You, J. Hu, B. Jiang, and J. W. Lee. Augmented virtual environments (ave): Dynamic fusion of imagery and 3d models. Proc. of the IEEE Virtual Reality 2003, pages [21] S. Prince, A. D. Cheok, F. Farbiz, T. Williamson, N. Johnson, M. Billinghurst, and H. kato. 3d live: Real time captured content for mixed reality. Proc. of the International Symposium on Mixed and Augmented Reality (ISMAR 02), pages 7 13, September [22] H. Saito, S. Baba, and T. Kanade. Appearance-based virtual view generation from multicamera videos captured in the 3-d room. IEEE Trans. on Multimedia, 5(3): , September [23] S. M. Seitz and C. R. Dyer. View morphing. Proc. of SIGGRAPH 96, pages 21 30, [24] S. M. Seitz and C. R. Dyer. Photorealistic scene reconstruction by voxel coloring. Proc. Computer Vision and Pattern Recognition (CVPR1997), pages , [25] Y. Seo and K. Hong. Calibration-free augmented reality in perspective. IEEE Trans. on Visualization and Computer Graphics, 6(4): , [26] G. Simon, A. W. Fitzgibbob, and A. Zisserman. Markerless tracking using planar structures in the scene. Proc. of the International Symposium on Augmented Reality, pages , Oct [27] R. Y. Tsai. A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Journal of Robotics and Automation, RA-3(4): , August [28] M. D. Wheeler, Y. Sato, and K. Ikeuchi. Consensus surfaces for modeling 3d objects from multiple range images. DARPA Image Understanding Workshop, [29] S. Yaguchi and H. Saito. Arbitrary viewpoint video synthesis from multiple uncalibrated cameras. IEEE Trans. on Systems, Man and Cybernetics, PartB, 34(1): , February 2004.