A Novel Image-Based Rendering System With A Longitudinally Aligned Camera Array

Transcription

1 EUOGAPHICS 2 / A. de Sousa, J.C. Toes Shot Pesentations A Novel Image-Based endeing System With A Longitudinally Aligned Camea Aay Jiang Li, Kun Zhou, Yong Wang and Heung-Yeung Shum Micosoft eseach, China {jiangli, hshum}@micosoft.com Abstact This pape intoduces a novel image-based endeing system to captue, epesent and ende eal wold and synthetic scenes. In ou system, a longitudinally aligned camea aay is mounted on a otating am suppoted by a tipod. The cameas ae always aimed along the adial diection. The scene is captued by the camea aay that otates along a cicle. Each pixel of the captued images is indexed by 4 paametes, i.e. the otation angle of the camea aay, the longitudinal numbe of the camea, the image column numbe and the image ow numbe. Given the position and the viewing diection of an obseve, the system can geneate novel views by intepolating the captued pixels in eal time without any geometic epesentation. If the obseve is constained to move on a plane, the size of the scene data can be futhe educed to that of an appoximately 3.5D plenoptic function. Compaed with light field and Lumigaph, ou method povides an easie inside-looking-out captue configuation and a unifom spatial sampling patten. Ou system goes a step futhe than concentic mosaics by allowing uses to move continuously within a 3D cylindical space, thus uses can expeience significant lateal as well as longitudinal paallaxes and lighting changes of a scene. Moeove, ou method povides an image-based solution to the wandeing of a lage envionment though concatenation of vaious wandeing cicles. Ou technique has potential applications in entetainment, e-commece and communication. 1. Intoduction In ecent yeas, image-based endeing techniques have been developed to geneate novel views of an envionment fom a set of pe-acquied images. These techniques have contibuted significantly to the wandeing aound in vitual envionment. With the use of image-based endeing techniques, the cost of endeing a scene is independent of the scene complexity and tuly compelling photo-ealism can be achieved since the images can be diectly taken fom the eal wold. While some appoaches have been developed based on view intepolation 4, view mophing 1, and geometic ecovey 12, 13, a banch of appoaches which equies less inteaction while constucting is based on plenoptic functions. The oiginal 7D plenoptic function 2 was defined as the intensity of light ays passing though evey position, at evey possible angle, fo evey wavelength and at evey * This wok was completed while Kun Zhou and Yong Wang wee intens at Micosoft eseach, China. time instant. These ays ae used to econstuct vitual scenes in which people can wande and look aound in it. Howeve, at that time, many poblems including how to captue a scene and what a unifom sampling patten of a scene is have not been consideed. By ignoing time and wavelength, a 5D plenoptic function 9 is poposed. It is pefomed via intepolating a set of panoamic images at diffeent 3D locations, but difficult featue coespondence poblems emain to be solved. If the scene can be constained to a bounding box, a 5D plenoptic function can be educed to a 4D plenoptic function called the light field 8 o a Lumigaph 7. The Lumigaph has been used mostly to epesent small objects that ae viewed fom outside. To captue the light field o a Lumigaph, pecise camea poses have to be known o ecoveed. Since 4D data sets ae extemely lage and the sampling of a box is iegula, walkthoughs of a eal scene using the light field o a Lumigaph have not yet been fully demonstated. QuickTime V 5 using a collection of panoamas is a pactical system that lets uses stand at one position and look aound in an envionment. The small file size of a panoama, which epesents a 2D plenoptic function, makes

2 the system applicable. The weakness of the system is that uses have to jump between diffeent captue positions if they want to navigate in the envionment; theefoe the goal of continuously wandeing is still not achieved. As a futhe step, Concentic Mosaics 11 that epesent a 3D plenoptic function captue a scene by spinning an off-centeed camea on a otay table and ende novel views by combining appopiate captued ays. This method allows uses to move continuously in a cicula egion and obseve lateal paallax and lighting changes in the scene. Compaed with the light field o a Lumigaph, concentic mosaics ae easy to captue and have much smalle file size because only a 3D plenoptic function is constucted. P view P c P view D can be epoduced fom the captued ays, e.g. P c D, the viewing ays, e.g. P view F could neve be epoduced fom the captued ays, e.g. P c B since the cameas have neve captued the pat of the scene aound point F. The weakness of concentic mosaics lies in the less sampling of the vetical infomation of the scene. How can we captue moe infomation along vetical diection and still limit the size of data file below that of a 4D plenoptic function? The answe is staightfowad utilizing a camea aay instead of only one camea in the captue pocess. Using this setup, we can always etieve those viewing ays that ae off the captue plane fom cetain cameas in the vetical aay and no depth coections ae needed. Without depth coection, the system becomes a puely image-based endeing system that can automatically geneate novel views egadless of any geometic ecovey. In addition, lage envionments can be easily constucted via the concatenation of vaious captue cicles. At the same time, we obseved that not all the captued ays need to be stoed if uses wee constained to move on a plane as in the case of concentic mosaics. This makes the file size of this system equivalent to an appoximately 3.5D plenoptic function. (a) P view P c F A B C The emainde of this pape is oganized as follows. In Section 2, we intoduce the setup of ou captue system and discuss ou sampling consideations. The pocess that indexes and compesses the image data will be discussed in Section 3. Section 4 is devoted to the endeing of a novel view. The concatenation of vaious captue cicles will be discussed in Section 5. A demo that shows the wandeing in a oom is illustated in Section 6. Finally we conclude ou wok and discuss futue diections in section 7. D E otating am Balance weight (b) Figue 1: The poblems in concentic mosaics method. (a) Top view. (b) Side view. Tipod Camea aay Unfotunately, concentic mosaics method inevitably contains vetical distotions and lacks vetical paallax. As illustated in Figue 1, the view at any point P view within the captue cicle should be epoduced fom the images captued at some point P c on the cicle. Fo those viewing diections that ae paallel to the captue plane, the captued ays such as P c C ae identical to the equied viewing ays such as P view C. Howeve, fo othe viewing diections that ae not paallel to the captue plane, the captued ays such as P c B ae much diffeent to the equied viewing ays such as P view A. Even when depth coection is intoduced, not all the viewing ays can be epoduced fom the captued ays. As shown in Figue 1(b), although the viewing ays, e.g. Figue 2: The setup of the captue system. 2. The captue system Figue 2 illustates the setup of ou captue system. A longitudinally aligned camea aay is mounted on a hoizontal am suppoted by a tipod. The cameas ae always aimed along the adial diection. The scene is captued while the aay otates along a cicle, which is efeed to as the captue cicle. The plane swept out by the otating am is efeed as the captue plane.

3 y ϕ fovc camea aay should be so designed that the longitudinal field of view θ fovo of the obseve is still coveed by the camea aay even if he/she is located at the fa end of the wandeing cicle. We have O x H 2 θ fovo ( + ) tan 2 (2) Of couse, the longitudinal field of view of the obseve should not be wide than that of the captue camea, i.e. Figue 3: The elationship between the adius of the wandeing cicle and the lateal field of view of the captue camea. Let us fist explain some paametes that ae elated to the captue and endeing pocesses. As illustated in Figue 3, the adius of a cicle in which uses can feely move and view depends on both the lateal field of view ϕ fovc of the captue camea and the adius of the captue cicle. It is expessed as: = θ fovo ϕ fovc sin (1) 2 z o θ fovc Figue 4: The elationship between the height of the camea aay and the longitudinal field of view of the obseve. It is obvious that any ay that oiginates fom any viewpoint within the cicle and passes though the captue camea must be within the field of view of the captue camea. Theefoe, any novel view of the use can always be epoduced fom the captued images. We efe this cicle as the wandeing cicle. In addition, as shown in Figue 4, the height H of the x n n H θfovo θ fovc (3) Now let us conside how densely we should deploy the cameas on the aay and how many images each camea should take in one cicle. Assuming that the width and height of the novel image of the obseve ae w o and h o, espectively, the ideal longitudinal inteval d c between adjacent cameas should be d c 2 = ( + ) θ tan 2 h o fovo It means that each ow of the novel image coesponds to each camea on the aay. This guaantees that the longitudinal paallax of the scene will be epoduced. On the othe hand, the angula otation incement ϕ i of the camea aay should be appoximately ϕ (4) fovo ϕ i (5) wo It means that each column of the novel image coesponds to each angula position of the camea aay on the captue cicle. This guaantees the epoduction of the lateal paallax of the scene. Finally, the esolution of the captue camea should be chosen as that of the obseve s view. 3. Pocessing of captued image data Since the size of the captued image data of a scene is usually vey lage, it is necessay to compess the data file befoe the entie image data ae loaded into the main memoy of a compute. Compession makes sense because thee ae significant coelation and edundancy between adjacent images. The technique we choose is vecto quantization 6, which is a compession method with quick selective decoding 14. It is wothy to note that in the case whee the obseve is constained to move on a 2D cicula plane as in the case of concentic mosaics, we can futhe educe the captued image data by discading the pat of image aea that would neve be seen. As illustated in Figue 5, we daw a line

4 θ fovo z o θ fovc x n j j-1 The captued ays ae indexed by 4 paametes, i.e. the otation angle of the camea aay, the longitudinal numbe of the camea, the image column numbe and the image ow numbe. We will discuss the detemination of these 4 paametes in the following. 4.1 Detemination of the otation angle of the camea aay As illustated in Figue 6, an obseve is supposed to stand at point P. One of the viewing ays fom the obseve is denoted as V. P and V ae the pojection vectos of P and V on the captue plane espectively. y Figue 5: The eduction of the captued image data. connecting the fa end of the wandeing cicle and the (j-1)th camea. The novel viewing ays that oiginate within the wandeing cicle and pass though the inteval between camea j-1 and camea j should be epoduced by intepolating between captued ays of camea (j-1) and camea j. The elevation angles of these ays ae always lage than that of the line of camea j, which is paallel to the above connecting line. Theefoe, fo camea j, the captued ays with thei elevation angles smalle than that of the above paallel line would neve be used in the endeing. So we only need to stoe the pats of image ows with thei coesponding elevation angles lying between angles actan ( j 1) d c ( ) + and θ fovc /2 fo cameas with j > o actan ( j + 1) d c ( ) + and -θ fovc /2 fo cameas with j <. It is obvious that the fathe away is the camea fom the aay cente, the fewe ae the image ows needed to be stoed. Thus the amount of captued data as a 4D plenoptic function is effectively educed to the amount of an appoximate 3.5D plenoptic function. Compaing with light field and Lumigaph, this system significantly educes the data size without sacificing any 3D paallaxes. Contasting with concentic mosaics, this system eliminates vetical distotions and displays significant longitudinal paallax and lighting changes. 4. endeing novel views The pinciple of endeing novel views is to intepolate each novel viewing ay by finding the neaest captued ays. -n O ϕ Q Q P tv x i+1th otating position ith otating position Figue 6: The detemination of the otation angle of the camea aay. The intesection point Q of the viewing ay V and the captue cicle is obtained by Q = + (6) P tv whee t is the positive oot of the equation P = tv (7) The diection fom the cicle cente to the intesection point Q may eithe coincide with one of the otating positions of the camea aay o lie between two adjacent otating positions. In the latte case, the intepolation weights ae invesely popotional to the angula diffeences between the diection and the two otating positions. 4.2 Detemination of the image column numbe Accoding to geometic elation, the azimuth angle ϕ between V and the diection of camea at the above otating position is equal to the azimuth angle ϕ V of V minus the azimuth angle ϕ Q of the otating position Q. The angle ϕ may coespond to eithe one o two adjacent columns of the images captued by the camea aay at the otating position. In the latte case, the intepolation weights ae invesely popotional to the angula diffeences between ϕ and those of the two adjacent columns of the images.

5 n View pot of camea P z V Q j+1 j j-1 V A B D C V P o Q -(j-1) -j -(j+1) Figue 7: The detemination of the longitudinal camea numbe. -n O Figue 8: The detemination of the image ow numbe. 5. Concatenation of the captue cicles 4.3 Detemination of the longitudinal camea numbe Figue 7 shows a diagam of a section plane detemined by the point P and the viewing diection V and its pojection V. The intesection point Q of the viewing ay V and the captue cylinde can be obtained by Q = P + tv (8) whee t is the positive oot of Eq.(7). The height of the intesection point Q may be equal to the height(s) of eithe one o two longitudinally adjacent cameas. In the latte case, the intepolation weights ae invesely popotional to the diffeences between the height of Q and the heights of the two longitudinally adjacent cameas. (a) 4.4 Detemination of the image ow numbe Figue 8 shows a diagam of viewpot of a camea. Assume that the above viewing ay V intesects with image plane of the viewpot at point A. Let B be the pojection of A on the lateal axis of the viewpot, theefoe AOB is equal to the elevation angle of vecto V. Suppose that a line that is paallel to the lateal axis and passes though A intesects with the longitudinal axis of the viewpot at D. It is angle COD instead of angle AOB that diectly detemines the coesponding ows of the captued images. The elationship between COD and AOB is V Q B Q A B (b) P A ( COD) = tan( AOB) sec( BOC) tan (9) whee BOC is exactly the angula diffeence ϕ mentioned in section 4.2. Angle COD may eithe be equal to one of the coesponding angles of the image ows o lie between the coesponding angles of two adjacent image ows. In the latte case, the weights of the linea intepolation ae detemined by the angula diffeences of COD and the coesponding angles of two adjacent image ows. Figue 9: The concatenation of captue cicles. One of the advantages of this system is that it povides a theoetical and pactical image based solution to the wandeing of lage envionments. Pevious QuickTime V method equies obseves to jump between hot spots, and light field and Lumigaph ae all local methods and neve povide concatenation solutions. As illustated in Figue 9, in this system, it is vey easy to concatenate vaious captue cicles and extend the wandeing space of obseves. In

6 Figue 9, captue cicles with adius ae epesented by solid lines and wandeing cicles with adius ae dawn in dash lines. Figue 9(a) indicates one of the most spase situations of the concatenation of the captue cicles, which allows an obseve to continuously move fom one wandeing cicle to anothe wandeing cicle though thei tangent points, wheeas Figue 9(b) shows one of the most dense situations, which allows obseves to move feely between wandeing cicles without any estiction. Othe configuations may be designed accoding to the distibution of scene objects. The endeing of novel views is also vey easy when obseves ae walking though vaious wandeing cicles since this method does not ely on any geometic epesentation of the scene. If the obseve stands at a common aea of some wandeing cicles, essentially anyone of the elated captue cicles can be used in the endeing of novel view (See the definition of wandeing cicle in Section 2). Fo consistence, we usually choose the captue cicle neaest to the viewpoint. Fo example, assuming that an obseve stands within a common aea of wandeing cicles A and B (See Figue 9 (b)). The hoizontal pojection of one of the ays oiginated fom the field of view of the obseve is denoted as V. It intesects with cicle A and cicle B at Q A and Q B espectively. We use captue cicle A to epoduce pixel coesponding to the viewing ay in the novel view since Q A is neae than Q B to P. 6. Expeiment esults We simulate the captue pocess in a synthetic scene, which is modified fom the scene Bians Beach Bungalow downloaded fom 3DCAFE 1. We choose the adius of the captue cicle as 1.57 metes. Both of the vetical and lateal fields of view of the camea ae 45. Theefoe the adius of wandeing cicle is.6 metes accoding to Eq.(1). We aange 61 cameas in an aay of 2.7 metes high. Each camea captue 36 pictues with a esolution of pixels as the aay otates one ound. It costs about 3 hous to ende a total of 2196 images in a Pentium III 5 PC. The amount of the esultant aw data is about 4GB. Afte vecto quantization (12:1) and Lempel-Ziv coding (4:1), the size of the data file is educed to 8MB. Ou system achieves a fame ate of 15 fames pe second in the wandeing. As illustated in Figue 1, uses can move left (a 1 ) and ight (a 2 ), up (b 1 ) and down (b 2 ), fowad (c 1 ) and backwad (c 2 ), and look upwad (d 1 ) and downwad (d 2 ). One can see that lateal and longitudinal paallaxes obviously exist between these pictues. In addition, the vetical lines of a wall ae obviously inclined when the obseve looks upwad (d 1 ) and download (d 2 ). Theefoe, ou system coectly epoduces the pespective effects. Inteested eades can visit to view a video demo of the wandeing. 7. Conclusion In this pape we have descibed a novel image-based endeing system. Accoding to the configuation of the system, images of scenes ae captued using a longitudinally aligned camea aay otating along a cicle with its oientation kept outwad. Each pixel of the captued images, which epesents a viewing ay of a scene, is indexed by 4 paametes, the otation angle of the camea aay, the longitudinal numbe of the camea, the image column numbe and the image ow numbe. When the viewing position and diection of an obseve ae given, the system geneates novel views by intepolating the captued pixels in eal time without any geometic models. This system epesents a novel sampling patten of a 4D plenoptic function. Compaed to the light field, Lumigaph and othe sampling method of 4D plenoptic function 3, ou method povides an easie captue configuation, a unifom spatial sampling and an outwad looking expeience. Because ou method does not equie any geometic ecovey, it is suitable fo wandeing aound in a lage envionment. In the case that obseves only need to move inside a cicle on a plane, concentic mosaics povided a 3D plenoptic function solution. Howeve, vetical distotions exist and vetical paallax is absent due to insufficient sampling along the vetical diection. Indeed, fo the situation of wandeing on a 2D plane, a 4D plenoptic function is still needed with 2D fo position and 2D fo ay diection. athe than employing a standad 4D plenoptic function that captues ays at evey point on the plane along evey diection, ou method only captues ays by otating a vetically aligned camea aay. By discading pats of captued image aeas that will neve be seen by the obseve, ou method educes the file size to that of an appoximately 3.5D plenoptic function. We ae woking on a numbe of poblems towads the impovement and the pactical use of the system. Fist, since it is difficult to pactically mount dozens of cameas on a vetical ba, we ae designing a simila device using only one camea but elevating the otation am a cetain height in each ound of otation. If it costs 2 minutes to otate one ound, the total captue time would be seveal hous. Second, since thee ae significant coelation and edundancy between adjacent images, moe efficient compession methods such as pediction-based IB compession that employs MPEG4 codec techniques ae unde study. Thid, as the file size of the scene data is still much lage than the cuently available bandwidth on the Intenet, a andom access method that only etieves pats of the data necessay fo endeing cuent views is unde development. Finally, special pupose CCD aay equipments that can eplace the odinay camea aay in the captue setup could also be designed. Acknowledgements We would like to thank Min-Sheng Wu fo poviding compession code and Hong-Hui Sun fo implementing the fist pototype of the system. Yiying Tong, an inten fom Zhejiang Univesity also made his contibutions to the completion of the system. Finally, special thanks go to Ka Yan Chan fo patiently poofeading the pape.

7 efeences E. H. Adelson and J. Begen, The Plenoptic Function and the Elements of Ealy Vesion, Computational Models of Visual Poceeding, pp.3-2, MIT pess, Cambidge, MA, Emilio Camahot, Apostolos Leios and Donald Fussell, Unifomly Sampled Light Fields, endeing Techniques 98, pp , S. E. Chen and L. Williams, View Intepolation fo Image Synthesis, ACM Compute Gaphics, Poc. of SIGGAPH 1993, pp , S. E. Chen, QuickTime V An Image-based Appoach to Vitual Envionment Navigation, ACM Compute Gaphics, Poc. of SIGGAPH 1995, pp.29-38, A. Gesho,. M. Gay, Vecto Quantization and signal compession, Kluwe Academic Publishes, S. J. Gotle,. Gzeszczuk,. Szeliski, and M. F. Cohen, The Lumigaph, ACM Compute Gaphics, Poc. of SIGGAPH 1996, pp.43-54, M. Levoy and P. Hanahan, Light Field endeing, ACM Compute Gaphics, Poc. SIGGAPH 1996, pp.31-42, L. McMillan and G. Bishop. Plenoptic Modeling: An Image-based endeing System, ACM Compute Gaphics, Poc. of SIGGAPH 1995, pp.39-46, S. M. Seitz and C. M. Dye, View Mophing, ACM Compute Gaphics, Poc. of SIGGAPH 1996, pp.21-3, H. Y. Shum and L. W. He, endeing with Concentic Mosaics, ACM Compute Gaphics, Poc. of SIGGAPH 1999, pp , Y. Yu and J. Malik, ecoveing Photometic Popeties of Achitectual Scenes fom Photogaphs, ACM Compute Gaphics, Poc. of SIGGAPH 1996, pp , Y. Yu, P. Debevec, J. Malik and T. Hawkins, Invese Global Illumination: ecoveing eflectance Models of eal Scenes fom Photogaphs, ACM Compute Gaphics, Poc. of SIGGAPH 1999, pp , J. Ziv and A. Lempel, A Univesal Algoithm fo Sequential Data Compession, IEEE Tansactions on Infomation Theoy, Vol.23, pp , (a 1 ) (a 2 ) (b 1 ) (b 2 )

8 (c 1 ) (c 2 ) (d 1 ) (d 2 ) Figue 1: The wandeing in a living oom. A use can move left (a 1 ) and ight (a 2 ), up (b 2 ) and down (b 2 ), fowad (c 1 ) and backwad (c 2 ), look up (d 1 ) and down (d 2 ), and take othe conventional actions such as tuning left and ight, and zooming in and zooming out.