LAMP: 3D Layered, Adaptive-resolution and Multiperspective Panorama - a New Scene Representation

Transcription

1 Submission o Special Issue of CVIU on Model-based and Image-based 3D Scene Represenaion for Ineracive Visualizaion LAMP: 3D Layered, Adapive-resoluion and Muliperspecive Panorama - a New Scene Represenaion Zhigang Zhu Deparmen of Compuer Science The Ciy College, The Ciy Universiy of New York, New York, NY Allen R. Hanson Deparmen of Compuer Science Universiy of Massachuses Amhers, Amhers, MA Shor Running Tile: 3D LAMP Represenaion Conac Informaion: Prof. Zhigang Zhu Deparmen of Compuer Science The Ciy College of New York /CUNY Conven Avenue and 138h Sree, New York, NY Tel: (212) Fax: (212) zhu@cs.ccny.cuny.edu URL: hp://www-cs.engr.ccny.cuny.edu/~zhu/

2 Absrac A compac visual represenaion, called he 3D Layered, Adapive-resoluion and Muliperspecive Panorama (LAMP), is proposed for represening large-scale 3D scenes wih large variaions of dephs and obvious occlusions. Two kinds of 3D LAMP represenaions are proposed: he relief-like LAMP and he image-based LAMP. Boh ypes of LAMPs concisely represen almos all he informaion from a long image sequence. Mehods o consruc LAMP represenaions from video sequences wih dominan ranslaion are provided. The relief-like LAMP is basically a single exended muli-perspecive panoramic view image. Each pixel has a pair of exure and deph values, bu each pixel may also have muliple pairs of exure-deph values o represen occlusion in layers, in addiion o adapive resoluion changing wih deph. The image-based LAMP, on he oher hand, consiss of a se of muli-perspecive layers, each of which has a pair of 2D exure and deph maps, bu wih adapive ime-sampling scales depending on dephs of scene poins. Several examples of 3D LAMP consrucion for real image sequences are given. The 3D LAMP is a concise and powerful represenaion for image-based rendering. Keywords: image-based modeling and rendering, layered represenaion, muli-resoluion, muli-image processing, spaio-emporal image, epipolar plane image 1

3 1. Inroducion Relaed Work D LAMP: Overview of Our Approach Organizaion of he Paper Basic Panoramic Geomery Relief-like 3D LAMP Represenaion Image-Based 3D LAMP Represenaion Deph Layering Temporal Re-Sampling Deph, Occlusion and Resoluion Recovery Deph Recovery: he Panoramic EPI-Based Approach Deph Boundary Localizaion Occlusion Recovery Resoluion Recovery LAMP Consrucion and Rendering: Experimenal Resuls D LAMP Consrucion Resuls Exended Panoramic Image (XPI) Represenaion LAMP-Based Rendering Comparisons and Discussions Layered Represenaion Full Perspecive, Full Orhogonal, Muli-Perspecive and Mulivalued Represenaions Conclusions Acknowledgemens References

4 1. INTRODUCTION The problem of modeling and rendering real 3D scenes has received increasing aenion in recen years in boh he compuer vision and compuer graphics communiies [1-3, 5, 6, 8, 17-19, 22, 23, 27]. In order o build a visual represenaion from image sequences for re-rendering real 3D naural scenes, here are wo challenging issues ha need o be solved: he correspondence problem beween wo (or muliple) views, and a suiable geomeric represenaion of large scale scenes. Usually a suiable visual represenaion will ease he correspondence problem. Many of he successful image-based modeling and rendering approaches [1-3, 5, 6, 8-10] have ried o simplify or avoid he correspondence problem by using 2D image inerpolaion or video regisraion/mosaicing. On he oher hand, more general approaches need sophisicaed vision algorihms, such as in muli-view sereo [17, 18] or general moion analysis [10, 11, 22, 23] of an image sequence. Three classes of image-based represenaions have been proposed o represen video sequences of large-scale 3D scenes: panoramic (mosaic) represenaions, muli-view represenaion and layered represenaions. Building and mainaining a suiable visual represenaion of a large-scale scene remains an imporan research opic in image-based modeling and rendering Relaed Work Mosaic-based approach - A video mosaic is a composie of images creaed by regisering overlapping frames. Many of he curren successful image mosaic algorihms, however, only generae 2D mosaics from a camera roaing around is nodal poin [1,2,3]. Creaing muliperspecive sereo panoramas from one roaing camera off is nodal poin was proposed by Ishiguro, e al [4], Peleg, e al [5], and Shum & Szeliski [6]. A sysem for creaing a global view for visual navigaion by pasing ogeher columns from images aken by a smoohly ranslaing camera (comprising only a verical sli) has been proposed by Zheng & Tsuji [7]. The moving sli paradigm was laer used as he basis of he 2D manifold projecion approach for image mosaicing [8], he muliple-cener-of-projecion approach for image-based rendering [9], he epipolar plane analysis echniques for 3D reconsrucion [15], and he parallel-perspecive view inerpolaion mehods for sereo mosaicing [16, 27]. Muli-perspecive panoramas (or mosaics) exhibi very aracive properies for visual represenaion and epipolar geomery; however, 3D 3

5 recovery from sereo mosaics sill faces he same problems as radiional sereo mehods - he correspondence problem and occlusion handling. The 3D informaion for regions ha are occluded in one of he views is usually difficul o obain since no correspondences can be esablished for hose regions. Layered represenaion - In a layered represenaion, a se of deph surfaces is firs esimaed from an image sequence from a single camera, and hen combined o generae a new view. Wang and Adelson [10] addressed he problem as he compuaion of 2D affine moion models and a se of affine suppor layers from an image sequence. The layered represenaion ha hey proposed consiss of hree maps in each layer: a mosaiced inensiy map, an alpha map, and a velociy map. Occlusion boundaries are represened as disconinuiies in a layer's alpha map (opaciy). This represenaion is a good choice for image compression of a video sequence and for limied image synhesis of seleced layers. Recenly, Ke & Kanade [22] proposed a subspace approach o reliably exrac planar layers from images by aking advanage of he fac ha homographies induced by planar paches in he scene form a low dimensional linear space. Occlusion regions are excluded from heir layered represenaion. Sawhney and Ayer [11] proposed a muliple moion esimaion mehod based on a Minimum Descripion Lengh (MDL) principle. However, heir algorihms are compuaionally expensive and require a combinaorial search o deermine he correc number of layers and he "projecive deph" of each poin in a layer. Occlusion regions are no recovered in heir layered model. Baker, Szeliski & Anandan [12] proposed a framework for exracing srucure from sereo pairs and represened a scene as a collecion of approximaely planar layers. Each layer consiss of an explici 3D plane equaion, a exure map (a sprie), and a map wih deph offses relaive o he plane. The iniial esimaes of he layers are recovered using echniques from parameric moion esimaion and hen refined using a re-synhesis algorihm which akes ino accoun boh occlusion and mixed pixels. More recen work on 3D layer exracion [23] uses an inegraed Bayesian approach o auomaically deermine he number of layers and he assignmen of individual pixels o layers. For more complex geomery, a layered deph image (LDI) has been proposed [13] which is a represenaion of a scene from a single inpu camera view, bu wih muliple pixels along each line of sigh. 4

6 Muli-view approach - Raher han consrucing a single mosaic from a sequence of images, muli-view approaches represen a scene by muliple images wih deph and exure. Chang and Zakhor [17] proposed a mehod o obain deph informaion of some pre-specified reference frames of an image sequence capured by an uncalibraed camera scanning a saionary scene, hen o ransform he poins of reference frames ono an image of he desired virual viewpoin. However, reference frames were chosen quie heurisically; a synhesized image from a viewpoin far away from ha of he reference frames leads o erroneous resuls since occluded or uncovered regions canno be well represened. Chang and Zakhor laer exended his work o a mulivalued represenaion (MVR) [26], which is auomaically consruced wih respec o a single reference frame from a se of dense deph maps. They poined ou ha occlusion levels come naurally from he dense deph informaion and argued ha because of visibiliy limiaions, real-world scenes ypically do no have more han hree occlusion levels. This disinguishes heir MVR from he layered represenaions, which usually have a much larger number of layers when using affine moion o group regions. Szeliski [18] presened a new approach o compuing dense deph and moion esimaes from muliple images. Raher han esimaing a single deph or moion map, a deph or moion map is associaed wih each inpu image (or some subse of hem). Furhermore, a moion compaibiliy consrain is used o ensure consisency beween hese esimaes, and occlusion relaionships are mainained by compuing pixel visibiliy D LAMP: Overview of Our Approach The goal of our work is o consruc a layered and panoramic represenaion of a large-scale 3D scene wih large variaions of dephs and obvious occlusions from primarily ranslaing video sequences. Our approach in his paper is based on a muli-perspecive panoramic view image (PVI) [7] and a se of epipolar plane images (EPIs) [14] exraced from a long image sequence, ogeher wih a panoramic deph map generaed by analyzing he EPIs [15, 21]. To accomplish his, we need o solve four problems: 1) generaing seamless PVIs and EPIs from video under more general moion han pure ranslaion in a large and real scene, 2) analyzing he huge amoun of daa in EPIs robusly and efficienly o obain dense deph informaion, 3) enhancing resoluion and recovering occlusions in a parallel-perspecive PVI represenaion, and 4) represening a large scale 3D scene wih differen dephs and occlusions efficienly and 5

7 compacly. While he firs wo issues are very imporan in consrucing a 3D model of a scene, hey have been discussed in our previous work [15, 21], so his paper will mainly focus on he las wo issues. We propose a new compac represenaion - 3D Layered, Adapive-resoluion and Muliperspecive Panorama (LAMP). The moivaion for layering is o represen occluded regions and he differen spaial resoluions of objecs wih differen deph ranges; meanwhile he model is represened in he form of a seamless muli-perspecive mosaic (panorama) wih viewpoins spanning a large disance. Two kinds of 3D LAMP represenaions are consruced: he relief-like LAMP and he 2D image-based LAMP. The relief-like LAMP is basically a single, exended, muli-perspecive PVI wih boh exure and deph values, bu each pixel has muliple values of exure-deph pairs o represen resuls of occlusion recovery and resoluion enhancemen. The image-based LAMP, on he oher hand, consiss of a se of muli-perspecive layers, each of which has boh exure and deph maps wih adapive ime-sampling scales depending on dephs of scene poins. The LAMP represenaion is relaed o such represenaions as he panoramic view image (PVI) [7, 15], sprie [12], and he layered deph image (LDI) [13]. However, i is more han a muli-perspecive PVI in ha deph, adapive-resoluion and occlusion are added in our represenaion. I is differen from he sprie (or he layered deph image) since he laer is a view of a scene from a single inpu camera view and is wihou adapive image resoluion. The 3D LAMP represenaion is capable of synhesizing images of new views wihin a reasonably resriced bu arbirary moving space, as is inensiy and deph maps conain almos all he informaion ha could be obained from an image sequence Organizaion of he Paper This paper is organized as follows. In secion 2, we describe he basic panoramic represenaion wih parallel-perspecive geomery, including boh exure and deph maps. Based on his supporing panoramic represenaion, we propose a relief-like LAMP represenaion in Secion 3, which feaures muliple layers, adapive resoluion, muli-perspecive, and panoramic views. In Secion 4, an image-based LAMP represenaion is presened wih jus a se of 2D images in order o simplify he 3D scene represenaion. Seps o conver a relief-like LAMP o an imagebased LAMP are provided. Secion 5 discusses a unique EPI-based approach o consruc a relief-like LAMP. This approach does no explicily exrac feaures, rack loci, or mach 6

8 correspondences. Insead, a spaio-emporal-frequency domain cross-analysis is performed o creae dense deph maps, o localize accurae deph boundaries, and o exrac occluded regions ha canno be seen in he seleced panoramic view. In Secion 6, experimenal resuls and some pracical consideraions are given for 3D LAMP consrucion, and 3D rendering based on LAMP represenaions are discussed. In Secion 7, we make wo comparisons. Firs, we compare our LAMP represenaion based on parallel-perspecive projecion wih hose represenaions having full perspecive or full orhogonal projecions. Second, we compare our image-based LAMP represenaion wih several exising layered represenaions. The las secion is a summary of his work. 2. BASIC PANORAMIC GEOMETRY For compleeness, we give a brief inroducion o he consrucions and represenaions of Panoramic View Images (PVIs) and Epipolar Plane Images (EPIs) and o he reconsrucion of a supporing PVI surface - a panoramic map wih boh deph and exure - from PVIs and EPIs. y Z y PVI (x=0) (a) X Y x O f V (b) o x EPI (y=0) Fig. 1. ST image model. A PVI is a y image inside he xy cube and an EPI is an x image Le us consider he siuaion in which a model of a large-scale scene will be consruced from a long and dense video sequence capured by a camera moving in a sraigh line whose line of sigh (i.e., opical axis) is perpendicular o he moion (Fig. 1). The resuling sequence obeys he following spaio-emporal (ST) perspecive projecion model X + V Y x ( ) = f, y( ) = f (1) Z Z where (X,Y,Z) represens he 3D coordinae of a poin a ime =0, f is he camera focal lengh, and V is is speed. A feaure poin (x,y) forms a sraigh locus and is deph value is 7

9 V Vd D = Z = f = f (2) v dx where v = dx / d (3) is he slope of he sraigh locus. In oher words, wo kinds of useful 2D ST images can be exraced: (1) Panoramic View Images (PVIs) - he y- inersecions in he xy cube (Fig. 1), which is a parallel-perspecive image including mos of he 2D informaion of he scene from muliple viewpoins, and (2) Epipolar Plane Images (EPIs) - he x- inersecions in he xy cube, whose ST locus orienaions represen dephs of scene poins. Fig. 1 illusraes he cenral PVI and one of he EPIs. (a) Frame 0 Frame 200 Frame 400 Frame 600 Frame 1000 (b) (c) Fig. 2. A few frames of he sabilized Main Building (MB) sequence in (a), and a pair of sereo PVIs: (b) x = 0 and (c) x= -56. Whie lines indicae maches. In real scenes, he moion of a camera usually will be composed of a dominan ranslaion plus unpredicable variaions. In addiion o deph recovery and occlusion handling, a pre-processing sep is needed o generae a sabilized image sequence wih only a 1D ranslaion by using image sabilizaion and image mosaicing echniques [15, 21]. Fig. 2 shows wo PVIs ha are exraced from x=0 and x=-56 of a 128*128*1024 xy image cube of a building scene - he sabilized Main Building (MB) sequence. The MB sequence was capured by a camcorder mouned on a ripod carried by a moving vehicle. Oher han he small vibraion of he vehicle on a normal oudoor 8

10 road, he ranslaional velociy was mosly consan. Fig. 2a shows a few frames afer video sabilizaion; image ransformaions for removing he camera vibraion produce blank regions (shown in black) around he borders of he sabilized frames, paricularly visible on he righ sides of Frame 0 and Frame The camera vibraion is also demonsraed by he irregular boundaries of he mosaic in Fig. 7. Muli-perspecive PVIs provide a compac represenaion for a large-scale scene, and sereo PVIs can be used o esimae he deph informaion of he scene [7, 6, 16, 27]. In panoramic sereo (Eq. (2), Fig. 2), he "dispariy" dx is fixed; and he disance D is proporional o d, he sereoscopic displacemen in. This indicaes ha deph resoluions are he same for differen dephs. However, we sill face wo problems in order o use sereo PVIs o recover 3D informaion - he correspondence problem and he occlusion problem. EPI (y=9) PVI (x=0) y x o occluded side PVI (x=-56) Fig. 3. Each EPI is a composie image of a scanline from all possible views of an image sequence, and includes informaion abou deph, occlusion and spaial resoluion of 3D scene poins Our soluion o hese wo problems is o effecively use he informaion ha is coninuous beween wo views, i.e., he epipolar plane images, o obain a supporing PVI surface - a panoramic view image wih boh exure and deph maps, and hen he occluded regions ha canno be seen from he PVI. Fig. 3 shows an EPI from which he occluded (and side) regions as well as dephs can be recovered; he algorihms o exrac deph and o recover occlusion/resoluion will be discussed in Secion 5. The supporing PVI surface is a 2D panoramic view image wih boh exure and deph map (Fig. 4a) and is he base for our LAMP represenaion. The exure map is a y- inersecion plane in he xy image cube, and he deph map has deph values for each pixel. In principle, we can exrac he supporing PVI surface from 9

11 any x locaion inside he xy cube. In his paper, we usually selec a cenral PVI (wih x = 0) ha has orhogonal parallel rays along he direcion of moion. For consrucing a dense deph map for he supporing PVI, we need o process all he EPIs inside he xy cube. While he 1D moion model will be used o develop our EPI-based approach in his paper, LAMP represenaions can be consruced under a more general [15, 16, 27] or differen moion [5,6], and/or using oher approaches [10, 11, 12, 22, 23]. (a) Texure and deph maps of he supporing PVI surface (back of he relief). y -x y -x (b) Texure and deph maps of he relief surface (fron of he relief). Fig. 4. The base layer of he relief-like LAMP represenaion for he MB sequence. 3. RELIEF-LIKE 3D LAMP REPRESENTATION Based on he represenaion of a muli-perspecive supporing PVI surface (conaining boh exure map and deph map), we propose a compac and comprehensive represenaion called he 3D LAMP- Layered, Adapive-resoluion and Muli-perspecive Panorama. We will explain he LAMP model by an illusraive example shown in Fig. 5 using a simple 1D scene. Recall ha a 3D ST image (which is a 2D EPI for he 1D scene in Fig. 5) includes everyhing from an image 10

12 sequence. Firs, we will give a basic LAMP represenaion - he relief-like 3D LAMP - ha is direcly carved from he 3D ST image (xy image). The relief-like 3D LAMP basically cus ou some essenial par of he 3D ST image depending on he deph value for each pixel. I has he following four properies ha make i very suiable for image-based modeling and rendering: (a) d 1 s 1 d 2 s 2 d 3 ( o 1 o 2 ) 1D scene Z view direcions for LAMP d 4 X O V x side layer s 1 EPI loci occluded layer o 2 (b) 0 base layer d 1 0 s e d 2 d 3 supporing-surface relief-surface side layer s 2 occluded layer o 1 occluding objec d 4 Fig. 5. A 1D illusraive scene and is LAMP represenaion. (a) 1D scene wih four horizonal dephs (d 1 -d 4 ) and wo sides (s 1,s 2 ), (b) relief-like LAMP and deph layering for he image-based LAMP from an EPI. Noe ha in order o represen all he informaion presened in an image sequence of he scene, differen parallel viewing direcions in (a) are needed, which are refleced in he LAMP represenaion as muliple layers in (b). 1). I is a panoramic image-based represenaion wih 3D informaion. A large-scale scene will firs be represened by a base layer, which is indexed by a seamless 2D panoramic view image (PVI) consising of boh exure and deph maps. This 2D PVI (see Fig. 4a for a real example) is defined as he supporing surface (back surface) of he base layer in he relief-like LAMP, which makes he LAMP represenaion very efficien for archiving (modeling) and rerieval (rendering). 2). I has adapive image resoluion. In he base layer (which includes dephs d 1, d 2, d 3 and d 4 in Fig. 5b), each poin has an aached sreak of muliple pixels in he x direcion, wih boh exure and deph informaion, in order o represen he resoluion loss in he muli-perspecive PVI (he supporing surface). The slope of he locus, i.e. he image velociy v in Eq. (3), gives us he 11

13 number of pixels if i is greaer han 1 (v>1). Oherwise he number of pixels is 1. Real examples can be found in he righ hand side of he panoramic image in Fig. 4a where he doors of he buildings and he bushes in fron of i appear narrower han he real ones. In he LAMP represenaion, he number of pixels ( emporal resoluion ) a each poin in he supporing surface (where x = 0) adapively changes wih he deph of ha poin. The adapive emporal sampling in he horizonal direcion and he inheren adapive spaio-sampling of perspecive projecions in he verical direcion recover and preserve image resoluions of he original frames in a saisfying way. The name relief-like comes from he fac ha he appearance of he fron surface (relief surface) of his represenaion is somewha like a relief sculpure, in which forms and figures (wih image velociies v > 1) are disinguished from a surrounding planar surface (he supporing surface) (Fig. 5b). Fig. 4b shows he corresponding relief surface of he base layer of he relieflike LAMP represenaion for he MB sequence (refer o Fig. 14 for all he inernal daa). Each pixel in he relief-like LAMP is associaed wih a locaion wih (x,y,) coordinaes; for he supporing surface, we have x = 0. The end poin in he relief surface in he locaion (y,) is exacly conneced wih he sar poin in he supporing surface in he locaion (y, +1) (Fig. 5). This feaure allows us o generae seamless mosaics in he image-based LAMP represenaion in he nex secion. 3). I is a layered represenaion. Addiional occluded layers, which are smaller pieces of muliperspecive view images compared wih he complee based layer, represen he occluded and side regions (regions o 1, o 2, s 1 and s 2 in Fig. 5b) ha are no visible in he seleced parallel panoramic view of he base layer (which is orhogonal o he camera pah in Fig. 5). They are aached o he base layer wih he same represenaion (exure, deph and adapive resoluion) as he base layer. Each of he occluded x-segmens is aached o a deph boundary poin (y, 0 ), and has an x coordinae, a sar ime s and an end ime e ha mark is posiion in he xy image (Fig. 5b). We wan o emphasize ha in order o represen all he informaion presened in an image sequence of he scene, differen parallel viewing direcions (shown in Fig. 5a) are needed, which are refleced in he LAMP represenaion (Fig. 5b) as muliple layers wih differen x coordinaes. 12

14 4). I is a muliple perspecive image. The PVI is a muli-viewpoin perspecive image (i.e., he y- image). Each sub-image (a one-column sli-image in concep) in he muli-perspecive panorama is full perspecive, bu successive sli images have differen viewpoins. The muliperspecive represenaion acs as a bridge in he image-based modeling and rendering pipeline beween he modeling end from he original perspecive sequences and he rendering end for new perspecive images, boh wih changing viewpoins over a large disance. In conclusion, a relief-like LAMP is composed of a complee base layer and a se of occluded layer pieces. I can be viewed as an essenial par of he xy image cube in which each pixel has wo aribues - exure (represened by inensiy I ) and deph (represened by he image velociy v ) - conneced o is coordinaes (x,y,), where he x coordinae is implicily represened. The daa srucure of a relief-like LAMP is a spaio-emporal (ST) 2D array ha is defined as follows: sruc x-sreak { in x 0 ; // sar x coordinae, indicaing he viewing direcion of he supporing surface in xlen; // lengh of he x-sreak, indicae hickness of he relief in I[xlen]; // exure: inensiy array of he x-sreak floa v[xlen]; // deph: image velociy array of he x-sreak } sruc Occlusion-Segmen { char ype; // segmen ype: OCCLUDED or SIDE in s, e ; // sar frame and end frame of he segmen x-sreak r[ e - s +1]; // x-sreaks in his segmen } sruc Relief-LAMP-Elemen { x-sreak *r; // a x-sreak Occlusion-Segmen *occ; // a occluding segmen; NULL if no a deph boundary } Relief-LAMP-Elemen rlamp[ydim][tdim]; // rlamp[y][]: y=0~ YDIM, = 0~TDIM I is clear ha a relief-like LAMP is a 2D spaio-emporal array indexed by y and ; a each (y,) locaion here is an x-sreak, which is a 1D segmen along he x direcion inside he xy cube and an opional Occlusion_Segmen ha consiss of a number of x-sreaks. In he curren implemenaion, each x-sreak assigns he same image velociy v of he sar poin (x 0,y,) in he supporing surface. In oher words, for resoluion enhancemen, we only sample v pixels along he x direcion and assign hem he same v value. An Occlusion-Segmen is one of wo ypes - OCCLUDED ha has he same deph, or SIDE ha has linearly inerpolaed dephs. Clearly, he 13

15 represenaion here allows furher refinemen of he deph map in ha each (x,y,) pixel in he relief-like LAMP can has is own deph. The 3D coordinaes of each pixel (x,y,) in he relieflike LAMP can be recovered by he following equaion Z V V V = F, Y = y, X = x V (4) v v v given v(x,y,) in he relief-like LAMP represenaion. 4. IMAGE-BASED 3D LAMP REPRESENTATION A relief-like LAMP can be viewed as having adapive ime sampling for every single pixel in he panoramic supporing surface as well as including all he occluded regions represened in he same way. This represenaion is good for a complex scene ha has many small deph layers. However, in erms of daa srucures, he relief-like LAMP is an inhomogeneous represenaion raher han a se of homogeneous 2D image arrays. In addiion, he base layer usually includes objecs a differen deph levels, and he occluded layers are no merged ino he regions hey belong o. For example, regions o 1 and o 2 in Fig. 5 belong o he same deph surface as region d 3, bu hey are segmened ino wo addiional occluded layers. Hence a naural exension of he basic LAMP is o exrac from i a more concise represenaion - an image-based 3D LAMP. In an image-based 3D LAMP, a scene is represened as muliple layers of 2D mosaiced images in which each layer is ruly represened by wo 2D arrays - a exure map and a deph map. We wish o creae a panoramic mosaic image for each layer ha is boh seamless and preserves adequae image resoluions. There are wo seps necessary o fulfill his goal: deph layering and ime re-sampling Deph Layering The moivaion for layering is o represen occluded regions and differen spaial resoluions of objecs wih differen deph ranges in differen layers. An image based LAMP is layered according o occluding relaions raher han merely dephs, which is also used by oher researchers as a powerful observaion o generae a more compac image-based represenaion [26]. The scene pars wih varying (bu no disconinuous) dephs in a single layer will furher use adapive emporal sampling raes in he second sep o represen differen resoluions for 14

16 differen dephs along he direcion of parallel projecion. Concepually, he implemenaion of deph layering from a relief-like LAMP is sraighforward since we can easily deermine where he occluding regions in he relief-like base layer should be segmened and pu ino separae image-based layers. The same is rue for he occluded or side regions in he occluded layers. Generally speaking, layers eiher in he same deph range or wih coninuous dephs will be merged ino one single layer. Regions wih occluding boundaries will be divided. For example, in Fig. 5b, he wo occluded regions ( o 1 and o 2 ) will be merged ino he base layer wih deph d 3, while he occluding region (d 4 ) originally included in he base layer will be separaed ou as a new layer. Ideally, side regions ( s 1 and s 2 in Fig. 5b) should be insered ino he base layer since hey can connec well (in boh deph and exure) wih he base layer. However, in our curren implemenaion we pu hem ino wo separae layers for simpliciy. Afer deph layering, he original relief-like LAMP is divided ino several single-layer relief-like LAMPs, ready for creaing image-based layers. Each of hem only includes a base layer, wih an x coordinae o indicae where i is aken from in he xy cube. Differen x coordinaes in he relief-like LAMP imply differen viewing direcions of parallel projecions in order o beer capure surfaces wih differen orienaions (see Fig. 5a ). +1 x-segmen +1 Supporing-surface (back) w re-sampling Image-based LAMP Relief surface (fron) Relief-like LAMP y x Fig. 6. Temporal re-sampling and seamless mosaicing 4.2. Temporal Re-Sampling From each single-layer relief-like LAMP ha is obained by deph layering (as shown in he lef par of Fig. 6), we will generae a seamless 2D panoramic image wih boh exure and deph aribues. In each column () of such a relief-like LAMP (Fig. 6), he number of pixels in he x direcion of a poin in he supporing surface is inversely proporional o he deph of ha poin. Since dephs may change along he y direcion, he x-y slice a each column usually has varying widhs in he x direcion. In order o creae a regular 2D image, we wan o warp each irregular x- 15

17 y slice (in he relief-like LAMP) wih varying widhs in he x direcion ino a recangular -y slice of equal widh w (w 1) in he direcion, which will be siched ino a 2D seamless mosaic in he image-based LAMP (see he righ par of Fig. 6). The widh of he -y slice image a ime is calculaed as he dominan image velociy of all v(y, ) in column of he supporing surface, e.g. w = v( ) median{ v( y, )} (5) = y Noe ha each x-segmen of he x-y slice in a relief-like LAMP sars from x 0 and ends a x e (y,)= x 0 -v(y, ); for he base layer x 0 =0. A emporal re-sampling is performed for each x-segmen by urning i ino a -segmen of w -pixels (Fig. 6 and Eq. (5)). In his way, super-ime sampling is virually achieved wihin a frame s ime (usually 1/30 second as he uni) such ha each pixel of a ransformed w -pixel slice represens a poin capured in a ime uni of 1/w. Hence, he exure map and he deph map of a layer saring from ime 0 are represened as I(y,k) and v(y,k) where index k for ime is k [ 1 τ 1 v( τ ), v( τ ) + v( )], v( ) = 0, > (6) = 0 τ = For compuing 3D coordinaes, he super-sampled k corresponding o column k is sored in a 1D array as par of he image-based LAMP model: 1 k v( τ ) τ = 0 k = + [, + 1), v( 0) = 0, > 0 v( ) (7) In his manner, each column k in he image-based LAMP array is virually a 1-column perspecive image a he super viewpoin (ime) k. The densiies of viewpoins adapively change wih dephs of he scene. The parallel-perspecive views beween ime and +1 are approximaed by ransforming he corresponding par of he perspecive image in frame. Noe ha his is a correspondenceless approach o implemening view inerpolaion and is differen from he global parameric mehod in [24] or he local mach mehod in [16, 27] for image mosaicing. Fig. 7 shows a seamless adapive-resoluion panoramic mosaic (corresponding o he PVI shown in Fig. 4a), where he ime scale in each insan is deermined by he dominan 16

18 (median) deph of poins along he corresponding verical line (he y direcion) in frame. Noe ha he aspec raios of he objecs in he righ-hand side of Fig. 4a are resored. rees Fig. 7. Muli-viewpoin mosaic wih adapive ime scales (he righ edge of he upper par connecs wih he lef edge of he boom par). The widh of each verical slice from he corresponding original frame is deermined by he dominan image velociy v of pixels along he y-axis in he corresponding PVI (he supporing surface of he base layer). Circles in his figure indicae he corresponding OCCLUDED and SIDE regions ha are shown on he EPIs in Fig. 12. In conclusion, in an image-based 3D LAMP each layer is basically a 2D parallel-perspecive panorama wih an x coordinae (o indicae viewing direcion). I has hree componens: 1) exure map I(y,k); 2) deph map v(y,k); and 3) a 1D super ime-sampling array k =(k) (o indicae densiies of viewpoins, or adapive emporal resoluions). The daa srucure is sruc Image-LAMP-Layer { in I[YDIM][KDIM]; // exure map I[y][]: y = 0~YDIM; k = 0 ~KDIM floa v[ydim][kdim]; //deph map v[y][] : y = 0~YDIM; k = 0 ~KDIM floa [KDIM]; // super-ime index k ; k = 0~KDIM; } Image-LAMP-Layer ilamp[l]; // layer l = 0~L From his represenaion, we can find he corresponding 3D coordinaes of a poin (y,k) in he image-based LAMP by Z V V V = F, Y = y, X = x V k (8) v v v given v(y,k) and k = (k). 17

19 As a comparison, Table 1 shows he radeoff beween relief-like and image-based LAMP represenaion. While he former is more general and more appropriae for complex scenes, he laer is more compac. Table 1. Comparison beween relief-like LAMP and image-based LAMP represenaions Relief-like LAMP Image-based LAMP Scene general, complex scenes well-layered scenes, 2-3 layers Represenaion inhomogeneous homogeneous, more compac Consrucion inermediae level higher level Rendering fas faser 5. DEPTH, OCCLUSION AND RESOLUTION RECOVERY In his secion, we will discuss one approach our panoramic EPI approach [15, 21] - o obain a dense deph map for he supporing surface of he relief-like LAMP. Furher, we will show how o generae occluded layers and adapive resoluion by selecively using he mos essenial informaion from all he EPIs. The suppor surface is a PVI wih seleced parallel viewing direcion; for example, a viewing direcion orhogonal o he camera moion direcion when x=0. The panoramic EPI approach consiss of four imporan seps: locus orienaion deecion, moion/deph boundary localizaion, deph-exure fusion, and occlusion/resoluion recovery. The resuls of he firs hree seps are a panoramic deph map wih accurae deph boundaries as well as he corresponding exure map (Fig. 4a) Deph Recovery: he Panoramic EPI-Based Approach The panoramic EPI approach we proposed in [15, 21] uses a frequency-spaio-emporal crossanalysis o esimae loci s orienaions in each EPI, wihou explicily racking he loci in EPIs. We will give a brief summary of how his goal is achieved in he firs hree seps of he panoramic EPI approach. The mehod consiss of hree seps: frequency domain loci orienaion esimaion, spaio-emporal domain deph boundary localizaion, and deph-exure fusion. Since i is very imporan o accuraely localize he deph boundary for layered represenaion in imagebased rendering applicaions, we will focus on he mehod for accurae deph boundary localizaion. 18

20 Firs, a Gaussian-Fourier Orienaion Deecor (GFOD) is performed along a scanline (x=x 0 ) in he EPI, which is he inersecion line of his EPI wih he supporing PVI. The GFOD operaor uses Gaussian-windowed Fourier ransforms o deec orienaions of he image under he Gaussian window. A large window (e.g ) is used in order o deec accurae locus orienaions. Muliple orienaions are deeced for a cerain emporal range when he GFOD operaor moves across a deph boundary. Thus, he Gaussian window is applied o reduce his range by assigning higher weighs for pixels closer o he cener of he window. However, he response of muliple (wo in our curren implemenaion) orienaions does no only happen exacly a he poin on he deph boundary (see Fig. 8 for locaions wih wo peaks). x (a) θ (b) θ (c) y (d) Fig. 8. Muliple orienaion deecion by Gaussian-windowed Fourier Orienaion Deecor (GFOD), and deph boundary localizaion. (a). An x- image (i.e., EPI, wih y = -18) wih he processed poins (whie dos) and a Gaussian window (indicaed by a circle). (b) Orienaion energy disribuion map Pd(φ,) and loci orienaion deecion: he long dashed curve (red in color version) indicaes he firs seleced peaks, and several pieces of solid lines (blue in color version) indicae he second peaks. (c) Loci orienaion angles seleced afer deph boundary localizaion and deph inerpolaion. (d) Par of he corresponding PVI (x = 0); he horizonal line in he PVI corresponds o he EPI in (a). Significan deph boundaries are marked by verical black lines across (a) o (d). The Fourier ransform G ( ξ, ω) is obained for a Gaussian-windowed EPI paern cenered a (x 0, ) (shown as a circle in Fig. 8a). The energy specrum P ( ξ, ω) =log(1+g 2 ( ξ, ω) ) is mapped ino he polar coordinae sysem ( r,φ ) by a coordinae ransformaion 19

21 2 2 π ξ r = ξ + ω, φ = + arcan. From he resuling polar represenaion P ( r,φ ), an orienaion 2 ω hisogram is consruced as ( r, φ ) drφ [ 0 π ] P ( φ ) 2 P, (9) d = r r1 where φ corresponds o he orienaion angle of he ST exure cenered a (x 0, ) and [r 1,r 2 ] is a frequency range of he bandpass filer, which is seleced adapively according o he spaialemporal resoluion of he image. Iniially, r 1 and r 2 are se o 8 and 30, respecively, for a window. An orienaion energy disribuion map P d (φ,) can be consruced (Fig. 8b), which visually represens he dephs of he poins along he ime () axis, corresponding o he processed EPI. Fig. 9. Panoramic deph consrucion: raw deph map, refined deph map and he exure maps wih deph boundaries superimposed in black lines (red in color version). In he second sep, a Deph Boundary Localizer (DBL) is used o accuraely localize he deph boundary. I measures inensiy consisencies along he wo deeced orienaions wih he wo highes peaks, aking occlusion/reappearance relaions ino accoun. The bes consisen measuremen should be achieved righ a he deph boundary since oherwise one of he measuremens will cross boh locus paerns (Fig. 8 c; see also Fig. 10a and Fig. 10b). Then in he hird sep, a deph-exure fusion (DTF) algorihm is applied o reduce he errors produced in each EPI analysis (due o aperure problems, noise, ec.) and o refine deph 20

22 boundary localizaions. The refinemen is based on he observaion ha a deph boundary almos always coincides wih an inensiy boundary in a visual scene. This observaion is also used by ohers [22, 25]. Fig. 9 compares he raw deph map (afer he firs wo seps) and he refined deph map, and shows superimposed deph boundaries in he panoramic exure map Deph Boundary Localizaion Since muliple orienaions are deeced no only a bu also near he deph/moion boundaries by using he large GFOD operaor, a Deph Boundary Localizer is designed o deermine wheher or no he deph/moion boundary is righ in he cener of he Gaussian window. For he mehod o be valid for mos of he cases encounered in a naural scene and applicable o he EPIs generaed by a un-sabilized camera, we use an approach ha does no rely on locus racking (which ofen fails due o he non-ideal ST exures generaed from a complex scene wih changing illuminaions and un-smooh camera moion). In our algorihm, muliple scale inensiy similariies are measured along all he deeced orienaions θ ( k = 1, L K ) by he GFOD operaor. k, Among hem he orienaion wih he greaes similariy measuremen is seleced as he correc orienaion. Noe ha only a comparison-and-selecion operaion is used, wihou assuming any deecion of feaure poins or using any roublesome hresholds. Consider he case in which wo orienaions θ 1 and θ 2 ( θ 1 > θ2 ) are deeced wihin a Gaussian window. Dissimilariy (i.e., variance) measuremens along θ 1 and θ 2 for a given circular window of radius R cenered a he poin (x 0, 0 ) are defined as he variance of inensiy values (Fig. 10 (a) and (b); refer o Fig. 8) C 1 θ (k=1,2) (10) R ± k k R r= (, R) = I ( ± r, θ ) I ( θ, R) 2 2 where r = ( x x ) + ( ), I (, R) = I( r θ ) ± k 1 θ. R 0 0 ± k ±, k R r= 1 In he above equaions, subscrips + and - denoe he dissimilariy measuremens along he deeced orienaions in posiive (+) and negaive (-) x direcions respecively. This is designed for dealing wih he occlusion of a farher objec ( θ 2 ) by a closer one ( θ 1 ): he occluding (i.e., closer) objec can be seen in boh he posiive and he negaive x direcions, bu he occluded (i.e., farher) objec can only be seen in one of he wo direcions (Fig. 10a). The dissimilariy measuremens for closer and farher objecs are defined as 21

23 E E ( θ, R) = ( C ( θ, R) + C ( θ, R) )/ P ( θ ) 1 ( θ, R) = min( C ( θ, R), C ( θ, R) )/ P ( θ ) d d 1 2 (11) respecively. Noice he difference in he wo measuremens - he occluded objec only akes he smaller measuremen among he posiive and negaive direcions. In addiion, we give more weighs o sronger oriened exure paerns: P d ( θ k ) is he value of he orienaion hisogram (Eq. (9)) a θ k (k=1,2). The higher he value is, he lower he dissimilariy measuremen should be. x C + (θ 1,R) C + (θ 2,R) x C + (θ 1,R) C + (θ 2,R) x C + (θ 1,R) C + (θ 2,R) C - (θ 2,R) (x 0, 0 ) C - (θ 2,R) (x 0, 0 ) C - (θ 2,R) (x 0, 0 ) C - (θ 1,R) C - (θ 1,R) C - (θ 1,R) (a) occlusion: muli-peaks in hree cases o he lef, jus a and o he righ of a deph boundary x C + (θ 1,R) C + (θ 2,R) C - (θ 2,R) (x 0, 0 ) C - (θ 1,R) (x 0, 0 ) x θ 1 θ 2 θ3 R 3 θ 3 R R 2 1 Linear(θ 1, θ 2 ) (b) reappearance (c) muli-scale window (d) deph inerpolaion Fig. 10. Principle of he deph boundary localizaion and deph inerpolaion Fig. 10a shows how o use hese measuremens o localize a deph boundary when he farher objec will be occluded by he closer objec (which is he occlusion case). Muliple peaks are deeced by he GFOD operaor when he Gaussian window (indicaed by circles) is near he deph boundary. When he Gaussian window is o he lef of he deph boundary, he dissimilariy measuremen (i.e. variance) along he locus direcion of he occluding objec E, 1 ( θ R) will be larger, since he measuremen is performed across he loci paern of he o-beoccluded objec (lef of Fig. 10a). On he oher hand, he dissimilariy measuremen along he locus direcion of he o-be-occluded objec E ( θ, R) will be much smaller, since he measuremen is righ along he locus of he o-be-occluded objec. Whenever he cener of he Gaussian window is precisely a he deph boundary, boh measuremens will be small since boh 2 22

24 measuremens are along heir own loci s direcions. However, since he occluding boundary of he closer objec usually will be visually sronger han he ST paern of he occluded objec, he measuremen will be in favor of he closer objec a his locaion (middle of Fig. 10a). As he cener of he window moves ino he occluding (closer) objec, he dissimilariy measuremen of he occluded objec will be significanly increased, since he measuremen will cross he loci of he occluding objec, bu he dissimilariy measuremen for he occluding objec will remain small (righ of Fig. 10a). Similar argumens hold for he reappearance case, when he occluded objec reappears behind he occluding (closer) objec (Fig. 10b). Therefore, a simple verificaion crierion can be expressed as θ1, if E( θ1, R) E( θ2, R) θ = θ 2, Oherwise (12) In fac, he condiion of occlusion and reappearance can be judged eiher by comparing + ( R) and C (, R) C θ 2, θ 2 (see Fig. 10) or by analyzing he conex of he processing (i.e., he change of dephs). In he case of occlusion of a far objec by a near objec (far o near, Fig. 10(a)), we have C ( R) C (, R) ( θ R) C ( R) C + 2, < θ2,. θ 2, + θ2 <, and in reappearance (near o far, Fig. 10(b)) we have In order o handle cases of various objec sizes, differen moion velociies, and muliple objec occlusions, muliple scale dissimilariy measuremens E (, ) θ (e.g., i=1,2,3) are calculaed wihin muliple scale windows of radii R i (i=1,2,3), R 1 <R 2 <R 3. In our experimens, we have seleced R 1 =m/8, R 2 =m/4, R 3 =m/2 (m = 64 is he window size; see Fig. 10(c)). By defining he following raio D max ( E( θ1, Ri ), E( θ2, Ri )) ( E( θ, R ), E( θ, R ) k R i i = (13) min 1 i 2 i a scale p (p=1,2 or 3) wih maximum Dp is seleced for comparing he inensiy similariies. For example, in Fig. 10(c), R 2 will be seleced. The seleced orienaion angle θ can be refined by searching for a minimum dissimilariy measuremen for a small-angle range around θ. The accuracy of he orienaion angle, especially ha of a far objec, can be improved by using more frames. 23

25 In order o obain a dense deph map, inerpolaions are applied o exureless or weak-exured regions/poins where no orienaion can be deeced. The proposed inerpolaion mehod (Fig. 10(d)) is based on he fac ha a deph disconinuiy almos always implies an occluding boundary or shading boundary. The value θ () beween wo insans of ime 1 and 2 wih esimaed orienaion angles θ 1 and θ 2 is linearly inerpolaed for smooh deph change (i.e., θ, T dis is a hreshold), and is seleced as min( θ 1, θ2 ), i.e., he angle of he farher 1 θ2 < Tdis objec, for deph disconinuiy (i.e., θ 1 θ2 Tdis measuremens from a real x- image(epi) are shown in Fig. 8c. ). The processing resuls of dense deph 5.3. Occlusion Recovery Because he supporing PVI only conains informaion from a single viewing direcion (for example, he direcion perpendicular o he moion direcion when we selec a PVI wih x 0 = 0), some pars of he scene ha are visible in oher pars of images from an original (or a sabilized) video sequence are missing due o occlusion. They will be recovered by analyzing deph occlusion relaions in he EPIs. The basic algorihm is performed in each EPI afer he panoramic deph map and is deph boundaries have been obained. The algorihm consiss of he following seps (Fig. 11, Fig. 12): Sep 1. Find he locaion of a deph boundary - A poin on a deph boundary, p0(x 0, 0 ), and orienaion angles (θ 2 and θ 1 ) of he occluded (far) and occluding (near) objecs are encoded in he deph map. A poin is considered as a poin on he deph boundary wih a deph disconinuiy when, for example, o θ 1 θ 2 > 2. Sep 2. Localize he missing par - The missing (occluded) par is represened by a 1D (horizonal) spaio-emporal segmen p s p e in he EPI, on which poins have he same parallel viewing direcions bu from moving viewpoins. I is calculaed from he slopes of he wo orienaion paerns ha have generaed he deph boundary, and i is denoed by an x coordinae and sar/end frames (s/e). Basically, he larges possible angle of viewing direcion (indicaed by he x posiion in he EPI) from he viewing direcion of he PVI (indicaed by he x 0 coordinae) possesses he mos missing informaion, bu possible occlusions by oher nearby objecs should be considered. For example, he second missing region from he righ in Fig. 12b was deermined by checking he occlusion of he locus paerns of he missing par agains hose 24

26 of oher nearby foreground objecs (rees), resuling in an ST segmen wih smaller x coordinae, i.e. smaller viewing angle from he viewing direcion of he PVI. In his way, a riangular region p 0 p s p e can be deermined, and he 1D segmen p s p e will be used as he exure of he missing par ha is occluded by he foreground objecs. x p s p e θ 1 θ 2 p 0 x p 0 p s θ 1 θ 2 p e (a) (b) (c) x p 0 p x p 1 (0,+1) (x,) Fig. 11. Region classificaion and adapive resoluion. (a) Locus paerns near an occlusion boundary; (b) locus paerns of fron- side surfaces and (c) emporal resoluion enhancemen by using spaial resoluion. Sep 3. Verify he ype of he missing par. The riangular region also conains deph informaion of he missing par - he 1D segmen p s p e. In principle, similar reamens can be made here as for he basic deph map as in Eq. (10). For simpliciy, he missing pars are classified ino wo ypes in our curren implemenaion: OCCLUDED and SIDE. If he loci wihin he riangular region form a parallel paern of angle θ 2 (Fig. 11a), hen he missing par is classified as OCCLUDED; oherwise, as SIDE if he angle of he oriened paern θ2 θ1 θ = θ1 + ( s) (14) e s changes linearly inside he riangle (Fig. 11b). By assuming he missing par as each of he wo ypes, an overall consisen measuremen along he hypohesized locus orienaions (indicaed by arrows in Fig. 11a and b) can be calculaed wihin he riangle region (similar o measuremens in moion boundary localizaion [15]). The ype of missing par is seleced as he one wih he beer consisen measuremen of he wo hypoheses. The loci s angle θ of he OCCLUDED region will be he same angle θ 2 as he occluded objec, whereas loci s angle θ of he SIDE region gradually changes from θ 1 o θ 2 (or from θ 2 o θ 1 ), as expressed in Eq. (14). 25

27 In his way, he dephs of he occluded or side region can be assigned. Fig. 11 illusraes he siuaion of reappearance where he farher objec re-appears behind he closer objec. Fig. 12 shows oher siuaions (occlusion and side) wih real image.: Fig 12(a) shows an example of recovering an occluded region (building façade) behind a ree, as indicaed by he firs circle in Fig. 7. Fig 12(b) shows hree recovered side regions. The firs wo correspond o he firs side of he building indicaed by he second circle in Fig. 7, which is separaed ino wo regions by a ree in fron of i. The hird side region corresponds o he second side façade indicaed by he hird circle in Fig. 7. The x locaion of he second side region is much closer o he supporing PVI (wih x = 0), since i is occluded by he ree. (a) p 0 (b) rees p 0 p s p e p s p e OCCLUDED SIDE Fig. 12. Occlusion and resoluion recovery resuls in real EPIs. (a) an OCCLUDED region; (b) hree SIDE regions (wo of hem belong o a side facade separaed by rees). Circles in Fig. 7 show he corresponding OCCLUDED and SIDE regions in his figure Resoluion Recovery As we have shown, he panoramic view image (PVI) in he direcion is under-sampled if he image velociy v of a poin in he PVI is greaer han one pixel per frame (as in he righ hand par of Fig. 4a, where he images of doors of he building look hinner han he real ones). Το enhance emporal resoluion, a v-pixel segmen in he x direcion is exraced from he EPI (as opposed o a single pixel in a PVI, as in Fig. 2 and Fig. 4a). Fig. 11c shows he principle: a nice feaure is ha he fron (relief) poin p x (x,) of an x-segmen p 0 p x a ime will exacly connec wih he back (supporing) poin p 1 (0,+1) of he x-segmen a ime +1 in he relief-like LAMP represenaion, since boh image poins are on he same locus of a 3D poin. This propery has been used o generae seamless, adapive-ime panoramas in Fig. 7. The hickness of he red 26

28 (dark in B_W version) horizonal lines (including hose of SIDE and OCCLUDED regions) in he wo EPI iles in Fig. 12 indicaes he number of poins (pixels) o be exraced in he x direcion of his epipolar plane image. As shown in Fig 7, a seamless panoramic mosaic is generaed afer resoluion enhancemen, where much higher emporal resoluions are achieved in he second par of he mosaic. video (1a) video sabilizaion (1b)PVI & EPI generaion Panoramic exure map (2a) Deph Map Acquisiion EPI 1 Loci orienaion & moion boundary EPI 2 esimaion EPI H H: heigh of a frame Panoramic deph map (2b)Texure-deph fusion (3). Occlusion & resoluion recovery Relief-like LAMP Image-based LAMP Image-based rendering Fig. 13. Sysem diagram of 3D panoramic scene modeling (PVI: Panoramic View image; EPI: Epipolar Plane Image; LAMP: Layered, Adapive-resoluion and Muliperspecive Panorama) 6. LAMP CONSTRUCTION AND RENDERING: EXPERIMENTAL RESULTS D LAMP Consrucion Resuls Our 3D panoramic scene modeling approach consiss of hree modules: (1) video sabilizaion (and PVI/EPI generaion), (2) dense deph map acquisiion (wih exure-deph fusion), and (3) deph layer consrucion (wih occlusion/resoluion recovery). The sysem diagram is shown in Fig. 13. For using he EPI-based approach o recover dense deph maps, our mehod requires a dense, long, 1D ranslaional video sequence as he inpu in order o generae parallel-perspecive panoramic mosaics. I is paricularly effecive when he number of frames is larger han he widh of an original video frame in order o generae mosaics wih panoramic view. In he 27

29 ideal case, he represenaion of parallel projecion requires a pure 1D ranslaion along one image axis. However our algorihms sill work under more pracical moion given ha a video sabilizaion module is used o preprocess he sequence. Our 3D video sabilizaion module esimaes inerframe camera moion, models he camera moion of he image sequence as a 1D dominan ranslaion plus a small vibraion (he laer is modeled by a homography beween successive frames), and removes he vibraion by a moion filering and image warping sep so ha a recified video sequence wih virually 1D ranslaional moion can be creaed. Deailed algorihms and resuls can be found in [15, 21]. The deph map acquisiion module was summarized in Secion 5.1. LAMP represenaions and consrucion are presened in Secions 3 and 4, while he algorihms for occlusion/resoluion recovery has been discussed in Secions 5.2 and 5.3. In his secion, we will presen some consrucion resuls from real image sequences, look a some pracical consideraions in LAMP represenaions, and discuss he LAMP-based 3D rending issues. (a) occlusion (b) afer occlusion recovery Fig. 14. Inernal daa of (a) he base layer, and (b) all he layers of (par of) he LAMP of he MB sequence. The images are runcaed for fiing ino he page. The MB sequence was capured by a camcorder mouned on a ripod carried by a moving vehicle. Aside from he small vibraion of he vehicle on a normal oudoor road, he velociy of ranslaion was mosly consan. The small vibraion in he video sequence (which is demonsraed by he irregular boundaries of he mosaic in Fig. 7) was removed by a video sabilizaion process discussed in our previous work[15, 21]. The frequency-domain deph esimaion mehod is robus for dealing wih he non-ideal loci creaed from his real-world video 28

30 sequence wih vibraion. We have shown he supporing surface and he relief-surface of he relief-like LAMP represenaion consruced from he MB sequence; Fig. 14 shows he inernal daa for he base layer and all he layers. Recall ha a relief-like LAMP represenaion is par of a 3D xy cube. Tha is o say, for each pixel (y,) in he supporing surface, here is an x-segmen of v pixels in he x direcion. The inernal daa of he base layer is displayed in Fig. 14a as he sequenial head-ail arrangemen of all he segmens of he base layer in each row. We have found ha boh he exure and he deph map show almos seamless connecions beween successive segmens, and i is obvious ha higher resoluions han in he corresponding PVI are recovered for closer objecs. The inernal daa of he relief-like LAMP, including all he occluded layers, is displayed in Fig. 14b as similar head-ail arrangemens of all he x-segmens in all he layers, including all he occluded layers as well as he base layer. The x-segmens in he occluded layers are insered ino he locaions of heir corresponding deph boundaries. Comparing Fig. 14b o Fig. 14a, addiional daa for he occluded regions are shown. For example, he porion of he building s facade occluded by he rees and he side facades of he building are parially recovered. Noe ha since boh images are runcaed o he same lengh, leaving ou differen amouns of scene poins on he righ side. The compacness of he LAMP represenaion can be seen from he following key numbers for he MB sequence (W*H*F= widh*heigh*frames, P: number of pixel per locaion in he PVI): - Original video sequence: S 0 = W*H*F=128*128*1204 byes (gray level images) = 16MB - PVI supporing surface (boh exure and deph in 1 bye/pixel): S p =H*F*2 = 128*1024*2 byes = 256KB - LAMP represenaion (exure 1 bye/pixel, deph 4 bye/pixel, as in he x_sreak srucure in Secion 3): MB. The las number approximaely corresponds o P = 3 pixels per locaion and 5 byes per pixel on average in he PVI supporing surface, which coun for boh adapive resoluion and occlusion represenaion. This ends up wih an esimaed LAMP size of 128*1024*5*3 = 1.92MB, which is close o he real size. If we use 1 bye/pixel o represen deph, hen he LAMP size will be S l = H*F*2*P = 128*1024*2*3 = 768 KB, which is simply hree imes ha of he PVI supporing surface. 29

31 In conclusion, he size of he LAMP represenaion in his example is abou 1/10 (if floaing poin deph is saved) or 1/20 (if bye/pixel deph is saved) of he daa size of he original video sequence. In he general case, he LAMP-o-sequence size raio is S l /S o = 2P/W, where W is he widh of each frame in he original video sequence and P is he number of pixels per locaion of he PVI supporing surface Exended Panoramic Image (XPI) Represenaion We have shown in [15] how o make full use of he original image sequence by generaing an exended panoramic image (XPI). Suppose ha an image sequence has F frames of images of size W H. An example is he frequenly used flower garden (FG) sequence(w H F = ). A PVI and an EPI is shown in Fig. 15a and Fig. 15b. I is unforunae in his case ha he field of view of he panoramic view image urns ou o be narrow due o he small number of frames and large inerframe displacemens. Therefore, an exended panoramic image (XPI) is consruced. The XPI is composed of he lef half of frame m/2 (m is he GFOD window size), he PVI par formed by exracing cener verical lines from frame m/2 o frame F-m/2, and he righ half of frame F-m/2 (Fig. 15c). The oal widh of he XPI is W x =W/2 +(F m)+ W/2 = W+F-m. PVI XPI y EPI analysis along his zigzag line EPI occlusion x mxm a b c Fig. 15. PVI, EPI and XPI of he flower garden (FG) sequence Fig. 16 shows he resuls of 3D recovery of he XPI of he FG sequence. In he deph map, he ree runk sands ou disincly from he background, and he gradual deph changes of he 30

32 flowerbed are deeced. The occluded regions and resoluions are recovered in a way similar o ha for a PVI image, excep ha he EPI analysis is performed along a zigzag line (Fig. 15b). a b Fig. 16. Panoramic deph map for he FG sequence. (a) Isomeric deph lines overlaid in he inensiy map (b)panoramic deph map. x-y par -y par (a) background layer d 0.5 k x-y par (b) objec layer Fig. 17. Image-based LAMP model of he FG sequence Fig. 17 shows he wo exraced layers of he image-based LAMP represenaion for he FG sequence, each of which has boh exure and deph maps. These wo pairs of images, along wih a 1D emporal sampling rae array (Eq. (7)) for each layer, are consruced from he 115-frame FG image sequence. In he y par of he background layer of his exended image-based LAMP, he ime-sampling rae is adapively changed according o he dominan deph in each ime insan. The ime scales are less han 0.5 (frames); his means ha higher resoluion is achieved han ha of he original PVI (shown in Fig. 16) in boh exure and deph maps (he uneven 31

33 super-iming in Fig. 17a is due o quanizaion). Furhermore, he background regions occluded by he ree runk have been compleely recovered, and have been merged ino he background layer wih boh exure and deph values. The size of he XPI-based LAMP represenaion in he general case is S xl = (W+(F-m)*P)*H*2*L (byes). where W*H is he size of an original frame, F is he number of frames in he sequence, m is he size of he Gaussian window (m = 64 in our experimens), P is he number of pixels per locaion in he PVI par of he XPI image, and L is he oal number of layers. The real sizes of he imagebased represenaion of he FG sequence is S xl = (114*2 + 95*2) KB = 418 KB. Compared o he size of he original video W H F = = 9.265MB, he real compression raio is The esimaed size using he above equaion is S xl = (352+(115-64)*2)*240*2*2 = 423 KB assuming P = 2 and L = 2. The heoreical LAMP-o-sequence raio is S xl /S o = 2L/F + 2PL/W, assuming ha F is much larger han m LAMP-Based Rendering The 3D LAMP model is capable of synhesizing images from new viewpoins differing from he viewpoins of he original image sequence hanks o is muli-perspecive represenaion, adapive resoluion, occlusion represenaion, and deph informaion. The mapping from a pixel in a relief-like LAMP model o is 3D coordinae can be compued by using Eq. (4), and hen i is sraighforward o re-projec i o he desired view (in pracice an inverse mapping should be applied). Because of he neighborhood relaions of pixels in he LAMP represenaion (refer o Fig. 14), a rendering algorihm can easily perform inerpolaion beween neighborhood pixels. In a relief-like LAMP, an aached layer is always occluded by he layer o which i is aached. So, when rendering he aached layers should be drawn firs so ha an occluded region could be seen correcly in a new view. Thanks o he adapive resoluion represenaion, he higher resoluions in he original images can be applied o a new view whose viewpoin is close o hose of he original image sequence. Fig. 18 shows he preliminary rendering resuls from he relief-like LAMP of he MB sequence. The developmen of a rendering sysem based on relieflike represenaion is sill underway. 32

34 (a) (b) Fig. 18. Rendering resuls from relief-based LAMP represenaion of he MB sequence. (a) Normal rendering (wih correc occlusion checking of differen layers), where he building façade is occluded by he ree. (b) The par of he façade is seen hrough he ree. Noe ha ha par is in shadow in he original video sequence, and herefore i is darker. The rendering process is easier using an image-based LAMP model. For a LAMP model based on an XPI (i.e., a combinaion of x-y and y- images) in general, Eq. (4) and Eq. (8) should be used in he wo x-y pars and he one y- par, respecively. Forunaely, boh equaions are very simple and virually he same (bu reflec he differen ways o obain x and indices) so ha a fas implemenaion is feasible. In order o achieve a correc occlusion relaion, a rendering from a new view should begin wih he farhes layer and end wih he neares layer, from he viewpoin of a synheic image. A simple image based 3D rendering program has been developed o demonsrae he capabiliy of image synhesis of arbirary views using he image-based LAMP represenaion. Synheic sequences wih a virual camera of 6 DOF of moion generaed from he LAMP model (wih and wihou he objec layer) of he flower garden sequence can be found on our web page [20]. The compacness of he image-based LAMP represenaion enables he rendering algorihm o achieve real-ime performance. Fig. 19 shows some snapshos of he synhesis wih boh he background and foreground layers, wihou he foreground layers. The perspecive disorion shown a he borders of he synheic images in he second and he hird rows in Fig. 19 clearly reveals he accuracy in he recovered deph changes for he ree, house, and garden. The firs synheic images in boh he second and he hird rows were generaed from a viewpoin farher from he original camera pah so ha he enire scene can be seen. 33

35 Fig. 19. Synheic images from he image-based LAMP model. Firs row: Snapshos of original video sequence; second row: snapshos of he virual walk-hrough wih boh he background and he rees; hird row: snapshos of synheic resuls wihou he ree. The firs synheic images in he 2nd and 3rd rows are generaed from viewpoin ha can see all he scene poins in he LAMP model. 7. COMPARISONS AND DISCUSSIONS 7.1. Layered Represenaion The LAMP represenaion is relaed o such represenaions as muli-perspecive panoramic view images (PVIs), spries, and layered deph images (LDIs). However, i is more han a muliperspecive PVI in ha deph, adapive-resoluion and occlusion are added. I is differen from a "sprie" (or LDI) since he sprie or LDI is a view of a scene from a single inpu camera view and is wihou adapive image resoluion. We compare our resuls [17] wih oher layered represenaions [10, 11, 13, 22, 23]. A comparison in number of inpu images, resuls of layered represenaions, and algorihm performance for he flower garden sequence is summarized in Table 2. Usually, in a layered represenaion a se of deph surfaces is firs esimaed from an image sequence of a moving camera and hen combined o generae a new view. The layered represenaion proposed in [10] consiss of hree maps in each layer: a mosaicing inensiy map, 34

36 an alpha map, and a velociy map. The velociy map is acually a se of parameers of he affine ransformaion beween layers. Occlusion boundaries are represened as disconinuiies in a layer's alpha map (opaciy). In [10], he firs 30 frames of he flower garden sequence were used as inpu. The oupu were hree affine plane layers - he ree, he house, and he flowerbed. Occlusion (by he ree) was recovered, bu no deph informaion was provided (i.e., each layer is only modeled as an affine plane). Processing of he 30 frames of 720*480 images ook 40 minues in a HP 9000 series 700 worksaion. Table 2. Comparison of 4 layered represenaion resuls for he flower garden sequence Wang-Adelson [10] Ke-Kanade [22] Sawhney-Ayer Baker-Szeliski- Image-based LAMP [11] Anandan[12] Inpu firs 30 frames (720x480) wo frames a few frames firs 9 even frames all 115 frames (350x240) Rep. 3 planar layers (ree, flower bed, house) 4 planar layers (ree, branch, 4 layers (ree, flower bed, 6 layers of Spries (3 ree, 2 flower 2 layers of LAMP (ree, background ) house, flowerbed) house, sky ) bed, 1 house ) Deph no no no yes yes, each obained from 64 frames Occlusion recovered no recovered no recovered recovered recovered Muli-view affine mosaic rep. 2D layer rep. single view single view mosaic muliple-view mosaic mosaic Adapive res. no no no a perspecive image adapive resoluion Performance 40 mins / 30 frames in a HP 9000 series 700 worksaion no available no available no available 14 mins / 115 frames in a 400 MHz PC (general C code) The subspace approach [22] is an effecive mehod for 2D layer exracion from an uncalibraed image sequence. For he flower garden sequence, four planar layers, which roughly correspond o ree, branch, house, and flowerbed, are exraced from wo frames in he sequence. Deph boundaries are accurae afer he layer refinemen sep using color segmenaion resuls (similar o our deph-exure fusion sep). However, deph informaion of each layer is no obained since hey are 2D layers, and he occluded regions are no recovered since only wo frames are used. The muliple moion esimaion mehod based on MDL and EM algorihms in [11] is compuaionally expensive. I requires a combinaorial search o deermine he correc number of layers and he "projecive deph" of each poin in a layer. For he flower garden sequence, only a few frames were processed, and a six parameric moion model was used o segmen he scene 35

37 ino four layers - ree, house, flowerbed and sky. Occlusion regions are no recovered in he layered model, and deph was no esimaed in he flower garden example. In he sprie represenaion [12], each layer consiss of an explici 3D plane equaion, a exure map (a sprie), and a map wih deph offse relaive o he plane. For he flower garden sequence, eigh-parameer homography was used o fi he planar layers. The firs nine even frames were used in he experimen, and he resul was six layers of spries - hree for he ree, wo for he flowerbed, and one for he house. Deph informaion was aached o each pixel, and he occlusion (by he ree) was recovered. Each layer was a single perspecive view mosaic. More recen work on 3D layer exracion by he same research group [23] uses an inegraed Bayesian approach o auomaically deermine he number of layers and he assignmen of individual pixels o layers. The approach is a very general one. However he segmenaion resuls for he flower garden sequence are no as saisfacory as he resuls in [12]. As he auhors poined ou in heir paper, here was always a danger in choosing he prior in order o obain a desired resul of layering, and his remains a challenging research issue. For he flower garden sequence, our mehod segmens he scene ino wo complee layers of he image-based LAMP represenaion: he ree and he background. Each layer has boh a exure map and a dense deph map. The segmenaion is very accurae along he deph boundaries, he occluded regions in he seleced panoramic view are recovered from oher views, and he images have adapive resoluions. Our algorihms can process 115 frames of 350*240 images in 14 minues on a 400 MHz PC o obain he final LAMP model. The processing ime reduces o 2.6 minues on a Xeon 2.4 GHz dual-cpu Dell Linux worksaion. The ime includes reading all of he 240 EPIs from a SCSI hard disk and displaying he processed EPIs and resuling images on he screen Full Perspecive, Full Orhogonal, Muli-Perspecive and Mulivalued Represenaions There are a leas hree exising geomeric represenaions for large image sequences, based on parallel projecion (i.e., a exured digial elevaion map), single-view perspecive mosaicing (e.g. sprie), and parallel-perspecive panorama (i.e., PVI). A parallel projecion of he panoramic deph in Fig. 4a of he MB sequence is shown in Fig. 20a ha correcly shows he aspec raio of deph and size (heigh and widh) of he scene. However, he resoluions of he nearer objecs are significanly decreased (or los) because of his evenly sampled represenaion. 36

38 A single view perspecive represenaion, on he oher hand, is no a good represenaion mehod for surfaces of varying orienaions and over a large disance (Fig. 20b). A very small deph esimaion error in he single-view-based represenaion could be grealy enlarged when reprojecing o a mosaiced image of a single referenced viewpoin. (a) (b) Fig. 20. Two geomeric represenaions of he 3D panorama of he MB sequence. (a) Parallel projecion (b) Perspecive projecion from a viewpoin a he cener of he pah where he video was capured A parallel-perspecive panoramic view image (PVI) is a compac image-based represenaion ha beer represens he image daa capured by a camera ranslaing over a long disance. Based on he muli-perspecive panorama, we have proposed a sill more compac and more powerful represenaion - 3D layered, adapive-resoluion and muli-perspecive panorama (LAMP). To our knowledge, his seems o be he firs piece of work ha inegraes muli-perspecive panoramas and layered represenaions wih adapive image resoluions in a unified model. The relief-like LAMP is basically a single exended muli-perspecive panoramic view image (PVI) wih boh exure and deph values, bu each pixel has muliple values o represen resuls of occlusion recovery and adapive resoluion enhancemen. The image-based LAMP, on he oher hand, consiss of a se of muli-perspecive layers, each of which has boh exure and deph maps wih adapive densiies of viewpoins depending on dephs of scene poins. No assumpion is made on he srucures of a scene in consrucing is 3D LAMP represenaions. The LAMP represenaions are effecive for image-based rendering. We have noiced ha our LAMP represenaions are very similar o he muli-valued represenaion (MVR) proposed by Chang and Zakhor [26] in ha (1) dense dephs are 37