A Layered Approach to Stereo Reconstruction

Transcription

1 Appeared in the 1998 Conference on Compter Vision and Pattern Recognition, Santa Barbara, CA, Jne A Layered Approach to Stereo Reconstrction Simon Baker Richard Szeliski and P. Anandan Department of Compter Science Microsoft Research Colmbia Uniersity Microsoft Corporation New York, NY Redmond, WA Abstract We propose a framework for extracting strctre from stereo which represents the scene as a collection of approximately planar layers. Each layer consists of an explicit 3D plane eqation, a colored image with per-pixel opacity (a sprite), and a per-pixel depth offset relatie to the plane. Initial estimates of the layers are recoered sing techniqes taken from parametric motion estimation. hese initial estimates are then refined sing a re-synthesis algorithm which takes into accont both occlsions and mixed pixels. Reasoning abot sch effects allows the recoery of depth and color information with high accracy, een in partially occlded regions. Another important benefit of or framework is that the otpt consists of a collection of approximately planar regions, a representation which is far more appropriate than a dense depth map for many applications sch as rendering and ideo parsing. 1 Introdction Althogh extracting scene strctre sing stereo has long been an actie area of research, the recoery of accrate depth information still remains largely nsoled. Most existing algorithms work well when matching featre points or highly textred regions, bt perform poorly arond occlsion bondaries and in ntextred regions. A common element of many recent attempts to sole these problems is explicit modeling of the 3D olme of the scene [37, 13, 8, 25, 26, 30]. he scene olme is discretized, often in terms of eqal increments of disparity, rather than into eqally sized oxels. he goal is then to find the oxels which lie on the srfaces of the objects in the scene. he major benefits of sch approaches inclde, the eqal and efficient treatment of a large nmber of images [8], the possibility of modeling occlsions [13], and the detection of mixed pixels at occlsion bondaries to obtain sb-pixel accracy [30]. Unfortnately, discretizing space olmetrically introdces a hge nmber of degrees of freedom, and leads to sampling and aliasing artifacts. Another actie area of research is the detection of parametric motions within image seqences [1, 34, 12, 9, 15, 14, 24, 5, 36, 35]. Here, the goal is the decomposition of he research described in this paper was condcted while the first athor was a smmer intern at Microsoft Research. the images into sb-images, commonly referred to as layers, sch that the pixels within each layer moe in a manner consistent with a parametric transformation. he motion of each layer is determined by the ales of the parameters. An important transformation is the 8 parameter homography (collineation), becase it describes the motion of a rigid planar patch as either it or the camera moes [11]. While existing techniqes hae been sccessfl in detecting mltiple independent motions, layer extraction for scene modeling has not been flly deeloped. One fact which has not been exploited is that, when simltaneosly imaged by seeral cameras, each of the layers implicitly lies on a fixed plane in the 3D world. Another omission is the proper treatment of transparency. With a few exceptions (e.g. [27, 3, 18]), the decomposition of an image into layers that are partially transparent has not been attempted. In contrast, scene modeling sing mltiple partially transparent layers is common in the graphics commnity [22, 6]. In this paper, we present a framework for reconstrcting a scene as a collection of approximately planar layers. Each of the layers has an explicit 3D plane eqation and is recoered as a sprite, i.e. a colored image with per-pixel opacity (transparency) [22, 6, 32, 20]. o model a wider range of scenes, a per-pixel depth offset relatie to the plane is also added. Recoery of the layers begins with the iteration of seeral steps based on techniqes deeloped for parametric motion estimation, image registration, and mosaicing. he reslting layer estimates are then refined sing a resynthesis step which takes into accont both occlsions and mixed pixels in a similar manner to [30]. Or layered approach to stereo shares many of the adantages of the aforementioned olmetric techniqes. In addition, it offers a nmber of other adantages: he combination of the global model (the plane) with the local correction to it (the per-pixel depth offset) reslts in ery robst performance. In this respect, the framework is similar to the plane + parallax work of [19, 23, 29], the model-based stereo work of [10], and the parametric motion + residal optical flow of [12]. he otpt (a collection of approximately planar regions) is more sitable than a discrete collection of oxels for many applications, inclding, rendering [10] and ideo parsing [15, 24]. 434

2 y Layer sprite L 1 Plane n 1 x = 0 Residal depth Z 1 Layer sprite L 2 Plane n 2 x = 0 Residal depth Z 2 Inpt: Images I k and cameras Pk Initialize Layers (Section 2.1) z x Estimate Plane Eqations n l (Section 2.2) Image I k Camera Matrix P k Boolean Opacities Estimate Layer Sprites L l (Section 2.3) Iteration Estimate Residal Depth Z l (Section 2.4) Boolean Mask B k 1 Masked Image M k 1 Boolean Mask B k 2 Masked Image M k 2 Figre 1: Sppose K images I k are captred by K cameras P k. We assme that the scene can be represented by L sprite images L l on planes n l x =0with depth offsets Z l. he boolean masks B kl denote the pixels in image I k from layer L l and the masked images are gien by M kl = B kl I k. 1.1 Basic Concepts and Notation We se homogeneos coordinates for both 3D world coordinates x =(x, y, z, 1) and for 2D image coordinates =(,, 1). he basic concepts of or framework are illstrated in Figre 1. We assme that the inpt consists of K images I 1 ( 1 ),I 2 ( 2 ),...,I K ( K ) captred by K cameras with known projection matrices P 1, P 2,...,P K. (In what follows, we drop the image coordinates k nless they are needed to explain a warping operation explicitly.) We wish to reconstrct the world as a collection of L approximately planar layers. Following [6], we denote a layer sprite with pre-mltiplied opacities by: L l ( l ) = (α l r l,α l g l,α l b l,α l ) (1) where r l = r l ( l ) is the red band, g l = g l ( l ) is the green band, b l = b l ( l ) is the ble band, and α l = α l ( l ) is the opacity. We also associate a homogeneos ector n l with each layer (which defines the plane eqation of the layer ia n l x =0) and a per-pixel residal depth offset Z l( l ). Or goal is to estimate the layer sprites L l, the plane ectors n l, and the residal depths Z l. o do so, we wish to se techniqes for parametric motion estimation. Unfortnately, most sch techniqes assme boolean-aled opacities α l (i.e., niqe layer assignments). We therefore split or framework into two parts. In the first part, de- Real Valed Opacities Assign Pixels to Layers B kl (Section 2.5) Refine Layer Sprites L l (Section 3) Otpt: n l, L l, and Z l Figre 2: We wish to compte the layer sprites L l, the layer plane ectors n l, the residal parallax Z l, and the boolean mask images B kl. After initializing, we iteratiely compte each qantity in trn fixing the others. Finally, we refine the layer sprite estimates sing a re-synthesis algorithm. scribed in Section 2, we assme boolean opacities to get a first approximation to the strctre of the scene. If the opacities are boolean, each point in each image I k is only the image of a point on one of the layers L l. We therefore introdce boolean masks B kl which denote the pixels in image I k that are images of points on layer L l. So, in addition to L l, n l,andz l, we also need to estimate the masks B kl. Once we hae estimates of the masks, we immediately compte masked inpt images M kl = B kl I k (see Figre 1). In the second part of or framework, we se the initial estimates of the layers made by the first part as inpt into a re-synthesis algorithm which refines the layer sprites L l, inclding the opacities α l. his second step reqires a generatie or forward model of the image formation processandisdiscssedinsection3. In Figre 2 we illstrate the processing steps of the framework. Gien any three of L l, n l, Z l,andb kl,there are techniqes for estimating the remaining one. he first part of or framework therefore consists of first initializing these for qantities, and then iteratiely estimating each one while fixing the other three. After good initial estimates of the layers are obtained, we moe on to the second part of the framework in which we se real aled opacities and refine the entire layer sprites, inclding the opacities. 435

3 2 Initial Comptation of the Layers 2.1 Initialization of the Layers Initialization of the layers is a difficlt task, which is ineitably somewhat ad-hoc. A nmber of approaches hae been proposed in the parametric motion literatre: Randomly initialize a large nmber of small layers, which grow and merge ntil a small nmber of layers remain which accrately model the scene [34, 24, 5]. Iteratiely apply dominant motion extraction [15, 24], at each step applying the algorithm to the residal regions of the preios step. Perform a color segmentation in each image, match the segments, and se as the initial assignment [2]. Apply a simple stereo algorithm to get an approximate depth map, and then fit planes to the depth map. Get a hman to initialize the layers. (In many applications, sch as model acqisition [10] and ideo parsing [24], the goal is a semi-atomatic algorithm and limited ser inpt is acceptable.) In this paper, we assme a hman has initialized the layers. As discssed in Section 5, flly atomating the framework is left as ftre work. 2.2 Estimation of the Plane Eqations o compte the plane eqation ector n l we need to map the pixels in the masked images M kl onto the plane defined by n l x =0.Ifxisa3Dworld coordinate of a point and k is the image of x in camera P k,wehae: k = P k x (2) where eqality is in the 2D projectie space P 2 [11]. Since P k is of rank 3, it follows that: x = P k k + sp k (3) where P k = P k (P kp k ) 1 is the psedo-inerse of the camera matrix P k, s is an nknown scalar, and p k is a ector in the nll space of P k,(i.e. P k p k =0). If x lies on the plane n l x =0we hae: n l P k k + sn l p k = 0. (4) Soling this eqation for s, sbstitting into Eqation (3), and rearranging yields: x = ( (n l p k )I p k n ) l P k k. (5) he importance of Eqation (5) is that it allows s to map a pixel coordinate k in image M kl onto the point on the plane n l x =0, of which it is the image. So, we can now map this point onto its image in another camera P k : ( k = P k (n l p k )I p k n ) l P k k H l kk k (6) where H l kk is a homography (collineation of P2 [11]). Eqation (6) describes the mapping between the two images which wold hold if the pixels were all images of points on the plane n l x =0. Using this relation, we can warp all of the masked images onto the coordinate frame of one distingished image 1 (w.l.o.g. image M 1l ) as follows: ( ) ( ) H l 1k M kl (1 ) M kl H l 1k 1. (7) Here, H l 1k M kl is the masked image M kl warped into the coordinate frame of M 1l. he property which we se to compte n l is that, assming the pixel assignments to the layers B kl are correct, the world is piecewise planar, and the srfaces are Lambertian, the warped images H l 1k M kl shold agree with each other where they oerlap. here are a nmber of fnctions which can be sed to measre the degree of consistency between the warped images, inclding least sqares [4] and robst measres [9, 24]. In both cases, the goal is the same: find the plane eqation ector n l which maximizes the degree of consistency. ypically, this extremm is fond sing some form of gradient decent, sch as the Gass-Newton method, and the optimization is performed in a hierarchical (i.e. pyramid based) fashion to aoid local extrema [4]. o apply this standard approach [31], we simply need to derie the Jacobian of the image warp H l 1k with respect to the parameters of n l. his is straightforward from Eqation (6) becase we know the cameras matrices P k. 2.3 Estimation of the Layer Sprites Before we can compte the layer sprites L l, we need to choose 2D coordinate systems for the planes. Sch coordinate systems can be specified by a collection of arbitrary (rank 3) camera matrices Q l. 2 hen, similarly to Eqations (5) and (6), we can show that the image coordinates k of the pixel in image M kl which is projected onto the pixel l on the plane n l x =0isgien by: ( k = P k (n l q l )I q l n ) l Q l l H l k k (8) where Q l is the psedo-inerse of Q l,andq l is a ector in the nll space of Q l. he homography H l k can be sed to warp the image M kl forward onto the coordinate frame of the plane n l x =0, the reslt of which is denoted Hl k M kl. hen, we can estimate the layer sprite (with boolean opacities) by blending the warped images: K L l = H l k M kl (9) k=1 1 It is possible to add an extra 2D perspectie coordinate transformation here. Sppose H is an arbitrary homography. We cold warp each masked image onto H H l 1k M kl( 1 ) M kl (HH l 1k 1). he addition of the homography H can be sed to remoe the dependence on one distingished image, as adocated by Collins [8]. 2 A sitable choice for Q l wold be one of the camera matrices P k, in which case Eqation (8) redces to Eqation (6). Another interesting choice is one in which the nll space of Q l is perpendiclar to the plane defined by n l, and the psedo-inerse maps the coordinate axes onto perpendiclar ectors in the plane (i.e. a camera with a frontal image plane). Note that often we do not want a fronto-parallel camera, since it may nnecessarily warp the inpt images. 436

4 where is the blending operator. here are a nmber of ways in which blending cold be performed. One simple method wold be to take the mean of the color ales. A refinement wold be to se a feathering algorithm sch as [28], where the aerage is weighted by the distance of each pixel from the nearest inisible pixel (i.e. α =0)inM kl. Alternatiely, robst techniqes cold be sed to estimate L l. he simplest sch example is the median operator, bt more sophisticated alternaties exist. An nfortnate effect of the blending in Eqation (9) is that aeraging tends to increase image blr. Part of the case is non-planarity in the scene (which is modeled in Section 2.4), bt image noise and resampling error also contribte. One simple method of compensating for this effect is to deghost the sprites [28]. Another soltion is to se image enhancement techniqes sch as [16, 21, 7], which can een be sed to obtain sper-resoltion sprites. 2.4 Estimation of the Residal Depth In general, the scene will not be piecewise planar. o model any non-planarity, we allow the point l on the plane n l x =0to be displaced slightly. We assme it is displaced in the direction of the ray throgh l defined by the camera matrix Q l. he distance it is displaced is denoted by Z l ( l ), as measred in the direction normal to the plane. In this case, the homographic warps sed in the preios section are not applicable, bt sing a similar argment, it is possible to show (see also [19, 23]) that: k = H l k l + w( l )Z l ( l )t kl (10) where H l k = P ( ) k (n l q l )I q l n l Q l is the planar homography of Section 2.3, t kl = P k q l is the epipole, and it is assmed that the ector n l =(n x,n y,n z,n d ) has been normalized sch that n 2 x + n2 y + n2 z =1. he term w( l) is a projectie scaling factor which eqals the reciprocal of Q 3 l x,whereq3 l is the third row of Q l and x is the world coordinate of the point. It is possible to write w( l ) as a linear fnction of the image coordinates l, bt the dependence on Q l and n l is qite complicated and so the details are omitted. Eqation (10) can be sed to map plane coordinates l backwards to image coordinates k,ortomap the image M kl forwards onto the plane. We denote the reslt of this warp by (H l k, t kl,z l ) M kl, or more concisely Wk l M kl. Almost any stereo algorithm cold be sed to compte Z l ( l ), althogh it wold be preferable to se one faoring small disparities. Doing so essentially soles a simpler (or what Debeec et. al [10] term a model-based) stereo problem. o compte the residal depth map, we initially set Z l ( l ) to be the ale (in a range close to zero) which minimizes the ariance of (H l k, t kl,z l ) M kl across k. Afterwards a simple smoothing algorithm is applied to Z l ( l ). Once the residal depth offsets hae been estimated, the layer sprite images shold be re-estimated sing: L l = K (H l k, t kl,z l ) M kl = k=1 K Wk l M kl (11) k=1 rather than Eqation (9). 2.5 Pixel Assignment to the Layers he basis for the comptation of the pixel assignments is a comparison of the warped images Wk l M kl with the layer sprites L l. 3 If the pixel assignment was correct (and neglecting resampling isses) these images shold be identical where they oerlap. Unfortnately, comparing these images does not yield any information otside the crrent estimates of the masked regions. o allow the pixel assignments to grow, we take the old estimates of B kl and enlarge them by a few pixels to yield new estimates B kl. hese new assignments can be compted by iterating simple morphological operations, sch as setting B kl = 1 for the neighbors of eery pixel for which B kl =1. Enlarged masked images are then compted sing: M kl = B kl I k (12) and a new estimate of the layer sprite compted sing: L l = K Wk l M kl. (13) k=1 (Here, Z l is enlarged in Wk l so that it declines to zero smoothly otside the old masked region.) One small danger of working with M kl and L l is that occlded pixels may be blended together with nocclded pixels and reslt in poor estimates of the L l. A partial soltion to this problem is to se a robst blending operator sch as the median. Another part of the soltion is, dring the blend, weight pixels for which B kl =1more than those for which B kl =0(and B kl =1). he weights shold depend on the distance of the pixel from the closest pixel for which B kl =1, in a similar manner to the feathering algorithm of [28]. Gien L l, or approach to pixel assignment is as follows. We first compte a measre P kl ( l ) of the likelihood that a pixel in Wk l M kl ( l ) is the warped image of the pixel l in the enlarged sprite L l. here are a nmber of ways of defining P kl. Perhaps the simplest is the residal intensity difference [24]: P kl = L l W l k M kl. (14) Another is the magnitde of the residal normal flow: P kl = L l Wk l M kl. (15) L l 3 Alternatiely, we cold compare the inpt images I k with the layer sprite images warped back onto image coordinates (W l k ) 1 L l.his means comparing the inpt image with a twice resampled, blended image. Both blending and resampling tend to increase blr, so, een if the pixel assignment was perfect, these images may well differ sbstantially. 437

5 Locally estimated ariants of the residal normal flow hae been sed by Irani and coworkers [16, 17, 15]. A final possibility wold be to compte the optical flow between Wk l M kl and L l. hen a decreasing fnction of the magnitde of the flow cold be sed for P kl. Next, P kl is warped back into the coordinate system of the inpt image I k to yield: ˆP kl = (Wk l ) 1 P kl. (16) his warping tends to blr P kl, bt this is acceptable since we will want to smooth the pixel assignment anyway. 4 he new pixel assignment can then be compted by choosing the best possible layer for each pixel: { 1 if ˆPkl ( B kl ( k ) = k ) = min l ˆPkl ( k ) (17) 0 otherwise. 3 Layer Refinement by Re-Synthesis In this section, we describe how the estimates of the layer sprites can be refined, now assming that their opacities α l are real aled. We begin by formlating a generatie model of the image formation process. Afterwards, we propose a measre of how well the layers re-synthesize the inpt images, and show how the re-synthesis error can be minimized to refine the estimates of the layer sprites. 3.1 he Image Formation Process We formlate the generatie (forward) model of the image formation process sing image compositing operations [6], i.e. by painting the sprites one oer another in a back-to-front order. he basic operator sed to oerlay the spritesistheoer operator: F B F +(1 α F )B, (18) where F and B are the foregrond and backgrond sprites, and α F is the opacity of the foregrond [22, 6]. his definition of the oer operator assmes pre-mltiplied opacities, as in Eqation (1). he generatie model consists of the following two steps: 1. Using the camera matrices, plane eqations, and residal depths, warp each layer backwards onto the coordinate frame of image I k sing the inerse of the operator in Section 2.4. his yields the n-warped sprite: U kl = (Wk) l 1 L l. (19) Note that the opacities shold be warped along with the color ales [6]. 4 We may want to smooth P kl een more, e.g. sing an isotropic smoother sch as a Gassian. Other alternaties inclde, (1) performing a color segmentation of each inpt image and only smoothing within each segment in a similar manner to [2], and (2) smoothing P kl less in the direction of the intensity gradient since strong gradients often coincide with depth discontinities and hence layer bondaries. 2. Composite the n-warped sprites in back-to-front order (which can be compted from the plane eqations): S k = L U kl = U k1 U kl (20) l=1 to obtain the synthesized image S k.ifwehaesoled the stereo reconstrction problem, and neglecting resampling isses, S k shold match the inpt I k. his last step can be re-written as three simpler steps: 2a. Compte the isibility of each n-warped sprite [30]: V kl = V k(l 1) (1 α k(l 1) )= l 1 (1 α kl ) (21) l =1 where α kl is the alpha channel of U kl,andv k1 =1. 2b. Compte the masked images, M kl = V kl U kl. 2c. Sm p the masked images, S k = L l=1 M kl. In these last three sbsteps, the isibility map makes the contribtion of each sprite pixel to the image S k explicit. 3.2 Minimization of Re-Synthesis Error As mentioned aboe, if the layer estimates are accrate, the synthesized image S k shold be ery similar to the inpt image I k. herefore, we refine the layer estimates by minimizing the prediction error: C = S k ( k ) I k ( k ) 2 (22) k k sing a gradient descent algorithm. (In order to frther constrain the space of possible soltions, we can add smoothness constraints on the colors and opacities [30].) Rather than trying to optimize oer all of the parameters (L l, n l, and Z l ) simltaneosly, we only adjst the sprite colors and opacities in L l, and then re-rn the preios motion estimation steps to adjst n l and Z l (see Figre 2 and Section 2). he deriaties of the cost fnction C with respect to the colors and opacities in L l ( l ) can be compted sing the chain rle [30]. In more detail, the isibility map V kl mediates the interaction between the n-warped sprite U kl and the synthesized image S k, and is itself a fnction of the opacities in the n-warped sprites U kl. For a fixed warping fnction W l k, the pixels in U kl are linear combinations of the pixels in sprite L l. his dependence can either be exploited directly sing the chain rle to propagate gradients, or alternatiely the deriaties of C with respect to U kl can be warped back into the reference frame of L l [30]. 438

6 (a) (b) (c) (d) (e) (f) (g) (h) Figre 3: Reslts on the flower garden seqence: (a) first and (b) last inpt images; (c) initial segmentation into six layers; (d) and (e) the six layer sprites; (f) depth map for planar sprites (darker denotes closer); front layer before (g) and after (h) residal depth estimation. (a) (b) (c) (d) (e) (f) Figre 4: Reslts on the symposim seqence: (a) third of fie images; (b) initial segmentation into six layers; (c) recoered depth map (darker denotes closer); (d) and (e) the fie layer sprites; (f) residal depth image for fifth layer. 439

7 (a) (b) (c) Figre 5: 3D iews of the reconstrcted symposim scene: (a) re-synthesized third image (note extended field of iew). (b) noel iew withot residal depth; (c) noel iew with residal depth (note the ronding of the people). 4 Experiments o alidate or approach, we experimented on two mlti-frame data sets. he first of these data sets is a standard motion seqence of a scene containing no independently moing objects. he second consists of 40 images taken simltaneosly. he camera geometry is not gien for either seqence, so we sed point tracking and a standard strctre from motion algorithm to estimate the camera matrices. Or experiments do not yet inclde the reslts of applying the layer refinement step described Section 3. o initialize or algorithm, we first decided how many layers were reqired, and then performed a rogh assignment of pixels to layers by hand. Varios atomated techniqes for performing this initial labeling are described in Section 2.1. Next, the atomatic hierarchical parametric motion estimation algorithm described in [31] was sed to find the 8-parameter homographies between the layers and estimate the layer sprites. (For the experiments presented in this paper, we set Q l = P 1, i.e. we reconstrcted the sprites in the coordinate system of the first camera.) Using the compted homographies, we fond the best plane estimate for each layer sing a Eclidean strctre from motion algorithm [33]. he reslts of applying these steps to the MPEG flower garden seqence are shown in Figre 3. Figres 3(a) and (b) show the first and last image in the sbseqence we sed (the first nine een images). Figre 3(c) shows the initial pixel labeling into seen layers. Figres 3(d) and (e) show the sprite images corresponding to each of the seen layers, re-arranged for more compact display. (hese sprites are actally the ones compted after residal depth estimation.) Note that becase of the blending that takes place dring sprite constrction, each sprite is larger than its footprint in any one of the inpt images. Figre 3(f) shows a depth map compted by painting eery pixel with its corresponding grey coded Z ale, where darker denotes closer. Once we hae recoered the initial geometric strctre, we recompte the homographies by directly adjsting the plane eqations, as described in Section 2.2. We then rn the the residal depth estimation algorithm described in Section 2.4 and recompte the sprites. Since the correspondence is now mch better across images, the reslting sprites are mch less blrry. Figre 3(g) shows the original sprite obtained for the lower flower bed, while Figre 3(h) shows the same sprite after residal depth estimation. Or second set of experiments ses fie images of a 40- image stereo data set taken at a graphics symposim. Figre 4(a) shows the middle inpt image, Figre 4(b) shows the initial pixel assignment to layers, Figre 4(c) shows the recoered planar depth map, and Figre 4(f) shows the residal depth map for one of the layers. Figres 4(d) and (e) show the recoered sprites. Figre 5(a) shows the middle image re-synthesized from these sprites. Finally, Figres 5(b c) show the same sprite collection seen from a noel iewpoint (well otside the range of the original iews), first with and then withot residal depth correction. he gaps in Figre 5 correspond to parts of the scene which where not isible in any of the fie inpt images. 5 Discssion We hae presented a framework for stereo reconstrction which represents the scene as a collection of approximately planar layers. Each layer consists of a plane eqation, a layer sprite image, and a residal depth map. he framework exploits the fact that each layer implicitly lies on a fixed plane in the 3D world. herefore, we only need to recoer three plane parameters per layer, independently of the nmber of images. We also showed how an initial estimate of the scene strctre allows s to reason abot image formation. We proposed a forward model of image formation, and deried a measre of how well the layers re-synthesize the inpt images. Optimizing this measre allows the layer sprites to be refined, and their opacities estimated. Or initial reslts are ery encoraging, howeer frther work is reqired to complete an implementation of the entire framework. In particlar, we are crrently implementing the layer refinement algorithm described in Section 3. Other areas which we are exploring inclde atomatic initialization of the layers and more sophisticated pixel assignment strategies. 440

8 Acknowledgements We wold like to thank Michael Cohen, Mei Han, and Jonathan Shade for fritfl discssions, Harry Shm for his implementation of the residal depth estimation algorithm, and the anonymos reiewers for many sefl sggestions and comments. References [1] G. Adi. Determining three-dimensional motion and strctre from optical flow generated by seeral moing objects. PAMI, 17(4): , [2] S. Ayer, P. Schroeter, and J. Bigün. Segmentation of moing objects by robst parameter estimation oer mltiple frames. In 3rd ECCV, pages , [3] J. R. Bergen, P. J. Brt, R. Hingorani, and S. Peleg. A threeframe algorithm for estimating two-component image motion. PAMI, 14(9): , [4] J.R. Bergen, P. Anandan, K.J. Hanna, and R. Hingorani. Hierarchical model-based motion estimation. In 2nd ECCV, pages , [5] M.J. Black and A.D. Jepson. Estimating optical flow in segmented images sing ariable-order parametric models with local deformations. PAMI, 18(10): , [6] J.F. Blinn. Jim Blinn s corner: Compositing, part 1: heory. IEEE Compter Graphics and Applications, 14(5):83 87, September [7] M.-C. Chiang and.e. Bolt. Local blr estimation and sper-resoltion. In CVPR 97, pages , [8] R.. Collins. A space-sweep approach to tre mlti-image matching. In CVPR 96, pages , [9]. Darrell and A.P. Pentland. Cooperatie robst estimation sing layers of spport. PAMI, 17(5): , [10] P.E. Debeec, C.J. aylor, and J. Malik. Modeling and rendering architectre from photographs: A hybrid geometryand image-based approach. In SIGGRAPH 96, pages 11 20, [11] O.D. Fageras. hree-dimensional Compter Vision: A Geometric Viewpoint. MI Press, [12] S. Hs, P. Anandan, and S. Peleg. Accrate comptation of optical flow by sing layered motion representations. In ICPR 94, pages , [13] S.S. Intille and A.F. Bobick. Disparity-space images and large occlsion stereo. In 2nd ECCV, [14] M. Irani and P. Anandan. A nified approach to moing object detection in 2D and 3D scenes. In 12th ICPR, pages , [15] M. Irani, P. Anandan, and S. Hs. Mosiac based representations of ideo seqences and their applications. In 5th ICCV, pages , [16] M. Irani and S. Peleg. Image seqence enhancement sing mltiple motions analysis. In CVPR 92, pages , [17] M. Irani, B. Rosso, and S. Peleg. Detecting and tracking mltiple moing objects sing temporal integration. In 2nd ECCV, pages , [18] S.X. J, M.J. Black, and A.D. Jepson. Skin and bones: Mlti-layer, locally affine, optical flow and reglarization with transparency. In CVPR 96, pages , [19] R. Kmar, P. Anandan, and K. Hanna. Direct recoery of shape from mltiple iews: A parallax based approach. In 12th ICPR, pages , [20] J. Lengyel and J. Snyder. Rendering with coherent layers. In SIGGRAPH 97, pages , [21] S. Mann and R.W. Picard. Virtal bellows: Constrcting high qality stills from ideo. In 1st ICIP, pages , [22]. Porter and. Dff. Compositing digital images. SIG- GRAPH 84, pages , [23] H. S. Sawhney. 3D geometry from planar parallax. In CVPR 94, pages , [24] H.S. Sawhney and S. Ayer. Compact representations of ideos throgh dominant and mltiple motion estimation. PAMI, 18(8): , [25] D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffsion. In CVPR 96, pages , [26] S.M. Seitz and C.M. Dyer. Photorealistic scene reconstrction by space coloring. In CVPR 97, pages , [27] M. Shizawa and K. Mase. A nified comptational theory of motion transparency and motion bondaries based on eigenenergy analysis. In CVPR 91, pages , [28] H.-Y. Shm and R. Szeliski. Constrction and refinement of panoramic mosaics with global and local alignment. In 6th ICCV, [29] R. Szeliski and J. Coghlan. Hierarchical spline-based image registration. In CVPR 94, pages , [30] R. Szeliski and P. Golland. Stereo matching with transparency and matting. In 6th ICCV, [31] R. Szeliski and H.-Y. Shm. Creating fll iew panoramic image mosaics and textre-mapped models. In SIG- GRAPH 97, pages , [32] J. orborg and J.. Kajiya. alisman: Commodity realtime 3D graphics for the PC. In SIGGRAPH 96, pages , [33]. Viéille, C. Zeller, and L. Robert. Using collineations to compte motion and strctre in an ncalibrated image seqence. IJCV, 20(3): , [34] J.Y.A. Wang and E.H. Adelson. Layered representation for motion analysis. In CVPR 93, pages , [35] Y. Weiss. Smoothness in layers: Motion segmentation sing nonparametric mixtre estimation. In CVPR 97, pages , [36] Y. Weiss and E.H. Adelson. A nified mixtre framework for motion segmentation: Incorporating spatial coherence and estimating the nmber of models. In CVPR 96, pages , [37] Y. Yang, A. Yille, and J. L. Local, global, and mltileel stereo matching. In CVPR 93, pages ,