The SPS Algorithm: Patching Figural Continuity and Transparency by Split-Patch Search

Transcription

1 The SPS Algorithm: Pathing Figural Continuity and Transpareny y Split-Path Searh Astrat A. Criminisi and A. Blake Mirosoft Researh Ltd., 7 J J Thomson Ave, Camridge, CB3 0FB, UK antrim@mirosoft.om This paper desries a novel algorithm for the effiient synthesis of high-quality virtual views from only two input images. The emphasis is on the reovery of ontinuity of ojets oundaries (figural ontinuity) with faithful synthesis of transpareny effets. The ontriution of this paper is two-fold: i) the Split- Path Searh (SPS) tehnique is introdued for dense stereo whih handles transpareny effets y assigning multiple disparities to mixed pixels; ii) an effiient extension of exemplar-ased image synthesis to the ase of two-amera stereo is proposed. Furthermore, this paper presents an approximate ut effetive solution to the hallenging prolem of layer estimation and ompositing in the ase of small image pathes. The effetiveness of the proposed tehnique is demonstrated on a numer of stereo image pairs taken from twoamera video-onferening setups, where the quality of the synthesized talking heads is of paramount importane. Moreover, the improvement in the quality of image synthesis is quantified y omparing the output of the SPS algorithm with thirteen ground-truth images. 1 Introdution This paper deals with the prolem of effiiently generating good-quality virtual images from stereo pairs. The example in fig. 1 is used throughout the paper to explain the steps of the proposed algorithm. Many state of the art view-synthesis algorithms [3, 13, 15, 16] are prone to artefats suh as: i) aliasing and imperfet rendering of transpareny effets, ii) streaky or loky artefats whih disrupt figural ontinuity, iii) fattening or shrinking of foreground ojets (see the orrupted outline of the head in fig. 1). The goal of this paper is that of effiiently deteting and orreting those artefats. As oserved in [18], mixed pixels our along ojet oundaries of opaque ojets and where there is transpareny. In those situations, geometry-ased tehniques whih assume a single depth per pixel, are inadequate. The a Figure 1: High-quality two-amera virtual-view synthesis. (a,) Left and right input images (size ) with large disparities and olusions (60 pixel max disparity). () Virtual ylopean image (detail) reovered y a standard dense stereo algorithm [7]. Along the oundary of the head streaky and loky artefats and aliasing effets our. ( ) Virtual ylopean image (detail) after the proposed SPS enhanement step. The removal of artefats and the introdution of mixed pixels produe a more natural-looking syntheti image. prolem is exaerated in pratial stereo mathing when multi-pixel windows are used for orrespondene mathing. The window prolem may e mitigated y the use of split or shiftale windows [12, 17], ut proper modeling of transpareny effets is also needed. Our proposed approah for rendering an e seen as an extension of reent exemplar-ased synthesis tehniques [8, 11] to stereo. It is inspired y the work of Fitzgion et al. [9] who realised the potential of ditionaries of exemplars in prouring high quality detail at oundaries and over texture. Their virtual-view synthesis algorithm operates on a olletion of alirated input images (26 or more in their examples) to produe interpolated views of striking quality. They use a path ditionary, gathered from the sequene it- '

2 self, to define path priors. One important property of this approah is that images are rendered without the need for expliit matting, simply y stealing pixels from appropriate loations in their rih ditionary. However their approah is very slow owing to the use of a sustantial ditionary and a omprehensive ut expensive treatment of the data likelihood. In ontrast, our aim is to develop an effetive strategy for artefats whih is nonetheless effiient enough to e inluded, on the fly, with real-time stereo mathing. The SPS Split-Path Searh algorithm ahieves high omputational effiieny (quasi real-time) without sarifiing image quality. Effiieny is ahieved y a variety of means: restriting andidate pathes to those lying on orresponding (left or right) epipolar lines; onstraining the searh region using tight, geometri depth ounds; applying exemplar-ased synthesis sparsely, only where flagged y an inonsisteny test. Elsewhere, away from deteted artefats, synthesis is onventional and geometrially-ased, and hene effiient. This parsimonious approah would fail however in the algorithm of Fitzgion et al. whih relies on a plentiful supply of pathes to ahieve aurate rendering. Instead, repaired pathes in SPS are omposed of two part-pathes, one attriuted to the foreground and one to the akground. This in turn demands the detetion and representation of multiple depths, one for akground and one for foreground, and is ahieved y testing expliitly oth foreground and akground depth hypotheses. Transpareny effets are rendered y effetive ompositing of the foreground and akground portions to ahieve realisti-looking virtual images. 2 Prolem Statement and Notation This paper assumes given the two left and right input images I l and I r whih have een epipolar-retified (as opposed to the full amera aliration of [9]) and photometrially registered. Our goal. We seek an effiient (possily real-time) algorithm for the high-quality synthesis of the image that would e seen y a virtual amera plaed in a new viewpoint. For simpliity of explanation, the fous here is on the synthesis of ylopean images 1. The extension to the ase of general virtual viewpoint is straightforward. Notation. Image points are indiated y oldfae letters, e.g. p or q. Upperase typesets indiate matries, e.g. A. Capital letters indiate images or pathes (suimages), e.g. I or Π. Furthermore, Π p indiates a path entred on the 1 Cylopean image (denoted I) is defined as the image that would e seen y a amera positioned half-way etween the two input ameras. a Figure 2: Geometry-ased virtual view synthesis. (a) The disparity map D omputed from the two input images in fig. 1a,. The disparities are omputed with referene to the ylopean oordinate system. () The reonstruted virtual ylopean image (denoted I in the text). The left and right oluded regions (around the foreground head) have een filled with pixels extrated from the left and right input views y making use of the fronto-parallel akground assumption [7]. point p and Π p (q) denotes the olour (or grey-sale intensity) of point q ontained in the path Π p. Finally, supersripts f and indiate foreground and akground pathes, respetively. 3 New-View Synthesis y SPS This setion outlines our view-synthesis algorithm whih is omposed of two steps. In the first step a standard densestereo tehnique generates a rough virtual image I. In the seond phase, the image I is effiiently refined y the Split- Path Searh algorithm to produe the final virtual image I. The main ontriution of this paper is the SPS tehnique for effiient, exemplar-ased image synthesis Estimating disparity and olusion maps. Given the two input images I l and I r a disparity map D is generated with respet to the oordinate system defined y the desired virtual viewpoint and, at the same time, the virtual image I is synthesized (see fig. 2). For this purpose we use the algorithm in [7] ut, as demonstrated in the results setion, the refinement step of our algorithm is independent of the hoie of the speifi dense-stereo reonstrution tehnique. The main ontriution of this paper lies in the way the unavoidale artefats of I are removed. The next setions desrie: i) an algorithm for the detetion of the artefats in I, and, ii) an algorithm for the removal of suh artefats y guided exemplar-ased image re-synthesis Artefat detetion and ordering. Given the input left and right images (I l and I r, respetively), the disparity map D and the orresponding olusion map O (a produt of dense stereo), eah input image an e projeted into the new, desired viewpoint. Let us all Il w the result of projeting the left input image I l into the target viewpoint (see fig. 3a); and Ir w for the right input image (fig. 3). A pixelwise distane d(il w,iw r ) etween the two projeted images 2

3 a d Figure 3: Artefat detetion. (a,) Projetions of left and right input images into the target viewpoint, respetively (denoted Il w and Ir w in text). Green and red regions denote estimated half-olusions. () Aliasing-insensitive image distane map, d(il w,ir w ). Darker points orrespond to larger pixel distanes (pixel intensities have een resaled for improved visiility). (d) Deteted set of artefat pixels A ( 7% of image area). These pixels will e re-synthesized and orreted y the SPS algorithm. indiates the loation and entity of artefats (the dark points in fig. 3). Assuming low levels of image noise, large values of d(i w l,iw r ) our in plaes where the dense-stereo algorithm has failed to estimate the orret pixel orrespondene etween the two input images I l and I r. Note that inaurate disparities do not neessarily produe inaurate pixel synthesis; however, here, sine we are interested in quality of image synthesis, we are orretly measuring artefats in image spae rather than disparity spae. Furthermore, in order to overome issues related to the (often) disrete nature of the disparity map, it is onvenient to define the image distane d(i 1,I 2 ) etween two generi images I 1 and I 2 as a sampling-independent funtion [1], where half-olusions are ignored. Finally, artefats are defined as the set A of points p I suh that d(i w l,iw r ) > λ (fig. 3d) with λ a predefined value 2. Furthermore, we have found it helpful to augment A with a one-pixel-wide oundary of the foreground. This an e ahieved readily from the deteted left and right olusions. The algorithm then proeeds to the removal of the artefats of the ylopean image y a re-synthesis proess. As it will e learer later this refinement proedure an e interpreted as an extension of the many exemplar-ased texture and image synthesis algorithms [8, 11] to two-view stereo. The work of [2, 6, 10] has pointed out that exemplar- 2 Typially we hoose λ very small (e.g. λ =5intensity levels) sine the quality of image synthesis is fairly roust to large numers of false positives. ased synthesis enefits from proessing the most reliale pixels first. Here we follow the same philosophy and assign a priority value P (p) to eah of the pixels p Awith the synthesis proeeding from highest- to lowest-priority pixels. Similar to [2] we adopt P (p) to e proportional to the numer of already filled neighouring pixels although more sophistiated ordering algorithms may e employed [6, 10] Artefat removal y SPS and re-rendering. The algorithm proeeds as follows: We have given the first (orrupt) estimate of the ylopean image I, and the set of deteted artefats A. For eah point p Awe extrat the soure path Φ p (fig. 4, typially 5 5), entred on p and we seek a new, target path Ψ p whih is similar to Φ p ut where the artefats have een removed (f. fig. 6). Replaing Φ p with Ψ p for all the points p in A ahieves the desired orretion. The steps of one iteration of the artefat-removal algorithm are: Split-Path Searh: given Φ p entred on p, searh along the orresponding sanlines in I l and I r for the two pathes that are most similar to the foreground and akground portions of Φ p ; Compositing and Rendering: omine those pathes to generate the target path Ψ p and replae Φ p with Ψ p. The details of eah step are explained next Split-Path Searh. Given p A, its orresponding path Φ p and a low-pass filtered version D of the ylopean disparity map D, we ompute the foreground and akground weight arrays Ω f p and Ω p as follows: Ω f p(q) = D(q) min D D max D ; min Ω p(q) =1 Ω f p(q); q Φ p (1) with D min and D max respetively the minimum and maximum value of the (filtered) disparities within Φ p. Notie that larger values of Ω f p (f. fig. 4) our for points whih are loser to the oserver, and thus more likely to e foreground in a path straddling foreground and akground. Ω p is the omplement of Ω f p. These approximate foreground and akground weights are suffiient to drive the searh algorithm desried elow. The low-pass filtering of the disparities ahieves roustness of the searh algorithm y reduing the influene of high-frequeny disparity artefats (e.g. the horizontal streaks along the head oundary in fig. 2a). Moreover, roust variants of the weights in (1) may e defined, e.g. y means of an ativation-like sigmoid transformation. In pratie, though, we have found the definitions in (1) suffiient. The SPS algorithm proeeds y searhing for the two pathes that are most similar to the foreground and akground portions of the soure ylopean path Φ p.thereovered pair of left-view pathes are denoted L f p and L p, 3

4 Cylopean Left yright sanline a p px y Figure 4: Split-Path Searh. Given the soure path Φ p in the orrupt ylopean image I, we seek the two pathes in I l and I r whih are most similar to the foreground and akground regions of Φ p, respetively. Results of the automati searh are the pairs of pathes L f p and L p for the left input image and Rp f and Rp for the right input image. The automatially omputed value of δ y determines the small portion of the urrent sanline in whih the searh is performed. In this running example we use pathes of size for larity, however the SPS algorithm normally employs 5 5 pathes for effiieny. I Il Ir Curl and the right-view ones Rp f and Rp (fig. 4). It is important to stress that the searh is limited, for effiieny, to small segments along the orresponding left and right sanlines as follows: L f p = Lˆq with ˆq = arg min d (Ω f p Φ p, Ω f p L q ) p x q x p x+δ y L p = Lˆq with ˆq = arg min d (Ω p Φ p, Ω p L q ) p x q x p x+δ y Rp f = Rˆq with ˆq = arg min d (Ω f p Φ p, Ω f p R q ) p x δ y q x p x Rp = Rˆq with ˆq = arg min d (Ω p Φ p, Ω p R q ) p x δ y q x p x with L q and R q generi left-view and right-view pathes entred on the generi point q q y = p y. Here the symol denotes point-wise multipliation etween images (or pathes). The distane d (Π 1, Π 2 ) etween two generi pathes Π 1 and Π 2 is defined as the sum of squared distanes (SSD) of pixel values where artefat pixels are ignored. The value δ y whih restrits the searh region for Weighted SSD Searhing Fg path on Right image y Searhing Bg path on Right image Figure 5: Path distanes. These two plots show the values of the weighted SSD distane funtions d (Ω f p Φ p, Ω f p R q) and d (Ω p Φ p, Ω p R q) for varying values of q x [p x δ y,p x] along the right sanline. Φ p is the example soure path in fig. 4. Restriting the searh to a small sanline segment of length δ y ahieves effiieny. Our winner-take-all algorithm selets the pathes Rp f and Rp orresponding to the minima of the aove plots (marked in green and lue, f. fig. 4). These orrespond to the two orret foreground and akground disparities at point p. effiieny is a sanline-dependent value defined as δ y = max q I qy=y D(q)/2. Examples of suh automatially extrated pathes are shown in fig. 4. The SPS algorithm may e interpreted as a winner-takeall algorithm for dense stereo. However, unlike previous approahes of this kind, here the algorithm is applied twie, one to the foreground and one to the akground portions of the soure path Φ p. This has the effet of assigning two depth values to the artefat pixels in A. Notaly, in the ase of mixed pixels the two estimated depths orrespond to the depths of the foreground and akground omponents of the mix. The redued searh region and the large autoorrelation of the Φ p path 3 make the typially fragile winnertake-all algorithm suffiiently roust. Effiieny. It is important to stress that the SPS algorithm is eonomial sine for eah point p Athe searh region is restrited to a short sanline segment of length δ y. Figure 5 shows the typial ehaviour of the path distane funtion for varying values of the q x oordinate Seleting the est akground path. Figure 4 demonstrates the suessful detetion of the two left and right foreground pathes L f p and Rp f (the foreground hair url is present in oth and in the same position). However, due to olusion, only one of the two retrieved akground pathes is meaningful. In fat, the true akground of Φ p (the vertial door frame) is oluded in the left view I l, thus the retrieved path L p is meaningless. In ontrast, the right akground path Rp ontains the orret akground information. The atual hoie etween L p and Rp is performed automatially y retaining the path Π p whih is most similar to the akground of Φ p, i.e. : Π p = arg min Λ {L p,rp } d (Ω p Λ, Ω p Φ p ).Intheexample in fig. 4 the seleted akground path is Π p = Rp. 3 Artefats tipially our along high-ontrast ojet oundaries. y q x 4

5 soure Analysis Synthesis Semi-transparent hole (alphas) a target Figure 6: Path matting and ompositing. See text for notation. (a) One the two akgrounds ( ˆL p, ˆR p) assoiated to the two foreground pathes (L f p,r f p) have een estimated transparenies (Γ p) and foreground olours (Π f p) of the target path an e omputed. Finally, the target path Ψ p is omputed y the onventional ompositing equation (2). The path Ψ p is a lear improvement with respet to the original, orrupted ylopean path Φ p. () A 3D height-map visualization of the estimated opaities Γ p showing the orretly reovered semi-transparent hole in the hair url. Final omposite x y At this point we have omputed Π p whih is one of the elements needed for ompositing the artefat-free target path Ψ p (fig. 6). Further steps are: i) for eah pixel q Φ p estimating its (unontaminated) foreground olour Π f p(q) and transpareny Γ p (q). ii) omine foreground Π f p, akground Π p and transparenies Γ p to otain the desired target path Ψ p. These steps are desried next Path matting, ompositing and rendering. For eah point p A, we have desried how to extrat two foreground-registered pathes L f p and Rp. f If we knew their orresponding akgrounds we ould apply the tehnique desried in [19] to estimate pixel opaities and unontaminated foreground olours neessary to generate the target path Ψ p. This setion attaks this prolem; however, having availale only two input images makes the prolem ill-posed, and reasonale assumptions will e neessary. Segmentation-ased matting tehniques [4] are not suited here sine they target single-image ases. The additional information provided y the omparison of the two foreground pathes is exploited in this paper. We egin y noting that the path L f p extrated from the left input image an e interpreted itself as a omposite image. In our example, its akground ˆL p (the poster on the ak wall) is ompletely visile in the right input view and an e extrated y the following searh proess: ˆL p = Rˆq where ˆq = arg min d (Ω p L f p, Ω p R q ) p x q x p x+δ y and simmetrially for ˆR p. In our running example, however, the akground orresponding to the right foreground path R f p (the rown door in fig. 4) is oluded, in the left image, y the person. Thus, the automatially extrated path ˆR p is meaningless. This situation an e automatially deteted y heking the sign of the quantity defined as: =d (Ω p ˆR p, Ω p R f p) d (Ω p ˆL p, Ω p L f p). In fat, the situation presented in our example orresponds to a value > 0. Similarly, < 0 orresponds to the olusion of the left akground path ˆL p. The prolem of oluded akground pathes is of a general nature and assumptions are needed to estimate the missing information. In the ase of oluded ˆR p we proeed as follows: given the right foreground path Rp f and the akground filter Ω p, we extrat the pixels of Rp f whih elong to the akground and then fit a parametri surfae model (e.g. polynomial, spline et.) to the orresponding olour values 4. Finally, the fitted surfae model is used to extrapolate the olours of the pixels in the oluded portion of Rp. f We have found that for small pathes (5 5) extrapolation via a generi planar fit (generally not at onstant height) produes good results. More powerful extrapolation tehniques may e onsidered for larger and highly textured pathes. Figure 6 shows the estimated ˆR p path of the example; notie the extrapolated area ehind the hair url. Symmetrial reasoning applies when < 0. Now, similarly to [19] we have availale two known foreground-registered pathes (L f p and Rp) f and the two orresponding (different) akground pathes ( ˆL p and ˆR p). For L f p and Rp f the onventional ompositing equation generalized to the ase of pathes is: L f p =Γ p Π f p +(1 Γ p ) ˆL p R f p =Γ p Π f p +(1 Γ p ) ˆR p 4 we employ an RGB olour model. 5

6 with Γ p the opaities and Π f p the unontaminated foreground olours. Sine oth akground pathes are known, then oth Γ p and Π f p are uniquely determined. Opaities are assumed to apply equally to eah of the RGB hannels. Unfortunately, some of the orresponding pixels in the two akgrounds ˆL p and ˆR p may have very similar olours, thus making the aurate reovery of transparenies and foreground olours ill-posed [19]. Image noise an further exaerate this pathologial situation. However, reasonale estimates of transparenies and olours an e otained through the inorporation of prior information (e.g. on the distriution of alpha and olour values). This regularization effet an e ahieved either y means of a Bayesian approah [19] or, simply y a depth-driven, low-pass filtering of the transpareny and olour signals. We have found the latter to work suffiiently well in the ase of small image pathes. Examples of estimation of Γ p and Π f p are shown in fig. 6, where foreground olours have een omposited on a white akground for aided visualization. Finally, given the foreground Π f p, the opaities Γ p and the akground Π p, the target path Ψ p remains defined: Ψ p =Γ p Π f p +(1 Γ p ) Π p. (2) Figure 6 shows the results of running one iteration of the SPS algorithm on a real-image example: omparison etween the original path Φ p and the estimated target path Ψ p, demonstrates the effetive removal of artefats. In fig. 6 notie how the estimated transpareny map Γ p orretly aptures the semi-transparent nature of the hole in the hair url. The enhanement of the entire virtual image I is ahieved y opying the ontent of Ψ p inside Φ p for all pixels p A Φ p and repeating the steps aove until all the pixels in A have een re-synthesized. Figure 1 shows the result of applying the SPS enhanement algorithm to the entire ylopean image in fig. 1. In the urrent version of the algorithm a pixel p Amay e synthesized more than one sine it elongs to a numer of overlapping pathes. In this ase only the last value is retained. Moreover, we have found larger path sizes to yield etter quality of synthesis at the expense of CPU yles. The SPS algorithm is validated next on a numer of ground-truth data and further real-image examples. 4 Results and Comparisons This setion presents a quantitative and qualitative evaluation of the performane of the proposed SPS algorithm. The improvement in the quality of the syntheti image is measured y omparisons against ground-truth data. Further examples of virtual-image synthesis in typial two-amera video-onferening sessions are also presented. Comparison with ground truth. The performane of the SPS algorithm is measured as follows: given the input im- Figure 7: Additional ground-truth data. Sample frames from the two additional ground-truth sequenes used to quantify the performane of the SPS algorithm. The amera is translating horizontally with onstant veloity. (a) Oranges sequene. Image size is and max ojet olusion is aout 4% of the image width. () Cue sequene. Image size is and max ojet olusion is aout 3% of the image width. ages I l and I r and the orresponding ground-truth ylopean image I gt we first synthesize the ylopean view I using a standard dense-stereo algorithm. Seondly, we apply the outline-enhanement SPS algorithm to the image I and generate the improved image I. The proportionate quality improvement is measured as the ratio ρ = d(i,i gt) d(i,i gt ). d(i,i gt ) Thus, positive values of ρ indiate an atual improvement of the image quality (eause d(i,i gt ) <d(i,i gt )) and vie-versa for negative values of ρ. The image distane d(i 1,I 2 ) etween two generi images I 1 and I 2 is defined as the sampling-independent distane of [1]. These distanes are omputed only at the points laelled as artefats. We run our experiments on all nine groundtruth sequenes from the Middleury data set ( For eah sequene we hose the first and last frames (frames 0 and 8) as the left and right input images and the middle frame (frame 4) as the ground-truth ylopean image I gt. The results are as follows: data arn1 arn2 ull ρ +0.20% +1.57% +7.33% data ones poster sawtooth ρ +2.94% +6.59% +2.33% data teddy tsukua venus ρ +3.89% +0.8% +1.66% It an e oserved that in all the aove experiments the sign of ρ is positive, onfirming the atual improvement of image quality ahieved y the SPS algorithm. The Middleury dataset uses short aselines and, onsequently, is haraterized y very small oluded regions. This is not very representative of real-world stereo pairs. For instane, the stereo images in fig. 1 are haraterized y a maximum ylopean olusion of 8% of the image width (to e ompared to the 2.6% average of maximum ylopean olusions in the Middleury dataset). Thus, in order to test the SPS algorithm with larger olusions we generated our own ground-truth data y aquiring two sequenes from a horizontally-translating amera (fig. 7). From eah of the two sequenes we extrated two triplets of left-ylopean-right images (laelled as exp1 and exp2 ), and the measured values of ρ are as follows: 6

7 a a' Using different dense-stereo algorithms. To demonstrate the general nature of SPS we have applied it to the virtual images generated y different dense-stereo algorithms. Figures 9a, show the ylopean images otained from the disparities estimated y the algorithms in [5] and [14], re- a a' ' Figure 9: SPS with different stereo algorithms. (a,) Virtual images generated y the Dynami Programming algorithm in [5] and the Graph- Cut algorithm in [14], respetively. (a, ) The orresponding images after SPS enhanement. In oth ases artefats have een removed and the outline of the teddy ear enhaned. ' a ' d d' Figure 8: A furry toy example. (a) The ylopean image generated y a standard geometry-ased dense stereo algorithm. The input left and right images are not shown here. (a ) The ylopean image after SPS enhanement. (,,d) details of (a). (,,d ) orresponding details of (a ). The artefats in the donkey s hair, the donkey s nek and the teddy s nose have een removed and the quality of the syntheti images enhaned. data oranges, exp1 oranges, exp2 ρ +6.50% +3.52% data ue, exp1 ue, exp2 ρ +5.47% +9.95% One again, all the entries of the aove tale show positive values of ρ, thus onfirming the effiay of the SPS tehnique. The larger (on average) values of ρ in this seond set of experiments are explained y the fat that larger olusion regions are more likely to ause failure of the geometry-ased synthesis. Consequently, the ontriution of our exemplar-ased enhanement eomes more evident. A furry toy example. Figure 8 shows the results of synthesizing ylopean images on a diffiult furry toy example. The detail images highlight the removal of oundary loks and streaks, and the improved rendering of mixed pixels. a' ' ' Figure 10: A two-amera video-onferening example. (a) Cylopean view after first pass. The two input images are not shown here. (,) Details of (a) showing aliasing artefats along the head oundary. (a ) Cylopean view after SPS enhanement. (, ) Details of (a ) where the introdution of pixel mixing produes an enhaned outline of the foreground ojet. spetively. Figures 9a, show the orresponding SPSenhaned ylopean images. Video-onferening examples. Finally, we present two more video-onferening examples (of the kind in fig. 1). Comparing fig. 10 with fig. 10 and fig. 10 with fig. 10 highlights the enhaned quality. Figure 11 demonstrates that the SPS algorithm is partiularly useful for akground sustitution. Notaly, the unavoidale aliasing that arises from disparity-ased akground removal and sustitution is fixed y running the SPS algorithm along the oundary of the foreground ojet, thus produing a smooth and artefat-free foreground/akground transition. Image sequenes. Our experiments demonstrate that synthesized temporal sequenes also enefit from the SPS algorithm, however, due to spae onstraints we are unale to 7

8 a f.p.s. for artefats whih typially over less than 10% of the image area. The result is a nearly real-time algorithm for the synthesis of high-quality virtual views from only two input images. Areas of further researh inlude: i) integrating SPS into stereo mathing, ii) investigating a proailisti framework for aurate path matting in the hallenging two-image senario, iii) extending the SPS algorithm to expliitly impose temporal onsisteny in image sequenes. Aknowledgements. The authors would like to thank C. Rother and R. Szeliski for inspiring disussions, and G. Cross for the effiient implementation of our dense stereo and SPS algorithms. Referenes Figure 11: SPS enhanes akground sustitution. (a) Input left image, the right image is not shown here. () Desired new akground. () Replaing the desired akground in the syntheti ylopean view y simple depth thresholding produes unnaturally sharp foreground/akground transitions. Moreover, lak of pixel mixing make the head appear stuk upon the akground. ( ) The result of applying SPS to (). Boundary artefats (aliasing and lak of pixel mixing) have een fixed, thus yielding a more realisti-looking omposition. show results here. Numerous examples and results on oth stati and temporal data are availale at [20]. 5 Conlusion This paper has presented a novel tehnique for the effiient and aurate synthesis of virtual views from only two input images. The emphasis is on the synthesis of artefat-free ojets oundaries (figural ontinuity) with faithful pixel mixing. We employ a two-pass new-view synthesis algorithm whih omines the effiieny of disparity-ased tehniques with the quality of exemplar-ased synthesis algorithms. The main ontriution is the Split-Path Searh algorithm for virtual-image synthesis whih: i) detets the artefats generated y the geometry-ased synthesis and ii) removes them y means of a multiple-depth stereo algorithm. The atual image synthesis is performed in an exemplarased fashion, adapted to the ase of stereo images. Transpareny effets and mixed pixels are rendered y pathased matting and ompositing. Computational effiieny is ahieved y taking advantage of the geometry-ased reonstrution to onstrain tightly the searh for exemplar pathes in the SPS step. In our C++ implementation, whih exploits SSE2 instrutions, the first pass (dense stereo) runs at approximately 7.5 f.p.s. on a dual-proessor 3GHz Pentium IV with 1G RAM. The SPS refinement phase redues the speed down to aout 5.5 ' [1] S. Birhfield and C. Tomasi. A pixel dissimilarity measure that is insensitive to image sampling. IEEE Transations on Pattern Analysis and Mahine Intelligene,, 4(20): , [2] R. Bornard, E. Lean, L. Laorelli, and J.-H. Chenot. Missing data orretion in still images and image sequenes. In ACM Multimedia, Frane, Deemer [3] A. Broadhurst and R. Cipolla. A statistial onsisteny hek for the spae arving algorithm. In Pro. ICCV, [4] Y.-Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski. A Bayesian approah to digital matting. In Pro. CVPR, [5] I. J. Cox, S. L. Hingorani, S. B. Rao, and B. M. Maggs. A maximumlikelihood stereo algorithm. CVIU, 63(3): , May [6] A. Criminisi, P. Perez, and K. Toyama. Ojet removal y exemplarased inpainting. In Pro. CVPR, Madison, WI, Jun [7] A. Criminisi, J. Shotton, A. Blake, and P.H.S. Torr. Gaze manipulation for one-to-one teleonferening. In Pro. ICCV, Nie, Ot [8] A. Efros and T. Leung. Texture synthesis y non-parametri sampling. In Pro. ICCV, pages , Sep [9] A. Fitzgion, Y. Wexler, and A. Zisserman. Image-ased rendering using image-ased priors. In Pro. ICCV, Nie, Ot [10] P. Harrison. A non-hierarhial proedure for re-synthesis of omplex texture. In Pro. Int. Conf. Central Europe Comp. Graphis, Vis. and Comp. Vision, Plzen, Czeh Repuli, Feruary [11] A. Hertzman, C. E. Jaos, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. In Pro. ACM SIGGRAPH, Eugene Fiume, [12] H. Hirshmueller. Improvements in real-time orrelation-ased stereo vision. IJCV, [13] R. Koh. 3D surfae reonstrution from stereosopi image sequenes. In Pro. ICCV, pages , [14] V. Kolmogorov and R. Zaih. Computing visual orrespondene with olusions using graph uts. In Pro. ICCV, pages , [15] K. N. Kutulakos and S. M. Seitz. A theory of shape y spae arving. Tehnial Report CSTR 692, University of Rohester, [16] D. Sharstein. View Synthesis Using Stereo Vision, volume 1583 of Leture Notes in Computer Siene (LNCS). Springer-Verlag, [17] D. Sharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo orrespondene algorithms. IJCV, 47(1/2/3):7 42, [18] R. Szeliski and P. Golland. Stereo mathing with transpareny and matting. IJCV, [19] Y. Wexler, A. Fitzgion, and A. Zisserman. Bayesian estimation of layers from multiple images. In Pro. ECCV, Copenhagen, [20] 8