New Appearance Models for Natural Image Matting

Transcription

1 New Appearance Models for Natural Image Mattng Dheeraj Sngaraju Johns Hopkns Unversty Baltmore, MD, USA. Carsten Rother Mcrosoft Research Cambrdge, UK. Chrstoph Rhemann Venna Unversty of Technology Venna, Austra. Abstract Image mattng s the task of estmatng a fore- and background layer from a sngle mage. To solve ths ll posed problem, an accurate modelng of the scene s appearance s necessary. Exstng methods that provde a closed form soluton to ths problem, assume that the colors of the foreground and background layers are locally lnear. In ths paper, we show that such models can be an overft when the colors of the two layers are locally constant. We derve new closed form expressons n such cases, and show that our models are more compact than exstng ones. In partcular, the null space of our cost functon s a subset of the null space constructed by exstng approaches. We dscuss the bas towards specfc solutons for each formulaton. Experments on synthetc and real data confrm that our compact models estmate alpha mattes more accurately than exstng technques, wthout the need of addtonal user nteracton. 1. Introducton Image mattng addresses the problem of estmatng the partal opacty of each pxel n a gven mage. In partcular, one assumes that the ntensty I of the th pxel can be wrtten as the convex combnaton of a foreground ntensty F and a background ntensty B, as I = α F + (1 α )B, (1) where α s referred to as the pxel s partal opacty value or alpha matte. By defnton, ths value s constraned to take values n [0, 1. We note that for each pxel n color mages, (1) gves us 3 equatons n 7 unknowns. Consequently, the mage mattng problem s hghly under-constraned. To ths effect, the user s requred to provde some addtonal nformaton n order to make the problem well posed. Such nformaton s typcally provded n the form of a trmap by markng dfferent regons n the mage as (a) foreground; α = 1, (b) background; α = 0, and (c) unknown; α [0, 1. The goal of mage mattng algorthms s therefore to estmate the alpha mattes of the pxels n the unknown re- Ths work was supported n part by Mcrosoft Research Cambrdge through ts PhD Scholarshp Programme and a travel sponsorshp. gon. Incpent methods such as [12 use the trmap to construct basc color models for the foreground and background, whch are subsequently used to estmate the alpha mattes n the unknown regon as per (1). Due to ther nave modelng schemes, such algorthms fal on mages wth complex ntensty varatons. Methods such as [2, 15 solve such ssues by usng local propagaton technques to estmate the alpha mattes. However, ther good performance s subject to the use of a tght trmap. Subsequent research n mage mattng wtnessed the use of a number of algorthms orgnally ntended for mage segmentaton [11, 17, 3, 4, 1, 20. In most cases, these algorthms use a sparse trmap to estmate a bnary segmentaton, whch s then used to generate a tght trmap for the mage. The alpha mattes are estmated usng ths tght trmap and can then potentally be refned by alternatng between re-estmaton of the alpha mattes and the trmap. It s mportant to apprecate the fact that mattng and segmentaton are dfferent problems. In order to estmate the alpha mattes of an mage, one needs to develop extremely accurate models for the scene s appearance. Ths s not so for the case of segmentaton, where t suffces to defne mage features that help to dstngush the object from the background. Consequently, recent work n mage mattng has seen a surge of research towards developng algorthms that explot varous features specfc to the mattng problem [6, 19, 16, 7, 10, 9. It was shown n [6 that f one assumes the ntenstes of the foreground and background layers to vary lnearly n small mage patches, then the alpha mattes could be estmated n a closed form fashon. It was later demonstrated that the performance of local propagaton based methods such as [6 could be mproved by addtonally learnng global color models [19, 16, 9. Recent work has also focused on enforcng sparsty of the alpha mattes [7, 10. For a more detaled revew of mage mattng algorthms, we refer the reader to [18. In ths paper, we propose to mprove the state of the art for mage mattng by developng accurate models for a scene s appearance, and hence fundamentally mprove the buldng blocks of mattng algorthms. In partcular, we 1

2 focus on the Mattng Laplacan proposed by [6, whch s a matrx characterzng a cost functon for mage mattng. Ths cost functon s derved under the assumpton that the foreground and background layers of each mage patch exhbt lnear varaton n the ntenstes. As we show, ths assumpton can be an overft for the mage data, f the colors of ether layer are locally constant. Specfcally, we show that for small perturbatons n the mage data, [6 mght construct a null space of possble solutons, whch s larger than desred, thereby makng the problem more ambguous. We then show how one can construct more compact models for the alpha mattes, whch have a null space of provably smaller dmenson than that of [6. Furthermore, the dfferent formulatons have a dfferent bas towards specfc solutons, whch we wll dscuss n detal. Compellng experments on synthetc and real mages valdate our clams. Consequently, we present a new framework for closed form solutons to mage mattng, whch s theoretcally prncpled and yelds hgh qualty alpha mattes. Note that snce the Mattng Laplacan has been used n [6, 19, 16, 7, 10, 9 for regularzaton of the alpha mattes, our framework can be used to mprove the performance of these algorthms. Furthermore, our framework can be appled to alternatve applcatons such as lght mxture estmaton [5, snce they use varants of the Mattng Laplacan. 2. The Mattng Laplacan: A Revew Omer and Werman [8 emprcally showed that the dstrbuton of colors n real mages s locally lnear n RGB space. Inspred by ths work, Levn et al. [6 state that gven any small patch n the query composte mage, the ntenstes of the correspondng foreground and background layers can be assumed to le on lnes n RGB space. In partcular, for a small patch W centered around pxel, there exst colors (F 1, F 2, B 1, B 2 ) such that the foreground and background colors (F j, B j ) of each pxel j W can be expressed as F j = β F j F 1 + (1 β F j )F 2, and B j = β B j B 1 + (1 β B j )B 2. Under ths assumpton, [6 showed that there exst affne functons v = (a R, ag, ab, b ) characterstc to the patch W, such that the alpha matte α j of each pxel j W can be wrtten as (2) α j = a R I R j + a B I B j + a G I G j + b, (3) where Ij R, IG j and Ij B refer to the RGB values of pxel j. The problem of estmatng the mattes α n the mage can consequently be posed as one of fndng the mnmzer of J(α, v)= [ ( αj a R Ij R a B Ij B a G I G ) 2 j b, V j W (4) where V s the set of all pxels n the mage. Essentally, ths corresponds to mnmzng the resdual of the affne model v defned n (3) for every small patch W. Note that [6 actually uses a modfcaton of the cost functon J(α, v) by ntroducng an addtonal regularzaton term, as J ɛ (α, v) = J(α, v) + ɛ ( ) a R 2 + a G2 + a B2. (5) V The regularzaton term s ntroduced n order to enforce the affne functon (a R, ag, ab, b ) = (0, 0, 0, c), c [0, 1, or n other words, enforce constancy of alpha mattes over the patch W. The motvaton for ths s twofold. Frstly, the user provded scrbbles are typcally sparse and constran far fewer pxels than the perfectly tght trmap. Hence, for many pxels n the mage, an α of 0 or 1 s desred, ndependent of the appearance model. Secondly, real mages often have hghly textured patches that do not satsfy the color lne model, but nonetheless may have unform alpha mattes across the patch. The alpha mattes of such patches can be explaned by an affne model of the form v = (0, 0, 0, c), c [0, 1. Therefore, the model allows for certan complex cases beyond the color lne model of (2). Note that the constructed cost functon J ɛ (α, v) depends on two unknown quanttes; the alpha mattes α and the affne functons v. However, [6 showed that ths can be reduced to a cost functon that depends solely on the alpha mattes. For the sake of smplcty, let us defne matrces G R ( W +3) 4 and ᾱ R W +3. The frst W rows of G are gven by [ Ij R Ij G Ij B 1, j W and the last three rows are gven by [ (ɛ)i 3 0, where I n s an dentty matrx of sze n n. The frst W entres of ᾱ are gven by α j, j W and the last three entres are equal to 0. Gven ths notaton, J ɛ (α, v) can be rewrtten as J ɛ (α, v) = V G v ᾱ 2. (6) Now, we see that we can estmate the affne functon v for each patch W, as v = argmn v G v ᾱ 2 = (G G ) 1 G ᾱ. (7) Therefore, usng the expresson for v from (7), we see that the cost functon J ɛ (α, v) can be reduced to a cost functon dependent on the alpha mattes only, as J ɛ (α) = V = α Lα. [ᾱ (I W +3 G (G G ) 1 G )ᾱ The matrx L s referred to as the Mattng Laplacan and we refer the reader to [6 for a detaled dervaton of ts entres. Note that the constructed cost functon s quadratc n the alpha mattes. Therefore, the mnmzer of ths cost functon (8)

3 can be estmated by solvng a lnear system. Hence, we have a closed form soluton for the alpha mattes. What s the null space? It s of nterest to nspect the nature of the solutons to ths system for a small patch W n the mage. Let us defne a matrx M = I W +3 G (G G ) 1 G. Note that M G = G G = 0. Therefore, by constructon, the columns of the matrx G and ther lnear combnatons are null vectors of the matrx M. If the foreground and background layers truly satsfy the color lne model, we know that the vector of alpha mattes ᾱ s gven by G v. In other words, the vector of alpha mattes s gven by a lnear combnaton of the columns of G and hence les n the null space of M. Also, n such cases G s of rank 4, and therefore M has a null space of dmenson 4. As a result, the vector of alpha mattes s only one of the potental mnmzers of the constructed cost functon. For nstance, we have seen from our earler dscusson that the constant soluton s also part of the null space. It s precsely for ths reason that the user s requred to mark scrbbles n the mage and embed constrants, so that the algorthm can recover the true alpha mattes as the mnmzers of J ɛ (α) Lmtatons of the Mattng Laplacan In practce, t can be observed that [6 does not always recover the ground truth alpha mattes. Ths s due to several reasons, a few of whch are outlned below. 1. Volaton of the color-lne model: In natural mages, t often happens that the mage data does not satsfy the color lne model. In the case of complex ntensty varatons, t s obvous that the color lne model s too smple to explan the mage ntenstes. In such scenaros, one would have to resort to data drven schemes such as [19, 9 for generatng canddate foreground and background colors. However, we are partcularly nterested n the case when the ntensty varatons are much smpler than the color lne, such as beng locally constant. As we shall show later, the true dmenson of possble solutons for such mage patches s less than 4 and hence the algorthm of [6 provdes an overft. Snce such patches occur commonly n natural mages, t s of nterest to construct more compact models for the alpha mattes n such patches. 2. Insuffcent user nteracton: Recall that the system constructed by [6 has a 4-dmensonal null space for each local mage patch consdered by the algorthm. Hence, hgh level user nteracton s requred to resolve any ambgutes. One could potentally ncorporate pror knowledge n order to bas the system towards certan famly of solutons. [6 bases the mattes to be locally constant, whch can prove to be unsatsfactory n practce. As shown n Fgure 2(c,e) when the user marks pxels correspondng to one layer only, [6 assgns constant alpha mattes equal to 0 or 1 to all unmarked pxels, based on whether the scrbbles correspond to background or foreground respectvely. We wll address ths problem n detal later. Alternatvely, [7 bases the mattes towards 0 or 1 usng non-lnear prors. Ths resultng system s however prohbtvely slow n practce. Moreover, the mattes estmated at the scrbbled pxels do not necessarly match the values specfed by the user. Note that one could also predct the alpha value for each pxel wth a certan confdence value, as n [19, 9. However, such frameworks are beyond the scope of ths paper. 3. Appearance Models Beyond Color Lnes In ths work, we consder the cases when the color lne models are volated, such that at least one of the foreground or background layers le on a pont rather than a lne n color space. Specfcally, we nspect the rank of the matrx G ntroduced n Secton 2 and analyze the cases when the rank of the matrx s less than 4. We show that for these cases, the ntenstes of the composte mage le on lnear/affne spaces of dmenson less than or equal to 4. Snce n general, the Mattng Laplacan of [6 has a 4 dmensonal null space, we demonstrate that our method s more robust to nosy data. Further, we show that the soluton space of our formulaton ncludes the constant alpha soluton, whch s mportant for hghly textured areas. We wll also show that the model of [6 has a natural bas towards constant solutons, whle our model has a natural bas for a constant 0 (or 1) soluton. Both bases, our and [6 are not optmal, snce the deal bas s towards 0 and 1 smultaneously. Unfortunately, the deal bas leads to a non-lnear system, e.g. as shown n [7, whch s very challengng to optmze and hence [7 s not ranked very well n recent evaluatons [10. We wll see n secton 4, that our method performs on average favourably, whch suggests that robustness to nose overweghs the nfluences of the dfferent bas. Also, we wll show that for the specal case where the user specfed unknown regon s bounded by constrants of one type only, e.g. only foreground, the bas of our formulaton towards 0 s clearly preferable Lne-Pont Color Models We frst consder the case when the colors of exactly one layer satsfy the color lne model, whle the colors of the other layer are constant and hence satsfy a color pont model. Wthout loss of generalty, assume that the foreground ntenstes are constant and that the background ntenstes le on a color lne. It s easy to check that f the type of models were nterchanged, our followng analyss would result n the same cost functon. Now, by the hypothess, j W, F j = F and j W, B j = β j B 1 + (1 β j )B 2. Therefore, the composte ntensty I j of a pxel j W can be expressed as j W :I j = α j F + (1 α j )[β j B 1 + (1 β j )B 2 = α j (F B 2 )+(1 α j )β j (B 1 B 2 )+B 2. (9)

4 For ths scenaro, we derve two mportant results as gven by Theorem 1. Theorem 1 Consder an mage patch W around a pxel V, such that the RGB ntenstes of the pxels n the patch, satsfy the lne-pont color models. Defne [ a matrx G LP R W 3, whose rows are gven as I R j Ij G Ij B, j W. Also defne a matrx ᾱ R W, whose entres are gven by α j, j W 1. If the foreground color pont does not le on the background color lne and there are at least three pxels a, b, c W such that α a α b α c and (1 α a )β a (1 α b )β b (1 α c )β c, then Rk(G LP ) = If Rk(G LP ) = 3, the alpha matte α j of each pxel j W, can be expressed as a lnear functon of the pxel s ntenstes, va unque coeffcents v = (a R, ab, ag ) R3 characterstc to the patch W, as Proof. j W : α j = a R I R j + a B I B j + a G I G j = ᾱ = G LP v. (10) 1. Note that by defnton, we have... G LP [ = α j (1 α j )β j 1 F B2 B 1 B 2 B 2. }{{}... H }{{} Γ (11) Snce by hypothess, F does not le on the lne spanned by B 1 and B 2, the matrx H s full rank, and hence of rank 3. If the alpha mattes do not le n any crtcal confguraton, then the matrx Γ n (11) s also full rank and hence of rank 3. Consequently, the matrx G LP s also of rank If G LP s rank 3, we know that H also must be rank 3 and hence nvertble. Hence, we can rewrte (11) as... α j (1 α j )β j 1 = GLP H 1. (12).. Therefore, we see that there exsts a unque lnear functon v gven by the frst column of H 1, that relates the alpha matte of each pxel to ts RGB ntenstes. Observe that ths s dfferent from (3) derved under the color lne assumpton, where the alpha mattes were affne. functons of the ntenstes. Now, there s no constant term present n the expresson for the alpha mattes. As earler, we can estmate the unknowns by mnmzng the cost functon J 3 (α, v)= (α j v I j ) 2 = ᾱ G LP v 2. (13) V j W V If we assume that the alpha mattes of the pxels n the patch W are known, the coeffcents v for each patch W can be estmated by mnmzng the functon J 3 (α, v), as v =argmn v G LP v ᾱ 2 = (G LP G LP ) 1 G LP ᾱ.(14) Substtutng the expresson for v from (14), we see that the cost functon J 3 (α, v) can be reduced to a cost functon dependent on the alpha mattes only, as J 3 (α) = V [ = α L 3 α. ᾱ (I W G LP (G LP G LP ) 1 G LP )ᾱ (15) As earler, the constructed cost functon s quadratc n the alpha mattes. Therefore, the alpha mattes can be estmated n closed form by solvng a lnear system. What s the null space? We now repeat the exercse of nspectng the nature of the solutons to ths system for a small patch W n the mage. M LP = I W G LP (G LP G LP Let us defne a matrx ) 1 G LP. By an earler argument, we know that the columns of the matrx G LP and ther lnear combnatons, are null vectors of the matrx M LP. Recall from Theorem 1 that ᾱ = G LP v. Therefore, we can conclude that the vector of true alpha mattes les n the null space of M LP. Snce G LP s rank 3, we have that the null space of M LP s of dmenson 3. When the mage data exactly obeys the lne-pont model, the RGB ntenstes le on a plane spanned by the model parameters F, B 1 and B 2. Snce the locus of any pont x on the plane can be expressed n terms of the perpendcular to the plane d R 3 as d x = 1, we note that there exsts a lnear functon d such that j W : d I j = 1. Consequently, the matrx G constructed by Levn et al. [6 s also rank 3, because the last column comprsng of all 1s can be expressed as a lnear combnaton of the frst 3 columns that contan the mage ntenstes. Hence, the null space of the Mattng Laplacan s also 3 dmensonal. Note that n the statement of Theorem 1, we have mentoned that there must be at least three pxels n the wndow a, b, c W such that α a α b α c and (1 α a )β a (1 α b )β b (1 α c )β c. Ths corresponds to the condton that the composte ntenstes completely span the plane defned by the foreground color pont and the background color lne. However, when the alpha mattes of all the pxels n a wndow are constant,.e. j W, α j = k [0, 1,

5 the ntenstes span only a subset of the plane defned by the foreground color pont and the background color lne. Specfcally, they span a lne on ths plane and hence the rank of G LP s 2 n ths case. Snce ths lne s a part of the plane dscussed above, all the composte ntenstes do stll satsfy the locus j W : d I j = 1. Hence, we see that our constructed cost functon naturally allows for locally constant alpha mattes, because there exsts a lnear functon kd R 3 such that j W, α j = (kd) I j = k(d I j ) = k. Ths s an mportant property when dealng wth trmaps that are not tght. What happens on real, nosy mages? Unfortunately, real data s always corrupted by some nose. In case the mage data has a slght perturbaton from the exact color-lne model, the ntenstes do no longer le on a plane. Hence, there s no d R 3 such that j W : d I j = 1, and the matrx G constructed by [6 s rank 4. As a result, the null space of the Mattng Laplacan s rank 4. The null space obtaned usng our framework, however, stll has a null space of dmenson 3, by constructon. Snce the frst 3 columns of G are exactly the same as G LP, the column span of G LP s a subset of the column span of G. Therefore, the null space obtaned usng our framework s a strct subset of the null space constructed by [6. Ths mples that our proposed model s more compact than that of [6. Hence, the key observaton s that the extra degree of freedom of the Mattng Laplacan s used to explan mage nose. What s the bas? The space of solutons for the alpha matte gven by our model or the model of [6 s typcally qute large (see [7). However, there s an mplct bas towards the result gven by the lnear solver. In fact, ths bas s enforced naturally by the structure of the dfferent cost functons. Whle [6 naturally bases the mattes to be locally constant, our new cost functon pushes the alpha mattes towards 0. By constructon, the Mattng Laplacan L has the vector wth all equal entres, as ts trval null vector. Therefore, [6 s based towards estmatng locally smooth alpha mattes. On the other hand, our cost functon L 3 s a postve sem-defnte matrx and not necessarly a Laplacan matrx. It has a trval null vector whch has all entres equal to 0, and consequently our algorthm estmates alpha mattes wth a bas towards 0. Note that by solvng for 1 α rather than α, we can also bas the alpha mattes towards 1. Result on toy data. Fgure 1 llustrates the advantage of our proposed model. The yellow foreground s a pont n RGB space, and the background les on a color lne, varyng from lght to dark blue. Hence, we have a perfect lne-pont color model. We add some nose to the composte mage n order to slghtly perturb t from ths model. Notce that [6 produces erroneous alpha mattes due to ts larger null space, and our method recovers a much better alpha matte. As expected [6 has a bas towards locally smooth mattes, and careful nspecton shows that our result has a tny shft (a) Composte mage (b) Ground truth alpha (c) Trmap (d) Our result (e) Result of [6:ɛ = 0 (f) Result of [6:ɛ =10 6 Fgure 1. Comparson of our proposed framework wth that of [6 for the lne-pont case. towards 0. Furthermore, note that the trmap s not very tght, and our method correctly recovers those pxels whch should be truly 0 or 1. for nsuffcent user nput. We now demonstrate that our algorthm can recover the alpha mattes even when the user provdes scrbbles for only one of the layers. Recall that snce [6 prefers locally constant mattes, t wll produce a result wth all pxels havng α = 0 or α = 1. Therefore, for a far comparson, we propose a new verson of [6, n order to bas the alpha mattes towards 0. In partcular, we estmate the alpha mattes by mnmzng the cost functon J ɛ (α, v)=j(α, v)+ɛ V ( ) a R 2 + a G2 + a B2 + b 2. (16) In ths modfcaton, we are basng the affne models of (3) towards (0, 0, 0, 0). It can easly be checked that we can elmnate the unknown affne models n a smlar fashon as descrbed earler, and obtan a closed form soluton for the alpha mattes. We hence have a formulaton whch can potentally estmate the alpha mattes even when the user provdes scrbbles for one of the layers only. Fgure 2 shows a toy example for the lne-pont color model where the user marks scrbbles for the foreground only. We are able to recover vsually pleasng alpha mattes, snce our natural bas towards 0 s the desred bas n ths case. However, we do not get good results wth our proposed modfcaton of [6 even when we ncrease ɛ n (16). As dscussed, these resultng alpha mattes are based to be locally smooth due to the nature of the Mattng Laplacan Pont-Pont Color Models We now consder the case when the colors of both the layers are constant and hence satsfy the color pont model. By the hypothess, j W, F j = F and j W, B j = B. Therefore, the composte ntensty I j of a pxel j W can be expressed as j W : I j = α j F + (1 α j )B.

6 (a) (b) (c) (d) (e) (f) (g) Fgure 2. Comparson of our proposed framework wth that of [6, when the user provdes scrbbles for one layer only. (a) Composte mage (b) Ground truth alpha (c) Trmap (d) Our result (e) (g) Result of [6: ɛ = 0, ɛ = 10 4 and ɛ = 10 2 respectvely For ths scenaro, we derve two mportant results as gven by Theorem 2. Theorem 2 Consder an mage patch W around a pxel V, such that the RGB ntenstes of the pxels n the patch, satsfy the pont-pont color models. Defne [ a matrx G P P R W 3, whose rows are gven as I R j Ij G Ij B, j W. that for each pxel j n the patch W, I j = [ F B B [ α j 1 = [ [ [ 0 1 αj I 0 d fb. λ f λ b λ b 1 }{{} Λ (19) Defne the projecton functon as Π(I j ) = Ĩj = [Ĩ1 Ĩ 2 [ = I0 d fb Ij, where A denotes the pseudo-nverse of a matrx A. We then have j W : Ĩj = Λ [ αj 1 = [ αj 1 = Λ 1 Ĩ j. (20) Hence, we conclude that the alpha matte of each pxel s gven by a lnear combnaton of the pxels projected ntenstes, the coeffcents of combnaton beng unquely gven by the frst row of Λ If the alpha mattes of all the pxels n the patch are not equal and the color ponts of the two layers are dstnct, then Rk(G LP ) = If Rk(G P P ) = 2, there exsts a projecton Π : I R 3 Ĩ R2, such that the alpha matte α j of each pxel j W can be expressed as a lnear functon of the projected ntenstes Ĩ R 2, va unque coeffcents v = (a 1, a2 ) R2 characterstc to W, as Proof. j W : α j = a 1 Ĩ1 + a 2 Ĩ2. (17) 1. Note that by defnton, we have G P P =.. [ α j 1 F B B.. } {{ } Γ }{{} H. (18) Snce by hypothess, F B, the matrx H n (18) s full rank, and hence of rank 2. If the alpha mattes of all the pxels are not equal, then matrx Γ n (18) s also of rank 2. Hence, the matrx G P P has rank Let the color lne passng through the foreground and background ntenstes F and B be gven by d fb. Therefore, there exst parameters I 0 R 3 and λ f, λ b R such that F = I 0 + λ f d fb and B = I 0 + λ b d fb. Now the compostng equaton (1) states As earler, we defne a matrx G R W 2, the rows of whch are gven by Ĩj, j W, and also a matrx ᾱ R W, whose entres are gven by α j, j W. The problem of fndng the unknown alpha mattes can then be posed as one of mnmzng the cost functon J 2 (α, v)= (α j v Ĩj) 2 = ᾱ G v 2.(21) V j W V Recall that the coeffcents v for each patch W can be estmated n closed form as v = argmn G v ᾱ 2 = ( G v G ) 1 G ᾱ. (22) Substtutng the expresson for v from (22), we see that the cost functon J 2 (α, v) can be reduced to a cost functon dependent on the alpha mattes only, as J 2 (α) = [ ᾱ (I W G ( G G ) 1 G )ᾱ V (23) = α L 2 α. Snce the constructed cost functon s quadratc n the alpha mattes, the alpha mattes can be estmated n closed form by solvng a lnear system. What s the null space? We know that for each small mage patch W, snce G s of rank 2, the null space of the matrx M = I W G ( G G ) 1 G s of rank 2. Now, we can proceed as we dd n the rank 3 case, and verfy that the column span of G s a subset of the column span of the matrx G employed by [6. Consequently, the 2 dmensonal null space constructed by our framework s a subset of the

7 (a) Composte mage (b) Ground truth alpha (c) Trmap (d) Our result (e) Result of [6:ɛ = 0 (f) Result of [6:ɛ =10 6 Fgure 3. Comparson of our proposed framework wth that of [6 for the pont-pont case. 4 dmensonal null space constructed by [6, and our proposed model for the alpha mattes s more compact than that of [6. Also, snce the locus of a pont (x, y) on a lne can be represented as c 1 x + c 2 y = 1, we can always fnd models (a 1, a2 ) such that our framework admts locally constant solutons. It s also easy to show that, as for the rank 3 case, the Laplacan of [6 s rank 2 for nose free data. on toy data. Fgure 3 gves a toy example to llustrate the advantage of our proposed model. The yellow foreground and blue background consttute dstnct ponts n RGB space. Therefore, ths scenaro corresponds to the pont-pont color model. We add some nose to the composte mage. The conclusons are the same as n Fgure 1,.e. [6 produces a soluton whch s worse than ours. Agan, observe that [6 has based towards locally smooth mattes, whle ours has a small bas towards Experments In ths secton, we present a quanttatve and qualtatve comparson of our proposed framework wth that of [6, and show that our formulaton helps to estmate better mattes. Frst, we gve the detals of our numercal mplementaton and then present an analyss of our tests Numercal Implementaton Lke n [6, we consder mage patches of sze 3 3. Note that we need to estmate the rank of each patch W, and for ths, we frst construct a matrx G, the rows of whch are gven by [ I R j I G j I B j 1, j W. We then estmate the sngular value decomposton of ths matrx as G = UΣV, where the dagonal entres of Σ are gven by σ 1 > σ 2 > σ 3 > σ 4. Now, we normalze the sngular values as Σ = λσ, where λ = (σ σ σ σ 2 4) 0.5. Ths normalzaton ensures that the rank estmaton s not senstve to scale varatons n the mage. We nspect the normalzed sngular values σ = λσ, and estmate the rank as rank(w )= argmax k [ σ k > δ, where δ s a pre-defned tolerance value. We use δ = n all our experments. Gven the rank of a patch W, we choose the approprate cost functon C for the patch as dscussed n Secton 3. In partcular, f the rank of W s 2, 3 or 4, we construct the matrx C of sze W W as C = L 2 n (21), C = L 3 n (13), or C = L n (4) respectvely, by restrctng V =. In the case of rank 1, we construct C usng the cost functon of (4), whch essentally forces the alpha mattes n the patch to be equal. We then defne a vector ᾱ R W, the entres of whch are gven by {α j }, j W. We therefore need to estmate the alpha mattes of the mage by mnmzng ᾱ C ᾱ = α Lα, where α s the vector contanng the alpha mattes of all the pxels n the mage. Note that L s a postve sem-defnte matrx of sze α α obtaned by aggregatng the matrces C. Now, we need to mnmze ths cost functon subject to the constrants that the set of pxels scrbbled as foreground (say F) have α = 1 and the set of pxels scrbbled as background (say B ) have α = 0. As shown n [14, the soluton to ths problem α = argmn α Lα α s.t. α = 0 f F, and α = 1 f B. (24) s equvalent to solvng a lnear system. Hence, we have a new closed form soluton for the alpha mattes of an mage We perform our evaluaton on the database used n [10, whch contans 27 hgh qualty mages. For the purpose of testng, we dlate the perfect trmap by 22 pxels. Fgures 4 6 are typcal examples. In what follows, we compare the performance of the followng 4 algorthms: (a) Rank-adaptve: our proposed algorthm n whch we construct the cost functon by analyzng the rank of each mage patch; (b) Rank-adaptve-mod: a modfcaton of our algorthm, where we treat all rank 2 patches as rank 3; (c) Levn: the algorthm of [6; and (d) Levn-mod: the algorthm of [6 wth our proposed modfcaton of basng the mattes towards 0 as n (16), but only for those connected regons n the trmap whch have only 1 as boundary condtons. Note, for Levn-mod we tred dfferent values for ɛ n eqn. (16) and selected the best as ɛ = The performance of each algorthm s evaluated usng the followng three dfferent metrcs. Gven the computed α matte and the ground truth α, we compute the metrcs SAD: α α, MSE: (α α )2, and gradent error: ( α α )2. Table 1 shows these errors for dfferent methods (averaged over all test cases). The best result for each metrc s hghlghted n bold. We note that Levn-mod obvously outperforms Levn. Interestngly, Rank-adaptve-mod performs better than Rank-adaptve. Vsual nspecton shows that the bas towards 0 s more pronounced for the rank 2 case than for

8 rank 3. Ths ponts towards the fact that for real mages, our rank 3 formulaton can account for the rank 2 cases also and has much more stable performance. Note that ths s not a drawback of our formulaton, snce we can have a black box for rank estmaton, whch always gves values of 3 or 4. Importantly, the modfcaton of our algorthm Rank-adaptve-mod performs better than Levn as well as Levn-mod n 2 out of the 3 metrcs. These mprovements can also be observed vsually, snce these three error metrcs may not be representatve of the error observed by a human. Table 1. Mean errors for the estmaton of alpha mattes Method Levn Levn-mod Rank-adaptve Rank-adaptve-mod SAD MSE Gradent Fgures 4 6 show typcal results obtaned n the above error analyss. We show the results of Levn-mod and Rank-adaptve-mod snce these algorthms rank the best n the error metrcs. In the mages dsplayng the rank estmated by us, we use the followng color-codng: dark blue - marked pxels, lght blue - rank 1, green - rank 2, orange - rank 3, and red - rank 4. It s mportant to note that most of the unmarked pxels n the trmap have rank less than 4. In spte of ntroducng a bas towards 0 alpha mattes, Levn-mod cannot deal wth holes n the trmap. Ths s due to ts nherent bas to estmate locally smooth alpha mattes. Our method, however, has no such problem and s able to recover vsually pleasng alpha mattes. Moreover, n spte of an nherent bas of our method towards 0 alpha mattes, our algorthm s able to accurately estmate 1 alpha mattes n several regons of the trmap, such as the boundary of the ball, the grl and the leaves n Fgures 4 6. We now address the ssue of usng scrbbles vs. trmap as user nteracton. In Fgure 7, we see that when we use the same scrbbles as used n [6, our framework s able to capture fner detals of the alpha matte of the dandelon, as compared to [6. On the other hand, n Fgure 8, we see that when we use the same scrbbles used n [6, our method gves suboptmal performance. Specfcally, due to the nherent bas, our method tres to ft fractonal alpha even though the true alpha matte s 1 n large portons of the bear. However, ths can be easly fxed by connectng the scrbbles and flood fllng them to create a trmap, as shown n Fgure 8(d). Our result (Fg. 8(f)) wth ths trmap s comparable to that of [6 (Fg. 8(e)). In general, our method outperforms [6 when the user nput s a trmap, whch as exhbted n our experments, need not be tght. Note that ths s not a lmtaton, snce recent methods such as [1, 10, 9, 20 also use the gven scrbbles to generate a trmap, and then estmate the alpha mattes usng ths trmap. (a) Image (d) Estmated rank (b) Ground truth alpha (e) Levn-mod (c) Trmap of (f) of Rank-adaptve-mod Fgure 4. Comparson of our framework vs. [6 on an mage of a ball (a) Image (d) Estmated rank (b) Ground truth alpha (e) Levn-mod (c) Trmap of (f) of Rank-adaptve-mod Fgure 5. Comparson of our framework vs. [6 on an mage of a grl (a) Image (d) Estmated rank (b) Ground truth alpha (e) Levn-mod (c) Trmap of (f) of Rank-adaptve-mod Fgure 6. Comparson of our framework vs. [6 on an mage of a plant

9 (a) Image + scrbbles (b) Result of Levn (c) Result of Rank-adaptve-mod Fgure 7. Comparson of our framework vs. [6 on a dandelon s mage. (a) Image + scrbbles orgnally used n [6 (d) Trmap va connectng scrbbles+floodfll (b) Result of Levn (c) Result of Rank-adaptve-mod (e) Result of Levn (f) Result of Rank-adaptve-mod Fgure 8. Comparson of our framework vs. [6 on an mage of a bear. 5. Conclusons In ths work, we have presented new appearance models for the problem of mage mattng. By constructon, these appearance models are more compact than that proposed by [6, and as shown n our analyss, outperform the tradtonal color lne model of [6, wthout the need of any addtonal user nteracton. Future work entals the need of closed form solvers for the mattes of mage patches that have complex ntensty varaton and hence do not satsfy the color lne model or the color pont model. [6 A. Levn, D. Lschnsk, and Y. Wess. A closed-form soluton to natural mage mattng. IEEE Trans. Pattern Anal. Mach. Intell., 30(2): , , 2, 3, 4, 5, 6, 7, 8, 9 [7 A. Levn, A. Rav-Acha, and D. Lschnsk. Spectral mattng. IEEE Trans. Pattern Anal. Mach. Intell., 30(10): , , 2, 3, 5 [8 I. Omer and M. Werman. Color lnes: Image specfc color representaton. In CVPR (2), pages , [9 C. Rhemann, C. Rother, and M. Gelautz. Improvng color modelng for alpha mattng. In BMVC, , 2, 3, 8 [10 C. Rhemann, C. Rother, A. Rav-Acha, and T. Sharp. Hgh resoluton mattng va nteractve trmap segmentaton. In CVPR, , 2, 3, 7, 8 [11 C. Rother, V. Kolmogorov, and A. Blake. GrabCut : Interactve foreground extracton usng terated Graph Cuts. ACM Trans. Graph., 23(3): , [12 M. A. Ruzon and C. Tomas. Alpha estmaton n natural mages. In CVPR, pages , [13 D. Sngaraju, C. Rhemann, and C. Rother. New appearance models for natural mage mattng. In Techncal Report, Mcrosoft Research, Cambrdge, [14 D. Sngaraju and R. Vdal. Interactve mage mattng for multple layers. In CVPR, [15 J. Sun, J. Ja, C.-K. Tang, and H.-Y. Shum. Posson mattng. ACM Trans. Graph., 23(3): , [16 J. Wang, M. Agrawala, and M. F. Cohen. Soft scssors: An nteractve tool for realtme hgh qualty mattng. ACM Trans. Graph., 26(3):9, , 2 [17 J. Wang and M. F. Cohen. An teratve optmzaton approach for unfed mage segmentaton and mattng. In ICCV, [18 J. Wang and M. F. Cohen. Image and vdeo mattng: A survey. Foundatons and Trends n Computer Graphcs and Vson, 3(2), [19 J. Wang and M. F. Cohen. Optmzed color samplng for robust mattng. In CVPR, , 2, 3 [20 Y. Zheng, C. Kambhamettu, J. Yu, T. Bauer, and K. Stener. Fuzzymatte: A computatonally effcent scheme for nteractve mattng. In CVPR, , 8 References [1 X. Ba and G. Sapro. A geodesc framework for fast nteractve mage and vdeo segmentaton and mattng. In ICCV, , 8 [2 Y.-Y. Chuang, B. Curless, D. Salesn, and R. Szelsk. A Bayesan approach to dgtal mattng. In CVPR (2), pages , [3 L. Grady, T. Schwetz, S. Aharon, and R. Westermann. Random walks for nteractve alpha-mattng. In Proceedngs of the Ffth IASTED Internatonal Conference on Vsualzaton, Imagng and Image Processng, pages , [4 Y. Guan, W. Chen, X. Lang, Z. Dng, and Q. Peng. Easy mattng: A stroke based approach for contnuous mage mattng. Eurographcs, 25(3): , [5 E. Hsu, T. Mertens, S. Pars, S. Avdan, and F. Durand. Lght mxture estmaton for spatally varyng whte balance. ACM Trans. Graph., 27(3),