Pattern Recognition Letters

Transcription

1 Pattern Recognition Letters 33 (2012) Contents lists available at ScienceDirect Pattern Recognition Letters journal homepage: Dynamic curve color model for image matting Sunyoung Cho, Hyeran Byun Department of Computer Science, Yonsei University, Shinchon-Dong, Seodaemun-Gu, Seoul , Republic of Korea article info abstract Article history: Available online 18 April 2011 Keywords: Image matting Dynamic curve color model Bézier curve Alpha estimation Image matting is the process of estimating the foreground and background elements from a single image with limited user input. To solve this severely under-constrained problem, there exist various methods to construct color models for an image. Most previous color models can fail to estimate accurate mattes for complex images of nonlinear color distributions due to their simple color models. In this paper, we present a new dynamic curve color model for image matting that can handle nonlinear color distributions. We show that the colors of a local region can be fit to a curve when the local region includes three types of colors foreground, background, and unknown mixed colors. Based on these colors in the local region, we adaptively construct a curve color model using a quadratic Bézier curve model. Our curve model allows the derivation of a new closed-form matting equation for estimating alpha values of colors forming a curve. We show that our method estimates alpha mattes more accurately than recent existing methods through visual and quantitative comparisons. Ó 2011 Elsevier B.V. All rights reserved. 1. Introduction Image matting is the task of extracting the foreground and background elements from a single image by estimating the opacity called alpha for each pixel in the image. It is an important task in many image and video editing applications, because it can be used to accurately compose the foreground element into a new background scene. In general, the ith color of the input image C i is modeled as a linear combination of the corresponding colors of a foreground image F i and a background image B i, C i ¼ a i F i þð1 a i ÞB i where a i is the pixel s opacity component used to linearly blend the foreground and background. This task is a severely under-constrained problem because it has seven unknowns (three foreground colors, three background colors, and one foreground opacity) for a three-channel color image. Therefore, most previous matting approaches require additional information for the input image, i.e., the user-specified image, which indicates the foreground (a = 1), background (a = 0), and unknown (a = [0, 1]) regions. They estimate the alpha matte by constructing the color model from the information of a user-specified image. Initial methods in image matting use algorithms based on various color models (Chuang et al., 2001; C. CORPORATION, 2002; Mishima et al., 1993; Ruzon and Tomasi, 2000). They construct Corresponding author. Tel.: ; fax: addresses: sycho22@yonsei.ac.kr (S. Cho), hrbyun@yonsei.ac.kr (H. Byun). ð1þ their color model by collecting the nearby known foreground and background samples for each unknown pixel. Ruzon and Tomasi (2000) construct a non-oriented Gaussian color model for a local unknown region. They create a local window by dividing the skeleton of the unknown region into intervals and model the local window colors as a mixture of isotropic Gaussians. This color model produces large fitting errors for textured regions and discontinuities in the final alpha matte. Bayesian matting (Chuang et al., 2001) uses a mixture of oriented Gaussian models for foreground and background color distributions. Its results are better than those in Ruzon and Tomasi (2000) because the model includes not only unknown color, but also foreground and background colors to model the Gaussians. However, it fails to estimate an accurate alpha matte when the color model assumption is not satisfied and a coarse trimap is given. The knockout system (C. CORPORATION, 2002) uses nearby known colors by weighing spatial distances between unknown and known pixels. Overall, these initial methods fail to estimate an accurate alpha matte because an erroneous sample set can be constructed when there is complex variation in the images. To solve the problems of initial sampling-based color models, some successful methods have been proposed for the analytical modeling of color distributions (Grady, 2006; Levin et al., 2008; Singaraju et al., 2009; Sun et al., 2004; Wang et al., 2007). They use local image statistics by defining the affinities between neighbor pixels. Poisson matting (Sun et al., 2004) formulates the matting problem as solving Poisson equations and estimates the alpha matte using the matte gradient field and Dirichlet boundary conditions. However, the method still produces an erroneous /$ - see front matter Ó 2011 Elsevier B.V. All rights reserved. doi: /j.patrec

2 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) matte for complex foreground and background patterns. In closed form matting (Levin et al., 2008), locally linear colors are assumed in the foreground and background of a small image region. Based on this assumption, the matting cost function is defined and the matting Laplacian is analytically derived by eliminating the foreground and background color components. Eventually, a closed form solution to image matting is produced by analyzing the matting Laplacian. However, this method has an overfitting problem in locally constant regions. Singaraju et al. (2009) improve the performance of Levin et al. (2008) by modeling the new appearance models (line point and point point color models), demonstrating that the color line model can be overfit in the case of locally constant color distributions. Consequently, the new color model is proposed to consider a case in which the color layer lies on a point. Although these analytic color models produce a higher quality matte in complex images than sampling-based methods, large errors can be produced due to propagation and accumulation. Recently, various challenging and successful methods have been proposed by combining sampling-based and affinity-based color models (Guan et al., 2006; Rhemann et al., 2008; Wang and Cohen, 2005, 2007; Zheng and Kambhamettu, 2009). The iterative matting approach (Wang and Cohen, 2005) constructs a global color model based on known foreground and background colors. The method models an alpha matte as a Markov Random Field (MRF) and optimizes the alpha matte using iterative belief propagation. Although it does not require a well-specified trimap, the method has a limitation for ambiguous image colors and high time complexity. The robust matting (Wang and Cohen, 2007) results in an improved performance by combining the optimized color sampling scheme with the matting affinity defined in Levin et al. (2008). This method uses confidence values in order to balance the data cost and neighborhood cost in the color sampling step. As a result, it produces better mattes by yielding a low confidence value for bad sampling, however it produces an inaccurate matte for long and furry foreground structures. Zheng and Kambhamettu (2009) construct the alpha-color model based on the learning of local color distributions. They improve the performance by handling nonlinear as well as linear color distributions using the kernel trick. The shared matting (Gastal and Oliveira, 2010) improve the speed as well as accuracy from observation that pixels in small neighborhood tend to share similar attributes. In this paper, we propose a new dynamic curve color model for image matting that is the combination of sampling-based and affinity-based methods. We assume that the colors of a local region containing foreground, background and unknown mixed pixels form a curve in RGB color space. From this assumption, we first generate a local sample window which satisfies our assumption. After that, we construct a curve color model by defining the affinity in the local sample window using Bézier curve model and derive a new matting equation based on the colors within the local sample window. Since our color model is not a fixed but a dynamic model, more accurate mattes can be produced for images which do not fit into a single fixed color model. In addition, our method provides a closed-form solution for image matting by directly estimating the alpha of pixels within local region. In Section 2, we present the limitations of the previous color models on image matting. Section 3 provides a derivation to our dynamic curve color model from an assumption of color distributions in a local image region. In Section 4, we construct the proposed curve color model and derive a new matting equation. Section 5 presents an experimental evaluation of our color model via comparison with other color models. Finally, the paper is concluded in Section Limitations of the previous color models We first describe the limitations of the previous color models. We consider the previous sampling- and affinity-based methods and focus on the linear color model (Levin et al., 2008; Wang and Cohen, 2007) for affinity. We also briefly describe our idea that addresses the limitations noted below. 1. Simple color model: Most sampling-based methods consider the distance between unknown pixel and known foreground Fig. 1. The RGB plots of color distributions on local image regions. (a) Original images with white marked local region. (b) Magnified local image regions. (c) Color plots on local regions in RGB color space.

3 922 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) and background sample sets for alpha estimation. Therefore, this approach can be simple to represent the complex patterns of images because the matte result can be different based on the sample set and distance metrics. The linear color model does not often represent the image data when an image is too simple or complex in intensity variation. The color point model (Singaraju et al., 2009) tries to solve the overfit problem of the color line model in the case of simple intensity variations. However, the color point model also does not handle complex intensity variations. Consequently, the previous color models produce inaccurate mattes for complex image intensity variations. In order to solve the problem, we propose a curve color model that can handle nonlinear color distributions. Our model estimates more accurate alpha mattes using an affinity-based color model adapted to the image data and sample set. 2. Erroneous local region: The sampling-based methods collect the nearby foreground and background colors for each unknown pixel in order to construct the color model for the local region, resulting in an erroneous matte due to an improper sample set. The affinity-based methods consider the color models for a fixed local region, such as a 3 3 window, which may not satisfy their color models. Consequently, previous methods produce erroneous results from color models of the erroneous local region. We simply solve this problem using a data-driven approach. Our method determines the local region which satisfies our color model and then applies the color model to the local region. Each local region can have a different size and color composition according to the data of a user-specified image and input image. Therefore, our method reduces the incidence of erroneous results obtained from an erroneous local region. 3. Derivation to a dynamic curve color model Fig. 2. The conceptual sketches representing the similarity between the quadratic Bézier curve and our curve color model. (a) The quadratic Bézier curve. (b) Our curve color model. As mentioned above, since matting is a severely underconstrained problem, additional information or assumptions are Fig. 3. Alpha matte results extracted by our method and previous methods. (a,j,s) Input image, (b,k,t) Trimap, (c,l,u) Our method, (d,m,v) Bayesian, (e,n,w) Closed-form, (f,o) New-appearance, (g,p,x) Robust, (h,q,y) Learning-based, (i,r,z) Shared.

4 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) Fig. 3 (continued) required. We assume that color distributions in a local region can be approximated to a curve shape. The local region consists of foreground color F, background color B, and unknown mixed colors between F and B. In a matting problem, the colors of the unknown pixels are mainly mixed colors. In Fig. 1, we illustrate the validity of the assumption by plotting RGB color distributions on local image regions. As shown, colors in the local white marked region have a curved form in the RGB color space, in which the ends correspond to F and B, respectively, and the center plots are mixed colors between F and B. Therefore, we make the following assumption on image matting. Assumption (Curve color model). The color distributions of the local image region including foreground, background and mixed colors can be approximated by a curved shape in the RGB color space. Levin et al. (2008) and Singaraju et al. (2009) assume the line or point color model in the local image region and estimate the alpha matte using the matting Laplacian derived from an affine function under the assumption. Different from their approach, we simply and efficiently construct the curve color model and derive the matting equation to directly estimate the alpha matte. First, in order to approximate colors in the local region to the curve, we apply a Bézier curve model. The Bézier curve is a parametric curve that is widely used to model smooth curves. The advantage of using a Bézier curve is that it works similar to the matting technique, i.e. a curve is determined as t varies from 0 to 1 as an alpha is determined in the range of 0 and 1 in the matting technique. In particular, a quadratic Bézier curve is used because a curve with three control points is appropriate to represent colors in the local region, including our three types of colors, i.e., F, B, and mixed unknown. In the quadratic Bézier curve, the second control point is changed based on the data and the curve is changed along with the second control point. Our curve model is also adaptively constructed according to the color distributions as if a Bézier curve changes according to the data and second control point. Therefore, we call our color model a dynamic curve color model and arrive at the following theorem. Fig. 2 shows the concept of above description. Theorem (Dynamic curve color model for image matting). The dynamic curve color model works similar to a quadratic Bézier curve. From the curve color model assumption, the local image region can be represented by the quadratic Bézier curve, and the curve can be dynamically changed according to the color distributions of the local image region. In this work, we first generate a local sample window, which includes F, B, and unknown mixed colors between F and B that satisfy our color curve assumption. After that, we construct a curve color model that approximates colors in the sample window using the

5 924 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) Fig. 3 (continued) quadratic Bézier curve. Finally, a new matting equation is derived to estimate the alphas of curve-forming pixels. 4. Construction of a dynamic curve color model 4.1. Review of a quadratic Bezier curve We begin with a brief review of the quadratic Bézier curve. The quadratic Bézier curve is generally represented as follows: BðtÞ ¼ð1 tþ 2 P 0 þ 2tð1 tþp 1 þ t 2 P 2 ; t 2½0; 1Š where B(t) is an interpolated point at parameter value t and P 0, P 1, and P 2 are three control points. The curve begins at P 0 (t = 0) and ends at P 2 (t = 1) and the intermediate points between P 0 and P 2 are determined by varying t from 0 to 1. P 1 influences the shape of the curve; for instance, if P 1; is close to P 0 (or P 2 ), then the curve leans toward P 0 (or P 2 ). Note that this property is similar to the matting technique. If the local image region contains more colors close to the F (or B), then the curve color model leans toward F (or B) Construction of a dynamic curve color model To construct a dynamic curve color model, we first generate a local sample window which satisfies our assumption. For this, we ð2þ first identify the nearest foreground and background pixels P i F and P i B for each unknown pixel Pi U, and then compute a line joining P i F and Pi B. Subsequently, Pi F, Pi B, and all unknown pixels on the line are used to generate the local sample window. Here, all unknown pixels on the line have mixed colors between C i F of Pi F and Ci B of Pi B. Now the local sample window contains three types of colors, F, B, and mixed colors between F and B. Note that each local sample window consists of a different size and color composition according to the data obtained from user-specified images and input images. Next, we set P 0 as C i B and P 2 as C i F in the equation of the quadratic Bézier curve in order to derive a matting equation with parameter a from the Bézier curve equation with parameter t. In other words, we use the fact that B and F have a values of 0 and 1 to indicate that P 0 and P 2 have t values of 0 and 1, respectively. Now, we need to determine P 1 in the equation of the quadratic Bézier curve. First, we show that P 1 can be determined using the point on the curve. If we know P 0, P 2, and a point on the curve, P 1 can be computed according to the following procedure. For Eq. (2) of the quadratic Bézier curve, a point Q on the curve can be substituted as follows: Q ¼ð1 qþ 2 P 0 þ 2qð1 qþp 1 þ q 2 P 2 where q is a t value of Q. Then, Eq. (3) can be rewritten as ð3þ

6 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) Fig. 3 (continued) 1 1 q P 1 ¼ Q 2qð1 qþ 2q P q 0 2ð1 qþ P 2 Now, we can compute a unique P 1 using Eq. (4). Similar to this, we use a pixel within the local sample window to determine P 1. We assume that there exists a pixel P i M within each sample window with an intermediate color between P i B and P i F, i.e., Ci M ¼ðCi B þ Ci FÞ=2. This assumption is reasonable because mixed colors between F and B exist in the sample window. Then, we can substitute C i B, Ci F, and the color Ci M of Pi M into Eq. (2). C i M ¼ð1 m iþ 2 C i B þ 2m ið1 m i ÞP i 1 þ m2 i Ci F where m i is the t value of C i M (0 < m i < 1) and is determined as jc i ~ B C i M j=jci B Ci F j where XY is the distance between X and Y, and C ~ i M is the most similar color to the C i M among the colors within the local sample window. Considering these representations, Eq. (5) can be rewritten as: ð4þ ð5þ 4.3. Derivation of a matting equation We now derive our matting equation based on the above curve color model. We first define a new matting equation using the equation of the quadratic Bézier curve: I i ¼ð1 a i Þ 2 P i 0 þ 2a ið1 a i ÞP i 1 þ a2 i Pi 2 ; a i 2½0; 1Š where I i is the color of an unknown pixel within the local sample window that satisfies our curve model, and P i 1 is adaptively determined using Eq. (7) and colors in the sample window. Eq. (8) can be rewritten as: I i ¼ a 2 i Ci F þ 2a ið1 a i ÞP i 1 þð1 a iþ 2 C i B As mentioned above, C i F and Ci B are the colors of the nearest F and B; pixels in each sample window, respectively. The above equation can be rewritten for a i as: ð8þ ð9þ 2m i ð1 m i ÞP i 1 ¼ /where/ ¼ Ci M ð1 m iþ 2 C i B m2 i Ci F ð6þ ðc i F 2P 1 þ C i B Þa2 i þ 2ðP i 1 Ci B Þa i þðc i B I iþ¼0 ð10þ Hence, P i 1 can be determined as: P i 1 ¼ / 2m i ð1 m i Þ P i 1can be changed based on the colors in the sample window. This is to show that each curve color model can be adaptively constructed according to the input data pixels. ð7þ By applying the quadratic formula into Eq. (10), we obtain qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi a i ¼ ðci B Pi 1 Þ ðc i B Pi 1 Þ2 ðc i F 2Pi 1 þ Ci B ÞðCi B I iþ C i F 2Pi 1 þ Ci B ð11þ a i is chosen as positive solution of Eq. (11) when C i F is greater than C i B. Otherwise, a i is chosen as negative solution. Moreover, since a i

7 926 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) Fig. 3 (continued) is dependent upon three channel components, the final alpha can be computed as the weighted sum of each channel component: ^a i ¼ w r a r i þ w g a g i þ w b a b i ð12þ where a r i, ag, and i ab i are alpha values of red, green, and blue channel components, and w r, w g, and w b are alpha weights for each channel in the RGB color, respectively. We experimentally determine the values of alpha weights as 0.26, 0.03, and 0.71 for R, G, and B channel components, respectively, and describe the reason for such values in Section 5.3. Using Eq. (12), we can finally compute an alpha value for color I i of an unknown pixel in the sample window without estimating a reliable F or B Comparison to learning-based matting The learning-based matting method (Zheng and Kambhamettu, 2009) estimates mattes using an alpha-color model obtained by learning the local color distributions. This method assumes that the alpha value can be represented by a linear combination of the alpha values of its neighboring pixels. Therefore, the method computes the linear combination coefficients which fit well with the alpha-color model using local color distributions. The method can handle nonlinear local color distributions by using the kernel trick. Our method can also handle the nonlinear color distributions by modeling the color distributions with a Bézier curve. However, there are differences between local learning-based matting and our dynamic curve matting. First, the learning-based matting computes the alpha value by learning the linear combination coefficients for the color distributions in a fixed local region such as a 7 7 patch. Although this learning-based approach provides more improvements than specific color model approaches (Levin et al., 2008; Singaraju et al., 2009), the method can degrade the performance when large fitting errors result from the computation of model coefficients. Unlike the learning-based approach, our approach applies the color model not for the fixed region but for the dynamic region satisfying our color model, producing a more accurate matte because the fitting error is low. Second, learning-based matting computes the alpha by learning the alpha-color model for each unknown pixel, which requires very time consuming operations. However, our method identifies the local region for an unknown pixel and simultaneously computes the alpha values of pixels in the local region. In other words, our method does not require the construction of a color model for every unknown pixel. 5. Experiments In this section, we present visual and quantitative comparisons of our method with the six previous methods which are Bayesian matting (Chuang et al., 2001), Closed-form matting (Levin et al., 2008), New appearance matting (Singaraju et al., 2009), Robust

8 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) Fig. 3 (continued) matting (Wang and Cohen, 2007), learning-based matting (Zheng and Kambhamettu, 2009), and Shared matting (Gastal and Oliveira, 2010). We perform our evaluation on the images provided by alphamatting.com benchmark and the public test sets of Wang Cohen (2007). The trimap is used as a user-specified image, and the white and black regions indicate the foreground and background colors, respectively. The gray region indicates the unknown colors, which are likely to be foreground, background, or mostly mixed colors Visual comparisons We visually compare our algorithm with six previous algorithms on the test images provided by alphamatting.com benchmark which contain highly textured images. Fig. 3 shows the alpha mattes extracted from three test images using our method and compares our results to the results of the previous algorithms. Our method produces more visually pleasing mattes compared to that of previous methods. Commonly, all methods produce poor results in regions where there is an ambiguity between foreground and background color distributions. Furthermore, closed-form matting and learning-based matting cannot deal with holes, which are background regions surrounded by foreground regions, due to the assumption of local smoothness in the alpha matte. The new appearance matting solves the hole problem of closed-form matting by exploiting the color point model. As shown in our experiment, however, the closed-form matting is able to produce an alpha matte that is more visually pleasing than that of the new appearance matting in fine-grained regions. As shown in Fig. 3(a), the new appearance matting fails to produce an accurate matte in fur regions. The robust matting produces the poor mattes because of erroneous color sampling from complex color distributions. Fig. 3(j) shows an example of highly textured background in the region of the elephant trunk. Our method deals with the holes and produces more accurate results on complex patterns of the region of elephant trunk. Fig. 3(s) is an example of highly textured background as well as a fuzzy foreground. All methods fail to estimate accurate mattes on boundary region between book and hair. However, among all methods, the learning-based matting produces the best result and Bayesian matting produces the worst result. Our result is similar to that of the robust matting, however, our method is slightly better than the robust matting on boundary region. Our failure is caused by inaccurate F and B sampling in image of complex variations. In Fig. 4, we compare the mattes obtained from robust matting and our method. Generally, the robust matting produces inaccurate mattes for long and furry foreground structures since they consider the affinity between neighboring pixels. As shown in Fig. 4, the robust matting cannot extract accurate mattes while

9 928 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) Fig. 4. The comparison results between robust matting and our method. (a) Input image. (b) Robust. (c) Our method. our method produces accurate mattes in the ends of blurry hair region. Although our method is a combination of sampling-based and affinity-based approaches such as robust matting, our method outperforms the robust matting in long and furry foreground structures because we consider the affinity between samples which correctly satisfies our color model assumption. Learning-based matting improves performance by learning the nonlinear color distributions better than those in closed-form matting and robust matting which assume linear color distributions. However, the method cannot deal with holes because they estimate the mattes using smoothness modeling such as closed-form matting. The first row of Fig. 5 shows the matte results in holecontaining image regions. We can see that the learning-based matting cannot deal with holes, and furthermore, the method extracts blurry mattes on the ends of leaves. Our method, on the other hand, extracts accurate mattes on the ends of leaves and deals with the holes. The second row of Fig. 5 is an example of image region that more than two colors are roughly included in the background region and foreground and background have similar color distributions. The learning-based matting cannot extract an accurate matte by applying the learning result from inappropriate samples due to their color ambiguity between foreground and background. Fig. 5. The comparison results between learning-based matting and our method. (a) Input image. (b) Magnified region. (c) Learning-based. (d) Our method.

10 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) However, our method collects the samples towards satisfying our color model, and therefore, our method can deal with images of complex color variations. We now show that our color model works in simple images (constant/linear color distributions) as well as complex images (nonlinear color distributions). To show this, we use an image obtained from Singaraju et al. (2009) in which the yellow foreground is a point and the background lies on a line varying from light to dark blue in RGB space, i.e. an image satisfying the line point color model. Fig. 6 shows the matte extracted using our method and compares our result to those of closed-form matting and new appearance matting. The color line model produces an erroneous alpha matte in holes by providing an overfit, while the line point color model results in an accurate alpha matte. Although our model is not designed for simple color variations, unlike that in new appearance matting, our method outperforms the closed-form matting and is comparable to that in new appearance matting. The reason for this finding is that our method adaptively generates a local sample window which satisfies our color model, and the curve can be changed according to the data even with simple color variations. On the other hand, closed-form matting and new appearance matting produce erroneous alpha mattes when the color distributions do not satisfy their color models because they apply their color models to the fixed local window and use color models for simple color distributions Quantitative comparisons We quantitatively evaluate the performance of our method on the training and test images provided by alphamatting.com benchmark and the test set of Wang and Cohen (2007). Given the computed alpha matte a and the ground truth a, we use the following three metrics used by Singaraju et al. (2009): (1) SAD: P i ja i a i j, (2) MSE: P i ða i a i Þ2, and (3) gradient error: P i ðra i ra i Þ2. Table 1 shows the average errors for training images of alphamatting.com benchmark and the test set of Wang and Cohen in different methods. The best result for each metric is highlighted in bold. Bayesian matting has the highest errors in the three metrics, as shown in visual comparisons. Closed-form matting performs better than the new appearance matting in three error metrics. Although the new appearance matting properly treats the holes, the closed-form matting is likely to estimate more accurate alpha mattes in regions other than holes, though the visual difference is often very small. Robust matting has the best performance in the gradient error metric because it can better preserve the gradient of the alpha matte in regions such as hair or fur, though it has lower performance than closed-form matting and learning-based matting in terms of SAD and MSE metrics. Learning-based matting has comparatively good performance in three metrics. Our method performs better than five previous methods in two out of the three metrics. However, our method has quite high error in gradient metric since our method does not consider the affinity Table 1 Average errors of alpha mattes obtained from different methods. Method SAD MSE Gradient Bayesian Closed-form New appearance Robust Learning-based Our method between neighboring pixels, e.g., smoothness assumption in Levin et al. (2008), Wang and Cohen (2007), Zheng and Kambhamettu (2009), while the affinity between pixels in local region is considered. This fact leads to a possibility to improve the performance of our method by considering the alpha relation between neighboring pixels. Tables 2 5 show the evaluation results for test images on the alphamatting.com benchmark. In SAD and MSE metrics, our method has a decent performance for pineapple, plasticbag, and net images. However, overall performance is lower than performance of three images because our method has relatively high errors for remaining images. In gradient error and connectivity error metric, our method has relatively lower performance than SAD and MSE metrics, as aforementioned in Table 1. Although our method has a medium rank in quantitative comparisons, our method produces visually pleasing results through simple and novel algorithm. In our current version, we use the simple method to select the P0 and P2 of a curve and to collect the window samples. Therefore, the performance can be improved by applying the improved sampling method (e.g., robust matting). Also, gradient error or connectivity error can be reduced through matte optimization with smoothness assumption. Furthermore, above error metrics may not be representative of the error observed by a human. As shown in visual comparisons, our method produces the pleasing and comparable mattes compared to the high-rank methods. For instance, our method extracts more accurate mattes than robust matting in long and furry foreground regions. Also, our method handles the hole better than the learning-based matting and closed-form matting. (See Figs. 4 and 5) 5.3. Parameters In Section 4.3, we determined the alpha weights for each RGB channel of the alpha obtained from Eq. (11). In fact, any channel of alpha can be used to determine the final alpha although there is little difference in each channel. However, to improve the performance, we combine the three channel of alpha. To determine the alpha weights, we analyze the alpha matte of the color channel computed by Eq. (11). Fig. 7 shows an example alpha matte in the color channel. In most dataset images, we obtain that erroneous alpha mattes are produced in regions with more G channel Fig. 6. A comparison of results with a line point color image. (a) Input image. (b) Closed-form. (c) New appearance. (d) Our method.

11 930 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) Table 2 The sum of absolute difference errors over different methods. Table 3 The mean squared errors over different methods.

12 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) Table 4 The gradient errors over different methods. Table 5 The connectivity errors over different methods.

13 932 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) Fig. 7. Example of the color channel alpha matte. (a) Input image. (b) Color alpha matte. (c, d) Magnified color alpha matte. Table 6 The mean errors in two weighting schemes. Method SAD MSE Gradient Equal weighting scheme Our weighting scheme component. Furthermore, the region with more R channel component produces erroneous alpha matte though a few cases. The accurate alpha matte can be obtained from the region with a greater B channel component. We compute the mean of alpha for each color channel in the correct alpha matte region for all images in the dataset. Finally, we determine the alpha weights of Eq. (12) as the normalized means of the alpha for each color channel. The alpha weights are determined as 0.26, 0.03, and 0.71 for R, G, and B channel components, respectively. Note that the alpha of the B channel component is most reflected, because most correct alpha values have a greater B channel component. To show the efficiency of our weighting scheme, we evaluate the mean errors between equal weighting scheme and our weighting scheme in Table 6. As shown in Table 6, our weighting scheme improves the performance in three metrics though there is no significant improvement. Generally, matting accuracy can be decreased due to the ambiguous color difference in the case of images consisting of similar colors between F and B samples. The reason of performance improvement in our weighting scheme is because there exist many cases that have a large difference in b channel between F and B samples. Consequently, the color ambiguity problem can be reduced by imposing more weight in b channel. Therefore, for further performance improvement, we think that the alpha obtained by a channel with the largest difference between F and B samples can be used. Fig. 8 shows the examples of matte improvement in the weighted matte compared with the equally weighted matte. For the image of first column, our weighting scheme produces more accurate matte in foreground regions while equally weighting scheme produces a stained matte. For the image of second column, the equally weighting scheme also cannot extract an accurate result in some background region. Fig. 8. The examples of matte improvement between two weighting schemes. (a) Input image. (b) Equal weighting scheme. (c) Our weighting scheme.

14 S. Cho, H. Byun / Pattern Recognition Letters 33 (2012) Conclusions In this paper, we have derived a new matting equation based on the assumption that the colors of a local region containing foreground, background, and unknown mixed colors form a curve. To approximate the colors in a local region, we introduced the dynamic curve color model using a quadratic Bézier curve. We showed that our method could directly estimate the alpha matte from our closed-form matting equation. In addition, our experiments showed that our algorithm estimates the alpha matte more accurately or comparable than other algorithms. Future work would include a more robust curve color model that can represent complex variations of images. Acknowledgements This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (MEST) (NRF D00998). References Chuang, Y.-Y., Curless, B., Salesin, D., Szeliski, R., A Bayesian approach to digital matting. In: Proc. Computer Vision and Pattern Recognition, CVPR 01, vol. 2, pp C. CORPORATION, Knockout user guide. Gastal, E.S.L., Oliveira, M.M., Shared sampling for real-time alpha matting. Eurographics 29 (2), Grady, L., Random walks for image segmentation. IEEE Trans. Pattern Anal. Machine Intell. 28 (11), Guan, Y., Chen, W., Liang, X., Ding, Z., Peng, Q., Easy matting. Eurographics 25 (3), Levin, A., Lischinski, D., Weiss, Y., A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Machine Intell. 30 (2), Mishima, M., Matusik, W., Pfister, H., Hughes, J.F., Durand, F., Soft edge chroma-key generation based upon hexoctahedral color space. In: U. S. Patent 5,355,174. Rhemann, C., Rother, C., Rav-Acha, A., Sharp, T., High resolution matting via interactive trimap segmentation. In: Proc. Computer Vision and Pattern Recognition, CVPR 08, pp Ruzon, M.A., Tomasi, C., Alpha estimation in natural images. In: Proc. Computer Vision and Pattern Recognition, 2000, CVPR 00, 1, pp Singaraju, D., Rother, C., Rhemann, C., New appearance models for natural image matting. In: Proc. Computer Vision and Pattern Recognition, CVPR 09, pp Sun, J., Jia, J., Tang, C.-K., Shum, H.-Y., Poisson matting. In: Proc. SIGGRAPH ACM SIGGRAPH, pp Wang, J., Cohen, M., An iterative optimization approach for unified image segmentation and matting. In: Proceedings International Conference on Computer Vision, ICCV 05, pp Wang, J., Agrawala, M., Cohen, M.F., Soft scissors: An interactive tool for realtime high quality matting. In: Proc. SIGGRAPH ACM SIGGRAPH, 9, pp Wang, J., Cohen, M.F., Optimized color sampling for robust matting. In: Proc. Computer Vision and Pattern Recognition, CVPR 07., pp Zheng, Y., Kambhamettu, C., Learning Based Digital Matting. In: Proc. International Conference on Computer Vision, ICCV 09, pp