Divide and Conquer: A Self-Adaptive Approach for High-Resolution Image Matting

Transcription

1 Divide and Conquer: A Self-Adaptive Approach for High-Resolution Image Matting Guangying Cao,Jianwei Li, Xiaowu Chen State Key Laboratory of Virtual Reality Technology and Systems School of Computer Science and Engineering, Beihang University Beijing, China {cgy,jianwei.li,chen}@buaa.edu.cn Zhiqiang He Lenovo Research Beijing, China lirong2@lenovo.com Abstract This paper proposes an efficient patch-based image matting approach to reduce memory consumption in processing high-resolution images. Most existing image matting techniques employ a global optimization over the whole set of image pixels, incurring a prohibitively high memory consumption. Inspired by divide and conquer, the image is divided into small patches self-adaptively according to the distribution of pixels, then the matting algorithm is applied in patch level. The memory consumption is related to the size of patch. Relationships between patches are also considered by locally linear embedding to maintain consistency through the whole image. Experimental results show that our method significantly reduces memory consumption while maintaining high-fidelity matting results on the benchmark dataset. Keywords-image matting; patch division; high-resolution I. INTRODUCTION Image matting is to softly extract the foreground object from a single input image which is applied to a number of important applications such as layer extraction, dehazing, film making and so on. The pixel s color I is a convex combination of foreground F and background B: I = αf + (1 α)b where α represents the opacity (alpha matte), whose values lie in [0,1] with α = 1 indicating a foreground pixel and α = 0 denoting a background pixel. However, this problem is highly ill-posed. It is typical to include a trimap or some user scribbles indicating some definite foreground, definite background and unknown region to simplify the problem. At the same time, strong priors will help significantly improve the quality of results. Existing matting methods have already obtained exciting results. However, with the increasing demand of highresolution images produced by today s digital cameras, high memory consumption becomes a common problem for image matting algorithms. Global optimization based methods inevitably incur a high cost in memory consumption. For example, one of the best known matting method [3] requires to allocate about 6GB memory to process an image with 7.7M pixels, which exceeds the capability of many low-end *Corresponding authors: Xiaowu Chen (chen@buaa.edu.cn); Zhiqiang He (lirong2@lenovo.com). commodity computers. Matting method for high-resolution images [24] is also explored, but the results are not satisfactory with a low rank on the benchmark [1]. Our key observation is that to achieve efficient and scalable matting on high-resolution images, the operations might be performed on multiple small data inspired by divide and conquer. In this paper, we adopt a division strategy to solve the problem of high memory consumption in image matting. We divide the high-resolution image into small patches, the shapes of patches are adaptively adjusted according to the distribution of unknown pixels in the trimap. Then we apply an optimized matting method in each patch parallelly, and combine the matting results with a weighted optimization method. With this divide and conquer strategy, the memory consumption has no relation with the size of image, but only relates to the size of patch, which achieving efficiency and scalability. One of the problems of this strategy is that some patches may be short of known information. A general situation is that all pixels in a patch are in the unknown area. To address this problem, we merge some adjacent patches close to the unknown patch and redivide the merged region. Another problem is that solving each patch individually will bring edge artifacts between patches. We add relations between patches to solve this problem. When we build the affinity matrix in one patch, we consider the relationships not only between the pixels in this patch, but also some pixel outside the patch as remedy for nonlocal information of each patch. Our main contribution is a patch-based image matting framework to solve high-resolution images efficiently. A patch division algorithm is presented to divide the image into appropriate patches adaptively according to the unknown pixel distribution. Nonlocal relationships between patches are established to eliminate the edge artifacts. Experiments on the benchmark [1] demonstrate that our approach maintain high-fidelity matting results while significantly reducing memory consumption. II. RELATED WORK Existing image matting methods can be roughly categorized into sampling-based methods such as [11,12,13,14,15,16,23], propagation-based methods such

2 as[10,2,5,9,21], and combination of the two methods such as [7,10]. We only discuss some of the most relevant works here. A more comprehensive survey can be found in [6]. The sampling-based matting methods collect color samples from known foreground and background regions to represent the corresponding color distributions and determine the alpha value of an unknown pixel according to its closeness to these distributions. These methods employ different selection criteria to collect a subset of known F and B samples and different algorithm to search for the best (F, B) pair for each unknown pixel within the representative samples. Once the best pair is found, the final alpha matte is estimated as the equation (2). The propagation-based matting methods define an affinity matrix representing the similarity between pixels and propagate the alpha values of known pixels to the unknown ones. These approaches mostly differ from each other in their propagation strategies or affinity definitions. Knn matting [9] defines the affinity matrix by the weights of each unknown pixel with the K nearest neighbors in the feature space. Some other matting methods such as [3,5] defined the affinity matrix by the weights of the pixels in the local window. Factually, most methods combine the two methods which will make a good trade-off between the two approaches. Robust matting [7] first collected samples with high confidence, and then used the Random Walk [22] to minimize the matting energy constructed by matting Laplacian. The other sampling-based method [11,14,23,12] also apply local smooth as a postprocessing step to further improve the quality of the estimated alpha matte. The method [10] also combines sampling and propagation which also first collects samples with high confidence and then uses nonlocal and local constraint to propagate the alpha values. In conclusion, the recent approaches not only the propagation-based but also the sampling-based need to build a large sparse matrix such as matting Laplacian [2,3,7,11,12,13,14,23] or other sophisticated matrices[5,10,18,21,8,9]. However, solving this kind of large linear system is a very memory consuming operation, which makes it unpractical to deal with high-resolution images. III. OUR APPROACH We decompose the image into small patches to reduce the memory consumption to handle high-resolution inputs. Theoretically, any existing matting methods could be embedded into our framework in disposing each patch. In this paper, we adopt LNSP method [10] for some reasons. First, the LNSP method keeps in a very high rank in the benchmark for a very long time. Moreover, the nonlocal prior of this method can help our patch-based method by building relations between patches. We optimize the LNSP method in several aspects to fit our patch-based matting framework. A. Comprehensive Sampling To expand the known area in the trimap, we first use sampling algorithm to turn some unknown pixels into definite foreground or background pixels. Here we follow the comprehensive sampling method [14] which produces nice result. We only briefly describe the basic idea. For more details please refer to [14]. The selection of best pair is done through a brute-force optimization of an objective function which consists of three parts as follows: O z (F i, B i ) = K z (F i, B i ) S z (F i, B i ) C z (F i, B i ) (1) where K indicates chromatic distortion, S and C contain spatial and color statistics of the image, respectively. The equation (1) estimates the confidence of each candidate pair for pixel i. Finally, we use the pair with the highest confidence to be the best pair that can represent the true foreground and background colors. Given a selected foreground and background sample pair (F i, B i ) for pixel i, the alpha value of pixel i can be estimated as: α i = (C i B i )(F i B i ) F i B i 2 (2) After sampling, the initial alpha values and the corresponding confidence map are obtained. In the unknown area, alpha values with high confidence can be seen as definite known pixels. The confidence threshold is set to be 0.2 in all our experiments. The problem of the sampling process is that it is very time consuming to conduct on high-resolution images directly. Thus we scale the image into a smaller one to do the sampling, and rescale the sampling result to the original resolution. In our experiments, we scale the image into no more than 800 pixels in both width and height. For example, the image doll with 19.31% unknown pixels takes about 90 mins in sampling process on original resolution ( ), and only takes 6 mins on the smaller one ( ). We find that sampling on the smaller image can also produce good result while saving time cost. B. Patch Division After sampling process, the definite alpha values are going to be propagated to the unknown area. To reduce memory consumption, we divide the high-resolution image into overlapped patches. In our experiment, the patch size is by default, and the adjacent patches are 20 pixels overlapped. The patches without unknown pixels are regarded as known patches and do not need to be computed again. However, some divided patches encounter large unknown area and may lack enough information of known foreground or background pixels. This kind of patch will result in very bad alpha matte. For example, the patches labeled a and b in Figure1 (d) only have unknown pixels. We call this kind of patch as low-confidence patch. If the

3 percentages of two kinds of known pixels (foreground and background) are larger than 5% in one patch, this patch will be seen as high-confidence patch. Otherwise, the patch will be regarded as low-confidence patch. For the low-confidence patches, we would like to merge them with the adjacent patches to increase their confidence. According to our observation, if the more different the quantity of three kinds of pixels in the adjacent patches, the more information will be obtained if these patches are merged. Under this observation, we use the variance of quantity of pixels in different region of the adjacent patches to determine which patches are merged and redivided. The merging direction is horizontal or vertical, according to specific situation. The variance of patch i is computed as follows: variance h i = variance v i = j {f,b,u} j {f,b,u} N h1 ij N v1 ij N h2 ij N v2 ij (3) Here, N represents the number of different kinds of pixels, f, b, u represents the foreground, background and unknown region respectively. And h, v represents horizontal and vertical direction respectively, h 1 and h 2 represent the horizontally adjacent patches, v 1 and v 2 represents the vertically adjacent patches. The variance h i is the sum of difference of three kinds of pixels in the horizontally adjacent patches and the variance v i is the sum of difference of three kinds of pixels in the vertically adjacent patches. If the variance of horizontally adjacent patches is greater than vertically adjacent patches, the direction is horizontal, and vice versa. As shown in Figure 1(e). Once the direction is determined, the adjacent patches will be merged along this direction. The merge process will continue until the region have enough foreground and background known pixels or encounter known patch. To guarantee the computing scale,the region is redivided into blocks containing about pixels. As shown in Figure 1 (f), if the merging direction is horizontal, the width of block will be the same as the width of region and the height of block will be 10000/width. If the merging direction is vertical, the height of block will be the same as the height of region and the width of block will be about 10000/height. Figure 1 shows the process of patch division. The other redivided patches can be computed directly because those patches have enough known pixels or be ignored because those patches have no unknown pixels. C. Graph Model After patch division, we apply the optimized version of the LNSP method [10] to each patch. An optimized graph model is built for each patch. Now we take the patch ρ as an example. Figure 1. The process of patch division. (a) original image (b) cropped trimap (c) trimap is divided into some patches (d) determine the bad patches (e) patches merge together (f) the merged regions are redivided. 1) Nonlocal Term: Each patch will lock the nonlocal information since the image is divided into patches, so we find the K nearest neighbors for each pixel in the global RGBXY feature space as remedy for nonlocal information of each patch. But the number of all neighbors of each pixel is much larger than the number of pixels in this patch, which results in large extra memory consumption. Besides, the weak constraint of each unknown pixel outside the patch will lead to inaccurate alpha matte. Thus we only choose the neighbors in this patch and the neighbors with known alpha values outside this patch. Here the known pixels not only include the known pixels labeled by the trimap or user scribbles, but also the pixels with high confidence according the confidence map obtained by comprehensive sampling at the first step. After selection, the quantity of neighbors related to the pixel outside this patch will about The matrix representing nonlocal constraint is only 10% 20% larger than the matrix built only in this patch, but the quality of result will have a great improvement. We connect the pixel X i to the selected neighbors X i1, X i2,, X id with weights W LLE, where D represents the number of selected neighbors. As stated in [25], the can be computed by minimizing W LLE id W LLE = arg max W LLE id i=1 id N D X i d=1 W LLE id X id 2 (4) The W LLE will be an N H matrix, where N represents the number of pixels in the patch and (H N) is the number of selected neighbors outside this patch. The last (H N) columes in W LLE are the weights between the selected pixels outside the patch and the pixels in the patch. 2) Local Term: The local matting Laplacian enhances the local smoothness of the alpha matte. W Lap is defined as: W Lap = [W Lap Lap, 0], where W is the matting Laplacian in in

4 of N N representing the weights between pixels and neighbors in a 3 3 windows in this patch. More details about matting Laplacian please refer to [3]. And 0 is a zero matrix of N (H N). 3) Data Term: For the pixel i in patch, its data weights, W (i,f ) and W (i,b) which represents the probability of a pixel belonging to foreground and background, are defined between each pixel and two virtual nodes to enforce the data constraint. For pixel i, the data term is defined as: W (i,f ) = γα i, W (i,b) = γ(1 α i ). (5) The initial alpha values of this patch in the equation (5) are the results by comprehensive sampling before local smooth proposed in [14] and γ is set 0.5. Besides the pixels in this patch, the pixels connected to the visual nodes also include the selected neighbors in W LLE outside this patch. 4) Closed-form Solution for Each Patch: For the patch ρ, the energy function for solving alpha values is defined as: E ρ = λ H (α i g i ) 2 + ( W ij (α i α j )) 2 (6) i Ω i=1 i N i Where the set Ω includes visual nodes, the pixels of known alpha values in this patch after sampling and the selected pixels as neighbors in feature space outside this patch. H is the total number of the nodes in the graph model associated represents three kinds of weights,, local term W Lap ij and data term W (i,f ) and W (i,b). The set N i is the set of neighbors of pixel i, including neighboring pixels in 3 3 windows, the selected nearest neighbors in feature space, and the two virtual nodes. The above function (6) can be further written in a matrix form as: with the patch ρ. W ij containing nonlocal term Wij LLE E ρ = (α G) T Λ(α G) + α T L T Lα (7) In which W ii, if i = j L ij = W ij, if i and j are neighbors 0, otherwise Here, the weight W ii = W ij. Λ is a diagonal matrix i N of H H and Λ ii is 1000 if i Ω and 0 otherwise. G is a vector of H, and G i is set 1 if i belongs to foreground and 0 otherwise. The above equation (7) is a quardratic function about α, which can be minimized by solving the linear equation in closed-form solution: (Λ + L T L)α = ΛG (8) The first N values in the α vector are the alpha values of the pixels in the patch ρ. Finally, all patches are independent of each other so the above operations can be performed in parallel. This way could greatly reduce the time consumption. Figure 2. The result comparison. (a) the image; (b) and (c) are the adjacent patches including the cropped image, trimap and alpha matte; (d) the result of direct average of the adjacent patches; (e) the result of weighted average of the adjacent patches. D. Global Alpha Matte Since the alpha matte of every patch has been obtained, the goal in this section is to get the global consistent alpha matte. The main problem is how to compute the alpha values in the overlapped region of those adjacent patches and eliminate the edge artifacts. To get more accurate alpha values, we give the patches different weights according to the number of different kinds of pixels. The weight is defined as: W ρ = e (P f P b ) 2 +(Pf Pu) 2 +(P b Pu) 2 σ 1 e Pu σ 2 (9) Here, P d, d = {f, b, u} represents the percentage of different kinds of pixels. The σ 1 and σ 2 are constants trading off the effect of variance and the quantity of unknown pixels. The overlapped patches are blended according to their weights to generate the final result. Equation (9) is based on the observations: the alpha matte will be more accurate if this patch has enough foreground and background pixels and not too many unknown pixels. We use the variance to measure the number of pixels in different region. If the patch only has little foreground, the alpha values of unknown pixels might be much smaller. If the patch only has little background, it might result in oversmooth. The direct average alpha values in the overlapped region may lead to edge artifacts, as shown in Figure 2 (d). While weighted average alpha values will be more accurate, as shown in Figure 2 (e). IV. EXPERIMENTS We evaluate the proposed approach on a well established benchmark dataset [1], which contains 35 natural images. Among those images, 27 of them constitute the training set where the groundtruth alpha mattes are available. The remaining 8 images without groundtruth alpha matte are used for evaluation. A. and Comparison We choose four representative image matting methods to compare with our approach. The source codes of these methods are available on the internet. Table 1 lists the comparison of memory and time consumption with these methods. Only three examples (plant, elephant, plasticbag)

5 Table I T HE MEMORY AND TIME COMPARISON OF DIFFERENT METHOD ON DIFFERENT RESOLUTION IMAGES. Performance Comparison for Image Matting Image plant elephant plasticbag Resolution 667KB 8.83MB 9.14MB Unknowns 23.17% 9.74% 24.12% 25.15MB MB MB CF[3] 0.8min 2min 4min 50.26MB MB MB KNN[9] 0.7min 2min 4min 25.15MB MB MB LB[5] 0.7min 2min 4min 25.15MB MB MB CS[14] 8min 54min 115min 2.35MB 0.42MB 2.55MB Our 10min 18min 62min Table II R ANKS OF DIFFERENT MATTING METHODS WITH RESPECT TO THE THREE MEASUREMENTS ON BENCHMARK DATASET [1]. SAD Method 1.LNSP 2.GS 3.Our 4.KL-Sparse 5.TSPS-RV 6.IT 7.CS 8.SVR 9.CWCT 10.SC Rank MSE Method 1.LNSP 2.Our 3.KL-Sparse 4.CCM 5.GS 6.TSPS-RV 7.CS 8.SVR 9.CWCT 10.CSC Rank Gradient Method Rank 1.GS Our KL-Sparse LNSP CS CCM SVR SC Segment Global 13.9 Figure 3. (a) the original image; (b) the trimap; (c) the initial alpha with high confidence in the confidence map (d); (e) the result directly using LNSP in each patch; (f) the result by LNSP; (g) our result. are listed for saving space. Here we only illustrate the memory cost by large sparse matrices, which are the highest memory cost during solving process. Since we divide the images into patches, we only build a sparse matrix based on patch instead of the whole image. Because patches differ from each other, we only consider the average numbers of elements in different patches. As shown in Table 1, the methods using the matting Laplacian such as [3,14] or other sophisticated matrices such as [9,5] consume large memory. While our method consumes little memory which is only about 0.1% 1% of them. This is because the memory consumed by our method relies on the size of the patch and the percentage of unknown pixels and has little relation with the image resolution. The processing time might be a little large, however, we can speed up by parallel computing. B. Evaluation on Benchmark Database We firstly compare our method with LNSP method [7]. As shown in Figure 3, we directly use LNSP in each patch without considering the neighbors outside the patch. The result (e) has much clear edge artifacts and the alpha mattes in different patches are not consistent. But the result (g) by Figure 4. The errors compared with CF [3], KNN [9], LB [5] and CS [14] on the high-resolution dataset with different trimaps for MSE and SAD measurement. our method solves this problem and is similar with the result (f) by LNSP directly used on the whole image. Since the benchmark cannot evaluate on the highresolution dataset, we compare our method with other methods on the low-resolution dataset evaluated by [4]. Here We only show the top ten methods for each measurement. Our method ranks at the top three as the Table 2 shown. The test result has been published on the benchmark [1]. While our method has great advantage on memory consumption in solving high-resolution images. C. Comparison on High-resolution Images Because the benchmark dataset cannot evaluate on the high-resolution dataset, we use 27 images with groundtruth to compare our method with CF [3], KNN [9], LB [5] and CS [14] on different trimaps which has different width of unknown region. Two main different metrics namely SAD (Sum of Absolute Differences) and MSE (Mean Squared Error) are taken into consideration. As shown in Figure 4, the SAD1 and MSE1 are the testing result on first kind trimap, and SAD2 and MSE2 are the testing result on second

6 Figure 5. Visual comparison of our method with CF [3], KNN [9], LB [5] and CS [14] on the testing dataset of high-resolution without ground truth. kind trimap. Being limited by the space, we only show the evaluation of first 10 high-resolution images. The last column in Figure 4 is the average SAD and MSE on different trimaps. From the figure we can see that our approach outperforms other methods on high-resolution images. We choose two representative images from the 8 highresolution images without groundtruth to compare the visual quality of alpha matte with CF [3], KNN [9], LB [5] and CS [14]. The foreground and background in the elephant example has similar color distribution. Figure 5 shows that our method well distinguish foreground and background from the similar color distribution. But the edges of foreground and background by other four methods are blurry. The second example plasticbag has large semi-transparent region. Figure 5 shows that the alpha mattes produced by other four methods are not consistent but ours is consistent and smooth. We also choose two images from 27 images with groundtruth to show the results in Figure 6. The first example has tiny hair and there are many holes between hairs. CF [3] and LB [5] methods fail to distinguish the tiny hair, CS [14] takes some foreground as background and KNN [9] takes some background as foreground when they process the tiny hair and the space between hair. However, our method get accurate alpha matte in processing the tiny hair. The second example has many small holes. The methods CF [3], LB [5], KNN [9] and CS [14] produce oversmooth results when they process the small holes. However, our method successfully distinguishes the small holes and produces similar result with the groundtruth as Figure 6 shown. V. CONCLUSION In this paper, we proposed an image matting method via patch division which consuming little memory even on the high-resolution images. Instead of propagating alpha values to the whole images according to relationships among all pixels, by self-adaptive patch division, we apply the graph model in every patch with unknown pixels. Our method significantly improves the memory efficiency while maintaining a high-degree of visual accuracy in matting results. One limitation of our current method is that the time consumption is a little large. We would like to employ GPU to improve efficiency in the future. VI. ACKNOWLEDGEMENT We would like to thank the reviewers for their help in improving the paper. This work was partially supported by NSFC ( & ), SRFDP ( ) and the Lenovo Outstanding Young Scientists Program. REFERENCES [1] [2] K. He, J. Sun, and X. Tang. Fast matting using large kernel matting laplacian matrices. In CVPR, pages , [3] A. Levin, D. Lischinski, and Y. Weiss. A closed form solution to natural image matting. In CVPR, pages , 2006.

7 Figure 6. Visual comparison of our method with groundtruth alpha matte, CF [3], KNN [9], LB [5] and CS [14] on the training dataset of highresolution with ground truth. [4] C. Rhemann, C. Rother, J. Wang, M. Gelautz, P. Kohli, and P. Rott. A perceptually motivated online benchmark for image matting. In CVPR, pages , [5] Y. Zheng and C. Kambhamettu. Learning based digital matting. In ICCV, pages , [6] J. Wang and M. Cohen. Image and video matting: A survey. Foundations and Trends in Computer Graphics and Vision, [7] J.Wang and M. Cohen. Optimized color sampling for robust matting. In CVPR, pages 1-8. IEEE, [8] P. Lee and Y. Wu. Nonlocal matting. In CVPR, pages , [9] Q. Chen, D. Li, and C.-K. Tang. Knn matting. In CVPR, pages , [10] X. Chen, D. Zou, S. Z. Zhou, Q. Zhao, and P. Tan. Image matting with local and nonlocal smooth priors. In CVPR, pages , [11] K. He, C. Rhemann, C. Rother, X. Tang, and J. Sun. A global sampling method for alpha matting. In CVPR, pages , [12] J. Johnson, D. Rajan, and H. Cholakkal. Sparse codes as alpha matte. In BMVC, Vol. 1. No. 3, [13] E. Shahrian and D. Rajan. Weighted color and texture sample selection for image matting. In CVPR, pages , [14] E. Shahrian, D. Rajan, B. Price, and S. Cohen. Improving image matting using comprehensive sampling sets. In CVPR, pages , [15] Y. Shi, O. C. Au, J. Pang, K. Tang, W. Sun, H. Zhang, W. Zhu, and L. Jia. Color clustering matting. In ICME, pages 1-6, [16] J. Sun, J. Jia, C.-K. Tang, and H.-Y. Shum. Poisson matting. ACM Trans. Graph., 23(3): , [17] L. Grady, T. Schiwietz, S. Aharon, and R. Westermann. Random walks for interactive alpha-matting. In VIIP, pages , [18] B. He, G. Wang, C. Shi, X. Yin, B. Liu, and X. Lin. Iterative transductive learning for alpha matting. In ICIP, pages , [19] Y.-Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski. A bayesian approach to digital matting. In CVPR, volume 2, pages II264, [20] J. Wang and M. F. Cohen. An iterative optimization approach for unified image segmentation and matting. In ICCV, volume 2, pages , [21] Z. Zhang, Q. Zhu, and Y. Xie. Learning based alpha matting using support vector regression. In ICIP, pages , [22] L. Grady. Random walks for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 28(11), [23] Karacan L, Erdem A, Erdem E. Image Matting with KL- Divergence Based Sparse Sampling. In ICCV, pages , [24] C. Rhemann, C. Rother, A. Rav-Acha, and T. Sharp. High resolution matting via interactive trimap segmentation. In CVPR, pages 1 8, [25] Roweis S T, Saul L K. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500): , 2000.