Blockwise Matrix Completion for Image Colorization

Transcription

1 Blockwise Matrix Completion for Image Colorization Paper #731 Abstract Image colorization is a process of recovering the whole color from a monochrome image given that a portion of labeled color pixels. Using matrix completion to colorize is an attractive approach. Traditional matrix completion algorithms use the column (row) basis to capture the main information of the matrix, which are based on the low-rank assumption that the columns (or rows) of the matrix have high correlation. However, for the natural visual data, the correlation between blocks of the image data matrix is much stronger than the one between columns (or rows). Thus, utilizing a blockwise basis to recover the missing color data of the image is more reasonable. In this paper, we propose a novel model to solve the colorization problem by a blockwise matrix completion approach. In particular, we define the blockwise nuclear norm and use it to construct a blockwise low-rank model. Our formulation is a convex optimization problem which can be solved by the augmented Lagrange multiplier algorithm. Experiments on natural images data show that our model outperforms other state-of-the-art colorization approaches. 1 Introduction Before color printing was invented, people had taken a large amount of black and white photos and films. Recovering the color of old monochrome visual data is an attractive application. Since colorizing manually is time consug, computer-assisted colorization is a widely used artificial intelligence technique. Several computer-assisted colorization approaches have been proposed (Horiuchi 22; Levin, Lischinski, and Weiss 24; Yatziv and Sapiro 26; Luan et al. 27; Wang and Zhang 212). A representative colorization method is based on the local color consistency assumption (Levin, Lischinski, and Weiss 24); that is, neighboring pixels have similar colors when their intensities are similar. However, the local consistency does not work for an image with complex textures. Moreover, we usually have not enough labeled pixels for each patch. Thus, Wang and Zhang (212) proposed another colorization method by matrix completion. Based on the idea behind the robust principal component analysis model (Candès et al. 211), they supposed that the color image can be Copyright c 215, Association for the Advancement of Artificial Intelligence ( All rights reserved. represented as a low-rank matrix plus a sparse matrix, and formulated the colorization problem as a matrix completion problem. Wang and Zhang (212) also presented a preprocess step to introduce the local color consistency idea to their method, which can improve the performance when the number of the labeled pixels is not enough. Thus, it is desirable to have a matrix completion method for colorization method, which naturally captures the local consistency property. Most of the extant matrix completion models are built on the low-rank assumption. In other words, these models use a column (row) basis to approximate the completed matrix (Cai, Candès, and Shen 21; Candès et al. 211; Koltchinskii et al. 211). In fact, this type of methods have also been widely used in other computer vision problems such as background modeling (Wright et al. 29) and image classification (Cabral et al. 211). In contrast, we propose a blockwise matrix completion model for the colorization problem. Our motivation is that the natural image typically consists of several different objects (regions), and the neighbouring pixels or patches of one object (region) are similar. Indeed, one column (row) usually crosses more than one object, while the neighbouring blocks of one object would be similar, which implies that the correlation among blocks of the image is higher than the one among columns (rows). Moreover, a block can keep the local consistency among the pixels within it. Therefore using a block basis in a natural image matrix is more reasonable. In particular, we define a blockwise nuclear norm to capture the correlation among blocks of the image and formulate a blockwise low-rank model. We train our model by the alternating direction method of multipliers (ADMM) (Lin, Chen, and Ma 21). In summary, we offer the following contributions: We propose a blockwise framework to solve the colorization problem. We define the blockwise nuclear norm and construct a blockwise low-rank model for matrix completion. Our approach integrates the benefits from the lowrank assumption based methods and the local consistency assumption based methods. We propose an efficient algorithm to train our model, which does not need additional computational costs in comparison with the existing low-rank matrix completion methods.

2 The experimental results show that our blockwise method can capture the color information better than the traditional low-rank methods. This shows that our method is effective although the idea is simple. 2 Notation and Definitions We give some notation which will be used in this paper. Given a vector x = (x 1,..., x N ) T R N, we let its Euclid norm be x = ( i x2 i )1/2. We denote the cosine similarity between x and y with same length as ρ(x, y) = xt y x y. We let I p denote the p p identity matrix. For a matrix A = [a ij ] R M N, we denote its ith row and jth column of A as A i and A j, respectively. Suppose A has singular value decomposition (SVD) A = U A Σ A VA T, and its singular values are σ 1(A) σ 2 (A) σ {M,N} (A). Then we let D τ (Σ A ) = Diag{sgn(σ i )( σ i τ) + } {M,N} i=1, and define the shrinkage operator as S τ (A) = U A D τ (Σ A )VA T (Cai, Candès, and Shen 21). We let A 1 = i,j a ij be the l 1 -norm, A F = ( i,j a2 ij )1/2 be the Frobenius norm, A = max i,j a ij be the -norm, A = {M,N} i=1 σ i (A) be the nuclear norm, and A 2 = σ 1 (A) be the spectral norm. Additionally, we let column vectorization operator be vec(a) = [a 11, a a M1, a a M2... a 1N... a MN ] T. And A B = [a ij b ij ] represents the Hadamard product of two same size matrices A and B. If M = mp and N = nq, we partition A into A (11) A (12)... A (1n) A (21) A (12)... A (2n) A =....., (1). A (m1) A (m2)... A (mn) where A (kl) R p q, k = 1, 2,..., m and l = 1, 2,..., n. Using the Matlab colon notation, we can write A (kl) = A(p(k 1) + 1: pk, q(l 1) + 1: ql). Definition 1. For the mp nq matrix A, we define its p q block rank as the maximum number of linearly independent blocks given in (1), and denote it by rank p q (A). Definition 2. For the mp nq matrix A partitioned as (1), we define its p q blockwise transformation from R mp nq to R pq mn as follows B p q (A) = [ vec(a (11) ), vec(a (21) ),... vec(a (m1) ), vec(a (12) ), vec(a (22) ),... vec(a (m2) ), vec(a (1n) ), vec(a (2n) ),... vec(a (mn) ) ]. We define the corresponding inverse transformation as B 1, which reads Bp q[b 1 p q (A)] = A. Definition 3. For the mp nq matrix A, we define its p q blockwise nuclear norm as B p q (A). Remark 1. Notice that we can also represent the block rank by a standard matrix rank by using blockwise transformation, that is, rank p q (A) = rank(b p q (A)). However, rank p q (A) rank(a) in general. Remark 2. The entry-wise matrix norms are invariant under blockwise transformation, such as A F = B p q (A) F and A 1 = B p q (A) 1. However, we can not have that A = B p q (A). 3 Image Colorization Given an M N RGB color image, we let R, G and B R M N be the red, green and blue channels of the original color image, respectively. Then the unfold matrix L = [R, G, B] R M 3N contains the whole information of the image. Suppose the corresponding monochrome image is W = α 1 R + α 2 G + α 3 B, where α 1, α 2 and α 3 are constant weight coefficients for each channel. Along the typical treatment, we let α 1 = α 2 = α 3 = 1 3 in this paper. The goal of colorization is to recover the matrix L by using an incomplete labeled color matrix D = [d ij ] R M 3N. We let Ω = [ω ij ] {, 1} M 3N be the indicator matrix of the observed color labels, that is, ω ij = 1 when d ij is observed and ω ij = otherwise. Recently, Wang and Zhang (212) formulated image colorization as a matrix completion problem. Specifically, the formulation is given as L,S R M 3N L + λ Ω S 1 + η 2 LT W 2 F s.t. L + S = D, (2) where T = [α 1 I N, α 2 I N, α 3 I N ] T. There is another important class of colorizing methods, which are based on the local consistency assumption (Levin, Lischinski, and Weiss 24). In particular, the approach aims to imize the difference between the pixel and the weighted average of the colors at neighboring pixels on the chroance channels. Wang and Zhang (212) also suggested combining local color consistency with the low-rank model. This is an effective way especially when the given labels are very few. 4 Methodology Although the local consistency does not exist in the case of some complex texture, we observe that the natural image has a good property that its neighbouring blocks usually are similar. Hence, compared with the column (row) based representation, the blockwise based representation is more reasonable. Let us see Figure 1, which illustrates our observation clearly. Representing the RGB image as L = [R, G, B] R mp 3nq, we let its row similarity matrix be C row = [ρ(l T i, LT j )] Rmp mp and the column similarity matrix be C col = [ρ(l i, L j )] R 3nq 3nq. Let ˆL = B p q (L) R pq 3mn. Then the similarity matrix under the blockwise representation is defined as C blk = [ρ(ˆl i, ˆL j )] R 3mn 3mn. Figures 1 (b), (c), and (d) visualize these three similarity matrices. It is clearly seen that the blockwise representation has much higher correlation. Figures 1 (e) and (f) illustrate the visual result of two representations. In particular, Figure 1 (f) shows that the original image matrices are of almost full rank, while Figure 1 (e) shows that the image matrix after the blockwise transformation is nearly low-rank.

3 (a) Example image (b) C row for row bases (c) C col for column bases (d) C blk for block bases (d) Conventional representation L = [R, G, B] (e) Blockwise representation ˆL = B p q (L) Figure 1: Figure (a) is an example of RGB image. Figure (b), (c) and (d) display its similarity matrices under different bases. Here we choose p = q = 2 for block size. The lighter pixels in the figures mean the higher correlations. Figure (d) is the conventional unfold matrix representation. Figure (e) display the result of blockwise transformation. Figure 2 further reports the results of the original low-rank and blockwise low-rank approximations. The blockwise representation obtains much lower reconstruction error. More specifically, the results in Figure 2(c) and (e) look significantly better than those in Figure 2(b) and (d). It is worth noting that the blockwise representation can naturally capture the local consistency property. This motivates us to use a block basis instead of a column (row) basis to recover the missing data. In particular, we find the best k-rank approximation of L by its top k singular values and corresponding singular vectors. Since each column of ˆL corresponds a block of L. we obtain the best blockwise low-rank approximation by using SVD on ˆL. 4.1 Blockwise Matrix Completion In this section we develop a blockwise low-rank model for image colorization. Based on the blockwise representation mentioned earlier, we assume that L is blockwise low-rank. Equipped with the notation in Section 2, we define the following block rank imization problem: L,S R mp 3nqrank p q(l) + λ Ω S 1 + η 2 LT W 2 F s.t. L + S = D, (3) where T = [α 1 I nq, α 2 I nq, α 3 I nq ] T. Since the problem in (3) is intractable, we use the blockwise nuclear norm to relax the blockwise rank. In particular, we have the blockwise low-rank (B) model for colorization problem as follows L,S R mp 3nq B p q (L) + λ Ω S 1 + η 2 LT W 2 F s.t. L + S = D. (4) As mentioned in Remark 2, the blockwise transformation has the invariant property under the 1-norm and the Frobenius norm. Hence the solution of (4) can be obtained by solving the following problem: ˆL,Ŝ Rpq 3mn ˆL + λ ˆΩ Ŝ 1 + η 2 ˆL ˆT Ŵ 2 F s.t. ˆL + Ŝ = ˆD, (5) where Ŵ = B p q(w), ˆD = Bp q (D), ˆΩ = Bp q (Ω), and ˆT = [α 1 I mn, α 2 I mn, α 3 I mn ] T. The conventional lowrank method is a special case of our model if we set p = M, q = 1, or p = 1, q = 3N. In other words, for L R M 3N, we have rank(l) = rank M 1 (L) = rank 1 3N (L) and L = B M 1 (L) = B 1 3N (L). In terms of the suggestion of Wang and Zhang (212), our model can also be combined with the local consistency to add label pixels. Specifically, for any unlabeled pixel, we find its neighboring pixels whose gray level values are similar with it, then use the weighted average of its neighbor to label it. However, we will see in Section 5 that the ensemble way does not perform very well when the number of the labeled pixels is sufficient. 4.2 Training Algorithm Let the solution of Problem (5) be ˆL. Then the solution of Problem (4) is L = B 1 p q(ˆl ). Since Problem (5) can

4 8 7 column (row) bases block bases 6 approximation error proportion of entries for reconstruction (a) error of approximation (b) column (row), 2% (c) blockwise, 2% (d) column (row), 5% (e) blockwise, 5% Figure 2: Figure (a) compares the sqaure error by using trivial low-rank method and our blockwise low-rank method to approximate Figure 1(a). Figure (b), (c), (d), (e) display the approximation results of using 2% and 5% entries by two different ways. be solved by the alternating direction method of multipliers (ADMM) efficiently (Wang and Zhang 212) and the blockwise transformation is trivial, the blockwise low-rank model can be also solved efficiently. The detailed procedure to solve Problem (4) is summarized in Algorithm 1. Algorithm 1 The ADMM Algorithm 1: Input: W, D, Ω; λ, η, p and q; 2: Ŵ = B p q(w), ˆD = B p q(d), ˆΩ = B p q(ω); 3: ˆT = [α1i mn, α 2I mn, α 3I mn] T ; 4: ˆL() = Ŝ() = ˆX () = ; 5: µ () 1 >, µ () 2 > ; τ > 1; j = ; 6: Y () 1 = Y () 2 = sgn( ˆD)/ max{ ˆD 2, ˆD /λ}; 7: repeat 8: ZˆL = Y(j) 1 +Y(j) 2 +µ(j) 1 ( ˆD Ŝ(j) )+µ (j) 2 ˆX (j) µ (j) 1 +µ(j) 2 9: ˆL(j+1) = S (j) 1/(µ )(ZˆL); 1 +µ(j) 2 1: ZŜ = 1 µ (j) 1 Y (j) 1 + ˆD (j) ˆL (j+1) ; 11: Ŝ (j+1) = ˆΩ D (j) λ/µ (ZŜ) + (1 ˆΩ) ZŜ; 1 12: ˆX(j+1) = (ηŵ ˆT T Y (j) 2 + µ (j) ˆL (j+1) 2 ); 13: (η ˆT ˆT T + µ (j) 2 I3pq) 1 ; 14: Y (j+1) 1 = Y (j) 1 + µ (j) 1 ( ˆD ˆL (j+1) Ŝ(j+1) ); 15: Y (j+1) 2 = Y (j) 2 + µ (j) 2 ( ˆX (j+1) ˆL (j+1) ); 16: µ (j+1) 1 = τµ (j) 1, µ(j+1) 2 = τµ (j) 2 ; 17: j = j + 1; 18: until convergence 19: Output: L = B 1 p q (ˆL (j+1) ). Since the number of entries of L and ˆL are the same, our algorithm has no additional memory cost. The computation cost of Algorithm 1 is doated by the SVD on pq 3mn size matrix (line 9), which is comparable to low-rank model (SVD on mp 3nq) by choosing suitable p and q. ; 5 Experiments In this section we conduct empirical evaluations of our method. For the sake of convenience, we denote our blockwise low-rank model as B and denote the ensemble method which incorporates B and local color consistency as. In experiments, we compare our methods with Levin, Lischinski, and Weiss s model () and Wang and Zhang s two models: low-rank () method and combining low-rank and local color consistency () method. Recall that an RGB image can be represented as the unfold matrix L = [R, G, B] and the corresponding monochrome image is the average of three channels, that is, 1 3 (R + G + B). In experiments we uniformly sample some proportion of image pixels to be labeled, then treat them with the original monochrome image as inputs. Let the resulting unfold matrix from a method be L. We define the relative square error (RSE) to measure the performance. That is, RSE = L L F / L F. Following the suggestion of Wang and Zhang (212), we choose the parameters λ and η as 1 in their model. We demonstrate the sensitivity of hyperparameters for the lowrank model (B) in Figure 1(a) using 2%, 3%, 4% and 5% labeled pixels. We set λ = η = 1,p = q = 2 in the following experiments. We compare different methods on sample images in Figure 3, which are sampled from AAAI s web site and the Berkeley Segmentation Dataset (Martin et al. 21). We normalize them to (or 48 32) size RGB image. The results of RSE by labeled pixels with different proportion are shown in Figure 5. We repeat 1 times for each setting and take the average. The standard deviation is very s- mall, so we do not report it. Notice that compared with other methods, can not utilize enough labels. Our blockwise low-rank model (B) is significantly better than the traditional low-rank model (). Further combining the local consistency with the conventional or blockwise low-rank methods (i.e., and ) can perform well when the labels are very few, but this causes more noises when the labeled pixels are sufficient. Additionally, the performances of both and are very close. We also provide some visual examples of the scenic photo from the web page of AAAI 215 in Figure 6. To facilitate

5 (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) Figure 3: There are sample images from AAAI s web site and the Berkeley Segmentation Dataset (Martin et al. 21). relative square error % 3% 4% 5% relative square error % 3% 4% 5% relative square error % 3% 4% 5% η p and q λ (a) varying λ with η = 1, p = q = 2 (b) varying η with λ = 1, p = q = 2 (c) varying p, q with λ = η = 1 Figure 4: We display the RSE by varying λ, η and the block size (let the shape be square) with other hyperparameters fixed. illustration, we stack the differences of three color channels vertically in these figures. The figures display the recovered images and their differences between the original ones. The corresponding RSE values are given under the sub-figures. Clearly, our B method is the best. In fact, is ineffective for intensity and illuation, and the others have more noise. Combining with the local consistency achieves poor visual performance as well. 6 Conclusion and Future Work In this paper, we have studied the image colorization problem and proposed a blockwise matrix completion approach. In particular, we have introduced the blockwise rank and the blockwise nuclear norm to construct a blockwise low-rank model. The empirical evaluations have proved that our model has better performance than the state-of-art approaches. It is worth pointing out that the blockwise framework is not limited to the colorization problem, because mostly natural visual images have high correlation between blocks. Thus, we believe that the blockwise framework can be used to other visual applications. References Cabral, R. S.; Torre, F.; Costeira, J. P.; and Bernardino, A Matrix completion for multi-label image classification. In Advances in Neural Information Processing Systems, Cai, J.-F.; Candès, E. J.; and Shen, Z. 21. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization 2(4): Candès, E. J.; Li, X.; Ma, Y.; and Wright, J Robust principal component analysis. Journal of the ACM (JACM) 58(3):11. Horiuchi, T. 22. Estimation of color for gray-level image by probabilistic relaxation. In Pattern Recognition, 22. Proceedings. 16th International Conference on, volume 3, IEEE. Koltchinskii, V.; Lounici, K.; Tsybakov, A. B.; et al Nuclearnorm penalization and optimal rates for noisy low-rank matrix completion. The Annals of Statistics 39(5): Levin, A.; Lischinski, D.; and Weiss, Y. 24. Colorization using optimization. In ACM Transactions on Graphics (TOG), volume 23, ACM. Lin, Z.; Chen, M.; and Ma, Y. 21. The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arxiv preprint arxiv: Luan, Q.; Wen, F.; Cohen-Or, D.; Liang, L.; Xu, Y.-Q.; and Shum, H.-Y. 27. Natural image colorization. In Proceedings of the 18th Eurographics conference on Rendering Techniques, Eurographics Association. Martin, D.; Fowlkes, C.; Tal, D.; and Malik, J. 21. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Computer Vision, 21. ICCV 21. Proceedings. Eighth IEEE International Conference on, volume 2, IEEE. Wang, S., and Zhang, Z Colorization by matrix completion. In AAAI. Citeseer. Wright, J.; Ganesh, A.; Rao, S.; Peng, Y.; and Ma, Y. 29. Robust principal component analysis: Exact recovery of corrupted lowrank matrices via convex optimization. In Advances in neural information processing systems, Yatziv, L., and Sapiro, G. 26. Fast image and video colorization using chroance blending. Image Processing, IEEE Transactions on 15(5):

6 B B B.3.25 B (a) (b) B (d).16 B (c).16 B.18.3 B (e) (f) B B (h).2 B.1.1 (g).12 B (i) (j) (k) (l) Figure 5: These figures display the performance of the five methods for the sample images in Figure 3, where the x-axis presents the proportion of labeled pixels and the y-axis shows the relative square errors., 3%, 7.19%, 3%, 6.54%, 3%, 8.26% B, 3%,2.59%, 3%, 8.25%, 4%, 7.11%, 4%, 4.7%, 4%, 9.16% B, 4%,1.88%, 4%, 9.16%, 5%, 7.5%, 5%, 3.46%, 5%, 1.12% B, 5%,1.44%, 5%, 1.13%, 6%, 7.1%, 6%, 2.59%, 6%, 11.4% B, 6%,1.14%, 6%, 11.5% Figure 6: These figures display the recovering results and corresponding errors by different methods for Figure 2(d). The numbers under the sub-figure mean the proportion of labeled pixels and RSE of the result.