Recent Advances inelectrical & Electronic Engineering

Transcription

1 4 Send Orders for Reprints to Recent Advances in Electrical & Electronic Engineering, 017, 10, 4-47 RESEARCH ARTICLE ISSN: eissn: Image Inpainting Method Based on Total Variation Regularization Recent Advances inelectrical & Electronic Engineering Su Xiao* School of Computer Science and Technology, Huaibei Normal University, Huaibei, China Abstract: Background: Image inpainting is a technique that can be used to restore missing or damaged pixels in images. Owing to its high practical value, image inpainting has been a research field for many years. For image inpainting, the Total Variation (TV) model is always a powerful and popular tool. However, when TV norm is involved, most of the conventional image inpainting methods suffer from difficulty in the numerical solution. ARTICLEHISTORY Received: September 4, 016 Revised: April 14, 017 Accepted: May 0, 017 DOI: / Methods: To improve the speed and efficiency of handling TV-regularized image inpainting problem, this paper proposes a novel method that mainly employs variable splitting and alternating minimization. The proposed method first converts the classical TV model into an equivalent unconstrained minimization problem. Then, by applying variable splitting and alternating minimization, the minimization problem is decomposed into several subproblems with a smaller size. In an iterative process, by alternately addressing these subproblems with the help of corresponding appropriate methods, the optimal solution of the original problem can be efficiently obtained. In image inpainting application, the proposed method smoothly completes four damaged images with 50% of pixels lost, and the restored images illustrate good visual sense and high values of improved signal-to-noise ratio. Conclusion: Using numerical experiments, the effectiveness of the proposed method is validated as well as the advantages of the proposed method over three similar state-of-the-art methods. Keywords: Image inpainting, variable splitting, alternating minimization, total variation, soft-thresholding, TV model. 1. INTRODUCTION Since Bertalmio et al. [1] proposed the theory of image inpainting, numerous methods in this research area have emerged. Among the image inpainting methods, the current study is devoted to those based on Total Variation (TV) regularization or its variants. The key component of the TVregularized inpainting methods is the famous TV norm, which has achieved great success in suppressing noise and preserving edges. Chan et al. [] introduced TV regularization into image inpainting, and they reconstructed sharp images by computing the Euler-Lagrange equation of the proposed model. Because of the non-smoothness of the TV norm, the conventional methods suffer from the problems of numerical difficulty and low efficiency when addressing TV-regularized image inpainting problems. To efficiently handle the TV-regularized image inpainting models, Zuo et al. [3] proposed the Generalized Accelerated Proximal Gradient (GAPG) method, which was derived from the fast iterative shrinkage/thresholding method [4]. In the GAPG method, the authors employed an important technique called variable splitting to reduce the problem-solving difficulty. Variable splitting is also the key component of recent popular optimization methods such as Alternating *Address correspondence to this author at the School of Computer Science and Technology, Huaibei Normal University, Huaibei, China; Tel/Fax: ; s: csxiaosu@163.com /17 $ Direction Method (ADM) and Split Bregman Method (SBM). Chan et al. [5] used ADM to decouple the TV-regularized image inpainting problem in a wavelet domain and then attacked the resulting subproblems using corresponding simpler methods. Split Bregman is closely related to ADM, which has been discussed in pertinent literature [6]. Therefore, when Papafitsoros et al. [7] applied SBM to deal with a combined image inpainting model involving TV-regularizer, the proposed method adopts a similar strategy of problemsolving to the method in [5]. Compared with the previous methods, the image inpainting methods based on ADM and SBM indeed demonstrate higher efficiency. Because tackling TV norm needs to confront the problem of numerical computation, Wen et al. [8] used dual formulation of TV norm as a regularizer to model the image inpainting problem. The proposed primal-dual iterative method avoids TV denoising problem; thus, it is relatively fast. Inspired by nonlocal-means filter technology [9], Zhang et al. [10] developed a Nonlocal TV (NLTV) model for image inpainting. As a variant of the classical TV model, the NLTV model can utilize more global information than its prototype. Hence, the method used by Zhang et al. outperforms majority of TV-regularized methods in terms of visual effects. To reduce the cost of solving NLTVregularized image inpainting problem, Chang et al. [11] employed Domain Decomposition Methods (DDMs) [1] as the solver for this type of problem. With the help of DDMs, the original inpainting problem is decoupled into subproblems with smaller scale, and these subproblems can be efficiently 017 Bentham Science Publishers

2 Image Inpainting Method Based on Total Variation Regularization Recent Advances in Electrical & Electronic Engineering, 017, Vol. 10, No addressed by split Bregman and Bregmanized operator splitting. Based on a Color TV (CTV) model [13], Duan et al. [14] extended the application of the NLTV model to color texture image inpainting. By introducing a new variable and an iterative parameter, the authors redeveloped the SBM to adapt to the proposed nonlocal CTV model. The inpainting results illustrate that the method in [14] excellently performed in terms of color edge preservation. Although numerous efficient methods were proposed in the past decade, the development of fast and stable image inpainting methods is still an open issue. Following recent research route in this area, this study focuses on quickly and efficiently handling a TV-based image inpainting problem. To overcome its numerical difficulty, the TV model is first transformed to an equivalent formulation. This new formulation allows the applications of variable splitting and alternating minimization, which contribute to the development of a highly efficient method. Similar to what most recently proposed methods do, the original image inpainting problem is addressed by computing its subproblems. When applied to image inpainting, the proposed method appears to be faster and more stable than its competitors. The rest of this paper is organized as follows. In Section, a new image inpainting model is proposed, and a novel method is developed to solve this model. Section 3 presents a group of experiments to verify the effectiveness of proposed method and to show its advantages. The conclusions are presented in the last section.. MODELING AND PROPOSED METHOD.1. Modeling The TV model for image inpainting can be expressed as min u st.. Hu g, (1) u 1 where u R N is the vector form of an unknown sharp image, g R M is the vector form of the damaged image, H R M N denotes a linear operator that makes the pixels in u randomly missing or damaged, 1 and are the l 1- and l - norms, respectively, u 1 is the TV norm of u, and R L N denotes the gradient operator defined in [15]. To simplify the handling of problem (1), most of the image inpainting methods equate it to the following minimization problem: 1 min Hu g u 1, () u where parameter > 0, and 0.5 Hu-g + u 1 is called cost functional. Although, unconstrained problem () is easier to address than problem (1), problem () ignores parameter, which reflects the noise level. In addition, searching the proper value for parameter results in repeatedly tackling problem (), which lowers the efficiency of the corresponding method. Hence, problem () is probably not the best equivalent transformation of problem (1). From the formulation of problem (1), we can observe that the feasible solutions of this problem should be contained in the following ball: B(, H, g) { u: Hu g }. (3) Thus, problem (1) is equivalent to the following minimization problem: min u ( Hu g), (4) u 1 B(, I,0) where C(c) is an indicator function defined as () 0 if c C C c. (5) if c C Equation (4) gives the model proposed by this paper for image inpainting... Proposed Methods Because directly dealing with the TV-reguralized model rarely leads to efficient methods, problem (4) (i.e., the proposed model) will be first equivalently converted using variable splitting to avoid this issue. Variable splitting is a simple technique that is generally applicable to the minimization problem, which is expressed as min F( x) F ( Px), (6) x 1 where P is an arbitrary linear operator. By introducing auxiliary variables and constraints, variable splitting transforms problem (6) into min F( w) F ( v) st.. w x, v Px, (7) wv, 1 where w and v are auxiliary variables. Clearly, problems (4) and (7) have the same form; thus, applying variable splitting to problem (4) yields min x ( y) st.. x u, y Hu g, (8) x, y 1 B(, I,0) where x R L and y R M are definitely auxiliary variables. Because problem (8) has multiple unknown variables to estimate, the alternating minimization strategy is adopted to solve this multivariable minimization problem using the following: min u x Hu g y, (9) u k k 1 k 1 1 u x min x 1, (10) x k 1 Hu g y min B(, I,0) ( y), (11) y where integer k 0 denotes the kth iteration. Equations (9) to (11) are subproblems of original equation (8), and each subproblem estimates one unknown variable with the other variables fixed. Subproblems (10) and (11) are l 1-denoising and orthogonal-projection problems, respectively. Because the cost functional of problem (9) is quadratic, directly taking its derivative leads to:

3 44 Recent Advances in Electrical & Electronic Engineering, 017, Vol. 10, No. 3 Su Xiao u ( H H) ( x H ( g y )). (1) k 1 T T 1 T k T k 1 1 Although the solutions of subproblems (10) and (11) cannot be obtained by calculating the derivatives, recent research has indicated that these subproblems have the following close-formed solutions [16]: k 1 k 1 x soft u 1 y k (,1/ ) (13) and k 1 Hu g k 1 1 if Hu g k 1 Hu g, (14) k 1 k 1 Hu g if Hu g where equation (13) denotes the well-known softthresholding function [17] defined as: soft(, z ) sign()max( z z,0). (15) On the basis of equations (1) to (14), the proposed image inpainting method can be summarized as follows. INPUT: 1) and ; ) 1 and ; 3) max iterations K; 4) x 0, y 0, and g; 5) and H OUTPUT: u k+1 FOR k = 0 to K Compute u k+1 using equation (1) Compute x k+1 using equation (13) Compute y k+1 using equation (14) IF u k+1 1- u k 1 < Stop iteration END END 3. MATERIALS, RESULTS AND DISCUSSION In this section, to show the validity of the proposed method, we present the experiments on image inpainting conducted on a laptop, which is equipped with Intel Core GHZ and 4GB RAM. The standard test images shown in Fig. (1) are adopted as the original sharp images, and their corresponding damaged images are shown in Fig. (), where the universal Peak Signal-to-Noise Ratio (PSNR), defined as 10 log 10(N 55 / u-u k+1 ), is used as the objective criterion of image quality. These damaged images are produced according to the universal linear model g = Hu+n, where H is a special diagonal binary matrix with a portion of its pixels randomly set to zero and n R M denotes a zero-mean Gaussian noise with standard deviation of 0.5. All the arguments of the proposed method are set as follows: = 50, = , 1 = 0.01, = 3, and K = 80. In addition to the proposed method, the Zuo [3], Wen [8], and Zhang [10] methods are also introduced for comparison. The three competitors all concentrate on tackling TV-regularized inpainting problem, and they have been demonstrated to be state-of-the-art methods. The configurations of these methods for comparison are kept the same as those in the corresponding literatures. (a) (b) (c) (d) Fig. (1). Original sharp images. (a) Barbara, whose resolution is (b) House, whose resolution is (c) Boat, whose resolution is (d) Camera, whose resolution is (a) (b) (c) (d) Fig. (). Damaged images with 50% of pixels lost. (a) Damaged Barbara, whose PSNR value is 6.69 db. (b) Damaged House, whose PSNR value is 6.7 db. (c) Damaged Boat, whose PSNR value is 6.7 db. (d) Damaged Camera, whose PSNR value is 8.30 db.

4 Image Inpainting Method Based on Total Variation Regularization Recent Advances in Electrical & Electronic Engineering, 017, Vol. 10, No (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) (m) (n) (o) (p) Fig. (3). Reconstructed images. (a)-(d) Images acquired by the Zuo method. (e)-(h) Images acquired by the Wen method. (i)-(l) Images acquired by the Zhang method. (m)-(p) Images acquired by the proposed method. The four methods are run in the MATLAB simulation platform to process the images shown in Fig. (). The inpainting results obtained by the proposed method and its competitors are shown in Fig. (3) and listed in Table 1, where the improved signal-to-noise ratio (ISNR), which is defined as 10 log 10( u-g / u k+1 -u ), is employed to measure the quality of inpainting. The Average ISNR Values (db) and Average Time (s) columns show the corresponding mean values of 10 runs for each method. Both the validity of the proposed method and its superiority over the three competitors are clearly demonstrated by these results. The rationale behind this superiority is that the proposed method has fast convergence with the help of efficient computation of the subproblems, and the subproblems all have accurate close-formed solutions. Because the Wen method requires more computation to correct the dual variable and tune step sizes, it suffers from a disadvantage in terms of time performance compared with the proposed and the Zuo methods. To avoid the numerical difficulty, the TV norm is replaced with its dual form, which degrades the ability of the Wen method to obtain ideal results. Benefiting from the advantages of NLTV, the Zhang method obtains better results than the Wen method in term of PSNR. However, the Zhang method requires more computational efforts to find the suitable weight, and in contrast to the classical TV, all directional derivatives need to be calculated. Therefore, this method shows lower efficiency than the other methods. To experimentally verify the convergence of the proposed method, Fig. (4) shows the varying curves of the values of the Mean Square Error (MSE) and cost functional, where MSE is defined as u k+1 -u /N and the cost functional is defined as u k+1 1. The curves illustrate that when the damaged images shown in Fig. () are iteratively processed, the MSE and cost functional values continually decrease, which explicitly reveals the convergence of the proposed method. CONCLUSION This study is about our proposed novel image inpainting method that employs variable splitting and alternating minimization to address the difficulty in the numerical

5 46 Recent Advances in Electrical & Electronic Engineering, 017, Vol. 10, No. 3 Su Xiao Table 1. Average ISNR values and average time. Methods Average ISNR Values (db) Average Time (s) Barbara House Boat Camera Barbara House Boat Camera Zuo Wen Zhang Proposed Fig. (4). Varying curves of the MSE and cost functional values. solution suffered by most conventional image inpainting methods. The proposed method models image inpainting as a TV-regularized minimization problem. Through equivalent transformation and problem decomposition, the inpainting model is subsequently turned into three simpler subproblems. In this manner, the solution of the original TVbased image inpainting model can be obtained by alternately and iteratively solving these subproblems. We believe that our study makes a significant contribution to the literature because our results clearly show the effectiveness of the proposed method and its superior performance. Although the proposed method is used only for image inpainting, its application can be easily extended to other areas such as image deblurring, image denoising, and so on. CONSENT FOR PUBLICATION Not applicable. CONFLICT OF INTEREST The authors declare no conflict of interest, financial or otherwise. ACKNOWLEDGEMENTS This work is supported by Anhui Provincial Natural Science Foundation (No QF150). REFERENCES [1] M. Bertalmio, G. Sapiro, V. Caselles and C. Ballester, Image inpainting, In 7 th Annual Conference on Computer Graphics and Interactive Techniques, 000, pp [] T. Chan and J. Shen, Local inpainting models and TV inpainting, SIAM J. Appl. Math., vol. 6, pp , Mar [3] W. Zuo and Z. Lin, A generalized accelerated proximal gradient approach for total-variation-based image restoration, IEEE Trans. Image Process., vol. 0, pp , Oct [4] A. Chambolle and C. Dossal, On the convergence of the iterates of the Fast Iterative Shrinkage/Thresholding Algorithm, J. Optim. Theory App., vol. 166, pp , Sep [5] R. H. Chan, J. Yang, and X. Yuan, Alternating direction method for image inpainting wavelet domains, SIAM J. Imaging Sci., vol. 4, pp , Sept [6] T. Goldstein, B. O'Donoghue, S. Setzer, and R. Baraniuk, Fast alternating direction optimization methods, SIAM J. Imaging Sci., vol. 7, pp , Aug [7] K. Papafitsoros and B. Schonlieb, A combined first and second order variational approach for image reconstruction, J. Math. Imaging Vis., vol. 48, pp , Feb. 014.

6 Image Inpainting Method Based on Total Variation Regularization Recent Advances in Electrical & Electronic Engineering, 017, Vol. 10, No [8] Y. Wen, R. H. Chan, and A. M. Yip, A primal-dual method for total-variation-based wavelet domain inpainting, IEEE Trans. Image Process., vol. 1, pp , Jan. 01. [9] A. Buades, B. Coll, and J. M. Morel, Image denoising methods. a new nonlocal principle, SIAM Rev., vol. 5, pp , Feb [10] X. Zhang and T.F. Chan, Wavelet inpainting by nonlocal total variation, Inverse Probl. Imag., vol. 4, pp , Feb [11] H. Chang, X. Zhang, X.C. Tai, and D. Yang, Domain decomposition methods for nonlocal total variation image restoration, J. Sci. Comput., vol. 60, pp , July 014. [1] J. Xu, X. C. Tai, and L. L. Wang, A two-level domain decomposition method for image restoration, Inverse Probl. Imag., vol. 4, pp , Aug [13] P. Blomgren and T.F. Chan, Color TV: total variation methods for restoration of vector-valued images, IEEE Trans. Image Process., vol. 7, pp , Mar [14] J. Duan, Z. Pan, B. Zhang, W. Liu, and X. C. Tai, Fast algorithm for color texture image inpainting using the non-local CTV model, J. Global Optim., vol. 6, pp , Aug [15] J. F. Cai, B. Dong, S. Osher, and Z. Shen, Image restoration: total variation, wavelet frames, and beyond. J. Am. Math. Soc., vol. 5, pp , Oct. 01. [16] P. L. Combettes and J. C. Pesquet, Proximal splitting methods in signal processing, In Springer Optimization and Its Applications, Vol. 49, Fixed-Point Algorithms for Inverse Problems in Science and Engineering, H. H. Bauschke, R. S. Burachik, P. L. Combettes, V. Elser, D. R. Luke, and H. Wolkowicz, Ed. New York: Springer, 011, pp [17] A. Fawzi, M. Davies, and P. Frossard, Dictionary learning for fast classification based on soft-thresholding, Int. J. Comput. Vis., vol. 114, pp , Sep. 015.