An Iteratively Reweighted Norm Algorithm for Total Variation Regularization

Transcription

1 An Iteratively eweighted Norm Algorithm for Total Variation egularization Paul odríguez and Brendt Wohlberg Abstract Total Variation TV regularization has become a oular method for a wide variety of image restoration roblems, including denoising and deconvolution. ecently, a number of authors have noted the advantages, including suerior erformance with certain non-gaussian noise, of relacing the standard l data fidelity term with an l norm. We roose a simle but very flexible and comutationally efficient method, the Iteratively eweighted Norm algorithm, for minimizing a generalized TV functional which includes both the l -TV and and l -TV roblems. I. INTODUCTION Total Variation TV regularization was first introduced for image denoising [], and has since evolved into a more general tool for solving a wide variety of image restoration roblems, including deconvolution and inainting [], [3]. The l -TV regularized solution of the inverse roblem involving data b and forward linear oerator A the identity in the case of denoising, and a convolution for a deconvolution roblem, for examle is the minimum of the functional Ju = Au b + λ D x u + D y u where D x u + D y u is the total variation of u, and D x, D y denote the horizontal and vertical discrete derivative oerators resectively. While a variety of algorithms [4], [5] have been roosed to solve this otimization roblem, it remains a comutationally exensive task which can be rohibitively costly for large roblems and non-sarse forward oerator A. ecently, the modified TV functional with an l data fidelity term Ju = Au b + λ D x u + D y u has attracted attention [6], [7] due to a number of advantages, including suerior denoising erformance with salt and eer seckle noise [8]. The standard aroaches to solving roblem are not effective for roblem, for Paul odríguez is with T-7 Mathematical Modeling and Analysis, Los Alamos National Laboratory, Los Alamos, NM 87545, USA. rodrig@t7.lanl.gov, Tel: , ax: Brendt Wohlberg is with T-7 Mathematical Modeling and Analysis, Los Alamos National Laboratory, Los Alamos, NM 87545, USA. brendt@t7.lanl.gov, Tel: , ax: This work was carried out under the ausices of the National Nuclear Security Administration of the U.S. Deartment of Energy at Los Alamos National Laboratory under Contract No. DE-AC5-6NA5396 and was artially suorted by the NNSA s Laboratory Directed esearch and Develoment Program. which efficient time-erformance and memory reuirements algorithm develoment is not well advanced [7]. In this aer, we introduce the Iteratively eweighted Norm IN algorithm for solving the generalized TV functional Ju = Au b + λ D x u + D y u 3 for and we have found the algorithm to converge for smaller and values, but cannot rove that the algorithm converges in these cases. The IN algorithm is a simle but comutationally efficient and very flexible method which is cometitive with existing, well-established [4], [5] algorithms for l -TV, and while we have not comared it with all of the l -TV algorithms of which we are aware [7], [8], [9], [], [], it is significantly faster than those with which we have erformed comarisons, and in contrast to many of these other methods, easily allows the inclusion of forward oerator A for a more general inverse roblem than denoising. II. ITEATIVELY EWEIGHTED NOM APPOACH A. Previous elated Work The IN aroach is closely related to the Iteratively eweighted Least Suares ILS [], [3] method, which minimizes the l norm Ju = Au b 4 by aroximating it, within an iterative scheme, by a weighted l norm roblem, for which an algebraic solution is easily derived. It is also worth mentioning similar methods used for constructing sarse signal reresentations [4], [5]. A brief outline of the convergence roof for ILS rovides a useful introduction to the outline of the IN convergence roof resented in Section II-D. At each iteration k, define the functional Q k u = W k/ Au b + Ju k, where W k = diag Au k b, and u k is the current solution i.e., the solution obtained at the end of the revious iteration. It may be shown that Ju Q k u u Ju k = Q k u k Ju k = Q k u k, i.e. the uadratic functional Q k u is tangent to Ju at u = u k, where it is also an uer bound for Ju. As a

2 result [3], the iteration u k+ = A T W k A A T b, 5 which minimizes Q k u using the weights derived from the revious iteration, converges to the minimizer of Ju. When <, the definition W k = diag Au k b must be modified to avoid the ossibility of division by zero. or =, it may be shown see [6] that the choice W k n,n = r k ɛ n if r n k ɛ if r n k < ɛ, where r k = Au k b, and ɛ is a small ositive number, guarantees global convergence to the minimizer of ρ ɛ r n, n where ɛ ρ ɛ r n = rn if r n ɛ r n ɛ if r n > ɛ is the Huber function. We will address a similar uestion for the IN algorithm in the following sections. B. IN: Data idelity Term The data fidelity term of Euation 3 has the form of the ILS functional in Euation 4, and is handled in the same way, reresenting Au b by Au b W / with iteratively udated weights W. Since the choice of W as described in Section II-A gives infinite weights for zerovalued comonents of Au b, when <, we set W = diag f Au b, where f is defined for some small ɛ as x if x > ɛ f x = ɛ if x ɛ, which is a common aroach for ILS algorithms [7]. C. IN: egularization Term It is not uite as obvious how to exress the TV regularization term from Euation 3 as a weighted l norm. Given vectors ξ and χ we have using block-matrix notation so that when we have W / W / ξ χ = k W = diag ξ + χ / W / W / ξ χ w k ξ k + w k χ k = ξ + χ. We therefore set ξ = D x u, χ = D y u, and define the oerator D and weights W D = Dx D y W = W W so that W / Du = W / D xu + W / D yu with weights defined by W = diag Dx u + D y u / gives the desired term. Note that this is not the searable aroximation, as in [8], for examle, which is often used. As in the case of the data fidelity term, care needs to be taken when < and D x u + D y u has zero-valued comonents. We define x / if x > ɛ f x = if x ɛ for some small ɛ, and set W = diag f Dx u + D y u. Note that f sets values smaller than the threshold, ɛ, to zero, as oosed to f, which sets values smaller than the threshold, ɛ, to ɛ. The motivation for this choice is that a region with very small or zero gradient should be allowed to have zero contribution to the regularization term, rather than be clamed to some minimum value. In ractice, however, we have found that this choice does not give significantly different results than the standard ILS aroach reresented by f. D. Convergence of the IN algorithm Here we briefly sketch the roof of global convergence of the IN algorithm. Combining the terms described in Sections II-B and II-C, we define the generalized TV functional Q k u = W k/ Au b u k + λ + λ k/ W Du + u k, 6 where W k, W k, and uk are the current data fidelity weights, regularization weights, and solution resectively. The constant with resect to u terms u k = Au k b u k = D x u k + D y u k do not effect the IN iteration since they have zero gradient with resect to u, but are necessary so that Ju k = Q k u k. Note that Ju k = uk + λ uk, as in Euation 3. It is easily shown that Ju Q k u u Ju k = Q k u k Ju k = Q k u k Q k u = A T W k A + λdt k W D >

3 Algorithm IN algorithm - General case Initialize u = A T A + λd T D A T b Iterate W k W k = diag f Au k b = diag f D x u k + D y u k u k = A T W k +λdy T W k D y A + λdt x W k D x A T W k b ig Lena image. As in the ILS case Section II-A, the uadratic functional Q k u is tangent to Ju at u = u k, where it is also an uer bound for Ju, and it has a ositive definite Hessian. Using these results, it can be shown that the minimizer of Q k u, given by u = A T W k A + λdt W k D A T W k b, 7 converges to the minimizer of 3 as we iterate over k. III. IN ALGOITHMS The IN algorithm can be easily derived see Algorithm from Euation 7. The matrix inversion can be achieved using the Conjugate Gradient CG or Preconditioned CG PCG method such as Jacobi line relaxation JL or symmetric Gauss-Seidel line relaxation SLGS [9]. We have also found that a significant seed imrovement may be achieved by starting with a high CG/PCG tolerance, which is decreased with each main iteration until the final desired value is reached. or the results reorted hereafter, oerators D x and D y are defined by alying the same one-dimensional discrete derivative along image rows and columns resectively. Alied to vector u N, this discrete one-dimensional derivative is comuted as u k u k+ for the derivative at index k,,..., N }, and as u N 3 8u N / for the derivative at index N. Values in the ranges to 4 and 4 to 8 were used for constants ɛ and ɛ resectively. All rogram run times were obtained using the NUMIPAD library [] on a Linux 3.GHz Intel Pentium-4 rocessor. In the remainder of this aer we shall restrict our attention to the l -TV case =, =, but note that this flexible aroach is caable of efficiently solving other cases as well, including the standard l -TV case =, = where we have found it to have similar erformance to the lagged diffusivity algorithm [4]; for a 5 5 inut image igure, normalized between and, corruted with Gaussian noise σ =.5 lagged diffusivity reuired.69s for 5 iterations, giving a denoised image with 6.4 db, while the l -TV IN via PCG-SLGS reuired.69s for 5 iterations, giving a 6. db outut. The number of CG iterations er main loo needed by lagged-diffusivity and several flavors of the IN algorithm are shown in igure. CG Iterations l TV Lagged l TV IN CG l TV IN PCG JL l TV IN PCG SLGS Main Iteration Number ig.. A comarison of CG iterations for lagged-diffusivity and several flavors of the IN algorithm for l -TV denoising. The IN algorithm is slow when couled with a standard CG solver, but the use of reconditioned CG PCG strategies, such as Jacobi line relaxation JL and symmetric Gauss- Seidel line relaxation SLGS, dramatically reduces the number of reuired CG iterations. A. IN l -TV Denoising Direct alication of Euation 7 for l -TV denoising is slow due to the ill-conditioning of the system to be inverted. However, in this case we may aly the substitution ũ = W k/ ũ = u to Euation 7, giving I + λw k / D T k W / DW k / W k/ b. Alying this modification indirect IN algorithm to the general direct algorithm results in a significant reduction in the reuired number of CG iterations; moreover if a PCG strategy is alied e.g. JL or SLGS, the number of CG iterations is further reduced, as shown in igure 4, where a comarison between the number of CG iteration for 5 loos for the direct and indirect CG, PCG-JL and PCG-SLGS IN algorithms is carried out to denoise a 5 5 image igure corruted with 5% seckle noise. The run times were 8.s,.s,.3s

4 a Image with % seckle noise. SN: -.76dB. b Denoised l -TV λ =.5. SN: 7.4dB. Time:.63s c Image with 5% seckle noise. SN: -5.77dB. d Denoised l -TV λ =.5. SN: 9.6dB. Time:.6s ig. 3. Extreme examles for l -TV denoising using the IN algorithm: original image igure was corruted with % and 5% seckle noise. and.3s resectively, with denoised ualities of 7.36 db,.85 db,.35 db, and.36 db resectively. More extreme examles are resented in igure 3, with the original image corruted by % igure 3a and 5% igure 3c seckle noise. The result of l -TV denoising these images, using the IN via PCG-JL algorithm, is surrisingly good: 7.4 db igure 3b in.63s when denoising igure 3a and 9.6 db igure 3d in.6s when denoising igure 3c. B. IN l -TV Deconvolution We aly the IN algorithm to the roblem of deconvolution of an image convolved by a searable smoothing filter having 9 tas, aroximating a Gaussian with standard deviation of.. In this case A is the corresonding linear oerator, and the substitution alied in the revious section is no longer ossible. We constructed a test image by convolving the image in igure by the smoothing kernel, and adding 5% seckle noise see igure 5a, giving an image with an SN of 4.dB. Comaring the erformance of l -TV via the lagged diffusivity algorithm and l -TV via the IN method deconvolution, we obtain a.db reconstruction in 9.s and a 4.4dB reconstruction in 6.8s resectively see igure 5b. IV. CONCLUSIONS The Iteratively eweighted Norm IN aroach rovides a simle but comutationally efficient method for TV regularized otimization roblems, including both denoising and those such as deconvolution having a linear oerator in the data fidelity term. This framework is very flexible, and can be alied to regularized inversions with a wide variety of norms for the data fidelity and regularization terms, including the standard l -TV, and more recently roosed l -TV formulations. This method rovides a significantly faster algorithm for the l -TV formulation than any other algorithm of which we are aware. The NUMIPAD library [], is roving to be a useful tool for solving otimization roblems and can be used to reroduce the results resented in this aer.

5 8 6 4 IN l direct IN l indirect CG IN l indirect PCG JL IN l indirect PCG SLGS CG Iterations Main Iteration Number ig. 4. A comarison of CG iterations for the direct and several flavors of the indirect IN algorithm for l -TV denoising. Both reconditioned CG PCG strategies, Jacobi line relaxation JL and symmetric Gauss-Seidel line relaxation SLGS, dramatically reduced the number of reuired CG iterations. a Blurred image with 5% seckle noise. SN: 4.dB. ACKNOWLEDGMENT The authors thank Markus Berndt for valuable discussion on CG reconditioning methods. EEENCES [] L. udin, S. Osher, and E. atemi, Nonlinear total variation based noise removal algorithms, Physica D, vol. 6, , 99. [] C.. Vogel and M. E. Oman, ast, robust total variation-based reconstruction of noisy, blurred images, IEEE Transactions on Image Processing, vol. 7, , June 998. [3] T. Chan, S. Esedoglu,. Park, and A. Yi, ecent develoments in total variation image restoration, in The Handbook of Mathematical Models in Comuter Vision N. Paragios, Y. Chen, and O. augeras., eds., Sringer, 5. [4] C.. Vogel and M. E. Oman, Iterative methods for total variation denoising, SIAM Journal on Scientific Comuting, vol. 7,. 7 38, Jan [5] A. Chambolle, An algorithm for total variation minimization and alications, Journal of Mathematical Imaging and Vision, vol., , 4. [6] M. Nikolova, Minimizers of cost-functions involving nonsmooth datafidelity terms. alication to the rocessing of outliers, SIAM Journal on Numerical Analysis, vol. 4, no. 3, ,. [7] T.. Chan and S. Esedoglu, Asects of total variation regularized l function aroximation, SIAM Journal on Alied Mathematics, vol. 65, no. 5, , 5. [8] M. Nikolova, A variational aroach to remove outliers and imulse noise, Journal of Mathematical Imaging and Vision, vol., no. -,. 99, 4. 4th Conference on Mathematics and Image Analysis; Setember -3, ; Paris, rance. [9] J. Darbon and M. Sigelle, A fast and exact algorithm for total variation minimization, Lecture Notes in Comuter Science, vol. 35/5, , 5. [] D. Goldfarb and W. Yin, Second-order cone rogramming methods for total variation-based image restoration, SIAM J. Sci. Comut., vol. 7, no., , 5. [] J.-. Aujol, G. Gilboa, T. Chan, and S. Osher, Structure-texture image decomosition - modeling, algorithms, and arameter selection, International Journal of Comuter Vision, vol. 67, no.,. 36, 6. []. Wolke and H. Schwetlick, Iteratively reweighted least suares: Algorithms, convergence analysis, and numerical comarisons, SIAM Journal on Scientific and Statistical Comuting, vol. 9,. 97 9, Set [3] K. P. Bube and. T. Langan, Hybrid l /l minimization with alications to tomograhy, Geohysics, vol. 6, , July-August 997. [4] I.. Gorodnitsky and B. D. ao, A new iterative weighted norm minimization algorithm and its alications, in IEEE Sixth SP Worksho on Statistical Signal and Array Processing, 99, Victoria, BC, Canada, Oct. 99. b Deblurred l -TV λ =.3. SN: 4.4dB. Time: 6.8s ig. 5. l -TV deconvolution using the IN algorithm. The original image igure was convolved by a searable smoothing filter before the addition of 5% seckle noise. [5] B. D. ao and K. Kreutz-Delgado, An affine scaling methodology for best basis selection, IEEE Transactions On Signal Processing, vol. 47,. 87, Jan [6] S. A. uzinsky and E. T. Olsen, L and L minimization via a variant of Karmarkar s algorithm, IEEE Transactions on Acoustics, Seech and Signal Processing, vol. 37, , 989. [7] J. A. Scales and A. Gersztenkorn, obust methods in inverse theory, Inverse Problems, vol. 4,. 7 9, Oct [8] Y. Li and. Santosa, A comutational algorithm for minimizing total variation in image restoration, IEEE Transactions on Image Processing, vol. 5, , June 996. [9] W. L. Briggs, V. E. Henson, and S.. McCormick, A Multigrid Tutorial. SIAM Books, nd ed.,. [] P. odríguez and B. Wohlberg, Numerical methods for inverse roblems and adative decomosition NUMIPAD. Software library available from htt://numiad.sourceforge.net/.