On the regularizing power of multigrid-type algorithms

Size: px

Start display at page:

Download "On the regularizing power of multigrid-type algorithms"

Elisabeth Sharp
6 years ago
Views:

1 On the regularizing power of multigrid-type algorithms Marco Donatelli Dipartimento di Fisica e Matematica Università dell Insubria

2 Outline 1 Restoration of blurred and noisy images The model problem Properties of the PSF Iterative regularization methods 2 Multigrid regularization Multigrid Methods Iterative Multigrid regularization Computational Cost 3 Numerical experiments An airplane An astronomic example with nonnegativity constraint Strengthen the projector Direct multigrid regularization 4 Conclusions

3 Restoration of blurred and noisy images Outline 1 Restoration of blurred and noisy images The model problem Properties of the PSF Iterative regularization methods 2 Multigrid regularization Multigrid Methods Iterative Multigrid regularization Computational Cost 3 Numerical experiments An airplane An astronomic example with nonnegativity constraint Strengthen the projector Direct multigrid regularization 4 Conclusions

4 Restoration of blurred and noisy images The model problem Image restoration with Boundary Conditions Using Boundary Conditions (BCs), the restored image f is obtained solving: (in some way...) g = blurred image ξ = noise (random vector) Af = g + ξ A = two-level matrix depending on PSF and BCs BCs Dirichlet periodic Neumann (reflective) anti-reflective A Toeplitz circulant DCT III DST I + low-rank

5 Restoration of blurred and noisy images Properties of the PSF Generating function of PSF Gaussian PSF PSF Generating function z(x) The ill-conditioned subspace is mainly constituted by the high frequencies.

6 Restoration of blurred and noisy images Iterative regularization methods Iterative regularization methods Several iterative methods (e.g., Landweber, CGNE,...) have regularization properties. Reason They firstly reduce the error in the low frequencies (well-conditioned subspace). When they arrive to reduce the algebraic error in the high frequencies then the restoration error increases because of the noise.

7 Multigrid regularization Outline 1 Restoration of blurred and noisy images The model problem Properties of the PSF Iterative regularization methods 2 Multigrid regularization Multigrid Methods Iterative Multigrid regularization Computational Cost 3 Numerical experiments An airplane An astronomic example with nonnegativity constraint Strengthen the projector Direct multigrid regularization 4 Conclusions

8 Multigrid regularization Multigrid Methods Multigrid structure Multigrid Idea Project the system in a subspace, solve the resulting system in this subspace and interpolate the solution in order to improve the previous approximation. Multigrid components The Multigrid combines two iterative methods: Smoother: a classic iterative method, Coarse Grid Correction: projection, solution of the restricted problem, interpolation. At the lower level(s) it works on the error equation!

9 Multigrid regularization Multigrid Methods The algorithm Two-Grid Methods (TGM) The j-th iteration for the system Ax = b: (1) x = Smooth(A,x (j),b, ν) (2) r 1 = P(b A x) (3) A 1 = PAP H (4) e 1 = A 1 1 r 1 (5) x (j+1) = x (j) + P H e 1 Multigrid (MGM): the step (4) becomes a recursive application of the algorithm.

10 Multigrid regularization Multigrid Methods The Algebraic Multigrid (AMG) The AMG uses information on the coefficient matrix and no geometric information on the problem. Different classic smoothers have a similar behavior: in the initial iterations they are not able to reduce effectively the error in the subspace generated by the eigenvectors associated to small eigenvalues (ill-conditioned subspace) To obtain a fast solver the projector is chosen in order to project the error equation in such subspace. For instance, for Toeplitz and algebra of matrices, see [Aricò, Donatelli, Serra Capizzano, SIMAX, Vol pp ].

11 Multigrid regularization Multigrid Methods Algebraic analysis of the Geometric Multigrid The MGM is an optimal solver for elliptic PDEs For elliptic PDEs the Poisson s problem ill-conditioned subspace is made by low frequencies (complemen tary with respect to the gaussian blur) π 3.5 Projector P = 1 16 P T (full weighting) (linear interpolation)

12 Multigrid regularization Multigrid Methods Multigrid for structured matrices Preserve the structure In order to apply recursively the MGM it is necessary to keep the same structure at each level (Toeplitz,... ). For every structure arising from the proposed BCs there are projectors that preserve the same structure. Projector At i-th level P i = K Ni A Ni (p), where K Ni R N i 4 N i is the cutting matrix that preserves the structure at the lower level. p(x, y) is the generating function of the projector, which selects the subspace where to project the linear system.

13 Multigrid regularization Multigrid Methods Multigrid, structured matrices and images The cutting matrix K ni in 1D circulant Toeplitz&DST I DCT III [ ] [ ] [ ] Low-pass filter Project in the low frequencies reduces the noise effects: 2D p(x, y) = (1 + cos(x))(1 + cos(y)) ց Full weighting ր Linear interpolation

14 Multigrid regularization Iterative Multigrid regularization Image restoration and Multigrid In the image deblurring the ill-conditioned subspace is related to high frequencies, while the well-conditioned subspace is generated by low frequencies. In order to obtain a fast convergence the algebraic multigrid projects in the high frequencies where the noise lives = noise explosion already at the first iteration (it requires Tikhonov regularization [NLAA in press]). In this case the low-pass filter projects in the well-conditioned subspace (low frequencies) = it is slowly convergent but it can be a good iterative regularizer.

15 Multigrid regularization Iterative Multigrid regularization Iterative multigrid regularization The Multigrid as an iterative regularization method If we have an iterative regularization method we can improve its regularizing properties and/or accelerate its convergence using it as smoother in a Multigrid algorithm. Regularization The regularization properties of the smoother are preserved since it is combined with a low-pass filter.

16 Multigrid regularization Iterative Multigrid regularization Two-Level (TL) regularization Idea Project into the low frequencies and then apply an iterative regularization method. TL as a specialization of TGM Smoother: iterative regularization method Projector: low-pass filter TL Algorithm 1 No smoothing at step (1): x = x (j) 2 Step (4): e 1 = A 1 1 r 1 Smooth(A 1,e 1,r 1, 1)

17 Multigrid regularization Iterative Multigrid regularization Multigrid regularization (applying recursively the TL) V-cycle Using a greater number of recursive calls (e.g. W-cycle), the algorithm works more in the well-conditioned subspace but it is more difficult to define an early stopping criterium.

18 Multigrid regularization Computational Cost Computational Cost Precomputing Assumptions: images n n and PSF m m with m n. Computation and memorization of the PSFs (coefficient matrices) for each coarse grids (A i+1 = P T i A i P i ) O(log(n)) V-cycle The problem at level j has size n j n j with n j = n/2 j. Cost of projection at level j: ց 7 4 n2 j, ր 7 8 n2 j flops. Let W(n) the computational cost of one smoother iteration W(n) = cn 2 + O(n), c 1.

19 Multigrid regularization Computational Cost Computational Cost V-cycle (PSF m m with m n) The total cost at level j is about c j = W(n j ) n2 j flops. The total cost of one V-cycle iteration is: log 21 2 (n) 1 8 n2 + c j < 7 2 n2 + 4 ( n ) 3 W W(n). j=1

20 Multigrid regularization Computational Cost Computational cost for more complex cycles Increasing the number of recursive calls Only one recursive call (projection) V scheme. Increasing the number of recursive calls (consecutive projections) the algorithm works more in the low frequencies subspace, however it is more difficult to define a stopping criterium (the method accelerates). γ = number of recursive calls, the cost of one iteration is C(γ,n) C(γ,n) ( cγ ) n 2 4 γ { O(n 2 log(n)), γ = 4 O(n 2log 4 (γ) ), γ > W(n), γ = 1 W(n), γ = 2 3W(n), γ = 3

21 Numerical experiments Outline 1 Restoration of blurred and noisy images The model problem Properties of the PSF Iterative regularization methods 2 Multigrid regularization Multigrid Methods Iterative Multigrid regularization Computational Cost 3 Numerical experiments An airplane An astronomic example with nonnegativity constraint Strengthen the projector Direct multigrid regularization 4 Conclusions

22 Numerical experiments An airplane An airplane Periodic BCs Gaussian PSF (A spd) noise = 1% Original Image Inner part Observed image Restored with MGM

23 Numerical experiments An airplane Restoration error e j = f f (j) 2 / f 2 restoration error at the j-th iteration. Minimum restoration error Method min j) j=1,... arg min j) j=1,... CG Richardson TL(CG) TL(Rich) MGM(Rich, 1) MGM(Rich, 2) CGNE RichNE

24 Numerical experiments An airplane Noise = 10% For CG and Richardson it is better to resort to normal equations. Minimum restoration error Method min j) j=1,... arg min j) j=1,... CGNE RichNE TL(CGNE) TL(RichNE) MGM(RichNE,1) MGM(RichNE,2) MGM(Rich,1) MGM(Rich,2)

25 Numerical experiments An astronomic example with nonnegativity constraint Saturn Periodic BCs and dark background (exact) Gaussian PSF + noise = 5% Original Image PSF Blurred + SNR = 20

26 Numerical experiments An astronomic example with nonnegativity constraint Nonnegativity constraint Does Multigrid preserve the nonnegativity? The projector (full weighting and linear interpolation) preserves the nonnegativity. Also with a projected smoother, the multigrid does not preserve the nonnegativity (cause: direct resolution at the coarser level). Preserving the nonnegativity with multigrid Project the computed solution at each iteration. It is better if we use a projected smoother too.

27 Numerical experiments An astronomic example with nonnegativity constraint Minimum restoration error Method min j) j=1,... argmin j) j=1,... CG Rich MGM(Rich +,1) MGM(Rich,1) MGM(Rich +,1) MGM(Rich +,2) RichNE CGNE MGM(CGNE +,2) MGM(CGNE +,3)

28 Numerical experiments An astronomic example with nonnegativity constraint Graph of the restoration error

29 Numerical experiments An astronomic example with nonnegativity constraint Restored images Rich + (9 iter.). MGM(Rich +,2) + (12 iter.) CGNE + (885 iter.).

30 Numerical experiments An astronomic example with nonnegativity constraint Restored images MGM(CGNE +,2) + (109 iter.) CGNE + (885 iter.).

31 Numerical experiments Strengthen the projector Strengthen the projector With multigrid we can strengthen the filtering properties of the projector instead of to use a smoother for normal equations: p(x, y) = (1 + cos(x)) α (1 + cos(y)) α, α N +. Computational cost increases The computational cost of one W-Cycle iteration become C(α,n) q(α)n 2, q(x) polynomial of degree 2 Linear system conditioning: let f j be the generating function of A j and β j the order of its unique zero in [π, π], then β j = β 0 + 2jα (cond. numb. = O(n β j j )).

32 Numerical experiments Strengthen the projector Strengthen the projector Projection and generalized re-blurring The projector performs a double re-blurring since A i+1 = P T i A i P i. Minimum restoration error with MGM(Rich +,2) + varying α α min j) j=1,... argmin j) j=1,

33 Numerical experiments Strengthen the projector Strengthen the projector MGM(Rich +,2) + (α = 1 in 12 iter.) MGM(Rich +,2) + (α = 4 in 38 iter.) MGM(CGNE +,2) + (109 iter.)

34 Numerical experiments Direct multigrid regularization The γ regularization Varying γ, the proposed multigrid is a direct (one step) regularization method with regularization parameter γ. The airplane example with noise = 1% e 1 = restoration error after one iteration of MGM(Rich,γ). The CGNE reaches a minimum error equal to after 2500 iterations! γ e

35 Numerical experiments Direct multigrid regularization The γ regularization (the airplane with noise = 1%) MGM(Rich,1) MGM(Rich,2) MGM(Rich,4) MGM(Rich,6) Restoration error norm vs. number of iterations The computational cost increases with γ but not so much (e.g. γ = 8 O(N 1.5 ) where N = n 2 ).

36 Conclusions Outline 1 Restoration of blurred and noisy images The model problem Properties of the PSF Iterative regularization methods 2 Multigrid regularization Multigrid Methods Iterative Multigrid regularization Computational Cost 3 Numerical experiments An airplane An astronomic example with nonnegativity constraint Strengthen the projector Direct multigrid regularization 4 Conclusions

37 Conclusions Summarizing... Multigrid regularization method It is a general framework which can be used to improve the regularization properties of an iterative regularizing method. It leads to a smaller relative error and a more flat error curve with respect to the smoother applied alone. It is fast and it obtains a good restored image without resort to normal equations. It can be combined with several techniques (e.g., nonnegativity constraints) and it can lead to several generalizations (e.g., γ regularization). Reference M. Donatelli and S. Serra Capizzano, On the regularizing power of multigrid-type algorithms, SIAM J. Sci. Comput., to appear.

38 Conclusions Future work Theoretical Theoretical analysis of the regularization property. Application PSF strictly non-symmetric. Combination with techniques for edge enhancing (Wavelet, Total Variation,... ). Numerical/Simulating A complete experimentation with all the proposed BCs (multigrid methods already exist for the arising matrices).

Image deblurring by multigrid methods. Department of Physics and Mathematics University of Insubria

Image deblurring by multigrid methods Marco Donatelli Stefano Serra-Capizzano Department of Physics and Mathematics University of Insubria Outline 1 Restoration of blurred and noisy images The model problem