ART Exhibit. Per Christian Hansen. Tommy Elfving Touraj Nikazad Hans Henrik B. Sørensen. joint work with, among others

Size: px

Start display at page:

Download "ART Exhibit. Per Christian Hansen. Tommy Elfving Touraj Nikazad Hans Henrik B. Sørensen. joint work with, among others"

Chastity Morgan
5 years ago
Views:

1 ART Exhibit Per Christian Hansen joint work with, among others Tommy Elfving Touraj Nikazad Hans Henrik B. Sørensen

2 About Me Forward problem Interests: numerical methods for inverse problems and tomography, fast and reliable numerical regularization algorithms, matrix computations, image deblurring algorithms, signal processing, Matlab software, 2/27 P. C. Hansen ART Exhibit

3 Why are We Interested in ART? There are many ways to compute reconstructions in tomography: explicit inversion formulas, Bayesian methods, algebraic iterative methods, variational formulations,... I will focus on a particular algebraic iterative method, ART: surprisingly simple to formulate, has a simple geometric interpretation, works well for a number of applications, has fast initial convergence, easily allows simple constraints (e.g., nonnegativity). 3/27 P. C. Hansen ART Exhibit

4 What is ART? A simple iterative procedure for solving Ax = b where each iteration updates x via sweeps over the rows a T i of the matrix A. Kaczmarz (1937): orthogonally project x on the hyperplane defined by a T i and the corresponding element b i of the right-hand side: x P i x = x + b i a T i x ka i k 2 a i, i =1, 2,...,m. 2 Gordon, Bender, Herman (1970): coined the term ART and introduced a nonnegativity projection: ½ x max 0,x+ b i a T i x ¾ ka i k 2 a i, i =1, 2,...,m. 2 ART is now used synonymously with Kaczmarz s formulation with a relaxation parameter ω k and a projection P C on a convex set: µ b i a T i x P C x + ω x k ka i k 2 a i, i =1, 2,...,m. 2 4/27 P. C. Hansen ART Exhibit

5 Software for ART I am afraid that this list is far from complete. SNARK09: C++ package from NYU, 2D reconstructions. ASTRA: C++ & CUDA with Matlab wrapper, from Antwerp + CWI. Image reconstruction toolbox: Matlab package from Prof. Jeff Fessler, Univ. of Michigan AIR Tools: Matlab package from DTU. What did I miss? 5/27 P. C. Hansen ART Exhibit

Semi-convergence theory. Implementation of block ART.

6 Some Interesting ART Topics ART is a rich source for research problems! This list is quite biased towards my own work with the AIR Tools. Semi-convergence theory. Implementation of block ART. Choice of relaxation paremter. Stopping Rules. Extensions and variations of ART. This presentation 6/27 P. C. Hansen ART Exhibit

7 Semi-Convergence Notation: b = A x + e, x =exactsolution, e =noise. Initial iterations: the error k x x k k 2 decreases. Later: the error increases as x k (weighted) least squares solution. k x xkk2 7/27 P. C. Hansen ART Exhibit A few references: F. Natterer, The Mathematics of Computerized Tomography (1986) A. van der Sluis & H. van der Vorst, SIRT- and CG-type methods for the iterative solution of sparse linear least-squares problems (1990) M. Bertero & P. Boccacci, Inverse Problems in Imaging (1998) M. Kilmer & G. W. Stewart, Iterative Regularization And Minres (1999) H. W. Engl, M. Hanke & A. Neubauer, Regularization of Inverse Problems (2000)

8 Illustration of Semi-Convergence 8/27 P. C. Hansen ART Exhibit

9 Analysis of Semi-Convergence for ART Elfving, H, Nikazad, Semi-convergence properties of Kaczmarzs method, Inverse Problems, 30 (2014), DOI: / /30/5/ Let x be the solution to the noise-free problem, and let x k denote the iterates when applying ART to b. Then kx k xk 2 kx k x k k 2 + k x k xk 2. Noise error Iteration error The convergence theory for ART is well established and ensures that the iteration error x k x goes to zero. Our concern here is the noise error e N k = x k x k. We wish to establish that it increases, and how fast. 9/27 P. C. Hansen ART Exhibit

10 Sidetrack: Noise Error for Landweber Steepest descent for LSQ problem: x k+1 = P C xk + ωa T (b Ax k ). The unprojected case: x k is a filtered SVD solution: x k = P n i=1 ϕ[k] i u T i b σ i v i ϕ [k] i = 1 1 ωσ 2 i k. Filter factors ϕ [k] i =1 1 ωσ 2 i k With projection an SVD analysis is not possible; we obtain: kx k x k k 2 σ 1 (1 ωσn) 2 k kbk 2 σ n σ n and for ωσ 2 n 1wehave: kx k x k k 2 ω k kak 2 kbk 2. Elfving, H, Nikazad, /27 P. C. Hansen ART Exhibit

11 Noise Error for ART No Projection ART is equivalent to applying SOR to AA T y = b, x = A T y. Splitting: AA T = L + D + L T, M =(D + ωl) 1, where L is strictly lower triangular and D =diag(ka i k 2 2). Then: x k+1 = x k + ωa T M (b Ax k ). We introduce: e = b b =noiseindata, Q = I ωa T MA. Then simple manipulations show that the noise error is given by k 1 X e N k = x k x k = Qe N k 1 + ωa T Me= ω Q j A T Me. j=1 After some work (see the paper) we obtain the bound ke N k k 2 ωδ 1 qk 1 q = ω k kat Mek 2 + O(σ 2 r). 11/27 P. C. Hansen ART Exhibit

12 Noise Error Analysis A Tighter Bound Further analysis (see the paper) shows that the noise error in ART is bounded above as: ke N k k 2 kat Mek 2 Ψ k + O(σ σ r), 2 Ψ k = 1 (1 ωσ2 r) k. r σ r As long as ωσ 2 r < 1wehave Ψ k for ω =1 and thus ke N k k 2 Ψ k ω k ωka T Mek 2 σ r k + O(σ 2 r ). σ r σ r σ r σ r This also holds for projected ART provided that A and P satisfy y R(A T ) P(y) R(A T ). 12/27 P. C. Hansen ART Exhibit k

13 Numerical Results ( paralleltomo from AIR Tools) The point of semi-convergence arises when noise error iteration error. Test problem: phantom, 60 projections at 3,6,9,,180, m = 15,232, n = 40,000. 1% noise 1% noise We estimate ωka T Mek σ r Hence our bound is a wild over-estimate but it correctly tracks the noise error. 5% noise 5% noise 13/27 P. C. Hansen ART Exhibit

14 Implementation Issues SIRT (Cimmino): x P x + ω A T D 2 (b Ax) kx k xk2/k xk2 Slow convergence. ART can converge a lot faster than SIRT. 14/27 P. C. Hansen ART Exhibit

Performance 1 core kx k xk 2 /k xk 2 Relative error Iterations k ART Intel Xeon E5620 2.

15 Performance 1 core kx k xk 2 /k xk 2 Relative error Iterations k ART Intel Xeon E GHz (1 core) Test Problem: Parallel-beam tomography. 13 projections. 3D Shepp-Logan phantom, Schabel (2006). 15/27 P. C. Hansen ART Exhibit

16 Performance 1 core ART SIRT Intel Xeon E GHz (1 core) Same number of flops! The difference is due to the cache: ART uses row a i twice once it is loaded. 16/27 P. C. Hansen ART Exhibit

17 Performance 4 cores ART SIRT SIRT 1 ART core 4 cores Intel Xeon E GHz (4 cores) Four cores are better suited for block matrixvector operations. 17/27 P. C. Hansen ART Exhibit

18 Our Dilemma ART has faster convergence than SIRT i.e., more reduction of the error per iteration. SIRT can better take advantage of multi-core architecture than ART. How to achieve the best of both worlds? Block methods! H. H. B. Sørensen and P. C. Hansen, Multi-core performance of block algebraic iterative reconstruction methods, SIAM J. Sci. Comp., 36 (2014), pp. C524 C /27 P. C. Hansen ART Exhibit

19 Block Methods In each iteration we can: Treat the blocks sequentially or simultaneously (i.e., in parallel). Treat each block by an iterative or by a direct computation. We obtain several methods: Sequential processing + ART on each block classical ART Sequential processing + SIRT on each block Sequential processing + pseudoinverse of A l Parallel processing + ART on each block Parallel processing + SIRT on each block classical SIRT Parallel processing + pseudoinverse of A l 19/27 P. C. Hansen ART Exhibit

20 Block-Sequential Methods Algorithm: Block-Sequential Initialization: choose an arbitrary x 0 R n Iteration: for k =0, 1, 2,... x k,0 = x k 1 x k,` = P x k,` 1 + ω A T` M` (b` A` x k,` 1 ), x k = x k 1,p SART: Andersen, Kak (1984) Block-Iteration: Censor (1988) ` =1, 2,...,p The convergence depends on the number of blocks p: If p = 1, we recover SIRT If p = m, we recover ART Parallelism given by the tradeoff: Variant by Elfving (1980): M` =(A`A T` ) A T` M` = A ` 20/27 P. C. Hansen ART Exhibit

21 Block-Parallel Methods Algorithm: Block-Parallel Initialization: choose an arbitrary x 0 R n Iteration: for k =0, 1, 2,... for ` =1,...,p execute in parallel x k,` =ART-sweep(ω,A`,b`,x k 1 ) x k P = 1 /p p`=1 xk,`. String-Averaging: Censor, Elfving, Herman (2001) The convergence depends on p: If p = 1, we recover ART If p = m, we recover SIRT Parallelism is given by: Variants: Elfving (1980) inner step: CARP algorithm, Gordon & Gordon (2005): x k,` = P x k 1,` + ω A `(b` A` x k 1,`) x k = P p `=1 D` x k,`, D` depends on sparsity structure 21/27 P. C. Hansen ART Exhibit

Thus, a re-ordering of the rows can produce blocks with mutually orthogonal rows (= the traces of rays are

22 Blocks of Structurally Orthogonal Rows In 3D tomography, it is easy to find sets of rows that are orthogonal due to the structure of zeros/nonzeros. Thus, a re-ordering of the rows can produce blocks with mutually orthogonal rows (= the traces of rays are non-overlapping). When a block has structurally orthogonal rows then ART, SIRT and pinv are equivalent. It is worthwhile to utilize this! PART algorithm, Gordon (2006) 22/27 P. C. Hansen ART Exhibit

23 Block Sequential Intel Xeon E GHz (4 cores) ART SIRT Block-Seq. 4 blocks 23/27 P. C. Hansen ART Exhibit The building blocks are SIRT iterations, suited for multicore. The blocks are treated sequentailly! Hence the error reduction per iteration is close to that of ART.

24 Block Parallel Intel Xeon E GHz (4 cores) ART SIRT Block Seq. Block Par. 4 blocks 24/27 P. C. Hansen ART Exhibit

Multi-Core Results 4 Cores Intel Core i7-3820 3.60 GHz (4 cores) The advantage of PART over standard ART is due to the improved use of multicore architecture.

25 Multi-Core Results 4 Cores Intel Core i GHz (4 cores) The advantage of PART over standard ART is due to the improved use of multicore architecture. Block-Seq: block-sequential-sirt Block-Par: block-parallel-art (Censor, Elfving, Herman) CARP: block-parallel-art (Gordon, Gordon) PART utilizes struct. orthog. ART (1 thread) voxels 115 projections of pixels 25/27 P. C. Hansen ART Exhibit

26 Multi-Core Results 32 Cores 4 socket AMD Opteron 6282 SE 2.60 GHz (32 cores) With many cores, PART is a clear winner. Block-Seq: block-sequential-sirt Block-Par: block-parallel-art (Censor, Elfving, Herman) CARP: block-parallel-art (Gordon, Gordon) PART utilizes struct. orthog. ART (1 thread) voxels 133 projections of pixels 26/27 P. C. Hansen ART Exhibit

Conclusions Block algebraic iterative reconstruction

similar to that of ART, and with the smaller computing time of

With a suitable row ordering and choice of blocks, we can

PART has identical convergence to ART and very good scaling

27 Conclusions Block algebraic iterative reconstruction techniques are able to achieve initial convergence rate similar to that of ART, and with the smaller computing time of SIRT, because we can utilize the multicore architecture. With a suitable row ordering and choice of blocks, we can produce blocks of structurally orthogonal rows. PART has identical convergence to ART and very good scaling properties in practice. Next step: target GPUs (joint work with ASTRA group). 27/27 P. C. Hansen ART Exhibit

Column-Action Methods in Image Reconstruction

Column-Action Methods in Image Reconstruction Per Christian Hansen joint work with Tommy Elfving Touraj Nikazad Overview of Talk Part 1: the classical row-action method = ART The advantage of algebraic