THE DEVELOPMENT OF THE POTENTIAL AND ACADMIC PROGRAMMES OF WROCLAW UNIVERISTY OF TECH- NOLOGY ITERATIVE LINEAR SOLVERS

Transcription

1 ITERATIVE LIEAR SOLVERS. Objectives The goals of the laboratory workshop are as follows: to learn basic properties of iterative methods for solving linear least squares problems, to study the properties (convergence, computational complexity, sensitivity to errors) of the selected linear solvers, to train the skills in coding the selected algorithms in Matlab and in using the Matlab functions for solving systems of linear equations. The workshop is scheduled for 3 academic hours.. Introduction * Iterative linear solvers attempt to iteratively approximate the solution = [ x ] I the system of linear equations: M where = [ ] I a ij A is a coefficient matrix and [ ] M following updates: x to Ax = b, () b = b I is a data vector, using the k ( ) ( k+ ) ( x = f x ), A, b ( ) where x k is the approximation to the solution x in the k-th iterative step, and f (,, is the ( update function determined by the underlying iterative method, where k ) * lim f x, A, b x. i, k j ) ( ) This laboratory workshop focuses on the two main classes of iterative solvers: stationary iterative methods (Richardson, Landweber, ewton, Jacobi, Gauss-Seidel, SOR) and Krylov subspace methods (CG, BiCG, BiCGSTAB, GMRES, QMR, LSQR). 3. Preparation. The expected time needed for the preparation to this workshop is 9 hours. 3.. Reading []. A. Bjorck, umerical Methods for Least Squares Problems, SIAM, Philadelphia, 996, []. G. Golub, C. F. Van Loan, Matrix Computations, The John Hopkins University Press, (Third Edition), 996. [3]. Y. Saad, Iterative Methods for Sparse Linear Systems, SIAM, 003. [4]. J. ocedal, S. J. Wright, umerical Optimization, Springer, 999, pp. 63, Project co-financed by European Union within European Social Fund

2 [5]. C. D. Meyer, Matrix Analysis and Applied Linear Algebra, SIAM, 000, [6]. Ch. Zarowski, An Introduction to umerical Analysis for Electrical and Computer Engineers, Wiley, 004, 3.. Problems At the beginning of the laboratory workshop each student should know the answers to the following questions: How to check whether a given system of linear equations has none, one, or many solutions? What is an over-determined, determined, and under-determined system of linear equations? How to formulate a Least Square (LS) problem to a system of linear equations? How to derive a system of normal equations to an LS problem? What is a probabilistic and a geometric interpretation of LS approximations (draw an example of curve fitting)? How to estimate the coefficients of a linear combination of the basis functions to fit data in the LS sense (linear regression problem)? Give examples of the fundamental stationary iterative methods for solving systems of linear equations. Give examples of the Krylov subspace methods for solving systems of linear equations. How to define the spectral radius? Explain how the CG method works (give an example). What is a convergence rate between the steepest descent gradient method and the ewton method? What is the Hessian matrix and how to calculate it for the linear LS problem? What is preconditioning and how does it work? What is the difference between parallel and coordinate-wise implementation of iterative methods? How to solve the constrained linear LS problems? 3.3. Detailed preparation Each group of students ( 3 persons) is expected to accomplish the following tasks:. code the selected algorithms for solving systems of linear equations in Matlab,. solve the selected problems with the coded algorithms, 3. compare the results (convergence rate, elapsed time, computational cost, etc.) obtained with the coded algorithms and with in-built functions in Matlab, 4. draw the conclusions. Project co-financed by European Union within European Social Fund

3 Problems to be solved The following problems that are given or can be expressed in terms of systems of linear equations should be solved with the coded algorithms. Problem : Solve the following system with the selected iterative solvers u v= 0 u + v w = 0. v + w z = 0 w+ z = 5 Estimate the computational costs and convergence rates. Start the iterations from zero-value initial guess. Problem : Solve the following system of linear equations using the selected iterative solvers: x+ x + x3 = x+ x + x3 =. x + x + x3 = Compare and discuss the computational costs with respect to convergence rates. Start the iterations from x (0) = 0. Problem 3: Estimate the exact solution to the following system of linear equations using the Gaussian elimination: 0.835x x = 0.68, 0.333x+ 0.66x = 0.067, Then slightly perturb b from to and compute iterative approximations to the perturbed system using the selected iterative solvers. Draw the distance between the exact solution and the iterative approximations versus iterations. Start the iterations from (0) (0) x = x = 0. Estimate the computational costs for each tested methods and compare them with the costs for the Gaussian elimination. Problem 4: Solve the system of linear equations: Ax = b, where: = A, = [ ] b K T, Project co-financed by European Union within European Social Fund 3

4 with the selected iterative solvers and compare the results to the Gaussian elimination. Start the iterations from (0) ( k) ( ) x = 0. Draw the residual error: r = b-ax k and the approximation error: x -x, where the exact solution is obtained with the Gaussian elimination. Use any preconditioning and compare the results. Problem 5: Let A = a ij R, with [ ] T b = K R. Solve the system Ax = b a = ij i + j (Hilbert matrix), and for = 5, 0 and 0, using the selected it- (0) erative methods. Start the iterations from x = 0. Draw the residual error: ( k) ( k) r = b-ax. Which iterative solver and preconditioning are the most appropriate for solving this problem? Problem 6: For = 5, 0, 5, 0 solve the system of linear equations: Ax = b, where sin() sin() sin( ) T b = K R, and A is the Vandermonde matrix with the (-)- T th column equal to [ K ]. To generate this matrix, use the function vander(.) from Matlab. Then perturb the vector b with a zero-mean uncorrelated Gaussian error (SR = 30 db) and determine the approximation error versus the size. Problem 7: Let [ 0 ] T c = K R and = [ 0 + ] * is the first column and r is the first row of the Toeplitz, and [ ] r K R. Assuming c T x = K R, perform the forward projection: Ax * = b, and solve the system of linear equations with the selected iterative solvers. Draw the errors: r = b-ax and x -x. Which ( k) ( k) method gives the best approximations? Explain why the approximation error is large even for a large number of iterations. Problem 8: Let 0 O 0 O A = 0 O O 0 R, b [ K ] T = R Solve the system Ax = b with the selected iterative solvers for = 0, 00, 000, 0000, For each draw the residual error ( k r ) = b-ax ( k) versus the k-the iterative step. Explain the difference in the convergence behavior. T Problem 9: Let Ax = b, where A= I C C, the symbol denotes the Kronecker product, I M M R is an identity matrix, C R is a random matrix with a uniform distribution, M = 00, and = 50, and x ( 0, I M ). Find the iterative method that solves the above system of linear equations with the lowest computational cost. Estimate the cost with a Project co-financed by European Union within European Social Fund 4

5 roughly calculated number of flops and with the elapsed time. Draw the errors x -x ver- sus iterations. 0 0 Problem 0: Solve Ax = b, where A R is the Hilbert matrix, and x ( 0, I 0 ), with the selected iterative solvers and with the Gaussian elimination in Matlab (x := A\b). Then make a small change in an entry of A or b, and compare the solutions. Draw the errors x -x versus iterations for the perturbed system. Algorithms to be coded Students are expected to code the following algorithms and apply them to the abovementioned nonlinear problems. Algorithm : The Richardson s first-order or Landweber method: [], Chapter 7, Section 7..3., pp Algorithm : The Jacobi s method: [] Chapter 7, Section 7..3., pp ; [] Chapter 0, Section 0..-., pp. 50 5; [3] Chapter 4, Section 4.., pp Algorithm 3: The Gauss-Seidel method: [] Chapter 7, Section 7..3., pp ; [] Chapter 0, Section , pp ; [3] Chapter 4, Section 4.., pp Algorithm 4: The Successive Over-Relaxation (SOR) method: [] Chapter 7, Section 7..4., pp ; [] Chapter 0, Section , pp ; [3] Chapter 4, Section 4.., pp Algorithm 5: The Steepest Descent (SD) method: [] Chapter 0, Section 0..-., pp. 50 5; [3] Chapter 5, Section 5.3., pp. 3 33, Algorithm 5.. Algorithm 6: The Conjugate Gradient (CG) method: [] Chapter 0, Section 0.., pp , Algorithm 0...; [3] Chapter 6, Section 6.7, pp. 74 8, Algorithm 6.7.; [4] Chapter 5, pp. 0 33, Algorithm 5.. Algorithm 7: The Preconditioned Conjugate Gradient (PCG) method: [] Chapter 0, Section 0.3., pp , Algorithm 0.3..; [3] Chapter 9, Section 9., pp , Algorithm 9..; [4] Chapter 5, pp. 0 33, Algorithm 5.3. Algorithm 8: The CGLS method: [] Chapter 7, Section 7.4., pp , Algorithm Algorithm 9: The PCCGLS method: [] Chapter 7, Section 7.4., pp , Algorithm Algorithm 0: The LSQR method: [] Chapter 7, Section 7.6.3, pp , Algorithm Project co-financed by European Union within European Social Fund 5

6 Algorithm : The CGR method: [] Chapter 0, Section 0.4., pp , Algorithm 0.4..; [3] Chapter 8, Section 8.3, pp , Algorithm 8.4. Algorithm : The CGE method: [] Chapter 0, Section 0.4., pp , Algorithm 0.4..; [3] Chapter 8, Section 8.3, pp , Algorithm 8.5. Algorithm 3: The BiCG method: [3] Chapter 7, Section 7.3, pp. 09, Algorithm 7.3. Algorithm 4: The BiCGSTAB method: [3] Chapter 7, Section 7.4, pp. 6 9, Algorithm 7.6. Algorithm 5: The CGS method: [3] Chapter 7, Section 7.4, pp. 3 6, Algorithm 7.8. Algorithm 6: The GMRES method: [] Chapter 0, Section 0.4.4, pp , Algorithm 0.4.4; [3] Chapter 6, Section 6.5, pp. 57 7, Algorithm 6.9 (standard version), Algorithm 6.0 (Householder version), Algorithm 6. (restarted version). Algorithm 7: The QMR method: [] Chapter 0, Section 0.4.7, p ; [3] Chapter 7, Section 7.3, pp. 3, Algorithm 7.4. Algorithm 8: The Kaczmarz method: [5] Chapter 5, Section 5.3.9, pp Algorithm 9: The Oblique Projection method: [5] Chapter 5, Section 5.3.0, pp Algorithm 0: The ewton method: [6], Section 8.4, pp Content of report The report should contain: introductory page, detailed mathematical description of the analyzed problems, a basic description of the coded algorithms, the Matlab code (together with the detailed end-line comments) of the analyzed algorithm, discussion on convergence properties of the analyzed methods, the results obtained with the coded algorithms (the residual error versus iterations: ( k ) * b-ax ; approximation error (if possible): x -x where x is the exact solution; estimation of computational costs), the results obtained with the Matlab in-built functions, conclusions Project co-financed by European Union within European Social Fund 6