Chapter 14: Matrix Iterative Methods

Transcription

1 Chapter 14: Matrix Iterative Methods 14.1INTRODUCTION AND OBJECTIVES This chapter discusses how to solve linear systems of equations using iterative methods and it may be skipped on a first reading of this book without loss in continuity. We have included this chapter because iterative methods are an alternative to the direct methods that we introduced in Chapter 8. InChapter 8 we showed how LU decomposition is used to solve matrix systems. We restricted our attention to tridiagonal and block tridiagonal matrices. We employed direct methods to find a solution to the system of linear equations in a finite number of steps. In this chapter we take a completely different approach. Here we start with some initial approximation to the solution of the matrix system and we then construct a sequence of estimates to the solution where each estimate is closer to the exact solution than the previous one. We are thus in the realm of iterative methods for solving systems of linear equations. The main methods of interest are: The Jacobi method The Gauss Seidel method Successive Overrelaxation (SOR) The Conjugate Gradient method Projected SOR method (for matrix inequalities). These methods are particularly suitable for sparse matrices. Furthermore, we generalise these methods to solve Linear Complementarity Problems (LCP) of the form (14.1) HereA is a square positive-definite matrix, b and c are given vectors and we seek a solution x that satisfies the conditions in (14.1). Here we speak of vector inequality; by definition, a vector v 1 is greater than a vector v 2 if each element in v 1 is greater than the corresponding element in v 2. Please recall that C++ functions that test inequality relationships between vectors are discussed in Chapter 7 and you should use the ArrayMechanisms package if you plan to write your own LCP programs. It saves you having to reinvent the wheel. We propose an algorithm for solving (14.1). The method was invented by C.W. Cryer (1979) and has gained wide acceptance in the financial engineering world (see Wilmott et al., 1993). This algorithm is called the Projected SOR method, which we shall discuss in section We now give a general introduction to solving linear systems of equations using iterative methods. < Day Day Up >

2 14.2ITERATIVE METHODS In general we wish to find the solution of the linear system written in matrix form: (14.2) We give some motivation on finding iterative methods to solve (14.2) and show how they work. For a definitive and very clear discussion, see Varga (1962). Let us rewrite matrix A in the following equivalent form (14.3) whered is the diagonal matrix with value zero everywhere except on the main diagonal where the values are equal to the diagonal elements of A. The matrix I is the identity matrix, U is upper triangular andl is lower triangular. We can then rewrite the matrix equation Ax = b in the equivalent form (14.4) whereb = (L + U) and c = D 1 b. This equation contains the unknown x on both left- and right-hand sides. Now is the crux: we define a one-step sequence of vectors as follows: (14.5) Starting with some initial approximation to the solution we hope that the sequence will converge to the exact solution x as k increases. As mathematicians we must prove that the sequence does converge; for more information, we refer again to Varga (1962). There are a number of ways to choose the iteration in (14.4), and we shall discuss each one in turn. < Day Day Up >

3 14.3THE JACOBI METHOD This is the simplest iterative method. The terms Lx and Ux in (14.4) are both evaluated at level k and the Jacobi scheme in matrix form is given by: (14.6) or in component form: (14.7) We usually take the initial approximation to be that vector all of whose values are zero. With this method we do not use the improved values until after a complete iteration. < Day Day Up >

4 14.4GAUSS SEIDEL METHOD This method is similar to the Jacobi method except that the term Lx is evaluated at the level k + 1: (14.8) In component form, Gauss Seidel is: (14.9) We can rewrite (14.9) to produce a more algorithmic depiction that is suitable for C++ development: (14.10) Notice that, in contrast to the Jacobi method, the Gauss Seidel method uses the improved values as soon as they are computed. This is reflected in equation (14.10). < Day Day Up >

5 14.5SUCCESSIVE OVERRELAXATION (SOR) By a simple modification of the Gauss Seidel method it is often possible to make a substantial improvement in the rate of convergence, by which we mean the speed with which the sequence of approximations converges to the exact solution x of (14.2). To this end, we modify (14.10) by introducing a so-called relaxation parameter as a coefficient of the residual term: (14.11) For = 1 we get the Gauss Seidel method as a special case. A major result is Furthermore, it has also been shown by experiment that for a suitably chosen the number of approximations needing to be computed may be reduced by a factor of 100 in some cases. Indeed for certain classes of matrices this optimal value is known. See Dahlquist (1974) or Varga (1962) for more information. < Day Day Up >

6 14.6OTHER METHODS We discuss two methods that are related to the current discussion The conjugate gradient method This is a direct method that is useful in practice. We assume that A is positive definite having n rows andn columns. We start with an initial vector U 0. Then for j = 1, 2, 3,..., n compute (14.12a) Forj = 1,...,n compute: (14.12b) ThenU n will be the solution of the linear system AU = F if rounding errors are neglected Block SOR We can generalise the SOR method to the case where the matrix A is partitioned into sub-matrices (14.13) For example, A could be a block tridiagonal matrix (recall that we discussed this problem in Chapter 8). We thus propose the following block SOR method (see Cryer, 1979): (14.14) A special case is when A is a block tridiagonal matrix. We can then combine LU decomposition with the block SOR because the intermediate vector in (14.14) is a candidate for LU decomposition Solving sparse systems of equations When approximating multidimensional partial differential equations by finite difference methods the resulting matrices are often sparse. An example is when we discretise the Black Scholes equation for an option based on the maximum of two assets. When the matrix is sparse we can resort to sparse matrix solvers. There is a vast literature on this subject and a full treatment is outside the scope of this book. For applications to financial engineering, see Tavella and Randall (2000). It is possible to use both direct methods and iterative methods to solve such systems although iterative methods are possibly more popular. For an introduction to direct methods for sparse systems, see Duff et al. (1990). In general, if you discretise a multidimensional problem directly in all directions, an iterative solver is more flexible; on the other hand, if you use Alternating Direction Implicit (ADI) or some other splitting method, a direct method is more flexible because we solve the problem as a sequence of tridiagonal systems for which we use LU decomposition.

7

8 14.7THE LINEAR COMPLEMENTARITY PROBLEM We now give an introduction to solving problems as shown in (14.1). These are the so-called Linear Complementarity Problem (LCP) methods and they arise in financial engineering applications when we discretise Black Scholes equations with an early exercise option. For the moment, we present the Projected SOR algorithm (PSOR) that solves (14.1) (see Wilmott et al., 1993): 1. Choose: Check: The interface for the class that implements the PSOR method is: template <class V, class I> class ProjectedSOR // The Projected SOR method private: // Ingredient of problem, this is // // Ax >= b, x >= c // (x - c).(ax - b) == 0 (inner product) NumericMatrix<V, I>* A; Vector<V, I>* b; Vector<V, I>* c; // The matrix // The right-hand side vector // The lower-bound on the solution // Temporary work space Vector<V, I> OldVec; // The solution at level k Vector<V,I> NewVec; // The solution at level k+1 Vector<V,I> InterVec; // The intermediate vector V tol; // Determines how many iterations public: ; // For you my friend We leave it as an exercise to write the code for this class. It is a simple extension of the code for the Gauss Seidel method.

9

10 14.8IMPLEMENTATION The header file for the iterative scheme is: enum MatrixIterativeType Jacobi, GaussSeidel; template <class V, class I> class MatrixIterativeSolver private: // Input to class NumericMatrix<V, I>* A; Vector<V, I>* b; V tol; // The matrix to be inverted // The right-hand side vector // Tolerance for convergence MatrixIterativeType itype; MatrixIterativeSolver(); MatrixIterativeSolver (const MatrixIterativeSolver<V,I>& s2); MatrixIterativeSolver<V,I>& operator = (const MatrixIterativeSolver<V,I>& i2); // Temporary work space Vector<V, I> OldVec; // The solution at level k Vector<V,I> NewVec; // The solution at level k+1 // Nitty-gritty functions void calcjacobi(); void calcgaussseidel(); public: // Constructors and destructor MatrixIterativeSolver(NumericMatrix<V,I>& MyA, Vector<V,I>& myrhs); virtual ~MatrixIterativeSolver(); void settolerance(const V& tolerance); void setiterationtype(matrixiterativetype type); // Result; note that this vector INCLUDES BOTH end conditions Vector<V,I> solve();

11 ; The essential code for Jacobi and Gauss Seidel is: template <class V, class I> void MatrixIterativeSolver<V,I>::calcJacobi() V tmp; for (I j = (*A).MinRowIndex(); j <= (*A).MaxRowIndex(); j++) tmp = V(0.0); for (I i = (*A).MinColumnIndex(); i <= (*A).MaxColumnIndex(); i++) if (i!= j) tmp += (*A)(j,i) * OldVec[i]; NewVec[j] = (-tmp + (*b)[j]) / (*A)(j, j); template <class V, class I> void MatrixIterativeSolver<V,I>::calcGaussSeidel() V tmp1, tmp2; for (I j = (*A).MinRowIndex(); j <= (*A).MaxRowIndex(); j++) tmp1 = tmp2 = V(0.0); for (I i = (*A).MinColumnIndex(); i <= j-1; i++) tmp1 += (*A)(j,i) * NewVec[i]; for (i = j+1; i <= (*A).MaxColumnIndex(); i++) tmp2 += (*A)(j,i) * OldVec[i];

12 NewVec[j] = (-tmp1 -tmp2 + (*b)[j]) / (*A)(j, j); The reader can check the above code with the algorithms in (14.7) and (14.9). We have done our best to make the relationship as clear as possible. Sample code is to be found on the CD. Here is a snippet: // A matrix corresponding to boundary value problem u" + u = 0, u(0) = 0, u(1) = 1 J = 10; double h = 1.0 / double(j); Vector<double, int> A(J-1, 2, 1.0); // J-1 els, start == 2 Vector<double, int> B(J, 1, (h*h)); Vector<double, int> C(J-1, 1, 1.0); Vector<double, int> R(J, 1, 0.0); // Right-hand side R[R.MaxIndex()] = -1.0; NumericMatrix<double, int> A3 = creatematrix(a,b,c); print(a3); MatrixIterativeSolver<double, int> secondsolver(a3, R); secondsolver.settolerance(0.0001); secondsolver.setiterationtype(jacobi); Vector<double, int> Result4 = secondsolver.solve(); cout << "\nsolution" << endl; print(result4); Vector<double, int> exact(result4); double d = ::sin(1.0); double x = h; for (int i = exact.minindex(); i <= exact.maxindex(); i++) exact[i] = ::sin(x) / d; x += h; print(exact); The output from this program is:

13 MAT:[ Row 1 (-1.99, 1, 0, 0, 0, 0, 0, 0, 0, 0, ) Row 2 (1, -1.99, 1, 0, 0, 0, 0, 0, 0, 0, ) Row 3 (0, 1, -1.99, 1, 0, 0, 0, 0, 0, 0, ) Row 4 (0, 0, 1, -1.99, 1, 0, 0, 0, 0, 0, ) Row 5 (0, 0, 0, 1, -1.99, 1, 0, 0, 0, 0, ) Row 6 (0, 0, 0, 0, 1, -1.99, 1, 0, 0, 0, ) Row 7 (0, 0, 0, 0, 0, 1, -1.99, 1, 0, 0, ) Row 8 (0, 0, 0, 0, 0, 0, 1, -1.99, 1, 0, ) Row 9 (0, 0, 0, 0, 0, 0, 0, 1, -1.99, 1, ) Row 10 (0, 0, 0, 0, 0, 0, 0, 0, 1, -1.99, )] Solution [Vector: size 10, Min/Max indices: 1, , , , , , , ,0.8044, , ,] [Vector: size 10, Min/Max indices: 1, , ] < Day Day Up >

14 14.9CONCLUSIONS AND SUMMARY We have given an introduction to iterative methods for solving linear systems of equations. We looked at the Jacobi, Gauss Seidel and Successive Overrelaxation (SOR) methods. All of these methods start with an initial guess to the solution of the matrix problem and a sequence of improved estimates is calculated that hopefully converge to the exact solution. An important point is the speed or rate of convergence and to this end it is possible to choose an optimal overrelaxation parameter that improves the convergence rate. Having built up the theory of iterative methods we then introduce the Linear Complementarity Problem (LCP). This has to do with matrix inequalities and we discuss the Projected SOR method to solve this problem. LCP solvers are needed when we discretise the Black Scholes equation by finite differences and they are needed primarily because of the early exercise feature of American options. < Day Day Up >