Sparse matrices: Basics

Transcription

1 Outline : Basics Bora Uçar RO:MA, LIP, ENS Lyon, France CR-08: Combinatorial scientific computing, September /28 CR09

2 Outline Outline 1 Course presentation 2 2/28 CR09

3 CR08: What is and what is not? CR08 is about sparse matrices and computations with them. Sparse matrix: It is a matrix with many zeros which are not stored. We will follow: Lecture notes, articles, books. S Presentations (survey/research) A = Sparse matrix computations: Mostly operate on the nonzero elements. are everywhere: Google s PageRank, network analysis (social, biological), simulation science (circuits, numerical,...) CR08 3/28 CR09

4 CR08: What is and what is not? CR08 is about graphs. Nonzeros of a matrix can been seen as the edges of a bipartite graph, a directed graph, an undirected graph, or a hypergraph. Computations on sparse matrices can be modeled using graphs. CR08 is about this correspondence in a broader sense /28 CR09

5 CR08: What is and what is not?...not a scientific computing/linear algebra course....not a full-fledged combinatorial optimization course....not a parallel computing course. We will see some exciting problems having to do with topics from these three domains. 5/28 CR09

6 CR08: A sample problem Let A be an n n symmetric positive definite matrix The LL T factorization ( can ) be described by the equation: d1 v1 T A = H 0 = v 1 H 1 ( ) ( ) ( d1 ) d v T 1 = d1 v d1 1 I n 1 0 H 1 0 I n 1 = L 1 A 1 L T 1, where H 1 = H 1 v 1v T 1 d 1 The basic step is applied on H 1, H 2... to obtain : ( ) A = (L 1 L 2 L n 1 ) I n L T n 1 L T 2 LT 1 = LL T /28 CR09

7 The basic step: H 1 = H 1 v 1v T 1 d 1 What is v 1 v T 1 in terms of structure? vt 1 v 1 v 1 v T 1 : a dense sub-matrix in H 1. v 1 is a column of A, hence the neighbors of the corresponding vertex. v 1 v 1 T The neighbors of the corresponding vertex form a clique. Basic step modeled using a graph: Pick the vertex, form a clique from its neighbors, and remove the vertex. 7/28 CR09

8 CR08: A sample problem G 0 G 1 G H 0 H 1 H G(L + L T ) L + L T G H 3 8/28 CR09 What can we say about the final matrix by just looking at the original graph?

9 CR08: Planning Course presentation Instructor: Lecture notes (references to articles, books); problems. Students: Participation to solving problems; Presentations (can be a survey, of type expository, or about a research problem). We will decide the form according to the number of participants. Classes to catch up: BU is away 10 1 November. We will decide the dates later for the course of that week. Office: Contact: Course web: GN1 Nord 332 (LIP) bora.ucar@ens-lyon.fr Lecture notes will be posted. 9/28 CR09

10 Course presentation When we started putting these models together they became very large as compared to linear programming capability at that time... It struck me that our matrices were mostly full of zeros, and if you have a set of simultaneous equations that are mostly zeros, if you pick your pivots right, you could just solve it by hand. Then I thought, well, maybe we could get the computer to do the same thing. This led to Sparse Matrices. As far as I know, I coined the word Sparse Matrix. Harry Markowitz: I got a Nobel Prize... Work in 1952 got Nobel in I published [in this area, refers to a 1957 paper on selecting pivots for Gaussian elimination] and then forgot about it...[in a] decade, it started to catch on and expanded very fast. I was completely oblivious to what was going on and to what extent. 10/28 CR09

11 : Coordinate format There are many ways to store a sparse matrix. We will look at three standard representations which store only the nonzero entries. We use τ to denote the number of nnzs Coordinate (Triplet) format Two integer arrays (irn, jcn) and a double array A, each of size τ: irn = [ ] jcn = [ ] A = [ ] The kth entry a ij is stored as irn[k] = i, jcn[k] = j, a[k] = a ij. The storage is 2τ integers and τ doubles (or single or complex). In general, τ = O(m + n). 11/28 CR09

12 : Coordinate format Question: Let A be an m n matrix. R and C be two diagonal matrices of size m m and n n. A can be scaled as  = RAC. Write a function/routine that does this operation for matrices stored in the coordinate format. 12/28 CR09

13 : Compressed row storage There are many ways to store a sparse matrix. We will look at three standard representations which store only the nonzero entries. Compressed row storage Two integer arrays (ia, jcn) and a double array A: The nonzeros of the ith row are stored at the ia[i] ia[i+1]-1 positions of jcn and A. ia = [ ] jcn = [ ] A = [ ] For example the 3rd row: starts at ia[3] = and finishes at ia[3+1]-1 = 5. The column indices are therefore jcn[,5] = 1 3 and values are A[,5] = /28 CR09

14 : Compressed row storage There are many ways to store a sparse matrix. We will look at three standard representations which store only the nonzero entries Compressed row format Two integer arrays (ia, jcn) and a double array A: ia = [ ] jcn = [ ] A = [ ] The nonzeros of the ith row are stored at the ia[i] ia[i+1]-1 positions of jcn and A. Let matrix be of size m n, and τ be the number of nonzeros, then the storage is m τ integer and τ double (or single or complex). 1/28 CR09

15 : Compressed row storage Question: Let A be an m n matrix stored in the compressed row storage format. Where is nnz(a)? Question: Let A be an m n matrix stored in the compressed row storage format. Write a function/routine that displays A. Question: Let A be an m n matrix. R and C be two diagonal matrices of size m m and n n. A can be scaled as  = RAC. Write a function/routine that does this operation for matrices stored in the compressed row storage format. 15/28 CR09

16 : Compressed column storage There are many ways to store a sparse matrix. We will look at three standard representations which store only the nonzero entries. Compressed column storage Two integer arrays (irn, ja) and a double array A: ja = [ ] irn = [ ] A = [ ] The nonzeros of the jth column are stored at the ja[j] ja[j+1]-1 positions of irn and A. For example the 2nd col: starts at ja[2] = 3 and finishes at ja[2+1]-1 =. The row indices are therefore irn[3,] = 2 5 and values are A[3,] = /28 CR09

17 : Compressed column storage There are many ways to store a sparse matrix. We will look at three standard representations which store only the nonzero entries Compressed column format The nonzeros of the jth column are stored at the ja[j] ja[j+1]-1 positions of irn and A. Two integer arrays (irn, ja) and a double array A: ja = [ ] irn = [ ] A = [ ] Let matrix be of size m n, and τ be the number of nonzeros, then the storage is n τ integer and τ double (or single or complex). 17/28 CR09

18 : Compressed column storage Question: Let A be an m n matrix stored in the compressed column storage format. Where is nnz(a)? Question: Let A be an m n matrix stored in the coordinate format. Write a function/routine that creates A in csc format. 18/28 CR09

19 Reminder: Dense matrix vector multiplication Need to compute y Ax for an m n dense matrix A and suitable dense vectors y and x. Row-major order for i = 1 to m do y[i] 0.0 for j = 1 to n do y[i] y[i] + A[i, j] x[j] Column-major order for i = 1 to m do y[i] 0.0 for j = 1 to n do for i = 1 to m do y[i] y[i] + A[i, j] x[j] 19/28 CR09

20 : Sparse matrix vector multiplies Need to compute y Ax for an m n sparse matrix A and suitable dense vectors y and x. Algorithm when A is stored in the coordinate format. 20/28 CR09

21 : Sparse matrix vector multiplies Need to compute y Ax for an m n sparse matrix A and suitable dense vectors y and x. A is stored in the coordinate format. Coordinate format with τ nonzeros (irn, jcn, A) for i = 1 to m do y[i] 0.0 for k = 1 to τ do y[irn[k]] y[irn[k]] + A[k] x[jcn[k]] 21/28 CR09

22 : Sparse matrix vector multiplies Need to compute y Ax for an m n sparse matrix A and suitable dense vectors y and x. Algorithm when A is stored in the CSR format. 22/28 CR09

23 : Sparse matrix vector multiplies Need to compute y Ax for an m n sparse matrix A and suitable dense vectors y and x. Algorithm when A is stored in the CSR format. Compressed row storage (ia, jcn, A) for i = 1 to m do val 0.0 for k = ia[i] to ia[i + 1] 1 do val val + A[k] x[jcn[k]] y[i] val 23/28 CR09

24 : Sparse matrix vector multiplies Need to compute y Ax for an m n sparse matrix A and suitable dense vectors y and x. Algorithm when A is stored in the CSC format. 2/28 CR09

25 : Sparse matrix vector multiplies Need to compute y Ax for an m n sparse matrix A and suitable dense vectors y and x. Algorithm when A is stored in the CSC format. Compressed column storage (ja, irn, A) for i = 1 to m do y[i] 0.0 for j = 1 to n do xval x[j] for k = ja[j] to ja[j + 1] 1 do y[irn[k]] y[irn[k]] + A[k] xval 25/28 CR09

26 : Sparse matrix vector multiplies Characterizes a wide range of applications with irregular computational dependency. reduction operation from inputs (here entries of x) to outputs (here entries of y) A fine grain computation: each nnz is read/operated on once. Guaranteeing efficiency will guarantee efficiency in applications with a coarser grain computation. 2/28 CR09

27 : Sparse matrix vector multiplies SpMxV s of the form y Ax are the computational kernel of many scientific computations Solvers for linear systems, linear programs, eigensystems, least squares problems, Repeated SpMxV with the same large sparse matrix A, The matrix A can be symmetric, unsymmetric, rectangular, Sometimes multiplies are of the form y ADA T z with a diagonal D (in interior point methods for linear programs). computation proceeds (why?) as w A T z, then x Dw, then y Ax Sometimes we have multiplies with A and A T independent; y Ax and w A T z (QMR, CGNE, and CGNR methods with square unsymmetric A; rectangular A in Lanczos method). 27/28 CR09

28 Further questions Given a real matrix A of size m n, with τ nonzeros. Question: Write a function to create its transpose (three times: stored in the COO, CSR, and CSC formats). Question: Write a function to determine if A is symmetric. Assume A is stored in the CSC format. 28/28 CR09