Solvers for Linear Systems in Graph Laplacians

Transcription

1 CS7540 Spectral Algorithms, Spring 2017 Lectures #22-#24 Solvers for Linear Systems in Graph Laplacians Presenter: Richard Peng Mar, 2017 These notes are an attempt at summarizing the rather large literature on solving linear systems in graph Laplacians. They re designed to give a high level overview of the various solver algorithms, while giving pointers to papers for the key proofs. Richard s mile-away view of solvers is that there are really two types: Solver Flow/Cuts Based Multiscale / ILU Based Base Case Trees Expanders Intermediate Step Remove off-tree edges Matrix operations Intermediate Errors Large Small Recursive Call Structure W-cycle V-cycle Solution transfer Tree data structures Operator composition Figure 1: Comparison of High-Level Approaches to Solvers Each of these ideas also have a variety of different instantiations. A quick list of them in roughly chronological order is below: 1. Graph based: (a) Congestion-dilation embeddings: the now (in)famous repeatedly rejected manuscript by Vaidya [Vai91], with the incorporation of low-stretch spanning trees by Boman and Hendrickson [BH01]. (b) Routing based: Steiner tree preconditioners by Gremban [Gre96], oblivious routing schemes by Maggs et al. [MMP + 05], (c) Notion of ultra-sparsifiers and incorporation of Loewner orderings: O(m 1.31 ) time algorithm by Spielman and Teng [ST03], and subsequently the first nearlylinear time solver [ST14]. (d) Completely numerical construction of ultra-sparsifiers by Koutis, Miller and Peng [KMP10, KMP11]. (e) Routines that act on one off-tree edge at a time, with connections to stochastic gradient descent and data structures by Kelner, Orecchia, Sidford, and Zhu [KOSZ13], with extensions to accelerated methods by Lee and Sidford [LS13]. (f) Analysis of expected preconditioners that combine analyses of numerical routines with the randomized construction of sparsifiers by Cohen, Kyng, Miller, Pachocki, Peng, Rao, and Xu [CKM + 14]. 1

2 2. Motivated by multiscale methods and incomplete Cholesky factorization: (a) Repeated squaring by Peng and Spielman [PS14], with extensions to directed Laplacians by Cohen, Kelner, Peebles, Peng, Rao, and Vladu [CKP + 16]. (b) Sparsified block Cholesky factorization by Kyng, Lee, Peng, Sachdeva, and Spielman [KLP + 16]. (c) Incomplete Choleksy factorization with the (randomized) error of every vertex elimination analyzed using matrix martingales by Kyng and Sachdeva [KS16]. We will start with the second set of algorithms as they have strictly fewer pieces, but are a bit harder to motivate. 1 Some Terminology and Tools A key idea that underlie all these algorithms is the view of error as an integral part of computation, and tracking entire algorithms as linear operators that approximate some ideal algorithm. Matrix approximations need to be defined via. the Loewner ordering: A B if B A is positive semi-definite. We will say A κ-approximates B if there exists λ min and λ max such that λ max κλ min and λ min A B λ max A. The key fact from numerical analysis that we ll use repeatedly is: Fact 1.1. If A κ-approximates B, then a linear system in A can be solved by O( κ) iterations of: 1. Matrix-vector multiplication by A, 2. Solving a linear system by B. This gives a very powerful way of removing errors. In particular, it says that all we need to do is to provide access to a linear operator Z that 2-approximates A to solve systems in A to high accuracy. This idea that constant factor errors suffice also motivates us to define small errors. We use A ɛ B to denote that A exp(ɛ)-approximates B. Here we exponentiate ɛ because errors simply add upon compositions now. This notion of approximation is also useful because of sparsification by effective resistances, which can be abstracted as: Fact 1.2. In any graph, sampling edges by probabilities exceeding O(log n) times weights times effective resistances produces an O(1)-approximation 2

3 2 Sparsification Based Solvers Sparsification based solvers use a sequence of reductions to make the matrix well-conditioned, while accumulating small errors at each of these change steps. They can be further divided into: 1. Squaring based, which reduce to a well-conditioned matrix on the same set of original variables. 2. Schur complement based, which reduces the number of variables until none remain. 2.1 Repeated Squaring By adding self loops, and rescaling by degree, we can ensure that the graph Laplacian is represented as: I X. Furthermore, by some perturbation, let s assume that I X is full rank, so we can compute its inverse. This assumption can be removed by carefully juggling pseudoinverses, but that just complicates things. The key observation here is (I X ) 1 = ( I X 2) 1 (I + X ), that is, solving a system in I X reduces to solving I X 2 after one matrix-vector multiplication involving I + X. Furthermore, as X can be interpreted as a random walk here, I X 2 can also be approximated by a sparse graph. However, spectral approximations do not compose under one sided multiplications: there are matrices A, B and C such that A 0.1 B but AC and BC do not approximate each other. In fact, the only thing that we can use is: C T AC 0.1 C T BC, that is, composing symmetrically by a matrix C on both sides. This restriction may sound a bit arbitrary, but ends up being the crux of the problem here. Once we accept it, and manipulate algebra with it as the key criteria, we can produce the following identity instead: (I X ) 1 = 1 2 (I + (I + X ) ( I X 2) 1 (I + X ) ). Once again, this can be derived and verified using scalars, as I and X commute with each other. 3

4 2.2 Vertex Elimination A close cousin of sparsified squaring is block Cholesky elimination based methods. Such methods also have close connections with multigrid and multilevel methods. The main idea is to partition the vertices into two parts, C, the coarse grid that remains, and F, the fine grid that gets removed. We will use [, ] to denote blocks of the Laplacian corresponding to these vertex subsets, specifically a Laplacian gets partitioned into: [ ] L[F,F L = ] L [F,C], L [C,F ] L [C,C] and block-cholesky factorization is more or less doing Gaussian elimination by pretending each of the blocks are scalars. It produces the factorization: [ ] [ ] [ I 0 L[F,F ] 0 I L 1 L = L [C,F ] L 1 [F,F ] I 0 L [C,C] L [C,F ] L 1 [F,F ] L [F,F ] L ] [F,C]. [F,C] 0 I Here the only important thing to notice is that the terms on both sides are symmetric. Furthermore, by inductively removing vertices, we can show that the Schur complement, Sc(L, C) def = L [C,C] L [C,F ] L 1 [F,F ] L [F,C], is another graph Laplacian. This means it can be sparsified, and we can recurse on it. So the only obstacle is to find a subset F such that L [F,F ] is easy to invert. To this end, note that if F is an independent set, then L [F,F ] is just a diagonal matrix containing all the degrees (of vertices to C). Therefore what we re looking for is finding an almost independent set. This we can do via the following Lemma: Lemma 2.1. In any undirected weighted graph there is a set F of at least 0.1n vertices such that each vertex in F has at least 1 of its degree going to C = V \ S. 10 The construction is a simple (but somewhat subtle 1 ) use of randomization. Start by picking half of the vertices at random: each of them has expected weighted degree to C of at least half of their total weighted degree. So with probability at least 1/2 such edges have 1/4 of their weights leaving. Then note that removing vertices in F can only improve the ones that remain, so we get about 1/4n vertices in expectation. Such a set is useful because L [F,F ] can now be written as a Laplacian plus a diagonal, L [F,F ] = X + D A. As X is the excess degree, we have X 0.1D, which coupled with D A 2D gives: 21X L X, or that L [F,F ] is O(1)-approximated by X. This plus iterative methods then means it s easy to solve systems in L [F,F ]. 1 you d be amazed about the messes that Richard arrived at before being shown this construction... 4

5 3 Graph Based Solvers Graph based solvers on the other hand aim to reduce to the easy combinatorial case of a tree. They treat error as more of a computational resource to be traded against combinatorial steps. They rely on sampling by upper bounds of a effective resistanced produced by stretches w.r.t. a tree: Formally, stretch is useful because: str T (e) def = w e 1 w e e P ath T (e) 1. If T is a spanning tree of G, then r e w e str T (e). 2. For any graph G, there exist a spanning tree such that the total stretch of all edges, e str T (e) is at most O(m log n log log n). Algebraic proofs of the effective resistance bounds also show a very interesting connection: [ ] tr L G L T = str T (e). e If we squint and think about the m = n case, this means that on average, the eigenvalues of L G L T are about log n, while L T L G also gives that they re at least 1. So on average, an iterative method should work, except their convergences are governed by the worst approximation. Fast graph based solvers can be viewed as ways of deailng with these large eigenvalues throgh further combinatorial structures. 3.1 Ultra-Sparsifiers Ultra-sparsifiers are more or less tree plus k edges. They are useful because Fact 3.1. A tree plus k edges is more or less a graph on O(k) vertices / edges. This is proven by removing leaves, as well as intermediate vertices of degree 2. In both cases, the number of edges minus vertices remains invariant, and the initial invariant can then be used to upper bound the final size. The goal is then to construct ultra-sparsifiers with k off-tree edges that also O(k log 2 n log log n)- approximate L G. This plus iterative methods leads to a running time recurrence of the form: T (m) = O(k log 2 n log log n) (T (k) + m),. which solves to about O(m log 2 n log log n). variety of ways. This bound can be further improved in a 5

6 It remains to give a construction for an ultra-sparsifier. Directly applying sampling by stretch gives an O(1)-approximation but with O(m log 2 n log log n) off-tree edges. Our goal is to have fewer off-tree edges, but at the cost of a worse approximation. A way to introduce error that also reduces off-tree stretch is to simply scale up the tree by a factor of κ. This reduces the total off-tree stretch to ( ) m log n log log n O, κ which after some short calculations means setting gives the required bounds. 3.2 Single Cycle Toggling κ O ( k log 2 n log log n ) There are also single cycle toggling algorithms that can be viewed as variants of gradient descent. These weren t covered in class due to time constraints, and hopefully at some point Richard will go back to write notes about them. References [BH01] Eric Boman and Bruce Hendrickson. On spanning tree preconditioners. Manuscript, Sandia National Lab., [CKM + 14] Michael B. Cohen, Rasmus Kyng, Gary L. Miller, Jakub W. Pachocki, Richard Peng, Anup Rao, and Shen Chen Xu. Solving SDD linear systems in nearly m log 1/2 n time. In STOC, pages , [CKP + 16] [Gre96] [KLP + 16] Michael B. Cohen, Jonathan A. Kelner, John Peebles, Richard Peng, Anup Rao, Aaron Sidford, and Adrian Vladu. Almost-linear-time algorithms for markov chains and new spectral primitives for directed graphs. Accepted to STOC Preprint available at Keith D. Gremban. Combinatorial Preconditioners for Sparse, Symmetric, Diagonally Dominant Linear Systems. PhD thesis, Carnegie Mellon University, Rasmus Kyng, Yin Tat Lee, Richard Peng, Sushant Sachdeva, and Daniel A Spielman. Sparsified cholesky and multigrid solvers for connection laplacians. In Proceedings of the 48th Annual ACM SIGACT Symposium on Theory of Computing, pages ACM, Available at 6

7 [KMP10] Ioannis Koutis, Gary L. Miller, and Richard Peng. Approaching optimality for solving SDD linear systems. In Proceedings of the 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, FOCS 10, pages , Washington, DC, USA, IEEE Computer Society. Available at [KMP11] [KOSZ13] [KS16] [LS13] Ioannis Koutis, Gary L. Miller, and Richard Peng. A nearly-m log n time solver for SDD linear systems. In Proceedings of the 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science, FOCS 11, pages , Washington, DC, USA, IEEE Computer Society. Available at Jonathan A. Kelner, Lorenzo Orecchia, Aaron Sidford, and Zeyuan Allen Zhu. A simple, combinatorial algorithm for solving SDD systems in nearly-linear time. In Proceedings of the 45th annual ACM symposium on Symposium on theory of computing, STOC 13, pages , New York, NY, USA, ACM. Available at Rasmus Kyng and Sushant Sachdeva. Approximate gaussian elimination for laplacians: Fast, sparse, and simple. CoRR, abs/ , Available at: Yin Tat Lee and Aaron Sidford. Efficient accelerated coordinate descent methods and faster algorithms for solving linear systems. In Proceedings of the 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, FOCS 13, pages , Washington, DC, USA, IEEE Computer Society. [MMP + 05] Bruce M. Maggs, Gary L. Miller, Ojas Parekh, R. Ravi, and Shan Leung Maverick Woo. Finding effective support-tree preconditioners. In Proceedings of the seventeenth annual ACM symposium on Parallelism in algorithms and architectures, SPAA 05, pages , New York, NY, USA, ACM. [PS14] Richard Peng and Daniel A. Spielman. An efficient parallel solver for SDD linear systems. In Proceedings of the 46th Annual ACM Symposium on Theory of Computing, STOC 14, pages , New York, NY, USA, ACM. Available at [ST03] Daniel A. Spielman and Shang-Hua Teng. Solving sparse, symmetric, diagonally-dominant linear systems in time O(m 1.31 ). In Proceedings of the 44th Annual IEEE Symposium on Foundations of Computer Science, FOCS 03, pages 416, Washington, DC, USA, IEEE Computer Society. [ST14] D. Spielman and S. Teng. Nearly linear time algorithms for preconditioning and solving symmetric, diagonally dominant linear systems. SIAM Journal on Matrix Analysis and Applications, 35(3): , Available at 7

8 [Vai91] Pravin M. Vaidya. Solving linear equations with symmetric diagonally dominant matrices by constructing good preconditioners. A talk based on this manuscript was presented at the IMA Workshop on Graph Theory and Sparse Matrix Computation, October