Parallel FEM Computation and Multilevel Graph Partitioning Xing Cai

Transcription

1 Parallel FEM Computation and Multilevel Graph Partitioning Xing Cai Simula Research Laboratory

2 Overview Parallel FEM computation how? Graph partitioning why? The multilevel approach to GP A numerical example

3 Parallel FEM computation Main computational tasks: Discretization assembly of element matrices and vectors Solution of Divide & conquer: Global domain is divided into subdomains Discretization: on each subdomain The global linear system is represented by the set of local linear systems Parallelization of iterative linear system solvers: subdomain linear algebra operations + communication

4 Parallel efficiency Balanced partitioning the subdomain grids have approximately the same number of elements and grid points Low communication overhead very few neighbors for each subdomain and small size for send/receive messages Partitioning an unstructured FE grid into subdomain grids nontrivial

5 FE grid & graph FE grid partitioning reformulation as a graph partitioning problem For a given FE grid, create a corresponding graph, where one vertex corresponds to one grid element, and a weighted edge between vertices represents the number of shared grid points between neighboring grid elements.

6 An example graph

7 Graph partitioning When vertices is ready, we can partition the set of into equal-sized subsets achieve good load-balance), and that (to edge-cut is minimized (to keep down the inter-processor communication volume). The graph partitioning problem is NP-complete. But there are several algorithms that are able to find reasonably good partitions

8 Multilevel graph partitioning Efficient and flexible, consisting of three phases: Coarsening phase: a recursive process that generates a sequence of subsequently coarser graphs. Initial partitioning phase: partitioning the coarsest graph into subsets; Refining phase: a recursive process that projects the partition of backward to at the same time refining the partition.,

9 Coarsening a graph Go through all the vertices in a random order. For each vertex, we match it with one of its so far unvisited neighboring vertices. The two vertices then form a new super-vertex, and the edge connecting them is collapsed. Other edges that coming out from them are merged if necessary. A new graph that has fewer vertices and edges thus arises from the old graph.

10 Three matching heuristics For every, Random matching (RDM) - a random unchosen neighbor is chosen. Heavy edge matching (HEM) - an unchosen neighboring vertex is chosen, if the edge has the heaviest weight among all the unchosen neighbors. Gain vertex matching (GVM) - an unchosen neighboring vertex is chosen such that the added weight of the edges that come out from the new super-vertex is minimized.

11 HEM vs. GVM HEM GVM

12 Initial partitioning The coarsest graph vertices and edges. has a small number of We can therefore afford using a complex partitioning algorithm.

13 Refining the partition The partition of is projected backward to. Every super-vertex in is split back into two vertices in and the collapsed edges are recovered. We also refine the partition of to reduce edgecut, while maintaining the load balance quality. That is, vertices are moved between subsets. There are many different refinement algorithms.

14 A simple refinement algorithm Go through all the vertices randomly. If a vertex is lying on the boundary, we will move the vertex into one of its neighboring subsets if a largest possible decrease of edge-cut is achievable load-imbalance is improved or remains within a prescribed threshold

15 An example of vertex movement before movement after movement

16 Good GP=good FE grid part? Edge-cut gives a good indication of the total communication volume. More important: average number of neighbors, denoted by, where is the number of neighbors for subdomain.

17 A numerical example Graph arising from an unstructured D FE grid Number of vertices: 60,064; Number of edges: 1,564,19. CPU measurements of 500 parallel CG iterations for solving a linear system arising from discretizing the Poisson equation. Measurements are obtained on a cluster of 4 PC nodes each with dual Pentium III 500MHz processors, inter-connected by standard 100 Mbit/s ethernet.

18 Measurements CPU GVM HEM RDM GVM HEM RDM

19 Measurements (contd) CPU GVM HEM RDM GVM HEM RDM

20 Concluding remarks Multilevel graph partitioning algorithms are flexible and efficient There are different algorithm choices for the three phases A good graph partition normally gives a good FE grid partition Minimizing edge-cut is relevant for reducing the communication overhead Minimizing the (average) number of neighbors is at least equally important.