PhD Student. Associate Professor, Co-Director, Center for Computational Earth and Environmental Science. Abdulrahman Manea.

Size: px

Start display at page:

Download "PhD Student. Associate Professor, Co-Director, Center for Computational Earth and Environmental Science. Abdulrahman Manea."

Edwin Miller
5 years ago
Views:

and Environmental Science Energy Resources Engineering

1 Abdulrahman Manea PhD Student Hamdi Tchelepi Associate Professor, Co-Director, Center for Computational Earth and Environmental Science Energy Resources Engineering Department School of Earth Sciences Stanford University 1

2 Introduction Background 2D Black Box Geometric MG (GMG) 3D Semicoarsening Multigrid Future Work 2

Reservoir Simulation (Black Oil): Mass Conservation of Component α: Incompressible: Total Balance: Incompressible Pressure Equation: Solver is the most computationally expensive component Unknowns

3 Reservoir Simulation (Black Oil): Mass Conservation of Component α: Incompressible: Total Balance: Incompressible Pressure Equation: Solver is the most computationally expensive component Unknowns have varying nature Pressure (elliptic) vs. Saturation (Hyperbolic) Multistage preconditioning scheme Constraints Pressure Residual (CPR)* CPR with Multigrid as the first stage: very robust and widely used scheme Aramco s GigaPOWERS * Wallis, J.R., et al. SPE (1985) 3

4 Objective Design and Implement a massively Parallel Reservoir Simulation Multigrid on GPU Architectures Plan 1. Implement an optimized serial version of Multigrid to have a reasonable serial performance baseline 2. Design and implement a parallel version of Multigrid that harnesses the power the massively parallel GPU architectures 4

5 Introduction Background 2D Black Box Geometric MG (GMG) 3D Semicoarsening Multigrid Future Work 5

Descretized equation is A f x f = b f Basic

The Pre-smoothing Step x f smooth A f, b f, x 0,

The Coarse-Grid Correction Step r f = b f A f x f

6 Descretized equation is A f x f = b f Basic 2-Level Multigrid Algorithm (3 steps) 1. The Pre-smoothing Step x f smooth A f, b f, x 0, υ 1 presmoothing postsmoothing 2. The Coarse-Grid Correction Step r f = b f A f x f r c = I c f r f e c = A c 1 r c e f = I f c e c x f = x f + e f 3. The Post-smoothing Step: x f smooth(a f, b f, x f, υ 2 ) Solve the Problem on the Coarse Grid * Brandt, A. (1977) 6

7 The prolongation and restriction operators weights depends on the PDE discontinuous coefficients λ p = q i-1,j+1 i,j+1 i+1,j+1 nw T i,j n ne T i,j T i,j I-1,J+1 I,J+1 I+1,J+1 i-1,j+1 i,j+1 i+1,j+1 i+1,j T i,j w i,j e T i,j i+1,j I-1,J i-1,j I,J i+1,j I+1,J sw T i,j s T i,j se T i,j i-1,j-1 i,j-1 i+1,j-1 i-1,j-1 i,j-1 i+1,j-1 I-1,J-1 I,J-1 I+1,J-1 * Alcouffe, R.E., et al. (1981) 7

8 Coarse grid operator: Manual Explicit handling of PDE on each coarser level Automatic (Black Box Multigrid) Using grid transfer operators: A c = I c f A f I f c = (I f c ) T A f f I c No info. about coarser grid is needed Used in Algebraic multigrid Preserve operator symmetry In Black Box Multigrid, two stages: Setup Stage: The interpolation, restriction and coarse grid operators are calculated. Solution Stage: Carrying out the cycling process Anisotropic PDE Coefficients Line Relaxation (2D), plane relaxation (3D) Semicoarsening *Dendy, J.E, (1982), (1986), Schaffer, S., (1998) 8

9 To handle anisotropies in all three dimensions (x,y,z): Alternating plane relaxation (too expensive) Semicoarsening with plane relaxation (cheaper) One plane-solve, and semicoarsening in the dimension orthogonal to that plane. When semicoarsening approach is used, with exact grid transfer operators, MG becomes a direct solver (i.e. a Schur Complement). However, grid transfer operators are not sparse, where A more efficient way is to approximate the exact grid transfer operators using a sparse (block diagonal) operator. 2D MG is used to define the components of the operator between every two planes (details can be found in *) *Schaffer, S., (1998) 9

10 Introduction Background 2D Black Box Geometric MG (GMG) 3D Semicoarsening Multigrid Future Work 10

Need a Multigrid solver capable of handling highly heterogeneous and anisotropic structured 2D reservoirs, thus: 2D Black Box Multigrid, with Alternating line-relaxation Testing Solver s convergence

11 Need a Multigrid solver capable of handling highly heterogeneous and anisotropic structured 2D reservoirs, thus: 2D Black Box Multigrid, with Alternating line-relaxation Testing Solver s convergence behavior: Test the convergence ratio for the same problem with varying sizes (using grid refinement) Compare the performance with well-established and widely-used Multigrid solvers, e.g. SAMG: Algebraic Multigrid Solver form Fraunhofer Institute for Algorithms and Scientific Computing (SCAI) MGD9V, etc Test Models Geostatically Generated using the Stanford Geostatistical Modeling Software (SGeMS) Derived from SPE10 Comparative Solution Project Model. large permeability variations of 8-12 orders of magnitude 11

12 SPE10, Layer 70 Residual Reduction Factor = r k+1 2 r k 2 12

13 ¼ Million Cell 1 Million Cell Computational Time Comparison (SPE10 Layer 85 Refined to 1 Million Cells): GMG: ~ 4.5 sec SAMG: ~ 7.0 sec 13

14 Parallelization of every component of the algorithm Both setup stage and solution stage Does not sacrifice algorithmic scalability (convergence rate) Smoother Alternating zebra-line relaxation Effectively handles anisotropies Coarsest Solve 4-color GS relaxation (to handle 9-point stencils) 14

15 Shared-Memory Parallelization OpenMP Coarse threads Hence coarse-scale parallelization Multiple cells (multiple lines) per thread Sparse Matrix Format CSR for cache coherence Tridiagonal Solver: Thomas Algorithm Serial within each line (i.e. thread) but several lines are handled in parallel (zebra-coloring) Architecture: 12 Intel Xeon X GHz cores with 48 GB Memory 15

16 Fine Threads Fine-scale parallelization Single cell per thread Sparse Matrix Format Diagonal with column major ordering Ideal for structured problems Coalesces memory accesses Minimizing storage requirements Exploits the banded matrix structure for efficient data access Minimize expensive communication with host Fit the whole problem on the GPU (up to 16M double precision) 16

17 non-coalesced Tridiagonal Solver Parallel cyclic reduction (PCR) in Batch* to exploit: fine scale parallelism within the line coarse scale parallelism exposed by the zebra ordering of lines Threads operates in two stages: Preparation Stage Solution Stage For coalescing memory accesses during the Preparation Stage (NOTE: grid points are numbered along x-direction): In X-line Relaxation: Each x-line is assigned to a block of threads In Y-line Relaxation: Points with the same x-coordinate are assigned to a block of threads y x coalesced *Using NVIDIA CUSPARSE Library: ( 17

18 Criteria Multicore GPU Architecture Specs 12 Intel Xeon X GHz cores with 48 GB Memory Nvidia Fermi-Based C2070 with 448 CUDA Cores and 6 GB Memory Matrix Structure CSR Format for cache coherence Diagonal Format with column major format (for coalescing memory accesses) Parallelization API Parallelization Granularity Tridiagonal Solver Algorithm (for line relaxation) OpenMP Multiple cells per thread (coarse) Thomas Algorithm (serial within each line, but multiple lines are handled in parallel by zebra coloring) CUDA One cell per thread (fine) Parallel Cyclic Reduction in Batch (Parallel within each line and multiple lines are handled in parallel as well) 18

19 Homogeneous Permeability Case: Solved with just one V(0,1) cycle Residual reduction by 10 9 Focuses on the scalability of the setup stage Heterogeneous Permeability Case: Derived from SPE10 85 th Layer by grid refinement Solved with six V(0,1) cycles Residual reduction by 10 9 Focuses on the scalability of the solution stage Problem Sizes: 1 Million, 4 Million and 16 Million cells 19

20 20

21 21

22 22

23 23

24 Introduction Background 2D Black Box Geometric MG (GMG) 3D Semicoarsening Multigrid Future Work 24

25 In reservoir simulation, z-direction Huge variations due to natural deposition Severe anisotropy compared to x/y directions An effect of discretization (pancake models). Semicoarsening in z-direction, and plane relaxation in the x-y plane We can use 2D MG for both: Setup Stage: construction of grid transfer operators Solution Stage: x-y plane relaxation 25

26 Parallelize plane solve kernel in both: Setup Stage: construction of grid transfer operators Five V(0,1) cycles/plane for approximating an exact solve Solution Stage: red/black plane relaxation One V(0,1) cycle/plane for doing plan-relaxation z 2D MG for Plane Solve Note that those 2D V(0,1) cycles are already parallelized (using the 2D GMG algorithm explained earlier) Other kernels are amenable to parallelization on the GPU, but are not tackled yet (under progress). 26

27 Implementation: CPU: Use OpenMP threads to distribute the plane solves across multiple cores GPU: Use CUDA with OpenMP to distribute the plane solves to multiple GPU s Platform: CPU: 24 Intel(R) Xeon(R) CPU 2.80GHz with HT and 180 GB Memory GPU: 6 Nvidia Fermi-Based M2090 s Test cases: homogeneous and heterogeneous (SPE 10) with various sizes Results: average time for the plane solves for both setup and solution stages 27

Speed Up 22 20 18 16 14 12 10 8 16K x 129 ~ 2M cells 66K x 33 ~ 2M cells 1M x 17 ~ 18M

28 Speed Up K x 129 ~ 2M cells 66K x 33 ~ 2M cells 1M x 17 ~ 18M cells 4M x 17 ~ 71M cells cores 1 GPU 2 GPU's 3 GPU's 4 GPU's 5 GPU's 6 GPU's 28

Speed Up 20 18 16 14 12 10 8 16K x 129 ~ 2M cells 66K x 33 ~ 2M cells 1M x 17 ~ 18M

29 Speed Up K x 129 ~ 2M cells 66K x 33 ~ 2M cells 1M x 17 ~ 18M cells 4M x 17 ~ 71M cells cores 1 GPU 2 GPU's 3 GPU's 4 GPU's 5 GPU's 6 GPU's 29

30 Planes need to be sufficiently large ( > 1M cells) for a noticeable advantage This is good for reservoir simulation, as grid refinement studies are usually made by refining the horizontal planes. Beyond 2-3 GPU s, no performance is gained Could be due to number of planes, or plane size.. Needs more investigation and profiling 30

31 Accelerate other kernels of 3D Semicoarsening Multigrid using GPU s (such as coarse operator construction, etc) Algebraic Multiscale Solver on GPU s is Next! 31

32 Thank you for your listening Questions 32

Matrix-free multi-gpu Implementation of Elliptic Solvers for strongly anisotropic PDEs

Iterative Solvers Numerical Results Conclusion and outlook 1/18 Matrix-free multi-gpu Implementation of Elliptic Solvers for strongly anisotropic PDEs Eike Hermann Müller, Robert Scheichl, Eero Vainikko