Scalability of Elliptic Solvers in NWP. Weather and Climate- Prediction

Size: px

Start display at page:

Download "Scalability of Elliptic Solvers in NWP. Weather and Climate- Prediction"

Robert Walsh
5 years ago
Views:

1 Background Scaling results Tensor product geometric multigrid Summary and Outlook 1/21 Scalability of Elliptic Solvers in Numerical Weather and Climate- Prediction Eike Hermann Müller, Robert Scheichl University of Bath AMM03 Met Office Satellite Meeting, Oct 24 th 2012

Background Scaling results Tensor product geometric multigrid Summary and Outlook 2/21 NGWCP project Next Generation Weather and Climate Prediction project Selection of numerical algorithms to

2 Background Scaling results Tensor product geometric multigrid Summary and Outlook 2/21 NGWCP project Next Generation Weather and Climate Prediction project Selection of numerical algorithms to simulate the atmosphere in weather and climate prediction which take advantage of massively parallel architectures. Develop new dynamical core for the Met Office Unified Model which scales up to cores Substantial increase in global model resolution 25km few km degrees of freedom per atmospheric variable Model runtime 1hour for 5 day forecast Solve elliptic PDE for pressure correction in 1second

3 Background Scaling results Tensor product geometric multigrid Summary and Outlook 3/21 1 Background Elliptic PDE in implicit time stepping Model equation Multigrid solvers 2 Scaling results Massively parallel scaling on Hector 3 Tensor product geometric multigrid Parallel scaling results Weak scaling Strong scaling Implementation in DUNE-Grid 4 Summary and Outlook

Background Scaling results Tensor product geometric multigrid Summary and Outlook 4/21 Implicit timestepping Large scale atmospheric flow: Navier Stokes equations Du Dt = 2Ω u 1 p + g + Su ρ Dρ = ρ

4 Background Scaling results Tensor product geometric multigrid Summary and Outlook 4/21 Implicit timestepping Large scale atmospheric flow: Navier Stokes equations Du Dt = 2Ω u 1 p + g + Su ρ Dρ = ρ u,... Dt image source: NASA Implicit time stepping Unconditionally stable Larger integration time step t Solve 3d elliptic PDE for pressure correction π at every time step: (α t) 2 c 2 s (a π ) + bπ = RHS Davies et al. Q J Royal Met Soc, 131 (608): , 2005,... Significant proportion of model runtime

5 Background Scaling results Tensor product geometric multigrid Summary and Outlook 5/21 Does the solver scale and perform? Started by testing the following black box solvers: Distributed and Unified Numerics Environment (DUNE) ISTL Bastian et al. 2008, Blatt and Bastian 2007 & 2008 CG preconditioned with aggregation AMG + ILU0 smoother Hypre Developed at LLNL by U. Maier-Yang, R. Falgout and others CG preconditioned with BoomerAMG Matrix ( + AMG) setup costs? Matrix-free geometric multigrid Hand-written Fortran code based on tensor-product multigrid idea Börm, Hiptmair Numerical Algorithms. 26: DUNE-based code with indirect horizontal-, direct vertical-addressing

6 Background Scaling results Tensor product geometric multigrid Summary and Outlook 6/21 Does the solver scale and perform? Comparison of Multigrid solvers for model equation Weak scaling of total time +AMG setup time all times in seconds # proc # dof AMG (DUNE) BoomerAMG geo MG matrix setup time for AMG solvers

Background Scaling results Tensor product geometric multigrid Summary and Outlook 7/21 Model equation Simplified model equation for u π on spherical shell [ ω 2 (2d) + λ 2 1 ( r 2 r 2 )] u + u = RHS

7 Background Scaling results Tensor product geometric multigrid Summary and Outlook 7/21 Model equation Simplified model equation for u π on spherical shell [ ω 2 (2d) + λ 2 1 ( r 2 r 2 )] u + u = RHS r r Dimensional analysis: r [1, 1 + h] with h = H/R earth = 10 2 : ( ) 2 ω 2 cs α t λ 2 R earth (α t) 2 (N 0 ) 2 Acoustic waves: c s 550ms 1 Buoyancy frequency N 0 = 0.018s 1 Off-centering parameter α = 1 2 (fully implicit: α = 1, fully explicit: α = 0)

Background Scaling results Tensor product geometric multigrid Summary and Outlook 8/21 Model equation Properties 7 point FV discretisation Vertical grid graded r k = 1 + h(k/n z ) 2 R earth /H 100 λ

8 Background Scaling results Tensor product geometric multigrid Summary and Outlook 8/21 Model equation Properties 7 point FV discretisation Vertical grid graded r k = 1 + h(k/n z ) 2 R earth /H 100 λ 2 /h 2 1 Anisotropy ( λ/h x z ) 2 = O( ) Horizontal grid e.g. cubed sphere, icosahedral,... no pole singularity as in lat/lon grid Δx Δz h ω2 / x 2 = const. as t to keep Courant number c s t/ x fixed

9 Background Scaling results Tensor product geometric multigrid Summary and Outlook 9/21 Multigrid solvers Multigrid idea: Eliminate error on all scales Hierachy of grids h, 2h, 4h,... Apply simple smoother on all levels, restrict/prolongate between levels Residual equation on coarser grids A (H) e (H) = r (H) Work on coarse grids is cheap! Algorithmically optimal Cost(MG) = O(n) Robust & parallelisable h A u =b (h) (h) (h) A e =r (2h) (2h) (2h) A e =r (4h) (4h) (4h) A e =r (8h) (8h) (8h)

Background Scaling results Tensor product geometric multigrid Summary and Outlook 10/21 Setup Weak scaling 1/6 of cubed sphere grid (have also run on entire sphere) Horizontal partitioning only

10 Background Scaling results Tensor product geometric multigrid Summary and Outlook 10/21 Setup Weak scaling 1/6 of cubed sphere grid (have also run on entire sphere) Horizontal partitioning only (atmos. physics) # processors problem size n x 2n x, n y 2n y, n z = 128, p 4p Keep ν = c g t/ x = 8.44 fixed (i.e. t decreases) ω t x, λ 2 = (α t) 2 (N 0 ) 2 All runs carried out on Hector Cray XE6 supercomputer 2816 nodes of 2 AMD Opteron 16-core Interlagos 2.3GHz = 90,122 cores NB explicit scheme requires ν 1

11 Background Scaling results Tensor product geometric multigrid Summary and Outlook 11/21 Weak Scaling Black box AMG solvers: # iterations & time per iteration all times in seconds AMG (DUNE) BoomerAMG # proc # dof # iter t iter eff. # iter t iter eff [00%] [00%] [98%] [100%] [97%] [97%] [94%] [97%] [95%] [97%] [92%] [84%] [92%] [32%] as preconditioner for CG

12 Background Scaling results Tensor product geometric multigrid Summary and Outlook 12/21 Setup costs + Anisotropy AMG has coarse level & matrix setup costs Rotating anisotropy due to vertical grading top z y x bottom coarse #processors fine Grid-aligned anisotropy Operator well-behaved in horizontal direction Tensor-product matrix-free geometric multigrid Börm, Hiptmair Numerical Algorithms. 26:

13 Background Scaling results Tensor product geometric multigrid Summary and Outlook 13/21 Tensor-product multigrid Tensor product operator A = A (r) M (horiz) h Vertical eigenmodes A (r) e (r) = ω j t M (r) e (r) u(r, x) = j + M (r) A (horiz) h [for operator (α ) ] Börm, Hiptmair Numerical Algorithms. 26: n z j=1 Vertical line relaxation (e.g. RB Gauss-Seidel) Semi-coarsening in horizontal direction only 2d multigrid convergence rate ρ (2d) max j Meteorological application on 3d lat-lon grid: { } ρ (horiz) [e (r) ] j u j (x)e (r) (r) j Buckeridge, Cullen, Scheichl and Wlasak Q J Royal Met Soc 137 (657): R P

14 Background Scaling results Tensor product geometric multigrid Summary and Outlook 14/21 Geometric multigrid Implementation 1/6th of cubed sphere grid RB SOR with vertical line relaxation horizontal semi-coarsening Halo exchange after each smoothing step & prolongation N halo = 1 + 2(n presmooth + n postsmooth ) = 5 Overlap communication & calculation collect/distribute coarse grid data when # procs > # columns collect distribute collect distribute collect distribute

15 Background Scaling results Tensor product geometric multigrid Summary and Outlook 15/21 Geometric multigrid Parallel Multigrid: volume/interface ratio decreases on coarser levels Hülsemann et al., Lect. Notes in Comp. Science and Engineering (2005) BUT Problem well conditioned on coarser levels (see talk by John Thuburn yesterday) Horizontal coupling (vertical coupling irrelevant due to exact vertical solve): ω 2 x 2 l = ω2 x l 2 7 2l Reduce number of levels Coarsen to 1 column (MG) Coarsen to 1 column/processor (7 levels, shallow MG) 4 levels (very shallow MG) 1-level method to check robustness

16 Background Scaling results Tensor product geometric multigrid Summary and Outlook 16/21 Weak scaling results Reduced number of multigrid levels all times in seconds standard MG n lev = 7 n lev = 4 # proc # dof # t iter # t iter # t iter [00%] [00%] [00%] [99%] [99%] [99%] [98%] [98%] [99%] [97%] [97%] [98%] [96%] [98%] [97%] [95%] [97%] [97%] [93%] [95%] [97%] as preconditioner for CG

17 Background Scaling results Tensor product geometric multigrid Summary and Outlook 17/21 Strong scaling results Geometric multigrid Problem size: n n 128 parallel efficiency 100% 90% 75% 50% 25% time per iteration [s] # cores efficiency = p 0 T(p 0 ) p T(p) 100% # cores

addressing in horizontal direction by work in vertical direction MacDonald et al.

18 Background Scaling results Tensor product geometric multigrid Summary and Outlook 18/21 Multigrid on arbitrary spherical grids Grid structure Tensor product grid structure 2-sphere } {{ } host grid 1-column } {{ } directly addressed Hide indirect addressing in horizontal direction by work in vertical direction MacDonald et al., Int J of HPC Appl (2011) Naturally maps to DUNE data model: Attach vector of size n z to each cell of the 2d host grid Multigrid hierarchy only on host grid Size of vertical column O(100)

19 Background Scaling results Tensor product geometric multigrid Summary and Outlook 19/21 Comparison to DUNE geometric MG code Time per iteration [Intel(R) Core(TM)2 Duo CPU E GHz] time per iteration [s] ALUGrid SPGrid+GeometryGrid YaspGrid+GeometryGrid Fortran t iter = A(grid) + B n z n z Implemented together with Andreas Dedner (Warwick)

Background Scaling results Tensor product geometric multigrid Summary and Outlook 20/21 Spherical grids Parallel convergence history [preliminary] Cubed sphere r / r 0 10 0 10-1

20 Background Scaling results Tensor product geometric multigrid Summary and Outlook 20/21 Spherical grids Parallel convergence history [preliminary] Cubed sphere r / r CubedSphere SOR CubedSphere Jacobi Icosahedral SOR Icosahedral Jacobi cells, 96 cores Icosahedral grid iteration cells, 320 cores

21 Background Scaling results Tensor product geometric multigrid Summary and Outlook 21/21 Summary and outlook Summary Outlook Multigrid solvers for elliptic PDE in NWP implicit time stepping Verified weak & strong scaling to cores (HECToR) Geometric multigrid code avoids AMG- and matrix setup costs Anisotropy: Tensor product multigrid semi-coarsening + vertical line relaxation Problem well-conditioned on coarser grids use small number of multigrid levels Geometric multigrid robust Hybrid MPI+OpenMP parallelisation More realistic problems: non-symmetry, non-smoothness,... GPGPUs memory layout?

22 22/21 Strong scaling Strong scaling AMG (DUNE) Scaled efficiency re. 64 cores (1node = 32 cores on Hector) E(p) = t(64) 64 p t(p) # dof = = # proc data/halo t/iter E(p) data/halo t/iter E / / % 4096/ % / % 1024/ % / % 256/ % / % 64/ %

23 23/21 Anisotropy BoomerAMG scaling, Setup I: on 16 cores ( ) 2 vertical coupling γ horizontal coupling = λ2 z h 2 x all times in seconds # proc γ top γ middle γ bottom time/iteration t setup [00%] 2.6 [00%] [100%] 2.7 [95%] [97%] 2.8 [92%] [97%] 3.2 [81%] [97%] 3.6 [72%] [84%] 5.7 [45%] [32%] 7.1 [37%]

24 24/21 Anisotropy BoomerAMG scaling, Setup II: on 16 cores ( ) 2 vertical coupling γ horizontal coupling = λ2 z h 2 x all times in seconds # proc γ top γ middle γ bottom time/iteration t setup [00%] 0.61 [00%] [97%] 0.66 [93%] [96%] 0.72 [85%] [94%] 0.86 [71%] [93%] 1.2 [52%] [87%] 1.5 [40%] [82%] 3.0 [20%]

25 25/21 Anisotropy BoomerAMG scaling, Setup III: on 16 cores ( ) 2 vertical coupling γ horizontal coupling = λ2 z h 2 x all times in seconds # proc γ top γ middle γ bottom time/iteration t setup [00%] 5.7 [00%] [98%] 6.0 [93%] [96%] 6.2 [92%] [101%] 10.2 [55%] [87%] 11.4 [50%] [41%] 11.7 [48%] [13%] 12.5 [46%]

26 26/21 Implicit timestepping Semi-implicit semi-lagrangian time stepping [F α t G] (n+1) = [F + (1 α) t G] (n) departure point F (n) F (n+1) departure point u (n+1) + α t c p θ (n+1) π (n+1) = R (n) u θ (n+1) = R (n) θ ρ (n+1) + α t ρ (n+1) u (n+1) = R (n) ρ Linearisation π (n+1) = π + π Equation for pressure correction π : (α t) 2 c 2 s (a π ) + bπ = RHS (n)

27 27/21 Robustness Dependence on parameters ω 2 and λ 2 Coefficient of 2nd order term (time step size) ω 2 = ( cs α t R earth ) 2 Use ω 2, 10 ω 2, 100 ω 2 Vertical coupling (depth of atmosphere, buoyancy term) λ 2 /h 2 = ( Rearth ) 2 1 H 1 + (α t) (N 0 ) 2 Use λ 2, 10 2 λ 2, 10 2 λ 2 Cartesian grid in [0, 1] [0, 1] [0, h]

28 28/21 Robustness # of iterations - 1 level method (CG + line smoother) r k / r 0 < 10 3 Iterations (ω 2,λ 2 ) (ω 2,10 2 λ 2 ) (ω 2,10 2 λ 2 ) (10 1 ω 2,λ 2 ) (10 2 ω 2,λ 2 ) [solver does not converge in 200 iterations] (8.4e+06) 64 (3.4e+07) 256 (1.3e+08) 1024 (5.4e+08) 4096 Number of processes (2.1e+09) (8.6e+09) (3.4e+10)

29 29/21 Robustness Number of iterations - Multigrid Iterations (ω 2,λ 2 ) [MG] (ω 2,10 2 λ 2 ) [MG] (ω 2,10 2 λ 2 ) [MG] (10 1 ω 2,λ 2 ) [MG] (10 2 ω 2,λ 2 ) [MG] (ω 2,λ 2 ) [Shallow MG] (ω 2,10 2 λ 2 ) [Shallow MG] (ω 2,10 2 λ 2 ) [Shallow MG] (10 1 ω 2,λ 2 ) [Shallow MG] (10 2 ω 2,λ 2 ) [Shallow MG] (8.4e+06) 64 (3.4e+07) 256 (1.3e+08) 1024 (5.4e+08) 4096 Number of processes (2.1e+09) (8.6e+09) (3.4e+10)

30 30/21 Comparison to DUNE geometric MG code Implemented together with Andreas Dedner (Warwick) Comparison of sequential runtimes ω 2 = , λ 2 /h 2 = dof Time per iteration [Intel(R) Core(TM)2 Duo CPU E GHz] all times in seconds Code Fortran MG DUNE MG rel. difference Cartesian % Spherical % Is n z large enough to hide indirect addressing?

31 31/21 Comparison to DUNE geometric MG code Time per iteration (Cartesian grid) time per iteration [s] ALUGrid SPGrid YaspGrid Fortran t iter = A(grid) + B n z n z

Matrix-free multi-gpu Implementation of Elliptic Solvers for strongly anisotropic PDEs

Iterative Solvers Numerical Results Conclusion and outlook 1/18 Matrix-free multi-gpu Implementation of Elliptic Solvers for strongly anisotropic PDEs Eike Hermann Müller, Robert Scheichl, Eero Vainikko