Parallel edge-based implementation of the finite element method for shallow water equations

Transcription

1 Parallel edge-based implementation of the finite element method for shallow water equations I. Slobodcicov, F.L.B. Ribeiro, & A.L.G.A. Coutinho Programa de Engenharia Civil, COPPE / Universidade Federal do Rio de Janeiro, Brazil Abstract A parallel implementation of the fmite element method is presented in this paper, focusing on the fully coupled stabilized solutions of nonlinear systems arising from the discretization of shallow water equations. Edge-based data structures are used in order to optirnize matrix-vector products appearing in the GMRES iterative solver. Numerical examples, including the tidal flow simulation in the coastal lagoon of Araruama, Brazil, show the speed-up of the present implementation, designed for PC clusters running MPI. 1 Introduction In this paper we are interested in the numerical solution of the so-called vertically averaged (2DH) model of the shallow water equations (SW) which satisfactory describes the hydrodynamics (circulation of water) in a class of wellmixed estuaries and coastal embayments. One of the major numerical difficulties associated with the SWE is its convection-dominated character. Many numerical procedures rely on characteristic or semi-lagrangian based methods in order to circumvent this drawback. Another way to deal with this problem, in its full Eulerian description, is to use stabilized finite element methods [2,4,5,6]. In this work we adopt this methodology, using a semi-discrete variational formulation for the non-conservative form of the S W, written in terms of velocity-celerity

2 14 Coastal Engineering V1 variables. Following this approach, the CAU finite element method [1,3] is constructed. The finite element discretization of the SWE leads to a nonlinear coupled nonsymmetric system of algebraic equations. In this paper we discuss a parallel implementation using the GMRES iterative solver. Edge-based data structures are used in order to optimize matrix-vector products appearing in the GMRES. Edge-based data structures have been used before with success in the solution of the SWE [7,8]. We simulate the tidal flow in the coastal lagoon of Araruama, Brazil, for three successive refined meshes, showing the good speed-up of the present implementation, designed for PC clusters running MPI. 2 Governing equations The 2DH model for the SWE is obtained by vertical integration averaging of the three-dimensional Navier-Stokes equations, with bottom and surface boundary conditions included. The principal limitation of vertically averaged models is that they do not consider the effects of velocity and density variations in the vertical direction. However, the 2DH model can be adequate for the consideration of pronounced unsteady flows in shallow water bodies. Let U, (i = 1, 2) represent the horizontal average velocity components and c = J g the ~ wave celerity, where g is the gravity acceleration and H = 77 +h is the total water depth (h is the water depth of the undisturbed configuration and 77 is the water surface elevation). Using these definitions the 2DH velocity/celerity model reads: where U=(Ul,U,,2c) ; U,,=dU/Bt and A=[A, A,F, VU=[U,, U,, = du lax, and A.VU = ATVU, with u,*t; I' : :l and

3 Coastal Engineering V1 15 In the above relations, p is the fluid density, f is the Coriolis parameter, v is the eddy viscosity,.rw are the wind shear stresses components at the free surface, g 112 and y = --(U: + U:), where C is the Chezy coefficient. C 3 Finite element formulation Let Q c!r2 be an open (spatial) domain with boundary r, and (0, T) the time interval of interest. According to the classical semi-discrete finite element discretization procedure, the domain a is subdivided into nel "elements" a, such that For each discrete time tn E (0, T) ; (n = 1,2,...), the finite element subspaces of piecewise continuous weighting functions and admissible functions are given respectively by where pk is the set of polynomials of degree less than or equal to k, and g are the prescribed boundary conditions. Under the above assumptions the semi-discrete stabilized finite element formulation reads: for each time tn, n = 1,2,..., find U h E Uh such that yeh E Eh:

4 16 Coastal Engineering V1 where R(Uh) = U: + A(U").VU" -V. (DVU") - F(Uh) is the residual associated with the approximate solution Uh. Moreover, in this formulation the first integral corresponds to the Galerkin method, the second integral comes from the SUPG contribution and the thud represents the additional contribution of the discontinuity-capturing operator (CAU). The definitions of T and can be found in references [l]. The usual finite element approximation procedure leads to a set of first order ODE: where U, is the column vector of nodal values of U* and U, is the correspondent time rate. The time derivatives can be approximated by a finite difference scheme such as the trapezoidal rule The resulting system of algebraic equations can be solved using the following predictor multi-corrector algorithm: U:,, = U"+ At(1 -a)~,, U:,, = 0 (predictor phase) For i = 0, 1, 2,... I (multi- corrector phase) where M * = M + aatk.

5 4 Parallel edge-based implementation Coastal Engineering V1 17 For the parallel implementation, the original domain R, that is subdivided into nel "elements" Re, should be first partitioned into N local subdomains RLi, i = 0, 1,..., N-l, where N is the number of processors (figure 1). Each subdomain RLi is subdivided into nelloc "elements" Re such that N-l Q= U Q, Q, nrlk =0 for j+k i=o Figure l. For N = 4. A partitioning program for unstructured meshes named METIS was used in this paper. METIS is a software package for partitioning large irregular graphs, large meshes and computing fill-reducing orderings of sparse matrices. It provides two methods for partitioning meshes (e.g., those arising in finite element or finite volume methods) into k equal size parts. The program initially converts the mesh into a nodal graph (each node of the mesh becomes a vertex of the graph) or into a dual graph (each element of the mesh becomes a vertex of the graph) and then performs its partitioning. METIS currently supports four different types of elements, i.e., triangles, tetrahedra, hexahedra or quadrilaterals. Triangular elements were used in all simulations in this paper. Figure 2 shows that elements A and B, through the common edge connecting node i to node j, contribute to the global edge-based matrix coefficients S(i,j). Both contributions SA(itj) and SB(irj) are stored in the corresponding element matrices. This can be done performing a loop on the elements and assembling the coefficients by edges.

6 18 Coastal Engineering V1 Figure 2. Element contributions. After performing the loop in each subdomain, all the coefficients related to the boundary edges must be added and equalized in each processor. Figure 3. Boundary edges on each subdomain for N=4. 5 Numerical examples The numerical example chosen to illustrate the performance of the parallel edgebased implementation is the simulation of a tidal flow in the coastal lagoon of Araruama, Brazil. We used a Linux Red Hat 7.2 cluster with 24 Pentium I11 1 GHz processors, fast-ethernet, running LAM-MP1 and Lahey Fortran. Three meshes were used in order to have an estimate of the speed-up of the implementation. The second and third meshes were obtained from the uniform refinement of the fmt (small). Figure 4 shows the small mesh and figure 5 the METIS partition for 4 processors. In Table 1 we list the topological characteristics of the small, medium and large meshes. The performance is evaluated using 10 time steps of 1, 10, 20, 30, 40 and 50s, with 10 Krylov space vectors for GMRES algorithm. GMRES and nonlinear iteration tolerances were set to 10".

7 Coastal Engineering V1 19 Figure 4. Small mesh. Figure 5. METIS partition for 4 processors. Table 1. Topological data for Araruama Lagoon meshes. Medium 75, , , ,628 Large 296, , , ,866 Speed-up results can be seen in the plots of Figures 6-8, respectively for the small, medium and large meshes for several simulations with different time steps. We may verify that speed-up increases with mesh refinement. However, the optimum number of processors for all simulations varies with time step and mesh size. For instance, for the small mesh the best speed-up is achieved for 12 processors, whereas for the large mesh we observed that the best speed-up is for 16 processors. In all simulations speed-up decreases as the time step increases. We may associate this behavior to the increasing difficulty to solve the resulting nonlinear systems as the time step increases. Note that for the small mesh 500s (10 time steps with dt=50s) were best simulated in 40s in 8 processors, reducing

8 20 Coastal Engineering V1 real time by a factor of In the medium mesh 500s were best simulated in 172s in 12 processors, yielding a reduction factor of 2.9. However, no reduction was observed in the large mesh simulations. If we use more powerful processors in the cluster we foresee that similar reductions in the simulation time will be possible for the large mesh too Processors Figure 6. Small mesh. Processors Figure 7. Medium mesh.

9 Coastal Engineering V $ Processors Figure 8. Large mesh. 6 Conclusions In this paper we have presented a cluster-oriented parallel edge-based implementation for the semi-discrete stabilized finite element formulation for problems governed by the shallow water equations. All computation related to formation and updating of edge matrices and residuals were parallelized. Edgebased matrix-vector products needed in the GMRES iterative solver were parallelized as well. A performance study for a real life tidal flow case shows that large coupled nonlinear solutions can be efficiently obtained in low cost parallel rnachnes. References Almeida, R. C., GaleBo, A. C., "An Adaptive Petrov-Galerkin Formulation for the Compressible Euler and Navier-Stokes Equations". Comput. Meth. Appl. Mech. Engrg, vol. 129, , Bova, S. W., Carey, G. F., "An entropy Variable Formulation and Petrov- Galerkin Methods for the Shallow Water Equations", in: Finite Element Modeling of Environmental Problems-Su$ace and Subsurface Flow and Transport, ed. G. Carey, John Wiley, London, England, Galelo, A. C., Do Carmo, E. G., "A Consistent Approximate Upwind Petrov-Galerkin Method for Convection-Dominated Problems", Comput. Meth. Appl. Mech. Engrg, vol. 32, , 1988.

10 22 Coastal Engineering V1 4. Ribeiro, F. L. B., Galego, A. C. and Landau, L., "A Space-Time Finite Element Formulation for Shallow Water Equations", in: Development and Application of Computer Techniques to Eizvimnme~ztal Studies Vl, pp , Computational Mechanics Publications, Ribeiro, F. L. B., Castro, R. G. S., Galego, A. C., Loula, A. F. D. and Landau, L., "A Space-Time Finite Element Formulation for Shallow Water Equations with Shock-capturing Operator", IV World Congress, Argentina, Saleri, F., "Some Stabilization Techniques in Computational Fluid Dynamics", Proceedings of the grh Ixternational Corzfererzce on Finite Elements in Fluids, Venezia, Ribeiro, F. L. B., Galego, A. C. and Landau, L., "Edge-based fmite element method for shallow water equations", International Journal for Numerical Methods in Fluids, vol. 36, pp , Ribeiro, F.L.B., Castro, R.G.S., Galego, A.C., Landau, L., "Finite Elements for Shallow Water Equations: Stabilized Formulations and Computational Aspects", Advances in Fluid Mechanics 111, pp , Computational Mechanics Publications, 2000.