Parallel lattice fluid simulations for transport phenomena in porous media

Size: px

Start display at page:

Download "Parallel lattice fluid simulations for transport phenomena in porous media"

Samson Pope
5 years ago
Views:

Parallel lattice fluid simulations for transport phenomena in porous media T. Watanabel, K. Kono2 & H, 0hashi3 ljapan Atomic Energy Research Institute, Japan.

1 Parallel lattice fluid simulations for transport phenomena in porous media T. Watanabel, K. Kono2 & H, 0hashi3 ljapan Atomic Energy Research Institute, Japan. Fuji Research Institute Corporation, Japan. 3Univemity of Tokyo, Japan. Abstract Parallel simulation programs using the lattice fluid methods are developed for studying the transport phenomena in porous media. The domain decomposition technique is applied and the message passing interface library is used for parallelization. Sample problems are solved to validate the simulation programs and evaluate parallel efficiency. It is shown for the LBGK simulations, in which the amount of data transfer between two processors is constant, that the on-nodeparallel calculation using shared memory is effective for the calculation time and the memory usage in comparison with the purely distributed calculations. The efficiency of data transfer for the RLG simulations, in which the amount of data transfer varies every time step, is found to be better for sending the number of data and some amount of particle data at the same time. 1 Introduction It is of importance for safety assessment of geological disposal of high-level radioactive waste (HLW) to predict the transport phenomena in the natural barrier such as rock matrix. Numerical simulations of those phenomena are extremely difficult since the material transport mechanism of HLW is complicated due to geometrical complexity of rock matrix and multiple physical and chemical phenomena in groundwater. A joint research project is being carried out by University of Tokyo, Fuji Research Institute and Japan Atomic Energy Research Institute to establish a simulation system for geological disposal of HLW. The project consists of a development of a multi-scale simulation method using lattice fluid models and a

~~6 Applications of High-Performance Computing in Engineering VII development of virtual experiment environment using high performance computing and virtual reality techniques [1].

2 ~~6 Applications of High-Performance Computing in Engineering VII development of virtual experiment environment using high performance computing and virtual reality techniques [1]. The lattice fluid models, which are based on the dynamics of discrete fluid particles, have recently progressed a great deal. Motions of fluid particles are calculated instead of solving continuous fluid equations, and macroscopic flow variables are obtained from the particle motion. Among particle simulation methods, the lattice gas automata (LGA) is one of the simple techniques for simulating complicated flow phenomena, as well as macroscopic flow fields. In the LGA introduced by Frisch, Hasslacher and Pomeau [2], space and time are discrete, and identical particles of equal mass populate a triangular lattice. The particles travel to neighboring sites at each time step, and obey simple collision rules that conserve mass and momentum. Macroscopic flow fields are obtained by coarse-grain averaging in space and time. The LGA has, however, some inherent drawbacks such as the lack of Galilean invariance, and several lattice fluid models have been proposed to overcome the drawbacks. The lattice BGK (LBGK) method and the real-coded lattice gas (RLG) method are used in our project to simulate the complicated transport phenomena in the natural barrier. In the LBGK method, the particle occupation variables, which are Boolean variables in the LGA, are replaced with single-particle distribution functions [3], and the collision process of particles is simplified to become a relaxation process of the distribution fimction toward the local equilibrium [4], The distribution fimctions are real variables as the ensemble average of particle occupation, and the local equilibrium distribution is chosen to recover the Navier-Stokes equations [5]. The RLG method has been proposed by Malevanets and Kapral [6]. In this method, the velocities of particles are represented by real variables, while they are Boolean variables in the LGA. The motion of particles are thus not restricted on the lattice. A stochastic propagation scheme with a multi-particle collision rule that conserve mass, momentum and energy is applied to the particles. Since the algorithm and programming are simple, and complex boundary geometries are easy to represent, the LBGK and the RLG methods have been applied to numerical simulations of various kinds of hydrodpamic flows [7,8,9,10]. The complicated structure of rock matrix and various kinds of flows with physical and chemical phenomena are modeled at the microscopic level using the LBGK and the RLG methods in this project. The domain decomposition technique is applied and the message passing interface library is used for parallelization. In this study, characteristics of the LBGK and the RLG methods in parallelization are described and the parallel efficiency is evaluated. 2 Lattice fluid models 2.2 Lattice BGK method In the LBGK method, a particle distribution functions fi(x,t) at space x and time t are introduced to represent a fluid, where the subscript i indicates the direction of

3 Applications of High-Performance Computing in Engineering VII 117 moving particle on a lattice. In this study, the 3-D 15-direction model is used. The lattice Boltzrnann equation is given by fi(x+e~, t+l)-fi(x, t)= Q~(x, t), (1) where e is a particle velocity vector and the right hand side is the collision operator. The linearized collision operator with a single time relaxation parameter is used in the LBGK method: Qi.+.jfq. r Here f~q is the local equilibrium velocity, and ~ is a characteristic momentum must be satisfied: (2) state depending on the local density and relaxation time. Conservation of mass and (3) where p and u is the density and the local velocity of the fluid, respectively. Using the Chaprnan-Enskog multiscale expansion, the equilibrium distribution func;on is obtained by - for I ei 1=O, U) + ~(ei c24)2 :U2)I for ei 1=1, (4) 212 fieq =L[~+:(ei tl)+~(ei u) ;U )1 for I ei 1= J 3, 893 so that the macroscopic mass and momentum conservation equations for continuous fluid are recovered. 2.2 Real-coded lattice gas method In the RLG method, fluid particles have a continuous velocity in both spatial direction and magnitude. The trajectories of the particles are traced in a shorttime interval by decoupling interparticle collisions, In the translational process,

~~8 Applications of High-Performance Computing in Engineering VII locations of the particles are obtained using the velocities calculated in the previous collision process, r(t + At) = r(t)+ V(t),

4 ~~8 Applications of High-Performance Computing in Engineering VII locations of the particles are obtained using the velocities calculated in the previous collision process, r(t + At) = r(t)+ V(t), (5) where r and v represent the location and the velocity of a particle. In the collision process, velocities of particles are modified while their locations remain unchanged, The velocities of the particles are obtained using the average velocity in a collision cell, V, and the rotation operator, o: v(t + At) = ~ + C7(V(t) ~). (6) The vector of velocity difference between the average velocity and the particle velocity is rotated by some angles defined by the rotation operator. The mass, momentum and energy are conserved during the collision process. 3 Parallel efficiency The lattice fluid methods consist of a streaming process and a collision process: the streaming and the relaxation of the distribution fi,mction in the LBGK method, and the streaming of particles and the multi-particle collision in the RLG method, The streaming and the collision processes are calculated locally, and high efficiency of parallel computation is expected in the lattice fluid methods. The efficiency of parallel computation for the lattice fluid methods is studied here, The domain decomposition technique is applied and the message passing interface (MPI) library is used. The parallel computer system, HITACHI SR8000, with 4 nodes is used, where one node consists of a shared memory and 8 processor elements (PEs). In this systeq purely distributed calculations are possible using 32 PEs. Variables and calculation procedures are divided into 32 parts, and the data transfer among PEs is performed using the MPI even in the same node, The on-node-parallel calculations are also possible using the shared memory. Variables and calculation procedures are divided into 4 parts in this case, and the data transfer among nodes is performed using the MPI. Calculation procedures in the node are, then, divided into 8 parts using the shared memory, and the data transfer is not necessary inside the node. The purely MPI calculations and the (MPI + on-node-parallel) calculations are compared and the efficiency of data transfer is discussed in the following. The workstation cluster with 4 COMPAQ XP 1000 workstations are also used for comparison. 3.1 LBGK simulations The sample problem for the LBGK method is the phase separation process from a homogeneous mixture of two fluids in 64x64x192 lattice nodes. The twocomponent two-phase model of the LBGK is used [7], The periodic boundary

Applications of High-Performance Computing in Engineering VII 119 conditions are applied to the top, botto~ and side boundaries, Examples of the calculated results are shown in Figure 1, where the

5 Applications of High-Performance Computing in Engineering VII 119 conditions are applied to the top, botto~ and side boundaries, Examples of the calculated results are shown in Figure 1, where the formation of interface between two fluids is depicted every 200 time steps. It is shown that the phase separation occurs and the size of each fluid region increases. One of the separated phases shown in Figure 1 is used as the complicated structure of rock matrix. The simulation region consists of 64x64 lattice points in the horizontal direction, and 192 lattice points in the vertical direction. Variables and calculation procedures are divided into 4 small parts in the vertical direction using the domain decomposition method. Each small part is assigned to each node. The variables and calculation procedures in the node are further divided into 8 small parts for the purely MPI calculations, while the calculation procedures alone are divided for the (MPI + on-node-parallel) calculations. The amount of data transfer between two processors is constant for the purely MPI calculations and that between two nodes is constant for the (MPI + on-nodeparallel) calculations. Several specific commands are inserted in the simulation program in order to use the on-node-parallel functions, and thus some more modifications are necessary for the (MPI + on-node-parallel) calculations. The speedup is shown in Figure 2, where the (MPI + on-node-parallel) cases are compared with the purely MPI cases on SR8000 and on the workstation cluster. The average speedup of five calculations is shown. For the (MPI + onnode-parallel) calculations using 8 PEs in one node, the calculation time is about 693 sec and larger than 560 sec of the purely MPI calculations. The on-nodeparallel ilmction needs additional calculation time, though the data transfer is not necessary in the node since the shared memory is used. The speedup of the (MPI + on-node-parallel) calculations inone node is thus lower than that of the MPI calculations, The calculation time of the (MPI + on-node-parallel) calculations with 4 nodes is about221 sec and slightly smaller than 237 sec of the purely MPI calculations. The speedup ratio is about 17.3 for the (MPI + on-node-parallel) calculations with 4 nodes, and about 16.2 for the purely MPI calculations using 32 PEs, The time for data transfer is shown in Figure 3, where the rate of the data transfer time to the elapsed time is shown. Domain decomposition is performed in one direction, and the amount of data transfer for one PE is constant with increase in the number of PEs, though the amount of calculation is reduced, The time for data transfer using 32 PEs is about 45 A of the total calculation time for the purely MPI calculations, while about 35 % for the (MPI + on-node-parallel) calculations. The rate of the data transfer time to the elapsed time is smaller for the (MPI + on-node-parallel) calculations, but the additional calculations are necessary, and thus the speedup is not much different between the purely MPI and the (MPI + on-node-parallel) cases as shown in Figure 2. It is seen in Figure 2, however, that the on-node-parallel finction is effective as the number of nodes and PEs increases. The normalized memory size of simulation program is shown in Figure 4, The memory size for a single PE is reduced and the total memory size is increased for both cases. The memory size for the case using 8 PEs on one node is about 130

~20 Applications of High-Performance Computing in Engineering VII MB for the (MPI + on-node-parallel) calculations and 480 MB for the purely MPI calculations.

6 ~20 Applications of High-Performance Computing in Engineering VII MB for the (MPI + on-node-parallel) calculations and 480 MB for the purely MPI calculations. The difference of memory size is due to an additional buffer memory needed for the data transfer by the MPI. Variables are divided into 8 parts in one node for the purely MPI calculations, while they are not divided for the (MPI + on-node-parallel) calculations, The amount of buffer memory is thus about 8 times larger for the purely MPI calculations. It is found that the shared memorys ystem is effective from the viewpoint of the memory size. 3.2 RLG simulations In parallel RLG simulations, the domain decomposition method is applied and the MPI library is used. The fluid particles are used in the RLG simulations, and the particle data are exchanged among PEs during the calculations. The amount of data transfer between two PEs varies every time step, since the motion of the fluid particles is not restricted on the lattice. For the data transfer of the parallel RLG simulations, two send-receive procedures are generally required. The number of data is sent first and all the particle data are sent next from one PE to other PEs. In the following, the efficiency of data transfer is studied for sending the number of data and some amount of particle data fust and the remaining particle data next. The sample problem for the RLG method is the thermal convection using 1.28 million particles in 40x40x20 collision cells. The top and bottom boundaries are cooled and heated, respectively, while periodic boundary conditions are applied to the side boundaries. This sample problem known as the Rayleigh-Benard convection is used for a validation of the heat transfer calculations in the RLG method. The simulation region is divided two-dimensionally in the horizontal directions in this case. Examples of the calculated results are shown in Figure 5, where the formation of convection pattern at the midelevation of the simulation region is depicted. The particle velocities are averaged in the collision cells and the distribution of vertical flow velocities are shown. The speedup is shown in Figure 6, where the purely MPI calculations are performed, The number of particles sent from one PE to other PEs in one time step is about 250 for the case with 32 PEs. In Figure 6, some amount of particle data (Num Data) is sent f~st with the number of particles: Num Data Oindicates that the number of particles alone is sent fust and all the particle data are sent next, Num Data 5 indicates that 5 particle data are sent first with the number of particles and the remaining particle data next, Num Data 50 indicates that 50 particle data are sent first with the number of particles and the remaining particle data next, The speedup of these three cases is not much different as shown in Figure 6. The speedup is about 19.0 using 32 PEs. The time for data transfer in this case is about 11 YO of the total computation time. The number of particles assigned to one PE decreases as the number of PEs increases. The amount of data transfer, which is the number of particles sent from one PE to other PEs, decreases as well, and the time for data transfer is relatively smaller for the RLG method than for the LBGK method. The average data transfer time in the case with 32 PEs is shown in Figure 7

Applications of High-Performance Computing in Engineering VII 121 against the number of particle data sent fust, It is clearly seen that the data transfer time becomes small when the number of

7 Applications of High-Performance Computing in Engineering VII 121 against the number of particle data sent fust, It is clearly seen that the data transfer time becomes small when the number of particle data is 42 and 170. In our RLG progra~ each particle has 3 velocities and 3 coordinates in 4 bite real numbers, and thus 42 and 170 particle data correspond to 1024 B and 4096 B, respectively, These two values are the buffer size for the send-receive process of the MPI. Although the buffer size depends on the computer systems, the efficiency of data transfer would be better for sending some amount of particle data first with the number of data. 4 Conclusion Parallel simulation programs using the LBGK and the RLG methods have been developed for studying the transport phenomena in porous media. The domain decomposition technique has been applied and the MPI library was used. Sample problems were solved to validate the simulation programs and evaluate parallel efficiency. It was shown for the LBGK method, in which the amount of data transfer is constant, that the on-node-parallel calculation using shared memory was effective for the calculation time and the memory usage in comparison with the purely distributed calculations. The efficiency of data transfer for the RLG method, in which the amount of data transfer varied every time step, was found to be better for sending some amount of particle data fust with the number of data. Acknowledgments This work was a part of the IAE s (Institute of Applied Energy) IVNET (Innovative and viable nuclear energy technology) projects funded by the Agency of Natural Resources and Energy, Ministry of Economy, Trade and Industry (METI), References [1] Itoh, G., Nakamura, J., Kono, K., Watanabe, T., Ohashi, H., Chen, Y,, & Nagasaki, S. Pore-scale simulation for predicting material transport through porous media. Proc. of the 10 hint. Con$ on Nuclear Engineering, 2002, [2] Frisch, U., Hasslacher, B. & Pomeau, Y. Lattice-gas automata for the Navier-Stokes equations. Phys. Rev. Lett., 56, pp , [3] McNamara, G.G. & Zanetti, G. Use of the Boltzmann equation to simulate lattice-gas automata. Phys. Rev, Lett., 61, pp , [4] Bhatnagar, P,L., Gross, E,P, & Krook, M. A model for collision processes in gases. I: small amplitude processes in charged and neutral one-component system. Phys. Rev., 94, pp , [5] Chen, H., Chen, S. & Matthaeus, W.H, Recovery of the Navier-Stokes equations using a lattice-gas Boltzmarm method. Phys. Rev. A, 45, pp. R5339-R5342, [6] Malevanets, A, & Kapral, R, Continuous-velocity lattice-gas model for fluid

8 ~22 Applications of High-Performance Computing in Engineering VII flow. Europhys. Lett., 44, pp , [7] Rothman, D.H. & Zaleski, S. Lattice-gas models of phase separation: interfaces, phase transitions, and multiphase flow. Rev. Mod. Phys., 66, pp , 1994, [8] Chen, H. & Doolen, G.D, Lattice Boltzmann method for fluid flows, Annu. Rev. Fluid Mech., 30, pp , [9] Hashimoto, Y., Chen, Y. & Ohashi, H. Immiscible real-coded lattice gas. Computer Phys. Comm., 129, pp ,2000. [10] Sakai, T., Chen, Y. & Ohashi, H. Formation of micelle in the real-coded lattice gas. Computer Phys. Comm., 129, pp ,2000. Figure 1: Simulation of phase separation using the LBGK method.? I 15.0 Cl ,0 0,0 0 EKE!EEb 1 I I.. Y. D Number d Processors Figure 2: Speedup of the LBGK method.

9 Applications of High-Performance Computing in Engineering VII 123 ~MPl --U MPI + (h Node Parallel, -d I o Number of Processors Figure 3: Data transfer time vs. number of PEs L I J 1.00 Total D. n PE CH3MPI - u MPI + On Node Parallel 0.01 \ Number of Processors Figure 4: Memory size for the LBGK method,

10 ~24 Applications of High-Performance Computing in Engineering VII Figure 5: Simulation of thermal convection using the RLG method EEizJ 1,, 0.0 o Number of Processors Figure 6: Speedup of the RLG method I 1 I 20.0 o Number of Data Figure 7: Data transfer time vs. number of data.

A Particle Cellular Automata Model for Fluid Simulations

Annals of University of Craiova, Math. Comp. Sci. Ser. Volume 36(2), 2009, Pages 35 41 ISSN: 1223-6934 A Particle Cellular Automata Model for Fluid Simulations Costin-Radu Boldea Abstract. A new cellular-automaton