Computer Physics Communications. Multi-GPU acceleration of direct pore-scale modeling of fluid flow in natural porous media

Transcription

1 Computer Pysics Communications 183 (2012) Contents lists available at SciVerse ScienceDirect Computer Pysics Communications ournal omepage: Multi-GPU acceleration of direct pore-scale modeling of fluid flow in natural porous media Saeed Ovaysi, Moammad Piri Department of Cemical and Petroleum Engineering, University of Wyoming, Laramie, WY , USA a r t i c l e i n f o a b s t r a c t Article istory: Received 23 June 2011 Received in revised form 8 Marc 2012 Accepted 16 April 2012 Available online 25 April 2012 Keywords: GPU computing Parallel programming Moving Particle Semi-implicit Particle-based metods Porous media MMPS Modified Moving Particle Semi-implicit (MMPS) is a particle-based metod used to simulate pore-scale fluid flow troug disordered porous media. We present a multi-gpu implementation of MMPS for ybrid CPU GPU clusters using NVIDIA s Compute Unified Device Arcitecture (CUDA). Message Passing Interface (MPI) functions are used to communicate between different nodes of te cluster and ence teir respective GPUs. Te accuracy and stability of te GPU implementation of MMPS are verified troug careful comparison wit te results obtained on conventional CPU-only clusters. We ten examine te speedup and scalability of te GPU implementation for pore-scale flow simulations in samples wit various sizes taken from te same natural porous system. We acieve a 134 speedup wit 60 grapics cards compared to 6 CPU cores wile maintaining a linear scalability. Incompressible fluid flow simulation to reac steady-state troug a 1 mm 1 mm 8 mm microtomograpy image of Benteimer sandstone is also performed in less tan Elsevier B.V. All rigts reserved. 1. Introduction Fluid flow in porous media is of great importance in many areas of science and tecnology including petroleum production, ydrology, and environmental remediation. Better understanding of fluid flow in porous media, owever, requires examining te pysics of fluid flow at te pore level (micron level). Currently, it is very difficult to acieve tis goal troug only experimental means. Terefore, it is crucial to develop models tat are capable of reproducing te true pysics of fluid flow troug porous media at te pore level. Recently, Modified Moving Particle Semi-implicit (MMPS) as been developed to directly model fluid flow in disordered porous media at te pore level [1]. MMPS, wen applied on ig-resolution images obtained using X-ray microtomograpy, can sed ligt on transport penomena in natural porous media [2]. However, te ig resolutions required to capture te pore-level complexities of natural porous media make te simulations computationally expensive. We ave previously presented a parallel implementation of te metod wic scales linearly on distributed memory clusters [1]. Noneteless, several ours are still required to complete a pysically useful simulation on more tan 200 processing cores. Fortunately, wit te advances made in General Purpose computation on Grapics Processing Units (GPGPU), it is possible to reduce te compuational cost significantly at a considerably lower price. In tis paper, we use Compute Unified Device Arcitecture (CUDA) developed by NVIDIA [3] to perform te simulations on Grapics Processing Units (GPUs). CUDA, wic is an extension of C, is integrated wit Message Passing Interface (MPI) to allow simulations across a multi-gpu platform wit distributed memory. Te underlying arcitecture of te code is written in C++. In te following sections, we first briefly introduce te computational algoritm of MMPS. Next, our single-gpu algoritm is discussed. We ten integrate te single- GPU code wit te domain decomposition tecnique facilitated by MPI to finalize a multi-gpu code tat runs on distributed memory computer clusters. Tis is ten followed by our scalability results obtained using te above-mentioned code. Finally, we present a case study in wic fluid flow in a large sample is simulated using te multi-gpu code. 2. MMPS MMPS is a Lagrangian particle-based metod used to solve te incompressible Navier Stokes equations in disordered porous media. Te voxel image of te porous medium renders itself to a particle-based representation were te rock (void) space is mapped into solid Corresponding autor. Tel.: ; fax: addresses: sovaysi@uwyo.edu (S. Ovaysi), mpiri@uwyo.edu (M. Piri) /$ see front matter 2012 Elsevier B.V. All rigts reserved. doi: /.cpc

2 S. Ovaysi, M. Piri / Computer Pysics Communications 183 (2012) (fluid) particles. To solve te incompressible Navier Stokes equations, MMPS uses a dual-level approac known as te pressure proection metod to calculate te velocity of fluid particles. First, an explicit velocity is calculated using v E = v + 1 ρ P s + µ ρ 2 v + g t (1) were v is velocity, ρ is density, µ is viscosity and g is gravity. Subscript E stands for explicit and D Dt wit respect to time, t. P s, static pressure, is calculated using 2 P s = 0. Te explicit velocity is ten used to calculate dynamic pressure, P d, using 2 P d = ρ v E + 1 n v k t k t t k=1 wic is ten used in te following equation to calculate te implicit velocity: denotes te substantial derivative (2) (3) v I = t ρ P d. It must be noted tat te second term in te rigt and side of Eq. (3) takes into account any deviation from te incompressibility criterion in te previous time steps, see [1] for more details. Te above equations are linearized using te particle-based summations proposed by Kosizuka [4], wic define te gradient of te scalar field A for particle i as (4) i A = d N i N A A i (r r i )W i r 2 i (5) were d is te number of spatial dimensions, r denotes te coordinates, r i = r i r is te distance between particles i and, N i = N W i is te number density, and W is a kernel tat gives iger weigts to te particles close to i tan te particles at a distance. To reduce te computational and memory costs, tis summation is performed only over a predefined radius of neigborood, kernel size, from i wic is denoted by in Eq. (6). Te number of neigboring particles residing in tis neigborood is denoted by N. In tis work, we use te following polynomial kernel: 2 2ri r i < 2 W i = 2 2ri 2 2 r (6) i < 0 r i. Similar principles are used to compute te divergence operator of te vector field v and te Laplacian operator of te scalar field A for particle i: were λ = i v = d N i 2 i A = 2d λn i V W(r)r2 dv V W(r)dv. N (v v i ) (r r i ) r 2 i N (A A i )W i W i In summary, te MMPS algoritm includes 1. Initialization 2. Neigbor searc 3. Compute P s using Eq. (2) 4. Compute V E using Eq. (1) 5. Compute P d using Eq. (3) 6. Compute V I using Eq. (4) 7. v = v E + v I 8. r = r + vt 9. t = t + t 10. if t < t final go to step finis were te neigbor searc in step 2 uses a linked-list searc algoritm wit O(N) complexity. Furtermore, te Bi-Conugate Gradient Stabilized (BiCGStab) solver is used to solve te systems of linear equations encountered in steps 3 and 5. Tese systems of linear equations (7) (8)

3 1892 S. Ovaysi, M. Piri / Computer Pysics Communications 183 (2012) Fig. 1. Computational times required to perform te main numerical operations in one time step of MMPS. Te computational times are averaged over 1000 time steps for a system composed of particles on one Intel Xeon X5670 core. Fig. 2. Memory ierarcies in te CUDA grid wit M blocks per grid and N treads per block. are obtained from te extended forms of Eqs. (2) and (3) using Eqs. (5) (8). Details of te derivations are given in [1]. Here we present only te final Eqs. (9) and (10) for P s and P d, respectively. N M i Nˆ i P si P s W i = 0 (9) were M i represents te subset of particles in te neigborood of i tat do not influence its P s, i.e., solids and disconnected particles. N N i P di P d W i = ρλ N (v E, v E,i ) (r r i ) N (v n v n W i + ) i (r r i ) W i ρλn n 1 i v k 2t 2dt 2 i tk (10) r 2 i were superscripts n and k denote te previous time step and te time steps earlier tan n, respectively. Fig. 1 presents te contribution of eac maor computation in one time step of MMPS. Te computational times are averaged over 1000 time steps for a system comprised of 315,000 particles. Knowing tat te particles move only a fraction of teir size at every time step, it is not necessary to perform neigbor searc at every time step (for instance, once every 10 time steps ere). Tis reveals te solution of te system of linear equations at step 5 as te bottleneck of te simulations wit a degree of freedom equal to te total number of particles. Depending on te resolution at wic te porous medium is imaged, te total number of particles for a (1 mm) 3 sample is of te order of tens of millions. Also, considering te movement of particles from one time step to anoter and te resulting cange in te velocity of te particles, te system of linear equations at step 5 presents a non-symmetric system tat requires several iterations of te BiCGStab solver to converge. In te next section, we discuss a single-gpu approac to overcome te computational cost associated wit tese computations. 3. Single-GPU acceleration A GPU consists of undreds of stream processors wic are grouped in stream multi-processors, see Table 1 for te specifications of two NVIDIA grapics cards. Fig. 2 illustrates te memory ierarcies tat are explicitly accessible to te stream processors troug te CUDA API functions. To access te GPU (device) processors, te CPU (ost) must launc a kernel function. Tis moves te execution to te device were Number of blocks per grid Number of treads per block treads execute te same instruction in te kernel function. Launcing a kernel wit a greater number of treads tan te actual number of processors can potentially overlap computation wit ig latency memory access to te GPU s global memory. Utilizing te sared and constant memories as well as coalescing are oter efficient ways to speed up GPU memory access, see [5] for te details. In tis section, we briefly explain te tecniques used to speed up te main MMPS operations outlined in Section 2. r 2 i k=1

4 S. Ovaysi, M. Piri / Computer Pysics Communications 183 (2012) Table 1 Specifications of two NVIDIA grapics cards. Grapics card GeForce GTX 580 Tesla C2050 Number of multiprocessors Number of processors Clock rate (GHz) Global memory (Mbytes) 1572 and 3071 (from ZOTAC) 2687 Constant memory (bytes) Sared memory per multiprocessor (bytes) Registers per multiprocessor Memory bandwidt (Gbytes/s) Max number of blocks per grid Max number of treads per block Neigbor searc Before using te linked-list algoritm, te entire domain must be divided into subdomains (cubes in tree dimensional systems) as large as te kernel size,. As sown in Fig. 3, to find te neigbors of any particle i, only te subdomains in te immediate neigborood of i are searced. Below is a pseudo-code for te CUDA kernel, performing neigbor searc using te linked-list algoritm. np is te number of particles. Storing te coordinates of te particles in te sared memory at te beginning reduces te cost of future memory accesses to tese data. global void neigbor_searc(...) id = blockidx.x * blockdim.x + treadidx.x; if(id<np) store coordinates of particle[id] in te sared memory for(over te x coordinate of te closest subdomains) for(over te y coordinate of te closest subdomains) for(over te z coordinate of te closest subdomains) for(= all te particles in te subdomain) r = distance between particle[id] and particle[] if(r<) add index of particle[] to array of neigbors for particle[id] 3.2. System of linear equations Te most time-consuming operation in te BiCGStab solver is matrix vector multiplication. To perform tis efficiently, we allocate eac row of te sparse matrix to a block. We ten use te sared memory to calculate te summation of multiplications for eac row. Tis algoritm proves to be four times faster tan te case were eac row is allocated to one tread. Below, we present a pseudo-code to multiply a sparse matrix caracterized by its elements, i.e., mult, and element ids, i.e., mult id, by vector B. Te results are stored in vector C. It sould be noted tat, in calculating te global summation, especial care must be taken to minimize tread divergence and ence improve te acieved speedup. To do tat, we ave employed te global summation tecnique presented in [5]. global void multiply(...) id = blockidx.x; tx = treadidx.x; sared float s_c[dimension of te block]; if(id<np) s_c[tx] = mult[tx]* B[mult_id[tx]]; synctreads(); for(s=blockdim.x>>2; s>0; s>>=1) if(tx<s) s_c[tx] = s_c[tx+s]; synctreads(); if(!tx) C[id] = s_c[0];

5 1894 S. Ovaysi, M. Piri / Computer Pysics Communications 183 (2012) Fig. 3. In te linked-list algoritms only te immediate neigboring subdomains, sown in grey, are searced. P1 P2 P3 P4 Fig. 4. Te domain is decomposed along te subdomains of te linked-list algoritm (small squares). Eac process communicates only its grey squares wit its neigboring processes. 4. Multi-GPU acceleration Flow simulation in large systems requires a considerable amount of memory tat is beyond wat a single GPU can provide. For tis reason, and also to acieve iger speedups, it is necessary to develope a multi-gpu implementation. Here, we first discuss domain decomposion. Eac subdomain is ten allocated to one MPI process wic in turn ands over te simulation to one GPU. Message passing and syncronization of te GPUs are done by MPI functions. An optimized domain decomposition must lead to a balanced distribution of computational load and also minimum communication between te processes [6]. To acieve load balance, we decompose te domain into subdomains wit equal volumes. Since we model te flow of incompressible fluids, subdomains wit equal volumes implies anding over an equal number of particles to eac MPI process. Also, to optimize te neigbor searc step for eac process, domain decomposition must be done along te subdomains of te linked-list algoritm discussed in Section 3. Fig. 4 illustrates a case were te entire domain is decomposed between 4 processes. At te initialization pase, te grey linked-list subdomains (small squares in te figure) are marked and te neigbor searc is initiated at eac time step by passing te coordinates of te particles in te grey subdomains to oter processes. It sould be noted tat, in practice, only a fraction of te particles in te grey subdomains are te actual neigbors of te particles in te neigboring processes. Since eac iteration of te BiCGStab solver

6 S. Ovaysi, M. Piri / Computer Pysics Communications 183 (2012) Fig. 5. Te actual neigboring particles of process Pn are sown wit filled particles. Tis constitutes only 68% of te particles in te grey area. Fig. 6. GPU-computed velocity distribution in a 0.5 mm 0.5 mm 1 mm Berea sandstone X-ray image wit µm resolution and a total of 2,519,424 particles. Te sample is under 1 Pa pressure difference along te z direction (from left to rigt) and te figure illustrates te results after 0.01 s of te real time. requires twice updating te information in te oter processes, it is wise to first mark te actual neigboring particles of eac process and ten for te rest of te communications only pass te actual neigboring particles. In Fig. 5, te actual neigboring particles constitute only 68% of te particles in te grey zone. Considering tat several iterations are required to solve te system of linear equations arising from Eqs. (2) and (3), tis 32% reduction in te communication cost goes a long way in furter optimizing te code. More optimization is acieved by overlapping te computations wit communications using non-blocking MPI calls. For instance, wile performing neigbor searc, a non-blocking MPI call can pass te coordinates of te particles residing in te grey subdomains to te neigboring process at te beginning wit almost no cost. Te receiving process can start by calling a non-blocking receive operation and immediately begin searcing te inner subdomains tat do not need information from te neigboring processes and ten wait for te non-blocking receive operation to complete, wic ten allows it to proceed wit searcing te periperal subdomains. Also, furter overlapping is acieved by performing te communications tat are required to mark te actual neigbors in anoter tread of te same MPI process wile te master tread performs computations. 5. Scalability and results To evaluate te scalability of our GPU implementation of MMPS and te accuracy and stability of te results, we studied a 0.5 mm 0.5 mm 1 mm sample from te Berea sandstone image in [7]. Fig. 6 visualizes te GPU-computed velocity distribution in tis sample wit te same accuracy as te results obtained using only CPUs. Te accuracy of te CPU-only results as previously been verified in [1,2] against analytical/numerical/experimental data available in te literature. Te reported results in tis section are obtained using a computer cluster wit 30 nodes. Eac node is equipped wit 2 AMD Opteron 6128 CPUs and 2 NVIDIA GTX 580 GPUs and 32 GB of memory. In addition to tat, te ead node of tis cluster is equipped wit 2 Intel Xeon X5670 CPUs and 8 GPUs (4 NVIDIA Tesla C2050s and 4 NVIDIA GeForce GTX 580 GPUs). Depending on weter a parallel CPU-based code as been developed or not, te scalability results can be presented in two ways (see te next paragrap). Since, we ave already developed a parallel CPU-based code [1], te scalability results in tis work are presented in bot ways. Furtermore, all te simulations, i.e., bot CPU based and CPU GPU based, are performed using single precision floating point operations.

7 1896 S. Ovaysi, M. Piri / Computer Pysics Communications 183 (2012) Fig. 7. Speedup of te main numerical operations of an average time step of MMPS on a system comprised of 315,000 particles using a single Tesla C2050 GPU compared wit a single Intel Xeon X5670 core. Te numbers are averaged over 1000 time steps. Fig. 8. Scalability results for different systems. System size measures te relative number of particles compared to a base system comprised of 315,000 particles. CPU times are obtained using all te 6 cores of an Intel Xeon X5670 CPU and te GPU results are based on NVIDIA GTX 580 GPUs. In te first approac, we are interested to know ow muc performance could be gained if we were to perform te MMPS computations on a GPU rater tan using a sequential code tat utilizes only one core of a CPU. Fig. 7 summarizes te speedup acieved for te maor numerical operations in an average time step of MMPS on a single Tesla C2050 GPU for a system composed of 315,000 particles. Compared to Fig. 1, te tree main contributors to te computational time, namely neigbor searc, computation of P s, and computation of P d, ave gained, on average, more tan 26 speedup. However, tis speedup proved to be dependent on te system size. Te same simulation on a system wit twice te size of te base system, i.e., wit 630,000 particles, acieved more tan 28 speedup. Interestingly, te less expensive GTX 580 outperformed Tesla C2050 and acieved a 34 speedup on te base system wic translates to a 36% better performance. Having developed a CPU-based parallel code, in te second approac we are interested in examining ow fast te code runs on multiple GPUs wen compared to te performance gained using all te cores of a CPU. To do tat, we first compute te computational time on all te 6 cores of te Intel Xeon X5670 CPU on te ead node. Ten, we obtain te computational time for te same simulation on te computing nodes. Fig. 8 presents te scalability results for different system sizes relative to te base system tat is composed of 315,000 particles. As sown in tis figure, te multi-gpu code performs better wen larger systems are used. Tis observation, wic is consistent wit te results presented in [8], underscores te cost of expensive memory access in GPUs. Since eac particle is assigned to one CUDA tread, by increasing te system size, and ence te number of particles, one increases te number of treads tat operate at a given time. Tis allows a better utilization of te processing power as te idle treads can start processing wile oter treads wait to access te GPU s global memory. Te actual computational times are also reported in Table 2. Tese numbers reveal a 3.3 better performance for one NVIDIA GeForce GTX 580 over all te 6 cores of an Intel X5670 CPU. It sould be stressed tat, depending on te orientation of te processes in te x, y, and z directions, domain decomposition can sligtly increase/decrease te number of iterations required to obtain te same level of precision. 6. Case study In tis section we study pore-scale fluid flow in Benteimer sandstone. Te entire microtomograpy image, sown in Fig. 9, is a 4 mm 4 mm 9 mm cylinder wit µm resolution. We cut a 1 mm 1 mm 8 mm sample from te central part of tis image wic amounts to 43,740,000 particles and apply a 25 Pa pressure difference across its lengt, i.e., from left to rigt as sown in Fig. 10.

8 S. Ovaysi, M. Piri / Computer Pysics Communications 183 (2012) Fig. 9. Isosurface of a 4 mm 4 mm 9 mm Benteimer sandstone sample wit µm resolution. Fig. 10. Distribution of static pressure in a 1 mm 1 mm 8 mm sample from a Benteimer sandstone microtomograpy image wit µm resolution. Only te fluid particles are visualized. Fig. 11. Distribution of velocity in a 1 mm 1 mm 8 mm sample from a Benteimer sandstone microtomograpy image wit µm resolution. Only te fluid particles faster tan m/s are sown. Table 2 Average computational times, in seconds, for one time step of flow simulations. CPU results are obtained using all te 6 cores of an Intel Xeon X5670 CPU and te GPU results are based on NVIDIA GeForce GTX 580 GPUs. System size CPU GPU NA NA 2 GPUs NA 4 GPUs GPUs GPUs GPUs GPUs GPUs GPUs GPUs GPUs GPUs Fig. 11 sows te velocity distribution in te most active fluid cannels, i.e., faster tan m/s, at steady-state, i.e., t = s. For tis simulation wic completed in less tan 1 we used a s time step and let te simulation run for 2000 time steps. To cross-ceck te accuracy of te results, exactly te same simulation was run on te entire 480 CPU cores of te cluster. However, te CPU-based simulation, wic was as accurate as its ybrid counterpart, completed in Conclusions We presented an MPI CUDA implementation of MMPS to simulate fluid flow troug naturally-occurring porous media on ybrid CPU GPU clusters. Tis implementation as proved to be very efficient wit linear scalability and as enabled us to perform pysically meaningful simulations in a practical amount of time. Altoug te largest system size is still limited by te total global memory of all te GPU cards, tis implementation as enabled us to look at significantly larger samples of natural porous media wit iger resolutions. Te scalability results presented in tis paper lead us to conclude tat ybrid CPU GPU clusters can deliver te same computational power at

9 1898 S. Ovaysi, M. Piri / Computer Pysics Communications 183 (2012) a muc lower price tan conventional CPU clusters. Considering te current market prices of bot te NVIDIA GeForce GTX 580 grapics card and te Intel X5670 CPU and te results presented in Section 5, te same simulation would be 9.6 less expensive on an NVIDIA GeForce GTX 580 grapics card. Tis olds true if one were to develop a parallel code tat would utilize all te available 6 CPU cores on te Intel X5670 CPU. We believe tat tis opens up new doors to applications demanding ig computational power at lower cost. References [1] S. Ovaysi, M. Piri, J. Comput. Pys. 229 (2010) [2] S. Ovaysi, M. Piri, J. Contam. Hydrol. 124 (2011) [3] NVIDIA, NVIDIA CUDA programming guide, version 3.0, [4] S. Kosizuka, H. Tamako, Y. Oka, Int. J. Comput. Fluid Dyn. 4 (1) (1995) [5] D.B. Kirk, W.W. Hwu, Programming Massively Parallel Processors: A Hands-on Approac, Morgan Kaufmann Pub., [6] D.E. Culler, J.P. Sing, A. Gupta, Parallel Computer Arcitecture: A Hardware/Software Approac, Morgan Kaufman Pub., [7] H. Dong, Micro-CT imaging and pore network extraction, P.D. Diss., Dept. Eart Sci. Eng., Imperial College, London, [8] D.A. Jacobsen, J.C. Tibault, I. Senocak, An MPI-CUDA implementation for massively parallel incompressible flow computations on multi-gpu clusters, in: 48t AIAA Aerospace Sciences Meeting, 2010.