Solving the Savage Hutter equations for granular avalanche flows with a second-order Godunov type method on GPU

Transcription

1 INTERNATIONAL JOURNAL FOR NUMERICAL METHODS IN FLUIDS Int. J. Numer. Meth. Fluds 015; 77: Publshed onlne 18 December 014 n Wley Onlne Lbrary (wleyonlnelbrary.com) Solvng the Savage Hutter equatons for granular avalanche flows wth a second-order Godunov type method on GPU Jan Zha 1,LYuan 1, *,,WeLu 1 and Xntng Zhang 1 LSEC & NCMIS, Insttute of Computatonal Mathematcs and Scentfc/Engneerng Computng, Academy of Mathematcs and Systems Scence, Chnese Academy of Scences, Bejng , Chna École Centrale de Pékn, Behang Unversty, Bejng , Chna SUMMARY In ths artcle, we apply Davs s second-order predctor-corrector Godunov type method to numercal soluton of the Savage Hutter equatons for modelng granular avalanche flows. The method uses monotone upstream-centered schemes for conservaton laws (MUSCL) reconstructon for conservatve varables and Harten Lax van Leer contact (HLLC) scheme for numercal fluxes. Statc resstance condtons and stoppng crtera are ncorporated nto the algorthm. The computaton s mplemented on graphcs processng unt (GPU) by usng compute unfed devce archtecture programmng model. A practce of allocatng memory for two-dmensonal array n GPU s gven and computatonal effcency of two-dmensonal memory allocaton s compared wth one-dmensonal memory allocaton. The effectveness of the present smulaton model s verfed through several typcal numercal examples. Numercal tests show that sgnfcant speedups of the GPU program over the CPU seral verson can be obtaned, and Davs s method n conjuncton wth MUSCL and HLLC schemes s accurate and robust for smulatng granular avalanche flows wth shock waves. As an applcaton example, a case wth a teardrop-shaped hydraulc jump n Johnson and Gray s granular jet experment s reproduced by usng specfc frcton coeffcents gven n the lterature. Copyrght 014 John Wley & Sons, Ltd. Receved 8 Aprl 014; Revsed 6 November 014; Accepted 0 November 014 KEY WORDS: Savage Hutter equaton; stoppng crteron; HLLC scheme; MUSCL reconstructon; GPU computng 1. INTRODUCTION Landsldes, rock avalanches, and debrs flows are natural phenomena, whch frequently occur n mountanous areas. They may cause great loss of propertes and lves and have been extensvely studed by many researchers n varous dscplnes. However, these phenomena are multphase, polydsperse, multscale, erosve, and rheologcally complcated, entalng hgh complexty to scentfc studes. It s well known that the basc composton materals n the aforementoned geomorphologcal phenomena are granular materals. Granular materals are a collecton of a large number of dscrete sold partcles wth nterstces flled wth one or more fluds [1]. Motons of a plenty of granular materals make up granular flows, whch are a contnuum treatment used to descrbe real avalanches and debrs flows. The granular materals drven by the gravty slde and accelerate down the slope, formng granular avalanche flows that generally have small thcknesses but can travel very long dstances. At the present stage, the man models for granular avalanche flows are typfed by shallow granular flow models such as the Savage Hutter (SH) model [], the Iverson Denlnger two-phase *Correspondence to: L Yuan, Academy of Mathematcs and Systems Scence, Insttute of Computatonal Mathematcs and Scentfc/Engneerng Computng, Chnese Academy of Scences, Bejng , Chna. E-mal: lyuan@lsec.cc.ac.cn Copyrght 014 John Wley & Sons, Ltd.

2 38 J. ZHAI ET AL. mxture model [3, 4], and the two-phase thn layer model proposed by Ptman [5], to name a few. The dervaton of shallow flow models utlzes the depth average approach and ther canoncal equatons are often wrtten n a Cartesan coordnate system wth the axs normal (or approxmately normal) to the basal surface. The canoncal formulatons [, 3, 5] are not sutable for numercal calculaton of avalanche flows over general topography. To smulate granular avalanche flows on general three-dmensonal (3D) terrans, the SH model can be formulated n a general curvlnear coordnate system wth x;y axes tangental to and axs normal to the basal surface, but such forms contan many non-conservatve metrc terms and may lead to computatonal complexty. Bouchut and Westdckenberg [6] derved a shallow water models n whch the coordnate system s a global Cartesan system, but the soluton varables are the flow depth n the drecton normal to the topography and the velocty tangent to the topography. However, the pressure term can not be wrtten n conservatve form for varable bed slope angles. Denlnger and Iverson [7] reformulated the depthaveraged governng equatons n a global Cartesan coordnate system wth vertcal and accounted explctly for the effect of non-zero vertcal acceleratons on depth-averaged mass and momentum fluxes and stress states. Ther equatons provde a hope of usng global Cartesan formulaton to descrbe shallow granular flows over general 3D terrans. In ths paper, we focus on numercal method for the SH model formulated n a specal global curvlnear coordnate system that s straght n the transverse drecton [8, 9]. Ths formalsm has the same conservatve terms as n the Cartesan formulaton. We also consder the specal dffculty n smulatng granular avalanche flows, that s, the yeldng/stoppng propertes of granular materals [10 13]. Approprate statc resstance condton and stoppng crteron must be taken account nto the soluton procedure n order to model the yeldng/stoppng propertes. As depth averaged shallow flow models are n most stuatons hyperbolc equatons that allow for dscontnuous solutons, some well-known Godunov type methods such as Harten Lax van Leer (HLL), HLL contact (HLLC) (c.f. [14]), and wave propagaton schemes have been used n the lterature [3, 7, 15]. Other classes of hgh-resoluton shock-capturng methods such as nonoscllatory central schemes [16] were also employed [8, 9]. Wang et al. [8] demonstrated that the non-oscllatory central scheme wth Mnmod lmter or van Leer lmter was very sutable to handle the problem of avalanche dynamcs. Patra et al. [15] employed Davs s second-order predctorcorrector Godunov type method [17] wth van Leer s monotone upstream-centered schemes for conservaton laws (MUSCL) reconstructon (c.f. [14]) and HLL scheme to solve the Denlnger Iverson formulaton [4, 7]. However, the statc resstance condton and the stoppng crteron are mssng n ther numercal procedure. In ths paper, we add the statc resstance condton and the stoppng crteron to Davs s Godunov type method to smulate the yeldng/stoppng propertes of granular avalanche flows. Furthermore, we mplement graphcs processng unt (GPU) computng and test the correctness and effcency of the resultng code n several numercal examples. The SH model s D and ts numercal soluton tme s generally not long. However, for engneerng desgn and hazard rsk analyss, a large number of smulatons wth hgh grd resolutons must be completed n short tmes, thus necesstatng parallel computng. Wth the avalablty and development of NVIDIA s compute unfed devce archtecture (CUDA, for ntroducton, see [18]) as a programmng model, GPU computng has become one of the most popular choces for parallel computng because of GPU s massve parallel processng power and low energy consumpton per Gflops. CUDA s based on C language, thus makng nowadays general users learn easly. Owng to ts character of many-core and faster memory access, a GPU can speedup massvely data-parallel codes many tmes relatve to a central processng unt (CPU) of the same generaton. Furthermore, performance benefts can scale wth multple GPUs usng CUDA 4.0 above, OpenMP, or Pthread on a shared-memory computer nstalled wth several GPUs. There are numerous work on applyng GPU computng for computatonal flud dynamcs. Here, we only lsted some work related to shallow flow models [19 5]. Kuo et al. [19] presented applcaton of the splt HLL scheme for both the Euler and shallow water equatons to Tesla C1060 GPU (frst generaton general purpose GPU) usng CUDA and reported over 67 tmes speedup over an Intel Xeon X547 CPU core (3.0 GHz). Brodtkorb et al. [0] mplemented the second-order Kurganov-Petrova s central scheme for the shallow water equatons on NVIDIA GeForce GTX 480 graphcs card and combned the smulaton wth OpenGL vsualzaton on the same card. Sætra and Brodtkorb et al. [1, ]

3 SOLVING THE S H EQUATIONS WITH A SECOND-ORDER GODUNOV TYPE METHOD ON GPU 383 presented detaled multple-gpu mplementaton of a block-based AMR method for the same central scheme and a seres of performance and accuracy tests. In Ref. [1], they used the row decomposton of computatonal doman to put the transferred data n contnuous memory and the control thread-work thread strategy to enable mult-gpu computatons n a sngle node and performed tradtonal CUDA block decomposton wthn each GPU for further parallelsm. Ther mplementaton shows near perfect weak and strong scalng and enables smulaton of up to 35 mllon cells at a rate of over 1. ggacells per second usng four Ferm GPUs (second generaton). Vnas et al. [3] also presented mult-gpu mplementatons of a Roe type fnte volume method for shallow water equatons wth contamnaton transport usng MPI where the fastest example reached a speedup of 78 usng four M050 GPUs on an Infnband network wth respect to a parallel executon on a sx-core CPU. Smth and Lang [4] mplemented a Godunov-type MUSCL Hancock scheme for shallow water equatons on Ferm M075 GPU and sgnfcantly expedted the smulatons when compared to a tradtonal CPU approach. They also ponted out that there are sgnfcant localzed errors ntroduced by 3-bt floatng-pont precson. Vacondo et al. [5] parallelzed a well-balanced postve depth reconstructon fnte volume explct dscretzaton technque on GTX 580 and M070 GPUs for fast flood smulatons. In that work, they adopted a novel and effcent block deactvaton optmzaton procedure to ncrease the effcency of numercal soluton n the presence of wettng dryng fronts. Whle many researchers have appled GPU computng to the shallow water equatons for modelng flood events and obtaned good performance as mentoned earler, there are few smlar applcatons to granular avalanche flows. Ref. [] remarked that many technques used n GPU mplementaton for the shallow water equatons are transferable to other hyperbolc conservaton laws and numercal schemes. Therefore, t s worthwhle to apply GPU computng to the SH equatons for smulatng granular avalanche flows. In ths work, we use CUDA and a sngle GPU to mplement the numercal algorthm for the SH model. The paper s organzed as follows. In Secton, the SH model s formulated. In Secton 3, Davs s verson of the Godunov type method together wth the MUSCL reconstructon and HLLC scheme s gven, and the statc resstance condton and stoppng crteron are added to the numercal procedure. In Secton 4, a short ntroducton of CUDA mplementaton used s presented followed by a practce on allocatng global memory on GPU for D array for ease of code portng. In Secton 5, three numercal examples are used to verfy the correctness of the numercal model and test the performance of the GPU and CPU programs, and one granular jet example s used to demonstrate the applcaton of the developed code. The last secton gves the concluson. Fgure 1. The curvlnear coordnates Oxy s defned on the reference surface (dashed lnes) where the downslope nclnaton angle of the reference surface s a functon of the downslope coordnate x (reproduced from [8]).

4 384 J. ZHAI ET AL.. GOVERNING EQUATIONS In 1989, Savage and Hutter frst derved the SH equatons [] from the ncompressble Naver Stokes equatons by usng the depth-averaged approach. In the SH model, the granular materal s regarded as cohesonless Coulomb materal, that s, the shear stress only depends on the normal stress : D tan, where s the nternal frcton angle of the materal. The system of the equatons s of hyperbolc type akn to the shallow water equatons. There are varous forms of SH equatons accordng to dfferent coordnate systems chosen. Here, we use the same specal global curvlnear coordnate system gven n Ref. [8] as shown n Fgure 1, where there s no transverse varaton for the nclnaton angle n the y drecton. The correspondng non-dmensonal SH @x.hu/ D hu @y.huv/ D hs x; () hv C ˇyh D hs y ; (3) where h s the flow thckness n the drecton, and u and v are the depth-averaged velocty components n the x and y drectons, respectvely. ˇx and ˇy are defned as ˇx D " cos K x ; ˇy D " cos K y ; (4) where K x and K y are the earth pressure coeffcents n the x and y drectons accordng to the Mohr Coulomb yeld crteron, and " s aspect rato of the characterstc thckness to the characterstc downslope extent. The terms s x and s y represent the net drvng acceleratons n the x and y drectons, respectvely, s x D sn u jvj tan ı max 0; cos C u " ; (5) s y D v jvj tan ı max 0; cos C u " ; (6) where jvj D p u C v, ı s the basal Coulomb frcton angle, s the local curvature of reference surface, and s the local stretchng of the curvature. The shallow basal topography s defned by ts elevaton D b.x; y/ above the curvlnear reference surface as shown n Fgure 1. The earth pressure coeffcents are gven accordng to [6]: 0 K xact/pass K yact/pass D 1 s 1 1 cos A sec 1; (7) cos ı K x C 1 p.k x 1/ C 4 tan ı ; (8)

5 SOLVING THE S H EQUATIONS WITH A SECOND-ORDER GODUNOV TYPE METHOD ON GPU 385 where and ı are the nternal and bed frcton angles respectvely. The subscrpts act and pass denote actve ( ) and passve ( C ) stress states: ˆ< K xact ; K x > 0 ˆ: K xpass ; <0 ˆ< K yact > 0 K y D ˆ: K ypass : <0 3. NUMERICAL METHODS 3.1. Second-order Godunov type method Smlar to Ref. [15], a second-order predctor-corrector Godunov type method due to Davs [17] wth the MUSCL reconstructon for the conservatve varables U D.h; hu; hv/ T and HLLC numercal flux [14] s used. Defne the cell average: U ;j D 1 Udxdy: (11) x y j The method acheves second-order accuracy n tme and space by usng second-order predctorcorrector scheme and MUSCL reconstructon. To proceed, rewrte the system (1 C B@U D S.U/ ; where A and B are the Jacoban matrces of the physcal fluxes F and G, respectvely: 3 3 A 0 1 D 4 u C ˇxh u05 ; B uv v D 4 uv v u 5 : (13) v C ˇyh 0v The procedure of the predctor-corrector Godunov type method goes as follows. Step 1: Predctor step. Compute the soluton varables at the half tme level n C 1 by usng Taylor expanson and Equaton (1) (Lax Wendroff approach): U nc 1 ;j D U n t h ;j A n ;j x n n ;j C B n ;j y ;j S n ;j ; (14) where x ;j and y ;j are the lmted slopes of U n the x and y drectons, respectvely. The lmter here could use the Mnmod lmter x ;j D x ;j x ; x ;j D Mnmod.U ;j U 1;j ; U C1;j U ;j /: (15) Step : Data reconstructon. The left state U L C1= and the rght state of UR C1= at the cell nterface C 1= n the x drecton are constructed usng the pece-wse lnear MUSCL reconstructon for the conservatve varables obtaned n Step 1: U L C 1 D U nc 1 ;j C x x nc 1 ;j ; U R C 1 Smlarly, U L;R are reconstructed n the y drecton. j C1= D U nc 1 x C1;j x nc 1 C1;j : (16)

6 386 J. ZHAI ET AL. Step 3: Corrector step. Godunov type method s used to update the soluton, U nc1 ;j D U n t ;j x O F C 1 ;j OF 1 ;j where bf C 1 D bf 8 F bf ˆ< F HLLC U L ; U R D C 1 C 1 F t G O y ;jc 1 OG ;j 1 C ts nc1= ;j ; (17) U L ; U R s the numercal flux. We use the HLLC numercal flux C 1 C 1 ˆ: F U L C 1 U L C 1 U R C 1 U R C 1 ; S L > 0 C S L U L U L ;S C 1 C 1 L S C S R U R U R ;S C 1 C S R ; S R 6 0 : (18) The ntermedate varables n the start regon of the HLLC approxmate Remann solver are 3 n the x drecton, and U K D h K S K u K S K S U K D h K S K v K S K S 4 1 S v K 5 ; K D L; R; (19) u K 5 ; K D L; R; (0) S n the y drecton. The estmates for the left, mddle [4], and rght waves are S L D mn u L pˇlh L ;u R pˇrh R ; (1) S D 1.u L C u R / C pˇlh L pˇrh R ; () S R D max u L C pˇlh L ;u R C pˇrh R : (3) Ths Godunov type method s stable under the CFL condton CFL D t max 8;j ju ;j jc p.ˇxh/ ;j x In ths work, CFL number s taken to be 0.4. C jv ;jjc p.ˇyh/ ;j y! <1: (4) 3.. Statc resstance condton In some studes [8, 15], the Coulomb frcton law contaned n Eqs (5) (6) was used to determne the frcton force everywhere any tme. Ths s not correct when the granular mass s at statc condton: v D 0 or when the mass has lttle net drvng force [10, 11]. For the case v D 0, Ref. [1] gave a statc frcton calculaton method for one-dmensonal (1D) cases. Here, we extend the same treatment to D cases.

7 SOLVING THE S H EQUATIONS WITH A SECOND-ORDER GODUNOV TYPE METHOD ON GPU 387 Theoretcally, when the velocty s non-zero, that s, v 0, the frcton force s computed wth the Coulomb frcton law. When v D 0, the statc resstance begns to work. Under ths condton, the momentum Eqs () (3) reduce to 6 @t D Dx D y fx C ; (5) f y where the combned gravty topography pressure terms are D x D h sn h" and D y D h" @y. Here, we set K x D K y D 1 n ˇx and ˇy for statc cases the lterature. As the maxmum frcton that the materals can sustan s max D h tan ı, the statc resstance f D.f x ;f y / can be deduced from Equaton (5) as follows: q when v D 0 and Dx C D y < max; set f D D; when v D 0 and qdx C D y > max; set f D D jdj max: (6) 3.3. Stoppng crtera Stoppng crtera for movng granular materals are ntroduced based on a Coulomb threshold n a way smlar to Refs. [10, 11, 13]. The Coulomb threshold s max D h nc1 tan ı. After obtanng a tral soluton h nc1 ;. hu/ f nc1 ;. hv/ f nc1 by usng Equaton (17) wthout the source term, we set an admssble basal shear stress at cell and tme n C 1 as " Q T x QT y # nc1 D " fhu fhv # nc1 C t 4 sn h" h" 3 5 nc1 : (7) The stoppng crtera are nc1 When ˇ QT ˇ < max t, that s, the admssble basal shear stress s less than the Coulomb threshold value, and when jr.h C b/j < tan, the flow velocty.u; v/ nc1 s set to zero. Otherwse, solve for momentums.hu/ nc1 and.hv/ nc1 usng Equaton (17) wth the full source term that uses the Coulomb basal frcton law Flow front and half wet cell In ths artcle, we treat the flow front as Refs. [4, 15] dd. Take the x drecton as example. Suppose there s a flow front of whch the rght sde s where h D 0, then a left-gong rarefacton wave emanates from the front, and correspondng Remann nvarant s I L D uj L C c L D uj 0 C c 0 : (8) Near the flow front, c 0 approaches zero as h! 0, so the wet/dry front corresponds to the tal of the rarefacton and has exact propagatng speed uj 0 D uj L C c L. Ths speed s taken as the estmated rght-gong wave speed S R n the HLLC scheme. The exstence of dry beds s a partcular feature that must be properly addressed when desgnng numercal schemes for shallow water flows. In the ntal condtons, we set the granular thckness to zero where the debrs flow has not arrved. In order to ensure numercal stablty and avod spurous

8 388 J. ZHAI ET AL. dffuson of the flow front n updatng the problematc partal-wet cells, we defne a cell wth 0< h 6 D 10 6 as the partal-wet cell. The momentum equatons n the partal-wet cell are not solved and nstead the veloctes are extrapolated from the neghborng cell wth the largest h n a way smlar to Ref. [7]: ² 0; f h neghbor cells v D vj hmax ; else : (9) 4. GPU PARALLEL COMPUTING The most wdely used programmng model to mplement GPU computng s NVIDIA s CUDA [18]. A CUDA program ncludes two knds of codes, the seral codes and the parallel codes [8, 9]. The seral codes that run on the host (CPU) sde are responsble for varables declaraton, ntalzaton, data transmsson, and kernel nvocaton. The parallel codes (called kernel functons ) runnng on the devce (GPU) sde are executed n parallel by massve threads organzed to match the GPU hardware feature and to allow for mappng typcal data structures (arrays, matrces). A number of threads makes up a block, and many blocks make up a grd, whch s the counterpart of a kernel functon. The block may be organzed nto 1D, D, or 3D array of threads, whle the grd may be n 1D, D, or 3D array of blocks. D block and grd are llustrated n Fgure. Effcent GPU computng requres careful consderaton of parallelsm organzaton, memory management, and kernel schedulng [4]. In the present explct predctor-corrector Godunov type method, all the calculatons ncludng slope lmter, reconstructon, flux and source evaluaton, as well as soluton update could be run on GPU n parallel. They are executed n several dfferent kernel functons n order to keep each kernel small enough to use the fast regsters and shared memores effcently. Kernel global vod getlrud s used to calculate the predctor step and data reconstructon step, global vod RKresult s used to compute the corrector step ncludng flux and source evaluaton, and global vod Boundary s used to apply boundary condtons. Furthermore, a sngle tmestep t n Equaton (4) s used for all cells, whch s determned from the smallest permssble tmestep n the whole doman. It can be computed on GPU n a reducton kernel ( global vod getdeltat), whch utlzes the shared memory and round query strategy [30]. Ths process s shown n Fgure 3. As conventon, we map each mesh cell to a thread, and each thread controls one or several mesh cells dependng on whether the total number of computatonal cells Fgure. Sketch of organzaton of a compute unfed devce archtecture (CUDA) program.

9 SOLVING THE S H EQUATIONS WITH A SECOND-ORDER GODUNOV TYPE METHOD ON GPU 389 Fgure 3. Tmestep reducton process n GPU. Fgure 4. Allocaton of a second-rank ponter n CPU. Fgure 5. Allocaton of a second-rank ponter d_h wth the help of auxlary varable dev_h on GPU. Fgure 6. Copy data from GPU to CPU. exceeds the maxmum number of threads allowed by the GPU capacty. In the frst box n Fgure 3, every thread computes the maxmum egenvalue of ts own. In the second box, every block allocates a shared memory, all threads of a block load ther values nto the shared memory, then carry out bnary comparson, and the resultng maxmum value s stored at the frst thread of the block. In the thrd box, the frst thread of each block loads the block s max-value nto a global array, durng whch atom addton operator s used to record the number of blocks that have done the data load and mark the fnshng of data load. In the fourth box, all threads n the last block make bnary comparson n the data n the global array and gets the maxmum value of all cells. We then use ths value to compute the tme step from Equaton (4). A lot of GPU programmers wll use 1D memory allocaton, whch can make addressng and memory copy between CPU and GPU fast. In ths work, n order to make the addressng drect n the GPU program, we use D memory allocatng for soluton varables. In C++, the D memory allocaton of a second-rank ponter h on CPU s va new double (Fgure 4). Now, the CPU memory can be used drectly for the ponter h. But we could not use cudamalloc to allocate a second-rank ponter on GPU n the way smlar to new double n Fgure 4, because cudamalloc can only allocate a frst-rank ponter. In ths work, n order to use a second-rank ponter d_h on GPU, we allocate memores for the second class (pontng to ponter) of d_h and for the frst class (pontng to double number) of a temporary second-rank ponter dev_h on GPU(lne 3 and 5 n Fgure 5, respectvely), whle the second class of dev_h s allocated on CPU (lne ). We then copy the memory of the second class of dev_h from CPU to that of d_h on GPU, as shown n the last lne of Fgure 5. In ths way, the frst-class ponter of d_h s assocated wth that of dev_h on GPU. Now, we can use the second-rank ponter d_h as a D array drectly n GPU, but when we want to copy the data of d_h back to CPU for output, we must copy each dev_hœ, whch ponts to N y double numbers allocated on GPU, nstead of d_hœ, whch s not explctly allocated on GPU, back to hœ (Fgure 6). Whle ths method of allocatng D GPU memory s convenent for portng the seral program to the GPU program, the speed of accessng GPU memory s expected to be slower than that of the 1D memory allocaton. Table I shows comparson of the computatonal tmes between 1D and D

10 390 J. ZHAI ET AL. Table I. Comparson of the runnng tmes for matrx multplcaton C D A B. n 1D allocaton D allocaton D(seral code) ms 8. ms 10 ms ms 3.4 ms 140 ms ms 109. ms 1340 ms ms 716. ms 1040 ms memory allocaton methods to compute C D A B, wherea; B and C are n n matrces. The results were obtaned n double precson usng an NVIDIA Tesla C075 GPU wth 6 GB memory and an Intel Xeon Westmere GHz 6-core CPU wth 4 GB memory. Only global memory s used n the code. It can be seen from Table I that D memory allocaton method s much slower than the 1D counterpart, although t acheves consderable speedup relatve to D allocaton seral code for large n. Ths ndcates that 1D memory allocaton s desrable for effcent GPU computng. However, the dfference may not be so large for a practcal applcaton code because other bottlenecks occur even usng 1D memory allocaton, as wll be seen from Table IV. The D memory allocaton technque presented here may be helpful n portng a seral code to a GPU code. 5. NUMERICAL EXAMPLES We use three examples to verfy the accuracy of the numercal model and compare the effcency of both CPU and GPU programs. Meanwhle, we compare the effcency of present GPU mplementaton wth earler GPU codes for modelng shallow water flows n the lterature. Fnally, we smulate a granular jet example to demonstrate the applcaton of the developed flow solver. The computer we used s a server wth an Intel Xeon Westmere 5675 (3.06 GHz) CPU and an NVIDIA Tesla C075 GPU. All computatons use double precson Dam break problem For the dam break problem [31], the computatonal doman s set to Œ 1:8; 1:8 Œ 1:6; 1:6 m, and the ntal condtons are h0 x<0 v D 0; h D 0 x > 0 ; (30) where h 0 D 10 m; D 40 o ; and ı D 4:5 o. The orgnal example s a 1D problem, but we calculate t wth D code and cut a slce from the plane y D 0 to compare wth the exact soluton. In order to get the exact soluton, we rewrte the SH equatons n dmensonalzed form as n Ref. [31] (we correct the sgn errors n [31]): h t C uh x C hu x D 0; u t C g cos h x C uu x D gcos.tan ı tan / : (31) Notce that the x drecton s downward along the slope. Wth the change of varables D x C 1 g cos.tan ı tan /t ; U D u C g cos.tan ı tan /t; (3) Equaton (31) reduces to a homogeneous system of equatons; then by utlzng the Rtter soluton of a frctonless dam break, we can obtan the analytcal soluton of a granular dam break problem [31]:

11 SOLVING THE S H EQUATIONS WITH A SECOND-ORDER GODUNOV TYPE METHOD ON GPU 391 Fgure 7. Exact and numercal solutons on two grd numbers for the dam break problem after 0.5 s. Table II. Errors of the computed depth and orders of accuracy for the dam break problem. Grd L error order L 1 error Order h 0 ;0/; < c 0 t ˆ< h.h; U/ D 0 ;! 9 c 0 t 3 t C c 0 ; c 0 t 6 6 c 0 t : (33) ˆ:.0; 0/; > c 0 t where c 0 D p gh 0 cos. Fgure 7 shows the computed depth n comparson wth the exact soluton after 0.5 s. We can observe some dfferences only at the rght flow front and the left tal of the rarefacton wave. However, on the fner grd 51 3, the dfference s further reduced. In Table II, we gve numercal errors of the computed depth h relatve to the exact soluton. The errors are reduced wth ncreasng grd szes, but the numercal order s not second order. It may be related to the ntal data jump, the lmter, the numercal flux, the treatment of the partal wet cells, and so on. However, the numercal order s compatble wth other general convergence rates for smlar test models (e.g., [10]). 5.. Quescent equlbrum and start/stop flow condtons Ths example s used to llustrate the role of the stoppng crtera and statc resstance n smulatng quescent equlbrum and start/stop flow condtons. Accordng to Ref. [31], we gve the bed level b and the ntal free surface level H D b C h as b D p x C y ; H D b C h D 0: b: (34) Because h s non-negatve, the ntal granular mass only covers the regon where b.x; y/ 6 0:1. The computatonal doman s Œ 0:3; 0:3 Œ 0:3; 0:3 m, and the mesh used s 5656.Inthefrst case, we set ı D D 40 o. As the free surface slope angle of the mass s less than the nternal frcton angle, whch decdes the maxmum permssble value of the ple slope, the ntal ple wll keep ts statc state all the tme. In Table III, we gve numercal errors of the computed depth relatve to the

12 39 J. ZHAI ET AL. Table III. Errors of the computed depth and orders of accuracy for the statc ple problem at ı D 40 o, t D 1 s. Mesh L error Order L 1 error Order Fgure 8. The counter lnes of depth wth ı D 40 o and t D 0:4 s (left), t D 1 s (rght). Fgure 9. The counter lnes of depth wth ı D 0 o at t D 0:1 sandt D 4 s. exact soluton on varous mesh szes. The numercal orders of accuracy are mproved compared wth Table II. Fgure 8 shows the computed counter lnes of the depth at t D 0:4 sandt D 1 s, we can observe the mass at t D 1:0 s s bascally the same as at t D 0:4 s, wth a lttle smearng n the center. In the second case, we set ı D D 0 o. As the frcton angle s smaller than the ntal free surface slope angle, the mass begns to flow, and Fgure 9 shows the counter lne dstrbutons at t D 0:1 sandt D 4 s. The dstrbuton after t D 4 s wll reman unchanged as the flow has reached statc balance.

13 SOLVING THE S H EQUATIONS WITH A SECOND-ORDER GODUNOV TYPE METHOD ON GPU Inclned plane mergng contnuously nto a horzontal plane In ths subsecton, we present a smulaton example of fnte granular mass sldng down an nclned plane and mergng contnuously nto a horzontal plane [8]. The computatonal doman s x Œ0; 30 ; y Œ 7; 7 (dmensonless). The rato " and the stretchng parameter n Equaton (5) are both set to 1 n ths work. The nclned regon les x Œ0; 17:5, and the horzontal regon les x > 1:5. The transton zone between them s smooth and the nclnaton angle s 8 0 ; 0 6 x 6 17:5; ˆ< x 17:5.x/ D 0 1 ; 17:5 < x < 1:5; 4 ˆ: 0 o ; x > 1:5; (35) where 0 D 35 o.wesetı D D 30 o. An hemsphercal shell wth radus of r 0 D 1:85 centered at.x 0 ;y 0 / D.4; 0/ s released suddenly at t D 0. Fgure 10 shows the thckness contours of the flud at nne dfferent tmes as computed on meshes wthout usng the stoppng crtera so as to compare wth the results n Ref. [8]. The results at t D 6; 15, and 4 are comparable. The quanttatve dfference between present and Wang s results may attrbute to dfferent schemes, dfferent mesh szes, and dfferent " used. We may compare the dfference between wth and wthout the stoppng crtera for later stage of the avalanche. Fgure 11 shows results at t D 40 on three gradually refned meshes. The ple becomes wder as the mesh s refned. For ths partcular example, the dfference between wth and wthout the stoppng crtera s small. Fgure 10. Thckness contours of the flow at nne tmes as computed on meshes wthout the stoppng crtera. The last row s Wang s results [8]. The quanttatve dfference may be caused by dfferent schemes and mesh szes and possbly dfferent " used (Ref. [8] dd not gve mesh numbers and ").

14 394 J. ZHAI ET AL. Fgure 11. Comparson of calculated results between wth and wthout the stoppng crtera at t D 40 usng three consecutvely refned meshes..8 8/.30 14/ means there are 8 30 meshes n the x drecton and 8 14 meshes n the y drecton. Sold lnes denote results wth the stoppng crtera, whle dashed lnes denote wthout. Table IV. CPU and GPU tmes for run from t D 0 to t D 4 wth four meshes, where M 0 N 0 D Mesh CPU seral tme GPU tme (D) S p (D) GPU tme (1D) S p (1D) 4 4.M 0 N 0 / 130 ms ms ms M 0 N 0 / ms 3466 ms ms M 0 N 0 / ms ms ms M 0 N 0 / ms ms ms 5.4 1D, one-dmensonal memory allocaton ; D, two-dmensonal memory allocaton. We have also compared both CPU and GPU programs n ths example. The runnng tmes of both CPU and GPU programs are recorded n order to compare ther effcences. Defne the GPU/CPU speedup rato as S p D tme of CPU seral program tme of GPU program : (36) Tmngs for run from t D 0 to t D 4 wth four dfferent meshes are gven n Table IV. It can be seen that wth the ncrease of the mesh number, the speedup rato ncreases. Ths can be explaned as follows. In the GPU program, there are some parts that need to be seralzed. Hence, the runnng tme s composed of the seral tme and the parallel tme. Wth the ncrease of the mesh number, the parallel computaton work runnng on GPU wll ncrease and fnally saturate because of GPU s parallel capacty, whle the seral work runnng on CPU and the memory copy work wll not ncrease much. Therefore, the speedup rato wll ncrease untl a maxmum value s attaned. Table IV also compares the D and 1D memory allocatons. Unlke n prevous matrx multplcaton case, the 1D GPU memory allocaton code s only a bt faster than the D memory allocaton code for ths case. As to performance comparson wth prevous work, the best specfc tme (.e., tme per cell per tme step) of the present mplementaton s approxmately 5:10 8 s per cell per tme step (double precson) wth 1D memory allocaton on Tesla C075 for ths D example (Table V). Ths rate s a bt slower than but comparable to other CUDA GPU mplementatons for the shallow water equatons, for example, s per cell per tme step (n double precson) on Tesla M075 n Ref. [4], and 1: : 10 9 s per cell per tme step on Tesla M070 n Ref. [5]. Because the SH equatons have extra calculatons for K x ;K y, and stoppng crtera compared wth

15 SOLVING THE S H EQUATIONS WITH A SECOND-ORDER GODUNOV TYPE METHOD ON GPU 395 Table V. Specfc tmes for two-dmensonal and one-dmensonal memory allocatons wth four meshes, where M 0 N 0 D Mesh D specfc tme(s/cell/step) 1D specfc tme(s/cell/step) 4 4.M 0 N 0 / 1: : M 0 N 0 / 1: : M 0 N 0 / 9: : M 0 N 0 / 8: : D, one-dmensonal memory allocaton; D, two-dmensonal memory allocaton. Fgure 1. Expermental apparatus [3]. The granular jet mpnges on the nclned plane, spreads nto a regon of thn, fast flow (A), and then passes through a jump, becomng thcker and slower flow (B). the shallow water equatons, and dfferent papers had used dfferent numercal schemes and dfferent treatment of dry-wet fronts, a precse comparson of computatonal effcency s out of the scope of the present paper Granular jet Johnson and Gray [3] conducted a seres of experments of granular jet mpngng on an nclned plane. Accordng to ther results, there are steady-state teardrop shock, blunted shock, and unstable tme-dependent flows that are dependent on the nclned angle and the heght of the release poston of the jet relatve to the plane H f, as sketched n Fgure 1.

16 396 J. ZHAI ET AL. We smulate ths problem wth the D memory allocaton GPU program by gvng the ntal condtons n the crcular regon centered at the stagnaton pont on the plane as Ref. [3]. In the crcular regon wth a chosen radus R mp D R, the analytcal soluton [3] v D u jet cos ;u jet sn ; (37) h D R cos 3 r.1 sn cos / ; (38) s vald, where R s the radus of the cylndrcal jet,.r; / s the polar coordnates centered at the stagnaton pont, and u jet s the jet velocty. We use the same parameters D 6:7 o, u jet D 0:99 m/s, and R 0:5D D 7:5 mm as n Ref. [3] for the steady case wth a teardrop-shaped shock. If usng a constant basal frcton coeffcent, one can only obtan one steady flow of unform thckness at a sngle slope angle of D tan 1. But experments [33] have observed steady unform flows over a range of slope angles over a roughened bed. Ths motvated Johnson and Gray [3] to use a varable basal frcton law to model the experments. In 1999, Poulquen et al. [34] demonstrated that a steady flowng layer wth unform thckness can exst on a slope nclned at an angle wth a mnmum depth h Q stop./, below whch steady flow s not observed, here h Q denotes dmensonal flow depth. They also found an emprcal dependence of the rato of flow depth h Q to Qh stop./ on the Froude number, Fr jvj Qh p D ˇ h" cos Qh stop./ ; (39) where ˇ D 0:136 s a measured constant for glass beads. At steady flowng states, D tan holds, and the nverse functon of Q h stop./ together wth Equaton (39) can lead to an equaton for the frcton coeffcent stop D tanœ stop. Q hˇ=fr/, mplyng that stop D stop.h 0 /. Smlar to Ref. [3], we use the form for the functon stop.h 0 / as gven Ref. [34] and the form for the functon start.h 0 / as gven n Ref. [33], respectvely, wth expermentally measured frcton angles, stop.h 0 h0 / D tan 1 C.tan tan 1 / e ' ; (40) start.h 0 / D tan 3 C.tan tan 1 / 1 1 C h0 ' ; (41) where 1 D 1 o ; D 30:7 o ; and 3 D : o are measured frcton angles [33]. The functon stop.h 0 / s a ft to the expermental measurements of Q h stop./, and t takes the aforementoned form wth a transton between two frcton angles 1 and. The functon start.h 0 / s a statc frcton coeffcent and s calculated by measurng the maxmum nclnaton angle at whch a unform layer of statonary materal wth depth h 0 starts to move. The free parameter ' has the dmenson of length and depends on the granular materal and surface propertes of the plane and characterzes the depth of flow over, whch a transton between the two frcton angles 1 and occurs. In order to make the steady state ft the experment better, we take the free parameter ' D 6 mm n ths paper. The functons stop and start areusednthefollowngway. When Fr >ˇ, that s, for the steady regme where Q h> Q h stop, the frcton law (40) s used as D stop! hˇ Q : (4) Fr

17 SOLVING THE S H EQUATIONS WITH A SECOND-ORDER GODUNOV TYPE METHOD ON GPU 397 (a) present result (b) smulaton [3] (c) experment wth sand [3] Fgure 13. Comparson of (a) present result wth 5156, where the contours are of flow depth and shadng ndcates the flowng regon of materal, (b) smulaton by Johnson and Gray [3], where the contours are of flow depth at ntervals of 1 mm, wth dark contours at ntervals of 5 mm and shadng ndcates the flowng regon of materal, and (c) expermental results. Unt n x;y tcks n (a) and (b) s meter, whle grd squares n (c) are -cm ntervals. Dfferent from (b), n (a), we do not plot contours nsde the crcular regon wth radus R mp representng the granular jet, where the analytcal solutons (37)-(38) are mposed. When 0<Fr <ˇ, we follow the method [33] n nterpolatng between the statc and steadyflow frcton coeffcents wth a power functon D Fr stop. h/ ˇ Q start. h/ Q C start. h/; Q (43) where D 10 3 : The earth pressure coeffcents are set to unty n ths example as done n Ref. [3]. Fgure 13 shows the steady-state depth contours for a teardrop-shaped shock flow obtaned wth the present GPU code on mesh cells n comparson wth the numercal smulaton and expermental vsualzaton n Ref. [3]. We can clearly observe that the contour lnes, the shape and poston of the shock, and the statonary regon are comparable to the lterature. However, present shock s slghtly longer than that n [3], whch may be caused by dfferent values of the free parameter ' used. 6. CONCLUSION In ths work, we have presented an mproved numercal method for the SH model for modelng granular avalanche flows by takng nto account the statc resstance condton and the stoppng crtera. We apply Davs s second-order predctor-corrector Godunov method n conjuncton wth the MUSCL reconstructon and HLLC scheme to GPU computng. The accuracy of the resultng numercal algorthm s verfed through several typcal numercal examples and the computatonal speedup of the GPU code relatve to the CPU seral code s shown to be evdent. Comparsons between usng 1D and D memory allocatons and wth prevous GPU codes for the shallow water equatons are also represented. As an applcaton example, one case wth teardrop-shaped shock n the granular jet experments of Johnson and Gray s smulated, and the computed results are comparable to expermental and numercal results n the reference.

18 398 J. ZHAI ET AL. ACKNOWLEDGEMENTS The authors thank the support of state key program for developng basc scences (010CB731505) and Natural Scence Foundaton of Chna ( , ). REFERENCES 1. Hutter K, Rajagopal KR. On flows of granular materals. Contnuum Mechancs and Thermodynamcs 1994; 6(): Savage SB, Hutter K. The moton of a fnte mass of granular materal down a rough nclne. Journal of Flud Mechancs 1989; 199(1): Iverson RM, Denlnger RP. Flow of varably fludzed granular masses across three-dmensonal terran: 1. Coulomb mxture theory. Journal of Geophyscal Research 001; 106(B1): Denlnger RP, Iverson RM. Flow of varably fludzed granular masses across three-dmensonal terran:. Numercal predctons and expermental tests. Journal of Geophyscal Research 001; 106(B1): Ptman EB, Le L. A two-flud model for avalanche and debrs flows. Phlosophcal Transactons of the Royal Socety A: Mathematcal, Physcal and Engneerng Scences 005; 363(183): Bouchut F, Westdckenberg M. Gravty drven shallow water models for arbtrary topography. Communcatons n Mathematcal Scences 004; (3): Denlnger RP, Iverson RM. Granular avalanches across rregular three-dmensonal terran: 1. Theory and computaton. Journal of Geophyscal Research 004; 109:14. DOI: /003JF Wang YQ, Hutter K, Pudasan SP. The Savage-Hutter theory: a system of partal dfferental equatons for avalanche flows of snow, debrs, and mud. ZAMM-Journal of Appled Mathematcs and Mechancs/Zetschrft für Angewandte Mathematk und Mechank 004; 84(8): Ta Y-C, Noelle S, Gray JMNT, Hutter K. Shock-capturng and front-trackng methods for granular avalanches. Journal of Computatonal Physcs 00; 175(1): Mangeney-Castelnau A, Vlotte JP, Brsteau MO, Perthame B, Bouchut F, Smeon C, Yernen S. Numercal modelng of avalanches based on Sant Venant equatons usng a knetc scheme. Journal of Geophyscal Research 003; 108(B11):17. DOI: /00JB Mangeney-Castelnau A, Bouchut F, Vlotte JP, Lajeneusse E, Aubertn A, Prull M. On the use of Sant Venant equatons to smulate the spreadng of a granular mass. Journal of Geophyscal Research 005; 110:17. DOI: /004JB Fan YY, Wang SJ, Wang EZ. Characterstcs of statc and dynamc resstance of one-dmensonal debrs flow and ts numercal smulaton. Journal of Engneerng Geology 010; 18(6): (In Chnese). 13. Maeno F, Hogg AJ, Sparks R, Matson GP. Unconfned slumpng of a granular mass on a slope. Physcs of Fluds 013; 5:0330. DOI: / Toro EF. Remann Solvers and Numercal Methods for Flud Dynamcs: A Practcal Introducton. Sprnger: Dordrecht Hedelberg London, New York, Patra AK, Bauer AC, Nchta CC, Ptman EB, Sherdan MF, Bursk M, Rupp Bc, Webber A, Stnton AJ, Namkawa LM, Renschler CS. Parallel adaptve numercal smulaton of dry avalanches over natural terran. Journal of Volcanology and Geothermal Research 005; 139(1): Kurganov A, Tadmor E. New hgh-resoluton central schemes for nonlnear conservaton laws and convecton dffuson equatons. Journal of Computatonal Physcs 000; 160: Davs SF. Smplfed second order Godunov-type methods. Socety of Industral and Appled Mathematcs (SIAM) Journal on Scentfc and Statstcal Computng 1998; 9: CUDA: reference manual, verson 4.0.: Nvda, Santa Clara, CA, Kuo F-A, Smth M, Hseh C-W, Chou C-Y, Wu J-S. GPU acceleraton for general conservaton equatons and ts applcaton to several engneerng problems. Computers & Fluds 011; 45: Brodtkorb A, Sætra M, Altnakar M. Effcent shallow water smulatons on GPUs: mplementaton, vsualzaton, verfcaton, and valdaton. Computers & Fluds 01; 55: Sætra M, Brodtkorb A. Shallow water smulatons on multple GPUs. In Appled Parallel and Scentfc Computng, Jónasson K (ed.)., Vol of Lecture Notes n Computer Scence. Sprnger: Berln Hedelberg, 01; Sætra M, Brodtkorb A, Le K-A. Effcent GPU-mplementaton of adaptve mesh refnement for the shallow-water equatons. Journal of Scentfc Computng 014; 60:1 6. DOI: /s Garca-Rodrguez J, Vnas J, Loberas J, Fraguela B, Arenaz M, Amor M, Castro M, Doallo R. A mult-gpu shallowwater smulaton wth transport of contamnants. Concurrency and Computaton: Practce and Experence 013; 5(8): Smth LS, Lang QH. Towards a generalsed GPU/CPU shallow-flow modellng tool. Computers & Fluds 013: Vacondo R, Palù A, Mgnosa P. GPU-enhanced fnte volume shallow water solver for fast flood smulatons. Envronmental Modellng & Software 014; 57: Hutter K, Segel M, Savage SB, Nohguch Y. Two-dmensonal spreadng of a granular avalanche down an nclned plane part I. Theory. Acta Mechanca 1993; 100(1 ): Bradford SF, Sanders BF. Fnte-volume model for shallow-water floodng of arbtrary topography. Journal of Hydraulc Engneerng 00; 18(3):89 98.

19 SOLVING THE S H EQUATIONS WITH A SECOND-ORDER GODUNOV TYPE METHOD ON GPU Zhang S, Chu YL. GPU hgh performance computng wth CUDA, 009. Chnese Hydrology Pub. Hsouse, (In Chnese). 9. Sanders J, Kandrot E. CUDA by Example: An Introducton to General-Purpose GPU Programmng. Addson-Wesley Professonal: Boston, USA, Lang S, Lu W, Yuan L. Solvng seven-equaton model for compressble two-phase flow usng multple GPUs. Computers & Fluds 014; 99: Juez C, Murllo J, García-Navarro P. D smulaton of granular flow over rregular steep slopes usng global and local coordnates. Journal of Computatonal Physcs 013; 55: Johnson CG, Gray JMNT. Granular jets and hydraulc jumps on an nclned plane. Journal of Flud Mechancs 011; 675(1): Poulquen O, Forterre Y. Frcton law for dense granular flows: applcaton to the moton of a mass down a rough nclned plane. Journal of Flud Mechancs 00; 453: Poulquen O. Scalng laws n granular flows down rough nclned planes. Physcs of Fluds 1999; 11: