Doman decomposton n shallow-water modellng for practcal flow applcatons Mart Borsboom 1, Menno Genseberger 1, Bas van t Hof 2, and Edwn Spee 1 1 Introducton For the smulaton of flows n rvers, lakes, and coastal areas for the executve arm of the Dutch Mnstry of Infrastructure and the Envronment the shallow-water solver SIMONA s beng used [1]. Applcatons range from operatonal forecastng of floodng of the Dutch coast [3] and bg lakes [7], to the assessment of prmary water defences (coast, rvers, and lakes). These applcatons requre a robust and effcent modellng framework wth extensve modellng flexblty and good parallel performance. About two decades ago, a parallel mplementaton of SIMONA was developed [10, 11] based on doman decomposton wth maxmum overlap. In the same perod, non-overlappng doman decomposton wth optmzed couplng was consdered for Delft3D-FLOW [2], a shallow-water solver that s numercally very smlar to SIMONA. More recently, deas of the latter were adapted for ncorporaton n SIMONA for enhanced modellng flexblty and parallel performance. Ths wll be the subject of the present paper. The paper s organzed as follows. The numercal approach for modellng shallow-water flow as mplemented n SIMONA s outlned n secton 2. In secton 3 we show how doman decomposton has been ncorporated and whch refnements have been made. The parallel performance of the modfed method s llustrated n secton 4 for two practcal flow problems from cvl engneerng. 2 ADI-type shallow-water solvers The shallow-water equatons consst of a depth-ntegrated contnuty equaton and two horzontal momentum equatons. Vertcal momentum s replaced by the hydrostatc pressure assumpton,.e., the vertcal varaton of the pressure s assumed to depend solely on hydrostatc forces as determned by the poston of the free surface. For the numercal soluton of the shallow-water equatons SIMONA apples a so-called alternatng drecton mplct (ADI) method to ntegrate the equatons numercally n tme, usng an orthogonal staggered grd wth horzontal curvlnear coordnates ξ and η [1]. 1 Deltares, Delft, The Netherlands, e-mal: {Mart.Borsboom}{Menno.Genseberger}{Edwn.Spee}@deltares.nl 2 VORtech Computng, Delft, The Netherlands, e-mal: bas.vanthof@vortech.nl 1
2 Mart Borsboom, Menno Genseberger, Bas van t Hof, and Edwn Spee In the ADI method, each tme step s splt n two stages of half a tme step. In the frst stage, the water-level gradent s taken mplctly n the ξ -momentum equaton and explctly n the η-momentum equaton. The mass fluxes n the contnuty equaton are taken mplctly/explctly n ξ - and η-drecton as well, allowng the mplct terms to be combned to uncoupled trdagonal systems of equatons n ξ - drecton for the water level at the ntermedate tme level. In contrast, the evaluaton of the horzontal convecton terms and vscosty terms are respectvely explct and mplct n the ξ - and η-momentum equaton. In the second stage of the tme step, the mplct and explct dscretsatons are nterchanged. For stablty, dervatves n vertcal drecton and the bottom frcton term are always ntegrated mplctly. The ADI method requres the use of farly small tme steps to avod excessve splttng errors: u t O(1), v t O(1), x ξ x η gh t O(10), and x ξ gh t x η O(10). (1) Here, x ξ, x η are the grd szes and u, v the veloctes n the two horzontal curvlnear coordnate drectons ξ and η, t s the tme step, h the local water depth, and g the acceleraton due to gravty ( gh s the shallow-water wave celerty). Because of the condtons (1), the dscretzed equatons to be solved have a farly hgh dagonal domnance horzontally. Ths enables the use of sem-explct teratve methods horzontally, such as red-black Jacob to solve mplct convecton and vscosty. For the same reason, horzontal doman decomposton wth explct couplng, f desgned properly, can be very effcent. We remark that n the vertcal drecton grd szes x ξ, x η are used and the systems of equatons are much stffer. Vertcal dervatves are therefore always ntegrated mplctly n tme. 3 Doman decomposton technques for ADI-type shallow-water solvers About two decades ago, a parallel mplementaton of SIMONA was developed [10, 11] usng a mult-doman verson of the ADI method wth Drchlet-Drchlet couplng and maxmum overlap to ensure fast convergence. Ths approach s stll appled n the 2006 verson of SIMONA. Later on, for modellng flexblty, the possblty to use dfferent grd resolutons per subdoman has been ntroduced. For such a stuaton t s not that easy to deal wth an overlap between subdomans. Therefore, the overlap was removed. Ths concerns the overlap of the physcal area of the subdomans,.e., the area contanng the nner grd cells. For the mplementaton of boundary condtons and couplng condtons, vrtual grd cells were added outsde the physcal areas along boundares and DD nterfaces. So although the subdomans do not overlap, the subdoman grds do. Unfortunately, a Drchlet- Drchlet couplng wth mnmal overlap (only the vrtual grd cells overlap) has a very slow rate of convergence. See also panel (b) of Fg. 1. By re-usng deas from a non-overlappng doman decomposton approach wth optmzed couplng
Doman decomposton n shallow-water modellng for practcal flow applcatons 3 for Delft3D-FLOW [2], the good convergence behavor has been restored. Ths approach s mplemented snce 2010 n SIMONA. To llustrate how convergence errors due to doman decomposton propagate from one subdoman to another n a mult-doman ADI-type shallow-water solver, we consder a unform grd of sze x ξ, a unform depth h, and assume a small surface elevaton ζ and flow velocty u. The mplct systems n the ξ -drecton at the frst half tme step from t n to t n+1/2 are then of the form (dscretzed contnuty equaton and momentum equaton): ζ n+1/2 ζ n +h un+1/2 +1/2 un+1/2 1/2 x ξ =..., u n+1/2 +1/2 un +1/2 t/2 +g ζ n+1/2 +1 ζ n+1/2 =.... x ξ t/2 (2) At the second half tme step from t n+1/2 to t n+1, equatons n η-drecton ( j-ndex) are obtaned. By elmnatng u n+1/2 +1/2, the two equatons (2) can be combned to: ( ) ζ n+1/2 CFL 2 ζ n+1/2 +1 2ζ n+1/2 + ζ n+1/2 1 =..., (3) wth CFL number CFL = gh t/(2 x ξ ). To study the behavor of (3) n a DD framework, we consder the homogeneous equaton that s satsfed by the DD convergence error δζ n+1/2,m = ζ n+1/2,m ζ n+1/2, wth ζ n+1/2 the soluton that s sought and ζ n+1/2,m ts teratvely determned approxmaton at teraton m: δζ n+1/2,m CFL 2( ) δζ n+1/2,m +1 2δζ n+1/2,m + δζ n+1/2,m 1 = 0. (4) The nhomogeneous perturbaton of δζ n+1/2,m comes from the boundares of the subdomans where nformaton s updated explctly (Schwarz algorthm). Equaton (4) determnes how that nformaton spreads across a subdoman and reaches the opposte subdoman boundary. Ths becomes clear from the soluton of (4), whch s of the form: δζ n+1/2,m = C LR λ +C RL λ, (5) wth λ = (CFL 2 +1/2 CFL 2 + 1/4)/CFL 2. The soluton conssts of the superposton of two modes: one decayng from left to rght and one decayng from rght to left. Panel (a) n Fg. 1 llustrates ths for a subdoman of 8 grd cells at CFL = 2 (green), CFL = 5 (red), and CFL = 10 (blue). For CFL 1, we have λ 1/CFL 2. At such a hgh decay rate per grd cell, whch s due to the large dagonal domnance of (4), a Drchlet-Drchlet couplng s effcent. For CFL 1, however, we have λ 1 CFL 1 and hence a much lower decay rate. A Drchlet-Drchlet couplng s then not effcent anymore, unless a large overlap s used to compensate for the low decay rate. Ths s llustrated n panel (b) and (c) of Fg. 1. A much larger DD convergence speed s obtaned by only transferng from left to rght (rght to left) the nformaton that s movng n that drecton. Ths s realzed by the couplng:
4 Mart Borsboom, Menno Genseberger, Bas van t Hof, and Edwn Spee (a) (b) (c) (d) Fg. 1 Behavor of convergence error δζ n+1/2,m n subdomans consstng of 8 nner grd cells (whte) and 1, 2, or 3 added vrtual grd cells (grey) that overlap wth nner grd cells of neghbourng subdomans: (a) nsde a subdoman at CFL = 2 (green), CFL = 5 (red), and CFL = 10 (blue); (b) across 3 subdomans at CFL = 5 wth Drchlet boundary condton left, Neumann boundary condton rght, and multplcatve Schwarz Drchlet-Drchlet couplng wth mnmal overlap n between (red, blue, green ndcate subsequent DD teratons); (c) enhancement of DD convergence wth Drchlet-Drchlet couplng when usng a larger overlap (ncreasngly longer dotted lnes ndcate error reducton for 1-, 2-, and 3-cell overlap); (d) across 3 subdomans wth optmzed multplcatve Schwarz based on the decomposton of the convergence error (red lnes) n ts two soluton modes (blue and green lnes), cf. (5). Note that n (b, c) the arrows ndcate the transfer of Drchlet values from an nner grd cell to a vrtual grd cell; n (d) the arrows ndcate the transfer of optmzed couplng nformaton from nterface to nterface. (CFL+1/2)δζ n+1/2,m+1 R (CFL 1/2)δζ n+1/2,m+1 R +1 = (CFL+1/2)δζ n+1/2,m L 1 (CFL 1/2)δζ n+1/2,m L, (6) wth R the ndex of the left vrtual grd cell of the subdoman rght of the DD nterface under consderaton, and wth L the ndex of the rght vrtual grd cell of the subdoman left. Notce the explct nature of the couplng: the soluton of doman L at prevous teraton m determnes the value (rght-hand sde of (6)) of the condton to be mposed at the left boundary of doman R durng next teraton m+1 (left-hand sde of (6)). An equvalent procedure s used for the transfer of couplng nformaton n the other drecton, from doman R to doman L.
Doman decomposton n shallow-water modellng for practcal flow applcatons 5 Panel (d) of Fg. 1 llustrates the hgh DD convergence rate that can be obtaned wth an optmzed couplng; the convergence speed s about as hgh as would be obtaned wth a Drchlet-Drchlet couplng wth maxmum overlap (of half a subdoman, cf. panel (c)). However, because of the overlap, the amount of work per teraton n the latter would be twce as large. Furthermore, as mentoned before, t can not be combned easly wth local grd refnements for whch the grd cells n the overlap do not concde, contrary to the stuaton n panel (c). The fast DD convergence speed that for dagonally domnant problems can be obtaned wth an optmzed explct local DD couplng (optmzed Schwarz), and the lnk wth absorbng boundary condtons, s well known [8, 5, 4, 9, 6]. Because the splttng appled n the ADI method leads to ndependent 1D problems, we have the advantage that the optmzed couplng can not only easly be determned for constant x ξ and h, as we dd here, but also for the general case, by means of the LU decomposton of the resultng trdagonal systems that are of the form (3), but wth space- and tme-varyng coeffcents. The bdagonal L-matrces descrbe the decay of the soluton n ncreasng - (or j-) drecton. Ther last rows determne the combnatons of pars of ζ s at the subdoman nterface (one ζ n a vrtual grd cell, the other ζ n the adjacent nner grd cell) that do not specfy ths part of the soluton, and hence only specfy soluton modes decayng n decreasng - (or j-) drecton. Transferng these combnatons n decreasng - (or j-) drecton across DD nterfaces (the varable-coeffcent generalzaton of (6)) therefore ensures maxmum DD convergence speed. Lkewse for the bdagonal U-matrces and the exchange of couplng nformaton n the other drecton. 4 Applcatons There are many applcaton areas of SIMONA. Here we present two examples. Frst we show the effect of the optmzed couplng wthout overlap for a schematc model of the rver Waal n the Netherlands. Ths schematc model has a smple geometrc shape such that load balancng s straghtforward. Second we show the parallel performance of the approach for DSCM, a huge real-lfe hydrodynamc model n whch both load balancng and number of unknowns are an ssue. For the experments we consdered the followng hardware: H4 lnux-cluster at Deltares, nodes nterconnected wth Ggabt Ethernet, each node contans 1 AMD dual-core Athlon X2 5200B processor wth 2.7 GHz per core, H4+ lnux-cluster at Deltares, nodes nterconnected wth Ggabt Ethernet, each node contans 1 Intel quad-core 7-2600 processor wth 3.4 GHz per core and hyperthreadng (so effectvely 8 threads are used on 4 cores), and Lsa lnux-cluster at SURFsara, nodes nterconnected wth Infnband, each node contans 2 Intel quad-core Xeon L5520 processors wth 2.3 GHz per core. On the H4 lnux-cluster both the 2006 and 2010 verson of SIMONA were used. On the H4+ and Lsa lnux-cluster the 2010 verson of SIMONA was used. Recall
6 Mart Borsboom, Menno Genseberger, Bas van t Hof, and Edwn Spee (see secton 3) that the 2006 verson uses Drchlet-Drchlet couplng and maxmum overlap where the 2010 verson uses optmzed couplng wthout overlap. 4.1 Schematc model of rver Waal To study the effect of lowerng the groynes on desgn flood level, n [12] a schematsed rver reach was used that was based on characterstc dmensons of rver Waal n the Netherlands. Here, for the performance tests we wll use the detaled model of [12] n whch the groynes are represented as bed topography (see Fg. 2). The detaled model s a symmetrcal compound channel of 30 km length ncludng floodplan (wdth of 1200 m) and man channel (wdth of 600 m). We apply a depth averaged verson of SIMONA. The floodplan s schematsed wth grd cells of 2 m x 4 m and the man channel wth grd cells of 2 m x 2 m, resultng n more than 9 mllon unknowns. A tme step of 0.015 mnutes s used, resultng n 12000 tme steps for the 3 hour smulaton that we consder here for the performance tests. From Fg. 2 t can be observed that, n general, SIMONA scales well. Furthermore, on the H4 lnux-cluster the 2010 verson of SIMONA s about 20-30 % faster than the 2006 verson. Ths addtonal work can be explaned from the overlap n the 2006 verson whch s not n the 2010 verson (see secton 3). The dfference n performance for the 2010 verson of SIMONA on H4, Lsa, and H4+ lnux-cluster s because of the dfferent hardware. 4.2 Next generaton Dutch Contnental Shelf Model (DCSM) The current generaton of nested SIMONA models used for predctng water levels along the Dutch coast n an operatonal mode (see [3]) already requre hgh performance computng. At the Lsa lnux-cluster parallel performance of the 2010 verson of SIMONA was tested for a next generaton verson of the DCSM (North Sea and adjacent regon of the North Atlantc). Ths 3D (10 layer) hgher resoluton model ncludes salnty and temperature stratfcaton processes whch are essental for smulatng among others the spread of the freshwater Rhne plume along the Dutch coast. Ths new model requres a huge computatonal effort but smulaton tmes cannot ncrease for operatonal purposes. Although the North Sea model has an rregular geometry whch s not deal for scalablty, performance tests at Lsa showed lnear scalablty up to 100 processors. The left panel of Fg. 3 shows the parttonng of the doman n 96 subdomans of (about) the same number of grd cells that s obtaned by applyng orthogonal recursve bsecton (ORB). The rght panel shows the parallel performance on the Lsa lnux-cluster as a functon of the number of subdomans and cores, for parttonngs n strps and by means of ORB. The results show an optmal speed-up for the ORB parttonng and a small decay n performance for the larger strp decompostons. The latter s due to the shape of the strps. The strps become very thn wth wdths of less than a dozen grd cells as the number of domans ncreases, whch affects the valdty of the appled local couplng optmzaton.
Doman decomposton n shallow-water modellng for practcal flow applcatons 7 10 2 H4 lnux cluster, 2006 verson of SIMONA H4 lnux cluster, 2010 verson of SIMONA Lsa lnux cluster, 2010 verson of SIMONA H4+ lnux cluster, 2010 verson of SIMONA wall clock tme (hours) 10 1 10 0 10 0 10 1 10 2 computatonal cores Fg. 2 Schematc model of rver Waal: an excerpt of the model ncludng part of the floodplan (top), parallel performance for dfferent versons of SIMONA and on dfferent hardware (bottom). Acknowledgements We thank SURFsara (www.surfsara.nl) for ther support n usng the Lsa lnux-cluster.
8 Mart Borsboom, Menno Genseberger, Bas van t Hof, and Edwn Spee Fg. 3 DCSM. Left: parttonng of computatonal doman n 96 subdomans usng the orthogonal recursve bsecton (ORB) method. Rght: parallel performance on Lsa lnux-cluster for parttonngs n vertcal strps and ORB parttonngs. References 1. SIMONA WAQUA/TRIWAQ - two- and three-dmensonal shallow-water flow model. (2012) URL {http://apps.helpdeskwater.nl/downloads/extra/smona/ release/doc/techdoc/waquapublc/sm1999-01.pdf} 2. De Goede, E.D., Groeneweg, J., Tan, K.H., Borsboom, M.J.A., Stellng, G.S.: A doman decomposton method for the three-dmensonal shallow water equatons. Smulaton Practce and Theory 3, 307 325 (1995) 3. De Kleermaeker, S.H., Verlaan, M., Kroos, J., Zjl, F.: A new coastal flood forecastng system for the Netherlands. In: T. Van Djk (ed.) Hydro12 Conference, Rotterdam, The Netherlands, 2012. Hydrographc Socety Benelux (2012). URL {http://proceedngs.utwente. nl/246} 4. Dolean, V., Lanter, S., Nataf, F.: Convergence analyss of addtve Schwarz for the Euler equatons. Appl. Numer. Math. 49(2), 153 186 (2004) 5. Engqust, B., Zhao, H.K.: Absorbng boundary condtons for doman decomposton. Appl. Numer. Math. 27(4), 341 365 (1998) 6. Gander, M.J.: Optmzed Schwarz methods. SIAM J. Numer. Anal. 44(2), 699 731 (2006) 7. Genseberger, M., Smale, A., Hartholt, H.: Real-tme forecastng of flood levels, wnd drven waves, wave runup, and overtoppng at dkes around Dutch lakes. In: F. Kljn, T. Schweckendek (eds.) 2nd European Conference on FLOODrsk Management, Rotterdam, The Netherlands, 2012, Comprehensve Flood Rsk Management, pp. 1519 1525. Taylor & Francs Group (2013) 8. Japhet, C., Nataf, F., Roux, F.X.: Extenson of a coarse grd precondtoner to non-symmetrc problems. In: J. Mandel, C. Farhat, X.C. Ca (eds.) Doman Decomposton Methods 10, Contemporary Mathematcs, vol. 218, pp. 279 286. AMS (1998) 9. Maday, Y., Magoulès, F.: Absorbng nterface condtons for doman decomposton methods: A general presentaton. Comput. Meth. Appl. Mech. Eng. 195(29 32), 3880 3900 (2006) 10. Roest, M.R.T.: Parttonng for parallel fnte dfference computatons n coastal water smulaton. Ph.D. thess, Delft Unversty of Technology, The Netherlands (1997) 11. Vollebregt, E.A.H.: Parallel software development technques for shallow water models. Ph.D. thess, Delft Unversty of Technology, The Netherlands (1997) 12. Yossef, M.F.M., Zagonjoll, M.: Modellng the hydraulc effect of lowerng the groynes on desgn flood level. Tech. Rep. 1002524-000, Deltares (2010)