A MapReduce-Based Multiple Flow Direction Runoff Simulation

Transcription

1 A MapReduce-Based Multple Flow Drecto Ruoff Smulato Ahmed Sdahmed ad Gyozo Gdofalv GeoIformatcs, Urba lag ad Evromet, KTH Drottg Krstas väg Stockholm Telephoe: Emal:{sdahmed, kth.se 1. Itroducto Usg typcal Dgtal Elevato Models (DEM) structures, may algorthms have bee proposed, whch ca be grouped to two groups, sgle flow drecto (SDF), ad multple flow drecto (MFD). I SFD algorthms each cell has oly oe outlet. The flow s assged to the eghbor cell wth steepest dow-slope (Tarboto 1997, lesö ad Zhou 1997, Q et al 2007). The SFD s commo ad easy to apply (Tarboto 1997), ad smple terms of computato. However, SFD lacks the ablty to properly model the flow flat ad relatvely hgh dverget flows areas ad creates parallel flows (Q et al 2007); havg oly oe outlet must be cosdered as llogcal ad t mposes great smplfcato (lesö ad Zhou 1997). I the other had, MFD algorthms dstrbute the flow amog dowslope eghborg pxels proportoally to the slope. Hgh performace computg was early recogzed the feld of hydrologcal modelg. arflow presets a parallel soluto for groudwater flows 3D space (Ashby ad Falgout 1996). Ta et al (2008) gave examples of usg of Lad Iformato System (developed by NASA) for hgh-resoluto modelg of surface ruoff o a hgh performace computg Lux cluster. The obectve of ths paper s to propose ad develop a geeral computatoal methodology based o the MapReduce framework for grd-based MFD algorthms. 2. BACKGROUND AND RELATED WORK 2.1 MFD MFD algorthms s prove to be more accurate compared to SFD (Zhou ad Lu 2002), but t comes at a hgh computato cost ad complexty as the water the SFD oly flow towards oe cell whle MFD water ca flow towards eght eghbors. Qu et al (1991) proposed dstrbutg the flow proportoally to the slope weghted wth cotour legth L, as Equato 1 L ta( β ) f = (ta(β) > 0; <= 8) (1) L ta( β ) = 1 where f represets the fracto of flow towards cell umber, ta(f ) ad ta(f ) s the slope of cell ad ; L ad L s costats of cell ad ; the summato s oly for dow-

2 slopes. Ths method does ot cosder the degree of dverget formed by the ceter cell ad ts eght eghborg cells. Freema (1991) proposed smlar algorthm based o the slope ad a costat as Equato 2 ta( β ) f = = 1 ta( β ) (ta(β) > 0; <= 8) (2) ad he suggested to be 1.1, whch was proved later to ot be optmal for all surfaces (lesö ad Zhou 1997). Q et al (2007) argued that ths algorthm stll does ot cosder the local terra codtos. May Algorthms have bee suggested sce the; readers are referred to Q et al (2007) for more dscusso o MFD algorthms. Q et al (2007) proposed a algorthm that combe Equato 1 ad 2, as well as cosderg the local terra codto as Equato 3 L ta( β ) f = (ta(β) > 0; <= 8) (3) L ta( β ) = 1 the value s determed accordg to local maxmum dow slope, ths way the algorthm works better for wth both dverget ad coverget flows. The value of at ay gve 3 by 3 wdow ca be calculated usg the followg equato. = 8.9 m( e,1) (4) where e s maxmum local dow slope ad m(e,1) s a fucto that retur the mmum betwee e ad 1. Ths study cosdered ths algorthm to demostrate the suggested MapReduce approach. 2.2 MapReduce ad Hadoop MapReduce programmg framework makes t easly to process parallel massvely large data o large umber of computers. MapReduce works maly through two fuctos, Map fucto ad Reduce fucto; map fucto takes a set of key/value put ad map t to zero or more set of key/value, the reduce fucto the takes each uque key ad group ts assocated values to a uque key/value set (Dea ad Ghemawat, 2008). Hadoop s a mplemetato of the MapReduce framework. 2.3 MapReduce MFD The local flow of a cell s defed based o a 3 by 3 eghborhood terms of possble flow teractos ( ad outflow) betwee the ceter cell ad ts eghbors. The global flow ca the be acheved by calculatg the local flow of each cell by addg the flow subtract the outflow. The suggested MapReduce ca be coceptualzed as flowg: For each cell form costruct a lst l of e elemets, where the frst elemet represets the ceter cell ad the rest elemets represets the eght eghbor cells. Each elemet composed of (ID;dem;wh), where ID s uque ID for the cell, dem ad wh are DEM ad water heght respectvely at the cell. l s stored sgle le for the purpose of feedg MapReduce.

3 The frst mapper Map1 apply MFD algorthm o each l ad calculate the flow betwee the ceter cell ad all the eghbor cells wth dow slope; ad the frst reducer Reduce1 aggregate the flows for each cell. The secod mapper Map2 wrtes each cell to the all eght eghbor cells; ad the secod reducer Reduce2 recostruct l for each cell. Repeatg these processes by usg the output of Reduce2 as a put to Map1 for each tme step. The rafall s added at the frst MapReduce ob ad processed by the frst Reduce1 as a flow for each cell. 3. EXERIMENTS AND RESULTS 3.1 Cluster Setup Hadoop verso s stalled o a cluster of four odes. Each ode has two Itel Xeo E5620 rocessor, 1MB L2 Cache, 12M L3 Cache, 2.40 GHz speed, 24GB RAM, oe 1Gbps Full Duplex NIC, ad 400GB hard drve. Each processor has 4 cores, 8 threads. All odes are coected usg a Ggabt swtch. The operatg system s Ubutu LTS. 3.2 Dataset Four mathematcs surfaces are used to geerate the put grdded DEM. These surfaces are recommeded by Zhou et al [8] to evaluate the accuracy of flow algorthms, ad t has bee chose as t represets the commo tra codtos, ad the same tme t s easy to geerate for the purpose of future comparso. 3.3 Results To verfy the proposed approach two expermets are carred out to execute oe tme step, cluster ad data scalg. Nodes Ellps IvEllps Saddle lae Ideal Table 1. erformace for cluster scalg as rato to sgle ode for the 30 mllo cells dataset.

4 30 Mllo Tme (Secods) Ellps IvEllps Saddle lae Number of odes Fgure 1. The tme compared to the umber of server odes for the 30 mllo cells dataset. Fgure 1 ad Table 1 show the result of the cluster scalg. Fg. 1 shows that the tme decrease drectly related to the umber of odes for the 30 mllo dataset. Table 1 compares the acheved tme decrease percetage to the theoretcally deal tme; however the percetage of two ode s tme s false dcator because the frst ode s rug as a server worker ode same tme. Data Scalg Tme (Sec) Ellps IvEllps Saddle lae Number of cells (Mllo) Fgure 2. The tme of executg oe tmestamp for dfferet datasets szes for each surface, o a four ode cluster setup. Fg. 2 shows the executo tme for dfferet data szes o a four ode cluster; the relatvely flat graph betwee 1 ad 20 mllo datasets dcates that the cluster s ot fully utlzed, as opposte to relatvely costat creasg from 30 mllo ad more.

5 The results shows that the suggest approach s sutable for achevg a parallelzed MFD o a MapReduce framework, ad hece reduced executo tme ad large amout of data processg ablty. 2. Refereces Ashby, S. F. ad R. D. Falgout, 1996, A parallel multgrd precodtoed cougate gradet algorthm for groudwater flow smulatos. Nucl Sc Eg, 124: Dea, J. ad Ghemawat, S, 2008, MapReduce : Smplfed Data rocessg o Large Clusters. (L.. Dael, Ed.) Commucatos of the ACM, 51(1), do: / Freema, T. G., 1991, Calculatg catchmet area wth dverget flow based o a regular grd. Computer ad Geosceces, 17: lesö,. ad Zhou, Q., 1997, Theoretcal estmato of flow accumulato from a grdbased dgtal elevato model. roceedgs of GIS AM/FM ASIA 97 ad Geoformatcs 97 Coferece (Tape), Q C, Zhu A.-X, e T, l B, Zhou C ad Yag L, 2007, A adaptve approach to selectg a flowpartto expoet for a multple-flow-drecto algorthm. Iteratoal Joural of Geographcal Iformato Scece, 21(4): Qu, Beve K, Chevaler ad lacho O, 1991, The predcto of hllslope flow paths for dstrbuted hydrologcal modelg usg dgtal terra models.. Hydrologcal rocesses, 5:59-79 Tarboto D.G, 1997, A ew method for the determato of flow drectos ad upslope areas grd dgtal elevato models. Water Resources Research, 33(2): Ta Y, eters-ldard C.D, Kumar S.V, Geger J, Houser.R, Eastma J.L, Drmeyer, Dotye B ad Adams J, 2008, Hgh-performace lad surface modelg wth a Lux cluster, Computers & Geosceces, 34: ZHOU Q ad LIU X, 2002, Error assessmet of grd-based flow routg algorthms used hydrologcal models. Iteratoal Joural of Geographcal Iformato Scece, 16(8):