High performance computing in an operational setting. Johan

Size: px

Start display at page:

Download "High performance computing in an operational setting. Johan"

Malcolm Green
5 years ago
Views:

1 High performance computing in an operational setting Johan

years of dedicated research 21% of our resources are allocated to R&D to enhance our knowledge

2 DHI in short We re an independent, private and not-for-profit organisation Our people are highly qualified 80% of our 1,100 employees hold an MSc or a PhD degree Our knowledge represents 50 years of dedicated research 21% of our resources are allocated to R&D to enhance our knowledge and innovation We make this knowledge globally accessible through our local teams and unique software DHI

3 Water modelling DHI

4 MIKE Powered by DHI software for water environments The standard for modelling of all water environments An extensive and global user community Local support by experts DHI

5 Shallow Water equations (2D depth-integrated NS equations) u v w + + = Sc x y z 2 ζ u u vu wu ζ 1 pa g ρ u = fv g dz Fu υt ussc t x y z x ρ0 x ρ x z z z 2 ζ v v uv wv ζ 1 pa g ρ v = fu g dz Fv υt vssc t y x z y ρ0 y ρ y z z z Solved for velocities and water level DHI

6 Flexible mesh DHI

7 Case: Riverine flooding South Boulder Creek, Colorado, USA City of Boulder Rocky Mountains South Boulder Creek West Valley FIRM Floodplain

8 South Boulder Creek, Colorado, USA Recent research showed that flooding would occur in the West Valley West Valley However: West Valley flooding was not identified in the local flood mapping by the regulatory authority. West Valley had been developed without flood protection measures. This discovery lost public confidence and raised distrust in standard floodplain mapping. FIRM Floodplain

dramatically affects flooding Important flow splits in flood plain (conveyance

9 Modelling Approach Overbank areas represented with very high accuracy: Floodplain storage is a key issue Storage effects behind culverts and roadway embankments dramatically affects flooding Important flow splits in flood plain (conveyance paths) Flooding and drying occurring Calculates flood conditions in 2.3 million pixels

10 South Boulder Creek, Colorado, USA 1969 floodplain delineation Results and Mapping shows: backup at highways overflows into west valley flow into baseline reservoir West Valley FIRM Floodplain

11 South Boulder Creek, Colorado, USA 1969 floodplain animation:

12 South Boulder Creek, Colorado, USA Comparing against 1969 flood extent Manhattan Drive looking south South Boulder Creek at US 36

13 Urban flooding in City of London, UK Model layout using the flexible grid for 2D overland flow simulation Trafalgar Square

14 Urban flooding in City of London, UK

15 Software increases the efficiency Undertaking complex analysis faster Tackling larger models To get the job done properly and quickly

16 Taking modelling to the next level Software must handle Data accessibility explosion Fine scale resolution Real time aspects

17 Efforts in decreasing simulation execution times 2011 MPI 2012 LINUX - HPC 2014 GPU 2016 HPCaaS 2008 OpenMP Implicit schemes

18 I/O is handled on local level Parallelization distributed memory Message passing interface (MPI) standard interface used for communication between processors The distribution of work is based on the domain decomposition concept (physical sub-domains) Each processor integrates basic equation in sub-domain Data exchange between sub-domains is based on halo layer/elements concept

19 Basic concept

20 Parallelization test case marine application Benchmark Mediterranean sea DHI

21 High Performance Computing Parallelization Distributed memory approach Speed up factor Mesh No. elements 80, ,029 1,292,116 DHI Number of processors #21

22 High Performance Computing (HPC) In a MIKE context: The most resource demanding models can now make use of HPC The speed-up depends on the specific model setup. The larger the model, the better the speedup (linear).

23 High Performance Computing hybrid approach Combines GPU technology with the MPI technology (a cluster of GPU s) IT4Innovation s Anselm Cluster at Ostrava University (Czech Republic) DHI #23

24 Benchmark preliminary results double precision DHI

25 Hybrid Parallelization A new frontier Mediterranean Sea Double Precision Mesh No. elements 323,029 1,292,116 5,156,238 Number of GPUs DHI #25

26 Hybrid Parallelization A new frontier Bench marking using a flood model MPI GPU GPU vs MPI 1 GPU is about 5x faster than 16 cores 4 GPU s is about 4x faster than 64 cores 16 GPU s is nearly 3x faster than 256 cores DHI #26

27 Hybrid Parallelization A case study Christchurch, New Zealand Catchment area approx. 420 km 2 including three river systems in the model domain: Avon River Styx River Heathcote River DHI #27 2D model domain: 4.2 million elements 10 m x 10 m resolution flexible mesh (rectangular elements) Distributed rainfall-runoff with no losses (rain-on-grid) - 1% AEP event - 21 hour storm

28 Hybrid Parallelization A case study Christchurch, New Zealand Run time on desktop PC (MPI) is 8.9 hours: 16 core Dell Workstation 2 x Intel Xeon CPU ES- 2687W v2 (8 core, 3.40 GHZ) 32 GB of RAM Windows 7 operating system Run time with 1 x GeForce GTX TITAN GPU card is 3.1 hours Run time with 2 x GeForce GTX TITAN GPU card is 1.7 hours DHI #28

0,60 0,40 0,20 0,00 64; 1,31 128; 0,81 256; 0,45 512; 0,26 768; 0,23 1024; 0,17 21

29 Hybrid Parallelization Christchurch, New Zealand 1,40 Christchurch MIKE 21 FM model HPC Cluster simulation perfomance Simulation time [hours] 1,20 1,00 0,80 0,60 0,40 0,20 0,00 64; 1,31 128; 0,81 256; 0,45 512; 0,26 768; 0, ; 0,17 21 hour flood forecast Below 10 minutes Number of HPC Cores DHI #29

30 Remote simulation using HPC

31 The potential - HPC can address the toughest challenges in water environments Model/Resolution Size Statistics based on multiple model runs Analytics Time critical runs DHI

32 Main usage Execution of large fine resolution set-ups Multiple simulations scenario runs/uncertainty assessment Auto-calibration semi automatic model parameter selections Data assimilation ensemble runs DHI

33 Value Proposition Access to powerful hardware without a large upfront investment Scalable solution to match needs Model performance second to none Support from DHI DHI

34 Client side GUI on standard PC HPC as a Service Select HPC service Choose number of cores DHI

35 Hardware and software is rented Book through HPC portal Upload data Run simulations Off load results Shut down in portal Data is lost

36 Boosting work flow Performance - Parallelization Remote execution - Utilizing powerful hardware Software as a Service - Renting hardware HPC as a Service DHI

37 Conclusions The use of advanced parallelization techniques are key in delivering timely detailed hydrodynamic modelling results. Large detailed 1D/2D hydrodynamic models can be used in real-time and near real-time applications like Flood Forecasting and Disaster Risk Management. DHI Software takes full advantage of the next wave of hardware solutions with the Hybrid MPI/GPU approach. DHI #37

Carlson Stepan Kuchar Jan Martinovic Lars Sørensen Ole Sørensen Vit Vondrak

38 Thank you Johan The presenter acknowledges the work of: Thomas Bech David Bezdek Jesper Carlson Stepan Kuchar Jan Martinovic Lars Sørensen Ole Sørensen Vit Vondrak The work was funded as a part of DHIs performance contract with the The Danish Agency for Science, Technology and Innovation

Introduction to MIKE FLOOD

Introduction to MIKE FLOOD HYDROEUROPE, Sophia-Antipolis, February 2011 Julie Landrein, DHI Denmark Introduction to MIKE FLOOD - Introduction to MIKE FLOOD - 1D Modelling: MIKE 11, MIKE URBAN - 2D Modelling: