An update on the COSMO- GPU developments

Size: px

Start display at page:

Download "An update on the COSMO- GPU developments"

Hector Green
5 years ago
Views:

1 An update on the COSMO- GPU developments COSMO User Workshop 2014 X. Lapillonne, O. Fuhrer, A. Arteaga, S. Rüdisühli, C. Osuna, A. Roches and the COSMO- GPU team Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Name (change on Master slide) C2SM 1

The COSMO- GPU project Part of a larger project (HP2C iniqaqve): runs efficiently on different architectures : MulQcore CPUs (x86) and GPUs allow domain scienqst to easily bring new developments will

2 The COSMO- GPU project Part of a larger project (HP2C iniqaqve): runs efficiently on different architectures : MulQcore CPUs (x86) and GPUs allow domain scienqst to easily bring new developments will sqll be a good fit in the future Prototype GPU- version based on COSMO 4.19 available since 2013 A^er posiqve evaluaqon, the COSMO- consorqum agreed to bring back the GPU developments to the main trunk 5.X 4.19 GPU

3 Funding and related projects Finished HP2C COSMO- CLM (June 2010 June 2013) HP2C OPCODE (July 2011 June 2013) HP2C COCoNet (January 2012 June 2013) Ongoing PASC GridTools (January 2014 June 2017) SNF Sinergia (May 2015 April 2018) Planned PASC Focused : CLAW, submited H2020 (ECMWF) 3

GPU compuqng : some key aspects Why using GPUs : 7x higher peak performance 1 (double precision) 4x higher memory bandwidth Equivalent energy consumpqon Hybrid approach : offload parts of an

4 GPU compuqng : some key aspects Why using GPUs : 7x higher peak performance 1 (double precision) 4x higher memory bandwidth Equivalent energy consumpqon Hybrid approach : offload parts of an applicaqon, while the remainder stays on the CPU GPUs have thousands of compute cores : need to express fine grain parallelism GPU and CPU have (currently) separate physical memory requires specific data management data transfer may be a performance issue (slow transfer via PCI bus) 1 Intel Sandy Bridge CPU vs Nvidia K20 GPU 4

5 COSMO porqng strategy : full GPU port Avoid CPU- GPU copies by execu?ng all the?me loop computa?on on GPU [1] Input/Setup keep on CPU / copy to GPU Δt Prepare step Physics Dynamics Assimilation Halo-update Diagnostics Output OpenACC directives OpenACC directives STELLA Library (using CUDA) mixed CPU and GPU (OpenACC) GPU-GPU communication library (GCL) mixed CPU and GPU (OpenACC) mixed CPU and GPU (OpenACC) Finalize [1] Fuhrer, O. et al., Supercomputing Frontiers and Innovations, 1,

6 COSMO 5.X- GPU status : Ready for COSMO- 1 Parts Status Remark Physics Ready Only COSMO- 1 (no Flake) Fortran- C++ interface Ready Dynamical core Ready Only COSMO- 1 AssimilaQon Ready Only working with Cray compiler No LHN CommunicaQon Ready Use GCL for GPU Structure code (e.g. iniqalizaqon, lmorg.f90, ) Ready DiagnosQcs On going Minimal set Output On going Variables required for verificaqon Single precision On hold Issue in assimilaqon. Work will start in December 6

7 COSMO 5.X- GPU validaqon Technical validaqon : short run, GPU vs CPU agree within rounding error Meteorological validaqon : started daily run end of November. Scores of the GPU and CPU executable will be compared to reference code CPU GPU 24h daily COSMO-1 CPU and GPU runs on hybrid system Todi at CSCS 7

8 Performance results COSMO- 1 (1 km) future Meteoswiss configuraqon 24h run 1158x774x80 grid points dynamics physics : microphysics, radiaqon, turbulence, soil, shallow convecqon assimilaqon : nudging output Socket to Socket comparison, using same number of compute nodes, on XK7 (todi) : GPU nodes : AMD Opteron CPU (using 1 cores) + K20x GPU CPU nodes : AMD Opteron CPU (using all 16 cores) 8

9 Performance comparison preliminary results x2.3 Time (s) Lower is better CPU GPU 24h COSMO-1 Simulation, 120 compute nodes on hybrid system Todi! Overall run 2.3x faster on GPU for this configuration! Not yet optimized 9

Performance : influence of the number of nodes 12000 x2.6 Lower is better Total Time (s) 10000 8000 6000 4000 x2.

10 Performance : influence of the number of nodes x2.6 Lower is better Total Time (s) x2.3 CPU GPU # nodes 24h COSMO-1 simulation using 64 or 120 compute nodes on todi! The GPU is more efficient when increasing the number of grid points per MPI taks 10

11 Conclusions COSMO 5.X GPU ready for COSMO- 1 daily run Reasonable overall performance Ongoing/future work Meteorological validaqon using COSMO- 1 configuraqon OpQmizaQon PorQng of missing COSMO- 1 features (Flake, latent heat nudging) PorQng of missing COSMO- E (Ensemble 2km) features ( sppt, sso ) COSMO- GPU code will be used for the Synergia project Planned official COSMO release with GPU capability should be available Q If you want to use/try the COSMO-5.X GPU code contact us 11

12 Thank you

13 Current production code New code CPU Energy (kwh) 6.0 Cray XE6 (Nov 2011) Cray XK7 (Nov 2012) Cray XC30 (Nov 2012) New code GPU Cray XC30 hybrid (Nov 2013) x 1.75x 3.0 CPU 2.51x 1.49x 6.89x 1.5 CPU 2.64x GPU 0.0 GPU 13

Physical parametrizations and OpenACC directives in COSMO

Physical parametrizations and OpenACC directives in COSMO Xavier Lapillonne Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Name (change on Master slide)