Status of the COSMO GPU version

Transcription

1 Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Status of the COSMO GPU version Xavier Lapillonne

2 Contributors in 2015 (Thanks!) Alon Shtivelman Andre Walser Andrea Arteaga Andreas Pauling Anne Roches Ben Cumming Carlos Osuna Christian Zeeman Christoph Angerer Guilherme Peretti-Pezzi Guy de Morsier Katherine Osterried Lukas Mosimann Michael Baldauf Pascal Spörri Peter Messmer Petra Baumann Roman Cattaneo Stefan Rüdisühli Simon Förster-Binz Ulrich Schättler Xavier Lapillonne 2

3 GPU-code ready for operation Main activities in 2015 : finalize merge of GPU code with COSMO 5.0+ Some new features supported on GPU: sppt, latent heat nudging, flake, pollen Migration to git : git@github.com:meteoswiss-apn/cosmo-pompa.git 3

4 COSMO-1 and E vs COSMO-7 and 2 Computational cost = 40 x ECMWF-Model 9 to 18 km gridspacing 2 to 4 x per day COSMO km gridspacing 8 x per day 1 to 2 d forecast 13 x 20 x COSMO-E 2.2 km gridspacing 2 x per day 5 d forecast 21 members 7 x Ensemble data assimilation: LETKF 4

5 Production with CSCS Cray XE6 (Albis/Lema) MeteoSwiss operational system Since ~4 years Next-generation system Accounting for Moore s law (factor 4) 5 Images: CSCS

6 New MeteoSwiss HPC system Piz Kesch (Cray CS Storm) brid system with a mixture of CPUs and GPUs Fat compute nodes with 2 Intel Xeon E (Haswell) and 8 Tesla K80 (each with 2 GK210) Only 12 out of 22 possible compute nodes Fully redundant (failover for research and development) First GPU system for operational weather prediction worldwide Compute Rack Compu 1 AFCO CRAYPD01 AFCO C MotivAir Mo 48 48P GigE (Ops) 48P Gi 47 48P GigE (Mgmt) 48P Gig 46 Mellanox FDR IB Mellan 45 BLANK BL OSS MDS/MGS NetApp E2760 ESMS MN MN PPN/Login PPN/Login dra dra dra dra dra dra dra dra dra dra dra dra 2 BLANK BL 1 BLANK BL O MDS NetAp ES PPN PPN M M 0 0 6

7 Performance results comparison with CPU only system Piz Dora Piz Kesch (Cray CS Storm) Installed at CSCS in July 2015 Public announcement today brid system with a mixture of CPUs and GPUs Fat compute nodes with 2 Intel Xeon E (Haswell) and 8 Tesla K80 (each with 2 GK210) Only 12 out of 22 possible compute nodes Fully redundant (failover for research and development) Piz Dora (Cray XC40) Traditional CPU based system Compute nodes with 2 Intel Xeon E v3 (Haswell) Pure compute rack Rack has 192 compute nodes Very high density (supercomputing line) 7

8 Results Relative to Old Code ( Old = no C++ dycore, double precision) Piz Dora Piz Kesch Factor Sockets at required timeto-solution ~26 CPUs ~7 GPUs 3.7 x Energy per member 10.0 kwh 2.06 kwh 4.8 x Time with 8 sockets per member s 5980 s 3.8 x Cabinets required to run ensemble at required time-to-solution x 8

9 Code maintenance Recent development have increased the complexity of the model CPU, GPU, single precision, double precision, various compiler, systems the number of contributions/developers Code review with dedicated tools (e.g. Github) Automatic testing : Jenkins (now integrated with github) 9

10 Testing and validation Unittest - Component testing - Dycore only Freq: Daily (Jenkins) Testsuite - Full model, many configuration - Compare short run to reference Freq: Daily (Jenkins) Performance Benchmark - Test performance of operational configuration Freq: Daily (Jenkins) Meteorological verification - Run several seasons - Compare scores Freq: When required TOT_PREC1 Mean Error CPU, double precision GPU, double precision GPU, single precision 10

11 Next steps COSMO priority POMPA project extension until Merge into the official COSMO version Further GPU porting (e.g. common physics) Maintenance of C++ dynamical core Assimilation in single precision Support, training and documentation 11

12 Conclusions GPU capable version of COSMO model has been developed New co-design hybrid system Piz-Kesch for operational COSMO runs Problem size increase 40x in 4 years Many interesting consequences Code maintenance Knowhow transfer Changing hardware New tools Model development 12

13 Thanks 13

14 Validation Both CPU and GPU implement IEEE-754 compliant floating point operations a+b or a*b will give the same results = bitwise identical However : Transcendental functions are not defined, compiler dependent Compiler may use so called : fused add multipy In general CPU and GPU results are not bitwise identical Note : CPU codes compiled with two compilers are in general not bitwise identical Validations: requires to define acceptable thresholds, e.g. by perturbing reference CPU results to the last bit 14

15 G2G communication IB DRAM QPI E E IOH IOH PLX PLX PLX PLX PCIe DRAM 15

16 COSMO on GPU Avoid CPU- GPU transfer, full port strategy Transfer to CPU for output 16

17 Managment summary Key ingredients Improvement in CPU performance ~2.8 x Port to accelerators (GPUs) ~2.3 x Code improvement ~1.7 x Increase utilization of system ~2.8 x Increase in number of sockets ~1.3 x Target system architecture to application Note Separating hardware investments from software and workflow investments does not make sense! 17 Image: Cray