Work on NEMO optimisation

Transcription

1 Work on NEMO optimisation Cyril Mazauric 2016, july 28th Juillet 2016

2 Disclaimer Atos, the Atos logo, Atos Consulting, Atos Worldgrid, Worldline, BlueKiwi, Canopy the Open Cloud Company, Yunano, Zero , Zero Certified and The Zero Company are registered trademarks of Atos. May Atos. Confidential information owned by Atos, to be used by the recipient only. This document, or any part of it, may not be reproduced, copied, circulated and/or distributed nor quoted without prior written approval from Atos. Atos may make changes to specifications and product descriptions at any time, without notice. All products, computer systems, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. This is not a binding offer. Atos hardware and software products may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Copyright 2016, Atos. All rights reserved. 02

3 H2020:EsiWace Center of Excellence in NWP Goal: provide and support end-to-end NWP Workflow. Promote interactions How: improve efficiency and productivity Consortium: DKRZ, CNRS, MPG, ECMWF, Cerfacs, MetOffice, DWD, Univ Reading, BSC, STFC, Seagate, Bull/Atos, Allinea, ICHEC, CMCC [Italy] Bull Role: roadmap vision, optimisation, guidance 03

4 NEMO : oceanographical simulation

5 WRF analysis NEMO NEMO (Nucleus for European Modelling of the Ocean) is a state-of-the-art modeling framework for oceanographic research, operational oceanography seasonal forecast and climate studies. 05

6 WRF analysis NEMO NEMO (Nucleus for European Modelling of the Ocean) is a state-of-the-art modeling framework for oceanographic research, operational oceanography seasonal forecast and climate studies. NEMO is developed by an European consortium 06

7 Test case

8 NEMO : test case work has been done with the last version of NEMO trunk dev_r6519_hpc_4 we choose to work with GYRE60 HSW 12 c 2.6 GHz (up to 20 nodes) FDR interconnect no i/o icc version Intel(R) MPI Library 5.1.3

9 Work done for vectorization

10 NEMO : vectorization Intel advisor help us to determine where the vectorization is not efficient we add SIMD instructions to help Intel compiler to vectorized DO jj = 2, jpjm1!dir$ SIMD DO ji = fs_2, fs_jpim1! vector opt. zus = zcof * SQRT( taum(ji,jj) )! Stokes drift

11 NEMO : Vectorization (Intel Advisor)

12 NEMO : Vectorization 012

13 Work done for cache blocking

14 NEMO : cache blocking most useful loop could be modified the use the cache efficiently DO jk = 1, jpkm1 DO jj = 1, jpjm1 DO ji = 1, fs_jpim1! vector opt. zdit(ji,jj,jk) = ( ptb(ji+1,jj,jk,jn) - zdjt(ji,jj,jk) = ( ptb(ji,jj+1,jk,jn) - DO jk = 1, jpkm1,size_bk DO jj = 1, jpjm1,size_bj DO ji = 1,fs_jpim1,size_bi! vector opt. DO jbk=jk, min((jk+size_bk)-1,jpkm1) DO jbj=jj, min((jj+size_bj)-1,jpjm1) DO jbi=ji,min((ji+size_bi)-1,fs_jpim1) zdit(jbi,jbj,jbk) = ( ptb(jbi+1,jbj,jbk,jn) - zdjt(jbi,jbj,jbk) = ( ptb(jbi,jbj+1,jbk,jn) -

15 NEMO : cache blocking Cache blocking vecto 015

16 016 work done on masks and others

17 NEMO : masks et al replace vmask and umask by tmask to load only 1 table umask(i,j,k) =tmask(i,j,k)*tmask(i+1,j,k) vmask(i,j,k) =tmask(i,j,k)*tmask(i,j+1,k) zmsku = wmask(jbi,jbj,jbk) / MAX( umask(jbi,jbj,jbk-1) + umask(jbi-1,jbj,jbk) & & + umask(jbi-1,jbj,jbk-1) + umask(jbi,jbj,jbk), 1._wp ) zmskv = wmask(jbi,jbj,jbk) / MAX( vmask(jbi,jbj,jbk-1) + vmask(jbi,jbj-1,jbk) & & + vmask(jbi,jbj-1,jbk-1) + vmask(jbi,jbj,jbk), 1._wp ) zmsku = wmask(ji,jj,jk) / MAX( tmask(ji+1,jj,jk-1) + tmask(ji-1,jj,jk) & & + tmask(ji-1,jj,jk-1) + tmask(ji+1,jj,jk), 1._wp ) zmskv = wmask(ji,jj,jk) / MAX( tmask(ji,jj+1,jk-1)+tmask(ji,jj-1,jk) & & + tmask(ji,jj-1,jk-1) + tmask(ji,jj+1,jk), 1._wp )

18 NEMO : masks et al improve the communication of the 2d table with the existing lbc_lnk_multi CALL lbc_lnk( zcpx, 'U', 1._wp ) CALL lbc_lnk( zcpy, 'V', 1._wp ) CALL lbc_lnk_multi( zcpx, 'U', 1._wp, zcpy, 'V', 1._wp )

19 NEMO : masks et al masks et al vecto cache 019

20 020 work done on 3d table communication

21 NEMO : 3d tables communication improve the communication of the 3d tables with a new lbc_lnk_multi for 3d tables (GYRE/WORK/traadv_fct.F90) CALL lbc_lnk( zwx, 'U', -1. ) CALL lbc_lnk( zwy, 'V', -1. ) CALL lbc_lnk( zwz, 'W', 1. ) CALL lbc_lnk_multi( zwx, 'U', -1._wp, zwy, 'V', - 1._wp, zwz, 'W', 1._wp ) it is preferable to send huge tables to use the interconnect bandwidth

22 NEMO : 3d tables communication 3d comm vecto cache masks 022

23 023 Numerical precision

24 NEMO : Numerical precision the numerical precision is good. the difference between initial solver.stat file and the last one is small.

25 025 Conclusion

26 NEMO : Conclusion..

27 ESiWACE has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No

28 Trusted partner for your Digital Journey