The Swift simulation code

Transcription

1 The Swift simulation code ICC, Durham! : Intro to SPH Bert VandenBroucke: GIZMO Pedro Gonnet: Task-based parallelism Matthieu Schaller: Swift 1

2 Cosmological hydrodynamical simulations Eagle in Durham 2

3 Cosma 5 DataCentric 3 7 M CPU hours Tom Schaye Theuns +15

4 The Hubble Sequence 4 Schaye +15

5 Gadget-3 Tree-SPH code 5

6 6

7 Contents: 1.Lagrangian versus Eulerian dynamics 2.SPH - Smoothed Particle Hydrodynamics 3.Gadget (Tree - SPH) 4.HPC challenges 7

8 Continuity equation: Lagrangian time derivative Lagrangian continuity equation: 8

9 Eulerian versus Lagrangian hydrodynamics The Universe is isotropic! v = 600 km s -1 in terms of Mach number: v/c=600/2=300! 9

10 Enzo/PPM Enzo/Zeus Flash (non)-galilean invariance of Eulerian codes 10

11 Mesh has special directions Lagrangian distribution is isotropic on average 11

12 Time-step limitation (Courant step) sound speed and resolution vary by orders of magnitude use individual time-steps 12

13 SPH: Interpolation on a disordered grid original SPH papers: Gingold & Monaghan, MNRAS 181, 1977 Lucy, AJ 82, 1977 reviews: Springel, ARAA 48, 2010 Price, JComputPhys 231,

14 Smoothed Particle Hydrodynamics: basics sum over neighbours ( close enough ) differentiable compact support kernel function particle mass 14 algorithmic choice of kernel function other hydro terms (time) integration schema computational neighbour finding parallelisation

15 Other terms pressure gradient gravitational force radiative heating and cooling 15

16 Equations to compute LHS RHS need two neighbour loops 16

17 Neighbour finding using a tree 17

18 Neighbour finding using a tree 18

19 Neighbour finding using a tree 19 Springel 10

20 Domain decomposition 20 Springel 10

21 time integration D: Drift K: Kick t n t n+1/2 t n+1 x n x n+1 21 Tom Springel Theuns05

22 Implementation in Gadget-2 22

23 Equations to compute LHS RHS need two neighbour loops 23

24 density.c T0 T1 local contribution + export list export particles off task contribution import particles add imported contributions 24

25 local contribution + export list export particles off-task contribution re-import particles add imported contributions 25

26 local contribution + export list export particles MPI-synchronisation off-task contribution re-import particles add imported contributions 26

27 Gravity scaling: Aquarius (Springel+08) 27

28 Timings of Gadget-2 on Aquarius Aq-A-4 ( particles, DM only) speed / speed on 24 cores testing strong scaling expansion factor 28

29 total cpu time [s] tree walk tree walk + imbalance expansion factor 29

30 total cpu time [s] walk + imbalance (N=288) walk + imbalance (N=24) tree walk expansion factor 30

31 total cpu time [s] walk + imbalance (N=600) expansion factor 31

32 Timings of Gadget-3 on Aquarius Aq-A-2 ( particles, DM only) cpu time [s] pkd-grav has similar scaling gravity (total) tree walk + imbalance tree walk expansion factor 32

33 Timings of Gadget-3 on Aquarius Aq-A-2 ( particles, DM only) fraction of cpu time tree walk expansion factor 33

34 Timings of pkdgrav on G-halo from Stadel Imbalance 34

35 Gadget-3 Tree-SPH code load imbalance limits scaling 35

36 HPC limitations: tree-walk limited cache efficiency (random data access) limited vectorizability main integration loop domain()! gravity_tree! density()! hydro_force() load-imbalance build-up limits single core performance limits scaling 36

37 SPH issues: blob test SPH SPH AMR AMR 37 Tom Agertz+07 Theuns

38 SPH issues: Kelvin-Helmholtz instability 38 Agertz+07

39 SPH issues: entropy cusps galaxy-cluster entropy profile SPH vs AMR AMR codes SPH codes 39 Tom Frenk+99 Theuns

40 40

41 41

42 The report of SPH s death have been greatly exaggerated SPH Anarchy implementation 42 Dalla Vecchia

43 arxiv: arxiv:

44 Strong scaling test of SPH ideal scaling Swift Gadget-2 44

45 Strong scaling test of Barnes-Hut gravity implementation ideal scaling Swift Gadget-2 45

46 Summary: basics load-(im)balance scaling tests issues with standard SPH 46