Hybrid OpenMP-MPI Turbulent boundary Layer code over 32k cores

Transcription

1 Hybrid OpenMP-MPI Turbulent boundary Layer code over 32k cores T/NT INTERFACE y/ x/ z/ Juan A. Sillero, Guillem Borrell, Javier Jiménez (Universidad Politécnica de Madrid) and Robert D. Moser (U. Texas Austin) FUNDED BY: CICYT, ERC, INCITE, PRACE & UPM 99

2 Outline Motivations Numerical approach Computational setup & domain decomposition Node topology Code Scaling IO performance Conclusions

3 Motivations Differences between internal and external flows: Internal: Pipes and channels External: Boundary layers Effect of large-scale intermittency in the turbulent structures Energy consumption optimization: Skin friction is generated in the interface vehicleboundary layer Separation of scales: 3 Layers structure: inner, logarithmic and outer Achieved only with high Reynolds number Important advantages of simulations over experiments

4 Motivations: Some underlying physic INTERNAL FLOWS EXTERNAL FLOWS Non Turbulent Sections: Turbulent Duct Pipe! 0+

5 Motivations: Some underlying physic INTERNAL FLOWS EXTERNAL FLOWS Non Turbulent Sections: Turbulent Duct! 0+ Pipe Skin Friction (drag) (5% world energy consumption)

6 Numerical Approach Incompressible Navier-Stokes equations v p w u + Boundary Conditions Staggered grid

7 Numerical Approach Incompressible Navier-Stokes equations Semi-implicit RK-3 Linear pressure-gradient terms v p w u Non-Linear Terms Linear Viscous Terms Staggered grid

8 Numerical Approach Incompressible Navier-Stokes equations v p w u SPATIAL DISCRETIZATION: Compact Finite Diferences (X & Y) Pseudo-Spectral (fftw 3.1.2)

9 Numerical Approach Fractional Step Method Inlet conditions using [Lund et al.] recycling scheme approach Linear systems solved using LU decomposition Poisson equation for pressure solved using direct method 2nd order time accuracy and 4th order CFD * Jimenez Simens et al. JCP 228, 4218 (2009) et al. JFM 657, 335 (2010)

10 Computational setup & domain decomposition Tier-0 Blue Gene/P ZY XY XY 63 Mb ZY 11 Mb (16 R*8 buffers) Plane to Plane decomposition 4x450 PowerPC 2 Gb RAM (DDR2) INCITE project (ANL) PRACE project (Jugene)

11 Computational setup & domain decomposition New parallelization strategy + Hybrid OpenMP-MPI

12 Computational setup & domain decomposition Global transposes: Change the memory layout Collective communications: MPI_ALLTOALLV Messages are single precision (R*4) About 40% of the total time (when using Torus network)

13 Computational setup & domain decomposition 4 OpenMP threads Static Scheduling: Through private indexes Maximise data locality Good load balance Loop blocking in Y Tridiagonal LS LU solver Tuned for Blue Gene/P

14 Computational setup & domain decomposition Create 2 MPI groups (MPI_GROUP_INCL) Groups created based in 2 lists of ranks Split global communicator in 2 local ones Each group performs independently Some global operations: Time step: MPI_ALLREDUCE Inlet conditions: SEND/RECEIVE

15 Node topology How to map virtual processes onto physical processors? 8192 Nodes BL1=512 BL2=7680 Predefined Custom Twice Faster 3D Torus network is lost: Comm BL1 [ Comm BL2 = MPI COMM WORLD

16 Node topology COMM. BALANCE COMPUT.

17 CODE SCALING: Time [s] 40% Comm. 8% Transp. 52% Comp. Millions of points per node Across nodes (MPI) Time per message [s] Within node (OpenMP) Node occupation 2 kb 7 MB Linear weak scaling Size of message [Bytes]

18 IO Performances Checkpoint of the simulation: 0.5 TBytes (R*4) Every 3 hours (12 hours run) Velocity {u,v,w} and pressure {p} fields (4x84 GB+4x7.2 Gb) Correlation files {u} Different strategies for IO: Serial IO: Discarded Parallel Collective IO: Posix calls SIONLIB library (Juelich) HDF5 (GPFS & PVFS2) HDF5 Tuning for Blue Gene/P: GPFS & PVFS2 (cache OFF & ON respectively) Cache OFF, write: 2 Gb/sec (5-15 minutes) Cache ON, write: 16 Gb/sec (25-60 seconds) Forcing file system block size in GPFS: 16 Gb/sec

19 Conclusions Turbulent boundary layer code ported to hybrid OpenMP-MPI Memory optimized for Blue Gene/P: 0.5 GB/core Excellent weak linear scalability up to 8k nodes Big impact in performances using custom node topologies Parallel Collective IO (HDF5): Read 22 Gb/sec, Write 16 Gb/sec Low pressure isosurface at high Reynolds numbers