Efficient use of hybrid computing clusters for nanosciences

Transcription

1 International Conference on Parallel Computing ÉCOLE NORMALE SUPÉRIEURE LYON Efficient use of hybrid computing clusters for nanosciences Luigi Genovese CEA, ESRF, BULL, LIG 16 Octobre 2008 with Matthieu Ospici, Jean-François Méhaut, Thierry Deutsch

2 Outline 1 Electronic structure calculations Main operations, parallelisation 2 The library 3 4

3 Ab initio calculations with DFT Several advantages Ab initio: No adjustable parameters DFT: Quantum mechanical (fundamental) treatment Main limitations Approximated approach Requires high computer power, limited to few hundreds atoms in most cases Wide range of applications: nanoscience, biology, materials

4 Performing a DFT calculation (KS formalism) Find a set of orthonormal orbitals Ψ i (r) that minimizes: E = 1 2 N/2 i=1 with ρ(r) = Ψ i (r)ψ i (r) i Z (Kohn-Sham) DFT Actors Ψ i (r) 2 Ψ i (r)dr + 1 Z ρ(r)ρ(r ) 2 r r dr dr Z +E LDA xc [ρ(r)] + V ext (r)ρ(r)dr A set of wavefunctions ψ i, one for each electron A computational approach on a finite basis One array for each Ψ i A set of computational operations on these arrays which depend on the basis set A (good) computer...

5 A basis for nanosciences: the project STREP European project: ( ) Four partners, 15 contributors: CEA-INAC Grenoble, U. Basel, U. Louvain-la-Neuve, U. Kiel Aim: To develop an ab-initio DFT code based on Daubechies, to be integrated in ABINIT, distributed freely (GNU-GPL license) L. Genovese, A. Neelov, S. Goedecker, T. Deutsch, et al., Daubechies wavelets as a basis set for density functional pseudopotential calculations, J. Chem. Phys. 129, (2008)

6 A DFT code based on Daubechies wavelets A basis with optimal properties for expanding localised information Localised in real space Smooth (localised in Fourier space) Orthogonal basis Multi-resolution basis Adaptive Systematic From early 80 s Applied in several domains Interesting properties for DFT Daubechies x φ(x) ψ(x)

7 Wavelet properties: adaptivity Adaptivity Resolution can be refined following the grid point. The grid is divided in Low (1 DoF) and High (8 DoF) resolution points. Points of different resolution belong to the same grid. Empty regions must not be filled with basis functions. Localization property,real space description Optimal for big & inhomogeneous systems, highly flexible

8 Some ongoing applications Study of favorable adsorption sites of NaCl clusters on Si tips during AFM experiments ( 250 at.) (Group of S. Goedecker, Basel University) 350 at.: DFT calculation to assess the accuracy of force fields at. : Organic Cu surface. HOMO-LUMO chg. dens. are compared to STM expts.

9 Operations performed Different numerical operations performed: For each wavefunction (Hamiltonian application) Between wavefunctions (Linear algebra) Comput. operations Convolutions with short filters BLAS routines FFT (Poisson Solver) Interpolating Daubechies

10 Orbital distribution scheme Used for the application of the hamiltonian The hamiltonian (convolutions) is applied separately onto each wavefunction ψ 1 MPI 0 ψ 2 ψ 3 ψ 4 MPI 1 ψ 5 MPI 2

11 Coefficient distribution scheme Used for scalar product & orthonormalisation BLAS routines (level 3) are called, then result is reduced MPI 0 MPI 1 MPI 2 ψ 1 ψ 2 ψ 3 ψ 4 ψ 5 Communications are performed via MPI_ALLTOALLV

12 High Performance Computing Localisation & Orthogonality Data locality Principal code operations can be intensively optimised Optimal for application on supercomputers Little communication, big packets of data No need of fast network Optimal speedup Efficiency of the order of 90%, up to thousands of processors Efficiency (%) at 65 at 173 at 257 at 1025 at (with orbitals/proc) Number of processors Data repartition optimal for material accelerators (GPU) Graphic Processing Units can be used to speed up the computation

13 Separable convolutions We must calculate F(I 1,I 2,I 3 ) = = L j 1,j 2,j 3 =0 L j 1 =0 h j1 h j1 h j2 h j3 G(I 1 j 1,I 2 j 2,I 3 j 3 ) L j 2 =0 h j2 L j 3 =0 h j3 G(i 1 j 1,i 2 j 2,i 3 j 3 ) Application of three successive operations 1 A 3 (I 3,i 1,i 2 ) = j h j G(i 1,i 2,I 3 j) i 1,i 2 ; 2 A 2 (I 2,I 3,i 1 ) = j h j A 3 (I 3,i 1,I 2 j) I 3,i 1 ; 3 F(I 1,I 2,i 3 ) = j h j A 2 (I 2,I 3,I 1 j) I 2,I 3. Main routine: Convolution + transposition F(I,a) = h j G(a,I j) a ; j

14 GPU-ported operations in (double precision) 20 Convolutions (CUDA rewritten) GPU speedups between and 20 can be 6 obtained for different 4 sections 2 GPU speedup (Double prec.) locden locham precond Wavefunction size (MB) GPU speedup (Double prec.) DGEMM DSYRK Number of orbitals Linear algebra (CUBLAS library) The interfacing with CUBLAS is immediate, with considerable speedups

15 Distribute the data on hybrid supercomputer should be executed on hybrid CPU-GPU architectures CPU CPU GPU GPU Data transfer is still MPI-based Network CPU CPU GPU GPU Only internode communication between CPU Data distribution should depend on the presence of GPUs on the nodes

16 Data repartition on a node Non-hybrid case GPU not used homogeneous repartition GPU MPI 0 MPI 1 MPI 2 MPI 3 DATA DATA DATA DATA

17 Data repartition on a node Naive repartition All the cores use the GPU at the same time GPU MPI 0 MPI 1 MPI 2 MPI 3 DATA DATA DATA DATA

18 Data repartition on a node Inhomogeneous repartition Only one node use the GPU with more data GPU MPI 0 MPI 1 MPI 2 MPI 3 DATA DATA DATA DATA

19 Data repartition on a node The approach library manages GPU resource within the node GPU MPI 0 MPI 1 MPI 2 MPI 3 DATA DATA DATA DATA

20 library (M. Ospici, LIG / Bull / CEA) De-synchronisation of operations Two semaphores are activated for each card on the node: Data transfer (CPU GPU CPU) Calculation on the GPU Each operation (e.g. convolution of a wavefunction) is associated to a stream. Operation overlap Calculation and data transfer of different stream may overlap Operation are scheduled on a first come - first served basis Several advantages The time for memory transfers is saved Heavy calculation can be passed to the card one - by - one, avoiding scheduling problems

21 Example of a time chart The GPU can be viewed as a shared co-processor CPU GPU Calculation GPU CPU MPI 0 MPI 1 MPI 2 MPI 3 Time

22 Convenince of approach Different tests thanks to flexibility We have performed many tests, with different ratios GPU/CPU on the same node Speedup on the full code (exemples) is the best compromise speedup/easiness Examples: CPU -GPU Inhomogeneous (best) Full code tested on Multi-GPU platforms CINES -Iblis 48 GPU, Prototype calculations CCRT - Titane Up to 196 GPU (Grand challenge 2009)

23 on Hybrid architectures can run on hybrid CPU/GPU supercomputers In multi-gpu environments, double precision calculations No Hot-spot operations Different code sections can be ported on GPU up to 20x speedup for some operations, 7x for the full parallel code (under improvements) Percent Seconds (log. scale) CPU code (rel.) Number of cores 1 Time (sec) Comm Other PureCPU precond locham locden LinAlg

24 on Hybrid architectures can run on hybrid CPU/GPU supercomputers In multi-gpu environments, double precision calculations No Hot-spot operations Different code sections can be ported on GPU up to 20x speedup for some operations, 7x for the full parallel code (under improvements) LG: Prix Bull-Fourier 2009 (Bull-GENCI) Reference Paper: LG et al., J. Chem. Phys. 131, (2009) Grand Challenge 2009 on CEA Hybrid Cluster Titane (192 GPU, 768 Intel Xeon Nehalem cores)

25 : a modern approach for nanosciences Flexible, reliable formalism (wavelet properties) Conceived for massive parallel architecture Open a path toward the diffusion of Hybrid architectures The library (M. Ospici) Share efficiently the GPU resource between the cores Works efficiently on massive supercomputers Can be generalised for other applications 1.3 GNU-GPL license Lots of applications & developments with team: D. Caliste, T. Deutsch (L_Sim - CEA INAC Grenoble) S. Goedecker (U. Basel) M. Ospici, J-F. Méhaut (LIG INRIA UJF Bull Grenoble)