Parallel computing on the Grid

Transcription

1 arallel computing on the Grid and application porting examples Roberto Alfieri - arma University & INFN Italy Comput-er.it Meeting - arma 21/06/2011 1

2 Current MI support in Grid (glite middleware) Issue Solution Description Multiple CU request JDL file CUnumber=4 Multiple MI version/flavour support Get the MI machinefile from the job manager mpi-start tool mpi-start tool openmpi mpich mpich2 Automatic management (BS LSF SGE) Files distribution among nodes mpi-start tool Automatic management (home shared not shared) CUnumber=4; Executable= "mpi-start"; Arguments= my-mpi-prog OENMI"; InputSandbox="my-mpi-prog";. 2

3 MI-start MpiStart (developed by HLRS Stuttgart) is a set of scripts that ease the execution of MI programs by using a unique and stable interface to the middleware. The adoption of MpiStart in Grid comes from the EGEE MI-WG recommendations Mpi-start frameworks: 1) Batch scheduler - BS, LSF, SGE,.. 2) MI implementations - openmi, MpiCh2,.. 3) Workflow control - Files distribution (if needed) - user's pre/post execution scripts Mpi-start maintainer web page: 4/13/2011 arallel computing on the Grid CCR-InfnGrid Workshop

4 Memory architecture on multicore processors In modern multicore processors the memory architecture is NUMA - Cpu/memory affinity is the ability to bind a process to a specific CU/memory bank p p p p p Measured network performance (using NetIE): Comm Type Latency MAX Bandw. 1 Intra-socket 640 ns 14 GBytes/s 2 Intra-board 820 ns 12 GBytes/s 3 infiniband 3300 ns 11 GBytes/s Memory performance (peak): Memory Type Latency MAX Bandw. L3 cache 35 ns DDR3 50 ns 32 GBytes/s Numa (HT or QI) 90 ns 11 GBytes/s 4 4

5 Forthcoming MI support (EMI middleware) Issue Solution Description Multiple CU request with granularity selection JDL file New features WholeNodes=true HostNumber=2 SMgranularity=8 openm support Mpi-start MI_USE_OM=1 pnode pcore psocket CU Affinity support Mpi-start MI_USE_AFFINITY=1 #CUnumber=4; Executable= "mpi-start"; Arguments= -t openmpi -pcore -d MI_USE_OM=1 -- my-prog"; InputSandbox="my-prog"; Wholenodes=true; HostNumber=2; SMGranularity=8; 5

6 arallel job examples 1 MI rank per node Executable= "mpi-start"; Arguments= -t openmpi -- my-mpi-prog"; InputSandbox="my-mpi-prog"; CUnumber=2; SMGranularity=1; 2 whole nodes, 1 MI rank per core Executable= "mpi-start"; Arguments= -t openmpi -- my-mpi-prog"; InputSandbox="my-mpi-prog"; Wholenodes=true; HostNumber=2; SMGranularity=8; Multi-thread job Executable= "my-openmp-prog"; Arguments= "; InputSandbox="my-openmp-prog"; Wholenodes=true; HostNumber=1; SMGranularity=8; 1 MI rank per socket, 4 threads per MI rank Executable= "mpi-start"; Arguments= -t openmpi d MI_USE_OENM=1 d MI_USE_AFFINITY=1 psocket -- my-prog"; InputSandbox="my-prog"; Wholenodes=true; HostNumber=2; SMGranularity=8;

7 arallel scientific applications Widely used application examples: Name Description MI Multi thread Einstein tk relativistic astrophysics C/C++/F90 YES YES NO Q-Espresso electronic structure F90 YES experimental with openmp GU Beta (May2011) Namd molecular dynamics C++ YES YES YES Chroma lattice field theory C++ YES NO YES Gromacs molecular dynamics C YES since v.4.5 with pthreads YES (Aug 2010) Since the INFN-ARMA grid parallel cluster (8 WNs 8 cores each, BS, openmi) supports a preliminary version of the new JDL syntax (WholeNodes attributes) and of the new mpi-start tool (openm and affinity support), it has been possible to start porting on the grid some widely used parallel applications. 7

8 orting example: Quantum Espresso QUANTUM ESRESSO is an integrated suite of computer codes for electronic-structure calculations and materials modeling. The maintenance and further development is promoted by the DEMOCRITOS National Simulation Center of IOM-CNR under the terms of the GNU General ublic License. arallel execution: In Quantum Espresso several parallelization levels are implemented with MI. Typical execution command on C clusters: mpirun -np 16 $BINDIR/pw.x < file.in >file.out Explicit OpenM is a very recent addition, still at an experimental stage. For execution using OpenM on N threads: env OM_NUM_THREADS=N $BINDIR/pw.x < file.in > file.out Hybrid parallelization (MI/openM) should be executed carefully to prevent conflicts: env OM_NUM_THREADS=N mpirun -np 8 $BINDIR/pw.x < file.in > file.out 8

9 espresso.jdl Executable = "espresso.sh"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"espresso.sh, file.in }; OutputSandbox = {"std.err","std.out, file.out }; espresso.sh Running QE on the Grid WholeNodes=true HostNumber=2 Granularity selection SMGranularity=8 #Requirements=(other.GlueCEUniqueID=="cream-ce.pr.infn.it:8443/cream-pbs-parallel"); #!/bin/bash #install wget tar xzvf espresso tar.gz cd $ESRESSO/espresso /configure #-enable-openmp make all # or whatever you need #execute $W=$ESRESSO/espresso-4.3.1/bin/pw.x mpi-start -t openmpi -psoket -d MI_USE_OM=1 -d MI_USE_AFFINITY=1 -- $W < file.in > file.out flavour MI rocess/thread distribution 9

10 orting example: Einstein toolkit Eintstein toolkit is an open software for relativistic astrophysics with mixed MI and openm parallelization. Evolution of a stable general relativistic TOV-Star. Hydro-dynamical Simulation of a perfect fluid coupled to the full Einstein Equations (dynamical space-time) on a 3-dimensional grid with 5-level of refinement spanning an octant of radius 177 km with a maximum resolution within the star of 740 m. Newly support mpi-start has adaptive way to mix openm and MI process. The best performance depends on the Hardware characteristic of the node: MI M Time (s) mpi-start -pcore mpi-start -psocket mpi-start -pnode Thanks to R. De ietri (arma Univ.) 10

11 orting example: NAMD NAMD is a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems. arallel execution: NAMD uses the Charm++ communications layer to launch namd2 processes. SM builds of Charm++ can be used with multiple threads on a single node or across the network. Typical parallel execution command: charmrun namd2 +p16 +ppn 8 <configfile> In case of MI-based SM builds: mpiexec np 16 namd2 +ppn 8 <configfile> NAMD may use specific syntax to start parallel executions, so a customization of the execution script is needed to adapt this application in the Grid environment. 11

12 namd.jdl Running parallel NAMD on the Grid Executable = namd.sh"; InputSandbox = { namd.sh, NAMD_2.8_Source.tar.gz, configfile }; WholeNodes=true HostNumber=2 SMGranularity=8 namd.sh #!/bin/bash #install tar xzf NAMD_2.8_Source.tar.gz make #execute set NODES = `cat $BS_HOSTFILE` set NODELIST = namd2.nodelist echo group main >! $NODELIST foreach node ( $nodes ) echo host $node >> $NODELIST NUMROCS = 2 * $#NODES charmrun namd2 +p$numrocs ++nodelist $NODELIST <configfile> 12

13 Thank you for your attention! 13