Message Passing Interface (MPI-1)

Transcription

1 IPGP INSTITUT DE PHYSIQUE DU GLOBE DE PARIS Parallel Programming: Message Passing Interface (MPI-1) Geneviève Moguilny Institut de Physique du Globe de Paris December 2005

2 Architecture and programming models According to Flynn (1972), S: Single, I: Instruction, M: Multiple, D: Data. Sequential or Parallel Data Control (Inst.) S I S SISD MISD M SIMD MIMD Among MIMD, Networks of Workstations (NOWs). Parallel programming writing of one (or several) program(s) that will simultaneously run to reach a common goal: each processor will execute an instance of the program, data (message) exchange call to a routine of a message passing (or exchange) library like PVM (Parallel Virtual Machine) or MPI (Message Passing Interface). Programming models on NOWs with distributed memory (P = Program): MPMD for heterogeneous networks PVM or MPI-2, SPMD for homogeneous networks PVM (obsolete) or MPI-[1 2]. December 2005 MPI 2 / 21

3 SPMD application A program written with a traditional language (Fortran, C...). Each process receives the same instance of the program, but with conditional instructions, run the same code or a different part of the code, on the same data or on different data. The variables of the program are private and stay in the memory of the processor allocated to the process. Data is exchanged between processes through calls to routines of a message passing library like MPI. Mem 1 Mem 2 Mem n High throughput and low latency network: Myrinet, SCI, Infiniband... CPU 1 CPU 2 CPU n msgs, NFS,... msgs bandwidth 90 Mb/s, lat. 90 µs bandwidth 2 Gb/s, lat. 5 µs Program December 2005 MPI 3 / 21

4 The MPI message passing library Review: 1992: Need to create portable applications with good performance creation of a working group (mainly from the US and Europe) to adopt the HPF (High Performance Fortran) methods. 1994: Version 1.0 of MPI. 1997: Definition of a standard for MPI-2 (dynamic control of tasks, parallel I/O...), available today. Main open source implementations: LAM MPI (Indiana University), MPICH (Argonne National Laboratory); and based on, specific network libraries (MPI-GM: MPICH over Myrinet). December 2005 MPI 4 / 21

5 Concept of messages passing MPI application = set of independent processes that run their own code, and communicate by calling MPI routines. Communication Message exchange Message = identifier(sender process) + data type (simple or derived) + length + identifier(receiver process) + data Mem 1 Mem 2 CPU 1 sender message CPU 2 receiver Point to point communication December 2005 MPI 5 / 21

6 Main categories of the MPI API ➊ Environment management initialization of / exit from the environment (MPI INIT, MPI FINALIZE), processes identification (MPI COMM SIZE, MPI COMM RANK). ➋ Communications (message send and receive) point to point, collectives. ➌ Groups and communicators management Communicator: set of processes in which MPI operations are done, processors + communication context (managed by MPI). Communicator creation from: processes group created first by the programmer, another communicator (MPI COMM WORLD = all processes). December 2005 MPI 6 / 21

7 Very small example (1/2) Source: HelloMPI.f90 1: program HelloMPI 2: 3: implicit none 4: include mpif.h 5: integer :: nb_procs, rang, ierr 6: 7: call MPI_INIT(ierr) 8: 9: call MPI_COMM_SIZE(MPI_COMM_WORLD, nb_procs, ierr) 10: call MPI_COMM_RANK(MPI_COMM_WORLD, rang, ierr) 11: print *, I am the process number, rang, among,nb_procs 12: 13: call MPI_FINALIZE(ierr) 14: 15: end program HelloMPI December 2005 MPI 7 / 21

8 Very small example (2/2) Compilation and link: mpif90 HelloMPI.f90 -o HelloMPI 1: program HelloMPI 2: 3: implicit none 4: include mpif.h 5: integer :: nb_procs, rang, ierr 6: 7: call MPI_INIT(ierr) 8: 9: call MPI_COMM_SIZE(MPI_COMM_WORLD, nb_procs, ierr) 10: call MPI_COMM_RANK(MPI_COMM_WORLD, rang, ierr) 11: print *, I am the process number, rang, among,nb_procs 12: 13: call MPI_FINALIZE(ierr) 14: 15: end program HelloMPI ifort -c -Iincludes path HelloMPI.f90 ifort -Llibs path HelloMPI.o -static-libcxa -o HelloMPI -lmpichf90 -lmpich Execution: mpirun -np 4 -machinefile mf HelloMPI Run HelloMPI (with ssh or rsh) on 4 processors (np = number of processors) of hosts listed in the file named mf. Result: I am the process number 0 among 4 I am the process number 2 among 4 I am the process number 3 among 4 I am the process number 1 among 4 December 2005 MPI 8 / 21

9 Point to point communications Communications between 2 processes identified by their rank in the communicator. Transfer modes: standard: Synchronous: Buffered: Ready: MPI does (or does not) a temporary copy according to the implementation. send finished when receive terminated. temporary copy in a buffer created by the programmer, send finished when copy terminated. send done when reception started (client/server). Blocking send (coupled send and receive) or not, Immediate reception or not. Two kinds of exchange: ➊ distinct sender and receiver (MPI SEND, MPI SSEND, MPI IBSEND...): CPU 1 CPU 2 MPI_[mode]SEND MPI_[mode]RECV ➋ the sender also receives and vice versa (MPI SENDRECV, MPI SENDRECV REPLACE): CPU 1 CPU 2 MPI_SENDRECV MPI_SENDRECV December 2005 MPI 9 / 21

10 Simple example of point to point communication 1:!! point_a_point.f90 : Exemple d utilisation de MPI_SEND et MPI_RECV 2:!! Auteur : Denis GIROU (CNRS/IDRIS - France) <Denis.Girou@idris.fr> (1996) 3: program point_to_point 4: implicit none 5: include mpif.h 6: integer, dimension(mpi_status_size) :: statut 7: integer, parameter :: etiquette=100 8: integer :: rank,value,ierr 9: call MPI_INIT(ierr) 10: call MPI_COMM_RANK(MPI_COMM_WORLD,rank,ierr) 11: 12: if (rank == 0) then 13: value= : call MPI_SEND(value,1,MPI_INTEGER,2,etiquette,MPI_COMM_WORLD,ierr) 15: elseif (rank == 2) then 16: call MPI_RECV(value,1,MPI_INTEGER,0,etiquette,MPI_COMM_WORLD,statut,ierr) 17: print *, Me, process, rank,, I received,value, from the process 0. 18: end if 19: 20: call MPI_FINALIZE(ierr) 21: end program point_to_point CPU 0 CPU 1 CPU 2 CPU 3 mpif90 point to point.f90 -o point to point message: 1234 mpirun -np 4 -machinefile mf point to point Me, process 2, I received 1234 from the process 0. December 2005 MPI 10 / 21

11 Collective communications Set of point to point communications inside a group, done by one operation. Synchronizations (BARRIER) Data motions General spreadings (BCAST) or selective ones (SCATTER) CPU 0 CPU 1 CPU 2 CPU 3 MPI_SCATTER(sendbuf, sendcount, sendtype, recvbuf, & recvcount, recvtype, root, comm) Distributed collects (GATHER) or general ones (ALLGATHER) Cross exchanges (ALLTOALL) Reductions (REDUCE with SUM, PROD, MAX, MIN, MAXLOC, MINLOC, IAND, IOR, IXOR) CPU 0 CPU 1 CPU 2 CPU 3 val = 1 val = 2 val = 3 val = 4 sum = 10 MPI_REDUCE(val, sum, count, datatype, MPI_SUM, root, comm) December 2005 MPI 11 / 21

12 Communicators creation: topologies For the applications that use a domain decomposition, it is interesting to define a virtual grid of the processes to facilitate their manipulation: number of sub-domains = number of processes, work on the sub-domains then finalization with computations at the frontiers. creation of new communicators based on these topologies. These topologies can be: cartesian: periodic (or not) grid, identification of the processes by their coordinates, graph type for more complex topologies Real CPUs (0,2) (1,2) (2,2) (0,1) (1,1) (2,1) (0,0) (1,1) (2,0) Neighbors(3) = 1, 2, 4 MPI_CART_CREATE(...) MPI_GRAPH_CREATE(...) December 2005 MPI 12 / 21

13 MPI application development Why MPI? Allow to change the order of magnitude of the problem to solve, by a possible quasi-unlimited increase of the available memory, with standard hardware and software. Optimization Communication s mode choice: avoid the unnecessary copies Contention of the messages Load balancing taking care of bottlenecks (I/O) Overlap computation and communication Use of persistent requests Development assistance Graphic debuggers with MPI: TotalView, DDT Visualization of communications: Trace Collector/Analyzer (was Vampir/VampirTrace), Blade Well parallelized MPI application execution time inversely proportional to the number of used processors December 2005 MPI 13 / 21

14 MPI documentation Parallel programming: intro/main.html Parallel Programming with MPI (including User s Guide): Final draft of the MPI standard as of 3/31/94: ftp://ftp.irisa.fr/pub/netlib/mpi/drafts/draft-final.ps MPI Subroutine Reference: Open source implementations of MPI: LAM : MPICH : French version of these slides with french language links: moguilny/grille/mpi December 2005 MPI 14 / 21

15 MPI application running on EGEE: SPECFEM3D Resolution of regional scale seismic wave propagation problems to model wave propagation at high frequencies and for complex geological structures, with use of the spectral-element method (SEM). Application first written by D. Komatitsch (Université de Pau), used in more than 75 laboratories in the world, especially for the study of earthquakes, seismic tomography, ultrasonic propagation in crystals, topography, sedimentary basins and valleys, interface waves, and crystal s anisotropy. More information: dkomati1 EGEE positioning: NA4, VO ESR, domain: Solid Earth Physics December 2005 MPI 15 / 21

16 SPECFEM3D: technical caracteristics lines of Fortran 90 / MPI. Very scalable application, ran on 1944 CPUs at the EarthSimulator (Japan). On EGEE, ran on 64 CPUs at Nikhef (NL), on 4 or 16 CPUs at SCAI (DE), LAL (FR), CPPM (FR), CGG (FR), SARA (NL), IISAS-Bratislava (SK), HG-01-GRNET (GR), TU-Kocise (SK) and AEGIS01-PHY-SCL (YU). Constraints: Memory optimization necessary recompilation and update of the input files on SE at each input parameter change. Lot of temporary outputs ( writes on the /tmp of each node), + outputs on shared directory to retrieve ( /home NFS mounted). Need to launch successively two mpirun that have to use the same nodes, allocated in the same order. December 2005 MPI 16 / 21

17 SPECFEM3D on EGEE: Fem4.jdl 1: # Fem4.jdl 2: NodeNumber = 4; 3: VirtualOrganisation = "esr"; 4: Executable = "Fem"; 5: Arguments = "xmeshfem3d4 xspecfem3d4 4"; 6: StdOutput = "Fem.out"; 7: StdError = "Fem.err"; 8: JobType = "mpich"; 9: # LRMS_type = "pbs"; 10: Type = "Job"; 11: InputSandbox = { "Fem", "xmeshfem3d4", "xspecfem3d4"}; 12: OutputSandbox = { "output_mesher.txt","output_solver.txt", "xmeshfem3d4.out", 13: "xspecfem3d4.out", "Fem.out","Fem.err","Par_file"}; 14: Requirements = (other.glueceinfototalcpus >= 4) 15: && Regexp("-pbs-",other.GlueCEUniqueID); December 2005 MPI 17 / 21

18 SPECFEM3D on EGEE: Fem script (1/2) 1: #!/bin/sh 2: # 3: EXE1=$1 4: EXE2=$2 5: CPU_NEEDED=$3 6: TMP_DIR=/tmp/DATABASES_MPI_DIMITRI4 7: institut= echo $HOSTNAME cut -f2-5 -d. 8: MPIHOME=/usr/bin 9: if [ -f "$PWD/.BrokerInfo" ] ; then 10: TEST_LSF= edg-brokerinfo getce cut -d/ -f2 grep lsf 11: else 12: TEST_LSF= ps -ef grep sbatchd grep -v grep 13: fi 14: if [ "x$test_lsf" = "x" ] ; then 15: HOST_NODEFILE=$PBS_NODEFILE 16: else 17: echo "LSF Hosts: $LSB_HOSTS" 18: HOST_NODEFILE= pwd /lsf_nodefile.$$ 19: fi 20: for host in cat ${HOST_NODEFILE} 21: do 22: ssh $host rm -rf $TMP_DIR 23: ssh $host mkdir $TMP_DIR 24: ssh $host chmod 775 $TMP_DIR 25: done 26: export LFC_HOST="mu35.matrix.sara.nl" 27: export LCG_CATALOG_TYPE="lfc" 28: LocalFn="DATA.tgz" 29: SeFn="lfn:/grid/esr/MoguilnyMPI/DATA${CPU_NEEDED}.tgz" 30: cmd="edg-rm --vo esr copyfile $SeFn file:// pwd /$LocalFn" 31: echo ">>> $cmd" 32: $cmd 33: 34: if [! -f $LocalFn ] ; then 35: echo "edg-rm failed, trying lcg-cp..." 36: export LCG_GFAL_INFOSYS="mu3.matrix.sara.nl:2170" 37: cmd="lcg-cp --vo esr lfn:data${cpu_needed}.tgz file:// pwd /$LocalFn" 38: echo ">>> $cmd" 39: $cmd 40: fi } Creation of temporary directories Input data retrieving from SE December 2005 MPI 18 / 21

19 SPECFEM3D on EGEE: Fem script (2/2) 41: if [! -f $LocalFn ] ; then 42: echo "$LocalFn not found." 43: exit 44: fi 45: # 46: echo "*************************************" 47: tar xfz $LocalFn 48: rm $LocalFn 49: rm -rf OUTPUT_FILES 50: mkdir OUTPUT_FILES 51: cp DATA/Par_file. 52: 53: chmod 755 $EXE1 54: chmod 755 $EXE2 55: ls -l 56: time $MPIHOME/mpirun -np $CPU_NEEDED -machinefile $HOST_NODEFILE \ 57: pwd /$EXE1 > $EXE1.out 58: cp OUTPUT_FILES/output_mesher.txt. 59: time $MPIHOME/mpirun -np $CPU_NEEDED -machinefile $HOST_NODEFILE \ 60: pwd /$EXE2 > $EXE2.out 61: echo "Size of OUTPUT_FILES :" 62: du -sk OUTPUT_FILES 63: for host in cat ${HOST_NODEFILE} 64: do 65: ssh $host echo "Size of $TMP_DIR on $host :" 66: ssh $host du -sk $TMP_DIR 67: ssh $host rm -rf $TMP_DIR 68: done 69: cp OUTPUT_FILES/output_solver.txt. 70: tar cfz OUTPUT_FILES${CPU_NEEDED}.tgz OUTPUT_FILES 71: lcg-del --vo esr -s grid11.lal.in2p3.fr lcg-lg --vo esr \ 72: lfn:/grid/esr/moguilnympi/output_files${cpu_needed}.tgz 73: lcg-cr --vo esr -d grid11.lal.in2p3.fr \ 74: file:// pwd /OUTPUT_FILES${CPU_NEEDED}.tgz \ 75: -l /grid/esr/moguilnympi/output_files${cpu_needed}.tgz 76: exit } Launch of the mpirun Output data storage on SE December 2005 MPI 19 / 21

20 SPECFEM3D on EGEE: run UI RB SE CE WN WN CPU 0 CPU 1 WN CPU 0 CPU 1 /tmp exec1 exec2 /tmp mpirun... exec1 mpirun... exec2 CPU 0 CPU 1 /tmp exec1 exec2 I/O messages Disk server /home Site December 2005 MPI 20 / 21

21 Perspectives Some successful tests done with glite 1.1. Short term wishes: Node CPU (pbs), incompatibility between lcgpbs / lcglsf and MPI another LRMS like SGE or OAR? Specific MPI queues with equivalent power CPUs. In the future... CPUs reservation. Inter-sites MPI (MPICH-G2, MPICH-V). High performance networks (Myrinet, Infiniband...). MPI on RCs that have hundreds of nodes. But globally, MPI / Fortran90 on EGEE works!! December 2005 MPI 21 / 21