Practical Introduction to Message-Passing Interface (MPI)

Transcription

1 1 Outline of the workshop 2 Practical Introduction to Message-Passing Interface (MPI) Bart Oldeman, Calcul Québec McGill HPC Bart.Oldeman@mcgill.ca Theoretical / practical introduction Parallelizing your serial code What is MPI? Why do we need it? How do we run MPI codes (on the Guillimin cluster)? How to program with MPI? Program structure Basic functions Examples Outline of the workshop 3 Parallelizing your serial code 4 Practical exercises on Guillimin Login, setup environment, launch MPI code Analyzing and running examples Modifying and tuning MPI codes Models for parallel computing (as an ordinary user sees it...) Implicit Parallelization minimum work for you Threaded libraries (MKL, ACML, GOTO, etc...) Compiler directives (OpenMP) Good for desktops and shared memory machines Explicit Parallelization work is required! You tell what should be done on what CPU Solution for distributed clusters (shared nothing!)

2 Distributed memory approach Process 0 A(10) 5 Address Space Different Variables! Process 1 A(10) Separate processes with own address spaces You need to make them interact and exchange information You tell what operation to do on what process MPI Message Passing Interface 6 Model for distributed memory approach Programmer manages memory by placing data in a particular process Programmer sends data between processes Programmer performs collective operations on sets of processes What is MPI for a user? MPI is NOT a language! MPI is NOT a compiler or specific product MPI is a specification for a standardized library: You use its subroutines You link it with your code History: MPI-1 (1994), MPI-2 (1997), MPI-3 (2012). Different implementations : 7 MPICH(2), MVAPICH(2), Open MPI, HP-MPI,... Your MPI code What MPI library do I link to? MPI Lib 8 InfiniBand Lib InfiniBand 10 Gbit/s Gigabit 1 Gbit/s When working on the Guillimin cluster, please use our version of the MVAPICH2 or Open MPI library! MPI library must match compiler used (Intel, PGI, or GCC) both at compile and at run time.

3 Basic features of MPI program 9 MPI is simple! (and complex...) 10 Include basic definitions (#include <mpi.h>, INCLUDE mpif.h, or ). Initialize and terminate MPI environment Get information about processes Send information between two specific processes (point-to-point communications) Send information between groups of processes (collective communications) You need to know these 6 functions: MPI Init : Initialize MPI environment MPI Comm size : How many CPUs do I have? MPI Comm rank : Identify each CPU MPI Send : Send data to another CPU MPI Recv : Receive data from another CPU MPI Finalize : Close MPI environment Fortran PROGRAM hello Example: Hello from N cores INTEGER err, rank, size CALL MPI Init(err) CALL MPI Comm rank (MPI COMM WORLD, & rank, ierr) CALL MPI Comm size (MPI COMM WORLD, & size, ierr) WRITE(*,*) Hello from processor,& rank, of, size CALL MPI Finalize(err) END PROGRAM hello C 11 #include <stdio.h> #include <mpi.h> int main (int argc, char * argv[]) { int rank, size; } MPI Init( &argc, &argv ); MPI Comm rank( MPI COMM WORLD, &rank ); MPI Comm size( MPI COMM WORLD, &size ); printf( "Hello from processor %d of %d\n", rank, size ); MPI Finalize(); return 0; General program structure 12 The header file (mpi.h or mpif.h) or module mpi must be included - contains definitions of MPI constants, types and functions All MPI programs start with MPI Init and end with MPI Finalize MPI COMM WORLD is a default communicator (defined in mpi.h) refers to the group of all processors in the job Each statement executes independently in each process

4 13 Compiling your MPI code Running your MPI code 14 NOT defined by the standard More or less similar for all implementations: Need to specify include directory and MPI library But usually a compiler wrapper (mpif90, mpicc) does it for you automatically. On the Guillimin cluster: user@guillimin>module add openmpi user@guillimin>mpif90 hello.f90 -o hello NOT defined by the standard A launching program (mpirun, mpiexec, mpirun rsh,...) is used to start your MPI code Particular choice of launcher depends on MPI implementation and on the machine used On the Guillimin cluster : mpiexec -n 4./hello 15 Exercise 1: Log in to Guillimin, setting up the environment 1) Log in to Guillimin: ssh username@guillimin.clumeq.ca 2) Check for loaded software modules: guillimin> module list 3) See all available modules: guillimin> module av 4) Load necessary modules: guillimin> module add ifort icc openmpi 5) Check loaded modules again 6) Verify that you have access to the correct mpi package: guillimin> which mpif90 output: /software/tools/openmpi intel/bin/mpif90 Exercise 2: Hello program, job submission 1) Copy hello.f90(c) and hello.pbs to your home directory: guillimin> cp /software/workshop/hello.f90. guillimin> cp /software/workshop/hello.pbs. 2) Compile your code: guillimin> mpif90 hello.f90 -o hello guillimin> mpicc hello.c -o hello 16

5 17 Exercise 2: Hello program, job submission Exercise 2: Hello program, job submission 18 3) Edit the file hello.pbs : #!/bin/bash #PBS -l nodes=1:ppn=2 #PBS -l walltime=00:05:00 #PBS -V #PBS -N hello cd $PBS O WORKDIR mpiexec -n 2./hello > hello.out 4) Submit your job: guillimin> msub -q class hello.pbs 5) Check the job status: guillimin> showq -u <username> 6) Check the output (hello.out) Exercise 3: Modifying Hello 19 MPI is simple! (and complex...) 20 Let s ask each CPU to do its own computation... IF (rank == 0) THEN a=sqrt(2.0) b=0.0 WRITE(*,*) a,b=,a,b, on proc,rank IF (rank == 1) THEN a=0.0 b=sqrt(3.0) WRITE(*,*) a,b=,a,b, on proc,rank Recompile as before and submit to the queue... You need to know these 6 functions: MPI Init : Initialize MPI environment MPI Comm size : How many CPUs I have? MPI Comm rank : Identify each CPU MPI Send : Send data to another CPU MPI Recv : Receive data from another CPU MPI Finalize : Close MPI environment

6 21 MPI Send / MPI Recv MPI basic datatypes (Fortran) 22 Passing message between two different MPI processes (point-to-point communication) If one process sends, another initiates the matching receive. The exchange data types are predefined for portability. (MPI has its own data types). MPI Send / MPI Recv is blocking! (There are also non-blocking versions) MPI Data type MPI INTEGER MPI REAL MPI DOUBLE PRECISION MPI COMPLEX MPI DOUBLE COMPLEX MPI LOGICAL MPI CHARACTER MPI PACKED MPI BYTE Fortran Data type INTEGER REAL DOUBLE PRECISION COMPLEX DOUBLE COMPLEX LOGICAL CHARACTER(1) Data packed with MPI Pack 8 binary digits MPI basic datatypes (C) MPI Data type MPI CHAR MPI SHORT MPI INT MPI LONG MPI UNSIGNED CHAR MPI UNSIGNED SHORT MPI UNSIGNED MPI UNSIGNED LONG MPI FLOAT MPI DOUBLE MPI LONG DOUBLE MPI PACKED MPI BYTE 23 C Data type signed char signed short int signed int signed long int unsigned char unsigned short unsigned int unsigned long int float double long double Data packed with MPI Pack 8 binary digits MPI: Sending a message Fortran: MPI Send (data, count, data type, dest, tag, comm, ierr) C: MPI Send (&data, count, data type, dest, tag, comm) Data : variable to send 24 Count : number of data elements to send Data type : type of data to send Dest : rank of the receiving process Tag : the label of the message Comm : communicator set of involved processes Ierr : error code (return value for C)

7 25 MPI: Receiving a message Fortran: MPI Recv (data, count, data type, source, tag, comm, status, ierr) C: MPI Recv (&data, count, data type, source, tag, comm, &status) Source is the rank of the sending process (or can be set to MPI any source) Tag must match the label used by sender (or can be set to MPI any tag) Status is an integer array with information in case of an error (source, tag, actual number of bytes received) MPI Send / MPI Recv are blocking! MPI Send / MPI Recv : Example PROGRAM mpipoint2point IMPLICIT NONE INTEGER :: rank, buffer, error, status( MPI status size ) CALL MPI init( error ) CALL MPI Comm rank( MPI COMM WORLD, rank, error ) IF( rank == 0 ) THEN buffer = 33 CALL MPI Send( buffer, 1, MPI INTEGER, 1, 10, & MPI COMM WORLD, error ) IF( rank == 1 ) THEN CALL MPI Recv( buffer, 1, MPI INTEGER, 0, 10, & PRINT*, Rank, rank, buffer=, buffer IF( buffer /= 33 ) Print*, fail CALL MPI Finalize( error ) END PROGRAM mpipoint2point 26 MPI Send / MPI Recv : Example 2 IF( rank == 0 ) THEN 27 IF( rank == 0 ) THEN Example 2 : Solution A 28 CALL MPI Send( buffer1, 1, MPI INTEGER, 1, 10, & MPI COMM WORLD, error ) CALL MPI Recv( buffer2, 1, MPI INTEGER, 1, 20, & ELSE IF( rank == 1 ) THEN CALL MPI Send( buffer2, 1, MPI INTEGER, 0, 20, & MPI COMM WORLD, error ) CALL MPI Recv( buffer1, 1, MPI INTEGER, 0, 10, & CALL MPI Send( buffer1, 1, MPI INTEGER, 1, 10, & MPI COMM WORLD, error ) CALL MPI Recv( buffer2, 1, MPI INTEGER, 1, 20, & ELSE IF( rank == 1 ) THEN CALL MPI Recv( buffer2, 1, MPI INTEGER, 0, 10, & MPI COMM WORLD, error ) CALL MPI Send( buffer1, 1, MPI INTEGER, 0, 20, & Will NOT work!!! Exchange Send/Recv order on one processor

8 IF( rank == 0 ) THEN 29 Example 2 : Solution B CALL MPI Isend( buffer1, 1, MPI INTEGER, 1, 10, & MPI COMM WORLD, REQUEST, error ) CALL MPI Recv( buffer2, 1, MPI INTEGER, 1, 20, & ELSE IF( rank == 1 ) THEN CALL MPI Isend( buffer2, 1, MPI INTEGER, 0, 20, & MPI COMM WORLD, REQUEST, error ) CALL MPI Recv( buffer1, 1, MPI INTEGER, 0, 10, & CALL MPI Wait( REQUEST, status, error )! Wait until send is complete Use non-blocking send: ISend Non-blocking Send: MPI Isend Differences from blocking Send: 30 Starts the transfer and returns control Between the start and the end of transfer the code can do other things You need to check for completion! REQUEST additional identifier for the comunication MPI Wait (REQUEST, Status, error) : The code waits for the completion of the transfer Exercise 3: MPI Send/Recv Copy send.f90(c) to your home directory: guillimin> cp /software/workshop/send.f90. Let s modify it: Send an array (or matrix) Ask 0 and 1 to exchange information Hints: /software/workshop/all/: send matrix.f90 exchange.f step beyond : Collective Communications 32 Involve ALL processes in the communicator MPI Bcast : Same data sent from root process to all the others MPI Reduce : root process collects data from the others and performs an operation (min, max, add, multiply, etc...) MPI Scatter : distributes variables from the root process to each of the others MPI Gather : collects variables from all processes to the root one

9 33 MPI Bcast: Example PROGRAM broadcast INTEGER ierr, myid, nproc, root INTEGER status(mpi STATUS SIZE) REAL A(2) CALL MPI Init(ierr) CALL MPI Comm size(mpi COMM WORLD, & nproc, ierr) CALL MPI Comm rank(mpi COMM WORLD, & myid, ierr) root = 0 IF( myid == 0 ) THEN a(1) = 2.0 a(2) = 4.0 CALL MPI BCAST(a, 2, MPI REAL, root, & MPI COMM WORLD, ierr) WRITE(*,*) myid, : a(1)=, a(1), & a(2)=, a(2) CALL MPI Finalize(ierr) END PROGRAM broadcast MPI Reduce : partial sum Other operations: product, min, max, etc. 34 MPI Reduce : Example PROGRAM reduce INTEGER ierr, myid, nproc, root INTEGER status(mpi STATUS SIZE) REAL A(2), res(2) CALL MPI Init(ierr) CALL MPI Comm size(mpi COMM WORLD, nproc, ierr) CALL MPI Comm rank(mpi COMM WORLD, myid, ierr) root = 0 a(1) = 2.0 a(2) = 4.0 CALL MPI REDUCE(a, res, 2, MPI REAL, MPI SUM, root, & MPI COMM WORLD, ierr) IF( myid == root ) THEN WRITE(*,*) myid, : res(1)=, res(1), & res(2)=, res(2) CALL MPI Finalize(ierr) END PROGRAM reduce 35 MPI Scatter MPI Scatter / MPI Gather 36 MPI Gather MPI Scatter: One-to-all communication different data sent from root process to all others in the communicator MPI Gather: data collected by the root process. Is the opposite of Scatter

10 Scatter 37 MPI Scatter / Gather : Example PROGRAM scatter INTEGER ierr, myid, nproc, nsnd, i, root INTEGER status(mpi STATUS SIZE) REAL A(16), B(2) CALL MPI Init(ierr) CALL MPI Comm size(mpi COMM WORLD, nproc, ierr) CALL MPI Comm rank(mpi COMM WORLD, myid, ierr) root = 0 IF( myid == root ) THEN DO i = 1, 16 a(i) = REAL(i) END DO nsnd = 2 CALL MPI Scatter (a, nsnd, MPI REAL, b, & nsnd, MPI REAL, root, & MPI COMM WORLD, ierr) WRITE(*,*) myid, : b(1)=, b(1), & b(2)=, b(2) CALL MPI Finalize(ierr) END PROGRAM scatter Gather PROGRAM gather INTEGER ierr, myid, nproc, nsnd, i, root INTEGER status(mpi STATUS SIZE) REAL A(16), B(2) CALL MPI Init(ierr) CALL MPI Comm size(mpi COMM WORLD, nproc, ierr) CALL MPI Comm rank(mpi COMM WORLD, myid, ierr) root = 0 b(1) = REAL( myid ) b(2) = REAL( myid ) nsnd = 2 CALL MPI Gather(b, nsnd, MPI REAL, a, nsnd,mpi REAL, root, MPI COMM WORLD, ierr) IF( myid == root ) THEN DO i = 1, (nsnd*nproc) WRITE(*,*) myid, : a(i)=, a(i) END DO CALL MPI Finalize(ierr) END PROGRAM gather Exercise 4: MPI Reduce Copy pi collect.f90(c) to your home directory: guillimin> cp /software/workshop/pi collect.f90. Let s add timings: IF (rank==0) THEN t1 = MPI Wtime()... IF (rank==0) THEN t2 = MPI Wtime() WRITE(*,*) Time =, t2-t Exercise 5: Replace collective calls in pi collect by point-to-point communication subroutines (MPI Send / MPI Recv) Exercise 6: take-home exercise Phase field crystal model A (Elder & Provatas): ψ t = M(W 2 2 ψ a 2 ψ a 4 ψ 3 ) here ψ(x, y) is discretized on a grid, e.g. 256x And/Or Implement a dot product routine in MPI.

11 41 Exercise 6: take-home exercise Discretized equation to solve PDE using Euler s method: ψ i,j,tn+1 = ψ i,j,tn + dtm(w 2 2 ψ a 2 ψ a 4 ψ 3 ) where 2 ψ (the Laplace operator) is discretized as ψ i+1,j +ψ i 1,j +ψ i,j+1 +ψ i,j ψ i+1,j+1+ψ i 1,j+1 +ψ i+1,j 1 +ψ i 1,j 1 4 3ψ i,j dx 2 with periodic boundary conditions. Exercise 6: take-home exercise Code in guillimin> cp /software/workshop/pf ModelA/f90/*. guillimin> make Split up grid among processors: column-wise, row-wise (implemented), or block-wise. All need neighbour-to-neigbour communication along the borders. Questions: Given 16 processors and a 256x256 grid, how many elements need to be transferred along the borders using 16 rows, and using 4x4 blocks, respectively, for every time step? What technique should scale better? 42 MPI routines we know... Startup: MPI Init, MPI Finalize Information on the processes: MPI Comm rank, MPI Comm size Point-to-point communications: MPI Send, MPI Recv MPI Irecv, MPI Isend, MPI Wait Collective communications: MPI Bcast, MPI Reduce, MPI Scatter, MPI Gather 43 Further readings: 44 The standard itself, news, development: Detailed MPI tutorials: mpi/tutorial/