character :: buffer(100) integer :: position real :: a, b integer :: n position = 0 call MPI_PACK(a, 1, MPI_REAL, buffer, 100, & position, MPI_COMM_WO

Transcription

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38 MPI_PACK and MPI_UNPACK Each communication incurs a latency penalty so it is best to group communications together Requires data to be contiguous in memory with no gaps between variables This is true for arrays (i.e. a[1:10] ) Not true when we mix many different variables (i.e. a, b, c) Use MPI_PACK to combine different data types into contiguous memory for sending Use MPI_UNPACK to unpack the data back into non-contiguous memory after being received

39 character :: buffer(100) integer :: position real :: a, b integer :: n position = 0 call MPI_PACK(a, 1, MPI_REAL, buffer, 100, & position, MPI_COMM_WORLD, ierr) call MPI_PACK(b, 1, MPI_REAL, buffer, 100, & position, MPI_COMM_WORLD, ierr) call MPI_PACK(n, 2, MPI_INTEGER, buffer, 100, & position, MPI_COMM_WORLD, ierr) Position variable is incremented after each step, recording where we are in the buffer: Now send the data: a a a a b b b b call MPI_BCAST(buffer, 100, MPI_PACKED, 0, MPI_COMM_WORLD, ierr) n n

40 On the receiving processors, receive the data: character :: buffer(100) call MPI_BCAST(buffer, 100, MPI_PACKED, 0, MPI_COMM_WORLD, ierr) Reset position to zero (start of buffer) position = 0 call MPI_UNPACK(buffer, 100, position, a, 1, & MPI_REAL, MPI_COMM_WORLD, ierr) call MPI_UNPACK(buffer, 100, position, b, 1, & MPI_REAL, MPI_COMM_WORLD, ierr) call MPI_UNPACK(buffer, 100, position, n, 2, & MPI_INTEGER, MPI_COMM_WORLD, ierr) Again position variable is automatically incremented after each step, recording where we are in the buffer

41 Groups and new communicators Sometimes we want to use global reduction operations but only on a subset of the processes in MPI_COMM_WORLD Can break up the processes within the MPI_COMM_WORLD communicator into sub groups Allocate the processors within the new group to a new communicator Include processes in new groups using: MPI_GROUP_INCL(group handle, size of new group, ranks of processes in new group, new group formed, ierr) Then create a new communicator with: MPI_COMM_CREATE(communicator, mpi group which is a subset of the communicator, new communicator, ierr) Each process within the new communicator now has a unique rank within that communicator different to the one it is allocated in MPI_COMM_WORLD. Find its new rank within the group using MPI_GROUP_RANK or within the world using MPI_COMM_RANK Remove processors from a group using MPI_GROUP_EXCL

42 program group include 'mpif.h' integer, parameter :: NPROCS = 8 integer rank, new_rank, sendbuf, recvbuf, numtasks, ierr integer orig_group, new_group, new_comm Integer ranks1 = /0, 1, 2, 3/, ranks2 = /4, 5, 6, 7/ call MPI_INIT(ierr) call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr) call MPI_COMM_SIZE(MPI_COMM_WORLD, numtasks, ierr) if (numtasks.ne. NPROCS) then print *, 'Must specify MPROCS= ',NPROCS,' Terminating.' call MPI_FINALIZE(ierr) stop Endif Sample Output: rank= 7 newrank= 3 recvbuf= 22 rank= 0 newrank= 0 recvbuf= 6 rank= 1 newrank= 1 recvbuf= 6 rank= 2 newrank= 2 recvbuf= 6 rank= 6 newrank= 2 recvbuf= 22 rank= 3 newrank= 3 recvbuf= 6 rank= 4 newrank= 0 recvbuf= 22 rank= 5 newrank= 1 recvbuf= 22 sendbuf = rank! Extract the original group handle call MPI_COMM_GROUP(MPI_COMM_WORLD, orig_group, ierr)! Divide tasks into two distinct groups based upon rank if (rank.lt. NPROCS/2) then call MPI_GROUP_INCL(orig_group, NPROCS/2, ranks1, & new_group, ierr) else call MPI_GROUP_INCL(orig_group, NPROCS/2, ranks2, & new_group, ierr) Endif call MPI_COMM_CREATE(MPI_COMM_WORLD, new_group, & new_comm, ierr) call MPI_ALLREDUCE(sendbuf, recvbuf, 1, MPI_INTEGER, & MPI_SUM, new_comm, ierr) call MPI_GROUP_RANK(new_group, new_rank, ierr) print *, 'rank= ',rank,' newrank= ',new_rank,' recvbuf= ', recvbuf call MPI_FINALIZE(ierr) end

43

44

45

46

47

48 -lmpi

49 mpi_selector MPI needs configuring on labserv or some libraries may not be visible. Use the mpi-selector menu command > mpi-selector 1. mvapich 2. openmpi >1 User or system wide settings (u/s) >u Overwrite old settings? > yes

50 Numerical Practical Exercise Parallelisation In order to demonstrate some of the basics of parallelisation, we will be parallelising your simple 1D Hydro code from the Hydrodynamics lecture. Remember that the flow of the code is as follows: 1. Set the initial conditions. 2. Begin the evolutionary timestep: a) Calculate the global minimum time step. b) Update the boundary Guardcells. c) Evolve the Hydrodynamic variables using the upwind advection scheme. d) Update the boundary Guardcells 3. If we have passed the maximum time, then output the resulting data to file. We now need to perform the following steps to parallelise the code. 1. Initialize MPI, determine the rank of the current processor and determine how many processes are running 2. Split the 400 cell (x = 0-1.) computational domain between the number of processes so that each CPU has to work on a roughly equal number of cells. (If you run on an odd number of processes, e.g. 3, it would be easiest to first change the number of cells to one which is divisible by that number)

51 3. Once the domain is divided amongst the CPUs you should set the initial conditions on each block. If you passed the x coordinates to the initial conditions subroutine in your original code then this will not change. 4. We now need to perform a global reduction operation after the local time step has been found in order to obtain the global minimum time step (we cannot evolve each block with a different time step). 5. You should update both the left and right guardcells of the local block stored on each processor at steps (a) and (d) above. This is best done with a collective communication. 6. The solving of the hydrodynamical equations with the upwind advection scheme should be the same and thus does not need changing (apart from the array indices which we loop between, i.e. before we looped between n=2 and N_Total-1, now we loop between n=2 and N_local-1) 7. Once we have reached the maximum time then we can output the data in the correct order. The data should be written in serial (i.e. just by the Master Processor), you should see what happens if you try to get all of the processors to write in parallel!

52 You should send the data to the Master Processor for the serial write using as a series of point-to-point operations. A single collective communication would not be possible for very large simulations, why? You may need to add a tag to each message so that you can ensure that you then write the data out in the correct order from the correct processors. The Report You should include a copy of your code (you can easily check it as it should produce the same results as for the standard non-parallelised hydro code). The report should briefly outline the parallelisation strategy and answer the questions posed above: What happens when you try and perform an unstructured parallel write and therefore why do we perform a serial write? Why is point-to-point communication necessary in most applications when you write the data out?

53