Parallelization. Tianhe-1A, 2.45 Pentaflops/s, 224 Terabytes RAM. Nigel Mitchell

Transcription

1 Parallelization Tianhe-1A, 2.45 Pentaflops/s, 224 Terabytes RAM Nigel Mitchell

2 Outline Pros and Cons of parallelization Shared memory vs. cluster computing MPI as a tool for sending and receiving messages Point-to-point communication Blocking vs. non-blocking communication Latency hiding Collective communication User defined reduction operations Application to Hydrodynamics

3 Memory The model disk galaxy on the left occupies a 60kpc 3 box and requires a peak resolution of 59pc to converge. This therefore requires a mesh of cells. For each cell we need to store the following hydrodynamic properties: ρ, v x, v y, v z, E int, E tot, P As well as the gravitational potential at the current and previous time step: Φ n, Φ n 1 For each cell we also store the fractional abundances of the nine dominant elements responsible for cooling in the ISM as well as a reserve fraction for the others that we don t explicitly track; H, He, C, N, O, Ne, Mg, Si, Fe, X others

4 In all this makes 19 double precision (REAL*8) variables per cell cells 19 variables 8 bytes = bytes bytes bytes per Gigabyte = 152Gb This is more memory than what is available on most single computers, therefore we need a bigger machine!

5 Different Types of Cluster Mainframes - Shared Memory Machines Represented the first supercomputers One giant system with all of the processors and memory based on one large motherboard Communication across the motherboard is extremely fast Memory is shared between all of the processors Usually smaller as these systems have to be specially designed implying that they cost a lot more

6 Distributed Grid Supercomputers Commonly referred to as clusters Dominate the top 500 supercomputers Cheaper and easier to build Easy to scale as they use conventional hardware connected using either Ethernet or Myrinet/Infiniband connections. Easy to replace damaged or worn out components without taking the entire system offline for long periods of time.

7 Network Portal Node 01 Headnode (controls cluster) Node 02 Node 03 Each node can have a few multicore processors Nodes are connected to data switches which handle the passing of data throughout the cluster Data Switch Node 04 Node 05 Node 05 Data Switch

8 Latency and Communication Speed When we wish to send a message to another processor we need to: initialise the connection, this takes time and is known as the latency of the system. The latency depends upon the hardware used to relay the signal. Ethernet has a very high latency ( 3 ms) Infiniband and Myrinet have much smaller latencies ( 0.2 μs) Send the data. The speed of this process depends upon the bandwidth. Ethernet ( 2 Gbs 1 ) Infiniband/Myrinet ( 300 Mbs 1 ) Finalise the communication by checking that the message was received (not always done)

9 MPI Message Passing Interface MPI is a library of subroutines which have almost identical syntax in different coding languages making it easy for programmers to switch between language types C: MPI_SEND(data, count, MPI_DOUBLE, send_to, tag, MPI_COMM_WORLD); FORTRAN: call MPI_SEND( data, count, MPI_DOUBLE_PRECISION, send_to, tag, MPI_COMM_WORLD, ierr) Fortran includes an extra integer ierr to identify whether the message was successfully sent or whether an error was identified.

10 MPI s prime goals: Provide source code portability code designed on one cluster should be able to compile and run on another without having to rewrite any of your source code. Allow efficient implementation across a wide range of architectures machine dependant effects are taken care of in the MPI software. Offer sufficient functionality to undertake all major tasks in a parallel program. Designed for heterogeneous clusters all computers within the cluster must run the same Operating System and have the same software libraries installed. You can mix hardware but your simulation will be limited to the slowest components.

11 What MPI does not: MPI does not control the loading of processes onto processors this is handled by mpiexec or mpirun, e.g. mpirun np 7./flash3 MPI does not allow for the spawning of additional processes during execution you cannot change the number of processors you are using part way through a run (unlike OpenMP which does allow for spawning but only runs locally on multithreaded chips). Debugging is handled by separate programs such as DDT. Parallel I/O is not explicitly included in the MPI libraries. The user must instead write routines to organise the writing of data from each processor, or the data must be sent to the master processor using MPI calls and then written out in serial by the one master processor.

12 Initialization and Finalizing We always begin by calling call MPI_INIT(ierr) ierr = integer used to return an error code. We always finish by calling call MPI_FINALIZE(ierr) MPI_FINALIZE does not guarantee that all messages have been sent or received before it stops the MPI so it is up to the user to design the code to ensure that this is true. You cannot restart MPI by calling MPI_INIT so MPI_FINALIZE must go at the end of your program. We also need to include the following line at the top of the file so that the compiler includes the necessary MPI libraries: include mpif.h

13 Communicators All processes that we wish to communicate with must belong to a communicator. A communicator is simply a list of the ranks identifying which processes we wish to communicate with at a given time. We can therefore have many different communicators with different groups of processes in them depending upon what we want to do MPI_COMM_WORLD this is a special communicator which includes all of the processes on the system that we are running.

14 Rank Each process on a parallel job is allocated a Rank which identifies it from the rest of the processes on the system. We can obtain it using the following command Call MPI_COMM_RANK(MPI_COMM_WORLD, rank) Ranks are numbered 0 N Processes 1 (this is the same in C and FORTAN) We can determine how many processes are running using: Call MPI_COMM_SIZE(MPI_COMM_WORLD, size) where size is an integer.

15 MPI Sends There are four types of send in MPI: 1. Standard Send 2. Synchronous Send 3. Buffered Send 4. Ready Send Until the send is complete the code will stop at the line with the call to the MPI send. By definition the send call can only complete once the send buffer it is using is free, ready to be used again. For each SEND call using MPI, you need a corresponding RECV statement to receive the message. The above sends therefore only differ in the way they treat the receipt of the message.

16 MPI Blocking SEND Syntax Call MPI_SEND(buffer, count, datatype, destination, tag, comm, ierr) Variable Buffer Count datatype Destination Tag comm Purpose Name of variable or array containing the data to be sent Number of elements of a given MPI datatype that the buffer contains MPI data type of the data being sent rank of the process to which the data is being sent specific identifier in case you wish to ensure that the message is received only by a specific RECV call. The communicator specified for both the sender and receiver, e.g. MPI_COMM_WORLD

17 Blocking form MPI_SEND MPI_SSEND MPI_BSEND MPI_RSEND Non-Blocking form MPI_ISEND MPI_ISSEND MPI_IBSEND MPI_IRSEND Standard Send Synchronous Send Buffered Send Ready Send Standard Send Synchronous Send Buffered Send Ready Send (no difference to blocking MPI_RSEND) The difference between a blocking and a non-blocking process is that a blocking process stops the code until it completes the send whereas a non-blocking process allows the code to continue running past it whilst it sends the data (before the send buffer has emptied).

18 Standard Send Completes as soon as the message is sent and the send buffer is free. Does not guarantee that the message was received, it may lie in the communication network between the sender and the receiver. Should therefore not assume that send will complete before the receive begins as two blocking sends exchanging messages could result in deadlock. Should not assume that message will complete after receive begins. If the order in which processes receive messages has a specific order this could be affected by this. The code should be written in a way that guarantees that the other processes are expecting the messages and thus wait to receive all of them. Otherwise the network could become overloaded with messages that have not been received.

19 Synchronous Send Only completes once the message has been received. Requires the sending of a hand shake from the receiving process to confirm that the message has been received. Is therefore slower as you need to wait for the message to be sent and the receiver to confirm receipt. Safer as you are guaranteed that the message is received. Useful if the order that messages are received in is important

20 Buffered Send Load the message into a temporary buffer in memory ready to be sent later once the system is ready. The message immediately completes after loading the message into the buffer unless the buffer is full. As the data is stored in a buffer that must be kept open whilst the system waits to send, it must be explicitly declared by the user and is not done automatically by the MPI send routine. Attach buffer space using the command Call MPI_BUFFER_ATTACH(buffer, size) where buffer is the name of a variable or array already allocated in memory and has size declared in bytes. Detach buffer space using the command Call MPI_BUFFER_DETACH(buffer, size)

21 Note only one buffer can be attached per process. Buffered space should not be altered by the user elsewhere in the code. Advantages: This allows the system to send the data later whilst the process continues running through the code. Therefore the code does not stand idle whilst we wait to send data. Guarantees the user that the sending and receiving will not be synchronised so the user does not assume anything. Cons: We may need to check that the message was received later. As a result blocking calls generally requires less MPI calls than the equivalent process with non-blocking sends Requires the allocation of extra memory for buffering as well as copying of data into and out of buffer space which can be costly.

22 Ready Send Sends message immediately without checking to see if a corresponding receive has been posted. Designed to remove the need for hand shakes and buffering between the sender and receiver. Therefore it provides faster communication at the expense of more robust checks Difficult to debug if things go wrong as it is possible messages are left undetected in the network May need to confirm that message was received. Proc 0 Non-Blocking Receive from 0 Proc 1 With tag 0 time Synchronous Send to 1 With tag 1 Blocking Receive from 0 With tag 1 Ready Send to 1 With tag 0 Message with tag 0 is received

23 MPI Blocking RECV Syntax Call MPI_RECV(buffer, count, datatype, source, tag, comm, status, ierr) Variable Buffer Count datatype source Tag comm status Purpose Name of variable or array into which received data should be written Number of elements of a given MPI datatype that buffer can contain (for MPI_RECV this can be an upper limit with the actual number of elements received being lower). MPI data type of the data being received rank of the process from which the data was sent. You can receive messages from any source using MPI_ANY_SOURCE specific identifier in case you wish to ensure that only a specific message is received by this call. You can receive messages from any tag using MPI_ANY_TAG The communicator specified for both the sender and receiver, e.g. MPI_COMM_WORLD If wildcards are specified for either source or tag, allowing the process to receive messages from any other process, then status returns the resulting source and tag of the sender

24 Call MPI_COMM_RANK(MPI_COMM_WORLD, mype) SenderID = 0 DestinationID = 1 if(mype.eq. SenderID) then! We are the sending process buffer=5678! Set the data to be sent Call MPI_Send(buffer, count, MPI_INTEGER, DestinationID, & tag, MPI_COMM_WORLD, ierr) write(*,*)"processor ",mype," sent ",buffer endif if(mype.eq. DestinationID) then! We are the receiving process Call MPI_Recv(buffer, count, MPI_INTEGER, SenderPE, & tag, MPI_COMM_WORLD, status,ierr) write(*,*)"processor ",mype," got ",buffer endif call MPI_FINALIZE(ierr) stop end

25 Message Order in Point-to-Point Communication The order that messages are sent in is preserved. Therefore if we send three messages from processor 0; a, b and c, then the receiver cannot receive them in any order other than a, b, c. Messages cannot overtake each other. If we communicate with multiple processors the messages can arrive in any intermixed order. Example: If we send a,b,c from process 0 and d, e, f from process 1, then the receiver on process 2 can receive them in any mixed order: a, b, d, e, f, c or a, d, e, b, c, f but not a, c, d, e, f, b as the message order from process 0 would have been broken. For this reason tags are useful when the order of messages are important

26 Testing for Completion of Non-Blocking Communication Call MPI_WAIT(request, status) Blocks the process whilst it waits for the communication specified with the request handle to complete. The request handle is returned by the non-blocking process we wish to check. Useful when you need the data before proceeding forward. Call MPI_WAIT_ALL(count, array_of_requests, array_of_statuses) Blocks the process whilst we wait for all processes in the provided communicator list to finish. Call MPI_TEST(request, flag, status) MPI_TEST only checks to see if the communication has completed but does not block the process. It returns either TRUE or FALSE in the variable flag.

27 Timing in MPI MPI_WTIME() double precision function which returns the wall clock time of the current process, useful for recording the time spent communicating. Note that the wall clock time has no well defined starting point so timings should be relative.

28 Collective Communication Collective communication always includes all processes in a given communicator Thus if we wish to act only on a few processes, then it may be better to define another communicator rather than use MPI_COMM_WORLD or perform point-to-point operations Point-to-point communicators cannot see collective communicators and vice versa (i.e. RECV cannot receive a message from a collective communication) Collective communications are all blocking, there are no nonblocking equivalents All processes within the communicator must call the collective communication subroutine As collective communications are blocking and all processes are involved, there is no need for tags as you can only perform one call at a time The sent message must fill the receive buffer unlike in point-topoint communications

29 MPI_BARRIER Call MPI_BARRIER(comm, ierr) Only blocks all processes until all processes within the communicator are at the MPI_BARRIER call. Used to synchronise all processes.

30 Before After ROOT B ROOT MPI_BCAST B B B B Broadcasts data from the root to all other processes within the communicator All processes must specify the same root and communicator buffer is used to send data on the root and used to receive the data on the receiving processes Call MPI_BCAST(buffer, count, datatype, root, comm, ierr)

31 Before After A A ROOT B ROOT B C D C D MPI_SCATTER A B C D Scatters data from the root to all other processes within the communicator Send_count is the number of elements sent to each process, not the total sent by the root. Send_ variables are dummy variables on receivers and recv_ variables are dummy variables on the root. Call MPI_SCATTER(send_buffer, send_count, send_datatype, recv_buffer, recv_count, recv_datatype, root, comm, ierr)

32 Before After ROOT MPI_GATHER A B C D ROOT A B C D A B C D Gathers a copy of data from all other processes and sends it to the root. All processes must specify the same root and communicator The recv_count will be different to the send_count send_ variables are dummy variables for receivers and vice versa for the recv_ variables Call MPI_GATHER (send_buffer, send_count, send_datatype, recv_buffer, recv_count, recv_datatype, root, comm, ierr)

33 Before After A ROOT ROOT MPI_ALLGATHER A B C D A B C D B C D A B C D A B C D A B C D Gathers copy of data from all processes and sends a copy to all processes Does not have a specific root process Send and receive details are important on all processes and may be different Amount of data sent from each process not necessarily the same Call MPI_ALLGATHER(send_buffer, send_count, send_datatype, recv_buffer, recv_count, recv_datatype, comm, ierr)

34 Before A After A ROOT ROOT MPI_ALL_TO_ALL B C D E F G H I J K L M N O P B C D E F G H I J K L M N O P A E I M B F J N C G K O D H L P Copies the n th element of the buffer array to the n th process, ordering them in the receiving array in ascending order according to which process sent them E.g. 1 st element of every process goes to the first process, ordered according to rank Call MPI_ALL_TO_ALL(send_buffer, send_count, send_datatype, recv_buffer, recv_count, recv_datatype, comm, ierr)

35 Before A B C D After Global Reduction Operations E I M F J N G H K L O P A E I B F J C D G H K L A selected reduction operation can be performed taking the n th buffer element from each process, e.g. SUM, MIN, MAX. The result is stored in the n th receiving element of the receiving buffer The order with which the operator acts on the array elements must not matter, i.e. M N O P AoEoIoM = AoMoEoI = AoIoMoE Reduction Operation

36 Reduction Operators Included With MPI MPI NAME MPI_MAX MPI_MIN MPI_SUM MPI_PRODUCT MPI_LAND MPI_BAND MPI_LOR MPI_BOR MPI_LXOR MPI_BXOR MPI_MAXLOC MPI_MINLOC Function Maximum Minimum Finds the Sum Determines the Product Logical AND Bitwise AND Logical OR Bitwise OR Logical exclusive OR Bitwise exclusive OR Maximum and its location (process rank) Minimum and its location (process rank)

37 Global Reduction Operations Call MPI_REDUCE(send_buffer, recv_buffer, count, datatype, operation_type, root, comm, ierr) - MPI_REDUCE only returns the result from the selected operator to the root process Call MPI_ALLREDUCE(send_buffer, recv_buffer, count, datatype, operation_type, comm, ierr) - MPI_ALLREDUCE returns the result from the selected operator to all processes and therefore no root argument is needed

38 Applying Parallelization to Hydro Codes

39 Ghost Cells Ghost cells or Guard Cells are cells either side of the computational domain that contain information on the boundary conditions. These are needed for some differencing operations - for example the gradient of the pressure: P = P i+1 P i 1 2Δx Ghost cells however are not updated by the advection scheme, unlike the other cells. Instead they are filled with pre-defined boundary data. P i=1 P i=2 P i=3 P i 1 P i P i+1 Guard cells (containing boundary information)

40 If we parallelize the problem and split the computational domain into groups of cells (blocks) then we need ghost cells on each block. Some of the ghost cells will contain the boundary information at the edge of the simulation volume we work in, whilst the others will contain copies of data from neighbouring blocks. Ghost cells that represent regions internal to the simulation volume contain data copied from their neighbouring blocks. Ghost cells on the outside of the computational domain contain user defined boundary conditions

41 Internal Cells Latency Hiding Guardcells containing boundary or neighbouring cell data If we use non-blocking sends we can perform some useful calculations on the internal cells whilst we wait for the transfer of guardcell data. Once we have received the guardcell data we can act upon data at the edges. This works well for processes such as cooling when we do not need to calculate gradients e.g. P As we still perform useful work instead of waiting idly for the communication to complete, we essentially hide the latency.

42 Morton Space Filling Curves Figure: Morton Space Filling Curve (University of Chicago, FLASH Centre for Computational Science) As each block of cells will require boundary data from neighbouring blocks, it is best to place blocks with the most common neighbours on the same CPU. This helps reduce communication across the network. Morton space filling curves like the one shown left determine the most ideal relation between blocks.

43 Labserv You will need to use labserv to compile and run your code Log into the system using: ssh Y student1@labserv.astro.univie.ac.at Labserv has mpif90 installed. Use following command to compile your code: mpif90 program.f90 o program.exe Run your code using the following command: mpirun np 2./program.exe (you can replace 2 with the desired number of processors) Labserv has 8 processors so occasionally you may have to wait for someone else to finish running their program

44 Parallelizing a Hydro Code Determine rank of each process and total number of processes Sub-divide the computational region (400 cells) across the processes and implement initial conditions Determine global minimum time step value Exchange guardcell data before and after hydro time step First try with synchronised sends then use non-blocking buffered sends When simulation is finished, send all of the data to the master processor which should then write the data out in order