Scientific Computing
|
|
- Laurence Berry
- 6 years ago
- Views:
Transcription
1 Lecture on Scientific Computing Dr. Kersten Schmidt Lecture 21 Technische Universität Berlin Institut für Mathematik Wintersemester 2014/2015
2 Syllabus Linear Regression, Fast Fourier transform Modelling by partial differential equations (PDEs) Maxwell, Helmholtz, Poisson, Linear elasticity, Navier-Stokes equation boundary value problem, eigenvalue problem boundary conditions (Dirichlet, Neumann, Robin) handling of infinite domains (wave-guide, homogeneous exterior: DtN, PML) boundary integral equations Computer aided-design (CAD) Mesh generators Space discretisation of PDEs Finite difference method Finite element method Discontinuous Galerkin finite element method Solvers Linear Solvers (direct, iterative), preconditioner Nonlinear Solvers (Newton-Raphson iteration) Eigenvalue Solvers Parallelisation Computer hardware (SIMD, MIMD: shared/distributed memory) Programming in parallel: OpenMP, MPI VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/2015 2
3 Generic distributed memory computer Clusters with partly distributed memory and shared memory A process is an instance of a program that is executing more or less autonomously on a physical processor. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/2015 3
4 Message passing Communication on parallel computers with distributed memory (multicomputers) is most commonly done by message passing. Processes coordinate their activities by explicitely sending and receiving messages. Assume that (as in MPI) processes are statically allocated. That is, the number of processors is set at the beginning of the program execution, and no further processes are created during execution. There is usually one process executing on one processor. Each process is assigned a unique integer rank in the range 0, 1,..., p 1, where p is the number of processes. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/2015 4
5 The Message Passing Interface: MPI Like OpenMP for shared memory programming, MPI is an application programmer interface to message passing. MPI extends programming languages (like C/C++ or Fortran) with a library of functions for point-to-point and collective communication and additional functions for managing the processes at the computation and for querying their status. MPI has become a de facto standard for message passing on multicomputers. Standardization by the MPI forum ( Implementations: MPICH: Open MPI: On clusters with many processors, VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/2015 5
6 First MPI demo program in C: mpi1st.c #include <stdio.h> #include <mpi.h> int main(int argc, char* argv[]) { } int rank; /* rank of process */ int p; /* number of processes */ MPI_Init(&argc, &argv); /* Start up MPI */ MPI_Comm_rank(MPI_COMM_WORLD, &rank); /* Find out proc rank */ MPI_Comm_size(MPI_COMM_WORLD, &p); /* Find out number of processes */ printf("proc %d from %d is ready.\n", rank, p); MPI_Finalize(); /* Shut down MPI */ Calling % mpicc mpi1st.c -o mpi1st % mpirun -np 4 mpi1st Proc 1 from 4 is ready. Proc 2 from 4 is ready. Proc 3 from 4 is ready. Proc 0 from 4 is ready. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/2015 6
7 Simple send and receive commands A typical usage of sending and receiving is given by the following example, where process 0 sends a single float x to process 1. Process 0 executes MPI_Send(&x, 1, MPI_FLOAT, 1, 0, MPI_COMM_WORLD); while process 1 executes MPI_Recv(&x, 1, MPI_FLOAT, 0, 0, MPI_COMM_WORLD, &status); Process 0 and process 1 execute different statements. However, the single-program multi-data (SPMD) programming model permits individual processes executing different statements by means of conditional branches. float x = 0; if (rank == 0) { x = 1; /* e.g. read from an input file */ MPI_Send(&x, 1, MPI_FLOAT, 1, 0, MPI_COMM_WORLD); } else if (rank == 1) MPI_Recv(&x, 1, MPI_FLOAT, 0, 0, MPI_COMM_WORLD, &status); VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/2015 7
8 Simple send and receive commands int MPI_Send(void* buffer /* in */, int count /* in */, MPI_Datatype datatype /* in */ int destination /* in */, int tag /* in */, MPI_Comm communicator /* in */) int MPI_Recv(void* buffer /* out */, int count /* in */, MPI_Datatype datatype /* in */ int destination /* in */, int tag /* in */, MPI_Comm communicator /* in */, MPI_Status* status /* out */) The communicators of MPI_Send and MPI_Recv have to match. The communicator indicates a collection of processes that can send messages to each other. The predefined communicator MPI_COMM_WORLD denotes the set of all processes that participate at the computation. Communicators are an important tool when writing library routines. Defining a communicator that is known only to these routines, messages issued by these routines cannot mix up with messages sent in other parts of the program (even if tags are identical). VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/2015 8
9 Simple send and receive commands The tag in the above example is 0. The tag of the send message must match the expected tag of the receive call. The tag is (can be) used to avoid confusion if several messages are communicated between sender and receiver, e.g., in an iteration. Receiver can wildcard. To receive from any source: use MPI_ANY_SOURCE. To receive with any tag: use MPI_ANY_TAG. The status of MPI_recv returns information on the data that was actually received. status is a (pointer to a) C structure with (at least) three members. status -> MPI_SOURCE status -> MPI_TAG If, e.g., tag or source have been set to be a wildcard, then status->mpi_source and status->mpi_tag return the actual values of these parameters. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/2015 9
10 Another demo program in C: greetings.c #include <stdio.h> #include <mpi.h> int main(int argc, char* argv[]) { int rank; /* rank of process */ int p; /* number of processes */ char message[100]; /* storage of the message */ MPI_Status status; /* return status for receiver */ MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &p); if (rank == 0) { sprintf(message, "Greetings from process %d!", rank); MPI_Send(message, strlen(message) + 1, MPI_CHAR, 0, 0, MPI_COMM_WOLRD); } else for (int source = 1; source < p; ++source) { MPI_Recv(message, 100, MPI_CHAR, source, 0, MPI_COMM_WOLRD, &status); printf("%s\n", message); } MPI_Finalize(); } /* main */ VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
11 Communication modes What happens if a send or receive command is issued? Standard communication : MPI_Send, MPI_Recv Sending and receiving is asynchron, blocking communication. MPI_Recv can be called before the associated MPI_Send is called. MPI_Recv blocks until the message has been received completely, blocks forever if message is not sent. MPI_Send can also be called before the associated MPI_Recv is called. MPI_Send blocks until the message is copied out of memory (wherever). The message might be copied directly into the matching receive buffer, or it might be copied into a temporary system buffer. MPI offers the choice of several communication modes that allow one to control the choice of the communication protocol. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
12 Communication modes What happens if a send or receive command is issued? Synchronous communication (avoid buffering) : MPI_Ssend, MPI_Srecv Sender waits until receiver is ready. Then the system copies from memory to memory. Both processors block forever if messages are not awaited or sent: deadlock, e.g. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
13 Communication modes Non-blocking communication : MPI_Isend, MPI_Irecv Messages are copied into a system buffer on the sender or receiver side (or both). MPI_Irecv returns immediately, even if nothing has been received yet. int MPI_Irecv(... MPI_Comm communicator /* in */, MPI_Request* request /* out */) Additional output parameter request to check whether a message actually has been received (i.e. has been copied in memory) int MPI_Test(MPI_Request* request /* in */, int* flag /* out */, MPI_Status* status /* out */) Instead of waiting of receiving data from other processors some other calculations can be performed in meantime. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
14 Communication modes Non-blocking communication : MPI_Isend, MPI_Irecv Messages are copied into a system buffer on the sender or receiver side (or both). MPI_Irecv returns immediately, even if nothing has been received yet. int MPI_Irecv(... MPI_Comm communicator /* in */, MPI_Request* request /* out */) Additional output parameter request. If, finally, one wants to wait until receiving the message, i.e., switching it to a blocking one, use int MPI_Wait(MPI_Request* request /* in */, MPI_Status* status /* out */) VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
15 Collective modes Let s assume that process 0 reads some input data that it needs to make available to all other processes in the group. We know how process 0 could proceed: for (dest = 1; dest < p; ++dest) MPI_Send(data, size, MPI_INT, dest, tag, MPI_COMM_WOLRD); In this approach p 1 messages are sent, all with the same sender. We know that there are more elegant (and in general more efficient) algorithms to do the above: a tree-structured algorithm (remember the hypercube) VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
16 Broadcast implements the tree-structured algorithm int MPI_Bcast(void* message /* in/out */, int count /* in */, MPI_Datatype datatype /* in */, int root /* in */, MPI_Comm communicator /* in */) Message data is sent from the source process (root) to all other processes. So, data is input data in the source process and output data otherwise. Remark: There is no tag, as broadcasts have been used historically for synchronization. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
17 Reduction: as OpenMP, MPI provides a reduction function (that uses a tree-structured algorithm) int MPI_Reduce(void* operand /* in */, void* result /* out */, int count /* in */, MPI_Datatype datatype /* in */, MPI_Op operator /* in */, int root /* in */, MPI_Comm communicator /* in */) combines the operands and stores the result in *result in process root. Both operand and result refer to count memory locations with data type datatype. MPI_Reduce must be called by all processes in the communicator comm, and count, datatype, operator, and root must be the same in each invocation. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
18 Operations for MPI_Reduce Operation name Meaning MPI_MAX MPI_MIN MPI_SUM MPI_PROD MPI_LAND MPI_BAND MPI_LOR MPI_BOR MPI_LXOR MPI_BXOR MPI_MAXLOC MPI_MINLOC Maximum Minimum Sum Product Logical and Bitwise and Logical or Bitwise or Logical exclusive or Bitwise exclusive or Maximum and location of maximum Minimum and location of minimum VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
19 Example: dot product Serial version float Serial_dot( float x[] /* in */, floag y[] /* in */, int n /* in */) { } float sum = 0.0; for (i = 0; i < n; ++i) sum = sum + x[i] * y[i]; return sum; Parallel version float sum = 0.0; float local_sum = Serial_dot(local_x, local_y, local_n); MPI_Reduce(&local_sum, &sum, 1, MPI_FLOAT, MPI_SUM, 0, MPI_COMM_WORLD); Use MPI_Allreduce instead of MPI_Reduce and all processes get the result (combination of MPI_Reduce and MPI_Bcast). There is no parameter root for MPI_Allreduce. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
20 Matrix-vector multiplication Let s consider what integrients we need to do a matrix-vector multiplication, y = A x. For simplicity we assume that A is a n n square matrix. Then, n 1 y k = a ki x i, 0 k < n. i=0 In OpenMP we could parallize this by #pragma omp parallel for(k = 0; k < n; ++k) { y[k] = 0.0; for(i = 0; i < n; ++i) { y[k] = y[k] + a[k,i] * x[i]; } VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
21 As the outer-most loop is parallized and we access the matrix row-wise, we can visualise the matrix-vector product as follows. This is not quite correct as all processes access all of y. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
22 How can we do this in a distributed memory environment? Let us assume that the matrix A and the vectors x, y are distributed in the block-wise fashion as displayed on the previous page. Let x k R m, y k R m, A k R m n, m = n p, 0 k < p, be portions of x, y, and A, respectively, stored in the process k (usually on processor k). Then, y k = A k x. Thus, each element of the vector y is the result of the inner product of a row of A with the vector x. In order to form the inner product of each row of A with x we either have to gather all of x onto each process or we have to scatter each (block-)row of A across the processes. In our previous OpenMP code the former has been done. If we had parallelized the inner loop then the latter approach had been taken. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
23 Gather (sammeln) vector, take parts from all processors and send to one VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
24 In MPI gathering vector x on process 0 can be done by the call /* Space allocated in calling program */ float local_x[]; /* Local storage for x */ float global_x[]; /* Storage for all x */ MPI_Gather(local_x, n/p, MPI_FLOAT, global_x, n/p, MPI_FLOAT, 0, MPI_COMM_WORLD); The syntax is MPI_Gather(void* send_data /* in */, int send_count /* in */, MPI_Datatype send_type /* in */, void* recv_data /* out */, int recv_count /* in */ MPI_Datatype recv_type /* in */, int dest /* in */, MPI_Comm comm /* in */) MPI_Gather: Collecting pieces of a distributed vector on a single processor. MPI_Allgather: Collecting pieces of a distributed vector on all processors. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
25 The alternative to gathering vector x is to scatter matrix A. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
26 The syntax of MPI_Scatter is int MPI_Scatter(void* send_data /* in */, int send_count /* in */, MPI_Datatype send_type /* in */, void* recv_data /* out */, int recv_count /* in */ MPI_Datatype recv_type /* in */, int origin /* in */, MPI_Comm comm /* in */) MPI_Scatter splits the data referenced by send_data on the process with rank origin in p segments, each of which consists of send_count elements of type send_type. The first segment is sent to process 0, the second to process 1, etc. VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
27 If A is stored block-wise, we have Here, x k R m, y k R m, A k R n m VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
28 Formally, we have to do the following 1. Compute (locally) y k = A k x x, 0 k < p 2. Reduce y = p 1 k=0 y k with root process 0 (for example), 3. Scatter y on the p processes. The last steps can be combined by the call MPI_Reduce_Scatter VL Scientific Computing WS 2014/2015, Dr. K. Schmidt, 02/03/
NUMERICAL PARALLEL COMPUTING
Lecture 5, March 23, 2012: The Message Passing Interface http://people.inf.ethz.ch/iyves/pnc12/ Peter Arbenz, Andreas Adelmann Computer Science Dept, ETH Zürich E-mail: arbenz@inf.ethz.ch Paul Scherrer
More informationScientific Computing
Lecture on Scientific Computing Dr. Kersten Schmidt Lecture 20 Technische Universität Berlin Institut für Mathematik Wintersemester 2014/2015 Syllabus Linear Regression, Fast Fourier transform Modelling
More informationDistributed Systems + Middleware Advanced Message Passing with MPI
Distributed Systems + Middleware Advanced Message Passing with MPI Gianpaolo Cugola Dipartimento di Elettronica e Informazione Politecnico, Italy cugola@elet.polimi.it http://home.dei.polimi.it/cugola
More informationCPS 303 High Performance Computing
CPS 303 High Performance Computing Wensheng Shen Department of Computational Science SUNY Brockport Chapter 5: Collective communication The numerical integration problem in Chapter 4 is not very efficient.
More informationMPI. (message passing, MIMD)
MPI (message passing, MIMD) What is MPI? a message-passing library specification extension of C/C++ (and Fortran) message passing for distributed memory parallel programming Features of MPI Point-to-point
More informationOutline. Communication modes MPI Message Passing Interface Standard
MPI THOAI NAM Outline Communication modes MPI Message Passing Interface Standard TERMs (1) Blocking If return from the procedure indicates the user is allowed to reuse resources specified in the call Non-blocking
More informationOutline. Communication modes MPI Message Passing Interface Standard. Khoa Coâng Ngheä Thoâng Tin Ñaïi Hoïc Baùch Khoa Tp.HCM
THOAI NAM Outline Communication modes MPI Message Passing Interface Standard TERMs (1) Blocking If return from the procedure indicates the user is allowed to reuse resources specified in the call Non-blocking
More informationCS 470 Spring Mike Lam, Professor. Distributed Programming & MPI
CS 470 Spring 2017 Mike Lam, Professor Distributed Programming & MPI MPI paradigm Single program, multiple data (SPMD) One program, multiple processes (ranks) Processes communicate via messages An MPI
More informationMessage Passing Interface
Message Passing Interface by Kuan Lu 03.07.2012 Scientific researcher at Georg-August-Universität Göttingen and Gesellschaft für wissenschaftliche Datenverarbeitung mbh Göttingen Am Faßberg, 37077 Göttingen,
More informationRecap of Parallelism & MPI
Recap of Parallelism & MPI Chris Brady Heather Ratcliffe The Angry Penguin, used under creative commons licence from Swantje Hess and Jannis Pohlmann. Warwick RSE 13/12/2017 Parallel programming Break
More informationMessage Passing Interface. most of the slides taken from Hanjun Kim
Message Passing Interface most of the slides taken from Hanjun Kim Message Passing Pros Scalable, Flexible Cons Someone says it s more difficult than DSM MPI (Message Passing Interface) A standard message
More informationCS 470 Spring Mike Lam, Professor. Distributed Programming & MPI
CS 470 Spring 2018 Mike Lam, Professor Distributed Programming & MPI MPI paradigm Single program, multiple data (SPMD) One program, multiple processes (ranks) Processes communicate via messages An MPI
More informationParallel Programming. Using MPI (Message Passing Interface)
Parallel Programming Using MPI (Message Passing Interface) Message Passing Model Simple implementation of the task/channel model Task Process Channel Message Suitable for a multicomputer Number of processes
More informationMessage Passing Interface
Message Passing Interface DPHPC15 TA: Salvatore Di Girolamo DSM (Distributed Shared Memory) Message Passing MPI (Message Passing Interface) A message passing specification implemented
More informationMPI: Parallel Programming for Extreme Machines. Si Hammond, High Performance Systems Group
MPI: Parallel Programming for Extreme Machines Si Hammond, High Performance Systems Group Quick Introduction Si Hammond, (sdh@dcs.warwick.ac.uk) WPRF/PhD Research student, High Performance Systems Group,
More informationPractical Scientific Computing: Performanceoptimized
Practical Scientific Computing: Performanceoptimized Programming Programming with MPI November 29, 2006 Dr. Ralf-Peter Mundani Department of Computer Science Chair V Technische Universität München, Germany
More informationMPI MESSAGE PASSING INTERFACE
MPI MESSAGE PASSING INTERFACE David COLIGNON, ULiège CÉCI - Consortium des Équipements de Calcul Intensif http://www.ceci-hpc.be Outline Introduction From serial source code to parallel execution MPI functions
More informationMessage Passing Interface
MPSoC Architectures MPI Alberto Bosio, Associate Professor UM Microelectronic Departement bosio@lirmm.fr Message Passing Interface API for distributed-memory programming parallel code that runs across
More informationCollective Communications
Collective Communications Reusing this material This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike 4.0 International License. http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_us
More informationHolland Computing Center Kickstart MPI Intro
Holland Computing Center Kickstart 2016 MPI Intro Message Passing Interface (MPI) MPI is a specification for message passing library that is standardized by MPI Forum Multiple vendor-specific implementations:
More informationCSE 613: Parallel Programming. Lecture 21 ( The Message Passing Interface )
CSE 613: Parallel Programming Lecture 21 ( The Message Passing Interface ) Jesmin Jahan Tithi Department of Computer Science SUNY Stony Brook Fall 2013 ( Slides from Rezaul A. Chowdhury ) Principles of
More informationCS 179: GPU Programming. Lecture 14: Inter-process Communication
CS 179: GPU Programming Lecture 14: Inter-process Communication The Problem What if we want to use GPUs across a distributed system? GPU cluster, CSIRO Distributed System A collection of computers Each
More informationBasic MPI Communications. Basic MPI Communications (cont d)
Basic MPI Communications MPI provides two non-blocking routines: MPI_Isend(buf,cnt,type,dst,tag,comm,reqHandle) buf: source of data to be sent cnt: number of data elements to be sent type: type of each
More informationMPI MESSAGE PASSING INTERFACE
MPI MESSAGE PASSING INTERFACE David COLIGNON CÉCI - Consortium des Équipements de Calcul Intensif http://hpc.montefiore.ulg.ac.be Outline Introduction From serial source code to parallel execution MPI
More informationDistributed Memory Parallel Programming
COSC Big Data Analytics Parallel Programming using MPI Edgar Gabriel Spring 201 Distributed Memory Parallel Programming Vast majority of clusters are homogeneous Necessitated by the complexity of maintaining
More informationHPC Parallel Programing Multi-node Computation with MPI - I
HPC Parallel Programing Multi-node Computation with MPI - I Parallelization and Optimization Group TATA Consultancy Services, Sahyadri Park Pune, India TCS all rights reserved April 29, 2013 Copyright
More informationIPM Workshop on High Performance Computing (HPC08) IPM School of Physics Workshop on High Perfomance Computing/HPC08
IPM School of Physics Workshop on High Perfomance Computing/HPC08 16-21 February 2008 MPI tutorial Luca Heltai Stefano Cozzini Democritos/INFM + SISSA 1 When
More informationWhat s in this talk? Quick Introduction. Programming in Parallel
What s in this talk? Parallel programming methodologies - why MPI? Where can I use MPI? MPI in action Getting MPI to work at Warwick Examples MPI: Parallel Programming for Extreme Machines Si Hammond,
More informationProgramming Using the Message Passing Paradigm
Programming Using the Message Passing Paradigm Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar To accompany the text ``Introduction to Parallel Computing'', Addison Wesley, 2003. Topic Overview
More informationCapstone Project. Project: Middleware for Cluster Computing
Capstone Project Project: Middleware for Cluster Computing Middleware is computer software that connects software components or applications. The software consists of a set of enabling services that allow
More informationTopics. Lecture 7. Review. Other MPI collective functions. Collective Communication (cont d) MPI Programming (III)
Topics Lecture 7 MPI Programming (III) Collective communication (cont d) Point-to-point communication Basic point-to-point communication Non-blocking point-to-point communication Four modes of blocking
More informationMPI Message Passing Interface
MPI Message Passing Interface Portable Parallel Programs Parallel Computing A problem is broken down into tasks, performed by separate workers or processes Processes interact by exchanging information
More informationParallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 4 Message-Passing Programming Learning Objectives n Understanding how MPI programs execute n Familiarity with fundamental MPI functions
More informationParallel programming MPI
Parallel programming MPI Distributed memory Each unit has its own memory space If a unit needs data in some other memory space, explicit communication (often through network) is required Point-to-point
More informationHigh-Performance Computing: MPI (ctd)
High-Performance Computing: MPI (ctd) Adrian F. Clark: alien@essex.ac.uk 2015 16 Adrian F. Clark: alien@essex.ac.uk High-Performance Computing: MPI (ctd) 2015 16 1 / 22 A reminder Last time, we started
More informationStandard MPI - Message Passing Interface
c Ewa Szynkiewicz, 2007 1 Standard MPI - Message Passing Interface The message-passing paradigm is one of the oldest and most widely used approaches for programming parallel machines, especially those
More informationIntroduction to MPI. HY555 Parallel Systems and Grids Fall 2003
Introduction to MPI HY555 Parallel Systems and Grids Fall 2003 Outline MPI layout Sending and receiving messages Collective communication Datatypes An example Compiling and running Typical layout of an
More informationProgramming with MPI Collectives
Programming with MPI Collectives Jan Thorbecke Type to enter text Delft University of Technology Challenge the future Collectives Classes Communication types exercise: BroadcastBarrier Gather Scatter exercise:
More informationCOMP 322: Fundamentals of Parallel Programming
COMP 322: Fundamentals of Parallel Programming https://wiki.rice.edu/confluence/display/parprog/comp322 Lecture 37: Introduction to MPI (contd) Vivek Sarkar Department of Computer Science Rice University
More informationIntroduction to the Message Passing Interface (MPI)
Introduction to the Message Passing Interface (MPI) CPS343 Parallel and High Performance Computing Spring 2018 CPS343 (Parallel and HPC) Introduction to the Message Passing Interface (MPI) Spring 2018
More informationExperiencing Cluster Computing Message Passing Interface
Experiencing Cluster Computing Message Passing Interface Class 6 Message Passing Paradigm The Underlying Principle A parallel program consists of p processes with different address spaces. Communication
More informationParallel Computing Paradigms
Parallel Computing Paradigms Message Passing João Luís Ferreira Sobral Departamento do Informática Universidade do Minho 31 October 2017 Communication paradigms for distributed memory Message passing is
More informationIntroduction to MPI: Part II
Introduction to MPI: Part II Pawel Pomorski, University of Waterloo, SHARCNET ppomorsk@sharcnetca November 25, 2015 Summary of Part I: To write working MPI (Message Passing Interface) parallel programs
More informationSlides prepared by : Farzana Rahman 1
Introduction to MPI 1 Background on MPI MPI - Message Passing Interface Library standard defined by a committee of vendors, implementers, and parallel programmers Used to create parallel programs based
More informationDistributed Memory Programming with MPI
Distributed Memory Programming with MPI Moreno Marzolla Dip. di Informatica Scienza e Ingegneria (DISI) Università di Bologna moreno.marzolla@unibo.it Algoritmi Avanzati--modulo 2 2 Credits Peter Pacheco,
More informationMPI - The Message Passing Interface
MPI - The Message Passing Interface The Message Passing Interface (MPI) was first standardized in 1994. De facto standard for distributed memory machines. All Top500 machines (http://www.top500.org) are
More informationTopics. Lecture 6. Point-to-point Communication. Point-to-point Communication. Broadcast. Basic Point-to-point communication. MPI Programming (III)
Topics Lecture 6 MPI Programming (III) Point-to-point communication Basic point-to-point communication Non-blocking point-to-point communication Four modes of blocking communication Manager-Worker Programming
More informationParallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 4 Message-Passing Programming Learning Objectives Understanding how MPI programs execute Familiarity with fundamental MPI functions
More informationIntroduction to MPI. Ricardo Fonseca. https://sites.google.com/view/rafonseca2017/
Introduction to MPI Ricardo Fonseca https://sites.google.com/view/rafonseca2017/ Outline Distributed Memory Programming (MPI) Message Passing Model Initializing and terminating programs Point to point
More informationMA471. Lecture 5. Collective MPI Communication
MA471 Lecture 5 Collective MPI Communication Today: When all the processes want to send, receive or both Excellent website for MPI command syntax available at: http://www-unix.mcs.anl.gov/mpi/www/ 9/10/2003
More informationChapter 4. Message-passing Model
Chapter 4 Message-Passing Programming Message-passing Model 2 1 Characteristics of Processes Number is specified at start-up time Remains constant throughout the execution of program All execute same program
More informationIntroduction to MPI. SuperComputing Applications and Innovation Department 1 / 143
Introduction to MPI Isabella Baccarelli - i.baccarelli@cineca.it Mariella Ippolito - m.ippolito@cineca.it Cristiano Padrin - c.padrin@cineca.it Vittorio Ruggiero - v.ruggiero@cineca.it SuperComputing Applications
More informationOutline. Introduction to HPC computing. OpenMP MPI. Introduction. Understanding communications. Collective communications. Communicators.
Lecture 8 MPI Outline Introduction to HPC computing OpenMP MPI Introduction Understanding communications Collective communications Communicators Topologies Grouping Data for Communication Input / output
More informationIn the simplest sense, parallel computing is the simultaneous use of multiple computing resources to solve a problem.
1. Introduction to Parallel Processing In the simplest sense, parallel computing is the simultaneous use of multiple computing resources to solve a problem. a) Types of machines and computation. A conventional
More informationLecture 6: Message Passing Interface
Lecture 6: Message Passing Interface Introduction The basics of MPI Some simple problems More advanced functions of MPI A few more examples CA463D Lecture Notes (Martin Crane 2013) 50 When is Parallel
More informationThe Message Passing Interface (MPI): Parallelism on Multiple (Possibly Heterogeneous) CPUs
1 The Message Passing Interface (MPI): Parallelism on Multiple (Possibly Heterogeneous) CPUs http://mpi-forum.org https://www.open-mpi.org/ Mike Bailey mjb@cs.oregonstate.edu Oregon State University mpi.pptx
More informationDepartment of Informatics V. HPC-Lab. Session 4: MPI, CG M. Bader, A. Breuer. Alex Breuer
HPC-Lab Session 4: MPI, CG M. Bader, A. Breuer Meetings Date Schedule 10/13/14 Kickoff 10/20/14 Q&A 10/27/14 Presentation 1 11/03/14 H. Bast, Intel 11/10/14 Presentation 2 12/01/14 Presentation 3 12/08/14
More informationIntroduction in Parallel Programming - MPI Part I
Introduction in Parallel Programming - MPI Part I Instructor: Michela Taufer WS2004/2005 Source of these Slides Books: Parallel Programming with MPI by Peter Pacheco (Paperback) Parallel Programming in
More informationCluster Computing MPI. Industrial Standard Message Passing
MPI Industrial Standard Message Passing MPI Features Industrial Standard Highly portable Widely available SPMD programming model Synchronous execution MPI Outer scope int MPI_Init( int *argc, char ** argv)
More informationCollective Communications I
Collective Communications I Ned Nedialkov McMaster University Canada CS/SE 4F03 January 2016 Outline Introduction Broadcast Reduce c 2013 16 Ned Nedialkov 2/14 Introduction A collective communication involves
More informationNon-Blocking Communications
Non-Blocking Communications Deadlock 1 5 2 3 4 Communicator 0 2 Completion The mode of a communication determines when its constituent operations complete. - i.e. synchronous / asynchronous The form of
More informationLesson 1. MPI runs on distributed memory systems, shared memory systems, or hybrid systems.
The goals of this lesson are: understanding the MPI programming model managing the MPI environment handling errors point-to-point communication 1. The MPI Environment Lesson 1 MPI (Message Passing Interface)
More informationCS 470 Spring Mike Lam, Professor. Distributed Programming & MPI
CS 470 Spring 2019 Mike Lam, Professor Distributed Programming & MPI MPI paradigm Single program, multiple data (SPMD) One program, multiple processes (ranks) Processes communicate via messages An MPI
More informationPyConZA High Performance Computing with Python. Kevin Colville Python on large clusters with MPI
PyConZA 2012 High Performance Computing with Python Kevin Colville Python on large clusters with MPI Andy Rabagliati Python to read and store data on CHPC Petabyte data store www.chpc.ac.za High Performance
More informationIntroduction to parallel computing concepts and technics
Introduction to parallel computing concepts and technics Paschalis Korosoglou (support@grid.auth.gr) User and Application Support Unit Scientific Computing Center @ AUTH Overview of Parallel computing
More informationDistributed Memory Systems: Part IV
Chapter 5 Distributed Memory Systems: Part IV Max Planck Institute Magdeburg Jens Saak, Scientific Computing II 293/342 The Message Passing Interface is a standard for creation of parallel programs using
More informationMessage-Passing Environments & Systems
Message-Passing Environments & Systems Origins of Cluster Computing or Commodity Supercomputing: Limitations of Homogeneous Supercomputing Systems Heterogeneous Computing (HC) Broad Issues in Heterogeneous
More informationMessage-Passing Computing
Chapter 2 Slide 41þþ Message-Passing Computing Slide 42þþ Basics of Message-Passing Programming using userlevel message passing libraries Two primary mechanisms needed: 1. A method of creating separate
More informationHigh Performance Computing Course Notes Message Passing Programming I
High Performance Computing Course Notes 2008-2009 2009 Message Passing Programming I Message Passing Programming Message Passing is the most widely used parallel programming model Message passing works
More informationMPI 5. CSCI 4850/5850 High-Performance Computing Spring 2018
MPI 5 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning Objectives
More informationMPI Tutorial Part 1 Design of Parallel and High-Performance Computing Recitation Session
S. DI GIROLAMO [DIGIROLS@INF.ETHZ.CH] MPI Tutorial Part 1 Design of Parallel and High-Performance Computing Recitation Session Slides credits: Pavan Balaji, Torsten Hoefler https://htor.inf.ethz.ch/teaching/mpi_tutorials/ppopp13/2013-02-24-ppopp-mpi-basic.pdf
More informationCS 426. Building and Running a Parallel Application
CS 426 Building and Running a Parallel Application 1 Task/Channel Model Design Efficient Parallel Programs (or Algorithms) Mainly for distributed memory systems (e.g. Clusters) Break Parallel Computations
More informationME964 High Performance Computing for Engineering Applications
ME964 High Performance Computing for Engineering Applications Parallel Computing with MPI Building/Debugging MPI Executables MPI Send/Receive Collective Communications with MPI April 10, 2012 Dan Negrut,
More informationReview of MPI Part 2
Review of MPI Part Russian-German School on High Performance Computer Systems, June, 7 th until July, 6 th 005, Novosibirsk 3. Day, 9 th of June, 005 HLRS, University of Stuttgart Slide Chap. 5 Virtual
More informationMPI Tutorial Part 1 Design of Parallel and High-Performance Computing Recitation Session
S. DI GIROLAMO [DIGIROLS@INF.ETHZ.CH] MPI Tutorial Part 1 Design of Parallel and High-Performance Computing Recitation Session Slides credits: Pavan Balaji, Torsten Hoefler https://htor.inf.ethz.ch/teaching/mpi_tutorials/ppopp13/2013-02-24-ppopp-mpi-basic.pdf
More informationPROGRAMMING WITH MESSAGE PASSING INTERFACE. J. Keller Feb 26, 2018
PROGRAMMING WITH MESSAGE PASSING INTERFACE J. Keller Feb 26, 2018 Structure Message Passing Programs Basic Operations for Communication Message Passing Interface Standard First Examples Collective Communication
More informationLecture 4 Introduction to MPI
CS075 1896 Lecture 4 Introduction to MPI Jeremy Wei Center for HPC, SJTU Mar 13th, 2017 1920 1987 2006 Recap of the last lecture (OpenMP) OpenMP is a standardized pragma-based intra-node parallel programming
More informationMPI point-to-point communication
MPI point-to-point communication Slides Sebastian von Alfthan CSC Tieteen tietotekniikan keskus Oy CSC IT Center for Science Ltd. Introduction MPI processes are independent, they communicate to coordinate
More informationIntermediate MPI. M. D. Jones, Ph.D. Center for Computational Research University at Buffalo State University of New York
Intermediate MPI M. D. Jones, Ph.D. Center for Computational Research University at Buffalo State University of New York High Performance Computing I, 2008 M. D. Jones, Ph.D. (CCR/UB) Intermediate MPI
More informationThe Message Passing Interface (MPI) TMA4280 Introduction to Supercomputing
The Message Passing Interface (MPI) TMA4280 Introduction to Supercomputing NTNU, IMF January 16. 2017 1 Parallelism Decompose the execution into several tasks according to the work to be done: Function/Task
More informationCEE 618 Scientific Parallel Computing (Lecture 5): Message-Passing Interface (MPI) advanced
1 / 32 CEE 618 Scientific Parallel Computing (Lecture 5): Message-Passing Interface (MPI) advanced Albert S. Kim Department of Civil and Environmental Engineering University of Hawai i at Manoa 2540 Dole
More informationCornell Theory Center. Discussion: MPI Collective Communication I. Table of Contents. 1. Introduction
1 of 18 11/1/2006 3:59 PM Cornell Theory Center Discussion: MPI Collective Communication I This is the in-depth discussion layer of a two-part module. For an explanation of the layers and how to navigate
More informationNon-Blocking Communications
Non-Blocking Communications Reusing this material This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike 4.0 International License. http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_us
More informationProgramming Using the Message-Passing Paradigm (Chapter 6) Alexandre David
Programming Using the Message-Passing Paradigm (Chapter 6) Alexandre David 1.2.05 1 Topic Overview Principles of Message-Passing Programming MPI: the Message Passing Interface Topologies and Embedding
More informationWorking with IITJ HPC Environment
Working with IITJ HPC Environment by Training Agenda for 23 Dec 2011 1. Understanding Directory structure of IITJ HPC 2. User vs root 3. What is bash_profile 4. How to install any source code in your user
More informationCS 6230: High-Performance Computing and Parallelization Introduction to MPI
CS 6230: High-Performance Computing and Parallelization Introduction to MPI Dr. Mike Kirby School of Computing and Scientific Computing and Imaging Institute University of Utah Salt Lake City, UT, USA
More informationHigh Performance Computing
High Performance Computing Course Notes 2009-2010 2010 Message Passing Programming II 1 Communications Point-to-point communications: involving exact two processes, one sender and one receiver For example,
More information4. Parallel Programming with MPI
4. Parallel Programming with MPI 4. Parallel Programming with MPI... 4.. MPI: Basic Concepts and Definitions...3 4... The Concept of Parallel Program...3 4..2. Data Communication Operations...3 4..3. Communicators...3
More informationMPI Tutorial. Shao-Ching Huang. IDRE High Performance Computing Workshop
MPI Tutorial Shao-Ching Huang IDRE High Performance Computing Workshop 2013-02-13 Distributed Memory Each CPU has its own (local) memory This needs to be fast for parallel scalability (e.g. Infiniband,
More informationChip Multiprocessors COMP Lecture 9 - OpenMP & MPI
Chip Multiprocessors COMP35112 Lecture 9 - OpenMP & MPI Graham Riley 14 February 2018 1 Today s Lecture Dividing work to be done in parallel between threads in Java (as you are doing in the labs) is rather
More informationBryan Carpenter, School of Computing
Bryan Carpenter, School of Computing 1 Plan Brief reprise of parallel computers and programming models Overview of MPI, with illustrative example. Brief comparison with OpenMP Use of MPI in GADGET 2 2
More informationThe Message Passing Interface (MPI): Parallelism on Multiple (Possibly Heterogeneous) CPUs
1 The Message Passing Interface (MPI): Parallelism on Multiple (Possibly Heterogeneous) s http://mpi-forum.org https://www.open-mpi.org/ Mike Bailey mjb@cs.oregonstate.edu Oregon State University mpi.pptx
More informationPeter Pacheco. Chapter 3. Distributed Memory Programming with MPI. Copyright 2010, Elsevier Inc. All rights Reserved
An Introduction to Parallel Programming Peter Pacheco Chapter 3 Distributed Memory Programming with MPI 1 Roadmap Writing your first MPI program. Using the common MPI functions. The Trapezoidal Rule in
More informationDistributed Memory Programming with MPI. Copyright 2010, Elsevier Inc. All rights Reserved
An Introduction to Parallel Programming Peter Pacheco Chapter 3 Distributed Memory Programming with MPI 1 Roadmap Writing your first MPI program. Using the common MPI functions. The Trapezoidal Rule in
More informationCSE 160 Lecture 15. Message Passing
CSE 160 Lecture 15 Message Passing Announcements 2013 Scott B. Baden / CSE 160 / Fall 2013 2 Message passing Today s lecture The Message Passing Interface - MPI A first MPI Application The Trapezoidal
More informationIntroduction to MPI. Jerome Vienne Texas Advanced Computing Center January 10 th,
Introduction to MPI Jerome Vienne Texas Advanced Computing Center January 10 th, 2013 Email: viennej@tacc.utexas.edu 1 Course Objectives & Assumptions Objectives Teach basics of MPI-Programming Share information
More informationMPI Collective communication
MPI Collective communication CPS343 Parallel and High Performance Computing Spring 2018 CPS343 (Parallel and HPC) MPI Collective communication Spring 2018 1 / 43 Outline 1 MPI Collective communication
More informationPoint-to-Point Communication. Reference:
Point-to-Point Communication Reference: http://foxtrot.ncsa.uiuc.edu:8900/public/mpi/ Introduction Point-to-point communication is the fundamental communication facility provided by the MPI library. Point-to-point
More informationMPI Programming. Henrik R. Nagel Scientific Computing IT Division
1 MPI Programming Henrik R. Nagel Scientific Computing IT Division 2 Outline Introduction Basic MPI programming Examples Finite Difference Method Finite Element Method LU Factorization Monte Carlo Method
More informationAn Introduction to Parallel Programming
Guide 48 Version 2 An Introduction to Parallel Programming Document code: Guide 48 Title: An Introduction to Parallel Programming Version: 2 Date: 31/01/2011 Produced by: University of Durham Information
More information