Hybrid MPI/OpenMP parallelization. Recall: MPI uses processes for parallelism. Each process has its own, separate address space.

Size: px
Start display at page:

Download "Hybrid MPI/OpenMP parallelization. Recall: MPI uses processes for parallelism. Each process has its own, separate address space."

Transcription

1 Hybrid MPI/OpenMP parallelization Recall: MPI uses processes for parallelism. Each process has its own, separate address space. Thread parallelism (such as OpenMP or Pthreads) can provide additional parallelism using a shared-memory model within each process As you know: OpenMP is simple and convenient for loop parallelization. In Pthreads the threads have to be created and managed explicitly by the user. This allows more complex and flexible models of parallelism. 1

2 Parallel Computers are Multicore Today s computers are multicore, i.e., shared memory machines. Today s supercomputers are made from commodity components, i.e., they are multicore machines connected by a fast network (for MPI). Supercomputers can therefore be used today using Pure MPI, i.e., MPI over the complete machine. MPI may use the fast network inbetween nodes and can move data using the shared memory within a node (may depend on the configuration!). For a large number of ranks (10 6 ) a scalable MPI may be required. Hybrid MPI/OpenMP, i.e., MPI between nodes and OpenMP within the node. Hybrid MPI/Pthreads, i.e., MPI between nodes and Pthreads within the node. Hybrid MPI/Pthreads/OpenMP, i.e., MPI between nodes and Pthreads+OpenMP within the node. Other configurations, e.g., mixed models auch as: # cores = 2x # MPI ranks, and each MPI rank spawns two threads. Hybrid programming can also be MPI+UPC threads. 2

3 Hybrid MPI/OpenMP parallelization Hybrid configurations, such as # Cores = n # MPI-ranks, combined with # Threads = n MPI-ranks make sense if the algorithmic building parallized with OpenMP does not scale beyond n threads. It has been observed that hybrid MPI/OpenMP can be SLOWER than pure MPI: Typically all threads are sleeping while master communicates (or even worse: critical region). Using MPI and OpenMP in one program is also called hybrid MPI/OpenMP programming. 3

4 Hybrid MPI/OpenMP parallelization MPI defines four possible levels of thread safety (0,1,2,3). The thread level is a parameter to MPI Init thread which replaces MPI Init The user promises to use hybrid programming according to the thread level. Errors will result if the promise is not kept. Used should use MPI Init thread(requested, &provided) instead of MPI Init. The user selects the level he asks for and the MPI library returns the level it supports. int required=mpi_thread_serialized; // Required level of MPI threading support int provided; // Provided level of MPI threading support MPI_Init_thread(&argc, &argv, required, &provided); Usually MPI Init thread is called before any OpenMP parallel region is entered. If MPI Init thread is called inside a parallel region then put the call inside a master region. The calling thread is also called the thread main (master). 4

5 Enable threading support in MPI lib Threading support sometimes needs to be enabled when compiling the MPI library. In OpenMPI configure --enable-mpi-threads will turn on support for MPI THREAD MULTIPLE. Additional option is --enable-progress-threads. 5

6 MPI Threading Support MPI libraries can (but do not necessary do) support different levels of threading. Thread levels are (0,1,2,3): MPI THREAD SINGLE: No multithreading, only one thread exists in the application. MPI THREAD FUNNELED: Multithreading, but only one thread may call MPI functions, i.e., the thread which has called MPI Init thread. MPI THREAD SERIALIZED: Multithreading, but only one thread at a time makes MPI calls. MPI THREAD MULTIPLE: Multithreading, and any thread can call MPI functions at any time. (some restrictions still apply). Using modules on the cluster module add gcc/4.8.2 openmpi/gcc/64/1.8.2 supports level 2. 6

7 MPI-2 Functions for Multithreading There are special MPI-2 function for multi-threaded use: int MPI_Init_thread(int * argc, char ** argv[], int required, int *provided); // Instead of MPI_Init int MPI_Query_thread(int *provided); // Returns the current level of thread support int MPI_Is_main_thread(int *flag); // Returns if the current thread is the one which called MPI_Init_thread // (the master) 7

8 Thread Levels: What is allowed (1) MPI THREAD SINGLE: No OpenMP multithreading is used. MPI THREAD FUNNELED: Only the OpenMP master thread makes calls to MPI functions: All MPI calls are, // MPI calls here #pragma omp parallel { // no MPi calls here #pragma omp master { // MPI calls here } } i.e., outside of an OpenMP parallel region, or inside an OpenMP master regions, or Verifies by calling MPI Is thread main that the calling thread is the one which called MPI Init thread. 8

9 Thread Levels: What is allowed (2) MPI THREAD SERIALIZED: #pragma omp parallel { // no MPi calls here } #pragma omp critical { // all MPI calls here } MPI THREAD MULTIPLE: // Any thread may call MPI functions with no restrictions at any time. Inofficial level THREAD MASTERONLY: MPI processes multithreaded but only master thread will call MPI lib AND other threads are sleeping. 9

10 Details on MPI THREAD MULTIPLE When multiple threads MPI functions in parallel then the outcome will be as if the calls executed sequentially in some sequential order Blocking MPI calls will block only the calling thread and will not prevent other threads from running or executing MPI functions It is the user s responsibility to prevent race conditions when threads in the same application post conflicting MPI calls, e.g., by accessing an info/lock object from one thread and freeing it from another thread. User must ensure that collective operations on the same communicator correctly ordered among threads: Cannot call a broadcast on one thread and a reduce on another thread on the same communicator. 10

11 Details on MPI THREAD MULTIPLE An implementation is not required to support levels higher than MPI THREAD SINGLE; that is, an implementation is not required to be thread safe A fully thread-safe implementation will support MPI THREAD MULTIPLE A program that calls MPI Init (instead of MPI Init thread) should assume that only MPI THREAD SINGLE is supported A threaded MPI program MUST CALL MPI Init thread. 11

12 Incorrect use of MPI THREAD MULTIPLE Process 0 Process 1 Thread 1 MPI_Bcast(MPI_COMM_WORLD) MPI_Bcast(MPI_COMM_WORLD) Thread 2 MPI_Barrier(MPI_COMM_WORLD) MPI_Barrier(MPI_COMM_WORLD) The user is responsible to ensure that either thread 1 or thread 2 gets scheduled first on both processes since otherwise a broadcast may be called combined with a barrier on the same communicator, which will block the program. 12

13 Using MPI THREAD FUNNELED Caveat: Calling MPI inside OpenMP master may not be enough due to lack of synchronization in OpenMP master; Therefore an OpenMP barrier is necessary to guarantee that data from all threads is available before the MPI call (barrier flushes all data) #pragma omp barrier #pragma omp master MPI_Bcast(...) #pragma omp barrier 13

14 Remarks on the performance of MPI THREAD MULTIPLE Using MPI THREAD MULTIPLE seems a simple, desirable solution. But there is an overhead for the thread safety since the MPI implementation always must protect data structures and parts of code, e.g., with OpenMP critical sections. The user knows more about the algorithm than the MPI library can. Also remember: MPI is hard to program but threading is hard to debug. Therefore MPI+threading is hard to program AND hard to debug. 14

15 Hybrid MPI/OpenMP parallelization ~]$ mpirun -np 1./a.out MPI: Hello world! level 0 MPI: rank=0 size=1 OMP: Hello World from thread = 0 OMP: Number of threads = 1 [rheinba@node001 ~]$ mpirun -np 2./a.out MPI: Hello world! level 0 MPI: Hello world! level 0 MPI: rank=1 size=2 OMP: Hello World from thread = 0 OMP: Number of threads = 1 MPI: rank=0 size=2 OMP: Hello World from thread = 0 OMP: Number of threads = 1 15

16 Remark on MPI-3 shared memory In MPI-3 explicit shared memory communication is possible using windows. MPI Win allocate share can be used to define a shared memory window: Create an MPI Window object for one-sided communication and shared memory access, and allocate memory at each process. Synchronization function MPI Win fence. MPI_Win_allocate_shared (count, size, MPI_INFO_NULL, comm, &base_ptr, &win_sm) Input Parameters count: size of window in bytes (nonnegative integer) size: local unit size for displacements, in bytes (positive integer) info: info argument (handle) comm: communicator (must be a set of ranks that can create a shared memory segmen Output Parameters baseptr: initial address of window (choice) win: window object returned by the call (handle) base prt can be used to write/read. 16

17 MPI/OpenMP on the Cluster Not all MPI libraries may be compiled with threading support. Use module add gcc/4.8.2 openmpi/gcc/64/1.8.2 mpicc -fopenmp mpi_omp_hello.c 1 MPI process, 1 thread always works: [rheinba@login01 ~]$ mpirun -np 1./a.out MPI_THREAD_SINGLE:0 MPI_THREAD_FUNNELED:1 MPI_THREAD_SERIALIZED:2 MPI_THREAD_MULTIPLE:3 MPI: Hello world! I support level 2 MPI: rank=0 size=1 OMP: Hello World from thread = 0 OMP: Number of threads = 1 17

18 1 MPI process, 1 thread for each MPI process [rheinba@login01 ~]$ mpirun -np 2./a.out MPI_THREAD_SINGLE:0 MPI_THREAD_SINGLE:0 MPI_THREAD_FUNNELED:1 MPI_THREAD_SERIALIZED:2 MPI_THREAD_MULTIPLE:3 MPI: Hello world! I support level 2 MPI: rank=0 size=2 OMP: Hello World from thread = 0 OMP: Number of threads = 1 MPI_THREAD_FUNNELED:1 MPI_THREAD_SERIALIZED:2 MPI_THREAD_MULTIPLE:3 MPI: Hello world! I support level 2 MPI: rank=1 size=2 OMP: Hello World from thread = 0 OMP: Number of threads = 1 18

19 PBS-Queuing System select=number of chunks ncpus=cpu cores (not cpus!) in a chunk mpiprocs=mpi ranks in each chunk Number of cores is select x ncpus. #PBS -l select=2:ncpus=1:mem=2gb:mpiprocs=1 Selects 2 cpu cores; 2 MPI ranks, each on one cpu core #PBS -l select=2:ncpus=12:mem=2gb:mpiprocs=1 Selects 24 cpu cores; 2 MPI ranks, 12 threads: [rheinba@login01 ~]$ cat pbs_out.txt node130 node

20 PBS-Queuing System (2) Create a PBS file: #!/bin/bash #PBS -N MPI #PBS -l select=10:ncpus=1:mem=2gb:mpiprocs=1 #PBS -l walltime=00:02:00 #PBS -m abe #PBS -M oliver.rheinbach@math.tu-freiberg.de #PBS -o pbs_out.txt #PBS -e pbs_err.txt module add gcc/4.8.2 openmpi/gcc/64/1.8.2 PBS_O_WORKDIR=$HOME cd $PBS_O_WORKDIR cat $PBS_NODEFILE echo mpirun./mpi_first Creates 10 MPI ranks. No threading. All ranks on one node. 20

21 PBS-Queuing System (3) Reading the output: ~]$ cat pbs_out.txt node130 node130 node130 node130 node130 node130 node130 node130 node130 node130 Hello world! rank=0 size=10 Hello world! Hello world! rank=2 size=10 rank=1 size=10 21

22 Hello world! Hello world! rank=9 size=10 rank=8 size=10 Hello world! rank=4 size=10 Hello world! rank=5 size=10 Hello world! rank=6 size=10 Hello world! rank=3 size=10 Hello world! rank=7 size=10 22

23 More in the exercise! 23

Lecture 36: MPI, Hybrid Programming, and Shared Memory. William Gropp

Lecture 36: MPI, Hybrid Programming, and Shared Memory. William Gropp Lecture 36: MPI, Hybrid Programming, and Shared Memory William Gropp www.cs.illinois.edu/~wgropp Thanks to This material based on the SC14 Tutorial presented by Pavan Balaji William Gropp Torsten Hoefler

More information

OpenMP and MPI. Parallel and Distributed Computing. Department of Computer Science and Engineering (DEI) Instituto Superior Técnico.

OpenMP and MPI. Parallel and Distributed Computing. Department of Computer Science and Engineering (DEI) Instituto Superior Técnico. OpenMP and MPI Parallel and Distributed Computing Department of Computer Science and Engineering (DEI) Instituto Superior Técnico November 15, 2010 José Monteiro (DEI / IST) Parallel and Distributed Computing

More information

OpenMP and MPI. Parallel and Distributed Computing. Department of Computer Science and Engineering (DEI) Instituto Superior Técnico.

OpenMP and MPI. Parallel and Distributed Computing. Department of Computer Science and Engineering (DEI) Instituto Superior Técnico. OpenMP and MPI Parallel and Distributed Computing Department of Computer Science and Engineering (DEI) Instituto Superior Técnico November 16, 2011 CPD (DEI / IST) Parallel and Distributed Computing 18

More information

MPI & OpenMP Mixed Hybrid Programming

MPI & OpenMP Mixed Hybrid Programming MPI & OpenMP Mixed Hybrid Programming Berk ONAT İTÜ Bilişim Enstitüsü 22 Haziran 2012 Outline Introduc/on Share & Distributed Memory Programming MPI & OpenMP Advantages/Disadvantages MPI vs. OpenMP Why

More information

MPI and OpenMP. Mark Bull EPCC, University of Edinburgh

MPI and OpenMP. Mark Bull EPCC, University of Edinburgh 1 MPI and OpenMP Mark Bull EPCC, University of Edinburgh markb@epcc.ed.ac.uk 2 Overview Motivation Potential advantages of MPI + OpenMP Problems with MPI + OpenMP Styles of MPI + OpenMP programming MPI

More information

Advanced MPI Programming

Advanced MPI Programming Advanced MPI Programming Latestslidesandcodeexamplesareavailableat www.mcs.anl.gov/~thakur/sc13-mpi-tutorial Pavan%Balaji% Argonne'Na*onal'Laboratory' Email:'balaji@mcs.anl.gov' Web:'www.mcs.anl.gov/~balaji'

More information

Implementation of Parallelization

Implementation of Parallelization Implementation of Parallelization OpenMP, PThreads and MPI Jascha Schewtschenko Institute of Cosmology and Gravitation, University of Portsmouth May 9, 2018 JAS (ICG, Portsmouth) Implementation of Parallelization

More information

Our new HPC-Cluster An overview

Our new HPC-Cluster An overview Our new HPC-Cluster An overview Christian Hagen Universität Regensburg Regensburg, 15.05.2009 Outline 1 Layout 2 Hardware 3 Software 4 Getting an account 5 Compiling 6 Queueing system 7 Parallelization

More information

COMP528: Multi-core and Multi-Processor Computing

COMP528: Multi-core and Multi-Processor Computing COMP528: Multi-core and Multi-Processor Computing Dr Michael K Bane, G14, Computer Science, University of Liverpool m.k.bane@liverpool.ac.uk https://cgi.csc.liv.ac.uk/~mkbane/comp528 2X So far Why and

More information

Performance Considerations: Hardware Architecture

Performance Considerations: Hardware Architecture Performance Considerations: Hardware Architecture John Zollweg Parallel Optimization and Scientific Visualization for Ranger March 13, 29 based on material developed by Kent Milfeld, TACC www.cac.cornell.edu

More information

Communication and Optimization Aspects of Parallel Programming Models on Hybrid Architectures

Communication and Optimization Aspects of Parallel Programming Models on Hybrid Architectures Communication and Optimization Aspects of Parallel Programming Models on Hybrid Architectures Rolf Rabenseifner rabenseifner@hlrs.de Gerhard Wellein gerhard.wellein@rrze.uni-erlangen.de University of Stuttgart

More information

Advanced Message-Passing Interface (MPI)

Advanced Message-Passing Interface (MPI) Outline of the workshop 2 Advanced Message-Passing Interface (MPI) Bart Oldeman, Calcul Québec McGill HPC Bart.Oldeman@mcgill.ca Morning: Advanced MPI Revision More on Collectives More on Point-to-Point

More information

COMP4510 Introduction to Parallel Computation. Shared Memory and OpenMP. Outline (cont d) Shared Memory and OpenMP

COMP4510 Introduction to Parallel Computation. Shared Memory and OpenMP. Outline (cont d) Shared Memory and OpenMP COMP4510 Introduction to Parallel Computation Shared Memory and OpenMP Thanks to Jon Aronsson (UofM HPC consultant) for some of the material in these notes. Outline (cont d) Shared Memory and OpenMP Including

More information

High Performance Computing: Tools and Applications

High Performance Computing: Tools and Applications High Performance Computing: Tools and Applications Edmond Chow School of Computational Science and Engineering Georgia Institute of Technology Lecture 20 Shared memory computers and clusters On a shared

More information

Message Passing Interface. George Bosilca

Message Passing Interface. George Bosilca Message Passing Interface George Bosilca bosilca@icl.utk.edu Message Passing Interface Standard http://www.mpi-forum.org Current version: 3.1 All parallelism is explicit: the programmer is responsible

More information

Message Passing with MPI

Message Passing with MPI Message Passing with MPI PPCES 2016 Hristo Iliev IT Center / JARA-HPC IT Center der RWTH Aachen University Agenda Motivation Part 1 Concepts Point-to-point communication Non-blocking operations Part 2

More information

Introduction to MPI+OpenMP hybrid programming. SuperComputing Applications and Innovation Department

Introduction to MPI+OpenMP hybrid programming. SuperComputing Applications and Innovation Department Introduction to MPI+OpenMP hybrid programming SuperComputing Applications and Innovation Department Basic concepts Architectural trend Architectural trend In a nutshell: memory per core decreases memory

More information

Introduction to OpenMP. OpenMP basics OpenMP directives, clauses, and library routines

Introduction to OpenMP. OpenMP basics OpenMP directives, clauses, and library routines Introduction to OpenMP Introduction OpenMP basics OpenMP directives, clauses, and library routines What is OpenMP? What does OpenMP stands for? What does OpenMP stands for? Open specifications for Multi

More information

Lecture 14: Mixed MPI-OpenMP programming. Lecture 14: Mixed MPI-OpenMP programming p. 1

Lecture 14: Mixed MPI-OpenMP programming. Lecture 14: Mixed MPI-OpenMP programming p. 1 Lecture 14: Mixed MPI-OpenMP programming Lecture 14: Mixed MPI-OpenMP programming p. 1 Overview Motivations for mixed MPI-OpenMP programming Advantages and disadvantages The example of the Jacobi method

More information

Hybrid Computing. Lars Koesterke. University of Porto, Portugal May 28-29, 2009

Hybrid Computing. Lars Koesterke. University of Porto, Portugal May 28-29, 2009 Hybrid Computing Lars Koesterke University of Porto, Portugal May 28-29, 29 Why Hybrid? Eliminates domain decomposition at node Automatic coherency at node Lower memory latency and data movement within

More information

Shared Memory programming paradigm: openmp

Shared Memory programming paradigm: openmp IPM School of Physics Workshop on High Performance Computing - HPC08 Shared Memory programming paradigm: openmp Luca Heltai Stefano Cozzini SISSA - Democritos/INFM

More information

Hybrid Model Parallel Programs

Hybrid Model Parallel Programs Hybrid Model Parallel Programs Charlie Peck Intermediate Parallel Programming and Cluster Computing Workshop University of Oklahoma/OSCER, August, 2010 1 Well, How Did We Get Here? Almost all of the clusters

More information

PRACE Autumn School Basic Programming Models

PRACE Autumn School Basic Programming Models PRACE Autumn School 2010 Basic Programming Models Basic Programming Models - Outline Introduction Key concepts Architectures Programming models Programming languages Compilers Operating system & libraries

More information

Multithreading in C with OpenMP

Multithreading in C with OpenMP Multithreading in C with OpenMP ICS432 - Spring 2017 Concurrent and High-Performance Programming Henri Casanova (henric@hawaii.edu) Pthreads are good and bad! Multi-threaded programming in C with Pthreads

More information

Issues in Developing a Thread-Safe MPI Implementation. William Gropp Rajeev Thakur

Issues in Developing a Thread-Safe MPI Implementation. William Gropp Rajeev Thakur Issues in Developing a Thread-Safe MPI Implementation William Gropp Rajeev Thakur Why MPI and Threads Applications need more computing power than any one processor can provide Parallel computing (proof

More information

Introduction to GALILEO

Introduction to GALILEO Introduction to GALILEO Parallel & production environment Mirko Cestari m.cestari@cineca.it Alessandro Marani a.marani@cineca.it Domenico Guida d.guida@cineca.it Maurizio Cremonesi m.cremonesi@cineca.it

More information

OpenMPand the PGAS Model. CMSC714 Sept 15, 2015 Guest Lecturer: Ray Chen

OpenMPand the PGAS Model. CMSC714 Sept 15, 2015 Guest Lecturer: Ray Chen OpenMPand the PGAS Model CMSC714 Sept 15, 2015 Guest Lecturer: Ray Chen LastTime: Message Passing Natural model for distributed-memory systems Remote ( far ) memory must be retrieved before use Programmer

More information

Hybrid MPI and OpenMP Parallel Programming

Hybrid MPI and OpenMP Parallel Programming Hybrid MPI and OpenMP Parallel Programming Jemmy Hu SHARCNET HPTC Consultant July 8, 2015 Objectives difference between message passing and shared memory models (MPI, OpenMP) why or why not hybrid? a common

More information

Introduction to MPI-2 (Message-Passing Interface)

Introduction to MPI-2 (Message-Passing Interface) Introduction to MPI-2 (Message-Passing Interface) What are the major new features in MPI-2? Parallel I/O Remote Memory Operations Dynamic Process Management Support for Multithreading Parallel I/O Includes

More information

Scientific Programming in C XIV. Parallel programming

Scientific Programming in C XIV. Parallel programming Scientific Programming in C XIV. Parallel programming Susi Lehtola 11 December 2012 Introduction The development of microchips will soon reach the fundamental physical limits of operation quantum coherence

More information

Multicore Architecture and Hybrid Programming

Multicore Architecture and Hybrid Programming Multicore Architecture and Hybrid Programming Rebecca Hartman-Baker Oak Ridge National Laboratory hartmanbakrj@ornl.gov 2004-2009 Rebecca Hartman-Baker. Reproduction permitted for non-commercial, educational

More information

CS420: Operating Systems

CS420: Operating Systems Threads James Moscola Department of Physical Sciences York College of Pennsylvania Based on Operating System Concepts, 9th Edition by Silberschatz, Galvin, Gagne Threads A thread is a basic unit of processing

More information

Issues in Developing a Thread-Safe MPI Implementation. William Gropp Rajeev Thakur Mathematics and Computer Science Division

Issues in Developing a Thread-Safe MPI Implementation. William Gropp Rajeev Thakur Mathematics and Computer Science Division Issues in Developing a Thread-Safe MPI Implementation William Gropp Rajeev Thakur Mathematics and Computer Science Division MPI and Threads MPI describes parallelism between processes MPI-1 (the specification)

More information

Introduction to GALILEO

Introduction to GALILEO November 27, 2016 Introduction to GALILEO Parallel & production environment Mirko Cestari m.cestari@cineca.it Alessandro Marani a.marani@cineca.it SuperComputing Applications and Innovation Department

More information

MPI and OpenMP (Lecture 25, cs262a) Ion Stoica, UC Berkeley November 19, 2016

MPI and OpenMP (Lecture 25, cs262a) Ion Stoica, UC Berkeley November 19, 2016 MPI and OpenMP (Lecture 25, cs262a) Ion Stoica, UC Berkeley November 19, 2016 Message passing vs. Shared memory Client Client Client Client send(msg) recv(msg) send(msg) recv(msg) MSG MSG MSG IPC Shared

More information

Parallel Computing: Overview

Parallel Computing: Overview Parallel Computing: Overview Jemmy Hu SHARCNET University of Waterloo March 1, 2007 Contents What is Parallel Computing? Why use Parallel Computing? Flynn's Classical Taxonomy Parallel Computer Memory

More information

SHARCNET Workshop on Parallel Computing. Hugh Merz Laurentian University May 2008

SHARCNET Workshop on Parallel Computing. Hugh Merz Laurentian University May 2008 SHARCNET Workshop on Parallel Computing Hugh Merz Laurentian University May 2008 What is Parallel Computing? A computational method that utilizes multiple processing elements to solve a problem in tandem

More information

ECE 574 Cluster Computing Lecture 10

ECE 574 Cluster Computing Lecture 10 ECE 574 Cluster Computing Lecture 10 Vince Weaver http://www.eece.maine.edu/~vweaver vincent.weaver@maine.edu 1 October 2015 Announcements Homework #4 will be posted eventually 1 HW#4 Notes How granular

More information

Code Parallelization

Code Parallelization Code Parallelization a guided walk-through m.cestari@cineca.it f.salvadore@cineca.it Summer School ed. 2015 Code Parallelization two stages to write a parallel code problem domain algorithm program domain

More information

Lecture 4: OpenMP Open Multi-Processing

Lecture 4: OpenMP Open Multi-Processing CS 4230: Parallel Programming Lecture 4: OpenMP Open Multi-Processing January 23, 2017 01/23/2017 CS4230 1 Outline OpenMP another approach for thread parallel programming Fork-Join execution model OpenMP

More information

CS4961 Parallel Programming. Lecture 16: Introduction to Message Passing 11/3/11. Administrative. Mary Hall November 3, 2011.

CS4961 Parallel Programming. Lecture 16: Introduction to Message Passing 11/3/11. Administrative. Mary Hall November 3, 2011. CS4961 Parallel Programming Lecture 16: Introduction to Message Passing Administrative Next programming assignment due on Monday, Nov. 7 at midnight Need to define teams and have initial conversation with

More information

Parallel Programming

Parallel Programming Parallel Programming Lecture delivered by: Venkatanatha Sarma Y Assistant Professor MSRSAS-Bangalore 1 Session Objectives To understand the parallelization in terms of computational solutions. To understand

More information

Practical Introduction to Message-Passing Interface (MPI)

Practical Introduction to Message-Passing Interface (MPI) 1 Outline of the workshop 2 Practical Introduction to Message-Passing Interface (MPI) Bart Oldeman, Calcul Québec McGill HPC Bart.Oldeman@mcgill.ca Theoretical / practical introduction Parallelizing your

More information

15-440: Recitation 8

15-440: Recitation 8 15-440: Recitation 8 School of Computer Science Carnegie Mellon University, Qatar Fall 2013 Date: Oct 31, 2013 I- Intended Learning Outcome (ILO): The ILO of this recitation is: Apply parallel programs

More information

Parallel Programming. Jin-Soo Kim Computer Systems Laboratory Sungkyunkwan University

Parallel Programming. Jin-Soo Kim Computer Systems Laboratory Sungkyunkwan University Parallel Programming Jin-Soo Kim (jinsookim@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu Challenges Difficult to write parallel programs Most programmers think sequentially

More information

A short overview of parallel paradigms. Fabio Affinito, SCAI

A short overview of parallel paradigms. Fabio Affinito, SCAI A short overview of parallel paradigms Fabio Affinito, SCAI Why parallel? In principle, if you have more than one computing processing unit you can exploit that to: -Decrease the time to solution - Increase

More information

Shared Memory Programming with OpenMP. Lecture 8: Memory model, flush and atomics

Shared Memory Programming with OpenMP. Lecture 8: Memory model, flush and atomics Shared Memory Programming with OpenMP Lecture 8: Memory model, flush and atomics Why do we need a memory model? On modern computers code is rarely executed in the same order as it was specified in the

More information

OPEN MP and MPI on Kingspeak chpc cluster

OPEN MP and MPI on Kingspeak chpc cluster OPEN MP and MPI on Kingspeak chpc cluster Command to compile the code with openmp and mpi /uufs/kingspeak.peaks/sys/pkg/openmpi/std_intel/bin/mpicc -o hem hemhotlz.c -I /uufs/kingspeak.peaks/sys/pkg/openmpi/std_intel/include

More information

ECE 574 Cluster Computing Lecture 13

ECE 574 Cluster Computing Lecture 13 ECE 574 Cluster Computing Lecture 13 Vince Weaver http://web.eece.maine.edu/~vweaver vincent.weaver@maine.edu 21 March 2017 Announcements HW#5 Finally Graded Had right idea, but often result not an *exact*

More information

Barbara Chapman, Gabriele Jost, Ruud van der Pas

Barbara Chapman, Gabriele Jost, Ruud van der Pas Using OpenMP Portable Shared Memory Parallel Programming Barbara Chapman, Gabriele Jost, Ruud van der Pas The MIT Press Cambridge, Massachusetts London, England c 2008 Massachusetts Institute of Technology

More information

Chip Multiprocessors COMP Lecture 9 - OpenMP & MPI

Chip Multiprocessors COMP Lecture 9 - OpenMP & MPI Chip Multiprocessors COMP35112 Lecture 9 - OpenMP & MPI Graham Riley 14 February 2018 1 Today s Lecture Dividing work to be done in parallel between threads in Java (as you are doing in the labs) is rather

More information

OpenMP 4. CSCI 4850/5850 High-Performance Computing Spring 2018

OpenMP 4. CSCI 4850/5850 High-Performance Computing Spring 2018 OpenMP 4 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning Objectives

More information

Why Combine OpenMP and MPI

Why Combine OpenMP and MPI Why Combine OpenMP and MPI OpenMP might not require copies of data structures Can have some interesting designs that overlap computation and communication Overcome the limits of small processor counts

More information

Kommunikations- und Optimierungsaspekte paralleler Programmiermodelle auf hybriden HPC-Plattformen

Kommunikations- und Optimierungsaspekte paralleler Programmiermodelle auf hybriden HPC-Plattformen Kommunikations- und Optimierungsaspekte paralleler Programmiermodelle auf hybriden HPC-Plattformen Rolf Rabenseifner rabenseifner@hlrs.de Universität Stuttgart, Höchstleistungsrechenzentrum Stuttgart (HLRS)

More information

Computer Architecture

Computer Architecture Jens Teubner Computer Architecture Summer 2016 1 Computer Architecture Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de Summer 2016 Jens Teubner Computer Architecture Summer 2016 2 Part I Programming

More information

OpenMP and MPI parallelization

OpenMP and MPI parallelization OpenMP and MPI parallelization Gundolf Haase Institute for Mathematics and Scientific Computing University of Graz, Austria Chile, Jan. 2015 OpenMP for our example OpenMP generation in code Determine matrix

More information

Practical Introduction to Message-Passing Interface (MPI)

Practical Introduction to Message-Passing Interface (MPI) 1 Practical Introduction to Message-Passing Interface (MPI) October 1st, 2015 By: Pier-Luc St-Onge Partners and Sponsors 2 Setup for the workshop 1. Get a user ID and password paper (provided in class):

More information

Holland Computing Center Kickstart MPI Intro

Holland Computing Center Kickstart MPI Intro Holland Computing Center Kickstart 2016 MPI Intro Message Passing Interface (MPI) MPI is a specification for message passing library that is standardized by MPI Forum Multiple vendor-specific implementations:

More information

Parallelism paradigms

Parallelism paradigms Parallelism paradigms Intro part of course in Parallel Image Analysis Elias Rudberg elias.rudberg@it.uu.se March 23, 2011 Outline 1 Parallelization strategies 2 Shared memory 3 Distributed memory 4 Parallelization

More information

Using a Cluster as a Memory Resource: A Fast and Large Virtual Memory on MPI

Using a Cluster as a Memory Resource: A Fast and Large Virtual Memory on MPI Using a Cluster as a Memory Resource: A Fast and Large Virtual Memory on MPI DLM: Distributed Large Memory System IEEE Cluster 2009, New Orleans, Aug.31- Sep.4 Hiroko Midorikawa, Kazuhiro Saito Seikei

More information

Hybrid MPI+OpenMP Parallel MD

Hybrid MPI+OpenMP Parallel MD Hybrid MPI+OpenMP Parallel MD Aiichiro Nakano Collaboratory for Advanced Computing & Simulations Department of Computer Science Department of Physics & Astronomy Department of Chemical Engineering & Materials

More information

Hybrid MPI+OpenMP+CUDA Programming

Hybrid MPI+OpenMP+CUDA Programming Hybrid MPI+OpenMP+CUDA Programming Aiichiro Nakano Collaboratory for Advanced Computing & Simulations Department of Computer Science Department of Physics & Astronomy Department of Chemical Engineering

More information

OpenMP. António Abreu. Instituto Politécnico de Setúbal. 1 de Março de 2013

OpenMP. António Abreu. Instituto Politécnico de Setúbal. 1 de Março de 2013 OpenMP António Abreu Instituto Politécnico de Setúbal 1 de Março de 2013 António Abreu (Instituto Politécnico de Setúbal) OpenMP 1 de Março de 2013 1 / 37 openmp what? It s an Application Program Interface

More information

EPL372 Lab Exercise 5: Introduction to OpenMP

EPL372 Lab Exercise 5: Introduction to OpenMP EPL372 Lab Exercise 5: Introduction to OpenMP References: https://computing.llnl.gov/tutorials/openmp/ http://openmp.org/wp/openmp-specifications/ http://openmp.org/mp-documents/openmp-4.0-c.pdf http://openmp.org/mp-documents/openmp4.0.0.examples.pdf

More information

Scalasca performance properties The metrics tour

Scalasca performance properties The metrics tour Scalasca performance properties The metrics tour Markus Geimer m.geimer@fz-juelich.de Scalasca analysis result Generic metrics Generic metrics Time Total CPU allocation time Execution Overhead Visits Hardware

More information

Parallel Computing. Lecture 17: OpenMP Last Touch

Parallel Computing. Lecture 17: OpenMP Last Touch CSCI-UA.0480-003 Parallel Computing Lecture 17: OpenMP Last Touch Mohamed Zahran (aka Z) mzahran@cs.nyu.edu http://www.mzahran.com Some slides from here are adopted from: Yun (Helen) He and Chris Ding

More information

CS 470 Spring Mike Lam, Professor. OpenMP

CS 470 Spring Mike Lam, Professor. OpenMP CS 470 Spring 2018 Mike Lam, Professor OpenMP OpenMP Programming language extension Compiler support required "Open Multi-Processing" (open standard; latest version is 4.5) Automatic thread-level parallelism

More information

Introduction to PICO Parallel & Production Enviroment

Introduction to PICO Parallel & Production Enviroment Introduction to PICO Parallel & Production Enviroment Mirko Cestari m.cestari@cineca.it Alessandro Marani a.marani@cineca.it Domenico Guida d.guida@cineca.it Nicola Spallanzani n.spallanzani@cineca.it

More information

Systems Software for Scalable Applications (or) Super Faster Stronger MPI (and friends) for Blue Waters Applications

Systems Software for Scalable Applications (or) Super Faster Stronger MPI (and friends) for Blue Waters Applications Systems Software for Scalable Applications (or) Super Faster Stronger MPI (and friends) for Blue Waters Applications William Gropp University of Illinois, Urbana- Champaign Pavan Balaji, Rajeev Thakur

More information

Hybrid Programming with MPI and OpenMP. B. Estrade

Hybrid Programming with MPI and OpenMP. B. Estrade Hybrid Programming with MPI and OpenMP B. Estrade Objectives understand the difference between message passing and shared memory models; learn of basic models for utilizing both message

More information

Tutorial: parallel coding MPI

Tutorial: parallel coding MPI Tutorial: parallel coding MPI Pascal Viot September 12, 2018 Pascal Viot Tutorial: parallel coding MPI September 12, 2018 1 / 24 Generalities The individual power of a processor is still growing, but at

More information

Tutorial: Compiling, Makefile, Parallel jobs

Tutorial: Compiling, Makefile, Parallel jobs Tutorial: Compiling, Makefile, Parallel jobs Hartmut Häfner Steinbuch Centre for Computing (SCC) Funding: www.bwhpc-c5.de Outline Compiler + Numerical Libraries commands Linking Makefile Intro, Syntax

More information

EE/CSCI 451 Introduction to Parallel and Distributed Computation. Discussion #4 2/3/2017 University of Southern California

EE/CSCI 451 Introduction to Parallel and Distributed Computation. Discussion #4 2/3/2017 University of Southern California EE/CSCI 451 Introduction to Parallel and Distributed Computation Discussion #4 2/3/2017 University of Southern California 1 USC HPCC Access Compile Submit job OpenMP Today s topic What is OpenMP OpenMP

More information

Best Practice Guide for Writing MPI + OmpSs Interoperable Programs. Version 1.0, 3 rd April 2017

Best Practice Guide for Writing MPI + OmpSs Interoperable Programs. Version 1.0, 3 rd April 2017 Best Practice Guide for Writing MPI + OmpSs Interoperable Programs Version 1.0, 3 rd April 2017 Copyright INTERTWinE Consortium 2017 Table of Contents 1 INTRODUCTION... 1 1.1 PURPOSE... 1 1.2 GLOSSARY

More information

Parallele Numerik. Blatt 1

Parallele Numerik. Blatt 1 Universität Konstanz FB Mathematik & Statistik Prof. Dr. M. Junk Dr. Z. Yang Ausgabe: 02. Mai; SS08 Parallele Numerik Blatt 1 As a first step, we consider two basic problems. Hints for the realization

More information

Shared memory programming model OpenMP TMA4280 Introduction to Supercomputing

Shared memory programming model OpenMP TMA4280 Introduction to Supercomputing Shared memory programming model OpenMP TMA4280 Introduction to Supercomputing NTNU, IMF February 16. 2018 1 Recap: Distributed memory programming model Parallelism with MPI. An MPI execution is started

More information

1 of 6 Lecture 7: March 4. CISC 879 Software Support for Multicore Architectures Spring Lecture 7: March 4, 2008

1 of 6 Lecture 7: March 4. CISC 879 Software Support for Multicore Architectures Spring Lecture 7: March 4, 2008 1 of 6 Lecture 7: March 4 CISC 879 Software Support for Multicore Architectures Spring 2008 Lecture 7: March 4, 2008 Lecturer: Lori Pollock Scribe: Navreet Virk Open MP Programming Topics covered 1. Introduction

More information

CMSC 714 Lecture 4 OpenMP and UPC. Chau-Wen Tseng (from A. Sussman)

CMSC 714 Lecture 4 OpenMP and UPC. Chau-Wen Tseng (from A. Sussman) CMSC 714 Lecture 4 OpenMP and UPC Chau-Wen Tseng (from A. Sussman) Programming Model Overview Message passing (MPI, PVM) Separate address spaces Explicit messages to access shared data Send / receive (MPI

More information

Advanced MPI. Andrew Emerson

Advanced MPI. Andrew Emerson Advanced MPI Andrew Emerson (a.emerson@cineca.it) Agenda 1. One sided Communications (MPI-2) 2. Dynamic processes (MPI-2) 3. Profiling MPI and tracing 4. MPI-I/O 5. MPI-3 22/02/2017 Advanced MPI 2 One

More information

Hybrid Programming with MPI and OpenMP

Hybrid Programming with MPI and OpenMP Hybrid Programming with and OpenMP Fernando Silva and Ricardo Rocha Computer Science Department Faculty of Sciences University of Porto Parallel Computing 2017/2018 F. Silva and R. Rocha (DCC-FCUP) Programming

More information

CSS 534 Program 2: Parallelizing Wave Diffusion with MPI and OpenMP Professor: Munehiro Fukuda Due date: see the syllabus

CSS 534 Program 2: Parallelizing Wave Diffusion with MPI and OpenMP Professor: Munehiro Fukuda Due date: see the syllabus CSS 534 Program 2: Parallelizing Wave Diffusion with MPI and OpenMP Professor: Munehiro Fukuda Due date: see the syllabus 1. Purpose In this programming assignment, we will parallelize a sequential version

More information

MPI-2. Introduction Dynamic Process Creation. Based on notes by Sathish Vadhiyar, Rob Thacker, and David Cronk

MPI-2. Introduction Dynamic Process Creation. Based on notes by Sathish Vadhiyar, Rob Thacker, and David Cronk MPI-2 Introduction Dynamic Process Creation Based on notes by Sathish Vadhiyar, Rob Thacker, and David Cronk http://www-unix.mcs.anl.gov/mpi/mpi-standard/mpi-report-2.0/mpi2-report.htm Using MPI2: Advanced

More information

Message Passing Interface (MPI)

Message Passing Interface (MPI) CS 220: Introduction to Parallel Computing Message Passing Interface (MPI) Lecture 13 Today s Schedule Parallel Computing Background Diving in: MPI The Jetson cluster 3/7/18 CS 220: Parallel Computing

More information

Cluster Clonetroop: HowTo 2014

Cluster Clonetroop: HowTo 2014 2014/02/25 16:53 1/13 Cluster Clonetroop: HowTo 2014 Cluster Clonetroop: HowTo 2014 This section contains information about how to access, compile and execute jobs on Clonetroop, Laboratori de Càlcul Numeric's

More information

CS 426. Building and Running a Parallel Application

CS 426. Building and Running a Parallel Application CS 426 Building and Running a Parallel Application 1 Task/Channel Model Design Efficient Parallel Programs (or Algorithms) Mainly for distributed memory systems (e.g. Clusters) Break Parallel Computations

More information

Introduction to Parallel Programming with MPI

Introduction to Parallel Programming with MPI Introduction to Parallel Programming with MPI PICASso Tutorial October 25-26, 2006 Stéphane Ethier (ethier@pppl.gov) Computational Plasma Physics Group Princeton Plasma Physics Lab Why Parallel Computing?

More information

Introduction to GALILEO

Introduction to GALILEO Introduction to GALILEO Parallel & production environment Mirko Cestari m.cestari@cineca.it Alessandro Marani a.marani@cineca.it Alessandro Grottesi a.grottesi@cineca.it SuperComputing Applications and

More information

UBDA Platform User Gudie. 16 July P a g e 1

UBDA Platform User Gudie. 16 July P a g e 1 16 July 2018 P a g e 1 Revision History Version Date Prepared By Summary of Changes 1.0 Jul 16, 2018 Initial release P a g e 2 Table of Contents 1. Introduction... 4 2. Perform the test... 5 3 Job submission...

More information

HPC Parallel Programing Multi-node Computation with MPI - I

HPC Parallel Programing Multi-node Computation with MPI - I HPC Parallel Programing Multi-node Computation with MPI - I Parallelization and Optimization Group TATA Consultancy Services, Sahyadri Park Pune, India TCS all rights reserved April 29, 2013 Copyright

More information

MPI 1. CSCI 4850/5850 High-Performance Computing Spring 2018

MPI 1. CSCI 4850/5850 High-Performance Computing Spring 2018 MPI 1 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning Objectives

More information

JURECA Tuning for the platform

JURECA Tuning for the platform JURECA Tuning for the platform Usage of ParaStation MPI 2017-11-23 Outline ParaStation MPI Compiling your program Running your program Tuning parameters Resources 2 ParaStation MPI Based on MPICH (3.2)

More information

OpenMP. Today s lecture. Scott B. Baden / CSE 160 / Wi '16

OpenMP. Today s lecture. Scott B. Baden / CSE 160 / Wi '16 Lecture 8 OpenMP Today s lecture 7 OpenMP A higher level interface for threads programming http://www.openmp.org Parallelization via source code annotations All major compilers support it, including gnu

More information

Message Passing Interface

Message Passing Interface MPSoC Architectures MPI Alberto Bosio, Associate Professor UM Microelectronic Departement bosio@lirmm.fr Message Passing Interface API for distributed-memory programming parallel code that runs across

More information

30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy

30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Why serial is not enough Computing architectures Parallel paradigms Message Passing Interface How

More information

Topics. Introduction. Shared Memory Parallelization. Example. Lecture 11. OpenMP Execution Model Fork-Join model 5/15/2012. Introduction OpenMP

Topics. Introduction. Shared Memory Parallelization. Example. Lecture 11. OpenMP Execution Model Fork-Join model 5/15/2012. Introduction OpenMP Topics Lecture 11 Introduction OpenMP Some Examples Library functions Environment variables 1 2 Introduction Shared Memory Parallelization OpenMP is: a standard for parallel programming in C, C++, and

More information

Parallelization, OpenMP

Parallelization, OpenMP ~ Parallelization, OpenMP Scientific Computing Winter 2016/2017 Lecture 26 Jürgen Fuhrmann juergen.fuhrmann@wias-berlin.de made wit pandoc 1 / 18 Why parallelization? Computers became faster and faster

More information

Message Passing Interface

Message Passing Interface Message Passing Interface DPHPC15 TA: Salvatore Di Girolamo DSM (Distributed Shared Memory) Message Passing MPI (Message Passing Interface) A message passing specification implemented

More information

Threaded Programming. Lecture 9: Alternatives to OpenMP

Threaded Programming. Lecture 9: Alternatives to OpenMP Threaded Programming Lecture 9: Alternatives to OpenMP What s wrong with OpenMP? OpenMP is designed for programs where you want a fixed number of threads, and you always want the threads to be consuming

More information

Parallel Computing Using OpenMP/MPI. Presented by - Jyotsna 29/01/2008

Parallel Computing Using OpenMP/MPI. Presented by - Jyotsna 29/01/2008 Parallel Computing Using OpenMP/MPI Presented by - Jyotsna 29/01/2008 Serial Computing Serially solving a problem Parallel Computing Parallelly solving a problem Parallel Computer Memory Architecture Shared

More information

Introduction to the Message Passing Interface (MPI)

Introduction to the Message Passing Interface (MPI) Introduction to the Message Passing Interface (MPI) CPS343 Parallel and High Performance Computing Spring 2018 CPS343 (Parallel and HPC) Introduction to the Message Passing Interface (MPI) Spring 2018

More information