Concurrent Programming with OpenMP
|
|
- Augusta Pierce
- 6 years ago
- Views:
Transcription
1 Concurrent Programming with OpenMP Parallel and Distributed Computing Department of Computer Science and Engineering (DEI) Instituto Superior Técnico March 7, 2016 CPD (DEI / IST) Parallel and Distributed Computing / 42
2 Outline Shared Memory Concurrent Programming Review of Operating Systems: PThreads OpenMP Parallel Clauses Private / Shared Variables CPD (DEI / IST) Parallel and Distributed Computing / 42
3 Shared-Memory Systems Uniform Memory Access (UMA) architecture also known as Symmetric Shared-Memory Multiprocessors (SMP) P P P P Cache Cache Cache Cache Main Memory I / O CPD (DEI / IST) Parallel and Distributed Computing / 42
4 Fork/Join Parallelism Cheap creation/termination of tasks invites for Incremental Parallelization: process of converting a sequential program to a parallel program a little bit at a time. initially only master thread is active master thread executes sequential code Fork: master thread creates or awakens additional threads to execute parallel code Join: at end of parallel code created threads die or are suspended Master Thread Other Threads Fork Join Time Fork Join CPD (DEI / IST) Parallel and Distributed Computing / 42
5 Fork/Join Parallelism read(a, B); x = initx(a, B); y = inity(a, B); z = initz(a, B); for(i = 0; i < N_ENTRIES; i++) x[i] = compx(y[i], z[i]); for(i = 1; i < N_ENTRIES; i++){ x[i] = solvex(x[i-1]); z[i] = x[i] + y[i];... finalize1(&x, &y, &z); finalize2(&x, &y, &z); finalize3(&x, &y, &z); CPD (DEI / IST) Parallel and Distributed Computing / 42
6 Processes and Threads Process A Process B Environment Global Data, Shared Code System Resources Interprocess Communication Thread 1 Private Data Stack Thread 2 Private Data Stack Thread 3 Private Data Stack... Thread n Private Data Stack CPD (DEI / IST) Parallel and Distributed Computing / 42
7 POSIX Threads (PThreads): Creation int pthread_create(pthread_t *thread, pthread_attr_t *attr, void *(*start_routine)(void*), void *arg) Example: pthread_t pt_worker; void *thread_function(void *args) { /* thread code */ pthread_create(&pt_worker, NULL, thread_function, (void *) thread_args); CPD (DEI / IST) Parallel and Distributed Computing / 42
8 PThreads: Termination and Synchronization int pthread_exit(void *value_ptr) int pthread_join(pthread_t thread, void **value_ptr) CPD (DEI / IST) Parallel and Distributed Computing / 42
9 PThread Example: Summing the Values in Matrix Rows #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <pthread.h> int buffer[n][size]; void *sum_row (void *ptr){ int index = 0, sum = 0; int *b = (int *) ptr; while (index < SIZE - 1) sum += b[index++]; /* sum row*/ b[index]=sum; /* store sum in last col. */ pthread_exit(null); int main(void){ int i,j; pthread_t tid[n]; for(i = 0; i < N; i++) if(pthread_create(&tid[i], NULL, sum_row, (void *) &(buffer[i]))!= 0){ printf("error creating thread, id=%d\n", i); exit(-1); else printf ("Created thread w/ id %d\n", tid[i]); for(i = 0; i < N; i++) pthread_join(tid[i], NULL); printf("all threads have concluded\n"); for(i = 0; i < N; i++){ for(j = 0; j < SIZE; j++) printf(" %d ", buffer[i][j]); printf ("Row %d \n", i); exit(0); for(i = 0; i < N; i++) for(j = 0; j < SIZE-1; j++) buffer[i][j] = rand()%10; CPD (DEI / IST) Parallel and Distributed Computing / 42
10 PThreads: Synchronization int count; void *sum_row(void *ptr){ int index = 0, sum = 0; int *b = (int *) ptr; while(index < SIZE - 1) sum += b[index++]; /* sum row */ b[index] = sum; /* store sum in last col. */ count++; pthread_exit(null); Problem? CPD (DEI / IST) Parallel and Distributed Computing / 42
11 PThreads: Synchronization int pthread_mutex_init(pthread_mutex_t *mutex, pthread_mutexattr_t *attr); int pthread_mutex_lock(pthread_mutex_t *mutex); int pthread_mutex_unlock(pthread_mutex_t *mutex); Example: pthread_mutex_t count_lock; pthread_mutex_init(&count_lock, NULL); pthread_mutex_lock(&count_lock); atomic_function(); pthread_mutex_unlock(&count_lock); CPD (DEI / IST) Parallel and Distributed Computing / 42
12 PThreads: Synchronization Example int count; pthread_mutex_t count_lock; void *sum_row(void *ptr){ int index = 0, sum = 0; int *b = (int *) ptr; main() { /*...*/ pthread_mutex_init(&count_lock, NULL); while(index < SIZE - 1) sum += b[index++]; /* sum row */ b[index]=sum; /* store sum in last col. */ pthread_mutex_lock(&count_lock); count++; pthread_mutex_unlock(&count_lock); pthread_exit(null); CPD (DEI / IST) Parallel and Distributed Computing / 42
13 OpenMP What is OpenMP? Open specification for Multi-Threaded, Shared Memory Parallelism Standard Application Programming Interface (API): Preprocessor (compiler) directives Library Calls Environment Variables More info at CPD (DEI / IST) Parallel and Distributed Computing / 42
14 OpenMP vs Threads (Supposedly) Better than threads: Simpler programming model Separate a program into serial and parallel regions, rather than T concurrently-executing threads Similar to threads: Programmer must detect dependencies Programmer must prevent data races CPD (DEI / IST) Parallel and Distributed Computing / 42
15 Parallel Programming Recipes Threads: 1 Start with a parallel algorithm 2 Implement, keeping in mind: Data races Synchronization Threading syntax 3 Test & Debug 4 Goto step 2 CPD (DEI / IST) Parallel and Distributed Computing / 42
16 Parallel Programming Recipes Threads: 1 Start with a parallel algorithm 2 Implement, keeping in mind: Data races Synchronization Threading syntax 3 Test & Debug 4 Goto step 2 OpenMP: 1 Start with some algorithm 2 Implement serially, ignoring: Data races Synchronization Threading syntax 3 Test & Debug 4 Automagically parallelize with relatively few annotations that specify parallelism and synchronization CPD (DEI / IST) Parallel and Distributed Computing / 42
17 OpenMP Directives Parallelization directives: parallel region parallel for parallel sections task Data environment directives: shared, private, threadprivate, reduction, etc. Synchronization directives: barrier, critical CPD (DEI / IST) Parallel and Distributed Computing / 42
18 C / C++ Directives Format #pragma omp directive-name [clause,...] Case sensitive Long directive lines may be continued on succeeding lines by escaping the newline character with a \ at the end of the directive line Always apply to the next statement, which must be a structured block. Examples: #pragma omp... statement #pragma omp... { statement1; statement2; statement3; CPD (DEI / IST) Parallel and Distributed Computing / 42
19 Parallel Region #pragma omp parallel [clauses] Creates N parallel threads All execute subsequent block All wait for each other at the end of executing the block Barrier synchronization CPD (DEI / IST) Parallel and Distributed Computing / 42
20 How Many Threads? The number of threads created is determined by, in order of precedence: Use of omp set num threads() library function Setting of the OMP NUM THREADS environment variable Implementation default - usually the number of CPUs Possible to query number of CPUs: int omp_get_num_procs (void) CPD (DEI / IST) Parallel and Distributed Computing / 42
21 Parallel Region Example main() { printf("serial Region 1\n"); omp_set_num_threads(4); #pragma omp parallel { printf("parallel Region\n"); printf("serial Region 2\n"); Output? CPD (DEI / IST) Parallel and Distributed Computing / 42
22 Thread Count and Id API #include <omp.h> int omp_get_thread_num() int omp_get_num_threads() void omp_set_num_threads(int num) Example: #pragma omp parallel { if(!omp_get_thread_num() ) master(); else slave(); CPD (DEI / IST) Parallel and Distributed Computing / 42
23 Work Sharing Directives Always occur within a parallel region Divide the execution of the enclosed code region among the members of the team Do not create new threads Two main directives are parallel for parallel section CPD (DEI / IST) Parallel and Distributed Computing / 42
24 Parallel For #pragma omp parallel #pragma omp for [clauses] for( ; ; ) {... Each thread executes a subset of the iterations All threads synchronize at the end of parallel for Restrictions No data dependencies between iterations Program correctness must not depend upon which thread executes a particular iteration Paradigm of Data Parallelism. CPD (DEI / IST) Parallel and Distributed Computing / 42
25 Handy Shortcut #pragma omp parallel #pragma omp for for ( ; ; ) {... is equivalent to #pragma omp parallel for for ( ; ; ) {... CPD (DEI / IST) Parallel and Distributed Computing / 42
26 PThread Example Revisited #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <pthread.h> int buffer[n][size]; void *sum_row (void *ptr){ int index = 0, sum = 0; int *b = (int *) ptr; while (index < SIZE - 1) sum += b[index++]; /* sum row*/ b[index]=sum; /* store sum in last col. */ pthread_exit(null); int main(void){ int i,j; pthread_t tid[n]; for(i = 0; i < N; i++) if(pthread_create(&tid[i], 0, sum_row, (void *) &(buffer[i]))!= 0){ printf("error creating thread, id=%d\n", i); exit(-1); else printf ("Created thread w/ id %d\n", i); for(i = 0; i < N; i++) pthread_join(tid[i], NULL); printf("all threads have concluded\n"); for(i = 0; i < N; i++){ for(j = 0; j < SIZE; j++) printf(" %d ", buffer[i][j]); printf ("Row %d \n", i); exit(0); for(i = 0; i < N; i++) for(j = 0; j < SIZE-1; j++) buffer[i][j] = rand()%10; CPD (DEI / IST) Parallel and Distributed Computing / 42
27 PThread Example Revisited #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <omp.h> #pragma omp parallel for for(i = 0; i < N; i++) sum_row(buffer[i]); int buffer[n][size]; void *sum_row (void *ptr){ int index = 0, sum = 0; int *b = (int *) ptr; while (index < SIZE - 1) sum += b[index++]; /* sum row*/ b[index]=sum; /* store sum in last col. */ int main(void){ int i,j; printf("all threads have concluded\n"); for(i = 0; i < N; i++){ for(j = 0; j < SIZE; j++) printf(" %d ", buffer[i][j]); printf ("Row %d \n", i); exit(0); for(i = 0; i < N; i++) for(j = 0; j < SIZE-1; j++) buffer[i][j] = rand()%10; CPD (DEI / IST) Parallel and Distributed Computing / 42
28 Multiple Work Sharing Directives May occur within the same parallel region: #pragma omp parallel { #pragma omp for for( ; ; ) {... #pragma omp for for( ; ; ) {... Implicit barrier at the end of each for. CPD (DEI / IST) Parallel and Distributed Computing / 42
29 Parallel Sections Functional Parallelism: several blocks are executed in parallel #pragma omp parallel { #pragma omp sections { #pragma omp section { a=...; b=...; #pragma omp section /* <- delimiter! */ { c=...; d=...; #pragma omp section { e=...; f=...; #pragma omp section { g=...; h=...; /*omp end sections*/ /*omp end parallel*/ CPD (DEI / IST) Parallel and Distributed Computing / 42
30 Handy Shortcut #pragma omp parallel #pragma omp sections {... is equivalent to #pragma omp parallel sections {... CPD (DEI / IST) Parallel and Distributed Computing / 42
31 OpenMP Memory Model Concurrent programs access two types of data Shared data, visible to all threads Private data, visible to a single thread (often stack-allocated) Threads: Global variables are shared Local variables are private CPD (DEI / IST) Parallel and Distributed Computing / 42
32 OpenMP Memory Model Concurrent programs access two types of data Threads: Shared data, visible to all threads Private data, visible to a single thread (often stack-allocated) Global variables are shared Local variables are private OpenMP: All variables are by default shared. Some exceptions: the loop variable of a parallel for is private stack (local) variables in called subroutines are private By using data directives, some variables can be made private or given other special characteristics. CPD (DEI / IST) Parallel and Distributed Computing / 42
33 Private Variables #pragma omp parallel for private( list ) Makes a private copy for each thread for each variable in the list. No storage association with original object All references are to the local object Values are undefined on entry and exit Also applies to other region and work-sharing directives. CPD (DEI / IST) Parallel and Distributed Computing / 42
34 Shared Variables #pragma omp parallel for shared ( list ) Similarly, there is a shared data directive. Shared variables exist in a single location and all threads can read and write it It is the programmer s responsibility to ensure that all multiple threads properly access shared variables (will discuss synchronization next) CPD (DEI / IST) Parallel and Distributed Computing / 42
35 Example PThread vs OpenMP PThreads OpenMP // shared, globals int n, *x, *y; void loop() { int i; // private, stack for(i = 0; i < n; i++) x[i] += y[i]; #pragma omp parallel \ shared(n,x,y) private(i) { #pragma omp for for(i = 0; i < n; i++) x[i] += y[i]; CPD (DEI / IST) Parallel and Distributed Computing / 42
36 Example PThread vs OpenMP PThreads OpenMP // shared, globals int n, *x, *y; void loop() { int i; // private, stack #pragma omp parallel for { for(i = 0; i < n; i++) x[i] += y[i]; for(i = 0; i < n; i++) x[i] += y[i]; CPD (DEI / IST) Parallel and Distributed Computing / 42
37 Example of private Clause for(i = 0; i < n; i++) for(j = 0; j < n; j++) a[i][j] = b[i][j] + c[i][j]; Make outer loop parallel, to reduce number of forks/joins. Give each thread its own private copy of variable j. CPD (DEI / IST) Parallel and Distributed Computing / 42
38 Example of private Clause for(i = 0; i < n; i++) for(j = 0; j < n; j++) a[i][j] = b[i][j] + c[i][j]; Make outer loop parallel, to reduce number of forks/joins. Give each thread its own private copy of variable j. #pragma omp parallel for private(j) for(i = 0; i < n; i++) for(j = 0; j < n; j++) a[i][j] = b[i][j] + c[i][j]; CPD (DEI / IST) Parallel and Distributed Computing / 42
39 firstprivate / lastprivate Clauses As mentioned, values of private variables are undefined on entry and exit. A private variable within a region has no storage association with the same variable outside of the region firstprivate (list) Variables in list are initialized with the value the original variable had before entering the parallel construct lastprivate (list) The thread that executes the sequentially last iteration or section updates the value of the variables in list CPD (DEI / IST) Parallel and Distributed Computing / 42
40 Example of firstprivate / lastprivate Clauses main() { a = 1; #pragma omp parallel for private(i), firstprivate(a), lastprivate(b) for (i = 0; i < n; i++) {... b = a + i; /* a undefined, unless declared firstprivate */... a = b; /* b undefined, unless declared lastprivate */ CPD (DEI / IST) Parallel and Distributed Computing / 42
41 threadprivate Variables Private variables are private on a parallel region basis. threadprivate variables are global variables that are private throughout the execution of the program. #pragma omp threadprivate(x) Initial data is undefined, unless copyin is used copyin (list) data of the master thread is copied to the threadprivate copies CPD (DEI / IST) Parallel and Distributed Computing / 42
42 Example of threadprivate Clause #include <omp.h> int a, b, i, tid; float x; #pragma omp threadprivate(a, x) main () { printf("1st Parallel Region:\n"); #pragma omp parallel private(b,tid) { tid = omp_get_thread_num(); a = tid; b = tid; x = 1.1 * tid +1.0; printf("thread %d: a,b,x= %d %d %f\n",tid,a,b,x); /* end of parallel section */ printf("2nd Parallel Region:\n"); #pragma omp parallel private(tid) { tid = omp_get_thread_num(); printf("thread %d: a,b,x = %d %d %f\n",tid,a,b,x); /* end of parallel section */ CPD (DEI / IST) Parallel and Distributed Computing / 42
43 Review Shared Memory Concurrent Programming Review of Operating Systems: PThreads OpenMP Parallel Clauses Private / Shared Variables CPD (DEI / IST) Parallel and Distributed Computing / 42
44 Next Class More on OpenMP: Synchronism Conditional Parallelism Reduction Clause Scheduling Options CPD (DEI / IST) Parallel and Distributed Computing / 42
Parallel and Distributed Computing
Concurrent Programming with OpenMP Rodrigo Miragaia Rodrigues MSc in Information Systems and Computer Engineering DEA in Computational Engineering CS Department (DEI) Instituto Superior Técnico October
More informationParallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 17 Shared-memory Programming 1 Outline n OpenMP n Shared-memory model n Parallel for loops n Declaring private variables n Critical
More informationParallel Programming. OpenMP Parallel programming for multiprocessors for loops
Parallel Programming OpenMP Parallel programming for multiprocessors for loops OpenMP OpenMP An application programming interface (API) for parallel programming on multiprocessors Assumes shared memory
More informationProgramming Shared Memory Systems with OpenMP Part I. Book
Programming Shared Memory Systems with OpenMP Part I Instructor Dr. Taufer Book Parallel Programming in OpenMP by Rohit Chandra, Leo Dagum, Dave Kohr, Dror Maydan, Jeff McDonald, Ramesh Menon 2 1 Machine
More informationParallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 17 Shared-memory Programming 1 Outline n OpenMP n Shared-memory model n Parallel for loops n Declaring private variables n Critical
More informationShared Memory Parallel Programming
Shared Memory Parallel Programming Abhishek Somani, Debdeep Mukhopadhyay Mentor Graphics, IIT Kharagpur August 2, 2015 Abhishek, Debdeep (IIT Kgp) Parallel Programming August 2, 2015 1 / 46 Overview 1
More informationParallel Programming. Exploring local computational resources OpenMP Parallel programming for multiprocessors for loops
Parallel Programming Exploring local computational resources OpenMP Parallel programming for multiprocessors for loops Single computers nowadays Several CPUs (cores) 4 to 8 cores on a single chip Hyper-threading
More informationParallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 17 Shared-memory Programming Outline OpenMP Shared-memory model Parallel for loops Declaring private variables Critical sections Reductions
More informationOpenMP programming. Thomas Hauser Director Research Computing Research CU-Boulder
OpenMP programming Thomas Hauser Director Research Computing thomas.hauser@colorado.edu CU meetup 1 Outline OpenMP Shared-memory model Parallel for loops Declaring private variables Critical sections Reductions
More informationParallel Programming with OpenMP. CS240A, T. Yang
Parallel Programming with OpenMP CS240A, T. Yang 1 A Programmer s View of OpenMP What is OpenMP? Open specification for Multi-Processing Standard API for defining multi-threaded shared-memory programs
More information15-418, Spring 2008 OpenMP: A Short Introduction
15-418, Spring 2008 OpenMP: A Short Introduction This is a short introduction to OpenMP, an API (Application Program Interface) that supports multithreaded, shared address space (aka shared memory) parallelism.
More informationECE 574 Cluster Computing Lecture 10
ECE 574 Cluster Computing Lecture 10 Vince Weaver http://www.eece.maine.edu/~vweaver vincent.weaver@maine.edu 1 October 2015 Announcements Homework #4 will be posted eventually 1 HW#4 Notes How granular
More informationOpenMP. António Abreu. Instituto Politécnico de Setúbal. 1 de Março de 2013
OpenMP António Abreu Instituto Politécnico de Setúbal 1 de Março de 2013 António Abreu (Instituto Politécnico de Setúbal) OpenMP 1 de Março de 2013 1 / 37 openmp what? It s an Application Program Interface
More informationEPL372 Lab Exercise 5: Introduction to OpenMP
EPL372 Lab Exercise 5: Introduction to OpenMP References: https://computing.llnl.gov/tutorials/openmp/ http://openmp.org/wp/openmp-specifications/ http://openmp.org/mp-documents/openmp-4.0-c.pdf http://openmp.org/mp-documents/openmp4.0.0.examples.pdf
More informationOpenMP I. Diego Fabregat-Traver and Prof. Paolo Bientinesi WS16/17. HPAC, RWTH Aachen
OpenMP I Diego Fabregat-Traver and Prof. Paolo Bientinesi HPAC, RWTH Aachen fabregat@aices.rwth-aachen.de WS16/17 OpenMP References Using OpenMP: Portable Shared Memory Parallel Programming. The MIT Press,
More informationParallel Numerical Algorithms
Parallel Numerical Algorithms http://sudalab.is.s.u-tokyo.ac.jp/~reiji/pna16/ [ 8 ] OpenMP Parallel Numerical Algorithms / IST / UTokyo 1 PNA16 Lecture Plan General Topics 1. Architecture and Performance
More informationIntroduction to OpenMP. OpenMP basics OpenMP directives, clauses, and library routines
Introduction to OpenMP Introduction OpenMP basics OpenMP directives, clauses, and library routines What is OpenMP? What does OpenMP stands for? What does OpenMP stands for? Open specifications for Multi
More informationOpenMP - II. Diego Fabregat-Traver and Prof. Paolo Bientinesi WS15/16. HPAC, RWTH Aachen
OpenMP - II Diego Fabregat-Traver and Prof. Paolo Bientinesi HPAC, RWTH Aachen fabregat@aices.rwth-aachen.de WS15/16 OpenMP References Using OpenMP: Portable Shared Memory Parallel Programming. The MIT
More informationConcurrent Programming with OpenMP
Concurrent Programming with OpenMP Parallel and Distributed Computing Department of Computer Science and Engineering (DEI) Instituto Superior Técnico October 11, 2012 CPD (DEI / IST) Parallel and Distributed
More informationDPHPC: Introduction to OpenMP Recitation session
SALVATORE DI GIROLAMO DPHPC: Introduction to OpenMP Recitation session Based on http://openmp.org/mp-documents/intro_to_openmp_mattson.pdf OpenMP An Introduction What is it? A set of compiler directives
More informationLecture 4: OpenMP Open Multi-Processing
CS 4230: Parallel Programming Lecture 4: OpenMP Open Multi-Processing January 23, 2017 01/23/2017 CS4230 1 Outline OpenMP another approach for thread parallel programming Fork-Join execution model OpenMP
More informationITCS 4/5145 Parallel Computing Test 1 5:00 pm - 6:15 pm, Wednesday February 17, 2016 Solutions Name:...
ITCS 4/5145 Parallel Computing Test 1 5:00 pm - 6:15 pm, Wednesday February 17, 016 Solutions Name:... Answer questions in space provided below questions. Use additional paper if necessary but make sure
More informationDistributed Systems + Middleware Concurrent Programming with OpenMP
Distributed Systems + Middleware Concurrent Programming with OpenMP Gianpaolo Cugola Dipartimento di Elettronica e Informazione Politecnico, Italy cugola@elet.polimi.it http://home.dei.polimi.it/cugola
More informationOpenMP 2. CSCI 4850/5850 High-Performance Computing Spring 2018
OpenMP 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning Objectives
More informationModule 10: Open Multi-Processing Lecture 19: What is Parallelization? The Lecture Contains: What is Parallelization? Perfectly Load-Balanced Program
The Lecture Contains: What is Parallelization? Perfectly Load-Balanced Program Amdahl's Law About Data What is Data Race? Overview to OpenMP Components of OpenMP OpenMP Programming Model OpenMP Directives
More informationIntroduction to. Slides prepared by : Farzana Rahman 1
Introduction to OpenMP Slides prepared by : Farzana Rahman 1 Definition of OpenMP Application Program Interface (API) for Shared Memory Parallel Programming Directive based approach with library support
More informationUvA-SARA High Performance Computing Course June Clemens Grelck, University of Amsterdam. Parallel Programming with Compiler Directives: OpenMP
Parallel Programming with Compiler Directives OpenMP Clemens Grelck University of Amsterdam UvA-SARA High Performance Computing Course June 2013 OpenMP at a Glance Loop Parallelization Scheduling Parallel
More informationOpenMP Algoritmi e Calcolo Parallelo. Daniele Loiacono
OpenMP Algoritmi e Calcolo Parallelo References Useful references Using OpenMP: Portable Shared Memory Parallel Programming, Barbara Chapman, Gabriele Jost and Ruud van der Pas OpenMP.org http://openmp.org/
More information1 of 6 Lecture 7: March 4. CISC 879 Software Support for Multicore Architectures Spring Lecture 7: March 4, 2008
1 of 6 Lecture 7: March 4 CISC 879 Software Support for Multicore Architectures Spring 2008 Lecture 7: March 4, 2008 Lecturer: Lori Pollock Scribe: Navreet Virk Open MP Programming Topics covered 1. Introduction
More informationPOSIX threads CS 241. February 17, Copyright University of Illinois CS 241 Staff
POSIX threads CS 241 February 17, 2012 Copyright University of Illinois CS 241 Staff 1 Recall: Why threads over processes? Creating a new process can be expensive Time A call into the operating system
More informationOpenMP and MPI. Parallel and Distributed Computing. Department of Computer Science and Engineering (DEI) Instituto Superior Técnico.
OpenMP and MPI Parallel and Distributed Computing Department of Computer Science and Engineering (DEI) Instituto Superior Técnico November 16, 2011 CPD (DEI / IST) Parallel and Distributed Computing 18
More informationMango DSP Top manufacturer of multiprocessing video & imaging solutions.
1 of 11 3/3/2005 10:50 AM Linux Magazine February 2004 C++ Parallel Increase application performance without changing your source code. Mango DSP Top manufacturer of multiprocessing video & imaging solutions.
More informationAllows program to be incrementally parallelized
Basic OpenMP What is OpenMP An open standard for shared memory programming in C/C+ + and Fortran supported by Intel, Gnu, Microsoft, Apple, IBM, HP and others Compiler directives and library support OpenMP
More informationAn Introduction to OpenMP
Dipartimento di Ingegneria Industriale e dell'informazione University of Pavia December 4, 2017 Recap Parallel machines are everywhere Many architectures, many programming model. Among them: multithreading.
More informationOpenMP. A parallel language standard that support both data and functional Parallelism on a shared memory system
OpenMP A parallel language standard that support both data and functional Parallelism on a shared memory system Use by system programmers more than application programmers Considered a low level primitives
More informationMultithreading in C with OpenMP
Multithreading in C with OpenMP ICS432 - Spring 2017 Concurrent and High-Performance Programming Henri Casanova (henric@hawaii.edu) Pthreads are good and bad! Multi-threaded programming in C with Pthreads
More informationOpenMP and MPI. Parallel and Distributed Computing. Department of Computer Science and Engineering (DEI) Instituto Superior Técnico.
OpenMP and MPI Parallel and Distributed Computing Department of Computer Science and Engineering (DEI) Instituto Superior Técnico November 15, 2010 José Monteiro (DEI / IST) Parallel and Distributed Computing
More informationParallel Processing Top manufacturer of multiprocessing video & imaging solutions.
1 of 10 3/3/2005 10:51 AM Linux Magazine March 2004 C++ Parallel Increase application performance without changing your source code. Parallel Processing Top manufacturer of multiprocessing video & imaging
More informationIntroduction to OpenMP
Introduction to OpenMP Ricardo Fonseca https://sites.google.com/view/rafonseca2017/ Outline Shared Memory Programming OpenMP Fork-Join Model Compiler Directives / Run time library routines Compiling and
More informationMulti-core Architecture and Programming
Multi-core Architecture and Programming Yang Quansheng( 杨全胜 ) http://www.njyangqs.com School of Computer Science & Engineering 1 http://www.njyangqs.com Programming with OpenMP Content What is PpenMP Parallel
More information/Users/engelen/Sites/HPC folder/hpc/openmpexamples.c
/* Subset of these examples adapted from: 1. http://www.llnl.gov/computing/tutorials/openmp/exercise.html 2. NAS benchmarks */ #include #include #ifdef _OPENMP #include #endif
More informationParallel Programming: OpenMP
Parallel Programming: OpenMP Xianyi Zeng xzeng@utep.edu Department of Mathematical Sciences The University of Texas at El Paso. November 10, 2016. An Overview of OpenMP OpenMP: Open Multi-Processing An
More informationChapter 4 Concurrent Programming
Chapter 4 Concurrent Programming 4.1. Introduction to Parallel Computing In the early days, most computers have only one processing element, known as the Central Processing Unit (CPU). Due to this hardware
More informationShared Memory Parallelism - OpenMP
Shared Memory Parallelism - OpenMP Sathish Vadhiyar Credits/Sources: OpenMP C/C++ standard (openmp.org) OpenMP tutorial (http://www.llnl.gov/computing/tutorials/openmp/#introduction) OpenMP sc99 tutorial
More informationOverview: The OpenMP Programming Model
Overview: The OpenMP Programming Model motivation and overview the parallel directive: clauses, equivalent pthread code, examples the for directive and scheduling of loop iterations Pi example in OpenMP
More informationCS4961 Parallel Programming. Lecture 9: Task Parallelism in OpenMP 9/22/09. Administrative. Mary Hall September 22, 2009.
Parallel Programming Lecture 9: Task Parallelism in OpenMP Administrative Programming assignment 1 is posted (after class) Due, Tuesday, September 22 before class - Use the handin program on the CADE machines
More informationOpenMP Tutorial. Seung-Jai Min. School of Electrical and Computer Engineering Purdue University, West Lafayette, IN
OpenMP Tutorial Seung-Jai Min (smin@purdue.edu) School of Electrical and Computer Engineering Purdue University, West Lafayette, IN 1 Parallel Programming Standards Thread Libraries - Win32 API / Posix
More informationHigh Performance Computing: Tools and Applications
High Performance Computing: Tools and Applications Edmond Chow School of Computational Science and Engineering Georgia Institute of Technology Lecture 2 OpenMP Shared address space programming High-level
More informationShared Memory: Virtual Shared Memory, Threads & OpenMP
Shared Memory: Virtual Shared Memory, Threads & OpenMP Eugen Betke University of Hamburg Department Informatik Scientific Computing 09.01.2012 Agenda 1 Introduction Architectures of Memory Systems 2 Virtual
More informationHPC Practical Course Part 3.1 Open Multi-Processing (OpenMP)
HPC Practical Course Part 3.1 Open Multi-Processing (OpenMP) V. Akishina, I. Kisel, G. Kozlov, I. Kulakov, M. Pugach, M. Zyzak Goethe University of Frankfurt am Main 2015 Task Parallelism Parallelization
More informationCMSC 714 Lecture 4 OpenMP and UPC. Chau-Wen Tseng (from A. Sussman)
CMSC 714 Lecture 4 OpenMP and UPC Chau-Wen Tseng (from A. Sussman) Programming Model Overview Message passing (MPI, PVM) Separate address spaces Explicit messages to access shared data Send / receive (MPI
More informationby system default usually a thread per CPU or core using the environment variable OMP_NUM_THREADS from within the program by using function call
OpenMP Syntax The OpenMP Programming Model Number of threads are determined by system default usually a thread per CPU or core using the environment variable OMP_NUM_THREADS from within the program by
More informationAssignment 1 OpenMP Tutorial Assignment
Assignment 1 OpenMP Tutorial Assignment B. Wilkinson and C Ferner: Modification date Aug 5, 2014 Overview In this assignment, you will write and execute run some simple OpenMP programs as a tutorial. First
More informationTopics. Introduction. Shared Memory Parallelization. Example. Lecture 11. OpenMP Execution Model Fork-Join model 5/15/2012. Introduction OpenMP
Topics Lecture 11 Introduction OpenMP Some Examples Library functions Environment variables 1 2 Introduction Shared Memory Parallelization OpenMP is: a standard for parallel programming in C, C++, and
More informationOpenMP! baseado em material cedido por Cristiana Amza!
OpenMP! baseado em material cedido por Cristiana Amza! www.eecg.toronto.edu/~amza/ece1747h/ece1747h.html! What is OpenMP?! Standard for shared memory programming for scientific applications.! Has specific
More informationOpenMP threading: parallel regions. Paolo Burgio
OpenMP threading: parallel regions Paolo Burgio paolo.burgio@unimore.it Outline Expressing parallelism Understanding parallel threads Memory Data management Data clauses Synchronization Barriers, locks,
More informationCS510 Operating System Foundations. Jonathan Walpole
CS510 Operating System Foundations Jonathan Walpole Threads & Concurrency 2 Why Use Threads? Utilize multiple CPU s concurrently Low cost communication via shared memory Overlap computation and blocking
More informationProgramming with Shared Memory. Nguyễn Quang Hùng
Programming with Shared Memory Nguyễn Quang Hùng Outline Introduction Shared memory multiprocessors Constructs for specifying parallelism Creating concurrent processes Threads Sharing data Creating shared
More informationGCC Developers Summit Ottawa, Canada, June 2006
OpenMP Implementation in GCC Diego Novillo dnovillo@redhat.com Red Hat Canada GCC Developers Summit Ottawa, Canada, June 2006 OpenMP Language extensions for shared memory concurrency (C, C++ and Fortran)
More informationComputer Architecture
Jens Teubner Computer Architecture Summer 2016 1 Computer Architecture Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de Summer 2016 Jens Teubner Computer Architecture Summer 2016 2 Part I Programming
More informationA brief introduction to OpenMP
A brief introduction to OpenMP Alejandro Duran Barcelona Supercomputing Center Outline 1 Introduction 2 Writing OpenMP programs 3 Data-sharing attributes 4 Synchronization 5 Worksharings 6 Task parallelism
More informationOperating systems fundamentals - B06
Operating systems fundamentals - B06 David Kendall Northumbria University David Kendall (Northumbria University) Operating systems fundamentals - B06 1 / 12 Introduction Introduction to threads Reminder
More informationParallel and Distributed Programming. OpenMP
Parallel and Distributed Programming OpenMP OpenMP Portability of software SPMD model Detailed versions (bindings) for different programming languages Components: directives for compiler library functions
More informationIntroduction to OpenMP
Introduction to OpenMP Le Yan Scientific computing consultant User services group High Performance Computing @ LSU Goals Acquaint users with the concept of shared memory parallelism Acquaint users with
More informationProgramming Shared Address Space Platforms using OpenMP
Programming Shared Address Space Platforms using OpenMP Jinkyu Jeong (jinkyu@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu Topic Overview Introduction to OpenMP OpenMP
More informationCOMP4300/8300: The OpenMP Programming Model. Alistair Rendell. Specifications maintained by OpenMP Architecture Review Board (ARB)
COMP4300/8300: The OpenMP Programming Model Alistair Rendell See: www.openmp.org Introduction to High Performance Computing for Scientists and Engineers, Hager and Wellein, Chapter 6 & 7 High Performance
More informationCOMP4300/8300: The OpenMP Programming Model. Alistair Rendell
COMP4300/8300: The OpenMP Programming Model Alistair Rendell See: www.openmp.org Introduction to High Performance Computing for Scientists and Engineers, Hager and Wellein, Chapter 6 & 7 High Performance
More informationCOMP4510 Introduction to Parallel Computation. Shared Memory and OpenMP. Outline (cont d) Shared Memory and OpenMP
COMP4510 Introduction to Parallel Computation Shared Memory and OpenMP Thanks to Jon Aronsson (UofM HPC consultant) for some of the material in these notes. Outline (cont d) Shared Memory and OpenMP Including
More informationOpenMPand the PGAS Model. CMSC714 Sept 15, 2015 Guest Lecturer: Ray Chen
OpenMPand the PGAS Model CMSC714 Sept 15, 2015 Guest Lecturer: Ray Chen LastTime: Message Passing Natural model for distributed-memory systems Remote ( far ) memory must be retrieved before use Programmer
More informationLecture 4. Threads vs. Processes. fork() Threads. Pthreads. Threads in C. Thread Programming January 21, 2005
Threads vs. Processes Lecture 4 Thread Programming January 21, 2005 fork() is expensive (time, memory) Interprocess communication is hard. Threads are lightweight processes: one process can contain several
More informationOpenMP. OpenMP. Portable programming of shared memory systems. It is a quasi-standard. OpenMP-Forum API for Fortran and C/C++
OpenMP OpenMP Portable programming of shared memory systems. It is a quasi-standard. OpenMP-Forum 1997-2002 API for Fortran and C/C++ directives runtime routines environment variables www.openmp.org 1
More informationAdvanced C Programming Winter Term 2008/09. Guest Lecture by Markus Thiele
Advanced C Programming Winter Term 2008/09 Guest Lecture by Markus Thiele Lecture 14: Parallel Programming with OpenMP Motivation: Why parallelize? The free lunch is over. Herb
More informationUnix. Threads & Concurrency. Erick Fredj Computer Science The Jerusalem College of Technology
Unix Threads & Concurrency Erick Fredj Computer Science The Jerusalem College of Technology 1 Threads Processes have the following components: an address space a collection of operating system state a
More informationCS333 Intro to Operating Systems. Jonathan Walpole
CS333 Intro to Operating Systems Jonathan Walpole Threads & Concurrency 2 Threads Processes have the following components: - an address space - a collection of operating system state - a CPU context or
More informationRicardo Rocha. Department of Computer Science Faculty of Sciences University of Porto
Ricardo Rocha Department of Computer Science Faculty of Sciences University of Porto For more information please consult Advanced Programming in the UNIX Environment, 3rd Edition, W. Richard Stevens and
More informationLecture 19: Shared Memory & Synchronization
Lecture 19: Shared Memory & Synchronization COMP 524 Programming Language Concepts Stephen Olivier April 16, 2009 Based on notes by A. Block, N. Fisher, F. Hernandez-Campos, and D. Stotts Forking int pid;
More informationShared Memory Parallelism using OpenMP
Indian Institute of Science Bangalore, India भ रत य व ज ञ न स स थ न ब गल र, भ रत SE 292: High Performance Computing [3:0][Aug:2014] Shared Memory Parallelism using OpenMP Yogesh Simmhan Adapted from: o
More informationScientific Computing
Lecture on Scientific Computing Dr. Kersten Schmidt Lecture 20 Technische Universität Berlin Institut für Mathematik Wintersemester 2014/2015 Syllabus Linear Regression, Fast Fourier transform Modelling
More informationParallel Programming
Parallel Programming OpenMP Nils Moschüring PhD Student (LMU) Nils Moschüring PhD Student (LMU), OpenMP 1 1 Overview What is parallel software development Why do we need parallel computation? Problems
More informationCSE 333 SECTION 9. Threads
CSE 333 SECTION 9 Threads HW4 How s HW4 going? Any Questions? Threads Sequential execution of a program. Contained within a process. Multiple threads can exist within the same process. Every process starts
More informationCS510 Operating System Foundations. Jonathan Walpole
CS510 Operating System Foundations Jonathan Walpole The Process Concept 2 The Process Concept Process a program in execution Program - description of how to perform an activity instructions and static
More informationShared Memory Programming Model
Shared Memory Programming Model Ahmed El-Mahdy and Waleed Lotfy What is a shared memory system? Activity! Consider the board as a shared memory Consider a sheet of paper in front of you as a local cache
More informationParallel Programming using OpenMP
1 OpenMP Multithreaded Programming 2 Parallel Programming using OpenMP OpenMP stands for Open Multi-Processing OpenMP is a multi-vendor (see next page) standard to perform shared-memory multithreading
More informationParallel Programming using OpenMP
1 Parallel Programming using OpenMP Mike Bailey mjb@cs.oregonstate.edu openmp.pptx OpenMP Multithreaded Programming 2 OpenMP stands for Open Multi-Processing OpenMP is a multi-vendor (see next page) standard
More informationDPHPC: Introduction to OpenMP Recitation session
SALVATORE DI GIROLAMO DPHPC: Introduction to OpenMP Recitation session Based on http://openmp.org/mp-documents/intro_to_openmp_mattson.pdf OpenMP An Introduction What is it? A set
More informationLSN 13 Linux Concurrency Mechanisms
LSN 13 Linux Concurrency Mechanisms ECT362 Operating Systems Department of Engineering Technology LSN 13 Creating Processes fork() system call Returns PID of the child process created The new process is
More informationHPC Workshop University of Kentucky May 9, 2007 May 10, 2007
HPC Workshop University of Kentucky May 9, 2007 May 10, 2007 Part 3 Parallel Programming Parallel Programming Concepts Amdahl s Law Parallel Programming Models Tools Compiler (Intel) Math Libraries (Intel)
More informationHigh Performance Computing: Tools and Applications
High Performance Computing: Tools and Applications Edmond Chow School of Computational Science and Engineering Georgia Institute of Technology Lecture 4 OpenMP directives So far we have seen #pragma omp
More informationOpenMP Overview. in 30 Minutes. Christian Terboven / Aachen, Germany Stand: Version 2.
OpenMP Overview in 30 Minutes Christian Terboven 06.12.2010 / Aachen, Germany Stand: 03.12.2010 Version 2.3 Rechen- und Kommunikationszentrum (RZ) Agenda OpenMP: Parallel Regions,
More informationConcurrency and Synchronization. ECE 650 Systems Programming & Engineering Duke University, Spring 2018
Concurrency and Synchronization ECE 650 Systems Programming & Engineering Duke University, Spring 2018 Concurrency Multiprogramming Supported by most all current operating systems More than one unit of
More informationJANUARY 2004 LINUX MAGAZINE Linux in Europe User Mode Linux PHP 5 Reflection Volume 6 / Issue 1 OPEN SOURCE. OPEN STANDARDS.
0104 Cover (Curtis) 11/19/03 9:52 AM Page 1 JANUARY 2004 LINUX MAGAZINE Linux in Europe User Mode Linux PHP 5 Reflection Volume 6 / Issue 1 LINUX M A G A Z I N E OPEN SOURCE. OPEN STANDARDS. THE STATE
More informationThreads. Jo, Heeseung
Threads Jo, Heeseung Multi-threaded program 빠른실행 프로세스를새로생성에드는비용을절약 데이터공유 파일, Heap, Static, Code 의많은부분을공유 CPU 를보다효율적으로활용 코어가여러개일경우코어에 thread 를할당하는방식 2 Multi-threaded program Pros. Cons. 대량의데이터처리에적합 - CPU
More informationCOSC 6374 Parallel Computation. Introduction to OpenMP(I) Some slides based on material by Barbara Chapman (UH) and Tim Mattson (Intel)
COSC 6374 Parallel Computation Introduction to OpenMP(I) Some slides based on material by Barbara Chapman (UH) and Tim Mattson (Intel) Edgar Gabriel Fall 2014 Introduction Threads vs. processes Recap of
More informationIntroduction to OpenMP.
Introduction to OpenMP www.openmp.org Motivation Parallelize the following code using threads: for (i=0; i
More informationOpenMP examples. Sergeev Efim. Singularis Lab, Ltd. Senior software engineer
OpenMP examples Sergeev Efim Senior software engineer Singularis Lab, Ltd. OpenMP Is: An Application Program Interface (API) that may be used to explicitly direct multi-threaded, shared memory parallelism.
More informationParallel Computing Parallel Programming Languages Hwansoo Han
Parallel Computing Parallel Programming Languages Hwansoo Han Parallel Programming Practice Current Start with a parallel algorithm Implement, keeping in mind Data races Synchronization Threading syntax
More informationCSEN 602-Operating Systems, Spring 2018 Practice Assignment 2 Solutions Discussion:
CSEN 602-Operating Systems, Spring 2018 Practice Assignment 2 Solutions Discussion: 10.2.2018-15.2.2018 Exercise 2-1: Reading Read sections 2.1 (except 2.1.7), 2.2.1 till 2.2.5. 1 Exercise 2-2 In Fig.1,
More informationParallel Computing. Lecture 13: OpenMP - I
CSCI-UA.0480-003 Parallel Computing Lecture 13: OpenMP - I Mohamed Zahran (aka Z) mzahran@cs.nyu.edu http://www.mzahran.com Small and Easy Motivation #include #include int main() {
More informationCS 470 Spring Mike Lam, Professor. OpenMP
CS 470 Spring 2018 Mike Lam, Professor OpenMP OpenMP Programming language extension Compiler support required "Open Multi-Processing" (open standard; latest version is 4.5) Automatic thread-level parallelism
More informationPCS - Part Two: Multiprocessor Architectures
PCS - Part Two: Multiprocessor Architectures Institute of Computer Engineering University of Lübeck, Germany Baltic Summer School, Tartu 2008 Part 2 - Contents Multiprocessor Systems Symmetrical Multiprocessors
More information