Fork / Join Parallelism
|
|
|
- Ruth Walters
- 6 years ago
- Views:
Transcription
1
2 Fork / Join Parallelism Image courtesy of Speedup limited by linear portion Amdahl s Law, Speedup = 1 / [(1- F) + F/S] Synchronization wait time
3 OpenMP: API allows us to code multi- threaded programs Share work across threads to increase performance Use directives to create multi- threaded code segments #pragma omp parallel! Code inside block will be run on each thread Helper functions omp_get_num_threads() = Number of Threads running omp_get_thread_num() = Current Thread ID
4 Duplication #pragma omp parallel! for! {! for(int i=0; i<array_size; i++)! z[i] = x[i] + y[i];! }!! False Sharing #pragma omp parallel! {! int thread_id = omp_get_thread_num();! int num_threads = omp_get_num_threads();! for(int i=thread_id; i<array_size; i+=num_threads)! z[i] = x[i] + y[i];! }! Data Race #pragma omp parallel for! {! for(int i=0; i<array_size; i++)! sum += x[i];! }
5 int values[100000];! #pragma omp barrier #pragma omp parallel! is a directive that {!!int i = omp_get_thread_num();! requires all threads!int n = omp_get_num_threads();! to wait until other!for (; i < ; i += n) {! threads reach the!!values[i] = i;! barrier before!!#pragma omp barrier! proceeding.!!}! }!!! Suppose each core has a 32 KiB direct mapped data cache with 64- byte blocks. (Write- back, write- allocate) If the snippet is run with one thread, how many cache misses for the values array? 6250 If the snippet is run with two threads, how many cache misses for the values array? 100,000 Name the phenomenon that explains the difference between these? False Sharing Using two threads and removing the barrier, would the number of cache misses decrease? Why? Yes, one thread could get far enough ahead and start working on a different cache block.
6 Without changing the #pragma statement, alter this code to correctly take advantage of mulqthreading. (Ignore edge cases)! int values[100000];! #pragma omp parallel! {! int i = omp_get_thread_num();! int n = omp_get_num_threads();! for (; i < ; i += n)! values[i] = i;! }! int values[100000];! #pragma omp parallel! {! int i = omp_get_thread_num();! int n = omp_get_num_threads();! for (100000/n*i; i < /n*(i+1); i += 1)! values[i] = I;!! }!
7 Combinational Logic Circuit Diagram, Truth Table, Boolean Expression
8 Sequential Logic State elements, circuits that remember Controls flow of information between CL blocks
9 Sequential Logic Timing Setup Time: Input must be stable before the CLK trigger for proper read Hold Time: Input must be stable after the CLK trigger for proper read CLK- to- Q Delay: Takes time for output to change after the CLK trigger Must design stateful circuits with these in mind
10 Sequential Logic Timing
11 Garcia Spring 2007 F2 We are trying to implement an FSM that outputs a 1 iff the last 3 input bits form a palindrome. Assume that the input was all 1s before the circuit begins.! Example:! I: ! O: !
12 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0! 0! 0! S00! S10! 0! 0! 1! S01! S11! 0! 1! 0! 0! 1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 0! 1! 1! 1!
13 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! S00! S10! 0! 0! 1! S01! S11! 0! 1! 0! 0! 1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 0! 1! 1! 1!
14 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! S00! S10! 0! 0! 1! 0! 0! 1! S01! S11! 0! 1! 0! 0! 1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 0! 1! 1! 1!
15 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! S00! S10! 0! 0! 1! 0! 0! 1! S01! S11! 0! 1! 0! 1! 1! 0! 0! 1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 0! 1! 1! 1!
16 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! S00! S10! 0! 0! 1! 0! 0! 1! S01! S11! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 0! 1! 1! 1!
17 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! S00! S10! 0! 0! 1! 0! 0! 1! S01! S11! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 1! 0! 1! 1! 1!
18 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! S00! S10! 0! 0! 1! 0! 0! 1! S01! S11! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 0! 1! 1! 1! 0! 1! 1! 1!
19 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! S00! S10! 0! 0! 1! 0! 0! 1! S01! S11! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 0! 1! 1! 1! 0! 0! 1! 0! 1! 1! 1!
20 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! S00! S10! 0! 0! 1! 0! 0! 1! S01! S11! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 0! 1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 1! 1! 1!
21 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0/1! 0! 0! 0! 1! 0! 0! S00! S10! 0! 0! 1! 0! 0! 1! S01! S11! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 0! 1! 1/1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 1! 1! 1!
22 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0/1! 0! 0! 0! 1! 0! 0! S00! S10! 0! 0! 1! 0! 0! 1! S01! 1/1! 0/1! S11! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 0! 1! 1/1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 1! 1! 1!
23 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0/1! 0! 0! 0! 1! 0! 0! S00! 0/0! S10! 0! 0! 1! 0! 0! 1! 1/1! 0/1! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! S01! 1/0! S11! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 0! 1! 1/1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 1! 1! 1!
24 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0/1! 0! 0! 0! 1! 0! 0! S00! 0/0! S10! 0! 0! 1! 0! 0! 1! 1/0! 1/1! 0/1! 0/0! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! S01! 1/0! S11! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 0! 1! 1/1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 1! 1! 1!
25 Outputs a 1 iff the last 3 input bits form a palindrome P1! P0! I! O! N1! N0! 0/1! 0! 0! 0! 1! 0! 0! S00! 0/0! S10! 0! 0! 1! 0! 0! 1! 1/0! 1/1! 0/1! 0/0! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! S01! 1/0! S11! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 0! 1! 1/1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 1! 1! 1!
26 Provide a fully reduced Boolean expression for the Output as a function of P1, P0, and I! P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! 0! 0! 1! 0! 0! 1! 0! 1! 0! 1! 1! 0! What is the name of this circuit? 0! 1! 1! 0! 1! 1! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 0! 1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 1! 1! 1!
27 Provide a fully reduced Boolean expression for the Output as a function of P1, P0, and I! P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! 0! 0! 1! 0! 0! 1! 0! 1! 0! 1! 1! 0! What is the name of this circuit? 0! 1! 1! 0! 1! 1! 1! 0! 0! 0! 0! 0! 1! 0! 1! 1! 0! 1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 1! 1! 1!
28 Provide a fully reduced Boolean expression for the Output as a function of P1, P0, and I! P1! P0! I! O! N1! N0! 0! 0! 0! 1! 0! 0! 0! 0! 1! 0! 0! 1! 0! 1! 0! 1! 1! 0! 0! 1! 1! 0! 1! 1! 1! 0! 0! 0! 0! 0! What is the name of this circuit? XNOR! 1! 0! 1! 1! 0! 1! 1! 1! 0! 0! 1! 0! 1! 1! 1! 1! 1! 1!
29 What is the shortest length input stream that can guarantee we test this circuit exhaustively?! 0/1! 0/0! S00! S10! 1/0! 1/1! 0/1! 0/0! S01! 1/0! S11! 1/1!
30 What is the shortest length input stream that can guarantee we test this circuit exhaustively?! 0/1! S00! 0/0! S10! 1/1! 1/0! 0/1! 0/0! S01! 1/0! S11! Find a path through all transitions starting from S11:! 8 ( )! 1/1!
31 If we were to implement this circuit using our standard FSM model, what is the smallest clock period T we can drive our system with? Answer in terms of T_setup, T_hold, T_clk- to- q, and T_CL.
32 If we were to implement this circuit using our standard FSM model, what is the smallest clock period T we can drive our system with? Answer in terms of T_setup, T_hold, T_clk- to- q, and T_CL. T = T_clk-to-q + T_CL + T_setup!
33 Servers on a rack / rack part of cluster Process high volume of independent requests Computation partitioned across servers Web services rely on them
34 Issues Load balancing Power usage vs compute load (1/2 peak power when idle) Cope with failures gracefully (redundancy) Elaborate hierarchy of memories
35 Power Usage Efficiency PUE = Total Building Power / IT equipment power Where does power go? Cooling Power Distribution Lighting 1.0 = Ideal maximum
36 Application of WSC Scalable Add more machines, increase performance Strong: Speed increases with more processors Weak: Speed increases with larger problem size
37 Apply Map to user supplied key/value pairs Data is sliced into shards and distributed Emit intermediate key/value pairs Apply Reduce to all values that share the same key Produces a set of merged output values Let MapReduce library handle details, focus on implementing Map and Reduce functions
38 Implement Map and Reduce functions to calculate the number of unique words of certain lengths. Example input: The quick brown fox jumped Output: (3, 2) (5, 2) (6, 1) Map (String title, String text):!! Reduce ( key, list( ) values):!!!
39 Implement Map and Reduce functions to calculate the number of unique words of certain lengths. Example input: The quick brown fox jumped Output: (3, 2) (5, 2) (6, 1) Map (String title, String text):!! Reduce (Integer key, list(integer) values):!
40 Implement Map and Reduce functions to calculate the number of unique words of certain lengths. Example input: The quick brown fox jumped Output: (3, 2) (5, 2) (6, 1) Map (String title, String text):! dict = {}! for word in text:! if!dict.get(word):! emit(len(word), 1))! dict.put(word)! Reduce (Integer key, list(integer) values):!
41 Implement Map and Reduce functions to calculate the number of unique words of certain lengths. Example input: The quick brown fox jumped Output: (3, 2) (5, 2) (6, 1) Map (String title, String text):! dict = {}! for word in text:! if!dict.get(word):! emit(len(word), 1))! dict.put(word)! Reduce (Integer key, list(integer) values):! emit(key, sum(list))!
42 Calculate cost of customer s orders based on a list of transactions. Input: (CustomerID, ItemInfo{name, qty, price}) Map (Integer ID, ItemInfo info):!! Reduce ( key, list( ) values):!!
43 Calculate cost of customer s orders based on a list of transactions. Input: (CustomerID, ItemInfo{name, qty, price}) Map (Integer ID, ItemInfo info):!! Reduce (Integer key, list(integer) values):!!
44 Calculate cost of customer s orders based on a list of transactions. Input: (CustomerID, ItemInfo{name, qty, price}) Map (Integer ID, ItemInfo info):! emit(id, info.qty*info.price)! Reduce (Integer key, list(integer) values):!!
45 Calculate cost of customer s orders based on single transactions. Input: (CustomerID, ItemInfo{name, qty, price}) Map (Integer ID, ItemInfo info):! emit(id, info.qty*info.price)! Reduce (Integer key, list(integer) values):! emit(key, sum(values))!
Wide operands. CP1: hardware can multiply 64-bit floating-point numbers RAM MUL. core
RAM MUL core Wide operands RAM MUL core CP1: hardware can multiply 64-bit floating-point numbers Pipelining: can start the next independent operation before the previous result is available RAM MUL core
Some features of modern CPUs. and how they help us
Some features of modern CPUs and how they help us RAM MUL core Wide operands RAM MUL core CP1: hardware can multiply 64-bit floating-point numbers Pipelining: can start the next independent operation before
Parallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 17 Shared-memory Programming 1 Outline n OpenMP n Shared-memory model n Parallel for loops n Declaring private variables n Critical
OpenMP. A parallel language standard that support both data and functional Parallelism on a shared memory system
OpenMP A parallel language standard that support both data and functional Parallelism on a shared memory system Use by system programmers more than application programmers Considered a low level primitives
High Performance Computing Systems
High Performance Computing Systems Shared Memory Doug Shook Shared Memory Bottlenecks Trips to memory Cache coherence 2 Why Multicore? Shared memory systems used to be purely the domain of HPC... What
Parallel Programming with OpenMP. CS240A, T. Yang
Parallel Programming with OpenMP CS240A, T. Yang 1 A Programmer s View of OpenMP What is OpenMP? Open specification for Multi-Processing Standard API for defining multi-threaded shared-memory programs
Parallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 17 Shared-memory Programming Outline OpenMP Shared-memory model Parallel for loops Declaring private variables Critical sections Reductions
1 of 6 Lecture 7: March 4. CISC 879 Software Support for Multicore Architectures Spring Lecture 7: March 4, 2008
1 of 6 Lecture 7: March 4 CISC 879 Software Support for Multicore Architectures Spring 2008 Lecture 7: March 4, 2008 Lecturer: Lori Pollock Scribe: Navreet Virk Open MP Programming Topics covered 1. Introduction
EE/CSCI 451: Parallel and Distributed Computation
EE/CSCI 451: Parallel and Distributed Computation Lecture #7 2/5/2017 Xuehai Qian Xuehai.qian@usc.edu http://alchem.usc.edu/portal/xuehaiq.html University of Southern California 1 Outline From last class
Shared Memory Programming with OpenMP (3)
Shared Memory Programming with OpenMP (3) 2014 Spring Jinkyu Jeong (jinkyu@skku.edu) 1 SCHEDULING LOOPS 2 Scheduling Loops (2) parallel for directive Basic partitioning policy block partitioning Iteration
Amdahl s Law. AMath 483/583 Lecture 13 April 25, Amdahl s Law. Amdahl s Law. Today: Amdahl s law Speed up, strong and weak scaling OpenMP
AMath 483/583 Lecture 13 April 25, 2011 Amdahl s Law Today: Amdahl s law Speed up, strong and weak scaling OpenMP Typically only part of a computation can be parallelized. Suppose 50% of the computation
Introduction to OpenMP. OpenMP basics OpenMP directives, clauses, and library routines
Introduction to OpenMP Introduction OpenMP basics OpenMP directives, clauses, and library routines What is OpenMP? What does OpenMP stands for? What does OpenMP stands for? Open specifications for Multi
The Art of Parallel Processing
The Art of Parallel Processing Ahmad Siavashi April 2017 The Software Crisis As long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a
OpenMP - III. Diego Fabregat-Traver and Prof. Paolo Bientinesi WS15/16. HPAC, RWTH Aachen
OpenMP - III Diego Fabregat-Traver and Prof. Paolo Bientinesi HPAC, RWTH Aachen fabregat@aices.rwth-aachen.de WS15/16 OpenMP References Using OpenMP: Portable Shared Memory Parallel Programming. The MIT
Parallel Programming
Parallel Programming OpenMP Dr. Hyrum D. Carroll November 22, 2016 Parallel Programming in a Nutshell Load balancing vs Communication This is the eternal problem in parallel computing. The basic approaches
Sometimes incorrect Always incorrect Slower than serial Faster than serial. Sometimes incorrect Always incorrect Slower than serial Faster than serial
CS 61C Summer 2016 Discussion 10 SIMD and OpenMP SIMD PRACTICE A.SIMDize the following code: void count (int n, float *c) { for (int i= 0 ; i
Parallel Numerical Algorithms
Parallel Numerical Algorithms http://sudalab.is.s.u-tokyo.ac.jp/~reiji/pna16/ [ 8 ] OpenMP Parallel Numerical Algorithms / IST / UTokyo 1 PNA16 Lecture Plan General Topics 1. Architecture and Performance
Module 10: Open Multi-Processing Lecture 19: What is Parallelization? The Lecture Contains: What is Parallelization? Perfectly Load-Balanced Program
The Lecture Contains: What is Parallelization? Perfectly Load-Balanced Program Amdahl's Law About Data What is Data Race? Overview to OpenMP Components of OpenMP OpenMP Programming Model OpenMP Directives
ECE 574 Cluster Computing Lecture 10
ECE 574 Cluster Computing Lecture 10 Vince Weaver http://www.eece.maine.edu/~vweaver vincent.weaver@maine.edu 1 October 2015 Announcements Homework #4 will be posted eventually 1 HW#4 Notes How granular
CS 590: High Performance Computing. Parallel Computer Architectures. Lab 1 Starts Today. Already posted on Canvas (under Assignment) Let s look at it
Lab 1 Starts Today Already posted on Canvas (under Assignment) Let s look at it CS 590: High Performance Computing Parallel Computer Architectures Fengguang Song Department of Computer Science IUPUI 1
Shared Memory Programming Model
Shared Memory Programming Model Ahmed El-Mahdy and Waleed Lotfy What is a shared memory system? Activity! Consider the board as a shared memory Consider a sheet of paper in front of you as a local cache
CMSC 714 Lecture 4 OpenMP and UPC. Chau-Wen Tseng (from A. Sussman)
CMSC 714 Lecture 4 OpenMP and UPC Chau-Wen Tseng (from A. Sussman) Programming Model Overview Message passing (MPI, PVM) Separate address spaces Explicit messages to access shared data Send / receive (MPI
OpenMP Programming. Prof. Thomas Sterling. High Performance Computing: Concepts, Methods & Means
High Performance Computing: Concepts, Methods & Means OpenMP Programming Prof. Thomas Sterling Department of Computer Science Louisiana State University February 8 th, 2007 Topics Introduction Overview
Parallel Programming. OpenMP Parallel programming for multiprocessors for loops
Parallel Programming OpenMP Parallel programming for multiprocessors for loops OpenMP OpenMP An application programming interface (API) for parallel programming on multiprocessors Assumes shared memory
Parallel Programming in C with MPI and OpenMP
Parallel Programming in C with MPI and OpenMP Michael J. Quinn Chapter 17 Shared-memory Programming 1 Outline n OpenMP n Shared-memory model n Parallel for loops n Declaring private variables n Critical
ELE 455/555 Computer System Engineering. Section 4 Parallel Processing Class 1 Challenges
ELE 455/555 Computer System Engineering Section 4 Class 1 Challenges Introduction Motivation Desire to provide more performance (processing) Scaling a single processor is limited Clock speeds Power concerns
OpenMPand the PGAS Model. CMSC714 Sept 15, 2015 Guest Lecturer: Ray Chen
OpenMPand the PGAS Model CMSC714 Sept 15, 2015 Guest Lecturer: Ray Chen LastTime: Message Passing Natural model for distributed-memory systems Remote ( far ) memory must be retrieved before use Programmer
Parallel Computing. Lecture 13: OpenMP - I
CSCI-UA.0480-003 Parallel Computing Lecture 13: OpenMP - I Mohamed Zahran (aka Z) mzahran@cs.nyu.edu http://www.mzahran.com Small and Easy Motivation #include #include int main() {
HPC Practical Course Part 3.1 Open Multi-Processing (OpenMP)
HPC Practical Course Part 3.1 Open Multi-Processing (OpenMP) V. Akishina, I. Kisel, G. Kozlov, I. Kulakov, M. Pugach, M. Zyzak Goethe University of Frankfurt am Main 2015 Task Parallelism Parallelization
OpenMP Introduction. CS 590: High Performance Computing. OpenMP. A standard for shared-memory parallel programming. MP = multiprocessing
CS 590: High Performance Computing OpenMP Introduction Fengguang Song Department of Computer Science IUPUI OpenMP A standard for shared-memory parallel programming. MP = multiprocessing Designed for systems
CS 61C: Great Ideas in Computer Architecture. MapReduce
CS 61C: Great Ideas in Computer Architecture MapReduce Guest Lecturer: Justin Hsia 3/06/2013 Spring 2013 Lecture #18 1 Review of Last Lecture Performance latency and throughput Warehouse Scale Computing
High Performance Computing: Tools and Applications
High Performance Computing: Tools and Applications Edmond Chow School of Computational Science and Engineering Georgia Institute of Technology Lecture 2 OpenMP Shared address space programming High-level
Parallel Computing Parallel Programming Languages Hwansoo Han
Parallel Computing Parallel Programming Languages Hwansoo Han Parallel Programming Practice Current Start with a parallel algorithm Implement, keeping in mind Data races Synchronization Threading syntax
OpenMP - II. Diego Fabregat-Traver and Prof. Paolo Bientinesi WS15/16. HPAC, RWTH Aachen
OpenMP - II Diego Fabregat-Traver and Prof. Paolo Bientinesi HPAC, RWTH Aachen fabregat@aices.rwth-aachen.de WS15/16 OpenMP References Using OpenMP: Portable Shared Memory Parallel Programming. The MIT
Lecture 13: Memory Consistency. + a Course-So-Far Review. Parallel Computer Architecture and Programming CMU , Spring 2013
Lecture 13: Memory Consistency + a Course-So-Far Review Parallel Computer Architecture and Programming Today: what you should know Understand the motivation for relaxed consistency models Understand the
ITCS 4/5145 Parallel Computing Test 1 5:00 pm - 6:15 pm, Wednesday February 17, 2016 Solutions Name:...
ITCS 4/5145 Parallel Computing Test 1 5:00 pm - 6:15 pm, Wednesday February 17, 016 Solutions Name:... Answer questions in space provided below questions. Use additional paper if necessary but make sure
CS691/SC791: Parallel & Distributed Computing
CS691/SC791: Parallel & Distributed Computing Introduction to OpenMP 1 Contents Introduction OpenMP Programming Model and Examples OpenMP programming examples Task parallelism. Explicit thread synchronization.
Parallel Computing. Hwansoo Han (SKKU)
Parallel Computing Hwansoo Han (SKKU) Unicore Limitations Performance scaling stopped due to Power consumption Wire delay DRAM latency Limitation in ILP 10000 SPEC CINT2000 2 cores/chip Xeon 3.0GHz Core2duo
15-418, Spring 2008 OpenMP: A Short Introduction
15-418, Spring 2008 OpenMP: A Short Introduction This is a short introduction to OpenMP, an API (Application Program Interface) that supports multithreaded, shared address space (aka shared memory) parallelism.
OpenMP programming. Thomas Hauser Director Research Computing Research CU-Boulder
OpenMP programming Thomas Hauser Director Research Computing thomas.hauser@colorado.edu CU meetup 1 Outline OpenMP Shared-memory model Parallel for loops Declaring private variables Critical sections Reductions
COMP4510 Introduction to Parallel Computation. Shared Memory and OpenMP. Outline (cont d) Shared Memory and OpenMP
COMP4510 Introduction to Parallel Computation Shared Memory and OpenMP Thanks to Jon Aronsson (UofM HPC consultant) for some of the material in these notes. Outline (cont d) Shared Memory and OpenMP Including
CS 61C: Great Ideas in Computer Architecture. Amdahl s Law, Thread Level Parallelism
CS 61C: Great Ideas in Computer Architecture Amdahl s Law, Thread Level Parallelism Instructor: Alan Christopher 07/17/2014 Summer 2014 -- Lecture #15 1 Review of Last Lecture Flynn Taxonomy of Parallel
OpenMP Algoritmi e Calcolo Parallelo. Daniele Loiacono
OpenMP Algoritmi e Calcolo Parallelo References Useful references Using OpenMP: Portable Shared Memory Parallel Programming, Barbara Chapman, Gabriele Jost and Ruud van der Pas OpenMP.org http://openmp.org/
ECE Spring 2017 Exam 2
ECE 56300 Spring 2017 Exam 2 All questions are worth 5 points. For isoefficiency questions, do not worry about breaking costs down to t c, t w and t s. Question 1. Innovative Big Machines has developed
Multithreading in C with OpenMP
Multithreading in C with OpenMP ICS432 - Spring 2017 Concurrent and High-Performance Programming Henri Casanova (henric@hawaii.edu) Pthreads are good and bad! Multi-threaded programming in C with Pthreads
Concurrent Programming with OpenMP
Concurrent Programming with OpenMP Parallel and Distributed Computing Department of Computer Science and Engineering (DEI) Instituto Superior Técnico March 7, 2016 CPD (DEI / IST) Parallel and Distributed
COMP Parallel Computing. SMM (2) OpenMP Programming Model
COMP 633 - Parallel Computing Lecture 7 September 12, 2017 SMM (2) OpenMP Programming Model Reading for next time look through sections 7-9 of the Open MP tutorial Topics OpenMP shared-memory parallel
Programming Shared Memory Systems with OpenMP Part I. Book
Programming Shared Memory Systems with OpenMP Part I Instructor Dr. Taufer Book Parallel Programming in OpenMP by Rohit Chandra, Leo Dagum, Dave Kohr, Dror Maydan, Jeff McDonald, Ramesh Menon 2 1 Machine
2 GHz = 500 picosec frequency. Vars declared outside of main() are in static. 2 # oset bits = block size Put starting arrow in FSM diagrams
CS 61C Fall 2011 Kenny Do Final cheat sheet Increment memory addresses by multiples of 4, since lw and sw are bytealigned When going from C to Mips, always use addu, addiu, and subu When saving stu into
OpenMP I. Diego Fabregat-Traver and Prof. Paolo Bientinesi WS16/17. HPAC, RWTH Aachen
OpenMP I Diego Fabregat-Traver and Prof. Paolo Bientinesi HPAC, RWTH Aachen fabregat@aices.rwth-aachen.de WS16/17 OpenMP References Using OpenMP: Portable Shared Memory Parallel Programming. The MIT Press,
OpenMP 2. CSCI 4850/5850 High-Performance Computing Spring 2018
OpenMP 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning Objectives
CSE 613: Parallel Programming. Lecture 2 ( Analytical Modeling of Parallel Algorithms )
CSE 613: Parallel Programming Lecture 2 ( Analytical Modeling of Parallel Algorithms ) Rezaul A. Chowdhury Department of Computer Science SUNY Stony Brook Spring 2017 Parallel Execution Time & Overhead
Cache Awareness. Course Level: CS1/CS2. PDC Concepts Covered: PDC Concept Locality False Sharing
Cache Awareness Course Level: CS1/CS PDC Concepts Covered: PDC Concept Locality False Sharing Bloom Level C C Programming Knowledge Prerequisites: Know how to compile Java/C++ Be able to understand loops
Topics. Introduction. Shared Memory Parallelization. Example. Lecture 11. OpenMP Execution Model Fork-Join model 5/15/2012. Introduction OpenMP
Topics Lecture 11 Introduction OpenMP Some Examples Library functions Environment variables 1 2 Introduction Shared Memory Parallelization OpenMP is: a standard for parallel programming in C, C++, and
Principles of Software Construction: Objects, Design, and Concurrency. The Perils of Concurrency Can't live with it. Cant live without it.
Principles of Software Construction: Objects, Design, and Concurrency The Perils of Concurrency Can't live with it. Cant live without it. Spring 2014 Charlie Garrod Christian Kästner School of Computer
Acknowledgments. Amdahl s Law. Contents. Programming with MPI Parallel programming. 1 speedup = (1 P )+ P N. Type to enter text
Acknowledgments Programming with MPI Parallel ming Jan Thorbecke Type to enter text This course is partly based on the MPI courses developed by Rolf Rabenseifner at the High-Performance Computing-Center
Introduc)on to OpenMP
Introduc)on to OpenMP Chapter 5.1-5. Bryan Mills, PhD Spring 2017 OpenMP An API for shared-memory parallel programming. MP = multiprocessing Designed for systems in which each thread or process can potentially
Parallel Algorithm Engineering
Parallel Algorithm Engineering Kenneth S. Bøgh PhD Fellow Based on slides by Darius Sidlauskas Outline Background Current multicore architectures UMA vs NUMA The openmp framework and numa control Examples
OpenMP 4. CSCI 4850/5850 High-Performance Computing Spring 2018
OpenMP 4 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning Objectives
HPC Workshop University of Kentucky May 9, 2007 May 10, 2007
HPC Workshop University of Kentucky May 9, 2007 May 10, 2007 Part 3 Parallel Programming Parallel Programming Concepts Amdahl s Law Parallel Programming Models Tools Compiler (Intel) Math Libraries (Intel)
A common scenario... Most of us have probably been here. Where did my performance go? It disappeared into overheads...
OPENMP PERFORMANCE 2 A common scenario... So I wrote my OpenMP program, and I checked it gave the right answers, so I ran some timing tests, and the speedup was, well, a bit disappointing really. Now what?.
EPL372 Lab Exercise 5: Introduction to OpenMP
EPL372 Lab Exercise 5: Introduction to OpenMP References: https://computing.llnl.gov/tutorials/openmp/ http://openmp.org/wp/openmp-specifications/ http://openmp.org/mp-documents/openmp-4.0-c.pdf http://openmp.org/mp-documents/openmp4.0.0.examples.pdf
OpenMP. Diego Fabregat-Traver and Prof. Paolo Bientinesi WS16/17. HPAC, RWTH Aachen
OpenMP Diego Fabregat-Traver and Prof. Paolo Bientinesi HPAC, RWTH Aachen fabregat@aices.rwth-aachen.de WS16/17 Worksharing constructs To date: #pragma omp parallel created a team of threads We distributed
EE/CSCI 451: Parallel and Distributed Computation
EE/CSCI 451: Parallel and Distributed Computation Lecture #11 2/21/2017 Xuehai Qian Xuehai.qian@usc.edu http://alchem.usc.edu/portal/xuehaiq.html University of Southern California 1 Outline Midterm 1:
HPX. High Performance ParalleX CCT Tech Talk Series. Hartmut Kaiser
HPX High Performance CCT Tech Talk Hartmut Kaiser (hkaiser@cct.lsu.edu) 2 What s HPX? Exemplar runtime system implementation Targeting conventional architectures (Linux based SMPs and clusters) Currently,
High Performance Computing. Introduction to Parallel Computing
High Performance Computing Introduction to Parallel Computing Acknowledgements Content of the following presentation is borrowed from The Lawrence Livermore National Laboratory https://hpc.llnl.gov/training/tutorials
ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective
ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Data Center Software Architecture: Topic 3: Programming Models Piccolo: Building Fast, Distributed Programs
OpenMP. António Abreu. Instituto Politécnico de Setúbal. 1 de Março de 2013
OpenMP António Abreu Instituto Politécnico de Setúbal 1 de Março de 2013 António Abreu (Instituto Politécnico de Setúbal) OpenMP 1 de Março de 2013 1 / 37 openmp what? It s an Application Program Interface
Parallel Programming. Exploring local computational resources OpenMP Parallel programming for multiprocessors for loops
Parallel Programming Exploring local computational resources OpenMP Parallel programming for multiprocessors for loops Single computers nowadays Several CPUs (cores) 4 to 8 cores on a single chip Hyper-threading
CS61C Summer 2012 Final
Login: cs61c CS61C Summer 2012 Final Your Name: SID: Your TA (Circle): Raphael Paul Brandon Sung Roa Name of person to your LEFT: Name of person to your RIGHT: This exam is worth 92 points and will count
CS 61C: Great Ideas in Computer Architecture (Machine Structures) Thread-Level Parallelism (TLP) and OpenMP
CS 61C: Great Ideas in Computer Architecture (Machine Structures) Thread-Level Parallelism (TLP) and OpenMP Instructors: John Wawrzynek & Vladimir Stojanovic http://inst.eecs.berkeley.edu/~cs61c/ Review
Shared Memory Parallel Programming
Shared Memory Parallel Programming Abhishek Somani, Debdeep Mukhopadhyay Mentor Graphics, IIT Kharagpur August 2, 2015 Abhishek, Debdeep (IIT Kgp) Parallel Programming August 2, 2015 1 / 46 Overview 1
CPSC 3300 Spring 2016 Final Exam Version A No Calculators
CPSC 3300 Spring 2016 Final Exam Version A No Calculators Name: 1. Find the execution time of a program that executes 8 billion instructions on a processor with an average CPI of 2 and a clock frequency
Lecture 14: Mixed MPI-OpenMP programming. Lecture 14: Mixed MPI-OpenMP programming p. 1
Lecture 14: Mixed MPI-OpenMP programming Lecture 14: Mixed MPI-OpenMP programming p. 1 Overview Motivations for mixed MPI-OpenMP programming Advantages and disadvantages The example of the Jacobi method
Advanced C Programming Winter Term 2008/09. Guest Lecture by Markus Thiele
Advanced C Programming Winter Term 2008/09 Guest Lecture by Markus Thiele Lecture 14: Parallel Programming with OpenMP Motivation: Why parallelize? The free lunch is over. Herb
DPHPC: Introduction to OpenMP Recitation session
SALVATORE DI GIROLAMO DPHPC: Introduction to OpenMP Recitation session Based on http://openmp.org/mp-documents/intro_to_openmp_mattson.pdf OpenMP An Introduction What is it? A set of compiler directives
Distributed Systems CS /640
Distributed Systems CS 15-440/640 Programming Models Borrowed and adapted from our good friends at CMU-Doha, Qatar Majd F. Sakr, Mohammad Hammoud andvinay Kolar 1 Objectives Discussion on Programming Models
Introduction to OpenMP
Introduction to OpenMP Ekpe Okorafor School of Parallel Programming & Parallel Architecture for HPC ICTP October, 2014 A little about me! PhD Computer Engineering Texas A&M University Computer Science
Introduction to OpenMP
Introduction to OpenMP Ricardo Fonseca https://sites.google.com/view/rafonseca2017/ Outline Shared Memory Programming OpenMP Fork-Join Model Compiler Directives / Run time library routines Compiling and
Compiling for GPUs. Adarsh Yoga Madhav Ramesh
Compiling for GPUs Adarsh Yoga Madhav Ramesh Agenda Introduction to GPUs Compute Unified Device Architecture (CUDA) Control Structure Optimization Technique for GPGPU Compiler Framework for Automatic Translation
OpenMP. Dr. William McDoniel and Prof. Paolo Bientinesi WS17/18. HPAC, RWTH Aachen
OpenMP Dr. William McDoniel and Prof. Paolo Bientinesi HPAC, RWTH Aachen mcdoniel@aices.rwth-aachen.de WS17/18 Loop construct - Clauses #pragma omp for [clause [, clause]...] The following clauses apply:
CS4961 Parallel Programming. Lecture 5: More OpenMP, Introduction to Data Parallel Algorithms 9/5/12. Administrative. Mary Hall September 4, 2012
CS4961 Parallel Programming Lecture 5: More OpenMP, Introduction to Data Parallel Algorithms Administrative Mailing list set up, everyone should be on it - You should have received a test mail last night
An Introduction to OpenMP
An Introduction to OpenMP U N C L A S S I F I E D Slide 1 What Is OpenMP? OpenMP Is: An Application Program Interface (API) that may be used to explicitly direct multi-threaded, shared memory parallelism
Parallel Programming
Parallel Programming OpenMP Nils Moschüring PhD Student (LMU) Nils Moschüring PhD Student (LMU), OpenMP 1 1 Overview What is parallel software development Why do we need parallel computation? Problems
A brief introduction to OpenMP
A brief introduction to OpenMP Alejandro Duran Barcelona Supercomputing Center Outline 1 Introduction 2 Writing OpenMP programs 3 Data-sharing attributes 4 Synchronization 5 Worksharings 6 Task parallelism
Scientific Computing
Lecture on Scientific Computing Dr. Kersten Schmidt Lecture 20 Technische Universität Berlin Institut für Mathematik Wintersemester 2014/2015 Syllabus Linear Regression, Fast Fourier transform Modelling
CS 470 Spring Mike Lam, Professor. OpenMP
CS 470 Spring 2018 Mike Lam, Professor OpenMP OpenMP Programming language extension Compiler support required "Open Multi-Processing" (open standard; latest version is 4.5) Automatic thread-level parallelism
Shared Memory Programming Models I
Shared Memory Programming Models I Peter Bastian / Stefan Lang Interdisciplinary Center for Scientific Computing (IWR) University of Heidelberg INF 368, Room 532 D-69120 Heidelberg phone: 06221/54-8264
PARALLEL ALGORITHM FOR SECOND ORDER RESTRICTED WEAK INTEGER COMPOSITION GENERATION FOR SHARED MEMORY MACHINES
Electronic version of an article published as Parallel Processing Letters, Volume 23, Issue 3, 2013, Pages 1350010 DOI: 10.1142/S0129626413500102 c World Scientific Publishing Company, http://www.worldscientific.com/toc/ppl/23/03
OpenMP examples. Sergeev Efim. Singularis Lab, Ltd. Senior software engineer
OpenMP examples Sergeev Efim Senior software engineer Singularis Lab, Ltd. OpenMP Is: An Application Program Interface (API) that may be used to explicitly direct multi-threaded, shared memory parallelism.
OpenMP * Past, Present and Future
OpenMP * Past, Present and Future Tim Mattson Intel Corporation Microprocessor Technology Labs timothy.g.mattson@intel.com * The name OpenMP is the property of the OpenMP Architecture Review Board. 1 OpenMP
Lecture 4: OpenMP Open Multi-Processing
CS 4230: Parallel Programming Lecture 4: OpenMP Open Multi-Processing January 23, 2017 01/23/2017 CS4230 1 Outline OpenMP another approach for thread parallel programming Fork-Join execution model OpenMP
Parallel programming using OpenMP
Parallel programming using OpenMP Computer Architecture J. Daniel García Sánchez (coordinator) David Expósito Singh Francisco Javier García Blas ARCOS Group Computer Science and Engineering Department
CS 61C: Great Ideas in Computer Architecture (Machine Structures) Warehouse-Scale Computing
CS 61C: Great Ideas in Computer Architecture (Machine Structures) Warehouse-Scale Computing Instructors: Nicholas Weaver & Vladimir Stojanovic http://inst.eecs.berkeley.edu/~cs61c/ Coherency Tracked by
SHARCNET Workshop on Parallel Computing. Hugh Merz Laurentian University May 2008
SHARCNET Workshop on Parallel Computing Hugh Merz Laurentian University May 2008 What is Parallel Computing? A computational method that utilizes multiple processing elements to solve a problem in tandem
Programming at Scale: Concurrency
Programming at Scale: Concurrency 1 Goal: Building Fast, Scalable Software How do we speed up software? 2 What is scalability? A system is scalable if it can easily adapt to increased (or reduced) demand
CS 5220: Shared memory programming. David Bindel
CS 5220: Shared memory programming David Bindel 2017-09-26 1 Message passing pain Common message passing pattern Logical global structure Local representation per processor Local data may have redundancy
Distributed Systems + Middleware Concurrent Programming with OpenMP
Distributed Systems + Middleware Concurrent Programming with OpenMP Gianpaolo Cugola Dipartimento di Elettronica e Informazione Politecnico, Italy cugola@elet.polimi.it http://home.dei.polimi.it/cugola
Review. Lecture 12 5/22/2012. Compiler Directives. Library Functions Environment Variables. Compiler directives for construct, collapse clause
Review Lecture 12 Compiler Directives Conditional compilation Parallel construct Work-sharing constructs for, section, single Synchronization Work-tasking Library Functions Environment Variables 1 2 13b.cpp
Shared Memory Parallelism - OpenMP
Shared Memory Parallelism - OpenMP Sathish Vadhiyar Credits/Sources: OpenMP C/C++ standard (openmp.org) OpenMP tutorial (http://www.llnl.gov/computing/tutorials/openmp/#introduction) OpenMP sc99 tutorial