EE382N (20): Computer Architecture - Parallelism and Locality Spring 2015 Lecture 14 Parallelism in Software I
|
|
- Buck Sherman
- 5 years ago
- Views:
Transcription
1 EE382 (20): Computer Architecture - Parallelism and Locality Spring 2015 Lecture 14 Parallelism in Software I Mattan Erez The University of Texas at Austin EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez 1
2 Credits 2 Most of the slides courtesy Dr. Rodric Rabbah (IBM) Taken from IAP taught at MIT in 2007 Parallel Scan slides courtesy David Kirk (VIDIA) and Wen-Mei Hwu (UIUC) Taken from EE493-AI taught at UIUC in Sprig 2007 EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
3 Parallel programming from scratch 3 Start with an algorithm Formal representation of problem solution Sequence of steps Make sure there is parallelism In each algorithm step Minimize synchronization points Don t forget locality Communication is costly Performance, Energy, System cost More often start with existing sequential code EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
4 Reengineering for Parallelism 4 Define a testing protocol Identify program hot spots: where is most of the time spent? Look at code Use profiling tools Parallelization Start with hot spots first Make sequences of small changes, each followed by testing Patterns provide guidance EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
5 4 Common Steps to Creating a Parallel Program 5 Partitioning d e c o m p o s i t i o n a s s i g n m e n t UE 0 UE 1 UE 2 UE 3 o r c h e s t r a t i o n UE 0 UE 1 UE 2 UE 3 m a p p i n g UE 0 UE 1 UE 2 UE 3 Sequential Computation Tasks Units of Execution Parallel program Processors EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
6 Decomposition 6 Identify concurrency and decide at what level to exploit it Break up computation into tasks to be divided among processes Tasks may become available dynamically umber of tasks may vary with time Enough tasks to keep processors busy umber of tasks available at a time is upper bound on achievable speedup Main consideration: coverage and Amdahl s Law EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
7 Coverage 7 Amdahl's Law: The performance improvement to be gained from using some faster mode of execution is limited by the fraction of the time the faster mode can be used. Demonstration of the law of diminishing returns EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
8 Amdahl s Law 8 Potential program speedup is defined by the fraction of code that can be parallelized time Use 5 processors for parallel work 25 seconds + sequential 25 seconds + sequential 50 seconds + parallel 10 seconds + 25 seconds sequential 25 seconds sequential 100 seconds 60 seconds EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
9 Amdahl s Law 9 time Use 5 processors for parallel work 25 seconds + sequential 25 seconds + sequential 50 seconds + parallel 10 seconds + 25 seconds sequential 25 seconds sequential 100 seconds 60 seconds Speedup= old running time / new running time = 100 seconds / 60 seconds = 1.67 (parallel version is 1.67 times faster) EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
10 Amdahl s Law 10 p = fraction of work that can be parallelized n = the number of processor speedup old running new running time time (1 fraction of time to complete sequential work 1 p) p n EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez fraction of time to complete parallel work
11 speedup Implications of Amdahl s Law 11 Speedup tends to tends to infinity as 1 number of processors 1 p Super linear speedups are possible due to registers and caches Typical speedup is less than linear Parallelism number of only processors worthwhile when it dominates execution EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
12 Assignment 12 Specify mechanism to divide work among PEs Balance work and reduce communication Structured approaches usually work well Code inspection or understanding of application Well-known design patterns As programmers, we worry about partitioning first Independent of architecture or programming model? Complexity often affects decisions Architectural model affects decisions Main considerations: granularity and locality EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
13 Fine vs. Coarse Granularity 13 Fine-grain Parallelism Low computation to communication ratio Small amounts of computational work between communication stages High communication overhead Potential HW assist Coarse-grain Parallelism High computation to communication ratio Large amounts of computational work between communication events Harder to load balance efficiently EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
14 Load Balancing vs. Synchronization 14 Fine Coarse PE 0 PE 1 PE 0 PE 1 EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
15 Load Balancing vs. Synchronization 15 Fine Coarse PE 0 PE 1 PE 0 PE 1 Expensive sync coarse granularity Few units of exec + time disparity fine granularity EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
16 Orchestration and Mapping 16 Computation and communication concurrency Preserve locality of data Schedule tasks to satisfy dependences early Survey available mechanisms on target system Main considerations: locality, parallelism, mechanisms (efficiency and dangers) EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
17 Parallel Programming by Pattern 17 Provides a cookbook to systematically guide programmers Decompose, Assign, Orchestrate, Map Can lead to high quality solutions in some domains Provide common vocabulary to the programming community Each pattern has a name, providing a vocabulary for discussing solutions Helps with software reusability, malleability, and modularity Written in prescribed format to allow the reader to quickly understand the solution and its context Otherwise, too difficult for programmers, and software will not fully exploit parallel hardware EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
18 History 18 Berkeley architecture professor Christopher Alexander In 1977, patterns for city planning, landscaping, and architecture in an attempt to capture principles for living design EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
19 Example 167 (p. 783): 6ft Balcony 19 EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
20 Patterns in Object-Oriented Programming 20 Design Patterns: Elements of Reusable Object- Oriented Software (1995) Gang of Four (GOF): Gamma, Helm, Johnson, Vlissides Catalogue of patterns Creation, structural, behavioral EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
21 Patterns for Parallelizing Programs 21 4 Design Spaces Algorithm Expression Finding Concurrency Expose concurrent tasks Algorithm Structure Map tasks to processes to exploit parallel architecture Software Construction Supporting Structures Code and data structuring patterns Implementation Mechanisms Low level mechanisms used to write parallel programs EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez Patterns for Parallel Programming. Mattson, Sanders, and Massingill (2005).
22 Here s my algorithm. Where s the concurrency? 22 MPEG Decoder frequency encoded macroblocks ZigZag IQuantization MPEG bit stream VLD macroblocks, motion vectors split differentially coded motion vectors Motion Vector Decode IDCT Repeat Saturation spatially encoded macroblocks join motion vectors Motion Compensation Picture Reorder recovered picture Color Conversion Display EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
23 Here s my algorithm. Where s the concurrency? 23 MPEG Decoder frequency encoded macroblocks ZigZag IQuantization MPEG bit stream VLD macroblocks, motion vectors split differentially coded motion vectors Motion Vector Decode Task decomposition Independent coarse-grained computation Inherent to algorithm IDCT Saturation spatially encoded macroblocks join Motion Compensation Picture Reorder recovered picture Repeat motion vectors Sequence of statements (instructions) that operate together as a group Corresponds to some logical part of program Usually follows from the way programmer thinks about a problem Color Conversion Display EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
24 Here s my algorithm. Where s the concurrency? 24 MPEG Decoder frequency encoded macroblocks ZigZag MPEG bit stream VLD macroblocks, motion vectors split differentially coded motion vectors Task decomposition Parallelism in the application IQuantization IDCT Saturation Motion Vector Decode Repeat Pipeline task decomposition Data assembly lines spatially encoded macroblocks join motion vectors Producer-consumer chains Motion Compensation recovered picture Picture Reorder Color Conversion Display EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
25 Here s my algorithm. Where s the concurrency? 25 MPEG Decoder frequency encoded macroblocks ZigZag MPEG bit stream VLD macroblocks, motion vectors split differentially coded motion vectors Task decomposition Parallelism in the application IQuantization IDCT Saturation Motion Vector Decode Repeat Pipeline task decomposition Data assembly lines spatially encoded macroblocks join motion vectors Producer-consumer chains Motion Compensation Picture Reorder Color Conversion recovered picture Data decomposition Same computation is applied to small data chunks derived from large data set Display EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
26 Guidelines for Task Decomposition 26 Algorithms start with a good understanding of the problem being solved Programs often naturally decompose into tasks Two common decompositions are Function calls and Distinct loop iterations Easier to start with many tasks and later fuse them, rather than too few tasks and later try to split them EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
27 Guidelines for Task Decomposition 27 Flexibility Program design should afford flexibility in the number and size of tasks generated Tasks should not tied to a specific architecture Fixed tasks vs. Parameterized tasks Efficiency Tasks should have enough work to amortize the cost of creating and managing them Tasks should be sufficiently independent so that managing dependencies doesn t become the bottleneck Simplicity The code has to remain readable and easy to understand, and debug EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
28 Case for Pipeline Decomposition 28 Data is flowing through a sequence of stages Assembly line is a good analogy ZigZag IQuantization IDCT What s a prime example of pipeline decomposition in computer architecture? Instruction pipeline in modern CPUs Saturation What s an example pipeline you may use in your UIX shell? Pipes in UIX: cat foobar.c grep bar wc Other examples Signal processing Graphics EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
29 Guidelines for Data Decomposition 29 Data decomposition is often implied by task decomposition Programmers need to address task and data decomposition to create a parallel program Which decomposition to start with? Data decomposition is a good starting point when Main computation is organized around manipulation of a large data structure Similar operations are applied to different parts of the data structure EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
30 Common Data Decompositions 30 Geometric data structures Decomposition of arrays along rows, columns, blocks Decomposition of meshes into domains EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
31 Common Data Decompositions 31 Geometric data structures Decomposition of arrays along rows, columns, blocks Decomposition of meshes into domains Recursive data structures Example: decomposition of trees into sub-trees problem subproblem split split subproblem split compute subproblem compute subproblem compute subproblem compute subproblem merge merge subproblem subproblem merge EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez solution
32 Guidelines for Data Decomposition 32 Flexibility Size and number of data chunks should support a wide range of executions Efficiency Data chunks should generate comparable amounts of work (for load balancing) Simplicity Complex data compositions can get difficult to manage and debug EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
33 Data Decomposition Examples 33 Molecular dynamics Compute forces Update accelerations and velocities Update positions Decomposition Baseline algorithm is 2 All-to-all communication Best decomposition is to treat mols. as a set Some advantages to geometric discussed in future lecture EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez
34 Data Decomposition Examples 34 Molecular dynamics Geometric decomposition Merge sort Recursive decomposition subproblem split problem split subproblem split compute subproblem compute subproblem compute subproblem compute subproblem merge merge subproblem subproblem merge EE382: Parallelilsm and Locality, Spring Lecture 14 (c) Rodric Rabbah, Mattan Erez solution
EE382N (20): Computer Architecture - Parallelism and Locality Fall 2011 Lecture 11 Parallelism in Software II
EE382 (20): Computer Architecture - Parallelism and Locality Fall 2011 Lecture 11 Parallelism in Software II Mattan Erez The University of Texas at Austin EE382: Parallelilsm and Locality, Fall 2011 --
More informationEE382N (20): Computer Architecture - Parallelism and Locality Lecture 10 Parallelism in Software I
EE382 (20): Computer Architecture - Parallelism and Locality Lecture 10 Parallelism in Software I Mattan Erez The University of Texas at Austin EE382: Parallelilsm and Locality (c) Rodric Rabbah, Mattan
More informationEE382N (20): Computer Architecture - Parallelism and Locality Lecture 13 Parallelism in Software IV
EE382 (20): Computer Architecture - Parallelism and Locality Lecture 13 Parallelism in Software IV Mattan Erez The University of Texas at Austin EE382: Parallelilsm and Locality (c) Rodric Rabbah, Mattan
More informationEE382N (20): Computer Architecture - Parallelism and Locality Lecture 11 Parallelism in Software II
EE382 (20): Computer Architecture - Parallelism and Locality Lecture 11 Parallelism in Software II Mattan Erez The University of Texas at Austin EE382: Parallelilsm and Locality (c) Rodric Rabbah, Mattan
More informationPatterns for! Parallel Programming!
Lecture 4! Patterns for! Parallel Programming! John Cavazos! Dept of Computer & Information Sciences! University of Delaware!! www.cis.udel.edu/~cavazos/cisc879! Lecture Overview Writing a Parallel Program
More informationAn Introduction to Parallel Programming
An Introduction to Parallel Programming Ing. Andrea Marongiu (a.marongiu@unibo.it) Includes slides from Multicore Programming Primer course at Massachusetts Institute of Technology (MIT) by Prof. SamanAmarasinghe
More informationEE382N (20): Computer Architecture - Parallelism and Locality Fall 2011 Lecture 14 Parallelism in Software V
EE382 (20): Computer Architecture - Parallelism and Locality Fall 2011 Lecture 14 Parallelism in Software V Mattan Erez The University of Texas at Austin EE382: Parallelilsm and Locality, Fall 2011 --
More informationEE382N (20): Computer Architecture - Parallelism and Locality Spring 2015 Lecture 13 Parallelism in Software I
EE382N (20): Computer Architecture - Parallelism and Locality Spring 2015 Lecture 13 Parallelism in Software I Mattan Erez The University of Texas at Austin N EE382N: Parallelilsm and Locality, Spring
More informationPatterns for! Parallel Programming II!
Lecture 4! Patterns for! Parallel Programming II! John Cavazos! Dept of Computer & Information Sciences! University of Delaware! www.cis.udel.edu/~cavazos/cisc879! Task Decomposition Also known as functional
More informationParallel Programming Concepts. Parallel Algorithms. Peter Tröger
Parallel Programming Concepts Parallel Algorithms Peter Tröger Sources: Ian Foster. Designing and Building Parallel Programs. Addison-Wesley. 1995. Mattson, Timothy G.; S, Beverly A.; ers,; Massingill,
More informationParallelism I. John Cavazos. Dept of Computer & Information Sciences University of Delaware
Parallelism I John Cavazos Dept of Computer & Information Sciences University of Delaware Lecture Overview Thinking in Parallel Flynn s Taxonomy Types of Parallelism Parallelism Basics Design Patterns
More informationMarco Danelutto. May 2011, Pisa
Marco Danelutto Dept. of Computer Science, University of Pisa, Italy May 2011, Pisa Contents 1 2 3 4 5 6 7 Parallel computing The problem Solve a problem using n w processing resources Obtaining a (close
More informationCOSC 6374 Parallel Computation. Parallel Design Patterns. Edgar Gabriel. Fall Design patterns
COSC 6374 Parallel Computation Parallel Design Patterns Fall 2014 Design patterns A design pattern is a way of reusing abstract knowledge about a problem and its solution Patterns are devices that allow
More informationWorkloads Programmierung Paralleler und Verteilter Systeme (PPV)
Workloads Programmierung Paralleler und Verteilter Systeme (PPV) Sommer 2015 Frank Feinbube, M.Sc., Felix Eberhardt, M.Sc., Prof. Dr. Andreas Polze Workloads 2 Hardware / software execution environment
More informationPatterns of Parallel Programming with.net 4. Ade Miller Microsoft patterns & practices
Patterns of Parallel Programming with.net 4 Ade Miller (adem@microsoft.com) Microsoft patterns & practices Introduction Why you should care? Where to start? Patterns walkthrough Conclusions (and a quiz)
More informationMattan Erez. The University of Texas at Austin
EE382V (17325): Principles in Computer Architecture Parallelism and Locality Fall 2007 Lecture 12 GPU Architecture (NVIDIA G80) Mattan Erez The University of Texas at Austin Outline 3D graphics recap and
More informationParallelization Strategy
COSC 335 Software Design Parallel Design Patterns (II) Spring 2008 Parallelization Strategy Finding Concurrency Structure the problem to expose exploitable concurrency Algorithm Structure Supporting Structure
More informationParallelism in Software
Parallelism in Software Minsoo Ryu Department of Computer Science and Engineering 2 1 Parallelism in Software 2 Creating a Multicore Program 3 Multicore Design Patterns 4 Q & A 2 3 Types of Parallelism
More informationIntroduction to Parallel Programming
Introduction to Parallel Programming David Lifka lifka@cac.cornell.edu May 23, 2011 5/23/2011 www.cac.cornell.edu 1 y What is Parallel Programming? Using more than one processor or computer to complete
More informationAde Miller Senior Development Manager Microsoft patterns & practices
Ade Miller (adem@microsoft.com) Senior Development Manager Microsoft patterns & practices Save time and reduce risk on your software development projects by incorporating patterns & practices, Microsoft's
More informationIntroduction to Parallel Computing
Portland State University ECE 588/688 Introduction to Parallel Computing Reference: Lawrence Livermore National Lab Tutorial https://computing.llnl.gov/tutorials/parallel_comp/ Copyright by Alaa Alameldeen
More informationEE382N (20): Computer Architecture - Parallelism and Locality Fall 2011 Lecture 23 Memory Systems
EE382 (20): Computer Architecture - Parallelism and Locality Fall 2011 Lecture 23 Memory Systems Mattan Erez The University of Texas at Austin EE382: Principles of Computer Architecture, Fall 2011 -- Lecture
More informationReview: Creating a Parallel Program. Programming for Performance
Review: Creating a Parallel Program Can be done by programmer, compiler, run-time system or OS Steps for creating parallel program Decomposition Assignment of tasks to processes Orchestration Mapping (C)
More information6.189 IAP Lecture 5. Parallel Programming Concepts. Dr. Rodric Rabbah, IBM IAP 2007 MIT
6.189 IAP 2007 Lecture 5 Parallel Programming Concepts 1 6.189 IAP 2007 MIT Recap Two primary patterns of multicore architecture design Shared memory Ex: Intel Core 2 Duo/Quad One copy of data shared among
More informationMIT OpenCourseWare Multicore Programming Primer, January (IAP) Please use the following citation format:
MIT OpenCourseWare http://ocw.mit.edu 6.189 Multicore Programming Primer, January (IAP) 2007 Please use the following citation format: Rodric Rabbah, 6.189 Multicore Programming Primer, January (IAP) 2007.
More informationParallelization Strategy
COSC 6374 Parallel Computation Algorithm structure Spring 2008 Parallelization Strategy Finding Concurrency Structure the problem to expose exploitable concurrency Algorithm Structure Supporting Structure
More informationLecture 27 Programming parallel hardware" Suggested reading:" (see next slide)"
Lecture 27 Programming parallel hardware" Suggested reading:" (see next slide)" 1" Suggested Readings" Readings" H&P: Chapter 7 especially 7.1-7.8" Introduction to Parallel Computing" https://computing.llnl.gov/tutorials/parallel_comp/"
More informationEE382N (20): Computer Architecture - Parallelism and Locality Spring 2015 Lecture 09 GPUs (II) Mattan Erez. The University of Texas at Austin
EE382 (20): Computer Architecture - ism and Locality Spring 2015 Lecture 09 GPUs (II) Mattan Erez The University of Texas at Austin 1 Recap 2 Streaming model 1. Use many slimmed down cores to run in parallel
More informationDesigning Parallel Programs. This review was developed from Introduction to Parallel Computing
Designing Parallel Programs This review was developed from Introduction to Parallel Computing Author: Blaise Barney, Lawrence Livermore National Laboratory references: https://computing.llnl.gov/tutorials/parallel_comp/#whatis
More informationA Study of High Performance Computing and the Cray SV1 Supercomputer. Michael Sullivan TJHSST Class of 2004
A Study of High Performance Computing and the Cray SV1 Supercomputer Michael Sullivan TJHSST Class of 2004 June 2004 0.1 Introduction A supercomputer is a device for turning compute-bound problems into
More informationSome aspects of parallel program design. R. Bader (LRZ) G. Hager (RRZE)
Some aspects of parallel program design R. Bader (LRZ) G. Hager (RRZE) Finding exploitable concurrency Problem analysis 1. Decompose into subproblems perhaps even hierarchy of subproblems that can simultaneously
More informationAnalytical Modeling of Parallel Programs
2014 IJEDR Volume 2, Issue 1 ISSN: 2321-9939 Analytical Modeling of Parallel Programs Hardik K. Molia Master of Computer Engineering, Department of Computer Engineering Atmiya Institute of Technology &
More informationThinking parallel. Decomposition. Thinking parallel. COMP528 Ways of exploiting parallelism, or thinking parallel
COMP528 Ways of exploiting parallelism, or thinking parallel www.csc.liv.ac.uk/~alexei/comp528 Alexei Lisitsa Dept of computer science University of Liverpool a.lisitsa@.liverpool.ac.uk Thinking parallel
More informationAnalytical Modeling of Parallel Programs
Analytical Modeling of Parallel Programs Alexandre David Introduction to Parallel Computing 1 Topic overview Sources of overhead in parallel programs. Performance metrics for parallel systems. Effect of
More informationCo-synthesis and Accelerator based Embedded System Design
Co-synthesis and Accelerator based Embedded System Design COE838: Embedded Computer System http://www.ee.ryerson.ca/~courses/coe838/ Dr. Gul N. Khan http://www.ee.ryerson.ca/~gnkhan Electrical and Computer
More informationCS 31: Intro to Systems Threading & Parallel Applications. Kevin Webb Swarthmore College November 27, 2018
CS 31: Intro to Systems Threading & Parallel Applications Kevin Webb Swarthmore College November 27, 2018 Reading Quiz Making Programs Run Faster We all like how fast computers are In the old days (1980
More informationI-1 Introduction. I-0 Introduction. Objectives:
I-0 Introduction Objectives: Explain necessity of parallel/multithreaded algorithms. Describe different forms of parallel processing. Present commonly used architectures. Introduce a few basic terms. Comments:
More informationEE382N (20): Computer Architecture - Parallelism and Locality Fall 2011 Lecture 18 GPUs (III)
EE382 (20): Computer Architecture - Parallelism and Locality Fall 2011 Lecture 18 GPUs (III) Mattan Erez The University of Texas at Austin EE382: Principles of Computer Architecture, Fall 2011 -- Lecture
More informationParallel Programming Patterns. Overview and Concepts
Parallel Programming Patterns Overview and Concepts Outline Practical Why parallel programming? Decomposition Geometric decomposition Task farm Pipeline Loop parallelism Performance metrics and scaling
More informationLecture 13: Memory Consistency. + a Course-So-Far Review. Parallel Computer Architecture and Programming CMU , Spring 2013
Lecture 13: Memory Consistency + a Course-So-Far Review Parallel Computer Architecture and Programming Today: what you should know Understand the motivation for relaxed consistency models Understand the
More informationA Pattern-supported Parallelization Approach
A Pattern-supported Parallelization Approach Ralf Jahr, Mike Gerdes, Theo Ungerer University of Augsburg, Germany The 2013 International Workshop on Programming Models and Applications for Multicores and
More informationIntroduction to Parallel Programming
Introduction to Parallel Programming Linda Woodard CAC 19 May 2010 Introduction to Parallel Computing on Ranger 5/18/2010 www.cac.cornell.edu 1 y What is Parallel Programming? Using more than one processor
More informationPrinciples of Parallel Algorithm Design: Concurrency and Mapping
Principles of Parallel Algorithm Design: Concurrency and Mapping John Mellor-Crummey Department of Computer Science Rice University johnmc@rice.edu COMP 422/534 Lecture 3 17 January 2017 Last Thursday
More informationConstructing Virtual Architectures on Tiled Processors. David Wentzlaff Anant Agarwal MIT
Constructing Virtual Architectures on Tiled Processors David Wentzlaff Anant Agarwal MIT 1 Emulators and JITs for Multi-Core David Wentzlaff Anant Agarwal MIT 2 Why Multi-Core? 3 Why Multi-Core? Future
More informationComputer and Information Sciences College / Computer Science Department CS 207 D. Computer Architecture. Lecture 9: Multiprocessors
Computer and Information Sciences College / Computer Science Department CS 207 D Computer Architecture Lecture 9: Multiprocessors Challenges of Parallel Processing First challenge is % of program inherently
More informationParallel Computing. Slides credit: M. Quinn book (chapter 3 slides), A Grama book (chapter 3 slides)
Parallel Computing 2012 Slides credit: M. Quinn book (chapter 3 slides), A Grama book (chapter 3 slides) Parallel Algorithm Design Outline Computational Model Design Methodology Partitioning Communication
More informationDesign of Parallel Algorithms. Models of Parallel Computation
+ Design of Parallel Algorithms Models of Parallel Computation + Chapter Overview: Algorithms and Concurrency n Introduction to Parallel Algorithms n Tasks and Decomposition n Processes and Mapping n Processes
More informationComputer Architecture Lecture 27: Multiprocessors. Prof. Onur Mutlu Carnegie Mellon University Spring 2015, 4/6/2015
18-447 Computer Architecture Lecture 27: Multiprocessors Prof. Onur Mutlu Carnegie Mellon University Spring 2015, 4/6/2015 Assignments Lab 7 out Due April 17 HW 6 Due Friday (April 10) Midterm II April
More information18-447: Computer Architecture Lecture 30B: Multiprocessors. Prof. Onur Mutlu Carnegie Mellon University Spring 2013, 4/22/2013
18-447: Computer Architecture Lecture 30B: Multiprocessors Prof. Onur Mutlu Carnegie Mellon University Spring 2013, 4/22/2013 Readings: Multiprocessing Required Amdahl, Validity of the single processor
More informationPrinciples of Parallel Algorithm Design: Concurrency and Decomposition
Principles of Parallel Algorithm Design: Concurrency and Decomposition John Mellor-Crummey Department of Computer Science Rice University johnmc@rice.edu COMP 422/534 Lecture 2 12 January 2017 Parallel
More informationHigh Performance Computing on GPUs using NVIDIA CUDA
High Performance Computing on GPUs using NVIDIA CUDA Slides include some material from GPGPU tutorial at SIGGRAPH2007: http://www.gpgpu.org/s2007 1 Outline Motivation Stream programming Simplified HW and
More informationTask Graph. Name: Problem: Context: D B. C Antecedent. Task Graph
Graph Name: Graph Note: The Graph pattern is a concurrent execution pattern and should not be confused with the Arbitrary Static Graph architectural pattern (1) which addresses the overall organization
More informationHuge market -- essentially all high performance databases work this way
11/5/2017 Lecture 16 -- Parallel & Distributed Databases Parallel/distributed databases: goal provide exactly the same API (SQL) and abstractions (relational tables), but partition data across a bunch
More informationIntroduction to Parallel Programming
Introduction to Parallel Programming January 14, 2015 www.cac.cornell.edu What is Parallel Programming? Theoretically a very simple concept Use more than one processor to complete a task Operationally
More informationHigh Performance Computing
The Need for Parallelism High Performance Computing David McCaughan, HPC Analyst SHARCNET, University of Guelph dbm@sharcnet.ca Scientific investigation traditionally takes two forms theoretical empirical
More information6.189 IAP Lecture 12. StreamIt Parallelizing Compiler. Prof. Saman Amarasinghe, MIT IAP 2007 MIT
6.89 IAP 2007 Lecture 2 StreamIt Parallelizing Compiler 6.89 IAP 2007 MIT Common Machine Language Represent common properties of architectures Necessary for performance Abstract away differences in architectures
More informationComputer Architecture: Parallel Processing Basics. Prof. Onur Mutlu Carnegie Mellon University
Computer Architecture: Parallel Processing Basics Prof. Onur Mutlu Carnegie Mellon University Readings Required Hill, Jouppi, Sohi, Multiprocessors and Multicomputers, pp. 551-560 in Readings in Computer
More informationPerformance Optimization Part II: Locality, Communication, and Contention
Lecture 7: Performance Optimization Part II: Locality, Communication, and Contention Parallel Computer Architecture and Programming CMU 15-418/15-618, Spring 2015 Tunes Beth Rowley Nobody s Fault but Mine
More informationCache Aware Optimization of Stream Programs
Cache Aware Optimization of Stream Programs Janis Sermulins, William Thies, Rodric Rabbah and Saman Amarasinghe LCTES Chicago, June 2005 Streaming Computing Is Everywhere! Prevalent computing domain with
More informationΠαράλληλη Επεξεργασία
Παράλληλη Επεξεργασία Μέτρηση και σύγκριση Παράλληλης Απόδοσης Γιάννος Σαζεϊδης Εαρινό Εξάμηνο 2013 HW 1. Homework #3 due on cuda (summary of Tesla paper on web page) Slides based on Lin and Snyder textbook
More informationSerial. Parallel. CIT 668: System Architecture 2/14/2011. Topics. Serial and Parallel Computation. Parallel Computing
CIT 668: System Architecture Parallel Computing Topics 1. What is Parallel Computing? 2. Why use Parallel Computing? 3. Types of Parallelism 4. Amdahl s Law 5. Flynn s Taxonomy of Parallel Computers 6.
More informationLecture 1: Gentle Introduction to GPUs
CSCI-GA.3033-004 Graphics Processing Units (GPUs): Architecture and Programming Lecture 1: Gentle Introduction to GPUs Mohamed Zahran (aka Z) mzahran@cs.nyu.edu http://www.mzahran.com Who Am I? Mohamed
More informationCMSC Computer Architecture Lecture 12: Multi-Core. Prof. Yanjing Li University of Chicago
CMSC 22200 Computer Architecture Lecture 12: Multi-Core Prof. Yanjing Li University of Chicago Administrative Stuff! Lab 4 " Due: 11:49pm, Saturday " Two late days with penalty! Exam I " Grades out on
More informationParallel Programming Patterns Overview and Concepts
Parallel Programming Patterns Overview and Concepts Partners Funding Reusing this material This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike 4.0 International License.
More informationAn Introduction to Software Architecture. David Garlan & Mary Shaw 94
An Introduction to Software Architecture David Garlan & Mary Shaw 94 Motivation Motivation An increase in (system) size and complexity structural issues communication (type, protocol) synchronization data
More informationExploring different level of parallelism Instruction-level parallelism (ILP): how many of the operations/instructions in a computer program can be performed simultaneously 1. e = a + b 2. f = c + d 3.
More informationBasics of Performance Engineering
ERLANGEN REGIONAL COMPUTING CENTER Basics of Performance Engineering J. Treibig HiPerCH 3, 23./24.03.2015 Why hardware should not be exposed Such an approach is not portable Hardware issues frequently
More informationDatabase Management and Tuning
Database Management and Tuning Concurrency Tuning Johann Gamper Free University of Bozen-Bolzano Faculty of Computer Science IDSE Unit 8 May 10, 2012 Acknowledgements: The slides are provided by Nikolaus
More informationChapter 10: Performance Patterns
Chapter 10: Performance Patterns Patterns A pattern is a common solution to a problem that occurs in many different contexts Patterns capture expert knowledge about best practices in software design in
More informationPrinciples of Parallel Algorithm Design: Concurrency and Mapping
Principles of Parallel Algorithm Design: Concurrency and Mapping John Mellor-Crummey Department of Computer Science Rice University johnmc@rice.edu COMP 422/534 Lecture 3 28 August 2018 Last Thursday Introduction
More informationAn Introduction to Software Architecture By David Garlan & Mary Shaw 94
IMPORTANT NOTICE TO STUDENTS These slides are NOT to be used as a replacement for student notes. These slides are sometimes vague and incomplete on purpose to spark a class discussion An Introduction to
More informationProgrammazione Avanzata e Paradigmi Ingegneria e Scienze Informatiche - UNIBO a.a 2013/2014 Lecturer: Alessandro Ricci
v1.0 20140421 Programmazione Avanzata e Paradigmi Ingegneria e Scienze Informatiche - UNIBO a.a 2013/2014 Lecturer: Alessandro Ricci [module 3.1] ELEMENTS OF CONCURRENT PROGRAM DESIGN 1 STEPS IN DESIGN
More informationPorting an MPEG-2 Decoder to the Cell Architecture
Porting an MPEG-2 Decoder to the Cell Architecture Troy Brant, Jonathan Clark, Brian Davidson, Nick Merryman Advisor: David Bader College of Computing Georgia Institute of Technology Atlanta, GA 30332-0250
More informationIntroduction to parallel computers and parallel programming. Introduction to parallel computersand parallel programming p. 1
Introduction to parallel computers and parallel programming Introduction to parallel computersand parallel programming p. 1 Content A quick overview of morden parallel hardware Parallelism within a chip
More informationProgramming as Successive Refinement. Partitioning for Performance
Programming as Successive Refinement Not all issues dealt with up front Partitioning often independent of architecture, and done first View machine as a collection of communicating processors balancing
More informationCSE 613: Parallel Programming. Lecture 2 ( Analytical Modeling of Parallel Algorithms )
CSE 613: Parallel Programming Lecture 2 ( Analytical Modeling of Parallel Algorithms ) Rezaul A. Chowdhury Department of Computer Science SUNY Stony Brook Spring 2017 Parallel Execution Time & Overhead
More informationCSC630/CSC730 Parallel & Distributed Computing
CSC630/CSC730 Parallel & Distributed Computing Analytical Modeling of Parallel Programs Chapter 5 1 Contents Sources of Parallel Overhead Performance Metrics Granularity and Data Mapping Scalability 2
More informationMPEG-2 Decoding in a Stream Programming Language
MPEG-2 Decoding in a Stream Programming Language Matthew Drake, Hank Hoffmann, Rodric Rabbah and Saman Amarasinghe Massachusetts Institute of Technology IPDPS Rhodes, April 2006 1 http://cag.csail.mit.edu/streamit
More informationCS4961 Parallel Programming. Lecture 5: Data and Task Parallelism, cont. 9/8/09. Administrative. Mary Hall September 8, 2009.
CS4961 Parallel Programming Lecture 5: Data and Task Parallelism, cont. Administrative Homework 2 posted, due September 10 before class - Use the handin program on the CADE machines - Use the following
More informationParallelizing Graphics Pipeline Execution (+ Basics of Characterizing a Rendering Workload)
Lecture 2: Parallelizing Graphics Pipeline Execution (+ Basics of Characterizing a Rendering Workload) Visual Computing Systems Analyzing a 3D Graphics Workload Where is most of the work done? Memory Vertex
More informationDesign of Digital Circuits Lecture 21: GPUs. Prof. Onur Mutlu ETH Zurich Spring May 2017
Design of Digital Circuits Lecture 21: GPUs Prof. Onur Mutlu ETH Zurich Spring 2017 12 May 2017 Agenda for Today & Next Few Lectures Single-cycle Microarchitectures Multi-cycle and Microprogrammed Microarchitectures
More informationMPEG-4: Simple Profile (SP)
MPEG-4: Simple Profile (SP) I-VOP (Intra-coded rectangular VOP, progressive video format) P-VOP (Inter-coded rectangular VOP, progressive video format) Short Header mode (compatibility with H.263 codec)
More informationTechno Expert Solutions An institute for specialized studies! Introduction to Advance QTP course Content
Introduction to Advance QTP course Content NTRODUCTION TO AUTOMATION Automation Testing Benefits of Automation Testing Automation Testing Vs Manual Testing Automation Test Tools Tool selection criteria
More informationCSC630/COS781: Parallel & Distributed Computing
CSC630/COS781: Parallel & Distributed Computing Algorithm Design Chapter 3 (3.1-3.3) 1 Contents Preliminaries of parallel algorithm design Decomposition Task dependency Task dependency graph Granularity
More information1 Software Architecture
Some buzzwords and acronyms for today Software architecture Design pattern Separation of concerns Single responsibility principle Keep it simple, stupid (KISS) Don t repeat yourself (DRY) Don t talk to
More informationDesign Patterns. An introduction
Design Patterns An introduction Introduction Designing object-oriented software is hard, and designing reusable object-oriented software is even harder. Your design should be specific to the problem at
More informationOutline. Database Tuning. Ideal Transaction. Concurrency Tuning Goals. Concurrency Tuning. Nikolaus Augsten. Lock Tuning. Unit 8 WS 2013/2014
Outline Database Tuning Nikolaus Augsten University of Salzburg Department of Computer Science Database Group 1 Unit 8 WS 2013/2014 Adapted from Database Tuning by Dennis Shasha and Philippe Bonnet. Nikolaus
More informationOnline Course Evaluation. What we will do in the last week?
Online Course Evaluation Please fill in the online form The link will expire on April 30 (next Monday) So far 10 students have filled in the online form Thank you if you completed it. 1 What we will do
More informationParallelism and runtimes
Parallelism and runtimes Advanced Course on Compilers Spring 2015 (III-V): Lecture 7 Vesa Hirvisalo ESG/CSE/Aalto Today Parallel platforms Concurrency Consistency Examples of parallelism Regularity of
More informationCMSC 714 Lecture 6 MPI vs. OpenMP and OpenACC. Guest Lecturer: Sukhyun Song (original slides by Alan Sussman)
CMSC 714 Lecture 6 MPI vs. OpenMP and OpenACC Guest Lecturer: Sukhyun Song (original slides by Alan Sussman) Parallel Programming with Message Passing and Directives 2 MPI + OpenMP Some applications can
More informationPrinciple Of Parallel Algorithm Design (cont.) Alexandre David B2-206
Principle Of Parallel Algorithm Design (cont.) Alexandre David B2-206 1 Today Characteristics of Tasks and Interactions (3.3). Mapping Techniques for Load Balancing (3.4). Methods for Containing Interaction
More informationMPEG-2. And Scalability Support. Nimrod Peleg Update: July.2004
MPEG-2 And Scalability Support Nimrod Peleg Update: July.2004 MPEG-2 Target...Generic coding method of moving pictures and associated sound for...digital storage, TV broadcasting and communication... Dedicated
More informationUsing Industry Standards to Exploit the Advantages and Resolve the Challenges of Multicore Technology
Using Industry Standards to Exploit the Advantages and Resolve the Challenges of Multicore Technology September 19, 2007 Markus Levy, EEMBC and Multicore Association Enabling the Multicore Ecosystem Multicore
More informationMultimedia Decoder Using the Nios II Processor
Multimedia Decoder Using the Nios II Processor Third Prize Multimedia Decoder Using the Nios II Processor Institution: Participants: Instructor: Indian Institute of Science Mythri Alle, Naresh K. V., Svatantra
More informationCS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring Lecture 18: Naming, Directories, and File Caching
CS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring 2004 Lecture 18: Naming, Directories, and File Caching 18.0 Main Points How do users name files? What is a name? Lookup:
More informationCS 590: High Performance Computing. Parallel Computer Architectures. Lab 1 Starts Today. Already posted on Canvas (under Assignment) Let s look at it
Lab 1 Starts Today Already posted on Canvas (under Assignment) Let s look at it CS 590: High Performance Computing Parallel Computer Architectures Fengguang Song Department of Computer Science IUPUI 1
More informationECE 669 Parallel Computer Architecture
ECE 669 Parallel Computer Architecture Lecture 9 Workload Evaluation Outline Evaluation of applications is important Simulation of sample data sets provides important information Working sets indicate
More informationParallel Numerics, WT 2013/ Introduction
Parallel Numerics, WT 2013/2014 1 Introduction page 1 of 122 Scope Revise standard numerical methods considering parallel computations! Required knowledge Numerics Parallel Programming Graphs Literature
More informationCS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring Lecture 18: Naming, Directories, and File Caching
CS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring 2002 Lecture 18: Naming, Directories, and File Caching 18.0 Main Points How do users name files? What is a name? Lookup:
More informationApplication Architectures, Design Patterns
Application Architectures, Design Patterns Martin Ledvinka martin.ledvinka@fel.cvut.cz Winter Term 2017 Martin Ledvinka (martin.ledvinka@fel.cvut.cz) Application Architectures, Design Patterns Winter Term
More information