Chapter 1. Introduction: Part I. Jens Saak Scientific Computing II 7/348
|
|
|
- Ethan Powell
- 5 years ago
- Views:
Transcription
1 Chapter 1 Introduction: Part I Jens Saak Scientific Computing II 7/348
2 Why Parallel Computing? 1. Problem size exceeds desktop capabilities. Jens Saak Scientific Computing II 8/348
3 Why Parallel Computing? 1. Problem size exceeds desktop capabilities. 2. Problem is inherently parallel (e.g. Monte-Carlo simulations). Jens Saak Scientific Computing II 8/348
4 Why Parallel Computing? 1. Problem size exceeds desktop capabilities. 2. Problem is inherently parallel (e.g. Monte-Carlo simulations). 3. Modern architectures require parallel programming skills to be optimally exploited. Jens Saak Scientific Computing II 8/348
5 [Flynn 72] The basic definition of a parallel computer is very vague in order to cover a large class of systems. Important details that are not considered by the definition are: Jens Saak Scientific Computing II 9/348
6 [Flynn 72] The basic definition of a parallel computer is very vague in order to cover a large class of systems. Important details that are not considered by the definition are: How many processing elements? Jens Saak Scientific Computing II 9/348
7 [Flynn 72] The basic definition of a parallel computer is very vague in order to cover a large class of systems. Important details that are not considered by the definition are: How many processing elements? How complex are they? Jens Saak Scientific Computing II 9/348
8 [Flynn 72] The basic definition of a parallel computer is very vague in order to cover a large class of systems. Important details that are not considered by the definition are: How many processing elements? How complex are they? How are they connected? Jens Saak Scientific Computing II 9/348
9 [Flynn 72] The basic definition of a parallel computer is very vague in order to cover a large class of systems. Important details that are not considered by the definition are: How many processing elements? How complex are they? How are they connected? How is their cooperation coordinated? Jens Saak Scientific Computing II 9/348
10 [Flynn 72] The basic definition of a parallel computer is very vague in order to cover a large class of systems. Important details that are not considered by the definition are: How many processing elements? How complex are they? How are they connected? How is their cooperation coordinated? What kind of problems can be solved? The basic classification allowing answers to most of these questions is known as Flynn s taxonomy. It distinguishes four categories of parallel computers. Jens Saak Scientific Computing II 9/348
11 [Flynn 72] The four categories allowing basic answers to the questions on global process control, as well as the resulting data and control flow in the machine are Jens Saak Scientific Computing II 10/348
12 [Flynn 72] The four categories allowing basic answers to the questions on global process control, as well as the resulting data and control flow in the machine are 1. Single-Instruction, Single-Data (SISD) Jens Saak Scientific Computing II 10/348
13 [Flynn 72] The four categories allowing basic answers to the questions on global process control, as well as the resulting data and control flow in the machine are 1. Single-Instruction, Single-Data (SISD) 2. Multiple-Instruction, Single-Data (MISD) Jens Saak Scientific Computing II 10/348
14 [Flynn 72] The four categories allowing basic answers to the questions on global process control, as well as the resulting data and control flow in the machine are 1. Single-Instruction, Single-Data (SISD) 2. Multiple-Instruction, Single-Data (MISD) 3. Single-Instruction, Multiple-Data (SIMD) Jens Saak Scientific Computing II 10/348
15 [Flynn 72] The four categories allowing basic answers to the questions on global process control, as well as the resulting data and control flow in the machine are 1. Single-Instruction, Single-Data (SISD) 2. Multiple-Instruction, Single-Data (MISD) 3. Single-Instruction, Multiple-Data (SIMD) 4. Multiple-Instruction, Multiple-Data (MIMD) Jens Saak Scientific Computing II 10/348
16 Single-Instruction, Single-Data (SISD) The SISD model is characterized by a single processing element, Jens Saak Scientific Computing II 11/348
17 Single-Instruction, Single-Data (SISD) The SISD model is characterized by a single processing element, executing a single instruction, on a single piece of data, in each step of the execution. Jens Saak Scientific Computing II 11/348
18 Single-Instruction, Single-Data (SISD) The SISD model is characterized by a single processing element, executing a single instruction, on a single piece of data, in each step of the execution. It is thus in fact the standard sequential computer implementing, e.g., the von Neumann model. Jens Saak Scientific Computing II 11/348
19 Single-Instruction, Single-Data (SISD) The SISD model is characterized by a single processing element, executing a single instruction, on a single piece of data, in each step of the execution. It is thus in fact the standard sequential computer implementing, e.g., the von Neumann model. Examples desktop computers until Intel Pentium 4 era, early netbooks on Intel Atom basis, pocket calculators, abacus, embedded circuits Jens Saak Scientific Computing II 11/348
20 Single-Instruction, Single-Data (SISD) instruction pool PU data pool Figure: Single-Instruction, Single-Data (SISD) machine model Jens Saak Scientific Computing II 12/348
21 Multiple-Instruction, Single-Data (MISD) In contrast to the SISD model in the MISD architecture we have multiple processing elements, Jens Saak Scientific Computing II 13/348
22 Multiple-Instruction, Single-Data (MISD) In contrast to the SISD model in the MISD architecture we have multiple processing elements, executing a separate instruction each, all on the same single piece of data, in each step of the execution. Jens Saak Scientific Computing II 13/348
23 Multiple-Instruction, Single-Data (MISD) In contrast to the SISD model in the MISD architecture we have multiple processing elements, executing a separate instruction each, all on the same single piece of data, in each step of the execution. The MISD model is usually not considered very useful in practice. Jens Saak Scientific Computing II 13/348
24 Multiple-Instruction, Single-Data (MISD) instruction pool PU PU PU data pool Figure: Multiple-Instruction, Single-Data (MISD) machine model Jens Saak Scientific Computing II 14/348
25 Here the characterization is Single-Instruction, Multiple-Data (SIMD) multiple processing elements, execute the same instruction, on a multiple pieces of data, in each step of the execution. This model is thus the ideal model for all kinds of vector operations c = a + αb. Examples Graphics Processing Units, Vector Computers, SSE (Streaming SIMD Extension) registers of modern CPUs. Jens Saak Scientific Computing II 15/348
26 Single-Instruction, Multiple-Data (SIMD) The attractiveness of the SIMD model for vector operations, i.e., linear algebra operations, comes at a cost. Consider the simple conditional expression if (b==0) c=a; else c=a/b; The SIMD model requires the execution of both cases sequentially. First all processes for which the condition is true execute their assignment, then the other do the second assignment. Therefore, conditionals need to be avoided on SIMD architectures to guarantee maximum performance. Jens Saak Scientific Computing II 16/348
27 Single-Instruction, Multiple-Data (SIMD) instruction pool PU PU PU data pool Figure: Single-Instruction, Multiple-Data (SIMD) machine model Jens Saak Scientific Computing II 17/348
28 MIMD allows Multiple-Instruction, Multiple-Data (MIMD) multiple processing elements, to execute a different instruction, on a separate piece of data, at each instance of time. Jens Saak Scientific Computing II 18/348
29 MIMD allows Multiple-Instruction, Multiple-Data (MIMD) multiple processing elements, to execute a different instruction, on a separate piece of data, at each instance of time. Examples multicore and multi-processor desktop PCs, cluster systems. Jens Saak Scientific Computing II 18/348
30 Multiple-Instruction, Multiple-Data (MIMD): Hardware classes MIMD computer systems can be further divided into three class regarding their memory configuration: distributed memory Every processing element has a certain exclusive portion of the entire memory available in the system. Data needs to be exchanged via an interconnection network. shared memory All processing units in the system can access all data in the main memory. hybrid Certain groups of processing elements share a part of the entire data and instruction storage. Jens Saak Scientific Computing II 19/348
31 Multiple-Instruction, Multiple-Data (MIMD): Programming models Single Program, Multiple Data (SPMD) SPMD is a programming model for MIMD systems. In SPMD multiple autonomous processors simultaneously execute the same program at independent points. 1 This contrasts to the SIMD model where the execution points are not independent. This is opposed to Multiple Program, Multiple Data (MPMD) A different programming model for MIMD systems, where multiple autonomous processing units execute different programs at the same time. Typically Master/Worker like management methods of parallel programs are associated with this class. 1 Wikipedia: Jens Saak Scientific Computing II 20/348
32 Multiple-Instruction, Multiple-Data (MIMD) instruction pool PU PU PU data pool Figure: Multiple-Instruction, Multiple-Data (MIMD) machine model Jens Saak Scientific Computing II 21/348
33 Memory Hierarchies in Parallel Computers Repetition Sequential Processor Cloud Network Storage Local Storage Tape Hard Disk Drive (HDD) Solid State Disk (SSD) Main Random Access Memory (RAM) L3 Cache L2 Cache L1 Cache Registers slow and very slow medium fast Figure: Basic memory hierarchy on a single processor system. Jens Saak Scientific Computing II 22/348
34 Memory Hierarchies in Parallel Computers Shared Memory: UMA Machine (6042MB) Socket P#0 L3 (4096KB) Core P#0 Core P#2 PU P#0 PU P#1 Host: pc812 Indexes: physical Date: Mo 09 Jul :37:17 CEST Figure: A sample dual core Xeon setup Jens Saak Scientific Computing II 23/348
35 Memory Hierarchies in Parallel Computers Shared Memory: UMA + GPU Machine (7992MB) Socket P#0 PCI 10de:0402 L2 (4096KB) L2 (4096KB) PCI 8086:10c0 L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) eth0 L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) PCI 8086:2922 Core P#0 Core P#1 Core P#3 Core P#2 sda sr0 PU P#0 PU P#1 PU P#2 PU P#3 sdb sr1 Figure: A sample Core 2 Quad setup Jens Saak Scientific Computing II 24/348
36 Machine (1024GB) NUMANode P#0 (256GB) Socket P#0 L3 (24MB) Core P#0 PU P#0 Core P#1 NUMANode P#1 (256GB) Socket P#1 L3 (24MB) Core P#0 PU P#1 PU P#4 Core P#1 NUMANode P#2 (256GB) Socket P#2 L3 (24MB) Core P#0 PU P#2 PU P#5 Core P#1 NUMANode P#3 (256GB) Socket P#3 L3 (24MB) Core P#0 PU P#3 Host: editha Indexes: physical PU P#6 Core P#1 PU P#7 Date: Mo 09 Jul :38:22 CEST Core P#2 PU P#8 Core P#2 PU P#9 Core P#2 PU P#10 Core P#2 PU P#11 Core P#8 PU P#12 Core P#8 PU P#13 Core P#8 PU P#14 Core P#8 PU P#15 Core P#17 PU P#16 Core P#17 PU P#17 Core P#17 PU P#18 Core P#17 PU P#19 Core P#18 PU P#20 Core P#18 PU P#21 Core P#18 PU P#22 Core P#18 PU P#23 Core P#24 PU P#24 Core P#24 PU P#25 Core P#24 PU P#26 Core P#24 PU P#27 Core P#25 PU P#28 Core P#25 PU P#29 Core P#25 PU P#30 Core P#25 PU P#31 Memory Hierarchies in Parallel Computers Shared Memory: NUMA Figure: A four processor octa-core Xeon system Jens Saak Scientific Computing II 25/348
37 Memory Hierarchies in Parallel Computers Shared Memory: NUMA + 2 GPUs Machine (32GB) NUMANode P#0 (16GB) Socket P#0 PCI 14e4:165f L3 (20MB) eth2 PCI 14e4:165f L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) eth3 L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) PCI 14e4:165f Core P#0 Core P#1 Core P#2 Core P#3 Core P#4 Core P#5 Core P#6 Core P#7 eth0 PU P#0 PU P#2 PU P#4 PU P#6 PU P#8 PU P#10 PU P#12 PU P#14 PCI 14e4:165f eth1 PCI 1000:005b sda PCI 10de:1094 PCI 102b:0534 PCI 8086:1d02 sr0 NUMANode P#1 (16GB) Socket P#1 PCI 10de:1094 L3 (20MB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1d (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) L1i (32KB) Core P#0 Core P#1 Core P#2 Core P#3 Core P#4 Core P#5 Core P#6 Core P#7 PU P#1 PU P#3 PU P#5 PU P#7 PU P#9 PU P#11 PU P#13 PU P#15 Host: adelheid Indexes: physical Date: Mon Nov 19 16:37: Figure: A dual processor octa-core Xeon system Jens Saak Scientific Computing II 26/348
38 Memory Hierarchies in Parallel Computers General Memory Setting Main Memory system bus cache cache I/O P 1... P n Figure: Schematic of a general parallel system Jens Saak Scientific Computing II 27/348
39 Memory Hierarchies in Parallel Computers General Memory Setting Main Memory Accelerator Device system bus cache cache I/O P 1... P n Figure: Schematic of a general parallel system Jens Saak Scientific Computing II 27/348
40 Memory Hierarchies in Parallel Computers General Memory Setting Main Memory Accelerator Device system bus cache cache I/O Interconnect P 1... P n Figure: Schematic of a general parallel system Jens Saak Scientific Computing II 27/348
41 Communication Networks The Interconnect in the last figure stands for any kind of Communication grid. This can be implemented either as local hardware interconnect, or in the form of network interconnect. In classical supercomputers the first was mainly used, whereas in today s cluster based systems often the network solution is used in the one form or the other. Jens Saak Scientific Computing II 28/348
42 Communication Networks I Hardware MyriNet shipping since 2005 transfer rates up to 10Gbit/s lost significance ( % TOP500 down to 0.8% in 2011) Infiniband transfer rates up to 300Gbit/s most relevant implementation driven by OpenFabrics Alliance 2, usable on Linux, BSD, Windows systems. features remote direct memory access (RDMA) reduced CPU overhead can also be used for TCP/IP communication Omni-Path Jens Saak Scientific Computing II 29/348
43 Communication Networks II Hardware transfer rates up to 100Gbit/s introduced by Intel in 2015/2016 rising significance Intel s approach to address cluster of > nodes 2 Jens Saak Scientific Computing II 30/348
44 Communication Networks Topologies 1. Linear array: nodes aligned on a string each being connected to at most two neighbors. Jens Saak Scientific Computing II 31/348
45 Communication Networks Topologies 1. Linear array: nodes aligned on a string each being connected to at most two neighbors. 2. Ring: nodes are aligned in a ring each being connected to exactly two neighbors Jens Saak Scientific Computing II 31/348
46 Communication Networks Topologies 1. Linear array: nodes aligned on a string each being connected to at most two neighbors. 2. Ring: nodes are aligned in a ring each being connected to exactly two neighbors 3. Complete graph: every node is connected to all other nodes Jens Saak Scientific Computing II 31/348
47 Communication Networks Topologies 1. Linear array: nodes aligned on a string each being connected to at most two neighbors. 2. Ring: nodes are aligned in a ring each being connected to exactly two neighbors 3. Complete graph: every node is connected to all other nodes 4. Mesh and Torus: Every node is connected to a number of neighbours (2-4 in 2d mesh, 4 in 2d torus). Jens Saak Scientific Computing II 31/348
48 Communication Networks Topologies 1. Linear array: nodes aligned on a string each being connected to at most two neighbors. 2. Ring: nodes are aligned in a ring each being connected to exactly two neighbors 3. Complete graph: every node is connected to all other nodes 4. Mesh and Torus: Every node is connected to a number of neighbours (2-4 in 2d mesh, 4 in 2d torus). 5. k-dimensional cube / hypercube: Recursive construction of a well connected network of 2 k nodes each connected to k neighbors. Line for two, square for four, cube fore eight. Jens Saak Scientific Computing II 31/348
49 Communication Networks Topologies 1. Linear array: nodes aligned on a string each being connected to at most two neighbors. 2. Ring: nodes are aligned in a ring each being connected to exactly two neighbors 3. Complete graph: every node is connected to all other nodes 4. Mesh and Torus: Every node is connected to a number of neighbours (2-4 in 2d mesh, 4 in 2d torus). 5. k-dimensional cube / hypercube: Recursive construction of a well connected network of 2 k nodes each connected to k neighbors. Line for two, square for four, cube fore eight. 6. Tree: Nodes are arranged in groups, groups of groups and so forth until only one large group is left, which represents the root of the tree. Jens Saak Scientific Computing II 31/348
Parallel Architectures
Parallel Architectures Part 1: The rise of parallel machines Intel Core i7 4 CPU cores 2 hardware thread per core (8 cores ) Lab Cluster Intel Xeon 4/10/16/18 CPU cores 2 hardware thread per core (8/20/32/36
COSC 6374 Parallel Computation. Parallel Computer Architectures
OS 6374 Parallel omputation Parallel omputer Architectures Some slides on network topologies based on a similar presentation by Michael Resch, University of Stuttgart Edgar Gabriel Fall 2015 Flynn s Taxonomy
COSC 6374 Parallel Computation. Parallel Computer Architectures
OS 6374 Parallel omputation Parallel omputer Architectures Some slides on network topologies based on a similar presentation by Michael Resch, University of Stuttgart Spring 2010 Flynn s Taxonomy SISD:
Parallel Processors. The dream of computer architects since 1950s: replicate processors to add performance vs. design a faster processor
Multiprocessing Parallel Computers Definition: A parallel computer is a collection of processing elements that cooperate and communicate to solve large problems fast. Almasi and Gottlieb, Highly Parallel
COSC 6385 Computer Architecture - Multi Processor Systems
COSC 6385 Computer Architecture - Multi Processor Systems Fall 2006 Classification of Parallel Architectures Flynn s Taxonomy SISD: Single instruction single data Classical von Neumann architecture SIMD:
Parallel Architectures
Parallel Architectures CPS343 Parallel and High Performance Computing Spring 2018 CPS343 (Parallel and HPC) Parallel Architectures Spring 2018 1 / 36 Outline 1 Parallel Computer Classification Flynn s
Scalability and Classifications
Scalability and Classifications 1 Types of Parallel Computers MIMD and SIMD classifications shared and distributed memory multicomputers distributed shared memory computers 2 Network Topologies static
Introduction to parallel computing
Introduction to parallel computing 2. Parallel Hardware Zhiao Shi (modifications by Will French) Advanced Computing Center for Education & Research Vanderbilt University Motherboard Processor https://sites.google.com/
Computing architectures Part 2 TMA4280 Introduction to Supercomputing
Computing architectures Part 2 TMA4280 Introduction to Supercomputing NTNU, IMF January 16. 2017 1 Supercomputing What is the motivation for Supercomputing? Solve complex problems fast and accurately:
Types of Parallel Computers
slides1-22 Two principal types: Types of Parallel Computers Shared memory multiprocessor Distributed memory multicomputer slides1-23 Shared Memory Multiprocessor Conventional Computer slides1-24 Consists
Chapter 11. Introduction to Multiprocessors
Chapter 11 Introduction to Multiprocessors 11.1 Introduction A multiple processor system consists of two or more processors that are connected in a manner that allows them to share the simultaneous (parallel)
Parallel Architecture. Sathish Vadhiyar
Parallel Architecture Sathish Vadhiyar Motivations of Parallel Computing Faster execution times From days or months to hours or seconds E.g., climate modelling, bioinformatics Large amount of data dictate
CS4230 Parallel Programming. Lecture 3: Introduction to Parallel Architectures 8/28/12. Homework 1: Parallel Programming Basics
CS4230 Parallel Programming Lecture 3: Introduction to Parallel Architectures Mary Hall August 28, 2012 Homework 1: Parallel Programming Basics Due before class, Thursday, August 30 Turn in electronically
Parallel Systems Prof. James L. Frankel Harvard University. Version of 6:50 PM 4-Dec-2018 Copyright 2018, 2017 James L. Frankel. All rights reserved.
Parallel Systems Prof. James L. Frankel Harvard University Version of 6:50 PM 4-Dec-2018 Copyright 2018, 2017 James L. Frankel. All rights reserved. Architectures SISD (Single Instruction, Single Data)
Advanced Parallel Architecture. Annalisa Massini /2017
Advanced Parallel Architecture Annalisa Massini - 2016/2017 References Advanced Computer Architecture and Parallel Processing H. El-Rewini, M. Abd-El-Barr, John Wiley and Sons, 2005 Parallel computing
CS 590: High Performance Computing. Parallel Computer Architectures. Lab 1 Starts Today. Already posted on Canvas (under Assignment) Let s look at it
Lab 1 Starts Today Already posted on Canvas (under Assignment) Let s look at it CS 590: High Performance Computing Parallel Computer Architectures Fengguang Song Department of Computer Science IUPUI 1
BlueGene/L (No. 4 in the Latest Top500 List)
BlueGene/L (No. 4 in the Latest Top500 List) first supercomputer in the Blue Gene project architecture. Individual PowerPC 440 processors at 700Mhz Two processors reside in a single chip. Two chips reside
Multiple Issue and Static Scheduling. Multiple Issue. MSc Informatics Eng. Beyond Instruction-Level Parallelism
Computing Systems & Performance Beyond Instruction-Level Parallelism MSc Informatics Eng. 2012/13 A.J.Proença From ILP to Multithreading and Shared Cache (most slides are borrowed) When exploiting ILP,
Outline. Distributed Shared Memory. Shared Memory. ECE574 Cluster Computing. Dichotomy of Parallel Computing Platforms (Continued)
Cluster Computing Dichotomy of Parallel Computing Platforms (Continued) Lecturer: Dr Yifeng Zhu Class Review Interconnections Crossbar» Example: myrinet Multistage» Example: Omega network Outline Flynn
PARALLEL COMPUTER ARCHITECTURES
8 ARALLEL COMUTER ARCHITECTURES 1 CU Shared memory (a) (b) Figure 8-1. (a) A multiprocessor with 16 CUs sharing a common memory. (b) An image partitioned into 16 sections, each being analyzed by a different
Introduction II. Overview
Introduction II Overview Today we will introduce multicore hardware (we will introduce many-core hardware prior to learning OpenCL) We will also consider the relationship between computer hardware and
Parallel Numerics, WT 2013/ Introduction
Parallel Numerics, WT 2013/2014 1 Introduction page 1 of 122 Scope Revise standard numerical methods considering parallel computations! Required knowledge Numerics Parallel Programming Graphs Literature
Objectives of the Course
Objectives of the Course Parallel Systems: Understanding the current state-of-the-art in parallel programming technology Getting familiar with existing algorithms for number of application areas Distributed
CS Parallel Algorithms in Scientific Computing
CS 775 - arallel Algorithms in Scientific Computing arallel Architectures January 2, 2004 Lecture 2 References arallel Computer Architecture: A Hardware / Software Approach Culler, Singh, Gupta, Morgan
What is Parallel Computing?
What is Parallel Computing? Parallel Computing is several processing elements working simultaneously to solve a problem faster. 1/33 What is Parallel Computing? Parallel Computing is several processing
FLYNN S TAXONOMY OF COMPUTER ARCHITECTURE
FLYNN S TAXONOMY OF COMPUTER ARCHITECTURE The most popular taxonomy of computer architecture was defined by Flynn in 1966. Flynn s classification scheme is based on the notion of a stream of information.
Lecture 7: Parallel Processing
Lecture 7: Parallel Processing Introduction and motivation Architecture classification Performance evaluation Interconnection network Zebo Peng, IDA, LiTH 1 Performance Improvement Reduction of instruction
Taxonomy of Parallel Computers, Models for Parallel Computers. Levels of Parallelism
Taxonomy of Parallel Computers, Models for Parallel Computers Reference : C. Xavier and S. S. Iyengar, Introduction to Parallel Algorithms 1 Levels of Parallelism Parallelism can be achieved at different
Parallel Computing Introduction
Parallel Computing Introduction Bedřich Beneš, Ph.D. Associate Professor Department of Computer Graphics Purdue University von Neumann computer architecture CPU Hard disk Network Bus Memory GPU I/O devices
Computer and Information Sciences College / Computer Science Department CS 207 D. Computer Architecture. Lecture 9: Multiprocessors
Computer and Information Sciences College / Computer Science Department CS 207 D Computer Architecture Lecture 9: Multiprocessors Challenges of Parallel Processing First challenge is % of program inherently
CS 770G - Parallel Algorithms in Scientific Computing Parallel Architectures. May 7, 2001 Lecture 2
CS 770G - arallel Algorithms in Scientific Computing arallel Architectures May 7, 2001 Lecture 2 References arallel Computer Architecture: A Hardware / Software Approach Culler, Singh, Gupta, Morgan Kaufmann
Introduction to Parallel Processing
Babylon University College of Information Technology Software Department Introduction to Parallel Processing By Single processor supercomputers have achieved great speeds and have been pushing hardware
Module 5 Introduction to Parallel Processing Systems
Module 5 Introduction to Parallel Processing Systems 1. What is the difference between pipelining and parallelism? In general, parallelism is simply multiple operations being done at the same time.this
Overview. Processor organizations Types of parallel machines. Real machines
Course Outline Introduction in algorithms and applications Parallel machines and architectures Overview of parallel machines, trends in top-500, clusters, DAS Programming methods, languages, and environments
Serial. Parallel. CIT 668: System Architecture 2/14/2011. Topics. Serial and Parallel Computation. Parallel Computing
CIT 668: System Architecture Parallel Computing Topics 1. What is Parallel Computing? 2. Why use Parallel Computing? 3. Types of Parallelism 4. Amdahl s Law 5. Flynn s Taxonomy of Parallel Computers 6.
Lecture 2 Parallel Programming Platforms
Lecture 2 Parallel Programming Platforms Flynn s Taxonomy In 1966, Michael Flynn classified systems according to numbers of instruction streams and the number of data stream. Data stream Single Multiple
Multiprocessors - Flynn s Taxonomy (1966)
Multiprocessors - Flynn s Taxonomy (1966) Single Instruction stream, Single Data stream (SISD) Conventional uniprocessor Although ILP is exploited Single Program Counter -> Single Instruction stream The
Tools and techniques for optimization and debugging. Fabio Affinito October 2015
Tools and techniques for optimization and debugging Fabio Affinito October 2015 Fundamentals of computer architecture Serial architectures Introducing the CPU It s a complex, modular object, made of different
Introduction to Parallel Programming
Introduction to Parallel Programming January 14, 2015 www.cac.cornell.edu What is Parallel Programming? Theoretically a very simple concept Use more than one processor to complete a task Operationally
Dr. Joe Zhang PDC-3: Parallel Platforms
CSC630/CSC730: arallel & Distributed Computing arallel Computing latforms Chapter 2 (2.3) 1 Content Communication models of Logical organization (a programmer s view) Control structure Communication model
Motivation for Parallelism. Motivation for Parallelism. ILP Example: Loop Unrolling. Types of Parallelism
Motivation for Parallelism Motivation for Parallelism The speed of an application is determined by more than just processor speed. speed Disk speed Network speed... Multiprocessors typically improve the
Parallel Computing Platforms
Parallel Computing Platforms Jinkyu Jeong (jinkyu@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu SSE3054: Multicore Systems, Spring 2017, Jinkyu Jeong (jinkyu@skku.edu)
Course II Parallel Computer Architecture. Week 2-3 by Dr. Putu Harry Gunawan
Course II Parallel Computer Architecture Week 2-3 by Dr. Putu Harry Gunawan www.phg-simulation-laboratory.com Review Review Review Review Review Review Review Review Review Review Review Review Processor
Computer parallelism Flynn s categories
04 Multi-processors 04.01-04.02 Taxonomy and communication Parallelism Taxonomy Communication alessandro bogliolo isti information science and technology institute 1/9 Computer parallelism Flynn s categories
Lecture 7: Parallel Processing
Lecture 7: Parallel Processing Introduction and motivation Architecture classification Performance evaluation Interconnection network Zebo Peng, IDA, LiTH 1 Performance Improvement Reduction of instruction
represent parallel computers, so distributed systems such as Does not consider storage or I/O issues
Top500 Supercomputer list represent parallel computers, so distributed systems such as SETI@Home are not considered Does not consider storage or I/O issues Both custom designed machines and commodity machines
WHY PARALLEL PROCESSING? (CE-401)
PARALLEL PROCESSING (CE-401) COURSE INFORMATION 2 + 1 credits (60 marks theory, 40 marks lab) Labs introduced for second time in PP history of SSUET Theory marks breakup: Midterm Exam: 15 marks Assignment:
Parallel Computer Architectures. Lectured by: Phạm Trần Vũ Prepared by: Thoại Nam
Parallel Computer Architectures Lectured by: Phạm Trần Vũ Prepared by: Thoại Nam Outline Flynn s Taxonomy Classification of Parallel Computers Based on Architectures Flynn s Taxonomy Based on notions of
High Performance Computing. Leopold Grinberg T. J. Watson IBM Research Center, USA
High Performance Computing Leopold Grinberg T. J. Watson IBM Research Center, USA High Performance Computing Why do we need HPC? High Performance Computing Amazon can ship products within hours would it
Lecture 25: Interrupt Handling and Multi-Data Processing. Spring 2018 Jason Tang
Lecture 25: Interrupt Handling and Multi-Data Processing Spring 2018 Jason Tang 1 Topics Interrupt handling Vector processing Multi-data processing 2 I/O Communication Software needs to know when: I/O
A taxonomy of computer architectures
A taxonomy of computer architectures 53 We have considered different types of architectures, and it is worth considering some way to classify them. Indeed, there exists a famous taxonomy of the various
CS4961 Parallel Programming. Lecture 3: Introduction to Parallel Architectures 8/30/11. Administrative UPDATE. Mary Hall August 30, 2011
CS4961 Parallel Programming Lecture 3: Introduction to Parallel Architectures Administrative UPDATE Nikhil office hours: - Monday, 2-3 PM, MEB 3115 Desk #12 - Lab hours on Tuesday afternoons during programming
ARCHITECTURAL CLASSIFICATION. Mariam A. Salih
ARCHITECTURAL CLASSIFICATION Mariam A. Salih Basic types of architectural classification FLYNN S TAXONOMY OF COMPUTER ARCHITECTURE FENG S CLASSIFICATION Handler Classification Other types of architectural
COSC 6385 Computer Architecture - Thread Level Parallelism (I)
COSC 6385 Computer Architecture - Thread Level Parallelism (I) Edgar Gabriel Spring 2014 Long-term trend on the number of transistor per integrated circuit Number of transistors double every ~18 month
Tools and techniques for optimization and debugging. Andrew Emerson, Fabio Affinito November 2017
Tools and techniques for optimization and debugging Andrew Emerson, Fabio Affinito November 2017 Fundamentals of computer architecture Serial architectures Introducing the CPU It s a complex, modular object,
CSE Introduction to Parallel Processing. Chapter 4. Models of Parallel Processing
Dr Izadi CSE-4533 Introduction to Parallel Processing Chapter 4 Models of Parallel Processing Elaborate on the taxonomy of parallel processing from chapter Introduce abstract models of shared and distributed
Unit 9 : Fundamentals of Parallel Processing
Unit 9 : Fundamentals of Parallel Processing Lesson 1 : Types of Parallel Processing 1.1. Learning Objectives On completion of this lesson you will be able to : classify different types of parallel processing
BİL 542 Parallel Computing
BİL 542 Parallel Computing 1 Chapter 1 Parallel Programming 2 Why Use Parallel Computing? Main Reasons: Save time and/or money: In theory, throwing more resources at a task will shorten its time to completion,
Multicores, Multiprocessors, and Clusters
1 / 12 Multicores, Multiprocessors, and Clusters P. A. Wilsey Univ of Cincinnati 2 / 12 Classification of Parallelism Classification from Textbook Software Sequential Concurrent Serial Some problem written
MULTIPROCESSORS AND THREAD-LEVEL. B649 Parallel Architectures and Programming
MULTIPROCESSORS AND THREAD-LEVEL PARALLELISM B649 Parallel Architectures and Programming Motivation behind Multiprocessors Limitations of ILP (as already discussed) Growing interest in servers and server-performance
SMD149 - Operating Systems - Multiprocessing
SMD149 - Operating Systems - Multiprocessing Roland Parviainen December 1, 2005 1 / 55 Overview Introduction Multiprocessor systems Multiprocessor, operating system and memory organizations 2 / 55 Introduction
Overview. SMD149 - Operating Systems - Multiprocessing. Multiprocessing architecture. Introduction SISD. Flynn s taxonomy
Overview SMD149 - Operating Systems - Multiprocessing Roland Parviainen Multiprocessor systems Multiprocessor, operating system and memory organizations December 1, 2005 1/55 2/55 Multiprocessor system
MULTIPROCESSORS AND THREAD-LEVEL PARALLELISM. B649 Parallel Architectures and Programming
MULTIPROCESSORS AND THREAD-LEVEL PARALLELISM B649 Parallel Architectures and Programming Motivation behind Multiprocessors Limitations of ILP (as already discussed) Growing interest in servers and server-performance
10 Parallel Organizations: Multiprocessor / Multicore / Multicomputer Systems
1 License: http://creativecommons.org/licenses/by-nc-nd/3.0/ 10 Parallel Organizations: Multiprocessor / Multicore / Multicomputer Systems To enhance system performance and, in some cases, to increase
Parallel Computing Platforms. Jinkyu Jeong Computer Systems Laboratory Sungkyunkwan University
Parallel Computing Platforms Jinkyu Jeong (jinkyu@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu Elements of a Parallel Computer Hardware Multiple processors Multiple
Transputers. The Lost Architecture. Bryan T. Meyers. December 8, Bryan T. Meyers Transputers December 8, / 27
Transputers The Lost Architecture Bryan T. Meyers December 8, 2014 Bryan T. Meyers Transputers December 8, 2014 1 / 27 Table of Contents 1 What is a Transputer? History Architecture 2 Examples and Uses
Normal computer 1 CPU & 1 memory The problem of Von Neumann Bottleneck: Slow processing because the CPU faster than memory
Parallel Machine 1 CPU Usage Normal computer 1 CPU & 1 memory The problem of Von Neumann Bottleneck: Slow processing because the CPU faster than memory Solution Use multiple CPUs or multiple ALUs For simultaneous
COMP4300/8300: Overview of Parallel Hardware. Alistair Rendell. COMP4300/8300 Lecture 2-1 Copyright c 2015 The Australian National University
COMP4300/8300: Overview of Parallel Hardware Alistair Rendell COMP4300/8300 Lecture 2-1 Copyright c 2015 The Australian National University 2.1 Lecture Outline Review of Single Processor Design So we talk
Processor Architecture and Interconnect
Processor Architecture and Interconnect What is Parallelism? Parallel processing is a term used to denote simultaneous computation in CPU for the purpose of measuring its computation speeds. Parallel Processing
COMP4300/8300: Overview of Parallel Hardware. Alistair Rendell
COMP4300/8300: Overview of Parallel Hardware Alistair Rendell COMP4300/8300 Lecture 2-1 Copyright c 2015 The Australian National University 2.2 The Performs: Floating point operations (FLOPS) - add, mult,
Chap. 2 part 1. CIS*3090 Fall Fall 2016 CIS*3090 Parallel Programming 1
Chap. 2 part 1 CIS*3090 Fall 2016 Fall 2016 CIS*3090 Parallel Programming 1 Provocative question (p30) How much do we need to know about the HW to write good par. prog.? Chap. gives HW background knowledge
Online Course Evaluation. What we will do in the last week?
Online Course Evaluation Please fill in the online form The link will expire on April 30 (next Monday) So far 10 students have filled in the online form Thank you if you completed it. 1 What we will do
SHARED MEMORY VS DISTRIBUTED MEMORY
OVERVIEW Important Processor Organizations 3 SHARED MEMORY VS DISTRIBUTED MEMORY Classical parallel algorithms were discussed using the shared memory paradigm. In shared memory parallel platform processors
Embedded Systems Architecture. Computer Architectures
Embedded Systems Architecture Computer Architectures M. Eng. Mariusz Rudnicki 1/18 A taxonomy of computer architectures There are many different types of architectures, and it is worth considering some
Interconnection Network
Interconnection Network Jinkyu Jeong (jinkyu@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu SSE3054: Multicore Systems, Spring 2017, Jinkyu Jeong (jinkyu@skku.edu) Topics
Computer Organization. Chapter 16
William Stallings Computer Organization and Architecture t Chapter 16 Parallel Processing Multiple Processor Organization Single instruction, single data stream - SISD Single instruction, multiple data
Chapter 2 Parallel Computer Architecture
Chapter 2 Parallel Computer Architecture The possibility for a parallel execution of computations strongly depends on the architecture of the execution platform. This chapter gives an overview of the general
Introduction to Parallel Programming
Introduction to Parallel Programming David Lifka lifka@cac.cornell.edu May 23, 2011 5/23/2011 www.cac.cornell.edu 1 y What is Parallel Programming? Using more than one processor or computer to complete
Parallel and High Performance Computing CSE 745
Parallel and High Performance Computing CSE 745 1 Outline Introduction to HPC computing Overview Parallel Computer Memory Architectures Parallel Programming Models Designing Parallel Programs Parallel
Let s say I give you a homework assignment today with 100 problems. Each problem takes 2 hours to solve. The homework is due tomorrow.
Let s say I give you a homework assignment today with 100 problems. Each problem takes 2 hours to solve. The homework is due tomorrow. Big problems and Very Big problems in Science How do we live Protein
Lecture 9: MIMD Architectures
Lecture 9: MIMD Architectures Introduction and classification Symmetric multiprocessors NUMA architecture Clusters Zebo Peng, IDA, LiTH 1 Introduction A set of general purpose processors is connected together.
BlueGene/L. Computer Science, University of Warwick. Source: IBM
BlueGene/L Source: IBM 1 BlueGene/L networking BlueGene system employs various network types. Central is the torus interconnection network: 3D torus with wrap-around. Each node connects to six neighbours
CDA3101 Recitation Section 13
CDA3101 Recitation Section 13 Storage + Bus + Multicore and some exam tips Hard Disks Traditional disk performance is limited by the moving parts. Some disk terms Disk Performance Platters - the surfaces
Top500 Supercomputer list
Top500 Supercomputer list Tends to represent parallel computers, so distributed systems such as SETI@Home are neglected. Does not consider storage or I/O issues Both custom designed machines and commodity
Architectures of Flynn s taxonomy -- A Comparison of Methods
Architectures of Flynn s taxonomy -- A Comparison of Methods Neha K. Shinde Student, Department of Electronic Engineering, J D College of Engineering and Management, RTM Nagpur University, Maharashtra,
Lecture 9: MIMD Architectures
Lecture 9: MIMD Architectures Introduction and classification Symmetric multiprocessors NUMA architecture Clusters Zebo Peng, IDA, LiTH 1 Introduction MIMD: a set of general purpose processors is connected
Computer Systems Architecture
Computer Systems Architecture Lecture 23 Mahadevan Gomathisankaran April 27, 2010 04/27/2010 Lecture 23 CSCE 4610/5610 1 Reminder ABET Feedback: http://www.cse.unt.edu/exitsurvey.cgi?csce+4610+001 Student
Moore s Law. Computer architect goal Software developer assumption
Moore s Law The number of transistors that can be placed inexpensively on an integrated circuit will double approximately every 18 months. Self-fulfilling prophecy Computer architect goal Software developer
CS252 Graduate Computer Architecture Lecture 14. Multiprocessor Networks March 9 th, 2011
CS252 Graduate Computer Architecture Lecture 14 Multiprocessor Networks March 9 th, 2011 John Kubiatowicz Electrical Engineering and Computer Sciences University of California, Berkeley http://www.eecs.berkeley.edu/~kubitron/cs252
Programmation Concurrente (SE205)
Programmation Concurrente (SE205) CM1 - Introduction to Parallelism Florian Brandner & Laurent Pautet LTCI, Télécom ParisTech, Université Paris-Saclay x Outline Course Outline CM1: Introduction Forms of
Multiprocessors. Flynn Taxonomy. Classifying Multiprocessors. why would you want a multiprocessor? more is better? Cache Cache Cache.
Multiprocessors why would you want a multiprocessor? Multiprocessors and Multithreading more is better? Cache Cache Cache Classifying Multiprocessors Flynn Taxonomy Flynn Taxonomy Interconnection Network
Computer Systems Architecture
Computer Systems Architecture Lecture 24 Mahadevan Gomathisankaran April 29, 2010 04/29/2010 Lecture 24 CSCE 4610/5610 1 Reminder ABET Feedback: http://www.cse.unt.edu/exitsurvey.cgi?csce+4610+001 Student
2. Parallel Architectures
2. Parallel Architectures 2.1 Objectives To introduce the principles and classification of parallel architectures. To discuss various forms of parallel processing. To explore the characteristics of parallel
Chapter 1 Computer System Overview
Operating Systems: Internals and Design Principles Chapter 1 Computer System Overview Ninth Edition By William Stallings Operating System Exploits the hardware resources of one or more processors Provides
Multiprocessors and Thread-Level Parallelism. Department of Electrical & Electronics Engineering, Amrita School of Engineering
Multiprocessors and Thread-Level Parallelism Multithreading Increasing performance by ILP has the great advantage that it is reasonable transparent to the programmer, ILP can be quite limited or hard to
Message Passing Interface (MPI)
CS 220: Introduction to Parallel Computing Message Passing Interface (MPI) Lecture 13 Today s Schedule Parallel Computing Background Diving in: MPI The Jetson cluster 3/7/18 CS 220: Parallel Computing
Parallel Numerics, WT 2017/ Introduction. page 1 of 127
Parallel Numerics, WT 2017/2018 1 Introduction page 1 of 127 Scope Revise standard numerical methods considering parallel computations! Change method or implementation! page 2 of 127 Scope Revise standard
Approaches to Parallel Computing
Approaches to Parallel Computing K. Cooper 1 1 Department of Mathematics Washington State University 2019 Paradigms Concept Many hands make light work... Set several processors to work on separate aspects
5 Computer Organization
5 Computer Organization 5.1 Foundations of Computer Science ã Cengage Learning Objectives After studying this chapter, the student should be able to: q List the three subsystems of a computer. q Describe
RISC Processors and Parallel Processing. Section and 3.3.6
RISC Processors and Parallel Processing Section 3.3.5 and 3.3.6 The Control Unit When a program is being executed it is actually the CPU receiving and executing a sequence of machine code instructions.