Introduction to parallel computing

Transcription

1 Introduction to parallel computing 2. Parallel Hardware Zhiao Shi (modifications by Will French) Advanced Computing Center for Education & Research Vanderbilt University

2 Motherboard Processor site/fall12itessentialsblanco/ processor-facts motherboard1_large.gif 2

3 Multi-core Processors and Hyperthreading Computer Motherboard Socket A Processor A» Physical core 1» Logical core 1» Logical core 2» Physical core 2» Logical core 3» Logical core 4» Physical core 3». A processor is placed into a socket that is built into the motherboard. Some motherboards contain multiple sockets. Starting in the early 2000s, Intel introduced the notion of hyperthreading, where the OS treats a physical core (CPU) as multiple logical cores, providing performance benefits in some cases. However, a logical core does not provide the performance benefit of an actual physical core. 3

4 The von Neumann Architecture (CPUs)

5 Central processing unit (CPU) Divided into two parts: Control unit (CU) decides which instruction in a program should be executed, fetches data from main memory, moves data to and from the ALU, etc. (the boss) Arithmetic logic unit (ALU) - responsible for executing the actual instructions, operates on data, logical comparisons, etc. (the worker) 5

6 Computer Memory This is a collection of locations, each of which is capable of storing both instructions and data. Every location consists of an address, which is used to access the location, and the contents of the location. 6

7 Memory Hierarchy Hard Drive (e.g. HDD or SSD) slower larger Main Memory (RAM) L2 Cache L1 Cache Registers faster smaller 7

8 memory fetch/read CPU 8

9 memory write/store CPU 9

10 von Neumann bottleneck Read/write operations cannot occur at once because they share a common bus (interconnect/wire) 10

11 A programmer can write code to exploit PARALLEL HARDWARE 11

12 Flynn s taxonomy SISD Single instruction stream Single data stream (SIMD) Single instruction stream Multiple data stream MISD Multiple instruction stream Single data stream (MIMD) Multiple instruction stream Multiple data stream 12

13 SIMD Parallelism achieved by dividing data among the processors. Applies the same instruction to multiple data items. Called data parallelism. 13

14 SIMD example control unit n data items n ALUs x[1] x[2] x[n] ALU 1 ALU 2 ALU n for (i = 0; i < n; i++) x[i] += y[i]; 14

15 SIMD What if we don t have as many ALUs as data items? Divide the work and process iteratively. Ex. m = 4 ALUs and n = 15 data items. Round ALU 1 ALU 2 ALU 3 ALU 4 1 X[0] X[1] X[2] X[3] 2 X[4] X[5] X[6] X[7] 3 X[8] X[9] X[10] X[11] 4 X[12] X[13] X[14] 15

16 SIMD drawbacks All ALUs are required to execute the same instruction, or remain idle. In classic design, they must also operate synchronously. The ALUs have no instruction storage. Efficient for large data parallel problems, but not other types of more complex parallel problems. 16

17 MIMD Supports multiple simultaneous instruction streams operating on multiple data streams. Typically consist of a collection of fully independent processing units or cores, each of which has its own control unit and its own ALU. 17

18 Communication model of parallel platforms Two primary forms of data exchange between parallel tasks - accessing a shared data space or exchanging messages. Platforms that provide a shared data space are called: Shared-address-space machines Multiprocessors Platforms that support messaging are called: Message passing platforms Distributed memory system Multicomputers 18

19 Shared memory systems A collection of autonomous processors is connected to a memory system via an interconnection network. Each processor can access each memory location. The processors usually communicate implicitly by accessing shared data structures. Single processor model. 19

20 Shared memory systems Most widely available shared memory systems use one or more multicore processors. (multiple CPUs or on a single chip) 20

21 Distributed memory systems Comprise of a set of processors and their own (exclusive) memory. These platforms are programmed using (variants of) send and receive primitives. Libraries such as MPI provide such primitives. We will investigate MPI in this class Multi-processor model. 21

22 Distributed memory systems Clusters (most popular) A collection of commodity systems. Connected by a commodity interconnection network. Nodes of a cluster are individual computations units joined by a communication network. a.k.a. hybrid systems 22

23 Distributed memory systems 23

24 Interconnection networks Affects performance of both distributed and shared memory systems. Two categories: Shared memory interconnects Distributed memory interconnects 24

25 Shared memory interconnects Bus interconnect A collection of parallel communication wires together with some hardware that controls access to the bus. Communication wires are shared by the devices that are connected to it. As the number of devices connected to the bus increases, contention for use of the bus increases, and performance decreases. 25

26 Distributed memory interconnects Two groups Direct interconnect Each switch is directly connected to a processor memory pair, and the switches are connected to each other. Indirect interconnect Switches may not be directly connected to a processor. 26

27 More definitions Any time data is transmitted, we re interested in how long it will take for the data to reach its destination. Latency The time that elapses between the source s beginning to transmit the data and the destination s starting to receive the first byte. Bandwidth The rate at which the destination receives data after it has started to receive the first byte. 27

28 Message transmission time = l + n / b latency (seconds) length of message (bytes) bandwidth (bytes per second) 28