Computer parallelism Flynn s categories

Transcription

1 04 Multi-processors Taxonomy and communication Parallelism Taxonomy Communication alessandro bogliolo isti information science and technology institute 1/9 Computer parallelism Flynn s categories Data stream Single Data (SD) stream Multiple Data (MD) stream Instruction stream Single Instruction (SI) streams SISD SIMD Multiple Instruction (MI) streams MISD MIMD MultiProcessors are MIMD computers Exploit thread-level parallelism Offer great flexibility Can be made of off-the-shelf processors MultiProcessor architectures: Centralized shared memory Distributed memory alessandro bogliolo isti information science and technology institute 2/9 1

2 Centralized shared-memory All processors share a single centralized memory Centralized shared memory computers are also called: Symmetric MultiProcessors (SMPs) Uniform Memory Access (UMA) computers alessandro bogliolo isti information science and technology institute 3/9 Centralized shared-memory The shared memory is used for communication Processors communicate by means of load and write operations performed on the same locations in memory The shared memory (and bus) is a bottleneck that limits system scalability Simultaneous accesses to the shared resources give rise to conflicts that need arbitration and may end-up stalling the processors The architecture is suitable only for a few processors Overcoming the bottlneck More complex communication systems (multiple buses, switches, crossbars,...) Large caches alessandro bogliolo isti information science and technology institute 4/9 2

3 Distributed memory An interconnection network connects multiple nodes Each node contains at least a processor with local memory Each node may be a small SMP with some I/O alessandro bogliolo isti information science and technology institute 5/9 Distributed memory Distributed-memory systems are further classified based on their address space Shared address space All processors share the same address space, that is mapped on distributed physical memories Distributed-memory mutliprocessors with shared address space are called Distributed shared memory (DSM) multiprocessor Non-uniform memory access (NUMA) multiprocessor Private address space Each processor has a private address space mapped on its local memory that cannot be accessed by remote processors Multiprocessors with multiple private address spaces are called Multicomputers since each node is a separate computer connected to a cluster alessandro bogliolo isti information science and technology institute 6/9 3

4 Communication method (1) Shared Memory Distributed Address space Shared Private SMP / UMA DSM / NUMA Multicomputers Shared-memory communication Processors communicate by sharing data structures Message-passing communication Processors communicate by sending messages according to a given network protocol alessandro bogliolo isti information science and technology institute 7/9 Communication method (2) Shared-memory communication Is compatible with SMPs Is easy to program Has reduced overhead (due to the implicit nature of communication and to the HW implementation of protection) Benefits from HW-controlled caching Suffers from cache coherence issues Message-passing communication Is easy to understand (due to its explicit nature) Is easy to synchronize Makes it easier to implement sender-initiated communication Has a simpler HW alessandro bogliolo isti information science and technology institute 8/9 4

5 Communication performance Bisection bandwidth Sum of the bandwidth of the lines that cross an ideal cut that partitions the network into two equal parts Depends on the network topology and on the nature of the physical links I/O bandwidth at each node Depends on the communication mechanism, on the network and on the local resources Latency Depends on the HW/SW overheads at sender and receiver nodes, and on the time of flight Latency hiding Depends on the application alessandro bogliolo isti information science and technology institute 9/9 5