Why Multiprocessors?

Transcription

1 Why Multiprocessors? Motivation: Go beyond the performance offered by a single processor Without requiring specialized processors Without the complexity of too much multiple issue Opportunity: Software available Parallel programs Multi-programmed machines

2 Multiprocessors: The SIMD Model SISD: Single Instruction stream, Single Data stream Uniprocessor This is the view at the ISA level Tomasulo uncovers data stream parallelism SIMD: Single Instruction stream, Multiple Data streams ISA makes data parallelism explicit Special SIMD instructions Same instruction goes to multiple functional units, but acts on different data

3 SIMD Drawbacks SIMD useful for loop-level parallelism Model is too inflexible to accommodate parallel programs as well as multiprogrammed environments Cannot take advantage of uniprocessor performance growth SIMD architecture usually used in special purpose designs Signal or image processing

4 Multiprocessors: The MIMD Model MIMD: Multiple Instruction streams, Multiple Data streams Each processor fetches its own instruction and data Advantages: Flexibility: parallel programs, or multiprogrammed OS, or both Built using off-the-shelf uniprocessors

5 MIMD: The Centralized Shared- Memory Model P P P $ $ $ Main Memory I/O Bus Single bus connects a shared memory to all processors Also called Uniform Memory Access (UMA) machine Disadvantage: cannot scale very well, especially with fast processors (more memory bandwidth required)

6 MIMD: Physically Distributed Memory P+$ P+$ Independent memory for each processor M I/O M I/O High-bandwidth interconnection M Interconnection n/w I/O M P+$ P+$ I/O Adv: cost-effective memory bandwidth scaling Adv: lesser latency for local access Disadv: communication of data between nodes

7 Communication Models with Physically Distributed Memory Distributed Shared Memory (DSM) Memory address space is the same across nodes Also called scalable shared memory Also called NUMA: non-uniform memory access Communication is implicit via load/store Multicomputer, or Message Passing Machine Separate private address spaces for each node Communication is explicit, through messages Synchronous, or asynchronous Std. Message Passing Interface (MPI) possible

8 Multiprocessing: Classification Multiprocessing SIMD MIMD Centralized shared memory Physically distributed memory Distributed shared memory (DSM) Message passing machines

9 Multiprocessing: Classification Multiprocessing SIMD MIMD Centralized shared memory Physically distributed memory Distributed shared memory (DSM) Message passing machines

10 DSM vs. Message Passing Shared Memory Well understood mechanisms for programming Program independent of communication pattern Low overhead for communicating small items Message Passing Hardware simplicity Communication is explicit forces programmer to pay attention to what is expensive Hardware controlled caching

11 Achieving the Desired Communication Model Message Passing on top of Shared Memory Considerable easier Difficulty arises in dealing with arbitrary message lengths Shared Memory on top of Message Passing Harder since every load/store has to be faked Every memory reference may involve OS One promising direction: use of VM to share objects at page level: shared VM

12 Challenges in Parallel Processing Limited parallelism available in programs 90% parallelizable ==> max speed possible? Exception: super-linear speedup Increased memory/cache available Usually not very great however Large latency of communication clock cycles 0.5% instructions access remote memory ==> what is the increase in CPI?

13 Addressing the Challenges Limited parallelism Tackled mainly by redesigning the algorithm or software Avoiding large latency Hardware mechanism: caching Software mechanism: restructure to make more accesses local

14 Some Example Applications Two classes Parallel programs or program kernels Multi-programmed OS Spatial and temporal data access patterns are important Computation to communication ratio is important

15 Parallel Application Kernels The FFT kernel Used in spectral methods Data represented as array Computation involves 1D FFT on each row Transpose 1D FFT on each row again Each processor gets a few rows of data Main communication step is the transpose (all to all communication)

16 Parallel Application Kernels (continued) The LU kernel LU factorization of a matrix Blocking is used Computation (dense matrix multiply) is performed by processor which owns the destination block Communication happens at regular intervals

17 Parallel Applications Barnes application N-body problem Octree representation Each processor is allocated a subtree Tree expansion as required (communication in this process)

18 Parallel Applications (continued) Ocean application Influence of eddy and boundary currents on ocean flows Involves solving PDEs Ocean divided into hierarchy of grids (finer grid for more accuracy) Each processor gets a set of grids Communication to exchange boundary conditions, at each step of the process

19 Computation to Communication Ratios Application Computation scaling Communication scaling FFT nlogn/p n/p Logn LU n/p sqrt(n/p) sqrt(n/p) Barnes nlogn/p logn*sqrt(n/p) sqrt(n/p) Ocean n/p sqrt(n/p) sqrt(n/p) Scaling of computation to communication

20 Multiprogrammed OS workload Workload used here is: Two independent copies of the compilation of the Andrew benchmark Three steps: Compilation: compute intensive Installing object files in a library: I/O intensive Removing the object files: I/O intensive