Overview. Processor organizations Types of parallel machines. Real machines

Transcription

1 Course Outline Introduction in algorithms and applications Parallel machines and architectures Overview of parallel machines, trends in top-500, clusters, DAS Programming methods, languages, and environments Message passing (SR, MPI, Java) + HPF Applications N-body problems, search algorithms Higher-level language: Chapel By Clemens Grelck last 4 lectures

2 Learning Goals Be able to compare parallel computer systems (architectures), including which architectures are most suitable for which applications Be able to analyse the performance of network topologies of parallel systems and to compute the most relevant performance metrics Be able to apply the performance-metrics on parallel algorithms to assess their efficiency Understand the history and rationale behind the different types of parallel machines More details: Advances in Computer Architecture course by Andy Pimentel (UvA)

3 Parallel Machines Parallel Computing Techniques and Applications Using Networked Workstations and Parallel Computers (2/e) Section (part of) 1.4 Barry Wilkinson and Michael Allen Pearson, 2005

4 Overview Processor organizations Types of parallel machines Processor arrays Shared-memory multiprocessors Distributed-memory multicomputers Real machines Cluster computers Blue Gene DAS

5 Processor Organization Network topology is a graph A node is a processor An edge is a communication path Evaluation criteria Diameter (maximum distance) Bisection width (minimum number of edges that should be removed to split the graph into 2 -almost- equal halves) Number of edges per node

6 Key issues in network design Bandwidth: Number of bits transferred per second Latency: Network latency: time to make a message transfer through the network Communication latency: total time to send the message, including software overhead and interface delays Message latency (startup time): time to send a zero-length message Diameter influences latency Bisection width influences bisection bandwidth (collective bandwidth over the ``removed edges)

7 Mesh q-dimensional lattice q=2 -> 2-D grid Number of nodes: k² (k = SQRT(p)) Diameter 2(k - 1) Bisection width k Edges per node 4

8 Binary Tree Number of nodes: 2 k 1 (k = 2 LOG(P)) Diameter: 2 (k -1) Bisection width: 1 Edges per node: 3

9 Hypertree Tree with multiple roots, gives better bisection width 4-ary tree: Number of nodes 2 k ( 2 k+1-1) Diameter 2k Bisection width 2 k+1 Edges per node 6

10 CM-5 Engineering solution: fat tree Tree with more bandwidth at links near the root CM-5

11 Hypercube k-dimensional cube, each node has binary value, nodes that differ in 1 bit are connected Number of nodes Diameter 2 k Bisection width 2 k-1 Edges per node k k

12 Hypercube Label nodes with binary value, connect nodes that differ in 1 coordinate Number of nodes Diameter 2 k Bisection width 2 k-1 Edges per node k k

13 Comparison Mesh Tree Hypercube Diameter o + + Bisection width o - + #edges 4 3 unlimited

14 Types of parallel machines Processor arrays Shared-memory multiprocessors Distributed-memory multicomputers

15 Processor Arrays Instructions operate on scalars or vectors Processor array = front-end + synchronized processing elements

16 Processor Arrays Front-end Sequential machine that executes program Vector operations are broadcast to PEs Processing element Performs operation on its part of the vector Communicates with other PEs through a network

17 Examples of Processor Arrays CM-200, Maspar MP-1, MP-2, ICL DAP (~1970s) Earth Simulator (Japan, 2002, former #1 of top-500) Ideas now applied in GPUs, CPU-extensions (MMX), & Intel Xeon Phi Coprocessor Vector Microarchitecture

18 Shared-Memory Multiprocessors Bus easily gets saturated => add caches to CPUs Central problem: cache coherency Snooping cache: monitor bus, invalidate copy on write Write-through or copy-back Bus-based multiprocessors do not scale

19 Other Multiprocessor Designs (1/2) Switch-based multiprocessors (e.g., crossbar) Expensive (requires many very fast components)

20 Other Multiprocessor Designs (2/2) Non-Uniform Memory Access (NUMA) multiprocessors Memory is distributed Some memory is faster to access than other memory Example: Teras at Sara, former Dutch National Supercomputer (1024-node SGI) Ideas now applied in multi-cores

21 Distributed-Memory Multicomputers Each processor only has a local memory Processors communicate by sending messages over a network Routing of messages: Packet-switched message routing: split message into packets, buffered at intermediate nodes Store-and-forward Circuit-switched message routing: establish path between source and destination

22 Packet-switched Message Routing Messages are forwarded one node at a time Forwarding is done in software Every processor on path from source to destination is involved Latency linear to distance x message length Old examples: Parsytec GCel (T800 transputers), Intel Ipsc

23 Circuit-switched Message Routing Each node has a routing module Circuit set up between source and destination Latency linear to distance + message length Example: Intel ipsc/2

24 Modern routing techniques Circuit switching: needs to reserve all links in the path (cf. old telephone system) Packet switching: high latency, buffering space (cf. postal mail) Cut-through routing: packet switching, but immediately forward (without buffering) packets if outgoing link is available Wormhole routing: transmit head (few bits) of message, rest follows like a worm

25 Network latency Performance Packet switching Wormhole routing Circuit switching Distance (number of hops)

26 Distributed Shared Memory Shared memory is easier to program, but doesn t scale Distributed memory is hard to program, but does scale Distributed Shared Memory (DSM): provide sharedmemory programming model on top of distributed memory hardware Shared Virtual Memory (SVM): use memory management hardware (paging), copy pages over the network Object-based: provide replicated shared objects (Orca language) Was hot research topic in 1990s, but performance remained the bottleneck Revival with Partitioned Global Address Space (PGAS) languages, like X-10 and Chapel

27 Flynn's Taxonomy Single Instruction Multiple Instruction Single Data Multiple Data SISD MISD SIMD MIMD Instruction stream: sequence of instructions Data stream: sequence of data manipulated by instructions

28 Flynn

29 Flynn's Taxonomy Single Instruction Multiple Instruction Single Data Multiple Data SISD MISD SIMD MIMD SISD: Single Instruction Single Data Traditional uniprocessors SIMD: Single Instruction Multiple Data Processor arrays MISD: Multiple Instruction Single Data Nonexistent? Dataflow machines (Maxeler)? MIMD: Multiple Instruction Multiple Data Multiprocessors and multicomputers