Convergence of Parallel Architectures

Transcription

1

2 History Historically, parallel architectures tied to programming models Divergent architectures, with no predictable pattern of growth. Systolic Arrays Dataflow Application Software System Software Architecture Shared Memory SIMD Message Passing Uncertainty of direction paralyzed parallel software development! 2

3 Today Extension of computer architecture to support communication and cooperation OLD: Instruction Set Architecture NEW: Communication Architecture Defines Critical abstractions, boundaries, and primitives (interfaces) Organizational structures that implement interfaces (hw or sw) Compilers, libraries and OS are important bridges today 3

4 CAD Multiprogramming Modern Layered Framework Database Scientific modeling Parallel applications Shared address Compilation or library Message passing Data parallel Programming models Communication abstraction User/system boundary Operating systems support Communication hardware Hardware/software boundary Physical communication medium 4

5 Programming Model What programmer uses in coding applications Specifies communication and synchronization Examples: Multiprogramming: no communication or synch. at program level Shared address space: like bulletin board Message passing: like letters or phone calls, explicit point to point Data parallel: more regimented, global actions on data Implemented with shared address space or message passing 5

6 Communication Abstraction User level communication primitives provided Realizes the programming model Mapping exists between language primitives of programming model and these primitives Supported directly by hw, or via OS, or via user sw Lot of debate about what to support in sw and gap between layers Today: Hw/sw interface tends to be flat, i.e. complexity roughly uniform Compilers and software play important roles as bridges today Technology trends exert strong influence Result is convergence in organizational structure Relatively simple, general purpose communication primitives 6

7 Communication Architecture = User/System Interface + Implementation User/System Interface: Comm. primitives exposed to user-level by hw and system-level sw Implementation: Organizational structures that implement the primitives: hw or OS How optimized are they? How integrated into processing node? Structure of network Goals: Performance Broad applicability Programmability Scalability Low Cost 7

8 Evolution of Architectural Models Historically machines tailored to programming models Prog. model, comm. abstraction, and machine organization lumped together as the architecture Evolution helps understand convergence Identify core concepts Shared Address Space Message Passing Data Parallel Others: Dataflow Systolic Arrays Examine programming model, motivation, intended applications, and contributions to convergence 8

9 Shared Address Space (SAS) Architectures Any processor can directly reference any memory location Communication occurs implicitly as result of loads and stores Convenient: Location transparency Similar programming model to time-sharing on uniprocessors Except processes run on different processors Good throughput on multiprogrammed workloads Naturally provided on wide range of platforms History dates at least to precursors of mainframes in early 60s Wide range of scale: few to hundreds of processors Popularly known as shared memory machines/processors or model Ambiguous: memory may be physically distributed among processors 9

10 SAS Model Process: virtual address space plus one or more threads of control Portions of address spaces of processes are shared P 0 St or e P 1 P 2 Virtual address spaces for a collection of processes communicating via shared addresses Load P n Shared portion of address space Machine physical address space P n pr i vat e Common physical addresses P 2 pr i vat e Private portion of address space P 1 pr i vat e P 0 pr i vat e Writes to shared address visible to other threads (in other processes too) Natural extension of uniprocessors model: conventional memory operations for comm.; special atomic operations for synchronization OS uses shared memory to coordinate processes 10

11 Communication Hardware Also natural extension of uniprocessor Already have processor, one or more memory modules and I/O controllers connected by hardware interconnect of some sort Mem Mem Mem Mem Interconnect Interconnect I/O ctrl I/O ctrl I/O devices Processor Processor Memory capacity increased by adding modules, I/O by controllers Add processors for processing! For higher-throughput multiprogramming, or parallel programs 11

12 History Mainframe approach Motivated by multiprogramming Extends crossbar used for mem bw and I/O Originally processor cost limited to small later, cost of crossbar Bandwidth scales with p High incremental cost; use multistage instead Minicomputer approach Almost all microprocessor systems have bus Motivated by multiprogramming, TP Used heavily for parallel computing Called symmetric multiprocessor (SMP) Latency larger than for uniprocessor Bus is bandwidth bottleneck caching is key: coherence problem Low incremental cost I/O I/O I/O C P P C C I/O C M M M M M $ $ P P M 12

13 Example: Intel Pentium Pro Quad CPU Interrupt controller Bus interface 256-KB L 2 $ P-Pro bus (64-bit data, 36-bit address, 66 MHz) PCI I/O cards PCI bridge PCI bus P-Pro module PCI bridge PCI bus P-Pro module P-Pro module Memory controller MIU 1-, 2-, or 4-way interleaved DRAM All coherence and multiprocessing glue in processor module Highly integrated, targeted at high volume Low latency and bandwidth 13

14 Example: SUN Enterprise System Bus Architecture Jupiter Interconnect SPARC Enterprise M3000, M4000, M5000, M8000, and M9000 servers utilize a system interconnect designed to deliver massive bandwidth and consistent low latency between components. The SPARC Enterprise M5000 server is implemented within a single motherboard but features two logical system boards. Similar to the SPARC Enterprise M4000 server design, each logical system board contains two system controllers that connect to each other, as well as CPU modules, memory access controllers, and an IOU. In addition, each system controller connects to a corresponding system controller on the other logical system board 14

15 SPARC Enterprise M5000 server interconnect diagram * 15

16 Charakteristics of SMP Machines All processors see everything Memory, I/O, interrupts, etc. There is only one kernel The scheduler determines which applications are assigned to which processor Application can migrate between processors They do not typically share caches Although hybrid cache architectures are on the way 16

17 Example SMP Motherboard Quad-CPU AMD Opteron support 17

18 Problems with SMP SMP system can be complex to set up and maintain because of duplicated HW such as processor fans, etc. They tend to be noisy as well due to fans SMP does not scale perfectly Because the memory is shared can develop hot spots where multiple applications must serialize on a single piece of data Process migration can lead to poor cache utilization We need to flush the cache if a process migrates Multiple processors can lead to race conditions We need to provide for multi-processor synchronization 18

19 M M M $ P Net work $ P Dance hall Scaling Up $ M M $ $ Problem is interconnect: cost (crossbar) or bandwidth (bus) Dance-hall: bandwidth still scalable, but lower cost than crossbar latencies to memory uniform, but uniformly large Distributed memory or non-uniform memory access (NUMA) Construct shared address space out of simple message transactions across a general-purpose network (e.g. read-request, read-response) Caching shared (particularly nonlocal) data? P P Net work P M Distributed memory $ P 19

20 Example: Cray T3E External I/O XY P $ Mem ctrl and NI Switch Mem Z Scale up to 1024 processors, 480MB/s links Memory controller generates comm. request for nonlocal references No hardware mechanism for coherence (SGI Origin etc. provide this) 20

21 SMP vs NUMA 21

22 NUMA Opteron s Hammer architecture Because each chip has its own memory resources and does not have to fight for a single unified pool of memory, Opteron has what is known as a "NUMA" (Non-Unified Memory Architecture) scheme. The benefits of Opteron s NUMA design will be most clearly seen in n-way systems. 22

23 Multicore Is Bad News For Supercomputers With no other way to improve the performance of processors further, chip makers have staked their future on putting more and more processor cores on the same chip. Sandia National Laboratories, in New Mexico, Simulated future high-performance computers containing the 8-core, 16''core, and 32-core microprocessors that chip makers say are the future of the industry. The results are distressing. Because of limited memory bandwidth and memory-management schemes that are poorly suited to supercomputers, the performance of these machines would level off or even decline with more cores. The performance is especially bad for informatics applications data-intensive programs that are increasingly crucial to the labs national security function. 23

24 Adding cores slows data-intensive applications High-performance computing has historically focused on solving differential equations describing physical systems, such as Earth s atmosphere or a hydrogen bomb s fission trigger. These systems lend themselves to being divided up into grids, so the physical system can, to a degree, be mapped to the physical location of processors or processor cores, thus minimizing delays in moving data. For informatics, more cores doesn t mean better performance [see red line in Trouble Ahead ], according to Sandia s simulation. After about 8 cores, there s no improvement, says James Peery, director of computation, computers, information, and mathematics at Sandia. At 16 cores, it looks like 2. 24

25 Multicore Is Bad News For Supercomputers Trouble ahead 25

26 Heart of the trouble At the heart of the trouble is the so-called memory wall the growing disparity between how fast a CPU can operate on data and how fast it can get the data it needs. Although the number of cores per processor is increasing, the number of connections from the chip to the rest of the computer is not. So keeping all the cores fed with data is a problem. In informatics applications, the problem is worse because there is no physical relationship between what a processor may be working on and where the next set of data it needs may reside. Instead of being in the cache of the core next door, the data may be on a DRAM chip in a rack 20 meters away and need to leave the chip, pass through one or more routers and optical fibers, and find its way onto the processor. 26

27 Solution (?) The key to solving this bottleneck is tighter, and maybe smarter, integration of memory and processors, says Peery (James Peery, director of computation, computers, information, and mathematics at Sandia). For its part, Sandia is exploring the impact of stacking memory chips atop processors to improve memory bandwidth. 27

28 The rise (and limit) of Many-Core 28

29 Message Passing Architectures Complete computer as building block, including I/O Communication via explicit I/O operations Programming model: directly access only private address space (local memory), comm. via explicit messages (send/receive) High-level block diagram similar to NUMA shred memory approach But comm. integrated at IO level, needn t be into memory system Like networks of workstations (clusters), but tighter integration Programming model more removed from basic hardware operations Library or OS intervention 29

30 Message-Passing Abstraction Address X Local process address space Send X, Q, t Receive Y, P, t AddressY ProcessP Process Q Send specifies buffer to be transmitted and receiving process Recv specifies sending process and application storage to receive into Memory to memory copy, but need to name processes Optional tag on send and matching rule on receive User process names local data and entities in process/tag space too In simplest form, the send/recv match achieves pairwise synch event Other variants too Many overheads: copying, buffer management, protection Match Local process address space 30

31 Evolution of Message-Passing Machines Early machines: FIFO on each link Hw close to prog. Model; synchronous ops Replaced by DMA, enabling non-blocking ops Buffered by system at destination until recv Diminishing role of topology Store&forward routing: topology important Introduction of pipelined routing made it less so Cost is in node-network interface Simplifies programming

32 General interconnection network formed from 8-port switches Example: IBM SP-2 Power 2 CPU L 2 $ Memory bus Memory controller MicroChannel bus IBM SP-2 node 4-way interleaved DRAM NIC I/O i860 DMA NI DRAM Made out of essentially complete RS6000 workstations Network interface integrated in I/O bus (bw limited by I/O bus) 32

33 Example Intel Paragon Sandia s Intel Paragon XP/S-based Supercomputer i860 L 1 $ Memory bus (64-bit, 50 MHz) Mem ctrl 4-way interleaved DRAM i860 L 1 $ Driver Intel Paragon node DMA NI 2D grid network with processing node attached to every switch 8 bits, 175 MHz, bidirectional 33

34 IBM s Hydro Cluster IBM's new Power 575 supercomputer uses a new system of chip-level watercooling to keep its processors chilled. Nicknamed "Hydro Cluster", the machine actually uses 448 of the new 5GHz POWER6 processors. 34

35 Toward Architectural Convergence Evolution and role of software have blurred boundary Traditional MP operations (send/recv) are supported on most shared memory machines via buffers Send involves writing data into the buffer Receive involves reading the data from shared storage Flags or locks are used to control access to the buffer and indicate events such as message arrival On a MP machine, a user process may construct a global address space Access to such a global address can be performed in SW through an explicit message transaction. Most MP libraries allow a process to accept a message for any process, so each process can serve data request from the others. 35

36 Toward Architectural Convergence Evolution and role of software have blurred boundary On a MP machine, a user process may construct a global address space A logical read is realized by sending a request to the process containing the object and receiving a response. The actual message transaction may be hidden from the user; it may be carried out by compiler-generated code for access to a shared variable. A shared virtual address space can be established on a messagepassing machine at the page level. A collection of processes has a region of shared address but, for each process, only the pages that are local to it are accessible. Upon access to a missing (i.e. remote) page, a page fault occurs and the OS engages the remote node in a message transaction to transfer the page and map it into the user address sapce 36

37 Toward Architectural Convergence Evolution and role of software have blurred boundary Send/recv supported on shared memory machines via buffers Can construct global address space on MP using hashing Page-based (or finer-grained) shared virtual memory Hardware organization converging too Tighter network interface (NI) integration even for MP (lowlatency, high-bandwidth) Even clusters of workstations/smps are parallel systems Emergence of fast system area networks (SAN) Programming models distinct, but organizations converging Nodes connected by general network and communication assists Implementations also converging, at least in high-end machines 37

38 Data Parallel Systems Programming model Operations performed in parallel on each element of data structure Logically single thread of control, performs sequential or parallel steps Conceptually, a processor associated with each data element Architectural model Array of many simple, cheap processors with little memory each Processors don t sequence through instructions Attached to a control processor that issues instructions Specialized and general communication, cheap global synchronization Original motivations Matches simple differential equation solvers Centralize high cost of instruction fetch/sequencing Control processor PE PE PE PE PE PE PE PE PE 38

39 Evolution and Convergence Rigid control structure (SIMD in Flynn taxonomy) SISD = uniprocessor, MIMD = multiprocessor Popular when cost savings of centralized sequencer high 60s when CPU was a cabinet Replaced by vectors in mid-70s More flexible w.r.t. memory layout and easier to manage Revived in mid-80s when 32-bit datapath slices just fit on chip No longer true with modern microprocessors Other reasons for demise Simple, regular applications have good locality, can do well anyway Loss of applicability due to hardwiring data parallelism MIMD machines as effective for data parallelism and more general Prog. model converges with SPMD (single program multiple data) Contributes need for fast global synchronization Structured global address space, implemented with either SAS or MP 39

40 Dataflow Architectures Represent computation as a graph of essential dependences Logical processor at each node, activated by availability of operands Message (tokens) carrying tag of next instruction sent to next processor Tag compared with others in matching store; match fires execution a = (b +1) (b c) d = c e f = a d 1 b + a f c d e Dataflow graph Network Token store Program store Waiting Matching Instruction fetch Execute Form token Network Token queue Network 40

41 Evolution and Convergence Key characteristics Ability to name operations, synchronization, dynamic scheduling Problems Operations have locality across them, useful to group together Handling complex data structures like arrays Complexity of matching store and memory units Expose too much parallelism (?) Converged to use conventional processors and memory Support for large, dynamic set of threads to map to processors Typically shared address space as well But separation of progr. model from hardware (like data-parallel) Lasting contributions: Integration of communication with thread (handler) generation Tightly integrated communication and fine-grained synchronization Remained useful concept for software (compilers etc.) 41

42 Systolic Architectures Replace single processor with array of regular processing elements Orchestrate data flow for high throughput with less memory access M M PE PE PE PE Different from pipelining Nonlinear array structure, multidirection data flow, each PE may have (small) local instruction and data memory Different from SIMD: each PE may do something different Initial motivation: VLSI enables inexpensive special-purpose chips Represent algorithms directly by chips connected in regular pattern 42

43 Systolic Arrays (contd.) Example: Systolic array for 1-D convolution x8 x7 y(i) = w1 x(i) + w2 x(i + 1) + w3 x(i + 2) + w4 x(i + 3) x6 x5 x4 y3 y2 y1 x3 w4 xin yin x2 Practical realizations (e.g. iwarp) use quite general processors Enable variety of algorithms on same hardware But dedicated interconnect channels Data transfer directly from register to register across channel Specialized, and same problems as SIMD General purpose systems work well for same algorithms (locality etc.) x w x1 w3 w2 w1 xout yout xout = x x = xin yout = yin + w xin 43

44 Convergence: Generic Parallel Architecture A generic modern multiprocessor Mem $ P Communication assist (CA) Network Node: processor(s), memory system, plus communication assist Network interface and communication controller Scalable network Convergence allows lots of innovation, now within framework Integration of assist with node, what operations, how efficiently... 44

45 Fundamental Design Issues

46 Understanding Parallel Architecture Traditional taxonomies SIMD/MIMD (multiple general-purpose processors) Programming models not enough, nor hardware structures We cannot focus entirely on programming models since in many cases widely differing machine organizations support a common programming model. We cannot just look at HW structures either, since common elements are employed in many different ways. Instead, we ought to focus our attention on the architectural distinctions that make a difference to the software that is to run on the machine. Fundamental Design Issues 46

47 Understanding Parallel Architecture Programming models not enough, nor hardware structures In particular, we need to highlight those aspects that influence how a compiler should generate code from a high-level parallel language, how a library writer would code a well optimized library, or how an application would be written in a low-level parallel language. We can then approach the design problem as one that is constrained from above by how programs use the machine and from below by what the basic technology can provide. Fundamentally, we must understand the operations that are provided at the user-level communication abstraction, how various programming models are mapped to these primitives, and how these primitives are mapped to the actual hardware. Fundamental Design Issues 47

48 Understanding Parallel Architecture The communication abstraction forms the key interface between the programming model and the system implementation. Viewed from the SW side, it must have a precise, well-defined meaning so that the same program will run correctly on many implementations. The operations provided at this layer must be simple, composable entities with clear costs, so that the software can be optimized for performance. Viewed from the hardware side, it also must have a well defined meaning so that the machine designer can determine where performance optimizations can be performed without violating the software assumptions. Fundamental Design Issues 48

49 Understanding Parallel Architecture The communication abstraction forms the key interface between the programming model and the system implementation. While the abstraction needs to be precise, the machine designer would like it not to be overly specific, so it does not prohibit useful techniques for performance enhancement or frustrate efforts to exploit properties of newer technologies The communication abstraction is, in effect, a contract between the HW and the SW allowing each the flexibility to improve what it does, while working correctly together. To understand the terms of this contract, we need to look more carefully at the basic requirements of a programming model. Fundamental Design Issues 49

50 Understanding Parallel Architecture A parallel program consists of one or more threads of control operating on data. A parallel programming model specifies what data can be named by the threads, what operations can be performed on the named data, and what ordering exists among these operations. Fundamental Design Issues 50

51 Fundamental Design Issues At any layer, interface (contract) aspect and performance aspects Naming: How are logically shared data and/or processes referenced? Operations: What operations are provided on these data Ordering: How are accesses to data ordered and coordinated? Fundamental Design Issues Replication: How are data replicated to reduce communication? Communication Cost: Latency, bandwidth, overhead, occupancy Understand at programming model first, since that sets requirements Other issues Node Granularity: How to split between processors and memory?... 51

52 Understanding Parallel Architecture To make these issues concrete, consider the programming model for a uniprocessor. A thread can name the locations in its virtual address space and can name machine registers. In some systems the address space is broken up into distinct code, stack, and heap segments, while in others it is flat. Similarly, different programming languages provide access to the address space in different ways; for example, some allow pointers and dynamic storage allocation, while others do not. Regardless of these variations, the instruction set provides the operations that can be performed on the named locations. Fundamental Design Issues 52

53 Understanding Parallel Architecture To make these issues concrete, consider the programming model for a uniprocessor. For example, in RISC machines the thread can load data from or store data to memory, but perform arithmetic and comparisons only on data in registers. Older instruction sets support arithmetic on either. Compilers typically mask these differences at the hardware/ software boundary, so the user s programing model is one of performing operations on variables which hold data. The hardware translates each virtual address to a physical address on every operation. The ordering among memory operations is sequential program order. Fundamental Design Issues 53

54 Sequential program order The programmer s view is that variables are read and modified in the top-to-bottom, left-to-right order specified in the program. More precisely, the value returned by a read to an address is the last value written to the address in the sequential execution order of the program. This ordering assumption is essential to the logic of the program. However, the reads and writes may not actually be performed in program order, because the compiler performs optimizations when translating the program to the instruction set and the hardware performs optimizations when executing the instructions. Both make sure the program cannot tell that the order has been changed. Fundamental Design Issues 54

55 Sequential program order The compiler and hardware preserve the dependence order. If a variable is written and then read later in the program order, they make sure that the later operation uses the proper value. Collections of reads with no intervening writes may be completely reordered Writes to different addresses can be reordered as long as dependences from intervening reads are preserved. This reordering occurs at the compilation level, for example, when the compiler allocates variables to registers, manipulates expressions to improve pipelining, or transforms loops to reduce overhead and improve the data access pattern. It occurs at the machine level when instruction execution is pipelined, multiple instructions are issued per cycle, or when write buffers are used to hide memory latency. Fundamental Design Issues 55

56 Contract Sequential Programming Model Naming: Can name any variable in virtual address space Hardware (and perhaps compilers) does translation to physical addresses Operations: Loads and Stores Ordering: Sequential program order Performance Rely on dependences on single location (mostly): dependence order Compilers and hardware violate other orders without getting caught Compiler: reordering and register allocation Hardware: out of order, pipeline bypassing, write buffers Transparent replication in caches Fundamental Design Issues 56

57 SAS Programming Model Naming: Any process can name any variable in shared space Operations: loads and stores, plus those needed for ordering Simplest Ordering Model: Within a process/thread: sequential program order Across threads: some interleaving (as in time-sharing) Additional orders through synchronization Again, compilers/hardware can violate orders without getting caught Different, more subtle ordering models also possible (discussed later) Fundamental Design Issues 57

58 Mutual exclusion (locks) Synchronization Ensure certain operations on certain data can be performed by only one process at a time Room that only one person can enter at a time No ordering guarantees Fundamental Design Issues Event synchronization Ordering of events to preserve dependences e.g. producer > consumer of data 3 main types: point-to-point global group 58

59 Message Passing Programming Model Naming: Processes can name private data directly. No shared address space Operations: Explicit communication through send and receive Send transfers data from private address space to another process Receive copies data from process to private address space Must be able to name processes Ordering: Program order within a process Send and receive can provide pt to pt synch between processes Mutual exclusion inherent Can construct global address space: Process number + address within process address space But no direct operations on these names Fundamental Design Issues 59

60 Naming and Operations Naming and operations in programming model can be directly supported by lower levels, or translated by compiler, libraries or OS Example: Shared virtual address space in programming model Fundamental Design Issues Hardware interface supports shared physical address space Direct support by hardware through v-to-p mappings, no software layers Hardware supports independent physical address spaces Can provide SAS through OS, so in system/user interface virtual-to-physical mappings only for data that are local remote data accesses incur page faults; brought in via page fault handlers same programming model, different hardware requirements and cost model Or through compilers or runtime, so above sys/user interface shared objects, instrumentation of shared accesses, compiler support 60

61 Naming and Operations (contd) Example: Implementing Message Passing Direct support at hardware interface But match and buffering benefit from more flexibility Support at sys/user interface or above in software (almost always) Hardware interface provides basic data transport (well suited) Send/receive built in sw for flexibility (protection, buffering) Choices at user/system interface: OS each time: expensive OS sets up once/infrequently, then little sw involvement each time Or lower interfaces provide SAS, and send/receive built on top with buffers and loads/stores Fundamental Design Issues Need to examine the issues and tradeoffs at every layer Frequencies and types of operations, costs 61

62 Ordering Message passing: no assumptions on orders across processes except those imposed by send/receive pairs SAS: How processes see the order of other processes references defines semantics of SAS Ordering very important and subtle Uniprocessors play tricks with orders to gain parallelism or locality These are more important in multiprocessors Need to understand which old tricks are valid, and learn new ones How programs behave, what they rely on, and hardware implications Fundamental Design Issues 62

63 Replication Very important for reducing data transfer/communication Again, depends on naming model Uniprocessor: caches do it automatically Reduce communication with memory Message Passing naming model at an interface A receive replicates, giving a new name; subsequently use new name Replication is explicit in software above that interface SAS naming model at an interface A load brings in data transparently, so can replicate transparently Hardware caches do this, e.g. in shared physical address space OS can do it at page level in shared virtual address space, or objects No explicit renaming, many copies for same name: coherence problem in uniprocessors, coherence of copies is natural in memory hierarchy Fundamental Design Issues 63

64 Communication Performance Performance characteristics determine usage of operations at a layer Programmer, compilers etc make choices based on this Fundamentally, three characteristics: Latency: time taken for an operation Bandwidth: rate of performing operations Cost: impact on execution time of program If processor does one thing at a time: bandwidth 1/latency But actually more complex in modern systems Characteristics apply to overall operations, as well as individual components of a system, however small We ll focus on communication or data transfer across nodes Fundamental Design Issues 64

65 Simple Example Component performs an operation in 100ns Simple bandwidth: 10 Mops Internally pipeline depth 10 => bandwidth 100 Mops Rate determined by slowest stage of pipeline, not overall latency Delivered bandwidth on application depends on initiation frequency Fundamental Design Issues Suppose application performs 100 M operations. What is cost? op count * op latency gives 10 sec (upper bound) op count / peak op rate gives 1 sec (lower bound) assumes full overlap of latency with useful work, so just issue cost if application can do 50 ns of useful work before depending on result of op, cost to application is the other 50ns of latency 65

66 Linear Model of Data Transfer Latency Transfer time (n) = T 0 + n/b useful for message passing, memory access, vector ops etc As n increases, bandwidth approaches asymptotic rate B How quickly it approaches depends on T 0 Size needed for half bandwidth (half-power point): n 1/2 = T 0 / B Fundamental Design Issues But linear model not enough When can next transfer be initiated? Can cost be overlapped? Need to know how transfer is performed 66

67 Communication Cost Model Comm Time per message= Overhead + Assist Occupancy + Network Delay + Size/Bandwidth + Contention = o v + o c + l + n/b + T c Overhead and assist occupancy may be f(n) or not Fundamental Design Issues Each component along the way has occupancy and delay Overall delay is sum of delays Overall occupancy (1/bandwidth) is biggest of occupancies Comm Cost = frequency * (Comm time - overlap) General model for data transfer: applies to cache misses too 67

68 Summary of Design Issues Functional and performance issues apply at all layers Functional: Naming, operations and ordering Performance: Organization, latency, bandwidth, overhead, occupancy Replication and communication are deeply related Management depends on naming model Goal of architects: design against frequency and type of operations that occur at communication abstraction, constrained by tradeoffs from above or below Hardware/software tradeoffs Fundamental Design Issues 68

69 Recap Parallel architecture is important thread in evolution of architecture At all levels Multiple processor level now in mainstream of computing Exotic designs have contributed much, but given way to convergence Push of technology, cost and application performance Basic processor-memory architecture is the same Key architectural issue is in communication architecture How communication is integrated into memory and I/O system on node Fundamental design issues Functional: naming, operations, ordering Performance: organization, replication, performance characteristics Design decisions driven by workload-driven evaluation Integral part of the engineering focus Fundamental Design Issues 69