Multiple Processor Systems. Lecture 15 Multiple Processor Systems. Multiprocessor Hardware (1) Multiprocessors. Multiprocessor Hardware (2)

Transcription

1 Lecture 15 Multiple Processor Systems Multiple Processor Systems Multiprocessors Multicomputers Continuous need for faster computers shared memory model message passing multiprocessor wide area distributed system Multiprocessors Multiprocessor Hardware (1) Definition: A computer system in which two or more CPUs share full access to a common RAM, i.e., there is a shared memory -based multiprocessors 4 Multiprocessor Hardware (2) Multiprocessor Hardware (3) Multiprocessors using multistage switching networks can be built from 2x2 switches Multiprocessor using a crossbar switch (a) 2x2 switch (b) Message format 1

2 Multiprocessor Hardware (4) Multiprocessor Hardware (5) Omega Switching Network UMA Uniform Memory Access NUMA Nonuniform Memory Access NC-NUMA No-Cache NUMA CC-NUMA Cache-Coherent NUMA Multiprocessor Hardware (6) Multiprocessor Hardware (7) NUMA Multiprocessor Characteristics 1. Single address space visible to all CPUs 2. Access to remote memory via commands - LOAD - STORE 3. Access to remote memory slower than to local (a) 256-node directory based multiprocessor (b) Fields of 32-bit memory address (c) Directory at node 36 Each CPU has its own operating system Disks and I/O devices are shared Memory can be allocated accordingly Communication between processes is simple Each CPU has its own operating system System calls are handled locally No sharing of processes No sharing of pages Disk caches may have duplicated and inconsistet blocks 2

3 Master-Slave multiprocessors Single OS data structure Load balance Better distribution of pages Only 1 disk cache Master-Slave multiprocessors Master answers all the system calls Bottleneck if there is a large number of CPUs Doesn t scale (SMP) Symmetric MultiProcessors Used by most modern multiprocessor Systems One copy of the OS in memory, any CPU can run it Balances processes and memory dynamically No bottleneck Symmetric Multiprocessors Problem: Need synchronization Solution: Use mutual exclusion for critical regions and for critical tables Hard to accomplish! Need to prevent deadlocks by ordering accesses. Multiprocessor Synchronization Multiprocessor Synchronization Problem It takes 2 separate bus cycles to perform the memory read and memory write Solution Lock the bus! Modern computers have a special bus line for this purpose. 3

4 Multiprocessor Synchronization Multiprocessor Synchronization More problems spin lock cache thrashing Multiple locks used to avoid cache thrashing Multiprocessor Synchronization Spinning versus Switching In some cases CPU must wait waits to acquire ready list In other cases a choice exists spinning wastes CPU cycles switching uses up CPU cycles also possible to make separate decision each time locked mutex encountered Timesharing independent processes single data structure for scheduling, contention automatic load balance Timesharing Smart scheduling Do not preempt processes that are holding a spin lock Use a flag for that Affinity Scheduling Try to run processes in the same processors to be able to reuse cache and TLB entries Two-level Scheduling Algorithm For affinity, associate each CPU with a set of processes However, idle CPUs may execute other CPU s processes Space sharing multiple threads (that belong to the same process) at same time across multiple CPUs 4

5 Space sharing No multiprogramming Processes run in dedicated mode Processes take control of the CPUs they are assigned to until they finish A process starts only when there is enough CPUs available for all its threads Scheduling is done at the job level FCFS is hard to beat Space and Timesharing To eliminate the waste of CPU cycles in space sharing Problem Threads may run out of phase Problem with communication between two threads both belong to process A both running out of phase 1 request/reply every 200 msec Solution: Gang Scheduling Groups of related threads scheduled as a unit (a gang) All members of gang run simultaneously on different timeshared CPUs All gang members start and end time slices together Multicomputers Definition: Tightly-coupled CPUs that do not share memory Gang Scheduling Also known as cluster computers clusters of workstations (COWs) 5

6 Multicomputer Hardware Multicomputer Hardware Interconnection topologies (a) single switch (b) ring (c) grid (d) double torus (e) cube (f) hypercube Switching scheme store-and-forward packet switching Multicomputer Hardware Multicomputer Hardware Other Switching Schemes Circuit switching Wormhole routing Network interface boards in a multicomputer Low-Level Communication Software Low-Level Communication Software If several processes running on node need network access to send packets Map interface board to all process that need it problem: requires synchronization If kernel needs access to network Use two network boards one to user space, one to kernel Node to Network Interface Communication Use send & receive rings coordinates main CPU with on-board CPU 6

7 User Level Communication Software Remote Procedure Call Minimum services provided send and receive commands (a) Blocking send call These are blocking (synchronous) calls (b) Nonblocking send call Steps in making a remote procedure call the stubs are shaded gray Remote Procedure Call Implementation Issues Cannot pass pointers call by reference becomes copy-restore (but might fail) Weakly typed languages client stub cannot determine size Not always possible to determine parameter types Cannot use global variables may get moved to remote machine Distributed Shared Memory Note layers where it can be implemented hardware operating system user-level software Distributed Shared Memory Distributed Shared Memory Replication (a) Pages distributed on 4 machines (b) CPU 0 reads page 10 (c) CPU 1 reads page 10 False Sharing Must also achieve sequential consistency 7

8 Multicomputer Scheduling Multicomputer Scheduling Load Balancing Each node has its own memory and its own set of processes Scheduling is local and simple Any strategy works Gang scheduling helps However, allocation is important Goal: balance the load and decrease communication Process Graph-theoretic deterministic algorithm Load Balancing Load Balancing Sender-initiated distributed heuristic algorithm overloaded node tryes to get rid of excess work Receiver-initiated distributed heuristic algorithm under loaded node tryes to get work 8