Architecture or Parallel Computers CSC / ECE 506

Transcription

1 Architecture or Parallel Computers CSC / ECE 506 Summer 2006 Scalable Programming Models 6/19/2006 Dr Steve Hunter

2 Back to Basics Parallel Architecture = Computer Architecture + Communication Architecture Small-scale shared memory extend the memory system to support multiple processors good for multiprogramming throughput and parallel computing allows fine-grain sharing of resources Naming & synchronization communication is implicit in store/load of shared address synchronization is performed by operations on shared addresses Latency & Bandwidth utilize the normal migration within the storage to avoid long latency operations and to reduce bandwidth economical medium with fundamental BW limit CSC / ECE 506 2

3 Realizing Programming Models CAD Database Scientific modeling Parallel applications Multi-programming Shared address Message passing Data parallel Programming models Compilation or library Communication abstraction User/system boundary Operating systems support Communication hardware Hardware/software boundary Physical communication medium P 1 P n Conceptual Picture Memory CSC / ECE 506 3

4 Key Property Large number of independent communication paths between nodes Allow a large number of concurrent transactions using different wires Initiated independently No global arbitration Effect of a transaction only visible to the nodes involved Effects propagated through additional transactions CSC / ECE 506 4

5 Network Transaction Primitive Communication network Serialized data packet Output buffer Input buffer Source node Destination node One-way transfer of information from a source output buffer to a destination input buffer causes some action at the destination occurrence is not directly visible at source Deposit data, state change, reply CSC / ECE 506 5

6 Bus Transactions vs Net Transactions Issues: protection check V->P?? format wires flexible output buffering reg, FIFO?? media arbitration global local destination naming and routing input buffering limited many source action completion detection CSC / ECE 506 6

7 Shared Address Space Abstraction Source Destination (1) Initiate memory access (2) Address translation (3) Local/remote check (4) Request transaction Load r [Global address] Read request Read request (5) Remote memory access Wait Memory access (6) Reply transaction (7) Complete memory access Read response Read response Time fixed format, request/response, simple action CSC / ECE 506 7

8 Consistency is challenging A=1; flag=1; while (flag==0); print A; P 1 P 2 P 3 Memory Memory Memory A:0 flag:0->1 Delay 1: A=1 3: load A 2: flag=1 Interconnection network (a) P 2 P 3 (b) P 1 Congested path CSC / ECE 506 8

9 Synchronous Message Passing Source Destination (1) Initiate send Recv P src, local VA, len (2) Address translation on P src (3) Local/remote check (4) Send-ready request Send P dest, local VA, len Send-rdy req (5) Remote check for posted receive (assume success) Wait Tag check (6) Reply transaction (7) Bulk data transfer Source VA Dest VA or ID Recv-rdy reply Data-xfer req Time CSC / ECE 506 9

10 Asynchronous Message Passing: Optimistic (1) Initiate send (2) Address translation (3) Local/remote check (4) Send data (5) Remote check for posted receive; on fail, allocate data buffer Source Send (P dest, local VA, len) Data-xfer req Destination Tag match Allocate buffer Time Recv P src, local VA, len CSC / ECE

11 Asynchronous Message Passing: Conservative Source Destination (1) Initiate send (2) Address translation on P dest (3) Local/remote check (4) Send-ready request Send P dest, local VA, len Send-rdy req (5) Remote check for posted receive (assume fail); record send-ready Return and compute Tag check (6) Receive-ready request (7) Bulk data reply Source VA Dest VA or ID Recv P src, local VA, len Recv-rdy req Time Data-xfer reply CSC / ECE

12 Active Messages Request handler handler Reply User-level analog of network transaction Action is small user function Request/Reply May also perform memory-to-memory transfer CSC / ECE

13 Common Challenges Input buffer overflow N-1 queue over-commitment => must slow sources reserve space per source (credit)» when available for reuse? Ack or Higher level Refuse input when full» backpressure in reliable network» tree saturation» deadlock free» what happens to traffic not bound for congested dest? Reserve ack back channel drop packets CSC / ECE

14 Challenges (cont) Fetch Deadlock For network to remain deadlock free, nodes must continue accepting messages, even when cannot source them what if incoming transaction is a request?» Each may generate a response, which cannot be sent!» What happens when internal buffering is full? logically independent request/reply networks physical networks virtual channels with separate input/output queues bound requests and reserve input buffer space K(P-1) requests + K responses per node service discipline to avoid fetch deadlock? NACK on input buffer full NACK delivery? CSC / ECE

15 Summary Scalability physical, bandwidth, latency and cost level of integration Realizing Programming Models network transactions protocols safety» N-1» fetch deadlock Next: Communication Architecture Design Space How much hardware interpretation of the network transaction? CSC / ECE

16 The End CSC / ECE