EECS 570. Lecture 19 Interconnects: Flow Control. Winter 2018 Subhankar Pal

Transcription

1 Lecture 19 Interconnects: Flow Control Winter 2018 Subhankar Pal Slides developed in part by Profs. Adve, Falsafi, Hill, Lebeck, Martin, Narayanasamy, Nowatzyk, Reinhardt, Singh, Smith, Torrellas and Wenisch. Special acknowledgement to Prof. Jerger of U. Toronto. Slide 1

2 Readings For today: Mukherjee et al. The Alpha Network Architecture, Hot Interconnects Jerger & Peh. Interconnection Networks Ch. 4 For Friday 3/30: Jerger & Peh. On-Chip Networks Ch. 5 Scott & Thoreson. The Cray T3E Network: Adaptive Routing in a High Performance 3D Torus, Hot Interconnects Slide 2

3 Types of Routing Algorithms Slide 3

4 Oblivious Routing decisions are made without regard to network state Keeps algorithms simple Unable to adapt Main idea with oblivious routing Multiple routing choices in routing table Choose among them stochastically Deterministic algorithms are a subset of oblivious Slide 4

5 Valiant s Routing Algorithm To route from s to d Randomly choose intermediate node d Route from s to d and from d to d. Randomizes any traffic pattern All patterns appear uniform random Balances network load Non-minimal Destroys locality s d d Slide 5

6 Minimal Oblivious Valiant s: Load balancing but significant increase in hop count Minimal Oblivious: some load balancing, but use shortest paths d must lie within min quadrant 6 options for d Only 3 different paths s d Slide 6

7 Oblivious Routing Valiant s and Minimal Adaptive Deadlock free When used in conjunction with X-Y routing Randomly choose between X-Y and Y-X routes Oblivious but not deadlock free! Slide 7

8 Exploits path diversity Adaptive Uses network state to make routing decisions Buffer occupancies often used Coupled with flow control mechanism Local information readily available Global information more costly to obtain Network state can change rapidly Use of local information can lead to non-optimal choices Can be minimal or non-minimal Slide 8

9 Minimal Adaptive Routing d s Local info can result in sub-optimal choices Slide 9

10 Non-minimal adaptive Fully adaptive Not restricted to take shortest path Misrouting: directing packet along non-productive channel Priority given to productive output Some algorithms forbid U-turns Livelock potential: traversing network without ever reaching destination Simplest non-minimal adaptive routing scheme? Hot potato routing No buffers, just throw the packet to a neighboring node Useful in optical networks where messages made from light can not be stored in any medium Slide 10

11 Non-minimal routing example d d s s Longer path with potentially lower latency Mechanisms to guarantee forward progress? Limit number of misroutings Prioritize packets based on age Livelock: continue routing in cycle Slide 11

12 Adaptive Routing Example Should 3 route clockwise or counterclockwise to 7? 5 is using all the capacity of link 5 à 6 Queue at node 5 will sense contention but not at node 3 Backpressure: allows nodes to indirectly sense congestion Queue in one node fills up, it will stop receiving flits Previous queue will fill up If each queue holds 4 packets 3 will send 8 packets before sensing congestion Slide 12

13 Adaptive Routing: Turn Model DOR eliminates 4 turns N to E, N to W, S to E, S to W No adaptivity Some adaptivity by removing 2 of 8 turns Remains deadlock free (like DOR) West first Eliminates S to W and N to W West first Slide 13

14 Turn Model Routing Negative first Eliminates E to S and N to W North last Eliminates N to E and N to W Odd-Even Eliminates 2 turns depending on if current node is in odd or even col. Even column: E to N and N to W Odd column: E to S and S to W Deadlock free (disallow 180 turns) Better adaptivity Negative first North last Slide 14

15 Negative-First Routing Example (2,3) (0,3) (0,0) (2,0) Limited or no adaptivity for certain source-destination pairs Slide 15

16 Turn Model Routing Deadlock What about eliminating turns NW and WN? Not a valid turn elimination Resource cycle results Slide 16

17 Adaptive Routing and Deadlock Option 1: Eliminate turns that lead to deadlock Limits flexibility Option 2: Allow all turns Give more flexibility Must use other mechanism to prevent deadlock Rely on flow control (later) Escape Virtual Channels Slide 17

18 Routing Summary Latency paramount concern Minimal routing most common for NoC Non-minimal can avoid congestion and deliver low latency To date: NoC research favors DOR for simplicity and deadlock freedom On-chip networks often lightly loaded Buffers are expensive on-chip! We only covered unicast routing Recent work on extending on-chip routing to support multicast Slide 18

19 Interfaces Topology Routing Flow Control Router Microarchitecture Next Topic Slide 19

20 Flow Control Slide 20

21 Switching/Flow Control Overview Topology: determines connectivity of network Routing: determines paths through network Flow Control: determine allocation of resources to messages as they traverse network Buffers and links Significant impact on throughput and latency of network Lossless versus lossy flow control Which is preferable for on-chip networks? Lossless! Lossy would mean additional hardware to buffer messages and also more traffic in the network Slide 21

22 Terminology Message Composed of one or more packets If message size is <= maximum packet size only one packet created Each packet has enough information to be routed independently Packets Composed of one or more flits (flow control unit) Control packets, data packets Flit Flow control digit Phit Physical digit Subdivides flit into chunks = to link width Slide 22

23 Packets Message Header Payload Packet Route Seq# Head Flit Body Flit Tail Flit Flit Type VCID Head, Body, Tail, Head & Tail Phit Off-chip: channel width limited by pins Requires phits On-chip: abundant wiring means phit size == flit size Slide 23

24 Packets Cache line RC Type VCI D Add r Bytes 0-15 Bytes Bytes Bytes Head Flit Body Flits Tail Flit Coherence Command RC Type VCI D Add r Cmd Head & Tail Flit Packet contains destination/route information Flits may not à all flits of a packet must take same route Slide 24

25 Flow Control Control State Channel Bandwidth Buffer capacity Control state records: allocation of channels and buffers to packets current state of packet traversing node Channel bandwidth advances flits from this node to next Buffers hold flits waiting for channel bandwidth Slide 25

26 Types of Switching Slide 26

27 Switching Different flow control techniques based on granularity Message-based: allocation made at message granularity (circuit-switching) Packet-based: allocation made to whole packets Flit-based: allocation made on a flit-by-flit basis Slide 27

28 Message-Based Flow Control Coarsest granularity Circuit-switching Pre-allocates resources across multiple hops Source to destination Resources = links Buffers are not necessary Probe sent into network to reserve resources Slide 28

29 Circuit Switching Once probe sets up circuit Message does not need to perform any routing or allocation at each network hop Good for transferring large amounts of data Can amortize circuit setup cost by sending data with very low per-hop overheads No other message can use those resources until transfer is complete Throughput can suffer due setup and hold time for circuits Links are idle until setup is complete Slide 29

30 Circuit Switching Example 0 Configuration Probe 5 Data Circuit Acknowledgement Significant latency overhead prior to data transfer Data transfer does not pay per-hop overhead for routing and allocation Slide 30

31 Circuit Switching Example (2) Configuration Probe 5 Data Circuit Acknowledgement When there is contention Significant wait time Message from 1 à 2 must wait Slide 31

32 Time-Space Diagram: Circuit-Switching Location S 08 S 08 S 08 S 08 S 08 A08 A 08 A 08 D 0 8 D 0 8 D 0 8 D 0 8 S 28 A 08 D 0 D 0 D 0 D 0 T 08 S A 08 D 0 D 0 D 0 D 0 T D 0 8 D 0 8 D 0 8 D 0 8 T 08 T 08 S 28 8 D 0 8 D 0 8 D 0 8 D 0 8 T Time to setup+ack circuit from 0 to 8 Time Time setup from 2 to 8 is blocked Slide 32

33 Packet-based Flow Control Break messages into packets Interleave packets on links Better utilization Requires per-node buffering to store in-flight packets Two types of packet-based techniques Slide 33

34 Store and Forward Links and buffers are allocated to entire packet Head flit waits at router until entire packet is received before being forwarded to the next hop Not suitable for on-chip Requires buffering at each router to hold entire packet Packet cannot traverse link until buffering allocated to entire packet Incurs high latencies (pays serialization latency at each hop) Slide 34

35 Store and Forward Example 0 5 Total delay = 4 cycles per hop x 3 hops = 12 cycles High per-hop latency Serialization delay paid at each hop Larger buffering required Slide 35

36 Time-Space Diagram: Store and Forward 0 H B B B T 1 H B B B T Location 2 5 H B B B T H B B B T 8 H B B B T Time Slide 36

37 Packet-based: Virtual Cut Through Links and Buffers allocated to entire packets Flits can proceed to next hop before tail flit has been received by current router But only if next router has enough buffer space for entire packet Reduces the latency significantly compared to SAF But still requires large buffers Unsuitable for on-chip Slide 37

38 Virtual Cut Through Example 0 Allocate 4 flit-sized Allocate 4 flit-sized buffers before buffers head before head proceeds proceeds 5 Total delay = 1 cycle per hop x 3 hops + serialization = 6 cycles Lower per-hop latency Large buffering required Slide 38

39 Time-Space Diagram: VCT 0 1 H B B B T H B B B T Location 2 5 H B B B T H B B B T 8 H B B B T Time Slide 39

40 Virtual Cut Through Cannot proceed because only 2 flit buffers available Throughput suffers from inefficient buffer allocation Slide 40

41 Time-Space Diagram: VCT (2) 0 1 H B B B T H B B B T Location H B B B T Insufficient Buffers H B B B T H B B B T Time Slide 41

42 Flit-Level Flow Control Help routers meet tight area/power constraints Flit can proceed to next router when there is buffer space available for that flit Improves over SAF and VCT by allocating buffers on a flitby-flit basis Slide 42

43 Pros Wormhole Flow Control More efficient buffer utilization (good for on-chip) Low latency Cons Poor link utilization: if head flit becomes blocked, all links spanning length of packet are idle Cannot be re-allocated to different packet Suffers from head of line (HOL) blocking Slide 43

44 Wormhole Example Red holds this channel: channel remains idle until red proceeds Channel idle but red packet blocked behind blue Buffer full: blue cannot proceed Blocked by other packets 6 flit buffers/input port Slide 44

45 Time-Space Diagram: Wormhole 0 1 H B B B T H B B B T Location 2 5 Contention H B B B T H B B B T 8 H B B B T Time Slide 45

46 Virtual Channels First proposed for deadlock avoidance We ll come back to this Can be applied to any flow control First proposed with wormhole Slide 46

47 Virtual Channel Flow Control Virtual channels used to combat HOL blocking in wormhole Virtual channels: multiple flit queues per input port Share same physical link (channel) Link utilization improved Flits on different VC can pass blocked packet Slide 47

48 Virtual Channel Flow Control (2) A (in) B (in) A (out) B (out) Slide 48

49 Virtual Channel Flow Control (3) In1 AH A1 A2 A3 A4 A AT In2 BH B1 B2 B3 B4 B BT Out AH BH A1 B1 A2 B2 A3 B3 A4 B4 A5 B5 AT BT A Downstream B Downstream AH A1 A2 A3 A4 A5 AT BH B1 B2 B3 B4 B5 BT Slide 49

50 Virtual Channel Flow Control (3) Packets compete for VC on flit by flit basis In example: on downstream links, flits of each packet are available every other cycle Upstream links throttle because of limited buffers Does not mean links are idle May be used by packet allocated to other VCs Slide 50

51 Virtual Channel Example 6 flit buffers/input port Buffer full: blue cannot proceed Blocked by other packets 3 flit buffers/vc Slide 51

52 Summary of techniques Circuit- Switching Store and Forward Virtual Cut Through Links Buffers Comments Messages N/A (buffer-less) Setup & Ack Packet Packet Head flit waits for tail Packet Packet Head can proceed Wormhole Packet Flit HOL Virtual Channel Flit Flit Interleave flits of different packets Slide 52

53 Deadlock Slide 53

54 Deadlock Using flow control to guarantee deadlock freedom give more flexible routing Recall: routing restrictions needed for deadlock freedom If routing algorithm is not deadlock free VCs can break resource cycle Each VC is time-multiplexed onto physical link Holding VC implies holding associated buffer queue Not tying up physical link resource Enforce order on VCs Slide 54

55 Deadlock: Enforce Order A0 C A B A1 B1 B0 D D C B D 0 D 1 C1 C0 A All message sent through VC 0 until cross dateline After dateline, assigned to VC 1 Cannot be allocated to VC 0 again Slide 55

56 Deadlock: Escape VCs Enforcing order lowers VC utilization Previous example: VC 1 underutilized Escape Virtual Channels Have 1 VC that is deadlock free Example: VC 0 uses DOR, other VCs use arbitrary routing function Access to VCs arbitrated fairly: packet always has chance of landing on escape VC Assign different message classes to different VCs to prevent protocol level deadlock Prevent req-ackmessage cycles Slide 56

57 Buffer Backpressure Slide 57

58 Buffer Backpressure Need mechanism to prevent buffer overflow Avoid dropping packets Upstream nodes need to know buffer availability at downstream routers Significant impact on throughput achieved by flow control Two common mechanisms Credits On-off Slide 58

59 Credit-Based Flow Control Upstream router stores credit counts for each downstream VC Upstream router When flit forwarded Decrement credit count Count == 0, buffer full, stop sending Downstream router When flit forwarded and buffer freed Send credit to upstream router Upstream increments credit count Slide 59

60 Credit Timeline t1 t2 t3 Process Node 1 Node 2 Flit departs router Credit round trip delay t4 t5 Process Round-trip credit delay: Time between when buffer empties and when next flit can be processed from that buffer entry If only single entry buffer, would result in significant throughput degradation Important to size buffers to tolerate credit turn-around Slide 60

61 On-Off Flow Control Credit: requires upstream signaling for every flit On-off: decreases upstream signaling Off signal Sent when number of free buffers falls below threshold F off On signal Sent when number of free buffers rises above threshold F on Slide 61

62 On-Off Timeline t1 Node 1 Node 2 F off threshold reached F off set to prevent flits arriving before t4 from overflowing t2 t3 t4 Process F on set so that Node 2 does not run out of flits between t5 and t8 t5 t6 t7 t8 Process F on threshold reached Less signaling but more buffering On-chip buffers more expensive than wires Slide 62

63 Buffer Utilization Cycle Credit count Head Flit VA/ SA ST LT BW VA/ SA ST Body Flit 1 SA ST LT BW SA ST Credit (head) Body Flit 2 C C-LT C- Up SA ST LT Credit (body 1) Tail Flit C C-LT C- Up SA ST LT Slide 63

64 Buffer Sizing Prevent backpressure from limiting throughput Buffers must hold # of flits >= turnaround time Assume: 1 cycle propagation delay for data and credits 1 cycle credit processing delay 3 cycle router pipeline At least 6 flit buffers Slide 64

65 Actual Buffer Usage & Turnaround Delay Actual buffer usage Credit propagation delay Credit pipeline delay flit pipeline delay flit propagation delay Flit arrives at node 1 and uses buffer Flit leaves node 1 and credit is sent to node 0 Node 0 receives credit Node 0 processes credit, freed buffer reallocated to new flit New flit leaves Node 0 for Node 1 New flit arrives at Node 1 and reuses buffer Slide 65

66 Flow Control Summary On-chip networks require techniques with lower buffering requirements Wormhole or Virtual Channel flow control Avoid dropping packets in on-chip environment Requires buffer backpressure mechanism Complexity of flow control impacts router microarchitecture Slide 66