High Node Count - Scalability Challenges for Interconnection Networks

Transcription

1 High Node Count - Scalability Challenges for Interconnection Networks Professor Olav Lysne Simula Research Laboratory

2 Overview Congestion control Fault Tolerance Scalable Modular Routing State Of The Art: State of Technology State of Knowledge State of Problem

3 CONGESTION CONTROL

4 Shared network resources could lead to network congestion and head-of-line (HOL) blocking. BECN FEC N Host Congestion tree - CCT - CCT Index Increase - CCT Index Min HOL blocked traffic (Victim) Switch - Threshold The InfiniBand CC mechanism - CCT relies Index Limit on a closed loop - Marking feedback Rate control systems to remove the - CCT congestion Index Timer tree. - Packet Size

5 Experiments show that the HOL blocking leads to performance degradation when CC is not activated.

6 The InfiniBand CC mechanism is able to remove both the HOL blocking and the parking lot problem. Parameter Values: Threshold 15 Marking Rate 1 Packet Size 8 CCTI Increase 1 CCTI Limit 127 CCTI Min 0 CCTI Timer 150 Without CC With CC

7 The average throughput of the victim flow as a function of the Marking_Rate (sw) and the CCTI_Timer (host).

8 The average combined throughput of the contributors as a function of the Marking_Rate and the CCTI_Timer.

9 Contributors may experience unfairness if an unfortunate CCTI_Timer value is chosen Contributors experience unfairness among each other for an extended periode of time each time a new contributer is added when an unfortunate timer is chosen. = (max value) (min value) Max value Min value.. The treatment variation variable : TVV = Var( 1, 2,..., n)

10 The treatment variation variable rules out a large part of the parameter space. Parameter Values: Threshold 15 Marking Rate 1 Packet Size 8 CCTI Increase 1 CCTI Limit 127 CCTI Min 0 CCTI Timer 150

11 InfiniBand Congestion Control in M9 (SUN DATACENTER INFINIBAND SWITCH 648) Gbps IBTA Specification 1.2 compliant 648 QDR/DDR/SDR 4x InfiniBand ports Three-stage internal full Clos network (non-blocking) Gbps!HS,!CC HS,!CC HS, CC HS, CC HS, CC QP!HS, CC 20% of the nodes send to everyone 80% of the nodes send to 8 hotspots Further simulation studies: - Different traffic patterns - Other topologies (M24: SUN DATA CENTER SWITCH 3456)!HS,!CC HS,!CC HS, CC HS, CC HS, CC QP!HS, CC

12 Congestion Control - State of the art State of technology InfiniBand Congestion Control Fecn/Becn Datacenter Ethernet - TBD Much more to be expected State of Knowledge Regional Explicit Congestion Notification Improvements on Fecn/Becn Parametrizations Dynamics? Impact on applications? Much more to do

13 Fault Tolerance -Living with faults Static Reconfiguration-based End-to-End Rerouting Local Rerouting

14 What is network deadlock? Deadlock is a cycle of packets all waiting for the next packet in the cycle to proceed before it can proceed itselg Routing functions may be deadlock free topologies may not for almost all topologies there exist reasonable but deadlocking routing functions, as well as reasonable and deadlock free routing functions.

15 Static Fault Tolerance Checkpoint Reconfigure Rollback Restart Requires topology agnostic routing algorithms LASH, TOR, LASH/TOR, L-turn, Segment-based, Up*/Down

16 Dynamic Reconfiguration A deadlock will contain old packets waiting behind new packets as well as new packets waiting Dependencies of Rold Key idea: make sure that new packets never wait

17 Dynamic Reconfiguration TOKEN A deadlock will contain old packets waiting behind new packets as well as new packets waiting Dependencies of Rold Key idea: make sure that new packets never wait

18 Dynamic Reconfiguration Dependencies of Rold STOP Depend. of Rnew A deadlock will contain old packets waiting behind new packets as well as new packets waiting Key idea: make sure that new packets never wait

19 Dynamic Reconfiguration Dependencies of Rold TOKEN TOKEN A deadlock will contain old packets waiting behind new packets as well as STOP Depend. of Rnew new packets waiting Key idea: make sure that new packets never wait

20 Dynamic Reconfiguration Dependencies of Rold TOKEN TOKEN A deadlock will contain old packets waiting behind new packets as well as STOP Depend. of Rnew new packets waiting Key idea: make sure that new packets never wait

21 Dynamic Reconfiguration Dependencies of Rold TOKEN TOKEN A deadlock will contain old packets waiting behind new packets as well as STOP Depend. of Rnew new packets waiting Key idea: make sure that new packets never wait

22 Dynamic Reconfiguration Dependencies of Rold TOKEN TOKEN A deadlock will contain old packets waiting behind new packets as well as STOP Depend. of Rnew new packets waiting Key idea: make sure that new packets never wait

23 Dynamic Reconfiguration TOKEN Dependencies of Rold STOP Depend. of Rnew A deadlock will contain old packets waiting behind new packets as well as new packets waiting Key idea: make sure that new packets never wait

24 Dynamic Reconfiguration TOKEN Dependencies of Rold STOP Depend. of Rnew A deadlock will contain old packets waiting behind new packets as well as new packets waiting Key idea: make sure that new packets never wait

25 Dynamic Reconfiguration Dependencies of Rold STOP Depend. of Rnew A deadlock will contain old packets waiting behind new packets as well as new packets waiting Key idea: make sure that new packets never wait TOKEN

26 Dynamic Reconfiguration Depend. of Rnew A deadlock will contain old packets waiting behind new packets as well as new packets waiting Key idea: make sure that new packets never wait

27 Dynamic Reconfiguration Depend. of Rnew A deadlock will contain old packets waiting behind new packets as well as new packets waiting Key idea: make sure that new packets never wait

28 Views Fully Connected Subnetworks for endpoint fault-tolerance The fat-tree is divided into a set of sub-networks. Each of these constitute a view.

29 Views Fully Connected Subnetworks for endpoint fault-tolerance Close-up of a subtree with 3 views One link is present in one, and only one, view Any path through the network is contained entirely within one view Only bottom-tier switches (and the endnode-connections) will contain traffic for serveral views.

30 FROOTS Dynamic Fault Tolerance Configuration 1 Configuration 2

31 Full VL for the non affected traffic, and two VLs for the traffic affected by faults.

32 Handling faults

33 Fault tolerance State Of The Art State of technology Topology agnostic routing algorithms (OFED) Static Reconfiguration with LASH (OFED) Endpoint Dynamic Reconfiguration (APM in IBA) State of Knowledge Dynamic Reconfiguration Local Rerouting New Compatible Routing function

34 Modularity of routing

35 What is the problem?

36 Dependencies aggregate c 1 c 2 The aggregated dependencies in a switch fabric must either be identified and removed, or taken into consideration in how the fabric is used

37 So... A configuration of Network of networks is free from deadlocks if its channel dependency graph extended with the aggregated dependencies in the switches is free from deadlocks.

38 Well what about local fault tolerance?

39 Modularity of routing State of technology Not present State of Knowledge Wide open but there is an approach

40 There is a way to do it better find it! T. A. Edison Simplicity is the ultimate sophistication. L.DaVinci