GIAN Course on Distributed Network Algorithms. Network Topologies and Interconnects

Transcription

1 GIAN Course on Distributed Network Algorithms Network Topologies and Interconnects Stefan T-Labs, 2011

2 The Many Faces and Flavors of Network Topologies Gnutella P2P. Social Networks. Internet. Smart grid. Wireless network before topology control Datacenter network Nervous system.

3 The Many Faces and Flavors of Network Topologies Also depends on the abstraction: router level topology vs Autonomous System topology? Gnutella P2P. Social Networks. Internet. Smart grid. Wireless network before topology control Datacenter network Nervous system.

4 The Many Faces and Flavors of Network Topologies given (social network) semi or unstructured (according to simple join protocol) subject to optimization (datacenter or structured p2p) Gnutella 2001 (unstructured p2p system) Chord DHT (structured p2p system)

5 The Many Faces and Flavors of Network Topologies given (social network) semi or unstructured (according to simple join protocol) subject to optimization (datacenter or structured p2p) Gnutella 2001 (unstructured p2p system) Chord DHT (structured p2p system) What makes a good network topology?

6 Properties of a good network topology? It depends! E.g., support simple and efficient routing! e.g., low diameter and short routes (wrt #hops, latency, energy,...?), no state needed at routers (destination address defines next hop), good expansion (fast information dissemination and gossip), etc. E.g., scalable! e.g., small number of neighbors to store (and maintain?), low degree, large bisection bandwidth / cutwidth, redundant paths / no bottleneck links,... E.g, robustness (random or worst-case failures?): e.g., symmetric structure, no single point of failure, redundant paths, good expansion, large mincut, k-connectivity,... Etc.! 6

7 Small diameter does not necessarily imply short routes! Especially in distributed setting. Need good algorithm! Remember the maze Properties of a good network topology? It depends! E.g., support simple and efficient routing! e.g., low diameter and short routes (wrt #hops, latency, energy,...?), no state needed at routers (destination address defines next hop), good expansion (fast information dissemination and gossip), etc. E.g., scalable! e.g., small number of neighbors to store (and maintain?), low degree, large bisection bandwidth / cutwidth, redundant paths / no bottleneck links,... E.g, robustness (random or worst-case failures?): e.g., symmetric structure, no single point of failure, redundant paths, good expansion, large mincut, k-connectivity,... Etc.! 7

8 Example: Is the Gnutella P2P network robust?

9 Example: Is the Gnutella P2P network robust? It depends

10 Example: Is the Gnutella P2P network robust? It depends all connections 30% random peers removed: still mostly connected ( giant component ), robust to random failures / leaves 4% highest degree peers removed: many disconnected components, not robust Measurement study 2001 with ~2000 peers: [Saroiu et al. 2002]

11 Network Topologies = Graphs Network topologies are often described as graphs! Graph G=(V,E): V = set of nodes/peers/..., E= set of edges/links/...? d(.,.): distance between two nodes = shortest path, e.g. d(a,d)=? D(G): diameter (D(G)=max u,v d(u,v)), e.g. D(G)=? (U): neighbor set of nodes U (not including nodes in U) (U) = (U) / U (size of neighbor set compared to size of U) (G) = min U, U <V/2 (U): expansion of G (meaning?)? Expansion captures bottlenecks! A C D B

12 Network Topologies = Graphs Network topologies are often described as graphs! Graph G=(V,E): V = set of nodes/peers/..., E= set of edges/links/... 3 d(.,.): distance between two nodes = shortest path, e.g. d(a,d)=? D(G): diameter (D(G)=max u,v d(u,v)), e.g. D(G)=? (U): neighbor set of nodes U (not including nodes in U) (U) = (U) / U (size of neighbor set compared to size of U) (G) = min U, U <V/2 (U): expansion of G (meaning?) 3 Expansion captures bottlenecks! A C D B

13 Example: Expansion (U), (U)? A U C D B 13

14 Example: Expansion (U), (U)? A U bottleneck! C D B (U) = {C} Therefore: (U)=1/3 14

15 Example: Expansion (U), (U)? A U bottleneck! C D B (U) = {C} Therefore: (U)=1/3 Worst possible: so also expansion of the whole graph! 15

16 The Clique Complete network: pro and cons? Pro: robust, simpleand fast routing, small diameter... Cons: does not scale! (degree?, number of edges?,...) 16

17 The Line Line network: pro and cons? Degree? Diameter? Expansion? Pro: simple and fast routing (tree = unique paths!), small degree (2)... Cons: does not scale! (diameter = n-1) Expansion?

18 The Line Line network: pro and cons? Degree? Diameter? Expansion? Pro: simple and fast routing (tree = unique paths!), small degree (2)... Cons: does not scale! (diameter = n-1, expansion = 2/n,...) Expansion? U (n/2 nodes) (U) (= 1 node)

19 The Line Line network: pro and cons? Degree? Diameter? Expansion? Pro: simple and fast routing (tree = unique paths!), small degree (2)... Cons: does not scale! (diameter = n-1, expansion = 2/n,...) Expansion? Can we reduce diameter without increasing degree by much? U (n/2 nodes) (U) (= 1 node)

20 Good Topologies? Binary tree network: pro and cons? Degree? Diameter? Expansion? 20

21 Good Topologies? Binary tree network: pro and cons? Degree? Diameter? Expansion? Pro: easy and fast routing (tree = unique paths!), small degree (3), log diameter... Cons: bad expansion? 21

22 Good Topologies? Binary tree network: pro and cons? Degree? Diameter? Expansion? Pro: easy and fast routing (tree = unique paths!), small degree (3), log diameter... Cons: bad expansion = 2/n,... G(U) (= 1 node) Expansion: U (~n/2 nodes) All communication from left to right tree goes through root! (no «bisection bandwidth») 22

23 Datacenter Interconnects For this reason, datacenter inter connects form fat-trees: more bandwidth at higher layers which interconnect more nodes. 23

24 Datacenter Interconnects For this reason, datacenter inter connects form fat-trees: more bandwidth at higher layers which interconnect more nodes. More bandwidth for constant bisection bandwidth! Note: more wires, not thicker wires! (Due to ECMP similar effect though?) The Clos topology: one of the most common datacenter interconnects today. 24

25 Datacenter Interconnects For this reason, datacenter inter connects form fat-trees: more bandwidth at higher layers which Servers organized into racks, racks organized into pods. The core routers usually connect to the Internet. interconnect more nodes. The Clos topology: one of the most common datacenter interconnects today. 25

26 Datacenter Interconnects For this reason, datacenter inter connects form fat-trees: more bandwidth at higher layers which Servers organized into racks, racks organized into pods. The core routers usually connect to the Internet. interconnect more nodes. The Clos topology: one of the most common datacenter interconnects today. 26

27 The Mesh 2d Mesh: pro and cons? Degree? Diameter? Expansion? Pro: easy and fast routing (coordinates!), small degree (4), <2 n diameter... Cons: diameter =?, expansion =?

28 The Mesh 2d Mesh: pro and cons? Degree? Diameter? Expansion? Pro: easy and fast routing (coordinates!), small degree (4), <2 n diameter... Cons: diameter = n, expansion = ~2/ n,... (U) = n nodes U ~ n/2 nodes

29 The Hypercube Use identifier manipulation to describe topology and do routing! Here: binary strings Connected iff Hamming distance = 1: flip exactly one bit. 29

30 The Hypercube Formally: d-dim Hypercube: Nodes V = {(b 1,...,b d ), b i binary} (nodes are bitstrings!) Edges E = for all i: (b 1,..., b i,..., b d ) connected to (b 1,..., 1-b i,..., b d )

31 The Hypercube d-dim Hypercube: Nodes V = {(b 1,...,b d ), b i binary} (nodes are bitstrings!) Edges E = for all i: (b 1,..., b i,..., b d ) connected to (b 1,..., 1-b i,..., b d ) Degree? Diameter? Expansion? How to get from (100101) to (011110)?

32 The Hypercube d-dim Hypercube: Nodes V = {(b 1,...,b d ), b i binary} (nodes are bitstrings!) Edges E = for all i: (b 1,..., b i,..., b d ) connected to (b 1,..., 1-b i,..., b d ) d = n nodes, hence d = log(n): degree Diameter: fix one bit after another, so log(n) as well 32

33 The Hypercube d-dim Hypercube: Nodes V = {(b 1,...,b d ), b i binary} (nodes are bitstrings!) Edges E = for all i: (b 1,..., b i,..., b d ) connected to (b 1,..., 1-b i,..., b d ) Simply the maximal Hamming distance! d = n nodes, hence d = log(n): degree Diameter: fix one bit after another, so log(n) as well 33

34 The Hypercube d-dim Hypercube: Nodes V = {(b 1,...,b d ), b i binary} (nodes are bitstrings!) Edges E = for all i: (b 1,..., b i,..., b d ) connected to (b 1,..., 1-b i,..., b d ) Simply the maximal Hamming distance! d = n nodes, hence d = log(n): degree 100 What about expansion? 000 Diameter: fix one bit after another, so log(n) as well

35 Expansion of Hypercube d-dim Hypercube: Nodes V = {(b d,...,b 1 ), b i binary} Edges E = for all i: (b d,..., b i,..., b 1 ) connected to (b d,..., 1-b i,..., b 1 )... Expansion? Find small neighborhood! all nodes with 0x`1 all nodes with 1x`1 all nodes with 2x`1 Idea: nodes with i x`1 are connected nodes with (i-1) x`1 and (i+1) x`1...: 35

36 Expansion of Hypercube Idea: How many nodes? U (~n/2 nodes) (U) = binomial(d,d/2+1)... all nodes with 0x`1 all nodes with 1x`1 all nodes with 2x`1 all nodes with d/2 x`1 all nodes with d/2+1 x`1 36

37 Expansion of Hypercube Idea: How many nodes? U (~n/2 nodes) Number of possibilities to place d/2+1 `1 s at d positions. (U) = binomial(d,d/2+1)... all nodes with 0x`1 all nodes with 1x`1 all nodes with 2x`1 all nodes with d/2 x`1 all nodes with d/2+1 x`1 37

38 Expansion of Hypercube Idea: How many nodes? U (~n/2 nodes) Number of possibilities to place d/2+1 `1 s at d positions. (U) = binomial(d,d/2+1)... all nodes with 0x`1 all nodes with 1x`1 all nodes with 2x`1 all nodes with d/2 x`1 all nodes with d/2+1 x`1 Expansion 1/ (log n) then follows from computing the ratio... 38

39 Expansion of Hypercube Idea: How many nodes? U (~n/2 nodes) Number of possibilities to place d/2+1 `1 s at d positions. (U) = binomial(d,d/2+1)... all nodes with 0x`1 all nodes with 1x`1 all nodes with 2x`1 all nodes with d/2 x`1 Can it even be lower? Also for ball? all nodes with d/2+1 x`1 Expansion 1/ (log n) then follows from computing the ratio... 39

40 Many networks are hypercubic! Many computer networks are variants or generalizations of hypercubes! E.g., peer-to-peer systems (Chord, Pastry, Kademlia,...) E.g., datacenter topologies (container-based datacenters, BCube, MDCCube,...) E.g., parallel architectures (butterfly variants, etc.) 40

41 Many networks are hypercubic! Many computer networks are variants or generalizations of hypercubes! E.g., peer-to-peer systems (Chord, Pastry, Kademlia,...) E.g., datacenter topologies (container-based datacenters, BCube, MDCCube,...) E.g., parallel architectures (butterfly variants, etc.) 41

42 Butterfly: A rolled-out hypercube Bitstrings like hypercube, connected if one bit difference, but not at every position d=1: k={0,1} k=0: k=1: d=2: d+1 (first index) 2 2 d (other indices) but rolled out: just at this position! 42

43 Butterfly: A rolled-out hypercube Bitstrings like hypercube, connected if one bit difference, but not at every position d=1: k={0,1} d=2: Differ in first bit on this level! k=0: Differ in second bit on this level! k=1: d+1 (first index) 2 2 d (other indices) but rolled out: just at this position! 43

44 Butterfly: A rolled-out hypercube Accordingly: 2-dimensional identifier: with the rollout dimension. Bitstrings like hypercube, connected if one bit difference, but not at every position d=1: k={0,1} k=0: k=1: d=2: d+1 (first index) 2 2 d (other indices) but rolled out: just at this position! 44

45 Butterfly: A rolled-out hypercube Formally Butterfly graph: (e.g., for parallel architectures) Nodes V = {(k, b 1...b d ), k ϵ {0,...,d}, b ϵ {0,1} d } (2-dim: number+bitstring ) Undirected edges E = for all i: (k-1, b 1...b k...b d ) connected to (k, b 1...b k...b d ) and (k, b b k...b d ) (i.e., to nodes on next level with same and opposite bit at only this position) Essentially a rolled-out hypercube! d=1: k={0,1} d=2: 0 1 k=0: 0 k=1: Roll-out hypercube into multiple levels: connect i-th bit on i-th level! d (other indices) d+1 (first index) 45

46 Butterfly: A rolled-out hypercube Butterfly graph: (e.g., for parallel architectures) Nodes V = {(k, b 1...b d ), k ϵ {0,...,d}, b ϵ {0,1} d } (2-dim: number+bitstring ) Undirected edges E = for all i: (k-1, b 1...b k...b d ) connected to (k, b 1...b k...b d ) and (k, b b k...b d ) (i.e., to nodes on next level with same and opposite bit at only this position) Essentially a rolled-out hypercube! d=1: k={0,1} k=0: 0 k=1: d=2: d+1 (first index) Diameter, Degree, Expansion? How many nodes in total? 2 2 d (other indices) 46

47 Butterfly: A rolled-out hypercube Butterfly graph: (e.g., for parallel architectures) Nodes V = {(k, b 1...b d ), k ϵ {0,...,d}, b ϵ {0,1} d } (2-dim: number+bitstring ) Undirected edges E = for all i: (k-1, b 1...b k...b d ) connected to (k, b 1...b k...b d ) and (k, b b k...b d ) (i.e., to nodes on next level with same and opposite bit at only this position) Essentially a rolled-out hypercube! d=1: k={0,1} 0 1 k=0: k=1: 0 1 d=2: Diameter, Degree, Expansion? How many nodes in total? d (other indices) d+1 (first index) Degree 4, Diameter 2d (e.g., go to corresponding bottom, then up) 47

48 Butterfly: A rolled-out hypercube Butterfly graph: Nodes V = {(k, b 1...b d ), k ϵ {0,...,d}, b ϵ {0,1} d } Edges E = for all i: (k-1, b 1...b k...b d ) connected to (k, b 1...b k...b d ) and (k, b b k...b d ) Expansion: Halfs only connected where k=d: differ in last bit. So n/d nodes 1 2 U has n/2 nodes. Expansion ~ 1/d. 48

49 CCC: Hypercube with cyclic corners 0,10 1,10 1,11 0,11 0,00 0,01 1,00 1,01 49

50 CCC: Hypercube with cyclic corners Hypercube with round corners : cycles. 0,10 1,10 1,11 0,11 0,00 0,01 1,00 1,01 50

51 CCC: Hypercube with cyclic corners Formally Cube-Connected Cycles: Hypercube with replaced corners (split into d nodes!) Nodes V = {(k, b 1...b d ) k ϵ {0,...,d-1}, b ϵ {0,1} d } Edges E = for all i: (k, b 1...b k...b d ) connected to (k-1, b 1...b k...b d ), (k+1, b 1...b k...b d ) and (k, b b k...b d ) (for each dimension one node!) Example: 1,10 1,11 0,10 0,11 0,00 0,01 1,00 1,01 51

52 De Bruijn Graph: Pull-in bits from the back! Like rolled out hypercube but now it is not bit difference at a certain position which matters, but how strings are shifted wrt to each other!

53 De Bruijn Graph: Pull-in bits from the back! Like rolled out hypercube but now it is not bit difference at a certain position which matters, but how strings are shifted wrt to each other! Shift in Shift in

54 De Bruijn Graph: Pull-in bits from the back! De Bruijn Graph: Nodes V = {(b 1...b d ) ϵ {0,1} d } (bitstrings...) (Undirected) edges E = for all i: (b 1...b k...b d ) Formally connected to (b 2...b d 0) and (b 2... b d 1) ( shift left and add 0 and 1 ) Example (undirected version):

55 De Bruijn Graph: Pull-in bits from the back! De Bruijn Graph: Nodes V = {(b 1...b d ) ϵ {0,1} d } (bitstrings...) (Undirected) edges E = for all i: (b 1...b k...b d ) connected to (b 2...b d 0) and (b 2... b d 1) ( shift left and add 0 and 1 ) Example (undirected version): How to do routing on this graph? 55

56 De Bruijn Graph: Pull-in bits from the back! De Bruijn Graph: Nodes V = {(b 1...b d ) ϵ {0,1} d } (bitstrings...) (Undirected) edges E = for all i: (b 1...b k...b d ) connected to (b 2...b d 0) and (b 2... b d 1) ( shift left and add 0 and 1 ) Example (undirected version): How to do routing on this graph? Fill in destination bits from the left! 56

57 We have seen graphs with diameter-degree pairs: log(n)-log(n) log(n)-o(1) Is there a graph with: O(1)-log(n) max(diameter,degree)< O(log n)?

58 What is the degree-diameter tradeoff? Idea? Proof? Theorem Each network with n nodes and max degree d>2 must have a diameter of at least log(n)/log(d-1)-1. Implications: Constant diameter networks need a linear degree! But also: log(n)-log(n) is not the best tradeoff for minimizing the maximum of degree and diameter!

59 What is the degree-diameter tradeoff? Idea? Proof? Theorem Each network with n nodes and max degree d>2 must have a diameter of at least log(n)/log(d-1)-1. How to prove this? Implications: Constant diameter networks need a linear degree! But also: log(n)-log(n) is not the best tradeoff for minimizing the maximum of degree and diameter!

60 What is the degree-diameter tradeoff? Idea? Proof? Theorem Each network with n nodes and max degree d>2 must have a diameter of at least log(n)/log(d-1) d d-1... Proof by simply counting how many nodes can be reached given a certain degree!

61 What is the degree-diameter tradeoff? Idea? Proof? Theorem Each network with n nodes and max degree d>2 must have a diameter of at least log(n)/log(d-1) In two steps, at most d (d-1) additional nodes can be reached! So in k steps at most: Formally: need to reach n nodes (if connected, i.e., finite diameter). 1 d d-1... Proof by simply counting how many nodes can be reached given a certain degree! To ensure it is connected this must be at least n, so: Reformulating this yields the claim... 61

62 EXERCISE 1: Understand the Pancake Graph O(log n / loglog n) diamter and degree! 62

63 Exercise 1: Pancake Graphs P n 63

64 Pancake Graphs Graph which minimizes max(degree, diameter)! Both in O( log n / log log n ) Nodes = permutations of {1,...,d} Edges = prefix reversals # nodes? degree? d! many nodes and degree (d-1). Routing? E.g., from (3412) to (1243)? Fix bits at the back, one after the other, in two steps, so diameter also log n / log log n. d! = n, so by Stirling formula: d = log(n)/loglog(n) (insert it to d d =n, resp. to d log(d) = log(n)...) 64

65 Solution 1a: Pancake Graphs 65

66 Solution 1b: Degree Pancake Graphs There are n-1 non-trivial prefix reversals for an n-dimensional Pancake with n digits 66

67 Solution 1c: Diameter of Pancake Graphs How to get from node v=v 1...v n to w=w 1...w n? Idea: fix one digit after the other at the back! Two steps: Prefix reversal such that digit is at the front, then full prefix reversal such that digit is at the back Length of routing path: at most 2*(n-1) Papadimitriou and Bill Gates have shown that this is asymptotically also optimal (i.e., close to diameter) 67