Many-to-Many Communications in HyperCast

Transcription

1 Many-to-Many Communications in HyperCast Jorg Liebeherr University of Virginia Jörg Liebeherr, 2001

2 HyperCast Project HyperCast is a set of protocols for large-scale overlay multicasting and peer-to-peer networking Motivating Research Problems: How to organize thousands of applications in a virtual overlay network? How to do multicasting in very large overlay networks?

3 Acknowledgements Team: Past: Bhupinder Sethi, Tyler Beam, Burton Filstrup, Mike Nahas, Dongwen Wang, Konrad Lorincz Current: Jean Ablutz, Weisheng Si, Haiyong Wang, Jianping Wang, Guimin Zhang This work is supported in part by the National Science Foundation: D E N A L I

4 Applications with many receivers 1,000,000 1, Number of Senders Collaboration Tools Games Distributed Information Systems Peer-to-Peer Applications Streaming Software Distribution 10 1,000 1,000,000 Number of Receivers

5 Need for Multicasting? Maintaining unicast connections is not feasible Infrastructure or services needs to support a send to group

6 Problem with Multicasting NAK NAK NAK NAK NAK Feedback Implosion: A node is overwhelmed with traffic or state One-to-many multicast with feedback (e.g., reliable multicast) Many-to-one multicast (Incast)

7 Multicast support in the network infrastructure (IP Multicast) Reality Check (after 10 years of IP Multicast): Deployment has encountered severe scalability limitations in both the size and number of groups that can be supported IP Multicast is still plagued with concerns pertaining to scalability, network management, deployment and support for error, flow and congestion control

8 Overlay Multicasting Logical overlay resides on top of the Layer-3 network Data is transmitted between neighbors in the overlay No network support needed Overlay topology should match the Layer-3 infrastructure

9 Overlay-based approaches for multicasting Build an overlay mesh network and embed trees into the mesh: Narada (CMU) RMX/Gossamer (UCB) many more Build a shared tree: Yallcast/Yoid (NTT, ACIRI) AMRoute (Telcordia, UMD College Park) Overcast (MIT) many more Build an overlay using a logical coordinate spaces : Chord (UCB, MIT) not used for multicast CAN (UCB, ACIRI)

10 HyperCast Approach Build overlay network as a graph with known properties N-dimensional (incomplete) hypercube Delaunay triangulation Advantages: Achieve good load-balancing Exploit symmetry Routing in the overlay comes for free Claim: Can improve scalability of multicast and peer-topeer networks by orders of magnitude over existing solutions

11 Hypercast Software Applications organize themselves to form a logical overlay network with a given topology No central control Dynamic membership hypercube Delaunay triangulation

12 Data Transfer Data is distributed neighborto-neighbor in the overlay network

13 HyperCast Software: Overlay Socket Protocol for transport services in Peer-to-Peer Networks Socket-based API Overlay Socket Interface Streams messages Different reliability semantics Implementation done in Java Statistics Interface OLSocket Overlay Node Interface Overlay Node Adapter Interface Application Transmit Buffer Forwarding Engine Adapter Interface Application Receive Buffer Message Store Network Adapter Network Adapter Software available from: Messages of the Overlay Protocol Application Messages

14 Nodes in a Plane 0,6 4,9 10,8 Nodes are are assigned x-y x-y coordinates (e.g., based on on geographic location) 0,3 0,2 0,1 0,0 1,0 2,0 3,0 5,2 12,0

15 Voronoi Regions 0,6 4,9 10,8 The The Voronoi region of of a node is is the the region of of the the plane that that is is closer to to this this node than than to to any any other node. 5,2 12,0

16 Delaunay Triangulation 0,6 4,9 10,8 The The Delaunay triangulation has has edges between nodes in in neighboring Voronoi regions. 5,2 12,0

17 Delaunay Triangulation 0,6 4,9 10,8 An An equivalent definition: A triangulation such that that each circumscribing circle of of a triangle formed by by three vertices, no no vertex of of is is in in the the interior of of the the circle. 5,2 12,0

18 Locally Equiangular Property Sibson 1977: Maximize the minimum angle C For every convex quadrilateral formed by triangles ACB and ABD that share a common edge AB, the minimum internal angle of triangles ACB and ABD is at least as large as the minimum internal angle of triangles ACD and CBD. A α D β B

19 Next-hop routing with Compass Routing A node s parent in a spanning tree is its neighbor which forms the smallest angle with the root. A node need only know information on its neighbors no routing protocol is needed for the overlay. A Root B is the Node s Parent B Node

20 4,9 10,8 Spanning tree tree when node (8,4) is is root. root. The The tree tree can can be be calculated by by both both parents and and children. 0,6 8,4 5,2 12,0

21 Problem with Delaunay Triangulations Delaunay triangulation considers location of nodes, but not the network topology 2 heuristics to achieve a better mapping Jörg Liebeherr, 2001 UNL, December 2001

22 Hierarchical Delaunay Triangulation 2-level hierarchy of Delaunay triangulations The node with the lowest x-coordinate in a domain DT is a member in 2 triangulations

23 Multipoint Delaunay Triangulation Different ( implicit ) hierarchical organization Virtual nodes are positioned to form a bounding box around a cluster of nodes. All traffic to nodes in a cluster goes through one of the virtual nodes

24 Evaluation of Overlays Simulation: Network with 1024 routers ( Transit-Stub topology) hosts Performance measures for trees embedded in an overlay network: Degree of a node in an embedded tree Relative Delay Penalty : Ratio of delay in overlay to shortest path delay Stress : Number of duplicate transmissions over a physical link

25 Illustration of Stress and Relative Delay Penalty AA Stress = BB Stress = 2 Unicast delay A B : 4 Delay A B in overlay: 6 Relative delay penalty for A B: 1.5

26 Transit-Stub Network Transit-Stub GA Tech topology generator 4 transit domains 4 16 stub domains 1024 total routers 128 hosts on stub domain

27 Overlay Topologies Delaunay Triangulation and variants Hierarchical DT Multipoint DT Degree-6 Graph Similar to graphs generated in Narada Degree-3 Tree Similar to graphs generated in Yoid Logical MST Minimum Spanning Tree Hypercube

28 Average Relative Delay Penalty

29 90 th Percentile of Relative Delay Penalty Delaunay triangulation

30 Average Stress

31 90 th Percentile of Stress Delaunay triangulation

32 The DT Protocol Protocol which organizes members of a network in a Delaunay Triangulation Each member only knows its neighbors soft-state protocol Topics: Nodes and Neighbors Example: A node joins State Diagram Rendezvous Measurement Experiments

33 HelloHello Hello Hello 4,9 HelloHello Each node sends Hello messages to to its its neighbors periodically 10,8 0,6 Hello Hello HelloHello HelloHello 5,2 HelloHello 12,0

34 Each Hello contains the the clockwise (CW) and and counterclockwise (CCW) neighbors Receiver of of a Hello runs runs a Neighbor test test ( ( locally equiangular prop.) CW CW and and CCW are are used to to detect new new neighbors 4,9 10,8 Neighborhood Table of ,6 Neighbor CW CCW Hello CW = 12,0 CCW = 4,9 5,2 12,0 4,9 4,9 5,2 12,0 10,8 5,2 12,0

35 A node that that wants to to join join the the triangulation contacts a node that that is is close 4,9 10,8 0,6 New node Hello 8,4 5,2 12,0

36 Node (5,2) updates its its Voronoi region, and and the the triangulation 4,9 10,8 0,6 8,4 5,2 12,0

37 (5,2) sends a Hello which contains info info for for contacting its its clockwise and and counterclockwise neighbors 4,9 10,8 0,6 5,2 Hello 8,4 12,0

38 (8,4) contacts these neighbors ,9 Hello 10,8 0,6 8,4 5,2 Hello 12,0

39 which update their their respective Voronoi regions. 4,9 10,8 0,6 8,4 5,2 12,0

40 4,9 Hello (4,9) and and (12,0) send Hellos and and provide info info for for contacting their their respective clockwise and and counterclockwise neighbors. 10,8 0,6 8,4 5,2 Hello 12,0

41 (8,4) contacts the the new new neighbor (10,8) ,9 10,8 0,6 8,4Hello 5,2 12,0

42 which updates its its Voronoi region... 4,9 10,8 0,6 8,4 5,2 12,0

43 and responds with with a Hello 0,6 4,9 8,4 Hello 10,8 5,2 12,0

44 This This completes the the update of of the the Voronoi regions and and the the Delaunay Triangulation 4,9 10,8 0,6 8,4 5,2 12,0

45 Rendezvous Methods Rendezvous Problems: How does a new node detect a member of the overlay? How does the overlay repair a partition? Three solutions: 1. Announcement via broadcast 2. Use of a server 3. Use `likely members ( Buddy List )

46 Rendezvous Method 1: 1: 4,9 Announcement via via broadcast (e.g., using IP IP Multicast) 10,8 0,6 New node 5,2 Hello 8,4 12,0

47 4,9 Leader 10,8 Rendezvous Method 1: 1: A Leader is is a node with with a Y-coordinate higher than than any any of of its its neighbors. 0,6 5,2 12,0

48 Rendezvous Method 2: 2: New New node and and leader contact a server. Server keeps a cache of of some other nodes 4,9 10,8 0,6 Server Request New node Server Server Reply (12,0) 5,2 Hello 8,4 NewNode NewNode 12,0

49 Rendezvous Method 3: 3: Each node has has a list list of of likely members of of the the overlay network 4,9 10,8 0,6 5,2 Hello 8,4 NewNode New node with Buddy List: (12,0) (4,9) NewNode 12,0

50 State Diagram of a Node Stopped Application starts Leader without Neighbor Neighbor added (with smaller coordinates) All neighbors leave or timeout Neighbor added (with larger coordinates) All neighbors leave or timeout Application exits Leader with Neighbor A new neighbor with greater coordinates is added After removing some neighbor, this node has largest coordinates Not Leader Send Goodbye Send Goodbye Send Goodbye Leaving

51 Sub-states of a Node Stable Without Candidate Neighbor Node contained in NewNode passes neighbor test After handling the candidate neighbor, node remains stable After neighborhood updating, node becomes not stable After neighborhood updating, node becomes stable. Stable With Candidate Neighbor After neighborhood updating, node becomes not stable Not Stable A node is is stable when all all nodes that that appear in in the the CW CW and and CCW neighbor columns of of the the neighborhood table also also appear in in the the neighbor column

52 Measurement Experiments Experimental Platform: Centurion cluster at UVA (cluster of 300 Linux PCs) 2 to 10,000 overlay members members per PC Internet Switch 3 Gigabit Ethernet centurion183 centurion Switch 4 Switch 8 centurion centurion246 centurion250 centurion251 Switch 5 Switch 9 centurion centurion centurion249 centurion252 Switch 6 Switch 11 centurion centurion centurion Switch 7 Switch 10

53 Experiment: Adding Members How long does it take to add M members to an overlay network of N members? Time to Complete (sec) M+N members

54 Experiment: Throughput of Multicasting 100 MB bulk transfer for N=2-100 members (1 node per PC) 10 MB bulk transfer for N= members (10 nodes per PC) Average throughput (Mbps) Measured values Bandwidth bounds (due to stress) Number of Members N

55 Experiment: Delay 100 MB bulk transfer for N=2-100 members (1 node per PC) 10 MB bulk transfer for N= members (10 nodes per PC) Delay of a packet (msec) Number of Nodes N

56 Summary Use of Delaunay triangulations for overlay networks Delaunay triangulation observes coordinates but ignores network topology No routing protocol is needed in the overlay Ongoing effort: use delay measurements to determine parameters HyperCast Project website: