Scalable P2P architectures

Transcription

1 Scalable P2P architectures Oscar Boykin Electrical Engineering, UCLA Joint work with: Jesse Bridgewater, Joseph Kong, Kamen Lozev, Behnam Rezaei, Vwani Roychowdhury, Nima Sarshar

2 Outline Introduction to P2P models: DHT and Unstructured Query Systems Routing Packets on a Small World Properties of real P2P systems (e.g. Gnutella). A model for Power law graphs Percolating Messages on a Graph Design of a new P2P system: Brunet

3 Hash Table A hash table is a simple data structure which can lookup in O(1) time. Each object has a hash code. This hash code is used as an index into an array. As long as each object has a different hash code, lookup happens in O(1). If there are on average m duplicates, look up is O(m). Where's the P2P?

4 Distributed Hash Table (DHT) Rather than keep all the elements of the hash table locally, they could point to nodes in a network:

5 DHT Systems If each node does not have a pointer to every other node, routing schemes are introduced. Each node knows about k other nodes. All queries are routed through these k nodes. The query should be resolved in the fewest number of hops. Most academic work has focused on DHT systems.

6 Hyperspace Routing (Pastry and Tapestry) Messages are routed by matching the prefix of the destination to the current node, and sending to the node which matches the next element. Nodes need O( M (log n)/log M ) neighbors for an alphabet of size M, which gives O( log n/ log M) distance Examples: Routing 101 starting at 000: 000 > 100 > 101 Routing 101 starting at 010: 110 > 100 > 101 Routing 101 starting at 011: 011 > 111 > 101

7 Distance Based Routing A distance metric is defined on the key space. Nodes are connected to their nearest neighbors in the space and usually to remote nodes. Messages are routed to the node which is closest to the destination. Examples: System Space Latency Connections CAN M-dimension torus M N ^{1/M} Neighbors: M Chord Ring log N Neighbors exponentially increasing: log N Symphony Ring (log^2 N)/k Neighbors and k remote Viceroy log N stacked rings log N Neighbors

8 Small World Model High clustering Regular Grid p=0 Large diameter Small World High clustering Low diameter p=1 Classic Random Network D. Watts and S. Strogatz, "Collective dynamics of small-world networks," Nature 393, 440 (1998). Low clustering Low diameter

9 Making a Small World Routable Kleinberg (2000) suggested a connection algorithm to make a ring routable: The ring is of unit circumference with N nodes. Each node has an address which represents its position on the ring. It is connected to its closest neighbor on each side, and one remote node. The probability that the remote node is at a distance L is p (L) ~ 1/L (one can generalize to allow k such connections). By following the path which takes you closest to the

10 The red nodes have connections to distance L with P(L) ~ 1/L A Routable Small World How can we show it is routable?

11 Greedy Routing Works The probability of connections going a distance d: P(d)=1/d log N What's the probability that a connection takes us to a distance less than d: P = d 1 d 1 x log N dx= log 1 log N Distance d Source Distance = d Destination

12 Greedy Routing Works How many such connections are needed to get close: How many nodes (M) do we need to get lucky L times: M P =L M=L log 1 log N log N log d log log N M= log 1 log Since we must be prepared for d = N, then: M = O(log^2 N) Distance d L d=log N L= log log N d log Source Distance = d Destination

13 Almost all are unstructured query networks! DHT Summary DHTs have nice theory associated with them, but they are not suited to all problems. There are many proposals to get poly log communications complexity. Small World systems (Chord, Symphony) are relatively easy to implement. DHTs are excellent in applications where a user knows EXACTLY the object he wants (a specific file, a specific user, a specific node, etc...) However, almost no real P2P systems are DHTs!

14 Unstructured Query Systems Many users will want to perform a search based on query strings and get results that are close to this query string. A user may wish to make a query which is an intersection of two queries: ( mp3 and SIZE > 3MB). Each object in the system might have many properties, and a query might match any subset of those properties. It is an open problem of how to find a way to map this generality onto DHTs.

15 Broadcast Query Systems In a broadcast query system, each node has some records. To query the network, the node sends a query to ALL neighbors. Each query has an identifying number, responses are routed back the way the query came. To query the entire system, a query will need to cross all edges (E), thus query cost is O(E) and E > N for all connected networks.

16 How do we make scalable query systems? Gnutella is popular protocol for file sharing which uses the unstructured query model. To attempt to solve the scalability problems, they introduced UltraPeers, which are nodes that keep copies of all the records of their LeafPeers. Now, each query costs O(U), if U is the number of UltraPeers. But, if U is a constant fraction of N, then query costs are still O(N), only the constant has changed. Can we do better if we take advantage of network structure?

17 Scale Free Networks Many large networks with interacting nodes, are what is called scale free networks, or power law networks. Many mechanisms have been suggested which can account for such degree distributions. Power law distributions are called scale free because of the following feature: P k = k 1/k P k = k = / k 1/k

18 Power Law Networks: WWW Paper Citations Social Networks File Sharing Networks Networks a. outgoing edge distribution α = 2.41 b. ingoing edge distribution α=2.1 c. diameter of network as function of size. A. Albert, H. Jeong, and A.-L. Barabási, Diameter of the World Wide Web, Nature,401, (1999).

19 Example Parameters from Real Networks (Newman, 2002)

20 Preferential Attachment A simple model which gives rise to a power law degree distribution was proposed by Barabasi, Albert At each time step, a new node joins and selects a node to connect to. The target node is selected with a probability proportional to its degree. The probability we select a node of degree k: Assuming a steady state solution, we want to write a difference equation for the number of nodes with degree k: n k =q k 1 q k k,1 n k = k 1 n k 1 k n k 2 k,1 k 2 n k = k 1 n k 1 k,1 4 n k = k k 1 k 2 1/k 3 q k = k n k 2

21 (Bond) Percolation Problem: If we have a graph and we delete each edge with probability (1 p), as a function of p, what is the size of the largest connected component?

22 Bond Percolation on Random Graphs (with generating functions) Suppose we have a random graph with a constrained degree distribution: p(k). Each node has a degree selected according to this distribution, but its edges are randomly connected. We use a generating function to represent this distribution: P x = k x k p k The mean is the first derivative at x=1: P ' x = k x k 1 k p k P ' 1 = k k p k = k If the random variable Z is the sum of independent random variables: Z = K_1 + K_ K_m, then the generating function is the product: Q x = z x z p z = P x =[P x ] m We can put this together to compute expected cluster sizes!

23 Doing the Calculation Write a consistency equation: Turn it into a generating function: m p C=q 1 m m 1 m l=1 C 1 q 0 k H x =q x m m p m k [ H x ]m 1 Use the generating function to get the mean: m p H ' x =q m m k [ H m m 1 p x m x ]m 1 m [H x ] m 2 H ' x k k k 1 H ' 1 =q 1 H ' 1 k q H ' 1 = k k 1 1 q k Hence we have a threshold on q where size diverges: q k k k 1

24 Details m p C=q 1 m m 1 m l=1 C 1 q 0 k H x =q x m m p m k [ H x ]m 1 m p H ' x =q m m k [H m m 1 p x m x ]m 1 m [ H x ] m 2 H ' x k m p H ' 1 =q m m k [H m m 1 p m 1 ]m 1 m [H 1 ] m 2 H ' 1 k m p H ' 1 =q m m k m m 1 p m m k k k 1 H ' 1 =q 1 H ' 1 k q H ' 1 = k k 1 1 q Threshold: q k k k 1 H ' 1

25 Percolation Thresholds for Example p k = Zeta 3 k 3 k =Zeta 2 /Zeta k 2 =Zeta 1 = q c = k k 2 k =0 p k = Zeta 4 k 4 k =Zeta 3 /Zeta k 2 =Zeta 2 = 2 6 q c = k k 2 k = Graphs What does this mean? p k = Zeta 3.5 k 3.5 k =Zeta 2.5 /Zeta k 2 =Zeta 1.5 =2.61 q c = k k 2 k =0.83 p k = 1 k k = 1 k 2 = q c = k k 2 k = 1 2 We can predict how many edges need to pass a packet to reach a constant fraction of the nodes!

26 Percolation in P2P (due to Nima Sarshar) With probability p we send the query to each neighbor. Each node that gets the query responds with any matches, and sends the query to each of his neighbors with probability p. How small can p be? It must be bigger than q_c!

27 Getting Poly log Scaling in Unstructured Query Systems Assume we have a random network of N nodes, and a degree distribution ~ 1/k^2. There is a maximum degree k_max (which is O(N)). We can get such a network using the protocol from Sarshar, Roychowdhury (PRE 2004) What is the cost of a percolation query at the threshold? C=q c E= q c k N p k = /k 2 2 k = log k max q c = log k max k max log k max C= log k max k max log k max N log k max k 2 = k max Hence we get only O(log^2 N) cost for each query! log 2 k max C= k max / N log k max / N k max =O N C log 2 N

28 Caching Records on High Degree Nodes In fact, in the previous result, we did not explain why sending so few copies of a query would ever reach the node which has a record matching the query. In order to make sure the query finds the record, we cache each record on high degree nodes. Each record is implanted into the network on a random walk path of length Log(N). One can show that it almost surely finds a node of degree k_max/2. One can show that the percolation query almost surely reaches all nodes of degree k_max/2 or greater A Percolation Query almost always finds at least one node from any random walk!

29 Simulation Results A percolation search protocol on a network of size 40,000 with degree distribution ~ 1/k^2 (grown from double preferential attachment).

30 Simulation Results A percolation search protocol on a Gnutella network of size 39,730. The network structure was obtained by Limewire's

31 Brunet: A Hybrid P2P System DHTs cannot resolve general queries. Unstructured systems (usually) require large routing tables to return query hits. Brunet is a new P2P protocol which combines the advantages of both DHTs and Unstructured Power law networks. Brunet offers a general P2P foundation on which a wide variety of protocols and applications can

32 Brunet: A Hybrid P2P System Each node has a 160 bit address which can also be thought of as a 160 bit positive integer. A distance metric using the integer representation. Each node is situated on a routable small world ring with structured connections to its neighbors on the ring, and shortcuts to remote locations. Each node also is on an unstructured network and has unstructured connections to other nodes on that power law network (p_k ~ 1/k^2) Structured Subgraph (small world) Unstructured Subgraph (1/k^2)

33 Brunet Protocol Details Each message on the Brunet system is represented by a packet. The packet has a source address, destination address, hops, time to live, and payload type in the header. The payload can contain arbitrary data. Packets are sent over Edges. An Edge can use any transport (currently we have TcpEdge and UdpEdge). The Address space is partitioned into 160 classes (each half as large as the previous). Some addresses are used for the DHT aspect of the system, some are used for the unstructured system. Some address classes represent unstructured destinations. To send a packet as a percolation message, the address class 126 is used. This class has 32 bits which specify the percolation probability. All Source addresses (currently) are structured addresses. This means nodes do not need to cache routes to particular nodes. Queries can always

34 Stability: How robust is the network to perturbation. We know that random node deletion will not cause problems, but there are many scenarios left to consider (e.g. attack, capacity issues, variance in Practical Issues In practice there are many issues yet to receive full analytic treatment: Firewalls: our protocol connects two nodes as long as one is not firewalled, but we have not analyzed what effects will be manifest as firewalled nodes dramatically increase in number. Bootstrapping: When the network is first forming, there are some interesting problems related to nodes finding their proper place. It is analogous to a distributed sorting algorithm. Convergence: Can we get a proof (including practical issues) that the network properties always converge to the desired properties.

35 Brunet Implementation The first implementation of the Brunet protocol is being completed at UCLA's Complex Networks Group. The code is developed using GNU/Linux and the Mono C# development environment. In addition to a programming library which implements the Brunet protocol, we have developed other tools: Netmodeler: a general C++ network modeling package Brunet Verifier: a protocol debugger for Brunet implementations

36 Open Problems Can the DHT or unstructured systems be used to build an improved model of distributed computing (e.g. how can these P2P models help in mapping task graphs onto resources)? What common primitives can be implemented using P2P systems? (e.g. what kinds of communications costs are incurred building a P2P Database?) What results can be obtained about protocol

37 Summary Using models inspired from social contexts (such as small world and power law networks) we see how some computer networking systems and architectures can be improved. Statistical Mechanics tools (percolation) allow us to analyze some novel networking conditions. By engineering previously ignored structural details of P2P systems, poly log scaling is achieved. The Brunet P2P system puts the DHT model together with the percolation search to get state of the art scaling properties.