Large-scale distributed systems. Anne-Marie Kermarrec

Transcription

1 Large-scale distributed systems Anne-Marie Kermarrec

2 P2P applications Large contributor of Internet traffic (60-70% of Internet Traffic) Applications l File sharing applications (Gnutella, Kazaa, Edonkey, Bit Torrent ) l Archival systems l Application level multicast l Streaming protocols (Coolstreaming, Pplive, etc, Spotify) l Telco applications l P2P currency l Caches l Distributed computations

3 Roadmap 1. Structured Peer to peer overlays 2. Unstructured P2P overlays 3. P2P streaming protocols 4. P2P Content notification 5. Incentive and security

4 Evaluation Individual l Synthesis report: synthesis report on a selection of articles l Talk

5 Introduction

6 Distributed Systems Definition [Tan95] «A collection of independent computers that appears to its users as a single coherent system» Software: Unique image Distributed applications Distributed Systems Hardware: Autonomous computers Local OS Network

7 Why should we decentralize? Economical reasons Availability Resource aggregation Flexibility (load balancing) Privacy Growing need of working collaboratively, sharing and aggregating distributed (geographically) distributed

8 How to decentralize? Connect resources and users Ensure (some form) of distribution transparency Ensure scalability (Security and privacy) Cope with large-scale

9 Peer-to-Peer Systems

10 Historical perspective 1970s s: Birth of the Internet l Limited reach of the Internet l , FTP, Telnet l Share documents and resources between research centers l Central committees to organize and maintain it 1990s l Tremendous expansion & diffusion l l Killer apps: WWW and e-commerce Client/Server model Late 1990s P2P: An alternative to Client/Server l l Passive clients à active peers End-computers play a role, contribute, interact

11 How it all started: NAPSTER June 1999 l l l l Napster is born / 1 st generation of P2P Users not only download content but also provide content to others Users establish a virtual network, entirely independent of physical network and administrative authorities or restrictions Basis: UDP and TCP connections between the peers Architecture Central index server Peers register to a central server Server gives list of files to be shared Server knows all peers and files in the network Server DOS NOT store MP3 files Peers send queries to a central server: index all the file Search request: list ifles with peer infor (encoding rate, size of file, peer s bdt) File transfers between peers

12 Napster Pros l Consistent view of the network l Fast and efficient search l Accuracy Cons l Single point of failure/control l Single point of contention

13 Napster December 1999: RIAA files a lawsuit against Napster Inc. l l TARGET: the central lookup server of Napster ACHIEVEMENT: Napster popularity skyrocketed! February 2001: Peak operation l l 26.4M users 2.79 billion files / month July 2001: Judge orders Napster to pull the plug! l Napster network breaks down instantly l (Napter came back as a legal pay per song system in 2003)

14 How it continued March 2000 l l l l Nullsoft releases Gnutella as an open source project Fully decentralized Additionally to offering files, the peers also take over routing tasks No central lookup server à no single point to attack Later in 2000: Superpeer concept l l l l Hierarchical routing layer Significantly improves scalability and efficiency FastTrack (Morpheus, KaZaA) edonkey l l l l KaZaA loses ground (many defected files due to weak hash keys to identify files) edonkey and Gnutella regain popularity edonkey becomes most popular file-sharing network: 2-3M online users Gnutella v0.6 adopts superpeer architecture (ultrapeers in Gnutella terminology)

15 How it took off! 2002:First version of BitTorrent released 2003 l BitTorrent causes majority of the observed traffic l Downloads significantly faster, due to mechanism against free-riding Middle of 2003 l New P2P concepts develop l Skype is born: a P2P Voice-over-IP application In the meantime: More P2P domains explored! l P2P Routing l Network Storage l P2P Multicasting l Data aggregation l P2P Streaming l etc. Today: Major efforts are made to increase the reliability of P2P systems, to use P2P also in mobile networks, etc

16 Internet Traffic

17 What makes P2P interesting? End-nodes are promoted to active components Nodes participate, interact, contribute to the services they use. Harness huge pools of resources accumulated in millions of end-nodes

18 Refining the definition Resources available at the edges of the Internet are utilized: l l l l l Storage CPU cycles Bandwidth Content Human presence Service is carried out collectively l Nodes share both benefits and duties Irregularities and dynamicity are treated as the norm

19 Main advantages of P2P Inherently scalable: l higher demand à higher contribution Increased (massive) aggregate capacity Utilize otherwise wasted resources Distribute load and administration Designed to be fault tolerant Inherently handle dynamic conditions

20 The core: Overlay Networks B A C Overlay Network Physical Network

21 Definition: overlay End-users communicate among themselves Types of overlay l General-purpose overlay l Application specific overlays Overlay construction l Network topology l Network metrics

22 Overlay Why do we need overlays? l Service not provided at the network level (e.g. ALM) l Application-level service Peer l Computer/end-user/application l Peers are by potentially client and server (leecher vs seed) Peers form an overlay network

23 The P2P paradigm A P2P application forms an overlay network Not all overlays are P2P Why P2P became popular l Internet democratization l High-bandwidth connections l Shift from servers to end users True revolution (TCP/IP WEB)

24 Overlay types Unstructured P2P Centralized Pure P2P Hybrid P2P P2P Structured P2P DHT-Based Central entity necessary to manage the overlay Central entity is some kind of index/group database Napster No central entities Any node can be removed without loss of functionality Gnutella v0.4, Freenet Multiple & Dynamic central entities Any node can be removed without loss of functionality Example: Gnutella v0.6, Freenet No central entities Fixed links, determined by node IDs Any node can be removed without loss of functionality Examples: Chord, Pastry, CAN

25 Overlay types Unstructured P2P Structured P2P Any two nodes can establish a link Topology evolves at random Topology strictly determined by node IDs Topology reflects desired properties of linked nodes

26 Main Issues in P2P: Overlay maintenance Bootstrapping l how to join the system Continuous maintenance l how to handle changes, faults, etc

27 Main Issues in P2P: Scalability Avoid central server Distribute load on multiple peers Limit load per peer Computing Messaging Storage State

28 Main Issues in P2P: Fairness vs load balancing Load balancing Distribute load among peers, but how? l Evenly? l Proportionally to node capacity? l? User behavior l Users are selfish and independent (maximize own benefit) l Incentives for fair play

29 Main Issues in P2P: Dynamicity and adaptability Changing topology l nodes join and leave: node churn l network partitions Changing data l content is changed l files are added / deleted Changing profiles l users change interests l new semantic categories introduced Change in load l load rebalancing

30 Main Issues in P2P: Fault tolerance Robustness of the overlay Resilience to failures Resistance to node & link crashes Availability

31 Main Issues in P2P: Performance Efficiency l in searching l in routing steps l in discovering relationships l etc. Locality l reduce network latency

32 P2P Application Areas CPU Internet/Intranet Distributed Computing Grid Computing Content File sharing Information Mgmt Discover Collaboration Aggregate Filter Instant Messaging Shared whiteboard Co-review/edit/author Gaming Storage Network Storage Caching Replication Bandwidth Content Distribution Collaborative download Edge Services VoIP

33 Structured P2P overlay networks

34 Outline Distrivuted Hash Table Structured P2P l DHT l Interface PASTRY CAN

35 1-P2P routing infrastructure Overlay: network abstraction on top of IP Basic functionality: distributed hash table Applications l Content-delivery networks l Storage systems, Caching l Naming services l Multicast

36 Distributed Hash Table (DHT) Data d -> h(d) = key Nodes= users -> labels Routing: Search for keys Correspondance between keys and nodes

37 Distributed Hash Table (DHT) containers Operations: insert(k,v) lookup(k,v) k1,v1 k4,v4 k2,v2 k3,v3 Table of containers Store <key,value> pairs Efficient access to a value given a key k5,v5 k6,v6 Mapping key-value ensured by the table of containers

38 Distributed Hash Table Operations: send(m,k) P2P overlay network k1,v1 k4,v4 k2,v2 nodes k3,v3 k5,v5 k6,v6 Message sent to keys: implementation of a DHT P2P Infrastructure ensures mapping between keys and physical nodes Fully decentralized: peer to peer communication paradigm

39 Mapping key nodeid Identifier Space An identifier is associated to each node (nodeid) Large identifier space (keys and nodeid) A node is responsible for closest key to its nodeid

40 Sending messages to keys keys nodeid destination source Each node maintains a routing table (nodeid, IP adress) At each routing hop, the message gets closer to destination Infrastructure P2P: mapping between key and node

41 Correspondence Node x Data stored by x

42 Distributed Hash Table (K 1,V 1 ) K V K V K V K V K V K V K V insert (K 1,V 1 ) K V K V K V retrieve (K 1 )

43 Routing Interface Route (key,node,msg) Send (node,msg) NextHopSet = lookup (key, num) Upcalls l Deliver(key,msg) l Forward (key,msg, NexHopNode) l Update (neighbour set)

44 Routing Interface Route (key, node, msg) Route the msg to the node (alive) responsible of the key. Non blocking, (best effort), the message may be lost or duplicated The node parameter is optional and is interpreted as the first hop (using a good heuristic might deliver the message in one single hop) msg: application-specific key: numerical key (160 bits) node: encapsulates of the last node known associated to that key

45 Routing Interface nexthopset = lookup (key,num) Returns a set of nodes (num) from the routing table as potential next steps for the routing key: numerical 160 bit key num: number of nodes required in the set Result: the resulting set of nodes

46 Routing Interface Send(node,msg): sends a message to a specific node; fails if the node is not available, blocking and reliable l Msg: msg is application-specific l Node: encapsulate of the destination node l Result: true if the msg is delivered false otherwise

47 Routing Interface Deliver (msg,key) Upcall delivering the message to application. Assumes one application per node msg: message application-specific key: numerical 160 bit key

48 Routing Interface Forward (msg,key,nexthopnode) upcall invoked at each node on the message route (source and destination included) to inform an application that the msg and the key are about to be forwarded to NextHopNode. The application can modify msg, key, nexthopnode or terminate the message by setting nexthopnode to null Msg: application specific msg Key: numerical 160 bit key of the node to which the message is forwarded after the upcall has returned

49 Routing Interface Update(neighborSet): upcall invoked to inform the application of a change in the neighborhood of the local node. Neigborset: set of k nodes

50 Pastry (MSR/RICE) node key Id space NodeId = 128 bits Nodes and key place in a linear space (ring) Mapping : a key is associated to the node with the numerically closest nodeid to the key

51 Pastry (MSR/Rice) Naming space : l Ring of 128 bit integers l nodeids chosen at random Key/node mapping l key associated to the node with the numerically closest node id Routing table: l Identifiers are a set of digits in base 16 l Matrix of 128/4 lines et 16 columns l routetable(i,j): nodeid matching the current node identifier up to level I with the next digit is j Leaf set l 8 or 16 closest numerical neighbours in the naming space

52 Pastry: Routing table(#65a1fcx) 0 x 1 x 2 x 3 x 4 x 5 x 7 x 8 x 9 x a x b x c x d x e x f x 6 0 x 6 1 x 6 2 x 6 3 x 6 4 x 6 6 x 6 7 x 6 8 x 6 9 x 6 a x 6 b x 6 c x 6 d x 6 e x 6 f x x x x x x x x x x x 6 5 b x 6 5 c x 6 5 d x 6 5 e x 6 5 f x 6 5 a 0 x 6 5 a 2 x 6 5 a 3 x 6 5 a 4 x 6 5 a 5 x 6 5 a 6 x 6 5 a 7 x 6 5 a 8 x 6 5 a 9 x 6 5 a a x 6 5 a b x 6 5 a c x 6 5 a d x 6 5 a e x 6 5 a f x log 16 N liges Line 0 Line 1 Line 2 Line 3

53 Pastry: Routing Route(d46a1c) d46a1c d471f1 d467c4 d462ba d4213f d13da3 Properties log 16 N hops Size of the state maintained (routing table): O(log N) 65a1fc

54 Routing algorithm, notations R line l, 0 l L i l i D l :entry of the routing table R, 0 i 128/b :ith closest nodeid in the leafset : value of the l digits of key D SHL( A, B) :length of the shared prefix between A and B 2 b,

55 Routing algorithm (on node A) if ( L D L 2 L / 2 L / // D belongs to the spacecovered by the leafset forward to L, so that D-L else{ //use routing table entries Let l = SHL(D,A) ; if ( R D l i ){ null ) forward to R l D l else{ // hole in the routing table forward to T { L R}so that SHL(T,D) l T-D < A-D } } i minimal} l ;}

56 Node departure Explicit departure or failure Replacement of a node The leafset of the closest node in the leafset contains the closest new node, not yet in the leafset Update from the leafset information Update the application

57 Failure detection Detected when immediate neighbours in the name space (leafset) can no longer communicate Detected when a contact fails during the routing Routing uses an alternative route

58 Fixing the routing table of A Repair R d l A contacts another entry (at random) so that ( i from :entry of the routing table of A to repair R i l+ d) and asks for entry R d l 1 ( i d) if no node in line l R i l from the same line, otherwise another entry answers the request

59 State maintenance Leaf set l is aggressively monitored and fixed Routing table l are lazily repaired When a hole is detected during the routing l Periodic gossip-based maintenance

60 Reducing latency Random assignment of nodeid: Nodes numerically close are geographically (topologically) distant Objective: fill the routing table with nodes so that routing hops are as short (latency wise) as possible Topological Metric: latency d467c4 6fdacd d467f

61 Exploiting locality in Pastry Neighbour selected based of a network proximity metric: l Closest topological node l Satisfying the constraints of the routing table routetable (i,j): nodeid corresponding to the current nodeid courant up to level i next digit = j l nodes are close at the top level of the routing table l random nodes at the bottom levels of the routing tables

62 Proximity routing in Pastry Leaf set d467c4 d46a1c Route(d46a1c) d471f1 d467c4 d462ba d4213f d13da3 Topological space 65a1fc Naming space d462ba d4213f 65a1fc d13da

63 Locality Node X routes to node A l Path A,B, -> Z l Z numerically closest to X l Initialisation of the line i of the routing table with the contents of line i of the routing table of the ith node encountered on the path Improving the quality of the routing table l X asks to each node of its routing table its own routing state and compare distances l Gossip-based update for each line (20mn) Periodically, an entry is chosen at random in the routing table Corresponding line of this entry sent Evaluation of potential candidates Replacement of better candidates New nodes gradually integrated

64 Node insertion in Pastry d46a1c d471f1 d467c4 d462ba d4213f d467c4 Topological space Route(d46a1c) d13da3 65a1fc New node: d46a1c Naming space d462ba d4213f 65a1fc d13da

65 Node discovery: approximation of the closest node discover(seed) //destination nodes=getleafset(seed) //probes on the leaf set nearnode=pickclosest(nodes) //pick closest in the leafset depth=getmaxroutingtablelevel(nearnode) //highest line in RT closest=nil while (closest!= nearnode) closest=nearnode nodes = getroutingtable(nearnode, depth) nearnode =pickclosest(nodes) if (depth >0) depth = depth 1 end return closest Routing table property used to move exponentially close to the closest Pick the closest node at each level and get the the next level from it (bottom up) l Constant number of probes at each level l Probed nodes get exponentially closer at each level Fill routing table entries from the closest node

66 Performance 1.59 slower than IP on average Normal Routing Tables Perfect Routing Tables No locality Per-hop distance Hop Number

67 Content Adressable network (CAN)

68 Content Adressable Network -CAN UCB/ICIR Virtual coordinates Cartesian space The space is shared between peers l Each node is responsible for a part of the space (zone) Abstraction l CAN enables to store data at a given point in the space l CAN enable the routing from a point of the space to the other (DHT functionality) l A point is associated to the node owning the zone in which the point lies

69 Space organisation in CAN node Key Name space D-dimension space Routing: progression within the space towards The destination

70 CAN: Example

71 CAN: exemple

72 CAN: routing node X ::insert(k,v) (1) a = h x (K) b = h y (K) y = b (a,b) (2) route(k,v) -> (a,b) (3) (a,b) stores (K,V) (x,y) x = a

73 CAN: node insertion N (1) Bootstrap : discovery of a contact node already participating to the CAN overlay network (2) Selection of a random point (p,q) in the space (3) Routing to (p,q) and discovery of node Y (4) Zone splitting between Y and N (x,y) (p,q) Y Insertion affects only Y and its immediate neighbours

74 Routing information The joining node gets the of its neighbours from the previous owner of the zone Set of neighbours of the joining node is a sub-set of neighbours of the previous owner The previous owner updates its own list of neighbours The neighbours of the joining node should also be updated

75 Departure, arrival, maintenance Node departure l The leaving node L must make sure that the zone is taken over l A leaving node hands over explicitly its own zone (and associated database) to one of its neighbours l Failure: detected by periodic messages from a node to its neighbours (hearteat) containing its own coordinates and coordinates of its neighbours

76 CAN: properties Each node maintains pointers to its immediate neighbours = 2d O(d) Routing in a N node network l Number of hops in a d-dimension space ( l In case of failure: selection of an alternative neighbour Optimizations l Multiple dimensions l Multiple reality l RTT Measures l Zone splitting l Locality awareness: landmarks O d( n 1/ d ))

77 Failure resilience destination source

78 Failure resilience

79 Failure resilience destination

80 Failure resilience

81 Routing resilience

82 Failure resilience Node X::route(D) If (X cannot progress directly towards D) l Check if one neighbour can progress towards the destination l If so, forward the message

83 Multiple dimensions Increasing the number of dimensions l The average path length is improved l The number of neighbours increases linearly with the dimension l Enhanced availability: potentially more nodes available

84 Reality Multiple independent coordinate spaces Each node is associated to a different zone in each space (reality): r sets of coordinates Enhanced availability l DHT content can be replicated across realities Ex: a pointer to a file stored at (x,y,z) is stored on three nodes responsible of point (x,y,z) in 3 realities l Improves average path length as well: depending on the destination, the most relevant reality is chosen

85 RTT measures So far, the metric used to progress in the space in the path length in the Cartesian space Better criterion to take into account the underlying topology RTT to each neighbour Message forwarded to the neighbour for which the ratio progress/rtt is the best l Avoid long hops

86 Summary on structured overlay networks Chord, Pastry and Tapestry use a generalized hypercube routing: prefix matching l State maintained: O(Log(N)) l Number of routing hops: O(Log(N)) l Proximity routing in Pastry and Tapestry CAN uses progression in a multidimensional Cartesian space l State maintained: O(D) l Number of routing hops: O(N 1/D ) l Proximity routing more difficult to exploit DHT Functionality=Exact match interface

87 References A. Rowstron and P. Druschel, "Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems", Middleware'2001, Germany, November Scalable Content-Addressable Network (SIGCOMM 2001) Sylvia Ratnasamy Paul Francis Mark Handley Richard Karp Scott Shenker Many more: Google P2P structured overlay networks

88 Application-level multicast on top of structured P2P networks

89 Group Communication Common and useful communication paradigm Disseminating information within a group sharing interest l Consistency of replicated data l Publish/Subscribe systems Studied a lot in local area networks l Group management (join, leave, send) More scalability needed l Application-level multicast (for medium-size groups) not scalable l Network-level multicast not fully deployed

90 Group communication Important functionality of distributed systems l l l l Failure detection Membership management Coherence management Event notification systems Reliability is crucial Scalability l l Group size Geographical spreading Source Broadcast Protocol Group of processes (nodes)

91 Broadcast protocols A Centralized versus decentralized protocols l l Load balancing Performance A C D B C Evaluation metrics B l l l Delay from the source to the destination Network traffic Node load G F E D E l Failure resilience F G

92 ALM on P2P structured overlays Scalability l O(logN) hops routing with a O(logN) state l Load balancing Self-* properties (organizing, healing, ) l P2P overlay network automatically repaired upon peer joins and departures l Automatic load re-distribution Attractive support for large-scale application-level multicast

93 ALM on structured overlay networks Overlay network used for group naming and group localization Flooding-based multicast [CAN multicast]: l Creation of a specific network for each group l Message flooded along the overlay links Tree-based multicast [Bayeux, Scribe] l Creation of a tree per group l Flooding along the tree branches

94 Flooding-based multicast Group members join the network associated to a given group Messages sent over all links of the P2P overlay Specific mechanism to get rid of duplication

95 Flooding in Pastry-like systems Example l on receiving <flood, m, i> l for each routing table row i (i greater than i) send <flood, m, i > to nodes in row l i=0 for original message sender

96 Tree-based multicast Creation of a tree per group l The tree root is the peer hosting the key associated to that group l The tree is formed as the union of routes from every member to the root id space

97 Scribe Support multiple groups on a p2p prefixmatching infrastructure (Pastry, Tapestry, ) Support various applications (size-wise) on a single infrastructure potentially l Instant Messaging l Information dissemination (stock alerts) l Diffusion lists (Windows updates) Properties l Scalability l Efficient: low latency, low network link stress, low node load l Reliability: application-specific

98 Scribe Broadcast protocol Membership management SCRIBE P2P Infrastructure PASTRY Internet TCP/IP

99 Scribe: design Goals l Group creation l Membership maintenance l Messages dissemination within a group Construction of a multicast tree on top of a Pastry-like infrastructure l Reverse path forwarding l Messages flooded along the tree branches

100 Scribe Interface Create(group) Join(group) Leave(group) Multicast(group,m) The P2P infrastructure is used for group creation and join protocol

101 Scribe: group creation #G Create(#G) Root Each group is assigned an identifier groupid = Hash (name) Multicast tree root : node which nodeid is the numerically closest to the groupid Create(group): P2P routing using the groupeid as the key

102 Scribe: tree creation join(group) : message sent through Pastry using groupeid as the key Multicast tree : union of Pastry routes from the root to each group l Low latency: leverage Pastry proximity routing l Low network link stress: most packets are replicated low in the tree

103 Scribe : join(group)

104 Scribe: message dissemination Multicast(group, m) Routing through Pastry to the root key=groupeid E 1100 Flooding along the tree branches from the root to the leaves

105 Reliability «best effort» reliability guarantee l Tree maintenance when failures are detected l Stronger guarantee may also be implemented Node failure l Parents periodically send heartbeat messages to their descendants in the tree l When such messages are missed, nodes join the group again Local reconfiguration Pastry routes around failures

106 Tree maintenance Root

107 Tree maintenance New root 1111 Faulty root

108 Load balancing Specific algorithm to limit the load on each node l Size of forwarding tables Specific algorithm to remove the forwarders-only peers from the tree l small-size groups

109 Scribe performance Discrete event simulator Evaluation metrics l Relative delay penalty l l RMD: max delay app-mcast / max delay ip-mcast RAD: avg delay app-mcast / avg delay ip-mcast Stress on each network link Load on each node Number of forwarding tables Number of entries in the forwarding tables Experimental set-up l Georgia Tech Transit-stub model (5050 core routers) l nodes chosen at random among l Zipf distribution for 1500 groups l Bandwidth not modeled

110 Group distribution Windows Update Group size Stock Alert Instant Messaging Group rank

111 Delay/IP CDF of Groups RMD RAD 300 Mean = 1.81 Median = Delay penalty

112 Load balancing Number of nodes Number of forwarding tables

113 Load balancing Number of nodes Number of nodes Total number of entries in forwarding tables Total number of entries in forwarding tables

114 Network load Number of network links Scribe IP Multicast Maximum Stress

115 Summary Generic P2P infrastructures l Good support for large-scale distributed applications l ALM Infrastructure Scribe exhibits good performances/ip multicast l Large size groups l Large number of groups l Good load-balancing properties

116 CAN Multicast Flooding in a CAN network l Either l Or All CAN members are group members Mini CAN overlay creation/groupe

117 CAN multicast: group formation Subset of CAN network members forms a mini-can Group identifier associated to a point (x,y) in the CAN space. (x,y) is the bootstrap node for the mini-can Group join = mini-can join Iterations on the CAN join protocol

118 CAN multicast : message diffusion CAN network of dimension d: 1.d Each node maintains at least 2d neighbours Diffusion l Source node sends the message to all its neighbours l A node receiving a message from dimension i Forwards the message to its neighbours along the dimensions 1 (i-1) Forwards the message to neighbours of dimension 1 in in the opposite direction (from the one it receives the message) l A node does not forward the message along a given dimension if the message has already traversed half of that dimension l A node does not forward an already received message

119 Example

120 Can multicast : Performance CAN: 6 dimensions, group of 8192 nodes, transit-stub topology Relative delay penalty (RDP) l 5-6 for the majority of group members More details in the comparison

121 Comparison: delay penalty/ip RAD trees flooding Pastry CAN

122 Comparison: average (physical) link stress 3 link stress for multicast trees flooding link stress for joining: identical for trees much larger for flooding example: 281 on CAN 0 Pastry CAN

123 Trees versus flooding Tree-based multicast is more efficient l Lower delay and network stress during the multicast l Huge difference in the network trafic during group creation l Main drawback: some peers may be forwarders-only

124 Conclusion ALM can be sucessfully implemented over structured P2P networks