(C) Chukiat Worasucheep 1

Transcription

1 Slides for Chapter 10: Peer-to-Peer Systems From Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edition 4, Pearson Education 2005 Contents Motivations & characteristics P2P Applications Napster, Gnutella, Kazaa BitTorrent, Skype, MMOG P2P Architecture P2P Middleware Routing overlays Summary Distributed Information Systems Page 1 Distributed Information Systems Page 2 P2P Motivation Motivations Image courtesy of Traditional client-server systems provide access to resources located on a single server computer or a small cluster of tightly-coupled servers. Expanding service on a large scale is limited when all hosts must be owned and managed by service provider. Scale of service is limited by server hardware capcbility and network connectivity. With centralized design, few decisions are required about the placement and subsequent retrieval of the resources. Peer-to-peer systems provide access to computing resources (storage, cycles, content) available in personal computers and workstations on the Internet. The design aims to deliver service that is fully decentralized, selforganizing, and dynamically balancing storage and processing loads. Distributed Information Systems Page 3 Distributed Information Systems Page 4 (C) Chukiat Worasucheep 1

2 Characteristics (1/2) Each user contributes resources to the system. Although the contributed resources may differ, all the nodes have the same functional capabilities and responsibilities. System operation does not depend on any centrally administrative system. Characteristics (2/2) System operation may offer limited degree of anonymity to providers and users of resources. Efficiency relies on the algorithm for placement of data across many hosts and subsequent access in a manner that balances workload. Access individual resource cannot be guarantees as availability of the processes and computers is unpredictable. But the system can be designed to make probability of failure small. Distributed Information Systems Page 5 Distributed Information Systems Page 6 Contents Motivations & characteristics P2P Applications Napster, Gnutella, Kazaa BitTorrent, Skype, MMOG P2P Architecture P2P Middleware Routing overlays Summary Peer-to-Peer Applications Deployment of peer-to-peer service emerged when users had acquired always-on, broadband connections to the network, making their computers suitable platforms for resource sharing. P2P File Sharing the most popular application of P2P systems. Made P2P popular Napster music exchange service Gnutella, BitTorrent Distributed Information Systems Page 7 Distributed Information Systems Page 8 (C) Chukiat Worasucheep 2

3 Napster music exchange service Began in MP3, Copyright infringement. Napster Architecture Concept Based on central index server (farm) User register and give list of files to share Searching based on keywords Results: List of files with additional information, e.g. peer s bandwidth, encoding rate, file size. on Mac OS 9 Distributed Information Systems Page 9 Distributed Information Systems Page 10 Napster Architecture Diagram peers Napster server Index 1. File location request 2. List of peers offering the file 5. Index update 3. File request 4. File delivered Figure 10.2 Napster server Index Distributed Information Systems Page 11 Napster Architecture Insight Napster considered network locality the number of hops between client and server when allocating a server to a clients. Characteristics of music application simplified the design: Requirement on consistency among replicas was not strong so this did not hamper performance. Replicas of the unified index of all music files Replicas of the music files; files are never updated Availability of files was not a serious concern; files could be downloaded again later. Knowledge of the locations of resources should be partitioned and distributed throughout the network. Each node maintains location information in its responsibility and the knowledge of the topology of the entire resources name space. Distributed Information Systems Page 12 (C) Chukiat Worasucheep 3

4 Napster Termination Napster was shut down because of the infringement of the copyright ownership. Although copying process was performed entirely between users machines, the index servers were an essential part of the process and at a well-known address. Their operators were unable to remain anonymous and then targeted in lawsuits. Anonymity for the receivers and providers is a legitimate concern for designs of peer-to-peer systems to be able to deny responsibility for holding or supplying resources. Napster Strengths and Weaknesses Strengths Fast and efficient overall search Consistent view of network Weaknesses Central server is single point of failure expensive to maintain server Distributed Information Systems Page 13 Distributed Information Systems Page 14 Gnuttela (2000+, peak in ) Gnuttela Addresses some of Napster s weaknesses and introduces its own problems Originally developed at Nullsoft (AOL) Accidentally released on their website Open protocol specifications Idea Share any types of files (not just music) You ask your neighbors for files of interest. Neighbors ask their neighbors, and so on. TTL field quenches messages after a number of hops Users with matching files reply to you Distributed Information Systems Page 15 Distributed Information Systems Page 16 (C) Chukiat Worasucheep 4

5 Gnuttela - Concept Gnuttela - Architecture Image courtesy of Fully distributed Based on an overlay network All peers are fully equal A peer needs to know another peer, that is already in the network, to join Searching based on flooding Direct downloads Distributed Information Systems Page 17 Distributed Information Systems Page 18 Gnuttela Diagram Gnuttela Strengths and Weaknesses Image courtesy of SlideShare Decentralized No single point of failure Not as susceptible to denial of service Cannot ensure correct results Flooding queries Search is now distributed but still not scalable Distributed Information Systems Page 19 Distributed Information Systems Page 20 (C) Chukiat Worasucheep 5

6 Kazaa A music file sharing application, , based on FastTrack protocols. ( 2nd generation P2P Hybrid of centralized Napster and decentralized Gnutella Super-nodes act as local search hubs Each super-node is similar to a Napster server for a small portion of the network Super-nodes are automatically chosen by the system based on their capacities (storage, bandwidth, etc.) and availability (connection time) Super-peers periodically exchange file lists Kazaa FastTrack Protocol Each installed Kazaa client comes coded with a list of supernodes. Every time the Kazaa application is launched, the client computer registers with the central server, and then chooses from a list of currently active supernodes. The client request is funneled through the supernode. The supernode communicates with other supernodes, which in turn connect to regular nodes that in turn connect to even more regular nodes, to fulfill the request until the Time to Live (TTL) of 7 runs out. Once the correct file has been located, it is transferred directly from the file owner to the requester using HTTP. It doesn't have to go through a supernode. Distributed Information Systems Page 21 Distributed Information Systems Page 22 Kazaa UI Kazaa Strenghts and Weaknesses Strength Combined good points of Napster and Gnutella Weakness Still no overall search Security? Distributed Information Systems Page 23 Distributed Information Systems Page 24 (C) Chukiat Worasucheep 6

7 Contents Motivations & characteristics P2P Applications Napster, Gnutella, Kazaa BitTorrent, Skype, MMOG P2P Architecture BitTorrent Developed by Bram Cohen in 2001 Goal: Quickly replicate one file to a large number of clients New approach for sharing large files More appropriately called P2P content distribution instead of filesharing P2P Middleware Routing overlays Summary Distributed Information Systems Page 25 Distributed Information Systems Page 26 How BitTorrent Works How BitTorrent Works Based on decentralized network Files are divided into pieces or blocks Pieces can be of size from 64 kb to 4 MB Build a network for each file to distribute A torrent file metadata about files and folders to be distributed, and usually also a list of the network locations of trackers. Trackers are computers that help participants find each other and form efficient distribution groups called swarms. Distributed Information Systems Page 27 Distributed Information Systems Page 28 (C) Chukiat Worasucheep 7

8 BitTorrent Download Client gets torrent file from somewhere Client contacts tracker, gets lists of peers Client contacts other peers directly for download of specific chunks Tit-for-tat principle Encourages cooperation, discourages free-riding A peer serves peers that serve it Peers download rarest chunks first makes them attractive to other peers, they can then download in return also prevents chunks from disappearing BitTorrent Strengths and Weaknesses Strengths Works quite well, only slow in the beginning Efficient distribution mechanism Enforces contribution, discourages free-riding Weaknesses No searching Files have to be large Everyone must contribute Distributed Information Systems Page 29 Distributed Information Systems Page 30 Skype Developed by the same people as KaZaA > 600 Million users in Million concurrent users (in 2012) Bought by Microsoft in 2011 for $8.5 billions Very similar to KaZaA s architecture Supernodes and ordinary nodes Central server for login and billing Proprietary and encrypted protocol Source code is closed Executable uses a lot of mechanisms to protect against reverse engineering Anti-debugger code Partly encrypted Skype How Skype goes through Firewalls Skype server plays mediator takes IP addresses and UDP ports forwards them to other side Both clients start connections Firewalls think that packets from the other side are responses to started connections Distributed Information Systems Page 31 Distributed Information Systems Page 32 (C) Chukiat Worasucheep 8

9 Joost Brought to you by the guys who made KaZaA and Skype, in 2007 Uses P2P video streaming technology Tries to merge the best of TV with the best of the Internet Advertisement based, View on demand Beta test was running 2012, service is suspended! Massively multiplayer online game (MMO or MMOG) Rogue on Unix on PDP11 machine, 1980 Distributed Information Systems Page 33 Distributed Information Systems Page 34 Massively multiplayer online game (MMO or MMOG) P2P Support for Multiplayer online game On the design of multiplayer video game MMORPG Contents Motivations & characteristics P2P Applications Napster, Gnutella, Kazaa BitTorrent, Skype, MMOG P2P Architecture P2P Middleware Routing overlays Summary Distributed Information Systems Page 35 Distributed Information Systems Page 36 (C) Chukiat Worasucheep 9

10 Architecture of P2P P2P Architecture based on file listing P2P Architecture Centralized (Hybrid) File listing exists on central server All peers connect to this central server Examples: Napster, ICQ Based on file listing Centralized Decentralized (hybrid) Based on node connection Unstructured Structured Decentralized No Central Server Each node connected to many other nodes May be pure Peer-to-Peer or with Super Peers Examples: Gnutella, BitTorrent Distributed Information Systems Page 37 Distributed Information Systems Page 38 P2P Architecture based on node connection Unstructured Nodes created arbitrarily Each node copies link information from other connected nodes Simple, however not efficient Examples: FastTrack, Gnutella Structured Implements algorithms to ensure connection between nodes. Scalable guarantees on numbers of hops to answer a query. Maintain all other P2P properties (load balance, self-organization, dynamic nature) Efficient, however complex algorithms involved Examples: DHT, Chord, CAN, Tapestry, and Pastry model. Contents Motivations & characteristics P2P Applications Napster, Gnutella, Kazaa BitTorrent, Skype, MMOG P2P Architecture P2P Middleware Routing overlays Summary Distributed Information Systems Page 39 Distributed Information Systems Page 40 (C) Chukiat Worasucheep 10

11 is a class of a decentralized distributed lookup service similar to a hash table. (key, value) pairs are stored in a DHT, and any participating node can efficiently retrieve the value associated with a given key. The mapping from keys to values is distributed among the nodes, in such a way that a change in the set of participants causes a minimal amount of disruption. Highly scalable to extremely large numbers of nodes. Able to handle continual node arrivals, departures, and failures. Courtesy of wikipedia Distributed Information Systems Page 41 Distributed Information Systems Page 42 DHT Properties Keys should mapped evenly to all nodes in the network (load balance) Each node should maintain information about only a few other nodes (scalability, low update cost) Messages should be routed to a node efficiently (small number of hops) Node arrival/departures should only affect a few nodes DHT Implementation DHT is a generic interface There are several implementations of this interface Chord [MIT], 2001 Pastry [Microsoft Research UK, Rice University], 2001 Tapestry [UC Berkeley], 2003 Content Addressable Network (CAN) [UC Berkeley] SkipNet [Microsoft Research US, Univ. of Washington] Kademlia [New York University] Viceroy [Israel, UC Berkeley] P-Grid [EPFL Switzerland] Distributed Information Systems Page 43 Distributed Information Systems Page 44 (C) Chukiat Worasucheep 11

12 Looking up data in P2P systems (MIT Lab, Balakrishnan et al., 2003) Looking up data in P2P systems (MIT Lab, Balakrishnan et al., 2003) Distributed Information Systems Page 45 Distributed Information Systems Page 46 Contents Motivations & characteristics P2P Applications Napster, Gnutella, Kazaa BitTorrent, Skype, MMOG P2P Architecture P2P Middleware Routing overlays Peer-to-peer middleware (1/4) Application-independent management Automatic placement of resources (data, objects, files) on computers Holding information about locations of resources Routing of messages to computers Taking account of the availability of computers, their trustworthiness, and requirements for load balancing and locality of information storage and use Summary Distributed Information Systems Page 47 Distributed Information Systems Page 48 (C) Chukiat Worasucheep 12

13 Peer-to-peer middleware (2/4) A key problem in the design of P2P applications is to provide a mechanism to enable clients to access resources quickly and dependably wherever they are in the network. Napster provided a unified index of available files 2 nd generation P2P file storage systems, such Gnutella and Freenet employ partitioned and distributed indexes, but the algorithms used are specific to each system. Peer-to-peer middleware (3/4) P2P middleware are designed for Functional requirements Simplify the construction of services across many hosts enabling clients to locate and access resources even though the resources are widely distributed amongst the hosts. Ability to add new resources and to remove them at will as well as add and remove hosts to the service. Non-functional requirements (Kubiatowicz 2003) (next page ) Distributed Information Systems Page 49 Distributed Information Systems Page 50 Peer-to-peer middleware (4/4) Functional requirements (previous page) Non-functional requirements (Kubiatowicz 2003) Global scalable access millions of objects on hundreds of thousands of hosts. Load balancing with use of replicas of heavily-used resources. Optimization for local interactions between neighboring peers to reduce impacts from the latency due to network distance between nodes. Accommodating to highly dynamic host availability most P2P allows hosts to freely join or leave the system. Security of data in an environment with heterogeneous trust. Anonymity, deniability and resistance to censorship Contents Motivations & characteristics P2P Applications Napster, Gnutella, Kazaa BitTorrent, Skype, MMOG P2P Architecture P2P Middleware Routing overlays Summary Distributed Information Systems Page 51 Distributed Information Systems Page 52 (C) Chukiat Worasucheep 13

14 Routing overlays (1/2) The development of P2P middleware has been an active research topic since 1997 (Plaxton, Rajaraman & Richa 1997). A routing overlay takes responsibility for locating nodes and objects. Any node access any object by routing each request through a sequence of nodes, exploiting knowledge of at each of them to locate the destination object. The overlay maintains knowledge of the locations of all the available replicas and delivers requests to the nearest live node (i.e. one that has not failed) that has a copy of the relevant object. It s called overlay since it implements a routing mechanism in the application layer, which is over the IP layer Routing overlays (2/2) GUID is used to identified each object in the overlay. GUIDs are also known as opaque identifiers, since they reveal nothing about the locations of the objects to which hey refer. Routing overlays take responsibilities for: A client wishing to invoke an operation on an object submits a request including the object s GUID to the routing overlay, which routes the request to a node at which a replica of the object resides. A node wishing to make a new object available to a peer-to-peer service computes a GUID for the object and announces it to the routing overlay, which then ensures that the object is reachable by all other clients. Make the removed objects (by clients) unavailable. Nodes may join and leave the services. Distributed Information Systems Page 53 Distributed Information Systems Page 54 Figure 10.3: Distribution of information in a routing overlay Some routing overlays A s routing knowledge D s routing knowledge Pastry (Rowstron and Druschel 2001, Castro et al. 2002) is a message routing infrastructure, based on distributed hash table (DHT), and deployed in several applications e.g. C PAST (2001) an immutable file storage system implemented as a distributed hash table (DHT) A D Squirrel (Iyer et al. 2002) a web caching service on local networks of PCs Tapestry (Dabek et al. 2003, Zhao et al. 2004) conceals DHT behind Distributed Object Location and Routing (DOLR) Object: Node: B B s routing knowledge C s routing knowledge OceanStore (Kubiatowicz et al. 2000) a very large scale, incrementally-scaled persistent storage facility for mutable data objects with long term persistence and reliability in a constantly changing network and resources Ivy file system (Muthitachareon 2002) similar to OceanStore, but emulates a Sun NFS srver Distributed Information Systems Page 55 Distributed Information Systems Page 56 (C) Chukiat Worasucheep 14

15 Figure 10.4: Basic programming interface for a distributed hash table (DHT) as implemented by the API over Pastry GUID is generated from the object s values using a hash function (e.g. SHA-1). Thus the overlay is sometimes called distributed hash tables (DHT). Basic APIs to access objects are as follows. put(guid, data) The data is stored in replicas at all nodes responsible for the object identified by GUID. remove(guid) Deletes all references to GUID and the associated data. value = get(guid) The data associated with GUID is retrieved from one of the nodes responsible it. Figure 10.5: Basic programming interface for distributed object location and routing (DOLR) as implemented by Tapestry publish(guid ) GUID can be computed from the object (or some part of it, e.g. its name). This function makes the node performing a publish operation the host for the object corresponding to GUID. unpublish(guid) Makes the object corresponding to GUID inaccessible. sendtoobj(msg, GUID, [n]) Following the object-oriented paradigm, an invocation message is sent to an object in order to access it. This might be a request to open a TCP connection for data transfer or to return a message containing all or part of the object s state. The final optional parameter [n], if present, requests the delivery of the same message to n replicas of the object. Figure 10.9: Pastry s routing algorithm Figure 10.10: Tapestry routing From [Zhao et al. 2004] To handle a message M addressed to a node D (where R[p,i] is the element at column i, row p of the routing table): 1. If (L -l < D < L l ) { // the destination is within the leaf set or is the current node 2. Forward M to the element L i of the leaf set with GUID closest D to or the current node A. 3. } else { // use the routing table to despatch M to a node with a closer GUID 4. find p, the length of the longest common prefix D of and A. and i, the (p+1) th hexadecimal digit of D. 5. If (R[p,i] null) forward M to R[p,i] // route M to a node with a longer common prefix. 6. else { // there is no entry in the routing table 7. Forward M to any node in L or R with a common prefix of length i, but a GUID that is numerically closer. } } Instructor s Guide for Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edn. 4 Pearson Education 2005 Tapestry routings for 4377 publish path Location mapping for 4378 Routes actually taken by send(4378) Phil s Books E FE 4B4F EC 4377 (Root for 4378) 4A6D 437A AA Phil s Books Replicas of the file Phil s Books (G=4378) are hosted at nodes 4228 and AA93. Node 4377 is the root nod for object The Tapestry routings shown are some of the entries in routing tables. The publish path routes followed by the publish messages laying down cached location mappings for object The loc mappings are subsequently used to route messages sent to Instructor s Guide for Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edn. 4 Pearson Education 2005 (C) Chukiat Worasucheep 15

16 Figure 10.14: Ivy system architecture Applications of P2P Ivy node Many file sharing networks Napster, Freenet Application Application DHash server Gnutella, Kazaa Bittorrent DHash server Peer-to-Peer Content Delivery Networks (P2P-CDN) Squirrel web caching service, based on Pastry Ivy server DHash server OceanStore and Ivy file stores DHash server Modifled NFS Client module Kernel DHash server Massively Multiplayer Online Games (MMOG) Centralized/P2P/Hybrid? Instructor s Guide for Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edn. 4 Pearson Education 2005 Distributed Information Systems Page 62 Research Study Conduct a brief/review research of any P2P applications or routing overlays or any other related issues. Sources: online DB IEEE, ACM, ScienceDirect, SpringerLink online DBs en.wiki.pedia.com Make a professional presentation of up to (strictly) 20 min. long per group in next month. Provide a 4-or-6-slide-per-page handouts before the presentation. Present its background, key concepts, and future work direction. Don t forget to cite your references appropriately. Topic must be different to other groups. Distributed Information Systems Page 63 (C) Chukiat Worasucheep 16