internet technologies and standards

Transcription

1 Institute of Telecommunications Warsaw University of Technology 25 internet technologies and standards Piotr Gajowniczek Andrzej Bąk Michał Jarociński

2 Internet application layer peer-to-peer systems overview P2P overview part based on Introduction to P2P Systems slides of prof. Nalini Venkatasubramanian

3 pure P2P architecture no always-on server arbitrary end systems directly communicate, creating an overlay on network resources peers are intermittently connected and change IP addresses examples: file distribution (BitTorrent) streaming (P2P TV systems) VoIP (Skype)

4 overlay topology virtual edge TCP connection or simply a pointer to an IP address overlay maintenance periodically ping to make sure neighbor is still alive or verify aliveness while messaging if neighbor goes down, may want to establish new edge new incoming node needs to bootstrap could be a challenge under high rate of churn overlay types churn : dynamic topology and intermittent access due to node arrival and failure unstructured (e.g., new node randomly chooses existing nodes as neighbors) structured (e.g., edges arranged in restrictive structure)

5 example : BitTorrent (file sharing) created by Bram Cohen in 2 file divided into 256Kb chunks peers in a swarm send/receive file chunks seed: a peer that stores a complete file leecher: a peer that has incomplete file (and bad send/receive ratio) torrent: a file that has info about file chunks (URL of the tracker and hash function values for all chunks for integrity verification) tracker: server tracking peers participating in a swarm new peer 2 join peer list 3 tracker torrent file data request 4 web server peer Source: Jukka K. Nurminen

6 BitTorrent: requesting, and sending chunks data exchange while downloading, peer can (and should) also upload chunks to other peers churn: peers may come and go once peer becomes seed, it may (selfishly) leave or remain in the swarm requesting chunks: the principle: periodically, ask each peer for list of chunks that they have request missing chunks from peers, general rule: rarest first sending chunks: tit-for-tat send chunks to four peers from which the chunks are received at the highest rate other peers are choked (do not receive chunks) re-evaluate top 4 every secs every 3 secs: randomly select another peer, start sending chunks called optimistic unchoke newly chosen peer may join top 4

7 example 2: Skype ( VoIP) P2P components clients: skype peers connect directly to each other for VoIP call super nodes (SN): skype peers with special functions overlay network: among SNs to locate SCs login server Skype login server Skype clients (SC) supernode (SN) supernode overlay network hybrid P2P

8 P2P VoIP Skype client joins skype network by contacting SN (IP address cached) using TCP logs-in (usename, password) to centralized skype login server obtains IP address for callee from SN, SN overlay or client buddy list initiate call directly to callee or in reverse or via relay node NAT issues will be covered later Skype login server

9 P2P reliability Skype failures 27 software environment 2 overlay structure

10 example 3: P2P video two main types data driven tree-based BitTorrent-like organized in trees (single or multiple)

11 tree-based vs data-driven tree-based peers organized into trees (parentchild relationships) issues downstream data flow structure needs to be optimized... and able to reorganize without any central management loop-avoidance is required node churn may have significant impact on the structure leaf nodes do not actively participate data driven no explicit data delivery structure pull-based nodes keep partnership lists data availability information guides data flow BitTorrent-like data flow video transfer needs additional mechanisms segments need to be downloaded such that playback deadlines are met scheduling required (to fulfill realtime constraints)

12 BiToS (BitTorrent Streaming) system parameter p: probability that we will select the High Priority piece for download p can be adaptive; lower p can help in getting unique fragments but may lead to additional functions: estimation if a given piece can be downloaded in time change to Rarest First rule: from many pieces with the same rareness, select the one with closest playback time

13 CoolStreaming example no formal structure of data delivery mcache video stream split into fixed-length pieces Buffer Map describes availability of pieces at the node nodes exchange buffer maps with partners a partial list of IP addresses of active nodes kept by each node used when new node joins (source node delegates new peeer request to one of the nodes that behaves like a tracker shares some partners with new peer) peers periodically advertise their availability SCAM (Scalable Gossip Membership Protocol) message: <seq num, id, num partner, time to live> entry: <seq num, id, num partner, time to live, last update time>

14 CoolStreaming scheduling BitTorrent CoolStreaming the scheduling algorithm needs to meet two constraints: the playback deadline for each segment the heterogeneous streaming bandwidth from the partners solved by heuristic algorithm calculate the number of potential suppliers for each segment start from rarest segments if multiple suppliers, select the one with the highest bandwidth and enough available time Failure Recovery / Partnership Refinement graceful departure: departing node issues a departure message node failure: other node that detects the failure can advertise this info to others downloads can be rescheduled (using buffer maps from other partners)

15 P2P pros and cons why? why not? massive scalability autonomy: no single point of failure resilience to Denial of Service load distribution, efficient resource usage resistance to censorship hard to maintain under high rate of churn hard to assure reliability needs specific solutions for lookup procedures

16 management issue P2P network must be self-organizing. Join and leave operations must be self-managed. infrastructure is untrusted and the components are unreliable. number of faulty nodes grows linearly with system size. tolerance to failures and churn content replication, multiple paths leverage knowledge of executing application load balancing dealing with freeriders freerider : rational or selfish users who consume more than their fair share of a public resource, or shoulder less than a fair share of the costs of its production.

17 lookup issue how to effectively locate data/files/objects in a large P2P system, even if the structure of the network is unpredictable? unstructured lookup : Napster, Gnutella, Kazaa centralized lookup (Napster) centralized directory (lookup is centralized, files copied in P2P manner) fully decentralized lookup (Gnutella) flooding based (inefficient and wasteful with bandwidth) a large (linear) part of the network is covered irrespective of hits found enormous number of redundant messages all users do this in parallel: local load grows linearly with size what search protocols can we come up with in an unstructured network? controlling topology: Random walk, Degree-biased Random Walk controlling placement of objects: Replication structurizing P2P : Kazaa hybrid P2P with supernodes (Skype predecessor)

18 KaZaA: SuperNode architecture nodes that have more connection bandwidth and are more available are designated as supernodes each supernode acts as a mini-hub, tracking the content (files) and IP addresses of its descendants for each file: file name, file size, content hash, file descriptors (used for keyword matches during query) content hash: when peer A selects file at peer B, peer A sends ContentHash in HTTP request if download for a specific file fails (partially completes), ContentHash is used to search for new copy of file (automatic recovery) if file is found in multiple nodes, user can select parallel downloading HTTP byte-range header used to request different portions of the file from different nodes

19 Internet application layer peer-to-peer systems structured data storage & search

20 distributed databases in p2p systems data in p2p systems are distributed between peers problem efficient location of data items: how to quickly find a data item stored at some set of nodes in the system desirable properties of the system: self-maintained (distributed management of node churn etc.) load balanced (load distributed evenly between nodes) effective (quick search, no matter at which node we start) solution: DHT (Distributed Hash Table)

21 DHT general idea each object is identified by an ID (or key) created using a hash function (e.g. sha-) sha- turns a string of arbitrary length into fixed bit string (6 bit) large space of identifiers (2 6 unique values) each node has unique ID taken from the same namespace as objects (eg. by hashing IP address) node (one or more) stores an ID-resource pair (id, value), where the value can be the object itself the address (IP + port number) of the node where the object is located example ( Communication Breakdown (a74eb9d7b95ac677b3baa7d ad, , 8) node stores ID-resource pairs with IDs that are close to their own (in a sense of some metric)

22 example: Kademlia (used in BitTorrent) the measure of distance between two objects is given by a metric that is a bitwise XOR operation on their IDs, interpreted as an integer distance(a, b) = a XOR b for selected ID there are no two other ids with the same distance to it the metric is symmetric resource placement every node maintains information about objects with IDs close to its own node A will have the ID-resource pair if it has the largest number of significant bits that match the ID

23 Example ID = sha-(communication Breakdown) = a74eb9d7b95ac677b3baa7d ad Node A ID A = sha-( ): 67a83db c88c94976b9b4aeca492 Node B ID B = sha-( ): e6954b bdd257d3e4d5bc457bb3474 Node C ID C = sha-( ): 3afbeba427a7e962c87784a385538eba8a4882 ID A XOR ID = C6BF73F56FD77ECA87F7AF3DC8E32884DE9F ID B XOR ID = 47825DE93347E7BD52CE84E66C4DFF4E79 ID C XOR ID = 9BECA5DFA32E3A4CFCBF9F864A3CCE328F Node B is the closest (4<C and 4<9) and should store (ID, , 3465)

24 Kademlia: binary tree nodes are treated as leafs in binary tree other node s position in A s tree is determined by the longest postfix it shares with A ( logn subtrees) subtrees of interest for a node Space of 6-bit numbers Node / Peer Our node common prefix: 24 No common prefix common prefix: common prefix:

25 DHT routing principles each node maintains a set of links to other nodes (its neighbors) in a structure called a routing table neighbors are usually located at different distances, so that a partial but wide knowledge of the entire network can be held by each node a node picks its neighbors according to a certain structure, that represents the network's topology together, these "neighborhood" links form the overlay network each node either is assigned an ID which owns a given key or has a link to a node closer to this key to route a message to the owner of any key do recursively: at each step, forward the message to the neighbor whose ID is closest to k when there is no such neighbor, then the message must have arrived at the closest node, which is the owner of k

26 Kademlia routing objective: find node closest to a given ID (such as sha-(comunication Breakdown)) let ID(n) = n-th bit of ID let s start from node A with ID A that has a table R with 6 entries sha-(node A) = I.I if ID()= then go to R(2) else get R() = IP address of some node with id=xxxx and send lookup message if ID(2)= then go to R(3) else get R(2) = IP address of some node with id =XXX and send lookup message if ID(3)= then go to R(4) else get R(3) = IP address of some node with id =XXX and send lookup message... if ID(n)=I then go to R(n+) else get R(n) = IP address of some node with id =... and send lookup message OR if such node does not exist then node A is the closest

27 Kademlia routing this can be improved by keeping in R lists of k nodes from within a specific range (so called k-buckets) node sends back best a of k nodes both a and k are system-wide parameters k-bucket range: if x and y agree in the first i digits and disagree in the (i+) then 2 i d(x,y) 2 i+ - k-bucket stores k nodes that are at distance [2 i,2 i+ -] (distance in the routing table grows exponentially) k-buckets are ordered lists: least-recently used (LRU) at the end k-buckets are updated when lookups are performed such that LRU nodes are removed when node X joins, it: contacts a participating node and inserts it in the appropriate bucket performs a query for its own ID this adds the node X to many nodes lists and builds the lists of X itself

28 Kademlia: Lookup Consider a query for ID initiated by node 28 Node / Peer Our node

29 Kademlia: K-Buckets ` Consider routing table for a node with prefix Its binary tree is divided into a series of subtrees The routing table is composed of a k-bucket s corresponding to each of these subtrees Consider a 2-bucket example, each bucket will have at least 2 contacts for its key range A contact consist of <IP:Port, NodeID> Node / Peer Our node

30 routing effectiveness to store an ID-resource pair a participant locates the k nodes closest to the ID and sends them requests to store data therefore, ID lookup equals locating k nodes closest to the given ID (ends when some node returns a value) accomplished by recursive algorithm outlined earlier a search only needs to query at most 6 nodes actual complexity is O(log N) effective! what about other objectives? consistent hashing removal or addition of one node changes only the set of keys owned by the nodes with adjacent IDs effiiently supports high rates of churn is supposed to produce values that are equally spread across the ID space, so that the system is load balanced

31 DHT usage in p2p systems BitTorrent swarm with distributed tracker Skype the ID is the sha- hash of the metadata, uniquely identifying a torrent the data is a list of the peers in the swarm let each peer announce itself by looking up the 8 nodes closest to the ID of the torrent and sending an announce message to them those 8 nodes add the announcing peer to the peer list stored at that infohash a peer joins a torrent by looking up the peer list at a specific hash DHT allows effective mapping of usernames to last-known IP addresses