Lesson 9 Applications of DHT: Bittorrent Mainline DHT, the KAD network

Transcription

1 Lesson 9 Applications of DHT: Bittorrent Mainline DHT, the KAD network 20/03/2017 1

2 IMPLEMENTATION OF KADEMLIA Kademlia is the protocol used by the largest public DHTs. Vuze ( before known as Azureus), version introduced the DHT for trackerless clients Bittorrent Inc. introduces its own DHT; incompatible with that of Vuze. mainline DHT a recent measurement: by 2013 users of Mainline DHT are from 10 million to 25 million, with a daily churn of at least 10 million emule KAD network 2

3 THE BITTORRENT PROTOCOL (1) the first BitTorrent client was published in July 2001 by Bram Cohen. today, a wide spectrum of compatible clients is available, typically free of charge a user exploiting the BitTorrent protocol, not only download a shared file, but becomes also the distributor of the file each file is divided into parts which are exchanged between the users: each user tries to download the parts it does not own and shares the parts it already downloaded swarm: set of peers that participate in sharing and distributing a specific file at a specific time. 3

4 HOW CAN I FIND A TACKER? file are not indexed within the Bittorrent network (instead they are indexed within emule). cannot find the file descriptor by the file index retrieve the file descriptor (.torrent) by searching in the Internet, often through specialized search sites and engines: Pirate Bay ( Mininova ( contains several information, most important are the file s InfoHash: an identifier which is the SHA of some information regarding the file and uniquely identifies the file the IP addresses of one or more trackers other information (to be seen in next lessons...) 4

5 THE BITTORRENT PROTOCOL (2) a tracker: server that manages a file s swarm replies to HTTP GET request any peer that wants to join a swarm must retrieve the address of at least a tracker managing the swarm downloads the file descriptor (.torrent) and opens it through the BitTorrent client opens a connection to the tracker and informs it of its own existence obtain a list of other swarm peers from the tracker and start to download the file from them. peers exchange information about the parts of the file that they own and each peer asks for the parts of the file it is interested to. if a peer remains online when it has finished the file download, it goes on distributing the file 5

6 THE BITTORRENT PROTOCOL (6) Tracker: a centralized service for the registration and coordination of the peers knows the address of each peer of the swarm and enables the mutual discovery of the peers sends to a requesting peer P a random subset of the peers (IP+port) of the swarm. P builds an initial set of known peers then, other peers contact P because they have obtained its address from the tracker and are added to the initial set. the peers periodically contact the tracker to send statistical information: sent and received data if the number of known peers is under of a given threshold, the tracker is contacted again to find out new peers of the swarm 6

7 THE BITTORRENT PROTOCOL file popeye.mp4.torrent hosted at a (well-known) webserver the.torrent has address of tracker for file the tracker, which runs on a webserver as well, keeps track of all peers downloading file 7

8 THE BITTORRENT PROTOCOL file popeye.mp4.torrent hosted at a (well-known) webserver the.torrent has address of tracker for file the tracker, which runs on a webserver as well, keeps track of all peers downloading file 8

9 THE BITTORRENT PROTOCOL file popeye.mp4.torrent hosted at a (well-known) webserver the.torrent has address of tracker for file the tracker, which runs on a webserver as well, keeps track of all peers downloading file 9

10 EXCHANGE PROTOCOL The green blocks represent fully downloaded chunks (or initially shared), available for upload. The red blocks are the blocks currently being downloaded. Each peer can upload only two blocks at a time. 10

11 EXCHANGE PROTOCOL The green blocks represent fully downloaded chunks (or initially shared), available for upload. The red blocks are the blocks currently being downloaded. Each peer can upload only two blocks at a time. 11

12 EXCHANGE PROTOCOL The green blocks represent fully downloaded chunks (or initially shared), available for upload. The red blocks are the blocks currently being downloaded. Each peer can upload only two blocks at a time. 12

13 EXCHANGE PROTOCOL The green blocks represent fully downloaded chunks (or initially shared), available for upload. The red blocks are the blocks currently being downloaded. Each peer can upload only two blocks at a time. 13

14 EXCHANGE PROTOCOL The green blocks represent fully downloaded chunks (or initially shared), available for upload. The red blocks are the blocks currently being downloaded. Each peer can upload only two blocks at a time. 14

15 EXCHANGE PROTOCOL Several interesting characteristics choice of peers choice of pieces detection of free riders Will be shown in one of the next lessons 15

16 THE MAINLINE DHT First application of a DHT in Bittorrent was introduced by Vuze (previously Azureus) After a little, the standard Bittorrent client introduces its own DHT, the Mainline DHT, not compatible with Vuze both DHTs strongly based on Kademlia Many other clients support the Mainline DHT Torrent, BitComet, BitSpirit Mainline DHT is a BEP (BitTorrent Enhancement Proposal) subject to continous revisions Kademlia improvements routing table management look-up 16

17 THE MAINLINE DHT Now each node in the Bittorrent network implements two functionalities A "peer" is a client/server listening on a TCP port that implements the BitTorrent protocol. A "node" is a client/server listening on a UDP port implementing the distributed hash table protocol. The DHT is composed of nodes, and node store the location of peers belonging to a swarm 17

18 THE MAINLINE DHT DHT introduced to offer a trackerless peer discovery mechanism goal: to avoid the bottleneck of a single tracker for trackerless peer discovery, a DHT plays the role of the tracker and stores the file's infohash (key) the list of the peers in the swarm (value) 18

19 MAINLINE DHT: HIGH LEVEL INTERACTION get the peers currently in the swarm GETPEERS(InfoHash) register itself as a new member of the swarm, PUT(InfoHash, <IP address, port>) 19

20 THE MAINLINE DHT Node 8 wants to share a file with infohash=42 And needs to know if a swarm is active for that file 20

21 THE MAINLINE DHT Node 8 owns in its routing table a reference to node 12 and sends it the request getpeer(42), to find out peers of the swarm 21

22 THE MAINLINE DHT Node 12 does not know any peer in the swarm of the torrent with InfoHash 42, but it knows some nodes whose identifier is closer to the 42 22

23 THE MAINLINE DHT Node 8 retransmit the request GET_PEERS to nodes 48 and 40 23

24 THE MAINLINE DHT Nodes 40 e 48 have no peer in the swarm of the torrent 42, 48 resends the identifier 40 to node 8. Node 8 does not receive information on new peers and stops the search of swarm peers 24

25 THE MAINLINE DHT Node 40 stores the correspondence 42, 8, recording that peer 8 belongs to the swarm of the torrent with InfoHash 42 25

26 THE MAINLINE DHT Now peer wants to download the file identified by InfoHash 42 and has, in its routing table, a reference to 48 26

27 THE MAINLINE DHT Node 48 has no information about the swarm of the torrent 42, but it knows node 40, which ha san identifier which is closer to the InfoHash 42 27

28 THE MAINLINE DHT Node 50 propagates the GETPEER request to 40 28

29 THE MAINLINE DHT Node 40 returns peer , i.e. a peer belonging to the swarm di of the torrent 42 29

30 THE MAINLINE DHT 50 announces it partecipates to the swarm of the torrent with InfoHash 42 30

31 ROUTING TABLE MANAGEMENT buckets in rotuing tables are structured differently from those in Kademlia: instead of a list of 160 buckets, BitTorrent starts with only one bucket when a bucket becomes full, one of two things can happen: the bucket is split old nodes are pinged (like in Kademlia) replace the split bucket by two new buckets each with half the range of the old bucket nodes from the old bucket are distributed among the two new ones. Two benefits to this bucket implementation: less memory is used if there are less than 160 buckets when searching buckets, it is not necessary to retrieve additional nodes from adjacent buckets because it is guaranteed that there are enough in the current bucket 31

32 ROUTING TABLE MANAGEMENT (1) buckets are dynamically splitted (8 contacts for each bucket) initially, two buckets , , ID intervals [ ; ) and [ ; ] red dot= identifier corresponding to the owner of the routing table when the bucket B corresponding to the owner is full and another node has to be inserted, B is split 32

33 ROUTING TABLE MANAGEMENT (2) Each node N in the routing tables is characterized by a state, depending on the activity detected by owner O of the routing table in the last interval of times Three states: GOOD, if O has had a contact with N in the last 15 minutes; QUESTIONABLE, if O if no contact has happened in the last 15 minutes BAD, if the node fail to reply Periodic refresh of the state 33

34 ROUTING TABLE MANAGEMENT ALGORITHM addnode(node nnew) def Bucket bold = bucket where the new node nnew falls if ( bold is not full ) then add nnew to bold ; else if (bold contains myid ) then divide bold, creating two new buckets binf and bsup; distribute nodes in bold and the new node nnew to binf e bsup ; else if ( all nodes in bold are GOOD) then discard nnew; if ( bold includes a node n BAD ) then substitute nnew to n If ( bold includes nodes QUESTIONABLE) then managequestionablenodes(nnew,bold); 34

35 ROUTING TABLE MANAGEMENT ALGORITHM ManageQuestionableNodes (Node ngood, Bucket b) def qlist = node marked QUESTIONABLE in b. for ( Node nquestionable in qlist ) PING(nquestionable ); if (nquestionable reply) then change state of nquestionable to GOOD; remove nquestionable from qlist and examine next node; else PING(nquestionable ); if (nquestionable replies) then set the state of nquestionable to GOOD; remove nquestionable from qlist and examine next node; else remove nquestionable from b and add to ngood; 35

36 EMULE KAD KAD: a DHT based on Kademlia a real production DHT very popular, since part of emule network supported by the emule client (starting from the version 0.40) more than 1.5 millions peers connected introduced to limit the disadvantages of the emule server-based network avoid bottlenecks (30 servers manage the 90% of the network) increase the anonymity of the publishers general characteristics of the KAD protocol a Kademlia-based implementation with several updates with respect to the original proposal to deploy it on a real network 36

37 KAD PUBBLISHING MECHANISM the publication of an entire file or of some parts of a file on the DHT implies a large traffic overhead on the network. for this reason, when a peer shares some content C stores C on the local storage support does not exploit the DHT to publish C or parts of C exploits the DHT for indexing keywords- source files source files- responsible peers this mechanism is called 2-level publishing model in this way, each peer stores a part of the index and this part includes references to some files shared by other users 37

38 2-LEVEL PUBLISHING MODEL an identifier, the KadID is assigned to each peer file keyword for each content (file): First Level Information: FileID: hash of the file Source: logical identifier of the peers (in the identifier space of the DHT) which has published the file FileID and stores it Second Level Information includes meta data Keyword: hash of a keyword identifying the content FileID: content hash of files identified by that keyword 38

39 KAD KEYWORD REQUEST search a file by some keywords look-up only with first keyword in list key is hash function on this keyword query is routed to peer with Kad ID closest to this hash value. 39

40 KAD KEYWORD REQUEST peer responsible for this keyword returns different sources (files identified by that keywords), together with keywords only those files with entries that include remaining keywords of request are returned 40

41 KAD SOURCE REQUEST Peer can use this hash to find peer responsible for the file possibly many peers share the same content (same hahs) 41

42 KAD SOURCE REQUEST Peer provides requester with a list of peers storing a copy of the file 42

43 KAD DOWNLOAD Eventually, the requester can download the data from these peers. 43

44 DIFFERENCES/ADDS ON TO KADEMLIA 2-level publishing method Kademlia k-buckets are modified the binary-tree structure defining the routing table is modified to allow different buckets to have different maximum limits introduces mechanisms to avoid peer overloading management of peers behind a firewall or a NAT firewall check buddy peers high replication of references: each reference R to F is stored on a set of peers of the DHT. This set of nodes defines the tolerance zone of R. 44

45 KAD: THE EMULE INTERFACE 45

46 KAD: THE EMULE INTERFACE On the left: list of contacts contained in the routing table. For each contact: an icon (different colors identify younger/older contacts) contact identifier contact up time (a numerical code corresponds to different up-times) client type (Emule version) distance 46

47 KAD: THE EMULE INTERFACE On the right bootstrap information a chart showing the distribution of the contacts in the routing table x-axis: identifier space y-axis: number of contacts in a specific range of identifiers grey bar: area including the maximum number of contacts the identifier of the node is close to the grey bar 47

48 KAD: THE EMULE INTERFACE For each active look-up, from the first column: search type (the icons have been shown in the preceding slides) target ID file/keyword name which are looked-up search state (active/stopped) stopped: a sufficient number of replies have been received so that the search can be stopped for the moment being 48

49 KAD: THE ACTIVE LOOKS -UP Load: popularity of a published keyword: Look at the sixth column: (only for keywords): in the first field average popularity returned by the searches, if larger than 20, the keyword is popular and is no more published. in the second field (two numbers in parenthesis) number of nodes which have sent a reply sum of the loads of these nodes sent/received packets: useful as statistical values 49

50 KAD: THE INTERFACE Another interface allows to visualize in real-time the active look-ups each circle is a node the downword nodes are close to the target ID an arrow from node A to node B means that A has given information to B the colours of the nodes correspond to the type of reply 50

51 KAD: THE ACTIVE LOOKS -UP 51

52 DHT: OTHER APPLICATIONS File sharing Torrent Mainline DHT KAD network NoSQL databases Amazon Dynamo Cassandra Distributed file systems GlusterFS Content Distribution Coral CDN 52