The emule 1 ricardo.pereira@inesc-id.pt IST 17-9-2014 1 Images from The emule Specification, Yoram Kulbak and Danny Bickson
1 High level Granularity 2 3 Queue management Part selection
emule High level Granularity emule is the most popular implementation of the edonkey 2000 protocol, which includes a few extensions The network contains a set of independent servers There isn t a single P2P network but a set of them, which may intercept Clients use both TCP and UDP according to the task being executed
emule High level Granularity Servers: Are independent, do not communicate among themselves Are the network entry point Supply clients with a list of other servers. This list is defined by the server operators Make available a file search service Supply information about peers connected to the same server sharing the same file Make possible communicating with peers behind firewalls
emule High level Granularity Clients: Share files among themselves Maintain a connection to one server using TCP May query other servers using UDP Transfer files using TCP May query other peers using UDP Share information about servers, peers and files among themselves
Network example High level Granularity
File transmission High level Granularity Files: Are divided into parts (9,28MB) The sharing unit is the full part Peers share file parts, even if they don t yet have the full file Parts are divided into chunks (180KB)
TCP connection to server and back! Clients connect to a single server, using TCP Login states GUID, TCP listening port, protocol version, user nick... Server attempts to establish back a new connection, in order to test peer s ability to receive connections If successful, server responds with an High ID (peer IP)
Clients unable to receive connections Server responds with low id. Server message contains warning text, stating limitations Servers present two limits, (soft and hard) for the number of supported clients. After reaching the soft limit, no more low ID peers are accepted
Session start Client supplies list of shared files, which it may have to update during the session Client requests list of alternative servers Server provides status information (number of clients and servers) Server sends welcome text message
Session start Server supplies list of servers Server states name, version and comments Client requests peers sharing the files it wants to download Server supplies partial list of the peers it knows
File description For each of the shared files, the client states: ID Name Size Type (audio, video, images,...) (optional) Length (opcional) Bitrate (opcional) Codec (opcional)
Other servers - UDP server state Clients communicate with other servers in order to learn their state/existence. UDP messages limited to: 10 mgs/s in total 1 msg each 5s per server Servers which do not respond x times in a row are discarded. Reply returns random value sent in the request
File search Client sends search request Server sends reply After finding the intended file, user starts download. Client requests sources Server sends stats (number of clients and files) Server sends set of sources
File search When performing a search, the user may state: Search string (with logical operators). Sent using prefix notation File type (optional) Minimum and maximum file size (optional) Availability (minimum number of known sources) (optional) File extension (optional) When it receives a set of sources, the client adds them to its list. Peers are contacted in the order they are learned
UDP Enhanced file search Client may extend the file search to several servers, using UDP May search for files or sources Sent simultaneously to several servers Server only answers if it has data to send (UDP) File search decided by user Source search automatic when known sources are inferior to 100 No more than 1 msg/s
Connecting to another peer Client identifies itself and indicates to which server it is connected emule extension indicates support for: UDP, secure ID, peer exchange, compression Only one connection between two peers Connection closed after 40s of inactivity
Connecting to low ID peers A high ID peer may, using the server, request a low ID peer to connect back. This allows NAT and firewalls to be circumvented.
Verifying a file existence Client which started connection indicates the file it wants Other responds whether it has the file and which parts it has May request list of known sources (peer exchange) Files are identified using 16B file hash (MD4) Peer may request root hash (part hashes)
Download request After confirming existence of file, peer requests download Uploading peers assigns score to request Request queued If it s not its turn to be served, it will have to wait
Download start Download may start when request reaches the top positions of the queue If peer isn t connected, uploader will connect back Client pipelines requests (up to 3 outstanding chunks) If downloader already has the file, it will cancel the request
Transmitting file parts Sending part messages carry 5 to 15KB Messages may be compressed Requests to a peer are all for the same part
UDP Periodic Download request Sent every 20 minutes Reply may be queue position Reply may be queue full Reply may be file not found
Credit system Queue management Part selection Secure identification Client uses a random 16B value as ID. Each has a RSA private/public key. Public keys are exchanged on the first connection between two peers. Secure signatures allow peers to trust each others and remember past file exchanges. Credits are accounted individually between each pair of peers Credit Value from 1 to 10, calculated as the minimum of (units of MB): uploaded total 2 downloaded total uploaded total + 2
Upload queue Queue management Part selection Number of upload slots limited to ensure a minimum of 2,4KB/s/slot Slots assigned to the top positions of the queue Clients sorted by score score = (rating queued time seconds)/100 rating = initial rating credit file priority initial rating is 0 for banned users initial rating is for friends initial rating is usually 100 initial rating is 200 during the first 15 minutes after being assigned an upload slot file priority is user defined (from 0.3 to 1.8)
Part selection Queue management Part selection The requested part is chosen according to the following criteria (decreasing importance): Part rarity (discrete: very rare, rare and common) Parts which allow previewing the file (first and last) Parts which aren t being downloaded from another source Parts whose download is closer to terminate After downloading a part, its checksum is verified. Should it fail, the peer will attempt to replace chunks sequentially until the checksum becomes valid. Chucksum is a SHA1 hash over each file part. It is called root hash
Question Queue management Part selection Any doubts?